2024-02-12T16:30:01.240 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-12T16:30:01.240 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} --verbose --name yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705/orig.config.yaml 2024-02-12T16:30:01.272 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-12T16:30:01.352 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: null first_in_suite: false job_id: '7556705' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(MON_DOWN\) - \(OSDMAP_FLAGS\) mon_bind_msgr2: false sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-tracker-64343-pacific sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 owner: scheduled_yuriw@teuthology priority: 60 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGNBgCOZxJx7VBzwEveLje+ynBZm42SIb2RE2Ir7sdFsc6o+jF9U86hq0QVYFO5P4erO08mcMnu0BKROB+gAIV8= smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoot6WVbbPiPGVwo1bMdxjp4plJ1VsA3lBdEWR3Ar2pngRTH+vnPN00VB8rjh/eAQUbaEXdJ6Zn56XVs9YpSw0= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-12_15:26:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-12T16:30:01.353 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa; will attempt to use it 2024-02-12T16:30:01.353 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks 2024-02-12T16:30:01.353 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-12T16:30:01.355 INFO:teuthology.task.internal:Checking packages... 2024-02-12T16:30:01.377 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0e714d9a4bd2a821113e6318adb87bd06cf81ec1' 2024-02-12T16:30:01.377 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-12T16:30:01.377 INFO:teuthology.packaging:ref: None 2024-02-12T16:30:01.378 INFO:teuthology.packaging:tag: None 2024-02-12T16:30:01.378 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-12T16:30:01.378 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:30:01.378 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-12T16:30:01.519 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-521.g0e714d9a 2024-02-12T16:30:01.520 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-12T16:30:01.527 INFO:teuthology.task.internal:no buildpackages task found 2024-02-12T16:30:01.527 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-12T16:30:01.561 INFO:teuthology.task.internal:Saving configuration 2024-02-12T16:30:01.573 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-12T16:30:01.619 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T16:30:01.644 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 16:25:10.148164', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGNBgCOZxJx7VBzwEveLje+ynBZm42SIb2RE2Ir7sdFsc6o+jF9U86hq0QVYFO5P4erO08mcMnu0BKROB+gAIV8='} 2024-02-12T16:30:01.664 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 16:25:10.146465', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoot6WVbbPiPGVwo1bMdxjp4plJ1VsA3lBdEWR3Ar2pngRTH+vnPN00VB8rjh/eAQUbaEXdJ6Zn56XVs9YpSw0='} 2024-02-12T16:30:01.664 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-12T16:30:01.671 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-12T16:30:01.671 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-12T16:30:01.671 INFO:teuthology.run_tasks:Running task console_log... 2024-02-12T16:30:01.761 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f489b4a2f70>, signals=[15]) 2024-02-12T16:30:01.762 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-12T16:30:01.767 INFO:teuthology.task.internal:Opening connections... 2024-02-12T16:30:01.767 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-02-12T16:30:01.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:30:01.848 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-02-12T16:30:01.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:30:01.920 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-12T16:30:01.928 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-02-12T16:30:01.952 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-02-12T16:30:01.953 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-02-12T16:30:02.014 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-02-12T16:30:02.014 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="8" 2024-02-12T16:30:02.014 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-02-12T16:30:02.014 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="8" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el8" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-12T16:30:02.015 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-12T16:30:02.016 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-02-12T16:30:02.037 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-02-12T16:30:02.060 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-02-12T16:30:02.060 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-02-12T16:30:02.121 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-02-12T16:30:02.121 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="8" 2024-02-12T16:30:02.121 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-02-12T16:30:02.121 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-02-12T16:30:02.121 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="8" 2024-02-12T16:30:02.121 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el8" 2024-02-12T16:30:02.121 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-12T16:30:02.121 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-02-12T16:30:02.122 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-12T16:30:02.122 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-02-12T16:30:02.122 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-12T16:30:02.122 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-12T16:30:02.122 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-12T16:30:02.122 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-02-12T16:30:02.146 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-12T16:30:02.155 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-12T16:30:02.219 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-12T16:30:02.219 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-02-12T16:30:02.222 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-02-12T16:30:02.243 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-12T16:30:02.248 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-12T16:30:02.248 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-02-12T16:30:02.284 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-02-12T16:30:02.316 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-12T16:30:02.368 INFO:teuthology.run_tasks:Running task kernel... 2024-02-12T16:30:02.382 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-12T16:30:02.382 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-12T16:30:02.383 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-12T16:30:02.383 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-12T16:30:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.403 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-02-12T16:30:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.405 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-02-12T16:30:02.464 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-489.el8.x86_64 2024-02-12T16:30:02.464 INFO:teuthology.task.kernel:Running kernel on smithi094: 4.18.0-489.el8.x86_64 2024-02-12T16:30:02.464 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-02-12T16:30:02.466 INFO:teuthology.orchestra.run.smithi033.stdout:4.18.0-489.el8.x86_64 2024-02-12T16:30:02.467 INFO:teuthology.task.kernel:Running kernel on smithi033: 4.18.0-489.el8.x86_64 2024-02-12T16:30:02.467 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2024-02-12T16:30:11.416 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - AppStream 4.1 MB/s | 27 MB 00:06 2024-02-12T16:30:11.878 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - AppStream 4.0 MB/s | 27 MB 00:06 2024-02-12T16:30:17.325 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-12T16:30:17.653 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:01 2024-02-12T16:30:19.425 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2024-02-12T16:30:19.795 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-02-12T16:30:20.009 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2024-02-12T16:30:20.204 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2024-02-12T16:30:20.511 INFO:teuthology.orchestra.run.smithi033.stdout:Copr repo for python3-asyncssh owned by ceph 8.7 kB/s | 3.5 kB 00:00 2024-02-12T16:30:20.513 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-02-12T16:30:21.628 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-02-12T16:30:24.350 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 16 MB 00:03 2024-02-12T16:30:25.594 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 434 kB/s | 24 kB 00:00 2024-02-12T16:30:28.147 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 454 kB/s | 24 kB 00:00 2024-02-12T16:30:28.774 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-12T16:30:28.775 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-12T16:30:28.894 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:30:28.895 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:30:28.895 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-02-12T16:30:28.895 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:30:28.895 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:30:28.895 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-12T16:30:28.895 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:30:28.895 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-12T16:30:28.896 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-12T16:30:28.896 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:30:28.896 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:30:28.896 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:30:28.896 INFO:teuthology.orchestra.run.smithi033.stdout:Install 3 Packages 2024-02-12T16:30:28.896 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:30:28.896 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 90 M 2024-02-12T16:30:28.897 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 96 M 2024-02-12T16:30:28.897 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:30:30.970 INFO:teuthology.orchestra.run.smithi033.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 5.6 MB/s | 10 MB 00:01 2024-02-12T16:30:31.144 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-12T16:30:31.145 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-12T16:30:31.247 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:30:31.248 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:30:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout:Install 3 Packages 2024-02-12T16:30:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:30:31.250 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 90 M 2024-02-12T16:30:31.250 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 96 M 2024-02-12T16:30:31.250 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:30:32.645 INFO:teuthology.orchestra.run.smithi033.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2024-02-12T16:30:33.037 INFO:teuthology.orchestra.run.smithi033.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 11 MB/s | 44 MB 00:03 2024-02-12T16:30:33.059 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:30:33.060 INFO:teuthology.orchestra.run.smithi033.stdout:Total 22 MB/s | 90 MB 00:04 2024-02-12T16:30:33.090 INFO:teuthology.orchestra.run.smithi094.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 7.9 MB/s | 10 MB 00:01 2024-02-12T16:30:33.753 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:30:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:30:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:30:34.645 INFO:teuthology.orchestra.run.smithi094.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2024-02-12T16:30:35.275 INFO:teuthology.orchestra.run.smithi094.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 13 MB/s | 44 MB 00:03 2024-02-12T16:30:35.275 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:30:35.275 INFO:teuthology.orchestra.run.smithi094.stdout:Total 22 MB/s | 90 MB 00:04 2024-02-12T16:30:35.536 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:30:35.563 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:30:35.959 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:30:36.383 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:30:36.384 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:30:37.637 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:30:37.672 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:30:38.501 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:30:39.028 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:30:40.461 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:30:40.989 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:30:42.788 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:30:43.177 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:30:44.548 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:30:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:30:46.861 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:30:47.131 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:30:48.368 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:30:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:10.347 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:10.371 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:10.578 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:10.601 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:10.729 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:10.729 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:31:10.730 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:31:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:31:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:31:11.371 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:11.371 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:31:11.371 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:31:11.372 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-12T16:31:11.372 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-12T16:31:11.372 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:31:11.372 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:31:11.437 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:31:11.437 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:31:11.438 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:31:11.438 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-12T16:31:11.438 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-12T16:31:11.438 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:31:11.438 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:31:11.613 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-02-12T16:31:11.701 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-02-12T16:31:11.868 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-12T16:31:11.869 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-12T16:31:11.869 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-12T16:31:11.869 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-12T16:31:11.869 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-12T16:31:11.869 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-12T16:31:11.869 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2024-02-12T16:31:11.869 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-02-12T16:31:11.971 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-12T16:31:11.971 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-12T16:31:11.972 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-12T16:31:11.972 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-12T16:31:11.972 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-12T16:31:11.972 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-12T16:31:11.972 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-02-12T16:31:11.972 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-02-12T16:31:12.080 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-12T16:31:12.080 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-12T16:31:12.080 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-02-12T16:31:12.113 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-02-12T16:31:12.113 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-02-12T16:31:12.113 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.02 2024-02-12T16:31:12.113 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 129.el8 2024-02-12T16:31:12.113 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Group : System Environment/Base 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 9265586 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Relocations : (not relocatable) 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : CentOS BuildSystem 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-02-12T16:31:12.114 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-02-12T16:31:12.115 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:31:12.115 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-12T16:31:12.115 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-12T16:31:12.115 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-12T16:31:12.115 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-02-12T16:31:12.115 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:31:12.115 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-02-12T16:31:12.116 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-12T16:31:12.116 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 4.18.0-536.el8.x86_64 2024-02-12T16:31:12.116 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-12T16:31:12.192 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-12T16:31:12.192 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-12T16:31:12.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.02 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 129.el8 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Group : System Environment/Base 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 9265586 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-12T16:31:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Relocations : (not relocatable) 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : CentOS BuildSystem 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:31:12.228 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-02-12T16:31:12.229 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-12T16:31:12.229 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 4.18.0-536.el8.x86_64 2024-02-12T16:31:12.229 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-12T16:31:13.807 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-02-12T16:31:13.810 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-02-12T16:31:15.636 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-02-12T16:31:15.647 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-02-12T16:31:15.676 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.WTyPrnTRCh 2024-02-12T16:31:15.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WTyPrnTRCh 2024-02-12T16:31:15.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.WTyPrnTRCh 2024-02-12T16:31:15.875 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-02-12T16:31:15.876 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-02-12T16:31:15.888 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.WTyPrnTRCh is 6KB 2024-02-12T16:31:15.908 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.G3xIxJq8Ep 2024-02-12T16:31:15.908 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.G3xIxJq8Ep 2024-02-12T16:31:15.936 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.WTyPrnTRCh 2024-02-12T16:31:15.953 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-02-12T16:31:15.980 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.G3xIxJq8Ep 2024-02-12T16:31:16.037 INFO:teuthology.orchestra.run.smithi094.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-12T16:31:16.037 INFO:teuthology.orchestra.run.smithi094.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-12T16:31:16.037 INFO:teuthology.orchestra.run.smithi094.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-12T16:31:16.038 INFO:teuthology.orchestra.run.smithi094.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-12T16:31:16.039 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-12T16:31:16.119 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.G3xIxJq8Ep is 6KB 2024-02-12T16:31:16.156 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-02-12T16:31:16.167 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.G3xIxJq8Ep 2024-02-12T16:31:16.185 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-02-12T16:31:16.261 INFO:teuthology.orchestra.run.smithi033.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-12T16:31:16.261 INFO:teuthology.orchestra.run.smithi033.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-12T16:31:16.261 INFO:teuthology.orchestra.run.smithi033.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-12T16:31:16.261 INFO:teuthology.orchestra.run.smithi033.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-12T16:31:16.262 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-12T16:31:16.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-02-12T16:31:46.186 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-12T16:31:46.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-02-12T16:31:46.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:31:46.416 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-12T16:31:46.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-02-12T16:31:46.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:32:04.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-02-12T16:32:04.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-02-12T16:32:13.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-02-12T16:32:13.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:32:13.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-02-12T16:32:13.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:32:32.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-02-12T16:32:44.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-02-12T16:32:44.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:32:44.699 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-02-12T16:32:45.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-02-12T16:32:45.184 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-536.el8.x86_64"... 2024-02-12T16:32:45.184 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-02-12T16:32:45.202 INFO:teuthology.orchestra.run.smithi033.stdout:4.18.0-536.el8.x86_64 2024-02-12T16:32:45.202 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-12T16:32:45.202 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-12T16:32:45.202 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-12T16:32:46.203 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-12T16:32:46.204 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-12T16:32:46.278 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2024-02-12T16:32:46.296 DEBUG:teuthology.parallel:result is None 2024-02-12T16:33:13.675 DEBUG:teuthology.orchestra.remote:timed out 2024-02-12T16:33:25.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-02-12T16:33:25.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:33:26.082 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-02-12T16:33:26.357 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-02-12T16:33:26.357 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-536.el8.x86_64"... 2024-02-12T16:33:26.357 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-02-12T16:33:26.377 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-536.el8.x86_64 2024-02-12T16:33:26.377 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-12T16:33:26.377 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-12T16:33:26.377 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-12T16:33:27.378 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-12T16:33:27.378 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-12T16:33:27.454 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-02-12T16:33:27.466 DEBUG:teuthology.parallel:result is None 2024-02-12T16:33:27.467 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-12T16:33:27.474 INFO:teuthology.task.internal:Creating test directory... 2024-02-12T16:33:27.474 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-12T16:33:27.478 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-12T16:33:27.500 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-12T16:33:27.524 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-12T16:33:27.555 INFO:teuthology.task.internal:Creating archive directory... 2024-02-12T16:33:27.555 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-12T16:33:27.558 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-12T16:33:27.597 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-12T16:33:27.604 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-12T16:33:27.604 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-12T16:33:27.636 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-12T16:33:27.673 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T16:33:27.682 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T16:33:27.683 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T16:33:27.694 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T16:33:27.697 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-12T16:33:27.723 INFO:teuthology.task.internal:Configuring sudo... 2024-02-12T16:33:27.723 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-12T16:33:27.728 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-12T16:33:27.759 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-12T16:33:27.798 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-12T16:33:27.798 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-12T16:33:27.802 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-12T16:33:27.822 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T16:33:27.909 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T16:33:27.965 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T16:33:28.049 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T16:33:28.097 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:33:28.097 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-12T16:33:28.166 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T16:33:28.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T16:33:28.281 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T16:33:28.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T16:33:28.368 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:33:28.368 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-12T16:33:28.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-02-12T16:33:28.436 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-02-12T16:33:28.516 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-12T16:33:28.528 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-12T16:33:28.792 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-12T16:33:28.800 INFO:teuthology.task.internal:Starting timer... 2024-02-12T16:33:28.800 INFO:teuthology.run_tasks:Running task pcp... 2024-02-12T16:33:28.862 INFO:teuthology.run_tasks:Running task selinux... 2024-02-12T16:33:28.898 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-12T16:33:28.898 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-02-12T16:33:29.008 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-02-12T16:33:29.010 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-02-12T16:33:29.074 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-02-12T16:33:29.075 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-12T16:33:29.075 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-02-12T16:33:29.130 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-02-12T16:33:29.130 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-02-12T16:33:29.154 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-02-12T16:33:29.154 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi033.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi094.front.sepia.ceph.com': 'permissive'} 2024-02-12T16:33:29.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-12T16:33:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.184 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-12T16:33:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.220 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-12T16:33:29.220 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-02-12T16:33:29.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-02-12T16:33:29.288 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-12T16:33:29.297 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-12T16:33:29.316 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-12T16:33:29.317 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi094.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-12T16:39:26.751 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi094.front.sepia.ceph.com')] 2024-02-12T16:39:26.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-02-12T16:39:26.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:39:26.833 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-02-12T16:39:26.904 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-02-12T16:39:26.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-02-12T16:39:26.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:39:26.977 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-02-12T16:39:27.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-02-12T16:39:27.070 INFO:teuthology.run_tasks:Running task clock... 2024-02-12T16:39:27.082 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-12T16:39:27.082 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-12T16:39:27.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T16:39:27.085 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-12T16:39:27.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T16:39:27.118 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-12T16:39:27.124 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-12T16:39:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-12T16:39:27.138 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-12T16:39:27.203 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-02-12T16:39:27.214 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-02-12T16:39:27.215 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-02-12T16:39:27.226 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-02-12T16:39:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-12T16:39:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-12T16:39:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-12T16:39:27.254 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-12T16:39:27.306 INFO:teuthology.orchestra.run.smithi094.stderr:bash: ntpq: command not found 2024-02-12T16:39:27.310 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-12T16:39:27.310 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-02-12T16:39:27.310 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:39:27.310 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:39:27.310 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:39:27.310 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:39:27.324 INFO:teuthology.orchestra.run.smithi033.stderr:bash: ntpq: command not found 2024-02-12T16:39:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-12T16:39:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-02-12T16:39:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:39:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:39:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:39:27.328 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:39:27.329 INFO:teuthology.run_tasks:Running task pexec... 2024-02-12T16:39:27.340 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-12T16:39:27.341 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi033.front.sepia.ceph.com 2024-02-12T16:39:27.341 DEBUG:teuthology.orchestra.run.smithi033:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-12T16:39:27.341 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi094.front.sepia.ceph.com 2024-02-12T16:39:27.341 DEBUG:teuthology.orchestra.run.smithi094:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-12T16:39:28.129 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:32 ago on Mon 12 Feb 2024 04:37:55 PM UTC. 2024-02-12T16:39:28.188 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:25 ago on Mon 12 Feb 2024 04:38:03 PM UTC. 2024-02-12T16:39:30.111 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:39:30.111 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:39:30.112 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-02-12T16:39:30.112 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:39:30.112 INFO:teuthology.orchestra.run.smithi094.stdout:Resetting modules: 2024-02-12T16:39:30.112 INFO:teuthology.orchestra.run.smithi094.stdout: container-tools 2024-02-12T16:39:30.112 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:39:30.112 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:39:30.112 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:39:30.112 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:39:30.298 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout:Resetting modules: 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout: container-tools 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:39:30.299 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:39:30.368 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:39:30.573 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:39:31.203 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:36 ago on Mon 12 Feb 2024 04:37:55 PM UTC. 2024-02-12T16:39:31.444 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:28 ago on Mon 12 Feb 2024 04:38:03 PM UTC. 2024-02-12T16:39:33.158 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:39:33.193 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:39:33.193 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:39:33.193 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:39:33.193 INFO:teuthology.orchestra.run.smithi094.stdout:Installing group/module packages: 2024-02-12T16:39:33.193 INFO:teuthology.orchestra.run.smithi094.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-12T16:39:33.193 INFO:teuthology.orchestra.run.smithi094.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout:Installing module profiles: 2024-02-12T16:39:33.194 INFO:teuthology.orchestra.run.smithi094.stdout: container-tools/common 2024-02-12T16:39:33.195 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:39:33.195 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling module streams: 2024-02-12T16:39:33.195 INFO:teuthology.orchestra.run.smithi094.stdout: container-tools rhel8 2024-02-12T16:39:33.195 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:39:33.195 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:39:33.195 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:39:33.195 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-02-12T16:39:33.195 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:39:33.196 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 27 M 2024-02-12T16:39:33.196 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 91 M 2024-02-12T16:39:33.196 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:39:33.559 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:39:33.561 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): aardvark-dns-1.9.0-1.module_el8+804+f13 5.9 MB/s | 1.0 MB 00:00 2024-02-12T16:39:33.595 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): cockpit-podman-82-1.module_el8+804+f131 3.6 MB/s | 759 kB 00:00 2024-02-12T16:39:33.597 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:39:33.597 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:39:33.597 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:39:33.597 INFO:teuthology.orchestra.run.smithi033.stdout:Installing group/module packages: 2024-02-12T16:39:33.597 INFO:teuthology.orchestra.run.smithi033.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-12T16:39:33.597 INFO:teuthology.orchestra.run.smithi033.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-12T16:39:33.597 INFO:teuthology.orchestra.run.smithi033.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-12T16:39:33.597 INFO:teuthology.orchestra.run.smithi033.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout:Installing module profiles: 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: container-tools/common 2024-02-12T16:39:33.598 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:39:33.599 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling module streams: 2024-02-12T16:39:33.599 INFO:teuthology.orchestra.run.smithi033.stdout: container-tools rhel8 2024-02-12T16:39:33.599 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:39:33.599 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:39:33.599 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:39:33.599 INFO:teuthology.orchestra.run.smithi033.stdout:Install 13 Packages 2024-02-12T16:39:33.599 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:39:33.599 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 27 M 2024-02-12T16:39:33.600 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 91 M 2024-02-12T16:39:33.600 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:39:33.620 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 649 kB/s | 38 kB 00:00 2024-02-12T16:39:33.645 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 4.8 MB/s | 248 kB 00:00 2024-02-12T16:39:33.696 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 3.5 MB/s | 179 kB 00:00 2024-02-12T16:39:33.738 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 595 kB/s | 25 kB 00:00 2024-02-12T16:39:33.788 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2024-02-12T16:39:33.855 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 17 MB/s | 4.0 MB 00:00 2024-02-12T16:39:34.022 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): toolbox-0.0.99.4-8.module_el8+804+f1313 15 MB/s | 2.5 MB 00:00 2024-02-12T16:39:34.106 INFO:teuthology.orchestra.run.smithi033.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 2.4 MB/s | 759 kB 00:00 2024-02-12T16:39:34.139 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): buildah-1.34.0-1.module_el8+804+f13139 13 MB/s | 9.7 MB 00:00 2024-02-12T16:39:34.139 INFO:teuthology.orchestra.run.smithi033.stdout:(2/13): aardvark-dns-1.9.0-1.module_el8+804+f13 2.9 MB/s | 1.0 MB 00:00 2024-02-12T16:39:34.156 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 395 kB/s | 53 kB 00:00 2024-02-12T16:39:34.156 INFO:teuthology.orchestra.run.smithi033.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 756 kB/s | 38 kB 00:00 2024-02-12T16:39:34.181 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 979 kB/s | 41 kB 00:00 2024-02-12T16:39:34.198 INFO:teuthology.orchestra.run.smithi033.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 4.1 MB/s | 248 kB 00:00 2024-02-12T16:39:34.265 INFO:teuthology.orchestra.run.smithi033.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 2.6 MB/s | 179 kB 00:00 2024-02-12T16:39:34.299 INFO:teuthology.orchestra.run.smithi033.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 743 kB/s | 25 kB 00:00 2024-02-12T16:39:34.333 INFO:teuthology.orchestra.run.smithi033.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 2.8 MB/s | 94 kB 00:00 2024-02-12T16:39:34.348 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 16 MB/s | 8.8 MB 00:00 2024-02-12T16:39:34.348 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:39:34.348 INFO:teuthology.orchestra.run.smithi094.stdout:Total 24 MB/s | 27 MB 00:01 2024-02-12T16:39:34.616 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:39:34.633 INFO:teuthology.orchestra.run.smithi033.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 8.3 MB/s | 4.0 MB 00:00 2024-02-12T16:39:34.637 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:39:34.637 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:39:34.742 INFO:teuthology.orchestra.run.smithi033.stdout:(9/13): buildah-1.34.0-1.module_el8+804+f131391 10 MB/s | 9.7 MB 00:00 2024-02-12T16:39:34.792 INFO:teuthology.orchestra.run.smithi033.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 1.0 MB/s | 53 kB 00:00 2024-02-12T16:39:34.834 INFO:teuthology.orchestra.run.smithi033.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 984 kB/s | 41 kB 00:00 2024-02-12T16:39:34.881 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:39:34.884 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:39:34.892 INFO:teuthology.orchestra.run.smithi033.stdout:(12/13): toolbox-0.0.99.4-8.module_el8+804+f131 9.7 MB/s | 2.5 MB 00:00 2024-02-12T16:39:35.134 INFO:teuthology.orchestra.run.smithi033.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 11 MB/s | 8.8 MB 00:00 2024-02-12T16:39:35.135 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:39:35.135 INFO:teuthology.orchestra.run.smithi033.stdout:Total 18 MB/s | 27 MB 00:01 2024-02-12T16:39:35.411 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:39:35.434 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:39:35.434 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:39:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:39:35.707 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:39:35.710 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:39:35.792 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-12T16:39:35.970 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-12T16:39:36.179 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-12T16:39:36.406 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-12T16:39:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:39:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-12T16:39:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-12T16:39:36.960 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-12T16:39:37.014 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-12T16:39:37.281 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-12T16:39:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-12T16:39:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-12T16:39:38.181 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-12T16:39:38.377 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-12T16:39:38.641 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-12T16:39:38.872 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-12T16:39:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-12T16:39:39.133 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-12T16:39:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-12T16:39:39.594 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-12T16:39:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-12T16:39:40.029 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-12T16:39:40.267 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-12T16:39:40.452 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-12T16:39:40.872 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-12T16:39:40.872 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-12T16:39:40.872 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-12T16:39:40.872 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-12T16:39:40.872 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-12T16:39:40.872 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-12T16:39:40.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-12T16:39:40.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-12T16:39:40.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-12T16:39:40.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-12T16:39:40.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-12T16:39:40.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-12T16:39:40.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-12T16:39:41.025 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-12T16:39:41.264 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-12T16:39:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-12T16:39:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:39:41.365 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:39:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:39:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:39:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:39:41.366 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:39:41.471 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-12T16:39:41.659 DEBUG:teuthology.parallel:result is None 2024-02-12T16:39:41.925 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-12T16:39:41.926 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-12T16:39:41.927 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-12T16:39:42.423 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-12T16:39:42.435 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:39:42.435 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:39:42.435 INFO:teuthology.orchestra.run.smithi033.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:39:42.435 INFO:teuthology.orchestra.run.smithi033.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:39:42.435 INFO:teuthology.orchestra.run.smithi033.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-12T16:39:42.435 INFO:teuthology.orchestra.run.smithi033.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-12T16:39:42.435 INFO:teuthology.orchestra.run.smithi033.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-12T16:39:42.435 INFO:teuthology.orchestra.run.smithi033.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:39:42.436 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:39:42.749 DEBUG:teuthology.parallel:result is None 2024-02-12T16:39:42.749 INFO:teuthology.run_tasks:Running task install... 2024-02-12T16:39:42.759 DEBUG:teuthology.task.install:project ceph 2024-02-12T16:39:42.759 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}} 2024-02-12T16:39:42.759 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-12T16:39:42.760 INFO:teuthology.task.install:Using flavor: default 2024-02-12T16:39:42.766 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-12T16:39:42.766 INFO:teuthology.task.install:extra packages: [] 2024-02-12T16:39:42.767 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': None, 'wait_for_package': False} 2024-02-12T16:39:42.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:39:42.769 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': None, 'wait_for_package': False} 2024-02-12T16:39:42.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:39:43.034 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/ 2024-02-12T16:39:43.034 INFO:teuthology.task.install.rpm:Package version is 16.2.14-521.g0e714d9a 2024-02-12T16:39:43.047 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/ 2024-02-12T16:39:43.047 INFO:teuthology.task.install.rpm:Package version is 16.2.14-521.g0e714d9a 2024-02-12T16:39:43.159 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-12T16:39:43.159 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:39:43.159 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-12T16:39:43.161 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-12T16:39:43.162 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:39:43.162 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-12T16:39:43.191 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-12T16:39:43.192 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-12T16:39:43.196 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-12T16:39:43.196 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-12T16:39:43.260 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-12T16:39:43.267 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-12T16:39:43.338 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-12T16:39:43.348 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-12T16:39:43.405 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-02-12T16:39:43.406 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-02-12T16:39:43.417 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-02-12T16:39:43.418 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-02-12T16:39:43.685 INFO:teuthology.orchestra.run.smithi033.stdout:76 files removed 2024-02-12T16:39:43.686 INFO:teuthology.orchestra.run.smithi094.stdout:76 files removed 2024-02-12T16:39:43.714 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw 2024-02-12T16:39:43.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-02-12T16:39:44.451 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 235 kB/s | 76 kB 00:00 2024-02-12T16:39:44.528 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 199 kB/s | 76 kB 00:00 2024-02-12T16:39:44.768 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 55 kB/s | 15 kB 00:00 2024-02-12T16:39:44.912 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 45 kB/s | 15 kB 00:00 2024-02-12T16:39:45.194 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 3.6 kB/s | 1.4 kB 00:00 2024-02-12T16:39:45.405 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 3.1 kB/s | 1.4 kB 00:00 2024-02-12T16:39:51.153 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - AppStream 4.5 MB/s | 27 MB 00:05 2024-02-12T16:39:53.054 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - AppStream 3.5 MB/s | 27 MB 00:07 2024-02-12T16:39:57.670 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - BaseOS 6.5 MB/s | 10 MB 00:01 2024-02-12T16:39:58.997 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-12T16:39:59.829 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-02-12T16:40:00.263 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2024-02-12T16:40:00.931 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - PowerTools 8.2 MB/s | 4.8 MB 00:00 2024-02-12T16:40:01.222 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-02-12T16:40:01.823 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2024-02-12T16:40:02.699 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 8 - PowerTools 6.1 MB/s | 4.8 MB 00:00 2024-02-12T16:40:03.391 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 2.2 kB/s | 3.5 kB 00:01 2024-02-12T16:40:03.951 INFO:teuthology.orchestra.run.smithi033.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-02-12T16:40:04.642 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-02-12T16:40:07.829 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 16 MB 00:03 2024-02-12T16:40:08.959 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 42 kB/s | 24 kB 00:00 2024-02-12T16:40:11.877 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 463 kB/s | 24 kB 00:00 2024-02-12T16:40:12.471 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:40:12.473 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:40:12.473 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:40:12.473 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 12 M 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 4.2 M 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.1 M 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 24 M 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 26 k 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-12T16:40:12.474 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 825 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 494 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.8 M 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 45 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 99 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 215 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-12T16:40:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 389 k 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 369 k 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 114 k 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout:Install 21 Packages 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:40:12.476 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 58 M 2024-02-12T16:40:12.477 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:40:13.652 INFO:teuthology.orchestra.run.smithi094.stdout:(1/23): ceph-base-16.2.14-521.g0e714d9a.el8.x86 7.7 MB/s | 6.1 MB 00:00 2024-02-12T16:40:13.711 INFO:teuthology.orchestra.run.smithi094.stdout:(2/23): ceph-selinux-16.2.14-521.g0e714d9a.el8. 440 kB/s | 26 kB 00:00 2024-02-12T16:40:13.851 INFO:teuthology.orchestra.run.smithi094.stdout:(3/23): libcephfs2-16.2.14-521.g0e714d9a.el8.x8 5.8 MB/s | 825 kB 00:00 2024-02-12T16:40:14.097 INFO:teuthology.orchestra.run.smithi094.stdout:(4/23): libradosstriper1-16.2.14-521.g0e714d9a. 2.0 MB/s | 494 kB 00:00 2024-02-12T16:40:14.613 INFO:teuthology.orchestra.run.smithi094.stdout:(5/23): librgw2-16.2.14-521.g0e714d9a.el8.x86_6 7.4 MB/s | 3.8 MB 00:00 2024-02-12T16:40:14.880 INFO:teuthology.orchestra.run.smithi094.stdout:(6/23): ceph-common-16.2.14-521.g0e714d9a.el8.x 12 MB/s | 24 MB 00:02 2024-02-12T16:40:14.905 INFO:teuthology.orchestra.run.smithi094.stdout:(7/23): python3-ceph-argparse-16.2.14-521.g0e71 154 kB/s | 45 kB 00:00 2024-02-12T16:40:14.939 INFO:teuthology.orchestra.run.smithi094.stdout:(8/23): python3-ceph-common-16.2.14-521.g0e714d 1.7 MB/s | 99 kB 00:00 2024-02-12T16:40:14.956 INFO:teuthology.orchestra.run.smithi094.stdout:(9/23): python3-cephfs-16.2.14-521.g0e714d9a.el 4.2 MB/s | 215 kB 00:00 2024-02-12T16:40:15.014 INFO:teuthology.orchestra.run.smithi094.stdout:(10/23): python3-rados-16.2.14-521.g0e714d9a.el 5.1 MB/s | 389 kB 00:00 2024-02-12T16:40:15.040 INFO:teuthology.orchestra.run.smithi094.stdout:(11/23): python3-rbd-16.2.14-521.g0e714d9a.el8. 4.3 MB/s | 369 kB 00:00 2024-02-12T16:40:15.067 INFO:teuthology.orchestra.run.smithi094.stdout:(12/23): python3-rgw-16.2.14-521.g0e714d9a.el8. 2.2 MB/s | 114 kB 00:00 2024-02-12T16:40:15.207 INFO:teuthology.orchestra.run.smithi094.stdout:(13/23): ceph-radosgw-16.2.14-521.g0e714d9a.el8 4.9 MB/s | 12 MB 00:02 2024-02-12T16:40:15.274 INFO:teuthology.orchestra.run.smithi094.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.8 MB/s | 677 kB 00:00 2024-02-12T16:40:15.299 INFO:teuthology.orchestra.run.smithi094.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 479 kB/s | 44 kB 00:00 2024-02-12T16:40:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-02-12T16:40:15.525 INFO:teuthology.orchestra.run.smithi094.stdout:(17/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 188 kB/s | 47 kB 00:00 2024-02-12T16:40:15.584 INFO:teuthology.orchestra.run.smithi094.stdout:(18/23): leveldb-1.22-1.el8.x86_64.rpm 699 kB/s | 181 kB 00:00 2024-02-12T16:40:15.609 INFO:teuthology.orchestra.run.smithi094.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 990 kB/s | 306 kB 00:00 2024-02-12T16:40:15.659 INFO:teuthology.orchestra.run.smithi094.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 999 kB/s | 75 kB 00:00 2024-02-12T16:40:15.743 INFO:teuthology.orchestra.run.smithi094.stdout:(21/23): liboath-2.6.2-3.el8.x86_64.rpm 270 kB/s | 59 kB 00:00 2024-02-12T16:40:15.861 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:40:15.864 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:40:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:40:15.864 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:40:15.864 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:40:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 12 M 2024-02-12T16:40:15.864 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-02-12T16:40:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-12T16:40:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 4.2 M 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.1 M 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 24 M 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 26 k 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 825 k 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 494 k 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-12T16:40:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.8 M 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 45 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 99 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 215 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 389 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 369 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 114 k 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout:Install 21 Packages 2024-02-12T16:40:15.866 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 2 Packages 2024-02-12T16:40:15.867 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:40:15.867 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 58 M 2024-02-12T16:40:15.867 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:40:15.951 INFO:teuthology.orchestra.run.smithi094.stdout:(22/23): librados2-16.2.14-521.g0e714d9a.el8.x8 11 MB/s | 3.7 MB 00:00 2024-02-12T16:40:16.018 INFO:teuthology.orchestra.run.smithi094.stdout:(23/23): librbd1-16.2.14-521.g0e714d9a.el8.x86_ 12 MB/s | 4.2 MB 00:00 2024-02-12T16:40:16.020 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:40:16.020 INFO:teuthology.orchestra.run.smithi094.stdout:Total 16 MB/s | 58 MB 00:03 2024-02-12T16:40:16.065 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:40:16.111 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:40:16.111 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:40:16.467 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:40:16.470 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:40:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:40:17.679 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-12T16:40:17.857 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-12T16:40:18.261 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-12T16:40:18.491 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-12T16:40:18.570 INFO:teuthology.orchestra.run.smithi033.stdout:(1/23): ceph-base-16.2.14-521.g0e714d9a.el8.x86 2.4 MB/s | 6.1 MB 00:02 2024-02-12T16:40:18.585 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-12T16:40:18.703 INFO:teuthology.orchestra.run.smithi033.stdout:(2/23): ceph-radosgw-16.2.14-521.g0e714d9a.el8. 4.3 MB/s | 12 MB 00:02 2024-02-12T16:40:18.729 INFO:teuthology.orchestra.run.smithi033.stdout:(3/23): ceph-selinux-16.2.14-521.g0e714d9a.el8. 162 kB/s | 26 kB 00:00 2024-02-12T16:40:18.835 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-12T16:40:18.837 INFO:teuthology.orchestra.run.smithi033.stdout:(4/23): libradosstriper1-16.2.14-521.g0e714d9a. 4.5 MB/s | 494 kB 00:00 2024-02-12T16:40:18.871 INFO:teuthology.orchestra.run.smithi033.stdout:(5/23): libcephfs2-16.2.14-521.g0e714d9a.el8.x8 4.8 MB/s | 825 kB 00:00 2024-02-12T16:40:18.963 INFO:teuthology.orchestra.run.smithi033.stdout:(6/23): python3-ceph-argparse-16.2.14-521.g0e71 491 kB/s | 45 kB 00:00 2024-02-12T16:40:19.032 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-12T16:40:19.038 INFO:teuthology.orchestra.run.smithi033.stdout:(7/23): python3-ceph-common-16.2.14-521.g0e714d 1.3 MB/s | 99 kB 00:00 2024-02-12T16:40:19.122 INFO:teuthology.orchestra.run.smithi033.stdout:(8/23): python3-cephfs-16.2.14-521.g0e714d9a.el 2.5 MB/s | 215 kB 00:00 2024-02-12T16:40:19.264 INFO:teuthology.orchestra.run.smithi033.stdout:(9/23): python3-rados-16.2.14-521.g0e714d9a.el8 2.7 MB/s | 389 kB 00:00 2024-02-12T16:40:19.373 INFO:teuthology.orchestra.run.smithi033.stdout:(10/23): python3-rbd-16.2.14-521.g0e714d9a.el8. 3.3 MB/s | 369 kB 00:00 2024-02-12T16:40:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-12T16:40:19.456 INFO:teuthology.orchestra.run.smithi033.stdout:(11/23): librgw2-16.2.14-521.g0e714d9a.el8.x86_ 6.1 MB/s | 3.8 MB 00:00 2024-02-12T16:40:19.482 INFO:teuthology.orchestra.run.smithi033.stdout:(12/23): python3-rgw-16.2.14-521.g0e714d9a.el8. 1.0 MB/s | 114 kB 00:00 2024-02-12T16:40:19.632 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-12T16:40:19.691 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-12T16:40:19.715 INFO:teuthology.orchestra.run.smithi033.stdout:(13/23): ceph-common-16.2.14-521.g0e714d9a.el8. 6.5 MB/s | 24 MB 00:03 2024-02-12T16:40:19.866 INFO:teuthology.orchestra.run.smithi033.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 293 kB/s | 44 kB 00:00 2024-02-12T16:40:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-12T16:40:19.916 INFO:teuthology.orchestra.run.smithi033.stdout:(15/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.4 MB/s | 677 kB 00:00 2024-02-12T16:40:19.941 INFO:teuthology.orchestra.run.smithi033.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 565 kB/s | 259 kB 00:00 2024-02-12T16:40:20.135 INFO:teuthology.orchestra.run.smithi033.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 944 kB/s | 181 kB 00:00 2024-02-12T16:40:20.150 INFO:teuthology.orchestra.run.smithi033.stdout:(18/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 166 kB/s | 47 kB 00:00 2024-02-12T16:40:20.176 INFO:teuthology.orchestra.run.smithi033.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-02-12T16:40:20.201 INFO:teuthology.orchestra.run.smithi033.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 871 kB/s | 59 kB 00:00 2024-02-12T16:40:20.310 INFO:teuthology.orchestra.run.smithi033.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 471 kB/s | 75 kB 00:00 2024-02-12T16:40:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-12T16:40:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-12T16:40:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-12T16:40:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-12T16:40:20.893 INFO:teuthology.orchestra.run.smithi033.stdout:(22/23): librbd1-16.2.14-521.g0e714d9a.el8.x86_ 6.0 MB/s | 4.2 MB 00:00 2024-02-12T16:40:20.952 INFO:teuthology.orchestra.run.smithi033.stdout:(23/23): librados2-16.2.14-521.g0e714d9a.el8.x8 4.8 MB/s | 3.7 MB 00:00 2024-02-12T16:40:20.954 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:40:20.955 INFO:teuthology.orchestra.run.smithi033.stdout:Total 11 MB/s | 58 MB 00:05 2024-02-12T16:40:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 11/25 2024-02-12T16:40:20.998 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:40:21.050 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:40:21.051 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:40:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-12T16:40:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-12T16:40:21.441 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:40:21.444 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:40:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 14/25 2024-02-12T16:40:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-12T16:40:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-12T16:40:21.999 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-12T16:40:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-12T16:40:22.333 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-12T16:40:22.495 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:40:22.524 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 19/25 2024-02-12T16:40:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-12T16:40:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-12T16:40:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-12T16:40:23.490 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-12T16:40:23.587 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-12T16:40:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-12T16:40:24.015 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-12T16:40:24.453 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-12T16:40:24.656 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-12T16:40:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-12T16:40:24.944 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-12T16:40:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-12T16:40:25.223 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-12T16:40:25.467 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-12T16:40:25.639 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-12T16:40:25.723 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-12T16:40:25.925 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-12T16:40:25.937 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-12T16:40:26.138 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 11/25 2024-02-12T16:40:26.197 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-12T16:40:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-12T16:40:26.336 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-12T16:40:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-12T16:40:26.536 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-12T16:40:26.768 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 14/25 2024-02-12T16:40:26.897 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-12T16:40:26.978 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-12T16:40:27.227 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-12T16:40:27.415 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-12T16:40:27.624 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-12T16:40:27.790 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 19/25 2024-02-12T16:40:30.462 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-12T16:40:30.706 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-12T16:40:31.639 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-12T16:40:31.937 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-12T16:40:32.052 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-12T16:40:32.197 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-12T16:40:45.274 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /dev 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /run 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /dev 2024-02-12T16:40:45.275 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:40:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-12T16:40:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-12T16:40:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-12T16:40:46.723 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-12T16:40:46.723 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-12T16:40:47.388 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-12T16:40:47.388 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 1/25 2024-02-12T16:40:47.388 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2/25 2024-02-12T16:40:47.388 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-12T16:40:47.388 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-12T16:40:47.388 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-12T16:40:47.388 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 6/25 2024-02-12T16:40:47.388 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 7/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 8/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 9/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 10/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 11/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-12T16:40:47.389 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-12T16:40:47.390 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-12T16:40:47.390 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-12T16:40:47.390 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-12T16:40:47.390 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 24/25 2024-02-12T16:40:48.279 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-12T16:40:48.279 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:40:48.279 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-02-12T16:40:48.279 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: leveldb-1.22-1.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.280 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:40:48.281 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:40:48.594 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-02-12T16:40:49.345 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:41 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:40:50.862 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:40:50.863 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:40:50.863 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:40:50.863 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:40:50.863 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 52 M 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 174 k 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-02-12T16:40:50.864 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:40:50.865 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 53 M 2024-02-12T16:40:50.865 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 197 M 2024-02-12T16:40:50.865 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:40:51.091 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 1.3 MB/s | 203 kB 00:00 2024-02-12T16:40:51.132 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2024-02-12T16:40:51.191 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-02-12T16:40:51.233 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-02-12T16:40:51.296 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): libcephsqlite-16.2.14-521.g0e714d9a.el8. 488 kB/s | 174 kB 00:00 2024-02-12T16:40:52.213 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /dev 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /run 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /dev 2024-02-12T16:40:52.214 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:40:53.458 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-12T16:40:53.535 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-12T16:40:53.535 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-12T16:40:53.667 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): ceph-test-16.2.14-521.g0e714d9a.el8.x86_ 19 MB/s | 52 MB 00:02 2024-02-12T16:40:53.669 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:40:53.669 INFO:teuthology.orchestra.run.smithi094.stdout:Total 19 MB/s | 53 MB 00:02 2024-02-12T16:40:53.711 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:40:53.712 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-12T16:40:53.712 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-12T16:40:53.732 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:40:53.732 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:40:54.021 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:40:54.024 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:40:54.392 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 1/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 6/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 7/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 8/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 9/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 10/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 11/25 2024-02-12T16:40:54.393 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-12T16:40:54.394 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 24/25 2024-02-12T16:40:54.514 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:40:54.755 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-12T16:40:54.948 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-12T16:40:55.078 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-12T16:40:55.138 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-12T16:40:55.285 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-12T16:40:55.285 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: leveldb-1.22-1.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-12T16:40:55.286 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:40:55.287 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:40:55.325 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-12T16:40:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-12T16:40:55.557 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-test 2024-02-12T16:40:56.341 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:00:45 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:40:57.963 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:40:57.964 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:40:57.964 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:40:57.964 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:40:57.964 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:40:57.964 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 52 M 2024-02-12T16:40:57.964 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:40:57.964 INFO:teuthology.orchestra.run.smithi033.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 174 k 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2024-02-12T16:40:57.965 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:40:57.966 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 53 M 2024-02-12T16:40:57.966 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 197 M 2024-02-12T16:40:57.966 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:40:58.555 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 577 kB/s | 203 kB 00:00 2024-02-12T16:40:58.701 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): libcephsqlite-16.2.14-521.g0e714d9a.el8. 349 kB/s | 174 kB 00:00 2024-02-12T16:40:58.726 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-02-12T16:40:58.926 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 353 kB/s | 71 kB 00:00 2024-02-12T16:40:59.010 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.0 MB/s | 323 kB 00:00 2024-02-12T16:41:00.103 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-12T16:41:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-12T16:41:00.599 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-12T16:41:00.599 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 1/6 2024-02-12T16:41:00.599 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2/6 2024-02-12T16:41:00.600 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-12T16:41:00.600 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-12T16:41:00.600 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-12T16:41:01.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout: jq-1.6-8.el8.x86_64 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:01.050 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:41:01.143 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): ceph-test-16.2.14-521.g0e714d9a.el8.x86_ 18 MB/s | 52 MB 00:02 2024-02-12T16:41:01.145 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:01.145 INFO:teuthology.orchestra.run.smithi033.stdout:Total 17 MB/s | 53 MB 00:03 2024-02-12T16:41:01.187 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:41:01.210 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:41:01.210 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:41:01.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-02-12T16:41:01.514 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:41:01.517 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:41:01.980 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:53 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:41:02.044 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:41:02.236 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-12T16:41:02.446 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-12T16:41:02.575 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-12T16:41:02.640 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-12T16:41:02.841 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-12T16:41:03.017 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-12T16:41:03.475 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:41:03.478 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:03.478 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:41:03.478 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:03.478 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:41:03.478 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.4 k 2024-02-12T16:41:03.478 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:41:03.478 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 2.4 M 2024-02-12T16:41:03.478 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 1.6 M 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 245 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 5.2 M 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 19 M 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-12T16:41:03.479 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-12T16:41:03.480 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-12T16:41:03.481 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-12T16:41:03.481 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-12T16:41:03.481 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-12T16:41:03.481 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:03.481 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:41:03.481 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:03.481 INFO:teuthology.orchestra.run.smithi094.stdout:Install 32 Packages 2024-02-12T16:41:03.481 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:03.482 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 32 M 2024-02-12T16:41:03.482 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 104 M 2024-02-12T16:41:03.482 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:41:04.080 INFO:teuthology.orchestra.run.smithi094.stdout:(1/32): ceph-16.2.14-521.g0e714d9a.el8.x86_64.r 34 kB/s | 6.4 kB 00:00 2024-02-12T16:41:04.238 INFO:teuthology.orchestra.run.smithi094.stdout:(2/32): ceph-mds-16.2.14-521.g0e714d9a.el8.x86_ 7.0 MB/s | 2.4 MB 00:00 2024-02-12T16:41:04.538 INFO:teuthology.orchestra.run.smithi094.stdout:(3/32): ceph-mgr-16.2.14-521.g0e714d9a.el8.x86_ 2.5 MB/s | 1.6 MB 00:00 2024-02-12T16:41:04.647 INFO:teuthology.orchestra.run.smithi094.stdout:(4/32): ceph-mgr-modules-core-16.2.14-521.g0e71 2.2 MB/s | 245 kB 00:00 2024-02-12T16:41:04.748 INFO:teuthology.orchestra.run.smithi094.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2024-02-12T16:41:04.781 INFO:teuthology.orchestra.run.smithi094.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.1 MB/s | 39 kB 00:00 2024-02-12T16:41:04.823 INFO:teuthology.orchestra.run.smithi094.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-02-12T16:41:04.898 INFO:teuthology.orchestra.run.smithi094.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 760 kB/s | 57 kB 00:00 2024-02-12T16:41:04.966 INFO:teuthology.orchestra.run.smithi094.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 6.7 MB/s | 457 kB 00:00 2024-02-12T16:41:05.199 INFO:teuthology.orchestra.run.smithi094.stdout:(10/32): ceph-osd-16.2.14-521.g0e714d9a.el8.x86 19 MB/s | 19 MB 00:00 2024-02-12T16:41:05.308 INFO:teuthology.orchestra.run.smithi094.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 727 kB/s | 249 kB 00:00 2024-02-12T16:41:05.392 INFO:teuthology.orchestra.run.smithi094.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2024-02-12T16:41:05.458 INFO:teuthology.orchestra.run.smithi094.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 655 kB/s | 44 kB 00:00 2024-02-12T16:41:05.517 INFO:teuthology.orchestra.run.smithi094.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 3.1 MB/s | 185 kB 00:00 2024-02-12T16:41:05.559 INFO:teuthology.orchestra.run.smithi094.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 4.0 MB/s | 173 kB 00:00 2024-02-12T16:41:05.634 INFO:teuthology.orchestra.run.smithi094.stdout:(16/32): ceph-mon-16.2.14-521.g0e714d9a.el8.x86 3.3 MB/s | 5.2 MB 00:01 2024-02-12T16:41:05.660 INFO:teuthology.orchestra.run.smithi094.stdout:(17/32): python3-cryptography-3.2.1-7.el8.x86_6 1.4 MB/s | 641 kB 00:00 2024-02-12T16:41:05.685 INFO:teuthology.orchestra.run.smithi094.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.0 MB/s | 384 kB 00:00 2024-02-12T16:41:05.710 INFO:teuthology.orchestra.run.smithi094.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2024-02-12T16:41:05.735 INFO:teuthology.orchestra.run.smithi094.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 233 kB/s | 18 kB 00:00 2024-02-12T16:41:05.761 INFO:teuthology.orchestra.run.smithi094.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 645 kB/s | 49 kB 00:00 2024-02-12T16:41:05.795 INFO:teuthology.orchestra.run.smithi094.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 698 kB/s | 59 kB 00:00 2024-02-12T16:41:05.820 INFO:teuthology.orchestra.run.smithi094.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 265 kB/s | 16 kB 00:00 2024-02-12T16:41:05.845 INFO:teuthology.orchestra.run.smithi094.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2024-02-12T16:41:05.871 INFO:teuthology.orchestra.run.smithi094.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2024-02-12T16:41:05.896 INFO:teuthology.orchestra.run.smithi094.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 581 kB/s | 29 kB 00:00 2024-02-12T16:41:05.921 INFO:teuthology.orchestra.run.smithi094.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 548 kB/s | 27 kB 00:00 2024-02-12T16:41:05.947 INFO:teuthology.orchestra.run.smithi094.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-02-12T16:41:05.972 INFO:teuthology.orchestra.run.smithi094.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-02-12T16:41:05.997 INFO:teuthology.orchestra.run.smithi094.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-02-12T16:41:06.022 INFO:teuthology.orchestra.run.smithi094.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2024-02-12T16:41:06.048 INFO:teuthology.orchestra.run.smithi094.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-02-12T16:41:06.049 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:06.049 INFO:teuthology.orchestra.run.smithi094.stdout:Total 12 MB/s | 32 MB 00:02 2024-02-12T16:41:06.131 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:41:06.168 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:41:06.169 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:41:06.483 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:41:06.486 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:41:07.556 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:41:07.778 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-12T16:41:07.933 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-12T16:41:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-12T16:41:08.173 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-12T16:41:08.273 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-12T16:41:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-12T16:41:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 1/6 2024-02-12T16:41:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2/6 2024-02-12T16:41:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-12T16:41:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-12T16:41:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-12T16:41:08.483 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-12T16:41:08.664 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-12T16:41:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-12T16:41:09.021 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-12T16:41:09.039 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-12T16:41:09.039 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:09.039 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:41:09.039 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:09.040 INFO:teuthology.orchestra.run.smithi033.stdout: jq-1.6-8.el8.x86_64 2024-02-12T16:41:09.040 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:09.040 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-12T16:41:09.040 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-12T16:41:09.040 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-12T16:41:09.040 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:09.040 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:41:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-12T16:41:09.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph 2024-02-12T16:41:09.384 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-12T16:41:09.596 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-12T16:41:09.826 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-12T16:41:10.057 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:00:59 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:41:10.101 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-12T16:41:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-12T16:41:10.771 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-12T16:41:10.934 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-12T16:41:11.188 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-12T16:41:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-12T16:41:11.676 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.4 k 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 2.4 M 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 1.6 M 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 245 k 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 5.2 M 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 19 M 2024-02-12T16:41:11.680 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-12T16:41:11.681 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout:Install 32 Packages 2024-02-12T16:41:11.682 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:11.683 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-12T16:41:11.683 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 32 M 2024-02-12T16:41:11.683 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 104 M 2024-02-12T16:41:11.683 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:41:11.916 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-12T16:41:12.104 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-12T16:41:12.331 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-12T16:41:12.572 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-12T16:41:12.661 INFO:teuthology.orchestra.run.smithi033.stdout:(1/32): ceph-16.2.14-521.g0e714d9a.el8.x86_64.r 31 kB/s | 6.4 kB 00:00 2024-02-12T16:41:12.770 INFO:teuthology.orchestra.run.smithi033.stdout:(2/32): ceph-mgr-16.2.14-521.g0e714d9a.el8.x86_ 5.1 MB/s | 1.6 MB 00:00 2024-02-12T16:41:12.807 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-12T16:41:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-12T16:41:13.311 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-12T16:41:13.621 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-12T16:41:13.670 INFO:teuthology.orchestra.run.smithi033.stdout:(3/32): ceph-osd-16.2.14-521.g0e714d9a.el8.x86_ 21 MB/s | 19 MB 00:00 2024-02-12T16:41:13.737 INFO:teuthology.orchestra.run.smithi033.stdout:(4/32): ceph-mgr-modules-core-16.2.14-521.g0e71 3.6 MB/s | 245 kB 00:00 2024-02-12T16:41:13.837 INFO:teuthology.orchestra.run.smithi033.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2024-02-12T16:41:13.879 INFO:teuthology.orchestra.run.smithi033.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 933 kB/s | 39 kB 00:00 2024-02-12T16:41:14.005 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 27/32 2024-02-12T16:41:14.046 INFO:teuthology.orchestra.run.smithi033.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 621 kB/s | 103 kB 00:00 2024-02-12T16:41:14.088 INFO:teuthology.orchestra.run.smithi033.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-02-12T16:41:14.163 INFO:teuthology.orchestra.run.smithi033.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 5.9 MB/s | 457 kB 00:00 2024-02-12T16:41:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-12T16:41:14.522 INFO:teuthology.orchestra.run.smithi033.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 694 kB/s | 249 kB 00:00 2024-02-12T16:41:14.706 INFO:teuthology.orchestra.run.smithi033.stdout:(11/32): python3-cryptography-3.2.1-7.el8.x86_6 3.4 MB/s | 641 kB 00:00 2024-02-12T16:41:14.915 INFO:teuthology.orchestra.run.smithi033.stdout:(12/32): ceph-mds-16.2.14-521.g0e714d9a.el8.x86 1.0 MB/s | 2.4 MB 00:02 2024-02-12T16:41:14.940 INFO:teuthology.orchestra.run.smithi033.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 467 kB/s | 109 kB 00:00 2024-02-12T16:41:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:(14/32): ceph-mon-16.2.14-521.g0e714d9a.el8.x86 2.2 MB/s | 5.2 MB 00:02 2024-02-12T16:41:15.090 INFO:teuthology.orchestra.run.smithi033.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 249 kB/s | 44 kB 00:00 2024-02-12T16:41:15.141 INFO:teuthology.orchestra.run.smithi033.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 948 kB/s | 185 kB 00:00 2024-02-12T16:41:15.190 INFO:teuthology.orchestra.run.smithi033.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.0 MB/s | 173 kB 00:00 2024-02-12T16:41:15.225 INFO:teuthology.orchestra.run.smithi033.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2024-02-12T16:41:15.241 INFO:teuthology.orchestra.run.smithi033.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 347 kB/s | 18 kB 00:00 2024-02-12T16:41:15.275 INFO:teuthology.orchestra.run.smithi033.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.0 MB/s | 384 kB 00:00 2024-02-12T16:41:15.292 INFO:teuthology.orchestra.run.smithi033.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 727 kB/s | 49 kB 00:00 2024-02-12T16:41:15.309 INFO:teuthology.orchestra.run.smithi033.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 874 kB/s | 59 kB 00:00 2024-02-12T16:41:15.326 INFO:teuthology.orchestra.run.smithi033.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 467 kB/s | 16 kB 00:00 2024-02-12T16:41:15.343 INFO:teuthology.orchestra.run.smithi033.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 556 kB/s | 19 kB 00:00 2024-02-12T16:41:15.368 INFO:teuthology.orchestra.run.smithi033.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-02-12T16:41:15.385 INFO:teuthology.orchestra.run.smithi033.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 405 kB/s | 24 kB 00:00 2024-02-12T16:41:15.402 INFO:teuthology.orchestra.run.smithi033.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 497 kB/s | 29 kB 00:00 2024-02-12T16:41:15.419 INFO:teuthology.orchestra.run.smithi033.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 546 kB/s | 27 kB 00:00 2024-02-12T16:41:15.460 INFO:teuthology.orchestra.run.smithi033.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.2 MB/s | 251 kB 00:00 2024-02-12T16:41:15.486 INFO:teuthology.orchestra.run.smithi033.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2024-02-12T16:41:15.511 INFO:teuthology.orchestra.run.smithi033.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 927 kB/s | 85 kB 00:00 2024-02-12T16:41:15.536 INFO:teuthology.orchestra.run.smithi033.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-02-12T16:41:15.541 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:15.542 INFO:teuthology.orchestra.run.smithi033.stdout:Total 8.2 MB/s | 32 MB 00:03 2024-02-12T16:41:15.625 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:41:15.667 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:41:15.668 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:41:15.763 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-12T16:41:15.934 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-12T16:41:16.064 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:41:16.067 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:41:16.376 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-12T16:41:16.525 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-12T16:41:16.741 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-12T16:41:16.967 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-12T16:41:16.984 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-12T16:41:17.116 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-12T16:41:17.227 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:41:17.466 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-12T16:41:17.520 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-12T16:41:17.520 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 1/32 2024-02-12T16:41:17.520 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2/32 2024-02-12T16:41:17.520 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 3/32 2024-02-12T16:41:17.520 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 4/32 2024-02-12T16:41:17.520 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 5/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 6/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-12T16:41:17.521 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-12T16:41:17.522 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-12T16:41:17.698 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-12T16:41:17.979 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-12T16:41:18.203 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-12T16:41:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-12T16:41:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:18.368 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:41:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-12T16:41:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-2.6-1.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-12T16:41:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-12T16:41:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-12T16:41:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-12T16:41:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:18.371 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:41:18.377 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-12T16:41:18.561 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-12T16:41:18.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-02-12T16:41:18.776 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-12T16:41:18.954 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-12T16:41:19.174 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-12T16:41:19.400 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:11 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:41:19.436 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-12T16:41:19.683 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-12T16:41:19.916 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-12T16:41:20.338 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-12T16:41:20.593 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-12T16:41:20.756 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-12T16:41:20.855 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:41:20.901 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:41:20.903 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:41:20.903 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:41:21.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-02-12T16:41:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-12T16:41:21.266 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-12T16:41:21.550 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-12T16:41:21.765 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:13 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:41:21.804 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-12T16:41:21.985 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-12T16:41:22.211 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-12T16:41:22.458 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-12T16:41:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-12T16:41:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-12T16:41:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-12T16:41:23.268 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 85 k 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-02-12T16:41:23.269 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:23.270 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 85 k 2024-02-12T16:41:23.270 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 352 k 2024-02-12T16:41:23.270 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:41:23.434 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-16.2.14-521.g0e714d9a.el8.noarch.rpm 519 kB/s | 85 kB 00:00 2024-02-12T16:41:23.435 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:23.435 INFO:teuthology.orchestra.run.smithi094.stdout:Total 515 kB/s | 85 kB 00:00 2024-02-12T16:41:23.435 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:41:23.442 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:41:23.442 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:41:23.444 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:41:23.447 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:41:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:41:23.591 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-12T16:41:23.965 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-12T16:41:24.004 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 27/32 2024-02-12T16:41:24.186 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-12T16:41:24.186 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-12T16:41:24.416 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-12T16:41:24.781 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-12T16:41:24.781 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:24.781 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:41:24.781 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:24.781 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:24.781 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:41:25.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-02-12T16:41:25.766 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:17 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:41:25.803 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-12T16:41:26.013 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-12T16:41:26.495 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-12T16:41:26.629 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-12T16:41:26.858 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-12T16:41:27.269 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 161 k 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:27.270 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-02-12T16:41:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:27.271 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 161 k 2024-02-12T16:41:27.271 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 470 k 2024-02-12T16:41:27.271 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:41:27.458 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-16.2.14-521.g0e714d 864 kB/s | 161 kB 00:00 2024-02-12T16:41:27.459 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:27.459 INFO:teuthology.orchestra.run.smithi094.stdout:Total 858 kB/s | 161 kB 00:00 2024-02-12T16:41:27.459 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:41:27.469 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:41:27.469 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:41:27.480 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-12T16:41:27.502 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-12T16:41:27.517 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:41:27.520 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:41:27.631 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-12T16:41:27.649 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:41:27.832 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-12T16:41:28.067 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-12T16:41:28.067 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 1/32 2024-02-12T16:41:28.067 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2/32 2024-02-12T16:41:28.067 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 3/32 2024-02-12T16:41:28.067 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 4/32 2024-02-12T16:41:28.067 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 5/32 2024-02-12T16:41:28.067 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 6/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-12T16:41:28.068 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-12T16:41:28.069 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-12T16:41:28.069 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-12T16:41:28.069 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-12T16:41:28.069 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-12T16:41:28.069 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-12T16:41:28.069 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-12T16:41:28.069 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-12T16:41:28.069 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-12T16:41:28.070 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-12T16:41:28.070 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-12T16:41:28.070 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-12T16:41:28.070 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-12T16:41:28.231 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-12T16:41:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-12T16:41:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:28.628 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:41:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:28.628 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:41:28.861 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-02-12T16:41:29.045 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-12T16:41:29.045 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:29.045 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:41:29.045 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:29.045 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-12T16:41:29.046 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-2.6-1.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-12T16:41:29.047 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-12T16:41:29.048 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-12T16:41:29.048 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-12T16:41:29.048 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-12T16:41:29.048 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:29.048 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:41:29.299 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-base 2024-02-12T16:41:29.621 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:21 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:41:30.090 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:19 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:41:31.087 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:41:31.133 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:41:31.134 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:41:31.134 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:41:31.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-02-12T16:41:31.649 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:41:31.699 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:41:31.700 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:41:31.700 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:41:31.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install cephadm 2024-02-12T16:41:31.984 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:23 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:41:32.585 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:21 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:41:33.512 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:41:33.513 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 3.1 M 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 24 k 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 15 k 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-02-12T16:41:33.514 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:33.515 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.4 M 2024-02-12T16:41:33.515 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 22 M 2024-02-12T16:41:33.515 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:41:34.037 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): ceph-grafana-dashboards-16.2.14-521.g0e7 173 kB/s | 24 kB 00:00 2024-02-12T16:41:34.063 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): ceph-prometheus-alerts-16.2.14-521.g0e71 89 kB/s | 15 kB 00:00 2024-02-12T16:41:34.196 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 85 k 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 85 k 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 352 k 2024-02-12T16:41:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:41:34.321 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 150 kB/s | 43 kB 00:00 2024-02-12T16:41:34.380 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 106 kB/s | 34 kB 00:00 2024-02-12T16:41:34.494 INFO:teuthology.orchestra.run.smithi033.stdout:cephadm-16.2.14-521.g0e714d9a.el8.noarch.rpm 288 kB/s | 85 kB 00:00 2024-02-12T16:41:34.494 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:34.494 INFO:teuthology.orchestra.run.smithi033.stdout:Total 287 kB/s | 85 kB 00:00 2024-02-12T16:41:34.495 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:41:34.502 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:41:34.502 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:41:34.504 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:41:34.508 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:41:34.552 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:41:34.814 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 399 kB/s | 196 kB 00:00 2024-02-12T16:41:35.093 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-12T16:41:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-12T16:41:35.513 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): ceph-mgr-dashboard-16.2.14-521.g0e714d9a 1.9 MB/s | 3.1 MB 00:01 2024-02-12T16:41:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-12T16:41:35.516 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:35.516 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.7 MB/s | 3.4 MB 00:02 2024-02-12T16:41:35.527 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:41:35.538 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:41:35.538 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:41:35.718 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:41:35.721 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:41:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-12T16:41:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:35.896 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:41:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:35.896 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:41:35.996 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:41:36.125 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-immutable-object-cache 2024-02-12T16:41:36.212 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-12T16:41:36.474 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-12T16:41:36.675 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-12T16:41:36.796 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 4/6 2024-02-12T16:41:36.913 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:25 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:41:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 5/6 2024-02-12T16:41:38.096 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-12T16:41:38.361 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-12T16:41:38.361 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 1/6 2024-02-12T16:41:38.361 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 2/6 2024-02-12T16:41:38.361 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 3/6 2024-02-12T16:41:38.361 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-12T16:41:38.361 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-12T16:41:38.534 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:41:38.534 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 161 k 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-02-12T16:41:38.535 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:38.536 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 161 k 2024-02-12T16:41:38.536 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 470 k 2024-02-12T16:41:38.536 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:38.741 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:41:38.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-12T16:41:39.231 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-immutable-object-cache-16.2.14-521.g0e714d 232 kB/s | 161 kB 00:00 2024-02-12T16:41:39.232 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:39.232 INFO:teuthology.orchestra.run.smithi033.stdout:Total 231 kB/s | 161 kB 00:00 2024-02-12T16:41:39.233 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:41:39.246 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:41:39.246 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:41:39.304 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:41:39.307 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:41:39.427 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:41:39.610 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:31 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:41:39.618 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-12T16:41:40.042 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-12T16:41:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-12T16:41:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:40.474 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:41:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:41:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:40.474 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:41:40.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr 2024-02-12T16:41:41.110 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:41:41.111 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-02-12T16:41:41.111 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 8.6 M 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-12T16:41:41.112 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-12T16:41:41.113 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:41.113 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:41:41.113 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-02-12T16:41:41.113 INFO:teuthology.orchestra.run.smithi094.stdout:Install 8 Packages 2024-02-12T16:41:41.113 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:41.113 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 37 M 2024-02-12T16:41:41.113 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 210 M 2024-02-12T16:41:41.113 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:41:41.481 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:30 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:41:42.161 INFO:teuthology.orchestra.run.smithi094.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.6 MB/s | 4.9 MB 00:00 2024-02-12T16:41:42.478 INFO:teuthology.orchestra.run.smithi094.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 4.5 MB/s | 4.7 MB 00:01 2024-02-12T16:41:42.579 INFO:teuthology.orchestra.run.smithi094.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 9.0 MB/s | 3.7 MB 00:00 2024-02-12T16:41:42.612 INFO:teuthology.orchestra.run.smithi094.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2024-02-12T16:41:43.050 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:41:43.101 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:41:43.102 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:41:43.102 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:41:43.207 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-dashboard 2024-02-12T16:41:43.381 INFO:teuthology.orchestra.run.smithi094.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 840 kB/s | 645 kB 00:00 2024-02-12T16:41:43.498 INFO:teuthology.orchestra.run.smithi094.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.4 MB/s | 172 kB 00:00 2024-02-12T16:41:43.996 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:32 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:41:44.332 INFO:teuthology.orchestra.run.smithi094.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 8.1 MB/s | 14 MB 00:01 2024-02-12T16:41:44.757 INFO:teuthology.orchestra.run.smithi094.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-52 2.6 MB/s | 8.6 MB 00:03 2024-02-12T16:41:44.758 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:44.758 INFO:teuthology.orchestra.run.smithi094.stdout:Total 10 MB/s | 37 MB 00:03 2024-02-12T16:41:44.992 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:41:45.010 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:41:45.010 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:41:45.335 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:41:45.338 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:41:45.622 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:41:45.623 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:45.623 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:41:45.623 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:45.623 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 3.1 M 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 24 k 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 15 k 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2024-02-12T16:41:45.624 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:45.625 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 3.4 M 2024-02-12T16:41:45.625 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 22 M 2024-02-12T16:41:45.625 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:41:45.885 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:41:46.013 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-12T16:41:46.085 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): ceph-grafana-dashboards-16.2.14-521.g0e7 107 kB/s | 24 kB 00:00 2024-02-12T16:41:46.133 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-12T16:41:46.135 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): ceph-prometheus-alerts-16.2.14-521.g0e71 53 kB/s | 15 kB 00:00 2024-02-12T16:41:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-12T16:41:46.319 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 183 kB/s | 34 kB 00:00 2024-02-12T16:41:46.411 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 131 kB/s | 43 kB 00:00 2024-02-12T16:41:46.436 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2024-02-12T16:41:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-12T16:41:47.123 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:41:47.336 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): ceph-mgr-dashboard-16.2.14-521.g0e714d9a 2.1 MB/s | 3.1 MB 00:01 2024-02-12T16:41:47.340 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:47.340 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.0 MB/s | 3.4 MB 00:01 2024-02-12T16:41:47.352 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:41:47.363 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:41:47.363 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:41:47.548 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:41:47.552 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:41:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:41:47.823 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:41:47.903 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-12T16:41:48.026 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-12T16:41:48.296 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-12T16:41:48.487 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-12T16:41:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-12T16:41:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 4/6 2024-02-12T16:41:49.579 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-12T16:41:49.715 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 5/6 2024-02-12T16:41:49.821 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-12T16:41:50.148 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-12T16:41:50.451 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-12T16:41:50.451 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 1/6 2024-02-12T16:41:50.451 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 2/6 2024-02-12T16:41:50.451 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 3/6 2024-02-12T16:41:50.451 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-12T16:41:50.452 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:50.866 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:41:50.984 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-12T16:41:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-12T16:41:51.773 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:40 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:41:52.914 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-12T16:41:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-12T16:41:53.388 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:41:53.390 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-02-12T16:41:53.390 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:41:53.390 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-02-12T16:41:53.390 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:41:53.390 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 8.6 M 2024-02-12T16:41:53.391 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:41:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-12T16:41:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-12T16:41:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-12T16:41:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-12T16:41:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-12T16:41:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-12T16:41:53.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-12T16:41:53.392 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:53.392 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:41:53.392 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-02-12T16:41:53.392 INFO:teuthology.orchestra.run.smithi033.stdout:Install 8 Packages 2024-02-12T16:41:53.393 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:41:53.394 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 37 M 2024-02-12T16:41:53.394 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 210 M 2024-02-12T16:41:53.394 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:41:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-12T16:41:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 1/8 2024-02-12T16:41:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-12T16:41:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:41:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-12T16:41:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-12T16:41:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-12T16:41:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-12T16:41:53.824 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-12T16:41:53.825 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-12T16:41:53.825 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:53.825 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:41:53.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-02-12T16:41:54.316 INFO:teuthology.orchestra.run.smithi033.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-52 13 MB/s | 8.6 MB 00:00 2024-02-12T16:41:54.692 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:46 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:41:55.217 INFO:teuthology.orchestra.run.smithi033.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.1 MB/s | 3.7 MB 00:00 2024-02-12T16:41:55.594 INFO:teuthology.orchestra.run.smithi033.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 598 kB/s | 225 kB 00:00 2024-02-12T16:41:55.899 INFO:teuthology.orchestra.run.smithi033.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.2 MB/s | 4.9 MB 00:02 2024-02-12T16:41:56.145 INFO:teuthology.orchestra.run.smithi033.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 1.9 MB/s | 4.7 MB 00:02 2024-02-12T16:41:56.212 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:41:56.214 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 27 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-12T16:41:56.215 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-12T16:41:56.216 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-12T16:41:56.216 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-12T16:41:56.216 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:56.216 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:41:56.216 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:41:56.216 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-02-12T16:41:56.216 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:41:56.217 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.9 M 2024-02-12T16:41:56.217 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 24 M 2024-02-12T16:41:56.217 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:41:56.520 INFO:teuthology.orchestra.run.smithi033.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 645 kB 00:00 2024-02-12T16:41:56.670 INFO:teuthology.orchestra.run.smithi033.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 329 kB/s | 172 kB 00:00 2024-02-12T16:41:56.716 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 138 kB/s | 20 kB 00:00 2024-02-12T16:41:56.733 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 164 kB/s | 27 kB 00:00 2024-02-12T16:41:56.854 INFO:teuthology.orchestra.run.smithi033.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 11 MB/s | 14 MB 00:01 2024-02-12T16:41:56.854 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:56.854 INFO:teuthology.orchestra.run.smithi033.stdout:Total 11 MB/s | 37 MB 00:03 2024-02-12T16:41:56.908 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 659 kB/s | 126 kB 00:00 2024-02-12T16:41:56.933 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 549 kB/s | 110 kB 00:00 2024-02-12T16:41:56.959 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): ceph-mgr-rook-16.2.14-521.g0e714d9a.el8 70 kB/s | 27 kB 00:00 2024-02-12T16:41:57.093 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:41:57.112 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:41:57.112 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:41:57.243 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 138 kB/s | 43 kB 00:00 2024-02-12T16:41:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 94 kB/s | 33 kB 00:00 2024-02-12T16:41:57.385 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 325 kB/s | 155 kB 00:00 2024-02-12T16:41:57.463 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:41:57.467 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:41:57.560 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 330 kB/s | 81 kB 00:00 2024-02-12T16:41:57.585 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-certifi-2018.10.15-7.el8.noarc 48 kB/s | 16 kB 00:00 2024-02-12T16:41:57.677 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 552 kB/s | 64 kB 00:00 2024-02-12T16:41:57.753 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 367 kB/s | 61 kB 00:00 2024-02-12T16:41:58.028 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.8 MB/s | 1.2 MB 00:00 2024-02-12T16:41:58.031 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:41:58.031 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-02-12T16:41:58.038 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:41:58.087 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:41:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:41:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:41:58.191 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:41:58.194 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:41:58.207 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-12T16:41:58.355 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-12T16:41:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-12T16:41:58.785 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:41:58.996 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-12T16:41:59.119 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-12T16:41:59.169 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-12T16:41:59.257 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:41:59.364 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-12T16:41:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-12T16:41:59.760 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-12T16:41:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:41:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-12T16:42:00.032 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-12T16:42:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-12T16:42:00.500 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-12T16:42:00.773 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-12T16:42:01.098 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-12T16:42:01.525 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-12T16:42:01.672 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-12T16:42:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-12T16:42:01.993 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-12T16:42:02.177 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-12T16:42:02.335 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 1/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-12T16:42:02.631 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-12T16:42:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-12T16:42:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-12T16:42:02.632 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-12T16:42:03.071 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-12T16:42:03.071 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:03.071 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:42:03.071 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:42:03.071 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-12T16:42:03.071 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-12T16:42:03.071 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-12T16:42:03.071 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:03.072 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:03.178 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-02-12T16:42:03.554 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-12T16:42:03.928 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:55 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:04.925 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-12T16:42:05.131 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-12T16:42:05.449 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:05.450 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:05.450 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:42:05.450 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:05.450 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:42:05.450 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 116 k 2024-02-12T16:42:05.450 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-02-12T16:42:05.451 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:05.452 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 5.8 M 2024-02-12T16:42:05.452 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 24 M 2024-02-12T16:42:05.452 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:42:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-12T16:42:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 1/8 2024-02-12T16:42:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-12T16:42:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:42:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-12T16:42:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-12T16:42:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-12T16:42:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-12T16:42:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-12T16:42:05.907 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:05.907 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:05.908 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:05.937 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.0 MB/s | 538 kB 00:00 2024-02-12T16:42:06.031 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-rook 2024-02-12T16:42:06.045 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): ceph-mgr-cephadm-16.2.14-521.g0e714d9a.e 411 kB/s | 116 kB 00:00 2024-02-12T16:42:06.196 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 82 kB/s | 21 kB 00:00 2024-02-12T16:42:06.388 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 255 kB/s | 49 kB 00:00 2024-02-12T16:42:06.555 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.0 MB/s | 4.8 MB 00:00 2024-02-12T16:42:06.588 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 655 kB/s | 355 kB 00:00 2024-02-12T16:42:06.590 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:06.591 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.1 MB/s | 5.8 MB 00:01 2024-02-12T16:42:06.641 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:42:06.651 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:42:06.651 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:42:06.755 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:42:06.758 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:42:06.813 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:55 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:42:07.268 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-12T16:42:07.512 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-12T16:42:08.419 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-12T16:42:08.453 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 27 k 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-12T16:42:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout:Install 13 Packages 2024-02-12T16:42:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:08.457 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 1.9 M 2024-02-12T16:42:08.457 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 24 M 2024-02-12T16:42:08.457 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:42:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-12T16:42:09.018 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-12T16:42:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-12T16:42:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-12T16:42:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/6 2024-02-12T16:42:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-12T16:42:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-12T16:42:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-12T16:42:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-12T16:42:09.936 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-12T16:42:09.936 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:09.936 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:42:09.936 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:42:09.936 INFO:teuthology.orchestra.run.smithi094.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-12T16:42:09.936 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-12T16:42:09.937 INFO:teuthology.orchestra.run.smithi094.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-12T16:42:09.937 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-12T16:42:09.937 INFO:teuthology.orchestra.run.smithi094.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-12T16:42:09.937 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:09.937 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:10.060 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-02-12T16:42:10.812 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:02 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:10.859 INFO:teuthology.orchestra.run.smithi033.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 161 kB/s | 27 kB 00:00 2024-02-12T16:42:10.884 INFO:teuthology.orchestra.run.smithi033.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 105 kB/s | 20 kB 00:00 2024-02-12T16:42:11.018 INFO:teuthology.orchestra.run.smithi033.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 799 kB/s | 126 kB 00:00 2024-02-12T16:42:11.043 INFO:teuthology.orchestra.run.smithi033.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 693 kB/s | 110 kB 00:00 2024-02-12T16:42:11.068 INFO:teuthology.orchestra.run.smithi033.stdout:(5/13): ceph-mgr-rook-16.2.14-521.g0e714d9a.el8 73 kB/s | 27 kB 00:00 2024-02-12T16:42:11.344 INFO:teuthology.orchestra.run.smithi033.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 120 kB/s | 33 kB 00:00 2024-02-12T16:42:11.369 INFO:teuthology.orchestra.run.smithi033.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 131 kB/s | 43 kB 00:00 2024-02-12T16:42:11.436 INFO:teuthology.orchestra.run.smithi033.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 177 kB/s | 16 kB 00:00 2024-02-12T16:42:11.511 INFO:teuthology.orchestra.run.smithi033.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 314 kB/s | 155 kB 00:00 2024-02-12T16:42:11.712 INFO:teuthology.orchestra.run.smithi033.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 237 kB/s | 81 kB 00:00 2024-02-12T16:42:11.787 INFO:teuthology.orchestra.run.smithi033.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 820 kB/s | 61 kB 00:00 2024-02-12T16:42:11.840 INFO:teuthology.orchestra.run.smithi033.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 2.9 MB/s | 1.2 MB 00:00 2024-02-12T16:42:11.870 INFO:teuthology.orchestra.run.smithi033.stdout:(13/13): python3-rsa-4.9-2.el8.noarch.rpm 179 kB/s | 64 kB 00:00 2024-02-12T16:42:11.874 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:11.874 INFO:teuthology.orchestra.run.smithi033.stdout:Total 573 kB/s | 1.9 MB 00:03 2024-02-12T16:42:11.933 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:42:11.950 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:42:11.950 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:42:12.045 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:42:12.048 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:42:12.297 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:12.297 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:12.297 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-02-12T16:42:12.297 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 911 k 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-02-12T16:42:12.298 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:12.299 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 994 k 2024-02-12T16:42:12.299 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.8 M 2024-02-12T16:42:12.299 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:42:12.642 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:42:12.903 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-12T16:42:12.935 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 208 kB/s | 83 kB 00:00 2024-02-12T16:42:13.100 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-12T16:42:13.185 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-16.2.14-521.g0e714d9a.el8.x86_ 1.4 MB/s | 911 kB 00:00 2024-02-12T16:42:13.185 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:13.186 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.1 MB/s | 994 kB 00:00 2024-02-12T16:42:13.196 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:42:13.218 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:42:13.218 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:42:13.308 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:42:13.311 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:42:13.343 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-12T16:42:13.500 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:42:13.605 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-12T16:42:13.775 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-12T16:42:13.805 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-12T16:42:13.977 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-12T16:42:14.074 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-12T16:42:14.354 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-12T16:42:14.378 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-12T16:42:14.379 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/2 2024-02-12T16:42:14.631 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-12T16:42:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-12T16:42:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:14.777 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:42:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-12T16:42:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:14.777 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:15.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-02-12T16:42:15.747 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-12T16:42:15.769 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:07 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:16.045 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-12T16:42:16.294 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-12T16:42:16.488 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-12T16:42:16.686 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-12T16:42:17.004 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-12T16:42:17.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 1/13 2024-02-12T16:42:17.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-12T16:42:17.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-12T16:42:17.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-12T16:42:17.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-12T16:42:17.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-12T16:42:17.004 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-12T16:42:17.005 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-12T16:42:17.005 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-12T16:42:17.005 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-12T16:42:17.005 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-12T16:42:17.005 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-12T16:42:17.273 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 131 k 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:17.274 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 131 k 2024-02-12T16:42:17.275 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 432 k 2024-02-12T16:42:17.275 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:42:17.648 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-16.2.14-521.g0e714d9a.el8.x86_64 351 kB/s | 131 kB 00:00 2024-02-12T16:42:17.648 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:17.649 INFO:teuthology.orchestra.run.smithi094.stdout:Total 350 kB/s | 131 kB 00:00 2024-02-12T16:42:17.649 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:42:17.654 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:42:17.654 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:42:17.707 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:42:17.710 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:42:17.833 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:42:17.888 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-12T16:42:17.888 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:17.888 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:42:17.888 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:42:17.888 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:17.889 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:18.033 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-cephadm 2024-02-12T16:42:18.058 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:18.720 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:18.720 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:18.720 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:42:18.721 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:42:18.721 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:18.721 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:18.821 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:07 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:18.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-02-12T16:42:19.721 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:11 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:20.445 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:20.446 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:20.446 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:42:20.446 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:20.446 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:42:20.446 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 116 k 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2024-02-12T16:42:20.447 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:20.448 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 5.8 M 2024-02-12T16:42:20.448 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 24 M 2024-02-12T16:42:20.448 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:42:20.753 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 4.0 MB/s | 538 kB 00:00 2024-02-12T16:42:20.803 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): ceph-mgr-cephadm-16.2.14-521.g0e714d9a.e 638 kB/s | 116 kB 00:00 2024-02-12T16:42:20.995 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 88 kB/s | 21 kB 00:00 2024-02-12T16:42:21.171 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.7 MB/s | 4.8 MB 00:00 2024-02-12T16:42:21.185 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:21.187 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 255 kB/s | 49 kB 00:00 2024-02-12T16:42:21.231 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:21.232 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:42:21.232 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:21.246 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 803 kB/s | 355 kB 00:00 2024-02-12T16:42:21.248 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:21.248 INFO:teuthology.orchestra.run.smithi033.stdout:Total 7.3 MB/s | 5.8 MB 00:00 2024-02-12T16:42:21.301 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:42:21.311 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:42:21.311 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:42:21.432 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:42:21.435 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:42:21.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-02-12T16:42:21.721 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:42:22.009 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-12T16:42:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-12T16:42:22.449 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:14 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-12T16:42:23.674 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-12T16:42:23.948 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:23.949 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:23.949 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-02-12T16:42:23.949 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:23.949 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:42:23.949 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 25 k 2024-02-12T16:42:23.949 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:23.950 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:42:23.950 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:23.950 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-02-12T16:42:23.950 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:23.950 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 25 k 2024-02-12T16:42:23.950 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 107 k 2024-02-12T16:42:23.950 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:42:24.011 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-12T16:42:24.187 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-12T16:42:24.247 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-16.2.14-521.g0e714d9a.el8.x86_6 86 kB/s | 25 kB 00:00 2024-02-12T16:42:24.248 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:24.248 INFO:teuthology.orchestra.run.smithi094.stdout:Total 86 kB/s | 25 kB 00:00 2024-02-12T16:42:24.248 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:42:24.251 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:42:24.251 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:42:24.254 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:42:24.257 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:42:24.306 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:42:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-12T16:42:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/6 2024-02-12T16:42:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-12T16:42:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-12T16:42:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-12T16:42:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-12T16:42:24.490 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:24.752 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:24.910 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:24.911 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:25.037 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-fuse 2024-02-12T16:42:25.099 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:25.099 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:25.099 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:42:25.099 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:42:25.099 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:25.099 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:25.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-02-12T16:42:25.825 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:14 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:25.971 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:17 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:27.444 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:27.453 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 911 k 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:27.454 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:42:27.455 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:27.455 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2024-02-12T16:42:27.455 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:27.455 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 994 k 2024-02-12T16:42:27.455 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 2.8 M 2024-02-12T16:42:27.455 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:42:27.489 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:27.491 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:42:27.491 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:27.584 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-02-12T16:42:27.790 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): ceph-fuse-16.2.14-521.g0e714d9a.el8.x86_ 3.3 MB/s | 911 kB 00:00 2024-02-12T16:42:27.866 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 243 kB/s | 83 kB 00:00 2024-02-12T16:42:27.866 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:27.867 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.4 MB/s | 994 kB 00:00 2024-02-12T16:42:27.877 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:42:27.899 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:42:27.899 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:42:27.994 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:42:27.998 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:42:28.190 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:42:28.333 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:20 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:28.464 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-12T16:42:28.695 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-12T16:42:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-12T16:42:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/2 2024-02-12T16:42:29.500 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-12T16:42:29.500 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:29.500 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:42:29.500 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-12T16:42:29.500 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:29.500 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:29.721 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados-devel 2024-02-12T16:42:29.784 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:29.830 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:29.831 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:42:29.832 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:29.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-02-12T16:42:30.506 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:19 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:30.681 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:22 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:32.112 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:32.113 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:32.113 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-02-12T16:42:32.113 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:32.113 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 131 k 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 131 k 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 432 k 2024-02-12T16:42:32.114 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:42:32.153 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:32.200 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:32.201 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:42:32.201 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:32.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-02-12T16:42:32.736 INFO:teuthology.orchestra.run.smithi033.stdout:librados-devel-16.2.14-521.g0e714d9a.el8.x86_64 211 kB/s | 131 kB 00:00 2024-02-12T16:42:32.736 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:32.737 INFO:teuthology.orchestra.run.smithi033.stdout:Total 210 kB/s | 131 kB 00:00 2024-02-12T16:42:32.737 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:42:32.743 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:42:32.743 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:42:32.804 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:42:32.807 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:42:32.928 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:42:33.175 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:25 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:33.910 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:34.630 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:34.641 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:34.687 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:34.688 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:42:34.688 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:34.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-02-12T16:42:35.537 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:27 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:35.584 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:42:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:42:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:35.584 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:35.808 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs2 2024-02-12T16:42:36.600 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:25 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:36.997 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:37.042 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:37.043 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:42:37.043 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:37.147 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-02-12T16:42:37.896 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:29 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:38.168 INFO:teuthology.orchestra.run.smithi033.stdout:Package libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:38.217 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:38.218 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:42:38.218 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:38.324 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs-devel 2024-02-12T16:42:39.105 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:28 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:39.360 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:39.405 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:39.406 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-02-12T16:42:39.406 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:39.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-02-12T16:42:40.315 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:32 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:40.741 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:40.741 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 25 k 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-02-12T16:42:40.742 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:40.743 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 25 k 2024-02-12T16:42:40.743 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 107 k 2024-02-12T16:42:40.743 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:42:40.959 INFO:teuthology.orchestra.run.smithi033.stdout:libcephfs-devel-16.2.14-521.g0e714d9a.el8.x86_6 118 kB/s | 25 kB 00:00 2024-02-12T16:42:40.960 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:40.960 INFO:teuthology.orchestra.run.smithi033.stdout:Total 117 kB/s | 25 kB 00:00 2024-02-12T16:42:40.960 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:42:40.964 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:42:40.964 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:42:40.967 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:42:40.970 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:42:41.020 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:42:41.216 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:41.501 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:41.824 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 88 k 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-02-12T16:42:41.825 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:41.826 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 88 k 2024-02-12T16:42:41.826 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 240 k 2024-02-12T16:42:41.826 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:42:41.878 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:41.878 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:41.878 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:42:41.878 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:42:41.878 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:41.878 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:41.985 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados2 2024-02-12T16:42:42.054 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-16.2.14-521.g0e714d9a.el8.x86_64.rpm 387 kB/s | 88 kB 00:00 2024-02-12T16:42:42.054 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:42.055 INFO:teuthology.orchestra.run.smithi094.stdout:Total 385 kB/s | 88 kB 00:00 2024-02-12T16:42:42.055 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:42:42.064 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:42:42.064 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:42:42.115 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:42:42.118 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:42:42.231 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:42:42.405 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:42.736 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:42.773 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:31 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:43.257 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:43.257 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:43.258 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:42:43.258 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:42:43.258 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:43.258 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:43.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-02-12T16:42:44.215 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:36 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:44.345 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:44.395 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:44.396 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:42:44.396 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:44.494 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librbd1 2024-02-12T16:42:45.294 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:34 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:45.715 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:45.715 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:45.715 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-02-12T16:42:45.715 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:45.715 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:42:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-12T16:42:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:45.716 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:42:45.716 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:45.716 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-02-12T16:42:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:45.716 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.7 M 2024-02-12T16:42:45.717 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 15 M 2024-02-12T16:42:45.717 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:42:46.159 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-16.2.14-521.g0e714d9a.el8.x86_64.rpm 8.3 MB/s | 3.7 MB 00:00 2024-02-12T16:42:46.159 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:46.160 INFO:teuthology.orchestra.run.smithi094.stdout:Total 8.3 MB/s | 3.7 MB 00:00 2024-02-12T16:42:46.160 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:42:46.171 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:42:46.171 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:42:46.234 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:42:46.237 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:42:46.669 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:42:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:46.859 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:46.909 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:46.910 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:42:46.910 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:47.002 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rados 2024-02-12T16:42:47.254 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:47.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:47.633 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:47.633 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:42:47.633 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:42:47.633 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:47.633 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:47.791 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:36 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:47.832 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-02-12T16:42:48.600 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:40 ago on Mon 12 Feb 2024 04:40:08 PM UTC. 2024-02-12T16:42:49.367 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:49.416 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:49.418 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:42:49.418 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:49.519 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rgw 2024-02-12T16:42:50.125 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-02-12T16:42:50.125 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-02-12T16:42:50.125 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:50.126 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-02-12T16:42:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 175 k 2024-02-12T16:42:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:50.126 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-02-12T16:42:50.126 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-02-12T16:42:50.126 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-02-12T16:42:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:50.127 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 175 k 2024-02-12T16:42:50.127 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 523 k 2024-02-12T16:42:50.127 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-02-12T16:42:50.299 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:39 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:50.422 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-16.2.14-521.g0e714d9a.el8.x86_64.rpm 594 kB/s | 175 kB 00:00 2024-02-12T16:42:50.422 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:50.423 INFO:teuthology.orchestra.run.smithi094.stdout:Total 591 kB/s | 175 kB 00:00 2024-02-12T16:42:50.423 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-02-12T16:42:50.433 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-02-12T16:42:50.433 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-02-12T16:42:50.484 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-02-12T16:42:50.487 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-02-12T16:42:50.623 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-02-12T16:42:50.804 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:51.098 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:51.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:42:51.454 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:51.454 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-02-12T16:42:51.454 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:42:51.454 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:42:51.454 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-02-12T16:42:51.643 DEBUG:teuthology.parallel:result is None 2024-02-12T16:42:51.849 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:51.899 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:51.900 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:42:51.900 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:52.003 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-cephfs 2024-02-12T16:42:52.788 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:41 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:54.348 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:54.397 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:54.398 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:42:54.398 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:54.498 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rbd 2024-02-12T16:42:55.280 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:44 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:56.845 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-12T16:42:56.894 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:56.895 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-02-12T16:42:56.895 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:42:56.988 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-fuse 2024-02-12T16:42:57.771 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:46 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:42:59.394 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:42:59.394 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:59.394 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-02-12T16:42:59.394 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:59.394 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:42:59.395 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 88 k 2024-02-12T16:42:59.395 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:59.395 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:42:59.395 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:42:59.395 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-02-12T16:42:59.395 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:42:59.395 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 88 k 2024-02-12T16:42:59.396 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 240 k 2024-02-12T16:42:59.396 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:42:59.800 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-fuse-16.2.14-521.g0e714d9a.el8.x86_64.rpm 218 kB/s | 88 kB 00:00 2024-02-12T16:42:59.801 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:42:59.801 INFO:teuthology.orchestra.run.smithi033.stdout:Total 217 kB/s | 88 kB 00:00 2024-02-12T16:42:59.801 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:42:59.812 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:42:59.812 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:42:59.869 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:42:59.872 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:42:59.982 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:43:00.205 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:00.509 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:00.905 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:00.906 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:00.906 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:43:00.906 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:43:00.906 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:00.906 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:43:01.171 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-mirror 2024-02-12T16:43:01.956 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:50 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:43:03.592 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-02-12T16:43:03.593 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:03.594 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 3.7 M 2024-02-12T16:43:03.594 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 15 M 2024-02-12T16:43:03.594 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:43:05.094 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-mirror-16.2.14-521.g0e714d9a.el8.x86_64.rpm 2.4 MB/s | 3.7 MB 00:01 2024-02-12T16:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:43:05.095 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.4 MB/s | 3.7 MB 00:01 2024-02-12T16:43:05.096 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:43:05.110 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:43:05.183 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:43:05.186 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:43:05.685 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:43:05.895 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:06.339 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:06.728 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:06.728 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:06.728 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:43:06.728 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:43:06.728 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:06.728 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:43:07.028 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-nbd 2024-02-12T16:43:07.821 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:56 ago on Mon 12 Feb 2024 04:40:11 PM UTC. 2024-02-12T16:43:09.432 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-02-12T16:43:09.432 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:43:09.432 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-02-12T16:43:09.432 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:43:09.432 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-02-12T16:43:09.432 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 175 k 2024-02-12T16:43:09.433 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:09.433 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-02-12T16:43:09.433 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-02-12T16:43:09.433 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-02-12T16:43:09.433 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:09.433 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 175 k 2024-02-12T16:43:09.433 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 523 k 2024-02-12T16:43:09.434 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-02-12T16:43:09.756 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-nbd-16.2.14-521.g0e714d9a.el8.x86_64.rpm 544 kB/s | 175 kB 00:00 2024-02-12T16:43:09.756 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:43:09.756 INFO:teuthology.orchestra.run.smithi033.stdout:Total 542 kB/s | 175 kB 00:00 2024-02-12T16:43:09.757 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-02-12T16:43:09.769 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-02-12T16:43:09.769 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-02-12T16:43:09.823 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-02-12T16:43:09.826 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-02-12T16:43:09.938 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-02-12T16:43:10.120 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:10.426 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:10.822 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-12T16:43:10.822 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:10.822 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-02-12T16:43:10.822 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-12T16:43:10.822 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:43:10.822 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-02-12T16:43:11.001 DEBUG:teuthology.parallel:result is None 2024-02-12T16:43:11.001 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:43:11.144 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-12T16:43:11.168 INFO:teuthology.orchestra.run.smithi033.stdout:16.2.14-521.g0e714d9a.el8 2024-02-12T16:43:11.169 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-521.g0e714d9a.el8 2024-02-12T16:43:11.169 INFO:teuthology.task.install:The correct ceph version 16.2.14-521.g0e714d9a is installed. 2024-02-12T16:43:11.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:43:11.312 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-12T16:43:11.338 INFO:teuthology.orchestra.run.smithi094.stdout:16.2.14-521.g0e714d9a.el8 2024-02-12T16:43:11.339 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-521.g0e714d9a.el8 2024-02-12T16:43:11.339 INFO:teuthology.task.install:The correct ceph version 16.2.14-521.g0e714d9a is installed. 2024-02-12T16:43:11.341 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-12T16:43:11.341 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:43:11.341 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-12T16:43:11.370 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:43:11.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-12T16:43:11.411 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-12T16:43:11.411 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:43:11.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-02-12T16:43:11.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-12T16:43:11.508 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:43:11.508 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-02-12T16:43:11.538 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-12T16:43:11.608 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-12T16:43:11.609 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:43:11.609 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-12T16:43:11.637 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-12T16:43:11.704 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:43:11.705 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-12T16:43:11.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-12T16:43:11.803 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-12T16:43:11.803 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:43:11.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-02-12T16:43:11.831 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-12T16:43:11.898 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:43:11.899 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-02-12T16:43:11.926 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-12T16:43:11.993 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-12T16:43:12.068 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)'], 'mon_bind_msgr2': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'cephadm_mode': 'root'} 2024-02-12T16:43:12.068 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:43:12.069 INFO:tasks.cephadm:Cluster fsid is cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:12.069 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-12T16:43:12.069 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.33:6789]', 'mon.c': '[v1:172.21.15.33:6790]', 'mon.b': '[v1:172.21.15.94:6789]'} 2024-02-12T16:43:12.069 INFO:tasks.cephadm:First mon is mon.a on smithi033 2024-02-12T16:43:12.069 INFO:tasks.cephadm:First mgr is y 2024-02-12T16:43:12.069 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-12T16:43:12.069 DEBUG:teuthology.orchestra.run.smithi033:> sudo hostname $(hostname -s) 2024-02-12T16:43:12.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo hostname $(hostname -s) 2024-02-12T16:43:12.127 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 0e714d9a4bd2a821113e6318adb87bd06cf81ec1)... 2024-02-12T16:43:12.128 DEBUG:teuthology.orchestra.run.smithi033:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-12T16:43:12.509 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 12 16:43 /home/ubuntu/cephtest/cephadm 2024-02-12T16:43:12.510 DEBUG:teuthology.orchestra.run.smithi094:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-12T16:43:12.726 INFO:teuthology.orchestra.run.smithi094.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 12 16:43 /home/ubuntu/cephtest/cephadm 2024-02-12T16:43:12.727 DEBUG:teuthology.orchestra.run.smithi033:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-12T16:43:12.748 DEBUG:teuthology.orchestra.run.smithi094:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-12T16:43:12.779 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2024-02-12T16:43:12.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph 2024-02-12T16:43:12.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /etc/ceph 2024-02-12T16:43:12.888 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /etc/ceph 2024-02-12T16:43:12.914 INFO:tasks.cephadm:Writing seed config... 2024-02-12T16:43:12.915 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-12T16:43:12.916 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-12T16:43:12.917 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-12T16:43:12.918 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:43:12.918 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-12T16:43:12.948 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = cf6ad642-c9c5-11ee-95b9-87774f69a715 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-12T16:43:12.948 DEBUG:teuthology.orchestra.run.smithi033:mon.a> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.a.service 2024-02-12T16:43:12.991 DEBUG:teuthology.orchestra.run.smithi033:mgr.y> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.y.service 2024-02-12T16:43:13.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:43:13.020 INFO:tasks.cephadm:Bootstrapping... 2024-02-12T16:43:13.020 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -v bootstrap --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.33:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:43:13.048 INFO:journalctl@ceph.mgr.y.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:43:13.304 INFO:teuthology.orchestra.run.smithi033.stderr:-------------------------------------------------------------------------------- 2024-02-12T16:43:13.304 INFO:teuthology.orchestra.run.smithi033.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1', '-v', 'bootstrap', '--fsid', 'cf6ad642-c9c5-11ee-95b9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.33:6789]', '--skip-admin-label'] 2024-02-12T16:43:14.222 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: 4.8.3 2024-02-12T16:43:14.229 INFO:teuthology.orchestra.run.smithi033.stderr:Verifying podman|docker is present... 2024-02-12T16:43:14.253 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: 4.8.3 2024-02-12T16:43:14.261 INFO:teuthology.orchestra.run.smithi033.stderr:Verifying lvm2 is present... 2024-02-12T16:43:14.261 INFO:teuthology.orchestra.run.smithi033.stderr:Verifying time synchronization is in place... 2024-02-12T16:43:14.269 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-12T16:43:14.275 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: inactive 2024-02-12T16:43:14.282 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: enabled 2024-02-12T16:43:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: active 2024-02-12T16:43:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:Unit chronyd.service is enabled and running 2024-02-12T16:43:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:Repeating the final host check... 2024-02-12T16:43:14.310 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: 4.8.3 2024-02-12T16:43:14.321 INFO:teuthology.orchestra.run.smithi033.stderr:podman (/bin/podman) version 4.8.3 is present 2024-02-12T16:43:14.321 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl is present 2024-02-12T16:43:14.321 INFO:teuthology.orchestra.run.smithi033.stderr:lvcreate is present 2024-02-12T16:43:14.327 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-12T16:43:14.333 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: inactive 2024-02-12T16:43:14.339 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: enabled 2024-02-12T16:43:14.345 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: active 2024-02-12T16:43:14.346 INFO:teuthology.orchestra.run.smithi033.stderr:Unit chronyd.service is enabled and running 2024-02-12T16:43:14.346 INFO:teuthology.orchestra.run.smithi033.stderr:Host looks OK 2024-02-12T16:43:14.346 INFO:teuthology.orchestra.run.smithi033.stderr:Cluster fsid: cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:14.346 INFO:teuthology.orchestra.run.smithi033.stderr:Acquiring lock 140633424062952 on /run/cephadm/cf6ad642-c9c5-11ee-95b9-87774f69a715.lock 2024-02-12T16:43:14.346 INFO:teuthology.orchestra.run.smithi033.stderr:Lock 140633424062952 acquired on /run/cephadm/cf6ad642-c9c5-11ee-95b9-87774f69a715.lock 2024-02-12T16:43:14.347 INFO:teuthology.orchestra.run.smithi033.stderr:Verifying IP 172.21.15.33 port 6789 ... 2024-02-12T16:43:14.347 INFO:teuthology.orchestra.run.smithi033.stderr:Base mon IP(s) is [172.21.15.33:6789], mon addrv is [v1:172.21.15.33:6789] 2024-02-12T16:43:14.350 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.33 metric 100 2024-02-12T16:43:14.350 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.33 metric 100 2024-02-12T16:43:14.353 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-12T16:43:14.353 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-02-12T16:43:14.353 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1754sec hoplimit 64 pref medium 2024-02-12T16:43:14.356 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-12T16:43:14.356 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-12T16:43:14.356 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-12T16:43:14.357 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-12T16:43:14.357 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fea5:d294/64 scope link 2024-02-12T16:43:14.357 INFO:teuthology.orchestra.run.smithi033.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-12T16:43:14.357 INFO:teuthology.orchestra.run.smithi033.stderr:Mon IP `172.21.15.33` is in CIDR network `172.21.0.0/20` 2024-02-12T16:43:14.357 INFO:teuthology.orchestra.run.smithi033.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-02-12T16:43:14.358 INFO:teuthology.orchestra.run.smithi033.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-12T16:43:14.359 INFO:teuthology.orchestra.run.smithi033.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1... 2024-02-12T16:43:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1... 2024-02-12T16:43:14.637 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: Getting image source signatures 2024-02-12T16:43:14.642 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-12T16:43:14.642 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: Copying blob sha256:955bfa9dba4a4380f93be7209d227ce2b057a09dc4ade787f31ef12d8b977cea 2024-02-12T16:43:40.390 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: Copying config sha256:1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5 2024-02-12T16:43:40.539 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: Writing manifest to image destination 2024-02-12T16:43:40.836 INFO:teuthology.orchestra.run.smithi033.stderr:/bin/podman: 1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5 2024-02-12T16:43:41.481 INFO:teuthology.orchestra.run.smithi033.stderr:ceph: ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable) 2024-02-12T16:43:42.117 INFO:teuthology.orchestra.run.smithi033.stderr:Ceph version: ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable) 2024-02-12T16:43:42.117 INFO:teuthology.orchestra.run.smithi033.stderr:Extracting ceph user uid/gid from container image... 2024-02-12T16:43:42.628 INFO:teuthology.orchestra.run.smithi033.stderr:stat: 167 167 2024-02-12T16:43:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:Creating initial keys... 2024-02-12T16:43:43.815 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-authtool: AQC/SsplMVrhJxAASmWmrbAkUD0BSwEJVunnwA== 2024-02-12T16:43:45.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-authtool: AQDASsplIFaaNhAAF1ZOFBquYCCBAQKkrWiRBg== 2024-02-12T16:43:46.243 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-authtool: AQDCSsple9y8CBAA6keHAkZxJGH99Fox6XrDYQ== 2024-02-12T16:43:46.931 INFO:teuthology.orchestra.run.smithi033.stderr:Creating initial monmap... 2024-02-12T16:43:47.508 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-12T16:43:47.508 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:47.508 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-12T16:43:48.080 INFO:teuthology.orchestra.run.smithi033.stderr:monmaptool for a [v1:172.21.15.33:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-12T16:43:48.081 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/monmaptool: set fsid to cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:48.081 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-12T16:43:48.081 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:43:48.081 INFO:teuthology.orchestra.run.smithi033.stderr:Creating mon... 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.565+0000 7fb2bb845880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.565+0000 7fb2bb845880 1 imported monmap: 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: last_changed 2024-02-12T16:43:47.392898+0000 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: created 2024-02-12T16:43:47.392898+0000 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 0: v1:172.21.15.33:6789/0 mon.a 2024-02-12T16:43:48.665 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.689 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.565+0000 7fb2bb845880 0 /usr/bin/ceph-mon: set fsid to cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Compile date Feb 8 2024 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: DB SUMMARY 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.error_if_exists: 0 2024-02-12T16:43:48.690 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.create_if_missing: 1 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.env: 0x556386d7c080 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.fs: Posix File System 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.info_log: 0x556387d61a80 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.statistics: (nil) 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.use_fsync: 0 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T16:43:48.691 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.db_log_dir: 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T16:43:48.692 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.write_buffer_manager: 0x556387d67980 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.unordered_write: 0 2024-02-12T16:43:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.row_cache: None 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.wal_filter: None 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.two_write_queues: 0 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.atomic_flush: 0 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T16:43:48.694 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.max_open_files: -1 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T16:43:48.695 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Compression algorithms supported: 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: kZSTD supported: 0 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: kXpressCompression supported: 0 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: kZlibCompression supported: 1 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.566+0000 7fb2bb845880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-12T16:43:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.merge_operator: 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_filter: None 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556387c77d30) 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T16:43:48.697 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: block_cache: 0x556387cacd10 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-12T16:43:48.698 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-12T16:43:48.699 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compression: NoCompression 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.num_levels: 7 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T16:43:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T16:43:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T16:43:48.702 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.table_properties_collectors: 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T16:43:48.703 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.bloom_locality: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.ttl: 2592000 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.567+0000 7fb2bb845880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.568+0000 7fb2bb845880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.568+0000 7fb2bb845880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-12T16:43:48.704 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.570+0000 7fb2bb845880 4 rocksdb: DB pointer 0x556387d75800 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.571+0000 7fb2a46b7700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.571+0000 7fb2a46b7700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:43:48.705 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:43:48.706 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:43:48.707 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.572+0000 7fb2bb845880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.572+0000 7fb2bb845880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-12T16:43:48.708 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:43:48.572+0000 7fb2bb845880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-02-12T16:43:49.213 INFO:teuthology.orchestra.run.smithi033.stderr:create mon.a on 2024-02-12T16:43:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-12T16:43:49.331 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-12T16:43:49.428 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715.target. 2024-02-12T16:43:49.428 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715.target. 2024-02-12T16:43:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Failed to reset failed state of unit ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.a.service: Unit ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.a.service not loaded. 2024-02-12T16:43:49.648 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Created symlink /etc/systemd/system/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715.target.wants/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@.service. 2024-02-12T16:43:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:49 smithi033 systemd[1]: Starting Ceph mon.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:43:50.333 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: disabled 2024-02-12T16:43:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: inactive 2024-02-12T16:43:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:firewalld.service is not enabled 2024-02-12T16:43:50.341 INFO:teuthology.orchestra.run.smithi033.stderr:Not possible to enable service . firewalld.service is not available 2024-02-12T16:43:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:Waiting for mon to start... 2024-02-12T16:43:50.342 INFO:teuthology.orchestra.run.smithi033.stderr:Waiting for mon... 2024-02-12T16:43:50.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: cluster 2024-02-12T16:43:50.282581+0000 mon.a (mon.0) 0 : [INF] mkfs cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:50.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: cluster 2024-02-12T16:43:50.277960+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-12T16:43:50.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:50 smithi033 bash[39006]: 818c9db08c4ac7402fcaa7c3db44ec3b8265425c14f4a66af1384994880b480a 2024-02-12T16:43:50.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:50 smithi033 systemd[1]: Started Ceph mon.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:43:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: cluster: 2024-02-12T16:43:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: id: cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-12T16:43:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:43:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: services: 2024-02-12T16:43:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.659314s) 2024-02-12T16:43:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-12T16:43:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-12T16:43:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:43:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: data: 2024-02-12T16:43:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-12T16:43:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-12T16:43:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-12T16:43:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: pgs: 2024-02-12T16:43:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:43:51.588 INFO:teuthology.orchestra.run.smithi033.stderr:mon is available 2024-02-12T16:43:51.588 INFO:teuthology.orchestra.run.smithi033.stderr:Assimilating anything we can from ceph.conf... 2024-02-12T16:43:51.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: cluster 2024-02-12T16:43:50.281213+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-12T16:43:51.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: cluster 2024-02-12T16:43:50.281626+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.33:6789/0} removed_ranks: {} 2024-02-12T16:43:51.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: cluster 2024-02-12T16:43:50.284482+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-12T16:43:51.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: cluster 2024-02-12T16:43:50.288101+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-12T16:43:51.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: cluster 2024-02-12T16:43:50.288509+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-12T16:43:51.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: audit 2024-02-12T16:43:50.940494+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.33:0/3988228278' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-12T16:43:52.196 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:43:52.196 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: [global] 2024-02-12T16:43:52.196 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: fsid = cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:52.196 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mon_host = [v1:172.21.15.33:6789] 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: [mgr] 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: [osd] 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-12T16:43:52.197 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-12T16:43:52.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: audit 2024-02-12T16:43:52.190245+0000 mon.a (mon.0) 8 : audit [INF] from='client.? v1:172.21.15.33:0/2450876239' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-12T16:43:52.511 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: audit 2024-02-12T16:43:52.193119+0000 mon.a (mon.0) 9 : audit [INF] from='client.? v1:172.21.15.33:0/2450876239' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-12T16:43:52.730 INFO:teuthology.orchestra.run.smithi033.stderr:Generating new minimal ceph.conf... 2024-02-12T16:43:53.702 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: audit 2024-02-12T16:43:53.350337+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.33:0/3020514680' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:43:53.971 INFO:teuthology.orchestra.run.smithi033.stderr:Restarting the monitor... 2024-02-12T16:43:54.252 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:53 smithi033 systemd[1]: Stopping Ceph mon.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:43:54.252 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:54 smithi033 bash[39347]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon.a" found: no such container 2024-02-12T16:43:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: debug 2024-02-12T16:43:54.250+0000 7f69bc6f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T16:43:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39021]: debug 2024-02-12T16:43:54.250+0000 7f69bc6f2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-12T16:43:54.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:54 smithi033 bash[39387]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a 2024-02-12T16:43:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:54 smithi033 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.a.service: Succeeded. 2024-02-12T16:43:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:54 smithi033 systemd[1]: Stopped Ceph mon.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:43:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:54 smithi033 systemd[1]: Starting Ceph mon.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:43:56.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.859+0000 7fd9813e8880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T16:43:56.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.859+0000 7fd9813e8880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-12T16:43:56.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.859+0000 7fd9813e8880 0 pidfile_write: ignore empty --pid-file 2024-02-12T16:43:56.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.864+0000 7fd9813e8880 0 load: jerasure load: lrc load: isa 2024-02-12T16:43:56.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.864+0000 7fd9813e8880 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T16:43:56.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Compile date Feb 8 2024 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: DB SUMMARY 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: CURRENT file: CURRENT 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 90837 ; 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.error_if_exists: 0 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.create_if_missing: 0 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.env: 0x558dd181e080 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.fs: Posix File System 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.info_log: 0x558dd3c00200 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.statistics: (nil) 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.use_fsync: 0 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T16:43:56.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.db_log_dir: 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T16:43:56.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.write_buffer_manager: 0x558dd48f2390 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T16:43:56.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.unordered_write: 0 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.row_cache: None 2024-02-12T16:43:56.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.wal_filter: None 2024-02-12T16:43:56.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T16:43:56.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T16:43:56.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T16:43:56.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.two_write_queues: 0 2024-02-12T16:43:56.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T16:43:56.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.atomic_flush: 0 2024-02-12T16:43:56.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T16:43:56.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T16:43:56.288 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_open_files: -1 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Compression algorithms supported: 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: kZSTD supported: 0 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: kXpressCompression supported: 0 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: kZlibCompression supported: 1 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T16:43:56.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.merge_operator: 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_filter: None 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558dd3ba5d28) 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cache_index_and_filter_blocks: 1 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T16:43:56.290 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: pin_top_level_index_and_filter: 1 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: index_type: 0 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: data_block_index_type: 0 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: index_shortening: 1 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: data_block_hash_table_util_ratio: 0.750000 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: hash_index_allow_collision: 1 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: checksum: 1 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: no_block_cache: 0 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: block_cache: 0x558dd3bdaf10 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: block_cache_name: BinnedLRUCache 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: block_cache_options: 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: capacity : 536870912 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: num_shard_bits : 4 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: strict_capacity_limit : 0 2024-02-12T16:43:56.291 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: high_pri_pool_ratio: 0.000 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: block_cache_compressed: (nil) 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: persistent_cache: (nil) 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: block_size: 4096 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: block_size_deviation: 10 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: block_restart_interval: 16 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: index_block_restart_interval: 1 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: metadata_block_size: 4096 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: partition_filters: 0 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: use_delta_encoding: 1 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: whole_key_filtering: 1 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: verify_compression: 0 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: read_amp_bytes_per_bit: 0 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: format_version: 2 2024-02-12T16:43:56.292 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: enable_index_compression: 1 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: block_align: 0 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compression: NoCompression 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.num_levels: 7 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T16:43:56.294 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T16:43:56.295 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T16:43:56.296 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T16:43:56.296 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T16:43:56.296 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.table_properties_collectors: 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T16:43:56.297 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.bloom_locality: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.ttl: 2592000 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.865+0000 7fd9813e8880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.866+0000 7fd9813e8880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.866+0000 7fd9813e8880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-12T16:43:56.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.866+0000 7fd9813e8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756235866698, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.866+0000 7fd9813e8880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.866+0000 7fd9813e8880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.867+0000 7fd9813e8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756235867711, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 86703, "table_properties": {"data_size": 84958, "index_size": 350, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 11269, "raw_average_key_size": 48, "raw_value_size": 78711, "raw_average_value_size": 342, "num_data_blocks": 11, "num_entries": 230, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707756235, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.867+0000 7fd9813e8880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.867+0000 7fd9813e8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756235868190, "job": 1, "event": "recovery_finished"} 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.868+0000 7fd9813e8880 4 rocksdb: DB pointer 0x558dd3ca3800 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.868+0000 7fd9681d9700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.868+0000 7fd9681d9700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ** DB Stats ** 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T16:43:56.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ** Compaction Stats [default] ** 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: L0 2/0 86.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Sum 2/0 86.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ** Compaction Stats [default] ** 2024-02-12T16:43:56.300 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Cumulative compaction: 0.00 GB write, 23.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Interval compaction: 0.00 GB write, 23.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ** Compaction Stats [default] ** 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: L0 2/0 86.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Sum 2/0 86.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ** Compaction Stats [default] ** 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.1 0.00 0.00 1 0.001 0 0 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Cumulative compaction: 0.00 GB write, 23.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.869+0000 7fd9813e8880 0 starting mon.a rank 0 at public addrs v1:172.21.15.33:6789/0 at bind addrs v1:172.21.15.33:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.870+0000 7fd9813e8880 1 mon.a@-1(???) e1 preinit fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.870+0000 7fd9813e8880 0 mon.a@-1(???).mds e1 new map 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.870+0000 7fd9813e8880 0 mon.a@-1(???).mds e1 print_map 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: e1 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-12T16:43:56.303 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: legacy client fscid: -1 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: No filesystems configured 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.870+0000 7fd9813e8880 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.870+0000 7fd9813e8880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.870+0000 7fd9813e8880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.870+0000 7fd9813e8880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:43:53.350337+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.33:0/3020514680' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:43:55.870+0000 7fd9813e8880 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:43:55.887003+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:43:55.887055+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.33:6789/0} removed_ranks: {} 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:43:55.887468+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:43:55.887502+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-12T16:43:56.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:43:55.887918+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-12T16:43:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-12T16:43:57.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:56 smithi033 bash[39506]: d0c4b09de5c13421610014b3c06dbb913884e7071bdfacbf3086ccd865d3be7b 2024-02-12T16:43:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:56 smithi033 systemd[1]: Started Ceph mon.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:43:59.456 INFO:teuthology.orchestra.run.smithi033.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-12T16:43:59.457 INFO:teuthology.orchestra.run.smithi033.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:43:59.457 INFO:teuthology.orchestra.run.smithi033.stderr:Creating mgr... 2024-02-12T16:43:59.457 INFO:teuthology.orchestra.run.smithi033.stderr:Verifying port 9283 ... 2024-02-12T16:43:59.598 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Failed to reset failed state of unit ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.y.service: Unit ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.y.service not loaded. 2024-02-12T16:43:59.605 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: Created symlink /etc/systemd/system/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715.target.wants/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@.service. 2024-02-12T16:43:59.709 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:43:59 smithi033 systemd[1]: Starting Ceph mgr.y for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:44:00.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:43:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:43:58.862534+0000 mon.a (mon.0) 6 : audit [INF] from='client.? v1:172.21.15.33:0/2492572326' entity='client.admin' 2024-02-12T16:44:00.353 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: disabled 2024-02-12T16:44:00.361 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: inactive 2024-02-12T16:44:00.362 INFO:teuthology.orchestra.run.smithi033.stderr:firewalld.service is not enabled 2024-02-12T16:44:00.362 INFO:teuthology.orchestra.run.smithi033.stderr:Not possible to enable service . firewalld.service is not available 2024-02-12T16:44:00.371 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: disabled 2024-02-12T16:44:00.379 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: inactive 2024-02-12T16:44:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:firewalld.service is not enabled 2024-02-12T16:44:00.380 INFO:teuthology.orchestra.run.smithi033.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-12T16:44:00.381 INFO:teuthology.orchestra.run.smithi033.stderr:Waiting for mgr to start... 2024-02-12T16:44:00.381 INFO:teuthology.orchestra.run.smithi033.stderr:Waiting for mgr... 2024-02-12T16:44:00.603 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:00 smithi033 bash[39781]: 8dca4f58c25a945fb7081f52907b79d1cde9c9223341823d5d7cf1f9a4259fd0 2024-02-12T16:44:00.604 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:00 smithi033 systemd[1]: Started Ceph mgr.y for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:44:00.604 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:00.424+0000 7f92dbfe4680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T16:44:00.604 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:00.508+0000 7f92dbfe4680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T16:44:01.020 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:44:01.022 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: { 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "health": { 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 0 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "a" 2024-02-12T16:44:01.023 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T16:44:01.024 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T16:44:01.028 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "available": false, 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "iostat", 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "nfs", 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "restful" 2024-02-12T16:44:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "modified": "2024-02-12T16:43:50.282942+0000", 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T16:44:01.030 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: } 2024-02-12T16:44:01.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:00.922+0000 7f92dbfe4680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T16:44:01.687 INFO:teuthology.orchestra.run.smithi033.stderr:mgr not available, waiting (1/15)... 2024-02-12T16:44:02.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:01.018209+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.33:0/2415378739' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T16:44:02.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:01.729+0000 7f92dbfe4680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T16:44:02.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:01.811+0000 7f92dbfe4680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T16:44:02.527 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:02.031+0000 7f92dbfe4680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T16:44:03.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:03.019+0000 7f92dbfe4680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T16:44:03.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:03.151+0000 7f92dbfe4680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:44:03.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:03.232+0000 7f92dbfe4680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T16:44:03.777 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:03.325+0000 7f92dbfe4680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T16:44:03.777 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:03.527+0000 7f92dbfe4680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T16:44:04.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:03.821+0000 7f92dbfe4680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T16:44:04.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:03.927+0000 7f92dbfe4680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T16:44:04.343 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:44:04.343 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: { 2024-02-12T16:44:04.343 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T16:44:04.343 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "health": { 2024-02-12T16:44:04.344 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T16:44:04.344 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T16:44:04.344 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T16:44:04.344 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:04.344 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T16:44:04.344 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T16:44:04.344 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 0 2024-02-12T16:44:04.344 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "a" 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:04.345 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T16:44:04.346 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T16:44:04.346 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T16:44:04.349 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T16:44:04.349 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:04.350 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "available": false, 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "iostat", 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "nfs", 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "restful" 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "modified": "2024-02-12T16:43:50.282942+0000", 2024-02-12T16:44:04.351 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:44:04.352 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:04.352 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T16:44:04.352 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: } 2024-02-12T16:44:04.695 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:04.339136+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? v1:172.21.15.33:0/788001247' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T16:44:05.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:04.706+0000 7f92dbfe4680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T16:44:05.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:04.790+0000 7f92dbfe4680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T16:44:05.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:04.885+0000 7f92dbfe4680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T16:44:05.048 INFO:teuthology.orchestra.run.smithi033.stderr:mgr not available, waiting (2/15)... 2024-02-12T16:44:05.336 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:05.065+0000 7f92dbfe4680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T16:44:05.610 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:05.333+0000 7f92dbfe4680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T16:44:05.611 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:05.470+0000 7f92dbfe4680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T16:44:05.611 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:05.608+0000 7f92dbfe4680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:44:05.874 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:05.785+0000 7f92dbfe4680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T16:44:06.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:05.871+0000 7f92dbfe4680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T16:44:07.258 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:05.886141+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:05.891326+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00524048s) 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.892367+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.892551+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.892670+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.892792+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.892916+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.893041+0000 mon.a (mon.0) 16 : audit [DBG] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.893183+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.893330+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:05.906709+0000 mon.a (mon.0) 19 : cluster [INF] Manager daemon y is now available 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.915212+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' 2024-02-12T16:44:07.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.921543+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:44:07.260 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.923675+0000 mon.a (mon.0) 22 : audit [INF] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:44:07.260 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.927179+0000 mon.a (mon.0) 23 : audit [INF] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' 2024-02-12T16:44:07.260 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:05.929127+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 v1:172.21.15.33:0/617940426' entity='mgr.y' 2024-02-12T16:44:07.812 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:44:07.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: { 2024-02-12T16:44:07.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T16:44:07.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "health": { 2024-02-12T16:44:07.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T16:44:07.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T16:44:07.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T16:44:07.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:07.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 0 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "a" 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:07.814 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T16:44:07.815 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T16:44:07.815 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T16:44:07.818 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T16:44:07.818 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T16:44:07.818 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T16:44:07.818 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:07.818 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T16:44:07.819 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "available": true, 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "iostat", 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "nfs", 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "restful" 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ], 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "modified": "2024-02-12T16:43:50.282942+0000", 2024-02-12T16:44:07.820 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:44:07.821 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: }, 2024-02-12T16:44:07.821 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T16:44:07.821 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: } 2024-02-12T16:44:08.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:06.894715+0000 mon.a (mon.0) 25 : cluster [DBG] mgrmap e3: y(active, since 1.00863s) 2024-02-12T16:44:08.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:07.805828+0000 mon.a (mon.0) 26 : audit [DBG] from='client.? v1:172.21.15.33:0/3135154261' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T16:44:08.431 INFO:teuthology.orchestra.run.smithi033.stderr:mgr is available 2024-02-12T16:44:09.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:07.895418+0000 mon.a (mon.0) 27 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-02-12T16:44:09.925 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:44:09.925 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: [global] 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: fsid = cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: [mgr] 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: [osd] 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-12T16:44:09.926 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-12T16:44:10.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:09.887261+0000 mon.a (mon.0) 28 : audit [INF] from='client.? v1:172.21.15.33:0/1553937221' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-12T16:44:14.879 INFO:teuthology.orchestra.run.smithi033.stderr:Enabling cephadm module... 2024-02-12T16:44:17.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:17.100152+0000 mon.a (mon.0) 29 : audit [INF] from='client.? v1:172.21.15.33:0/2939986764' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-12T16:44:17.514 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ignoring --setuser ceph since I am not root 2024-02-12T16:44:17.514 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ignoring --setgroup ceph since I am not root 2024-02-12T16:44:17.514 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:17.313+0000 7fef95c52680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T16:44:17.514 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:17.397+0000 7fef95c52680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T16:44:18.040 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:17.778+0000 7fef95c52680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T16:44:18.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:17.153264+0000 mon.a (mon.0) 30 : audit [INF] from='client.? v1:172.21.15.33:0/2939986764' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-12T16:44:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:17.153306+0000 mon.a (mon.0) 31 : cluster [DBG] mgrmap e5: y(active, since 11s) 2024-02-12T16:44:18.643 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: { 2024-02-12T16:44:18.643 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-12T16:44:18.643 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "available": true, 2024-02-12T16:44:18.643 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-12T16:44:18.643 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-12T16:44:18.643 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: } 2024-02-12T16:44:18.891 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:18.583+0000 7fef95c52680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T16:44:18.892 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:18.669+0000 7fef95c52680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T16:44:19.158 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:18.889+0000 7fef95c52680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T16:44:19.334 INFO:teuthology.orchestra.run.smithi033.stderr:Waiting for the mgr to restart... 2024-02-12T16:44:19.334 INFO:teuthology.orchestra.run.smithi033.stderr:Waiting for mgr epoch 5... 2024-02-12T16:44:19.523 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:18.640838+0000 mon.a (mon.0) 32 : audit [DBG] from='client.? v1:172.21.15.33:0/3172579754' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-12T16:44:20.200 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:19.889+0000 7fef95c52680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T16:44:20.200 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:20.023+0000 7fef95c52680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:44:20.200 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:20.104+0000 7fef95c52680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T16:44:20.200 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:20.197+0000 7fef95c52680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T16:44:20.527 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:20.400+0000 7fef95c52680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T16:44:21.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:20.691+0000 7fef95c52680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T16:44:21.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:20.793+0000 7fef95c52680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T16:44:21.908 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:21.547+0000 7fef95c52680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T16:44:21.908 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:21.631+0000 7fef95c52680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T16:44:21.908 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:21.726+0000 7fef95c52680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T16:44:21.909 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:21.906+0000 7fef95c52680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T16:44:22.451 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:22.174+0000 7fef95c52680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T16:44:22.451 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:22.310+0000 7fef95c52680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T16:44:22.451 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:22.448+0000 7fef95c52680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:44:22.715 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:22.626+0000 7fef95c52680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T16:44:23.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:22.713+0000 7fef95c52680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:22.727226+0000 mon.a (mon.0) 33 : cluster [INF] Active manager daemon y restarted 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:22.727435+0000 mon.a (mon.0) 34 : cluster [INF] Activating manager daemon y 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:22.729994+0000 mon.a (mon.0) 35 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:22.730088+0000 mon.a (mon.0) 36 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00273987s) 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.731146+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.731726+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.732517+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.732677+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.732782+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:44:23.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:22.746811+0000 mon.a (mon.0) 42 : cluster [INF] Manager daemon y is now available 2024-02-12T16:44:23.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.760314+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:23.741 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: { 2024-02-12T16:44:23.741 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-12T16:44:23.741 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "initialized": true 2024-02-12T16:44:23.741 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: } 2024-02-12T16:44:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:44:22.757674+0000 mgr.y (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-12T16:44:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.849415+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.850417+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.851785+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.853361+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.868032+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:44:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:22.871030+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:44:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:23.733227+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e7: y(active, since 1.00587s) 2024-02-12T16:44:24.424 INFO:teuthology.orchestra.run.smithi033.stderr:mgr epoch 5 is available 2024-02-12T16:44:24.424 INFO:teuthology.orchestra.run.smithi033.stderr:Setting orchestrator backend to cephadm... 2024-02-12T16:44:25.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:23.732052+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 v1:172.21.15.33:0/351476973' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-12T16:44:25.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:23.739388+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14120 v1:172.21.15.33:0/351476973' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-12T16:44:26.089 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:24.852338+0000 mon.a (mon.0) 51 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-02-12T16:44:26.090 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:25.165666+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14126 v1:172.21.15.33:0/1834109787' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:26.090 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:25.170618+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:26.090 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:25.227479+0000 mon.a (mon.0) 53 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:26.526 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: value unchanged 2024-02-12T16:44:27.212 INFO:teuthology.orchestra.run.smithi033.stderr:Generating ssh key... 2024-02-12T16:44:27.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:26.524813+0000 mgr.y (mgr.14116) 5 : audit [DBG] from='client.14128 v1:172.21.15.33:0/638710483' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:28.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:27.921196+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 v1:172.21.15.33:0/1553270577' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:44:27.921405+0000 mgr.y (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: Generating public/private rsa key pair. 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: Your identification has been saved in /tmp/tmpxhj_pp5l/key. 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: Your public key has been saved in /tmp/tmpxhj_pp5l/key.pub. 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: The key fingerprint is: 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: SHA256:q9fadZAhsYONkY+vdMUqz8A/Qi6RQUyXbzg6937uSdw ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: The key's randomart image is: 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: +---[RSA 3072]----+ 2024-02-12T16:44:28.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | o. oo. | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | o.o= o | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | . o==.. | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | . + +ooo | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | =S+ oo | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | = B.+. o | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | B.@ + E | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | ..=.Bo.o | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: | .o.+o=+ | 2024-02-12T16:44:28.529 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: +----[SHA256]-----+ 2024-02-12T16:44:29.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:28.272350+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:28.274712+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:29.574 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChSxx/ikcCathouuIuu0YW9cvnKTtfoVLQAga5tLlreODjYpRGV25DGQWe5HxH2Rqca2/6TA/JHco54ounjJFF0PyACUg/G8IjzUbZOEWOxaEn9FpTcssbtNDAxcVBPmyugyzFWtdby0ZLk0GQdT258vjKnFGvnIhPhzCDAEr0yGaN/eSFUIPsVsfQQaCXEqzF9yPcFZEfi7z/QiTtlhCdtJPEFZF2s9OH6y8tNjTIJPF2YtX8ZY9gxHr/9yVff/3+hOKkdmrk4cR73SsS2nDsHRNVk3oxIQIUXDJl1gaVyPoifl3T76r7kXTt6usOuSwzcBDoJ+zRbFoCiZAWrXo0/PRYBo1etCEUq067V0VFEW4hdLNuKsggStE55904Y40vylWxMtPAsvO7C9oGZZrU4NJt1dftlk0zLIvdtrdxoZUfuvGdEPEVqqh/Ye2ewLn29//TQhNH2E4BeVQ8C7hDYeGLeIkdwO3+0y96VqQFQH4JXpgdyd+Ww/dtR8Y0Bz8= ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:44:30.176 INFO:teuthology.orchestra.run.smithi033.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-12T16:44:30.176 INFO:teuthology.orchestra.run.smithi033.stderr:Adding key to root@localhost authorized_keys... 2024-02-12T16:44:30.177 INFO:teuthology.orchestra.run.smithi033.stderr:Adding host smithi033... 2024-02-12T16:44:30.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:29.572732+0000 mgr.y (mgr.14116) 8 : audit [DBG] from='client.14132 v1:172.21.15.33:0/153238077' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:31.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [79B blob data] 2024-02-12T16:44:31.574 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: Added host 'smithi033' with addr '172.21.15.33' 2024-02-12T16:44:32.225 INFO:teuthology.orchestra.run.smithi033.stderr:Deploying unmanaged mon service... 2024-02-12T16:44:32.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:30.950399+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 v1:172.21.15.33:0/3059534176' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi033", "addr": "172.21.15.33", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:32.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:31.571781+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:32.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:31.604865+0000 mon.a (mon.0) 57 : audit [DBG] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:33.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:44:31.572058+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Added host smithi033 2024-02-12T16:44:33.281 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-12T16:44:33.777 INFO:teuthology.orchestra.run.smithi033.stderr:Deploying unmanaged mgr service... 2024-02-12T16:44:34.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:33.276234+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 v1:172.21.15.33:0/2401564864' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:34.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:44:33.277486+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-12T16:44:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:33.279837+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:34.813 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-12T16:44:36.066 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:34.808948+0000 mgr.y (mgr.14116) 13 : audit [DBG] from='client.14138 v1:172.21.15.33:0/2050601742' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:36.066 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:44:34.809848+0000 mgr.y (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-12T16:44:36.066 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:34.811974+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:36.067 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:35.372790+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:36.152143+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:36.617899+0000 mon.a (mon.0) 62 : audit [INF] from='client.? v1:172.21.15.33:0/349624019' entity='client.admin' 2024-02-12T16:44:39.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:38.012361+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14116 v1:172.21.15.33:0/83965582' entity='mgr.y' 2024-02-12T16:44:39.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:38.496750+0000 mon.a (mon.0) 64 : audit [INF] from='client.? v1:172.21.15.33:0/1976545857' entity='client.admin' 2024-02-12T16:44:39.858 INFO:teuthology.orchestra.run.smithi033.stderr:Enabling the dashboard module... 2024-02-12T16:44:41.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:40.660751+0000 mon.a (mon.0) 65 : audit [INF] from='client.? v1:172.21.15.33:0/2440722062' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-12T16:44:41.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ignoring --setuser ceph since I am not root 2024-02-12T16:44:41.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ignoring --setgroup ceph since I am not root 2024-02-12T16:44:41.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:40.872+0000 7f3996a98680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T16:44:41.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:40.957+0000 7f3996a98680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T16:44:41.658 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:41.336+0000 7f3996a98680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T16:44:42.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:40.713863+0000 mon.a (mon.0) 66 : audit [INF] from='client.? v1:172.21.15.33:0/2440722062' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-12T16:44:42.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:40.713912+0000 mon.a (mon.0) 67 : cluster [DBG] mgrmap e9: y(active, since 17s) 2024-02-12T16:44:42.247 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: { 2024-02-12T16:44:42.247 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-12T16:44:42.247 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "available": true, 2024-02-12T16:44:42.247 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-12T16:44:42.248 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-12T16:44:42.248 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: } 2024-02-12T16:44:42.452 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:42.144+0000 7f3996a98680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T16:44:42.452 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:42.229+0000 7f3996a98680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T16:44:42.718 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:42.451+0000 7f3996a98680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T16:44:43.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:42.245220+0000 mon.a (mon.0) 68 : audit [DBG] from='client.? v1:172.21.15.33:0/4070423693' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-12T16:44:43.777 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:43.440+0000 7f3996a98680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T16:44:43.777 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:43.574+0000 7f3996a98680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:44:43.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:43.657+0000 7f3996a98680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T16:44:44.249 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:43.751+0000 7f3996a98680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T16:44:44.250 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:43.954+0000 7f3996a98680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T16:44:44.312 INFO:teuthology.orchestra.run.smithi033.stderr:Waiting for the mgr to restart... 2024-02-12T16:44:44.312 INFO:teuthology.orchestra.run.smithi033.stderr:Waiting for mgr epoch 9... 2024-02-12T16:44:44.527 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:44.248+0000 7f3996a98680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T16:44:44.527 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:44.363+0000 7f3996a98680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T16:44:45.491 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:45.128+0000 7f3996a98680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T16:44:45.491 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:45.213+0000 7f3996a98680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T16:44:45.491 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:45.308+0000 7f3996a98680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T16:44:45.491 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:45.489+0000 7f3996a98680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T16:44:46.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:45.758+0000 7f3996a98680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T16:44:46.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:45.895+0000 7f3996a98680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T16:44:46.301 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:46.035+0000 7f3996a98680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:44:46.301 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:46.213+0000 7f3996a98680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T16:44:46.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:44:46.300+0000 7f3996a98680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T16:44:46.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [79B blob data] 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:46.313331+0000 mon.a (mon.0) 69 : cluster [INF] Active manager daemon y restarted 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:46.313570+0000 mon.a (mon.0) 70 : cluster [INF] Activating manager daemon y 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:46.316469+0000 mon.a (mon.0) 71 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:46.316574+0000 mon.a (mon.0) 72 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00309853s) 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.317713+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.318215+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.319013+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.319231+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.319443+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:46.332760+0000 mon.a (mon.0) 78 : cluster [INF] Manager daemon y is now available 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.431620+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.433050+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:47.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.457110+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:44:47.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:46.460870+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:44:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: { 2024-02-12T16:44:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-12T16:44:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: "initialized": true 2024-02-12T16:44:47.328 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: } 2024-02-12T16:44:48.310 INFO:teuthology.orchestra.run.smithi033.stderr:mgr epoch 9 is available 2024-02-12T16:44:48.311 INFO:teuthology.orchestra.run.smithi033.stderr:Generating a dashboard self-signed certificate... 2024-02-12T16:44:48.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:47.319613+0000 mgr.y (mgr.14146) 1 : audit [DBG] from='client.14150 v1:172.21.15.33:0/683737490' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-12T16:44:48.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:47.320562+0000 mon.a (mon.0) 83 : cluster [DBG] mgrmap e11: y(active, since 1.00709s) 2024-02-12T16:44:48.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:47.326265+0000 mgr.y (mgr.14146) 2 : audit [DBG] from='client.14150 v1:172.21.15.33:0/683737490' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-12T16:44:48.578 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:47.722772+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:44:49.211 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-12T16:44:50.198 INFO:teuthology.orchestra.run.smithi033.stderr:Creating initial admin user... 2024-02-12T16:44:50.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:49.172496+0000 mgr.y (mgr.14146) 3 : audit [DBG] from='client.14156 v1:172.21.15.33:0/333252406' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:50.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:49.204346+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:44:50.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:49.209662+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:44:50.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:49.323887+0000 mon.a (mon.0) 87 : cluster [DBG] mgrmap e12: y(active, since 3s) 2024-02-12T16:44:51.191 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$onnWKInm0prqcFxBUIfUsutPoAyDT/MZeoctCtL3ekUH8A122gwUG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707756291, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-12T16:44:51.894 INFO:teuthology.orchestra.run.smithi033.stderr:Fetching dashboard port number... 2024-02-12T16:44:52.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:50.943940+0000 mgr.y (mgr.14146) 4 : audit [DBG] from='client.14158 v1:172.21.15.33:0/1869301987' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:44:52.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:51.189186+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:44:52.661 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: 8443 2024-02-12T16:44:53.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:52.659840+0000 mon.a (mon.0) 89 : audit [DBG] from='client.? v1:172.21.15.33:0/1339636072' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-12T16:44:53.379 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: disabled 2024-02-12T16:44:53.388 INFO:teuthology.orchestra.run.smithi033.stderr:systemctl: inactive 2024-02-12T16:44:53.389 INFO:teuthology.orchestra.run.smithi033.stderr:firewalld.service is not enabled 2024-02-12T16:44:53.389 INFO:teuthology.orchestra.run.smithi033.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-12T16:44:53.393 INFO:teuthology.orchestra.run.smithi033.stderr:Ceph Dashboard is now available at: 2024-02-12T16:44:53.393 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:53.393 INFO:teuthology.orchestra.run.smithi033.stderr: URL: https://smithi033.front.sepia.ceph.com:8443/ 2024-02-12T16:44:53.393 INFO:teuthology.orchestra.run.smithi033.stderr: User: admin 2024-02-12T16:44:53.393 INFO:teuthology.orchestra.run.smithi033.stderr: Password: m5pklpj1iw 2024-02-12T16:44:53.393 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:53.393 INFO:teuthology.orchestra.run.smithi033.stderr:Enabling autotune for osd_memory_target 2024-02-12T16:44:54.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:44:53.191288+0000 mon.a (mon.0) 90 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-02-12T16:44:55.828 INFO:teuthology.orchestra.run.smithi033.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-12T16:44:56.601 INFO:teuthology.orchestra.run.smithi033.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-12T16:44:56.601 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:56.601 INFO:teuthology.orchestra.run.smithi033.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:44:56.601 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:56.601 INFO:teuthology.orchestra.run.smithi033.stderr:Or, if you are only running a single cluster on this host: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: ceph telemetry on 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr:For more information see: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T16:44:56.602 INFO:teuthology.orchestra.run.smithi033.stderr:Bootstrap complete. 2024-02-12T16:44:56.648 INFO:tasks.cephadm:Fetching config... 2024-02-12T16:44:56.648 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:44:56.649 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-12T16:44:56.670 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-12T16:44:56.670 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:44:56.670 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-12T16:44:56.729 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-12T16:44:56.729 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:44:56.729 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-12T16:44:56.799 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-12T16:44:56.799 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:44:56.799 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-12T16:44:56.858 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-12T16:44:56.859 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChSxx/ikcCathouuIuu0YW9cvnKTtfoVLQAga5tLlreODjYpRGV25DGQWe5HxH2Rqca2/6TA/JHco54ounjJFF0PyACUg/G8IjzUbZOEWOxaEn9FpTcssbtNDAxcVBPmyugyzFWtdby0ZLk0GQdT258vjKnFGvnIhPhzCDAEr0yGaN/eSFUIPsVsfQQaCXEqzF9yPcFZEfi7z/QiTtlhCdtJPEFZF2s9OH6y8tNjTIJPF2YtX8ZY9gxHr/9yVff/3+hOKkdmrk4cR73SsS2nDsHRNVk3oxIQIUXDJl1gaVyPoifl3T76r7kXTt6usOuSwzcBDoJ+zRbFoCiZAWrXo0/PRYBo1etCEUq067V0VFEW4hdLNuKsggStE55904Y40vylWxMtPAsvO7C9oGZZrU4NJt1dftlk0zLIvdtrdxoZUfuvGdEPEVqqh/Ye2ewLn29//TQhNH2E4BeVQ8C7hDYeGLeIkdwO3+0y96VqQFQH4JXpgdyd+Ww/dtR8Y0Bz8= ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-12T16:44:56.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:44:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:55.824504+0000 mon.a (mon.0) 91 : audit [INF] from='client.? v1:172.21.15.33:0/1854859702' entity='client.admin' 2024-02-12T16:44:56.941 INFO:teuthology.orchestra.run.smithi033.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChSxx/ikcCathouuIuu0YW9cvnKTtfoVLQAga5tLlreODjYpRGV25DGQWe5HxH2Rqca2/6TA/JHco54ounjJFF0PyACUg/G8IjzUbZOEWOxaEn9FpTcssbtNDAxcVBPmyugyzFWtdby0ZLk0GQdT258vjKnFGvnIhPhzCDAEr0yGaN/eSFUIPsVsfQQaCXEqzF9yPcFZEfi7z/QiTtlhCdtJPEFZF2s9OH6y8tNjTIJPF2YtX8ZY9gxHr/9yVff/3+hOKkdmrk4cR73SsS2nDsHRNVk3oxIQIUXDJl1gaVyPoifl3T76r7kXTt6usOuSwzcBDoJ+zRbFoCiZAWrXo0/PRYBo1etCEUq067V0VFEW4hdLNuKsggStE55904Y40vylWxMtPAsvO7C9oGZZrU4NJt1dftlk0zLIvdtrdxoZUfuvGdEPEVqqh/Ye2ewLn29//TQhNH2E4BeVQ8C7hDYeGLeIkdwO3+0y96VqQFQH4JXpgdyd+Ww/dtR8Y0Bz8= ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:44:56.953 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChSxx/ikcCathouuIuu0YW9cvnKTtfoVLQAga5tLlreODjYpRGV25DGQWe5HxH2Rqca2/6TA/JHco54ounjJFF0PyACUg/G8IjzUbZOEWOxaEn9FpTcssbtNDAxcVBPmyugyzFWtdby0ZLk0GQdT258vjKnFGvnIhPhzCDAEr0yGaN/eSFUIPsVsfQQaCXEqzF9yPcFZEfi7z/QiTtlhCdtJPEFZF2s9OH6y8tNjTIJPF2YtX8ZY9gxHr/9yVff/3+hOKkdmrk4cR73SsS2nDsHRNVk3oxIQIUXDJl1gaVyPoifl3T76r7kXTt6usOuSwzcBDoJ+zRbFoCiZAWrXo0/PRYBo1etCEUq067V0VFEW4hdLNuKsggStE55904Y40vylWxMtPAsvO7C9oGZZrU4NJt1dftlk0zLIvdtrdxoZUfuvGdEPEVqqh/Ye2ewLn29//TQhNH2E4BeVQ8C7hDYeGLeIkdwO3+0y96VqQFQH4JXpgdyd+Ww/dtR8Y0Bz8= ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-12T16:44:56.998 INFO:teuthology.orchestra.run.smithi094.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChSxx/ikcCathouuIuu0YW9cvnKTtfoVLQAga5tLlreODjYpRGV25DGQWe5HxH2Rqca2/6TA/JHco54ounjJFF0PyACUg/G8IjzUbZOEWOxaEn9FpTcssbtNDAxcVBPmyugyzFWtdby0ZLk0GQdT258vjKnFGvnIhPhzCDAEr0yGaN/eSFUIPsVsfQQaCXEqzF9yPcFZEfi7z/QiTtlhCdtJPEFZF2s9OH6y8tNjTIJPF2YtX8ZY9gxHr/9yVff/3+hOKkdmrk4cR73SsS2nDsHRNVk3oxIQIUXDJl1gaVyPoifl3T76r7kXTt6usOuSwzcBDoJ+zRbFoCiZAWrXo0/PRYBo1etCEUq067V0VFEW4hdLNuKsggStE55904Y40vylWxMtPAsvO7C9oGZZrU4NJt1dftlk0zLIvdtrdxoZUfuvGdEPEVqqh/Ye2ewLn29//TQhNH2E4BeVQ8C7hDYeGLeIkdwO3+0y96VqQFQH4JXpgdyd+Ww/dtR8Y0Bz8= ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:44:57.010 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-12T16:45:00.929 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-12T16:45:00.929 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-12T16:45:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:44:59.900177+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:00.222866+0000 mon.a (mon.0) 93 : audit [INF] from='client.? v1:172.21.15.33:0/1785838666' entity='client.admin' 2024-02-12T16:45:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:00.306091+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:45:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:00.309227+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:00.311181+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:45:01.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:00.314493+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:03.965 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi094 2024-02-12T16:45:03.965 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:45:03.965 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/etc/ceph/ceph.conf 2024-02-12T16:45:03.986 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:45:03.986 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:45:04.048 INFO:tasks.cephadm:Adding host smithi094 to orchestrator... 2024-02-12T16:45:04.049 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch host add smithi094 2024-02-12T16:45:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:03.373828+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:03.384461+0000 mon.a (mon.0) 99 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:45:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:03.385770+0000 mon.a (mon.0) 100 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:03.386312+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:45:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:45:03.387250+0000 mgr.y (mgr.14146) 6 : cephadm [INF] Updating smithi033:/etc/ceph/ceph.conf 2024-02-12T16:45:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:45:03.409251+0000 mgr.y (mgr.14146) 7 : cephadm [INF] Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:45:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:03.454110+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:03.456155+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:07.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:06.320618+0000 mgr.y (mgr.14146) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:08.320944+0000 mgr.y (mgr.14146) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:11.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:10.321273+0000 mgr.y (mgr.14146) 10 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:13.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:12.321546+0000 mgr.y (mgr.14146) 11 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:15.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:14.321875+0000 mgr.y (mgr.14146) 12 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:16.322249+0000 mgr.y (mgr.14146) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:18.322540+0000 mgr.y (mgr.14146) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:21.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:20.322799+0000 mgr.y (mgr.14146) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:22.323115+0000 mgr.y (mgr.14146) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:25.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:24.323506+0000 mgr.y (mgr.14146) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:26.323790+0000 mgr.y (mgr.14146) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:28.324157+0000 mgr.y (mgr.14146) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:30.324423+0000 mgr.y (mgr.14146) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:33.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:32.324746+0000 mgr.y (mgr.14146) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:34.325004+0000 mgr.y (mgr.14146) 22 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:37.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:36.325406+0000 mgr.y (mgr.14146) 23 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:37.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:45:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [89B blob data] 2024-02-12T16:45:37.476 INFO:teuthology.orchestra.run.smithi094.stdout:Added host 'smithi094' with addr '172.21.15.94' 2024-02-12T16:45:38.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:45:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [79B blob data] 2024-02-12T16:45:38.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-12T16:45:38.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:36.853993+0000 mgr.y (mgr.14146) 24 : audit [DBG] from='client.14170 v1:172.21.15.94:0/623239407' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi094", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:45:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:37.473235+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:45:37.473628+0000 mgr.y (mgr.14146) 25 : cephadm [INF] Added host smithi094 2024-02-12T16:45:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:37.515944+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:45:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:38.146445+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:38.325840+0000 mgr.y (mgr.14146) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:39.004416+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:40.001 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:45:40.001 INFO:teuthology.orchestra.run.smithi094.stdout:[{"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.94", "hostname": "smithi094", "labels": [], "status": ""}] 2024-02-12T16:45:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:39.998857+0000 mgr.y (mgr.14146) 27 : audit [DBG] from='client.14172 v1:172.21.15.94:0/64501309' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:45:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:40.326212+0000 mgr.y (mgr.14146) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:41.904 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-12T16:45:41.904 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-12T16:45:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:42.326526+0000 mgr.y (mgr.14146) 29 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:42.725204+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:43.263763+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:43.264543+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi094", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:45:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:43.265675+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:43.266496+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:45:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:43.309555+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:43.312171+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:44.734 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2024-02-12T16:45:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:45:43.267730+0000 mgr.y (mgr.14146) 30 : cephadm [INF] Updating smithi094:/etc/ceph/ceph.conf 2024-02-12T16:45:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:45:43.281749+0000 mgr.y (mgr.14146) 31 : cephadm [INF] Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:45:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:44.107745+0000 mon.a (mon.0) 115 : audit [INF] from='client.? v1:172.21.15.33:0/3820811956' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-12T16:45:45.319 INFO:tasks.cephadm:Adding mon.a on smithi033 2024-02-12T16:45:45.319 INFO:tasks.cephadm:Adding mon.c on smithi033 2024-02-12T16:45:45.319 INFO:tasks.cephadm:Adding mon.b on smithi094 2024-02-12T16:45:45.320 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch apply mon '3;smithi033:[v1:172.21.15.33:6789]=a;smithi033:[v1:172.21.15.33:6790]=c;smithi094:[v1:172.21.15.94:6789]=b' 2024-02-12T16:45:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:44.326804+0000 mgr.y (mgr.14146) 32 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:44.729223+0000 mon.a (mon.0) 116 : audit [INF] from='client.? v1:172.21.15.33:0/3820811956' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-12T16:45:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:44.729272+0000 mon.a (mon.0) 117 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:45:47.220 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled mon update... 2024-02-12T16:45:47.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:46.327151+0000 mgr.y (mgr.14146) 33 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:47.803 DEBUG:teuthology.orchestra.run.smithi033:mon.c> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.c.service 2024-02-12T16:45:47.806 DEBUG:teuthology.orchestra.run.smithi094:mon.b> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.b.service 2024-02-12T16:45:47.808 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-12T16:45:47.808 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:45:47.836 INFO:journalctl@ceph.mon.b.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:45:47.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:45:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:47.211820+0000 mgr.y (mgr.14146) 34 : audit [DBG] from='client.14176 v1:172.21.15.94:0/871552112' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:[v1:172.21.15.33:6789]=a;smithi033:[v1:172.21.15.33:6790]=c;smithi094:[v1:172.21.15.94:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:45:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:45:47.214069+0000 mgr.y (mgr.14146) 35 : cephadm [INF] Saving service mon spec with placement smithi033:[v1:172.21.15.33:6789]=a;smithi033:[v1:172.21.15.33:6790]=c;smithi094:[v1:172.21.15.94:6789]=b;count:3 2024-02-12T16:45:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:47.218315+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:47.236182+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:45:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:47.237863+0000 mon.a (mon.0) 120 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:47.238618+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:45:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:47.242133+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:47.243910+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:45:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:47.244582+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:45:47.245245+0000 mgr.y (mgr.14146) 36 : cephadm [INF] Deploying daemon mon.b on smithi094 2024-02-12T16:45:49.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:48.327465+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:50.517 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:45:50.517 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":1,"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","modified":"2024-02-12T16:43:47.392898Z","created":"2024-02-12T16:43:47.392898Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:45:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:dumped monmap epoch 1 2024-02-12T16:45:51.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:50.327739+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:50.515265+0000 mon.a (mon.0) 125 : audit [DBG] from='client.? v1:172.21.15.94:0/650118391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:45:51.977 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:51 smithi094 systemd[1]: Starting Ceph mon.b for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:45:52.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.496+0000 7f95d6576880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T16:45:52.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.496+0000 7f95d6576880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-12T16:45:52.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.496+0000 7f95d6576880 0 pidfile_write: ignore empty --pid-file 2024-02-12T16:45:52.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.500+0000 7f95d6576880 0 load: jerasure load: lrc load: isa 2024-02-12T16:45:52.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T16:45:52.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:52.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T16:45:52.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Compile date Feb 8 2024 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: DB SUMMARY 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: CURRENT file: CURRENT 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 476 ; 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:52.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.error_if_exists: 0 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.create_if_missing: 0 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.env: 0x5633e8234080 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.fs: Posix File System 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.info_log: 0x5633e9b6b980 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.statistics: (nil) 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.use_fsync: 0 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T16:45:52.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.db_log_dir: 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T16:45:52.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.write_buffer_manager: 0x5633ea8683f0 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T16:45:52.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.unordered_write: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.row_cache: None 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.wal_filter: None 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.two_write_queues: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.atomic_flush: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T16:45:52.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.max_open_files: -1 2024-02-12T16:45:52.999 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Compression algorithms supported: 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: kZSTD supported: 0 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: kXpressCompression supported: 0 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: kZlibCompression supported: 1 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T16:45:53.000 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.501+0000 7f95d6576880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.merge_operator: 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_filter: None 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5633e9b1bd28) 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cache_index_and_filter_blocks: 1 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: pin_top_level_index_and_filter: 1 2024-02-12T16:45:53.001 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: index_type: 0 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: data_block_index_type: 0 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: index_shortening: 1 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: data_block_hash_table_util_ratio: 0.750000 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: hash_index_allow_collision: 1 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: checksum: 1 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: no_block_cache: 0 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: block_cache: 0x5633e9b50f10 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: block_cache_name: BinnedLRUCache 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: block_cache_options: 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: capacity : 536870912 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: num_shard_bits : 4 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: strict_capacity_limit : 0 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: high_pri_pool_ratio: 0.000 2024-02-12T16:45:53.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: block_cache_compressed: (nil) 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: persistent_cache: (nil) 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: block_size: 4096 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: block_size_deviation: 10 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: block_restart_interval: 16 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: index_block_restart_interval: 1 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: metadata_block_size: 4096 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: partition_filters: 0 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: use_delta_encoding: 1 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: whole_key_filtering: 1 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: verify_compression: 0 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: read_amp_bytes_per_bit: 0 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: format_version: 2 2024-02-12T16:45:53.003 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: enable_index_compression: 1 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: block_align: 0 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compression: NoCompression 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.num_levels: 7 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T16:45:53.004 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T16:45:53.005 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T16:45:53.006 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T16:45:53.007 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.table_properties_collectors: 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.bloom_locality: 0 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T16:45:53.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.ttl: 2592000 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756352503765, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.502+0000 7f95d6576880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.503+0000 7f95d6576880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756352504219, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1402, "table_properties": {"data_size": 488, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707756352, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.503+0000 7f95d6576880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.009 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.503+0000 7f95d6576880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756352504591, "job": 1, "event": "recovery_finished"} 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.504+0000 7f95d6576880 4 rocksdb: DB pointer 0x5633e9c19800 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.504+0000 7f95bd367700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.504+0000 7f95bd367700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ** DB Stats ** 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ** Compaction Stats [default] ** 2024-02-12T16:45:53.010 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ** Compaction Stats [default] ** 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:45:53.011 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ** Compaction Stats [default] ** 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-12T16:45:53.012 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ** Compaction Stats [default] ** 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:45:53.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.504+0000 7f95d6576880 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.504+0000 7f95d6576880 0 using public_addrv v1:172.21.15.94:6789/0 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.504+0000 7f95d6576880 0 starting mon.b rank -1 at public addrs v1:172.21.15.94:6789/0 at bind addrs v1:172.21.15.94:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.505+0000 7f95d6576880 1 mon.b@-1(???) e0 preinit fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 0 mon.b@-1(synchronizing).mds e1 new map 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: e1 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: legacy client fscid: -1 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.014 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: No filesystems configured 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bb363700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95c5377700 4 rocksdb: (Original Log Time 2024/02/12-16:45:52.526872) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95c5377700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95c5377700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756352526915, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1060, "num_deletes": 13, "total_data_size": 3589524, "memory_usage": 3609696, "flush_reason": "Manual Compaction"} 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95c5377700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-12T16:45:53.015 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.525+0000 7f95bfb6c700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:50.327739+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:50.515265+0000 mon.a (mon.0) 125 : audit [DBG] from='client.? v1:172.21.15.94:0/650118391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T16:45:52.526+0000 7f95bfb6c700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 bash[40842]: 708c2929edfaab741daf485135ea4932f191454dfcb5a2a6cc43aa4317767e1c 2024-02-12T16:45:53.016 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:52 smithi094 systemd[1]: Started Ceph mon.b for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:45:53.016 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-12T16:45:53.017 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:45:57.831 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:45:57.831 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":2,"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","modified":"2024-02-12T16:45:52.531121Z","created":"2024-02-12T16:43:47.392898Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-12T16:45:57.833 INFO:teuthology.orchestra.run.smithi094.stderr:dumped monmap epoch 2 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:52.328094+0000 mgr.y (mgr.14146) 39 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:52.536090+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:52.536314+0000 mon.a (mon.0) 128 : cluster [INF] mon.a calling monitor election 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:52.552465+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:53.530695+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:54.328441+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:54.530568+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:54.532117+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:55.530853+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:57.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:56.328740+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:56.530763+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:57.530948+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:57.566287+0000 mon.a (mon.0) 135 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:57.571105+0000 mon.a (mon.0) 136 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.33:6789/0,b=v1:172.21.15.94:6789/0} removed_ranks: {} 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:57.582913+0000 mon.a (mon.0) 137 : cluster [DBG] fsmap 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:57.582958+0000 mon.a (mon.0) 138 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:57.583169+0000 mon.a (mon.0) 139 : cluster [DBG] mgrmap e13: y(active, since 71s) 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:57.586418+0000 mon.a (mon.0) 140 : cluster [INF] overall HEALTH_OK 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:57.588755+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:57.589949+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:45:57.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:57.592441+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:52.328094+0000 mgr.y (mgr.14146) 39 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:52.536090+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:52.536314+0000 mon.a (mon.0) 128 : cluster [INF] mon.a calling monitor election 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:52.552465+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:53.530695+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:54.328441+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:54.530568+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:54.532117+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:55.530853+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:56.328740+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:56.530763+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:57.530948+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:57.566287+0000 mon.a (mon.0) 135 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:57.571105+0000 mon.a (mon.0) 136 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.33:6789/0,b=v1:172.21.15.94:6789/0} removed_ranks: {} 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:57.582913+0000 mon.a (mon.0) 137 : cluster [DBG] fsmap 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:57.582958+0000 mon.a (mon.0) 138 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:57.583169+0000 mon.a (mon.0) 139 : cluster [DBG] mgrmap e13: y(active, since 71s) 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:57.586418+0000 mon.a (mon.0) 140 : cluster [INF] overall HEALTH_OK 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:57.588755+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:57.589949+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:45:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:57.592441+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:45:57.597681+0000 mgr.y (mgr.14146) 42 : cephadm [INF] Deploying daemon mon.c on smithi033 2024-02-12T16:45:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:57.829796+0000 mon.a (mon.0) 144 : audit [DBG] from='client.? v1:172.21.15.94:0/1752992662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:45:58.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:45:58.531196+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:45:57.597681+0000 mgr.y (mgr.14146) 42 : cephadm [INF] Deploying daemon mon.c on smithi033 2024-02-12T16:45:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:57.829796+0000 mon.a (mon.0) 144 : audit [DBG] from='client.? v1:172.21.15.94:0/1752992662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:45:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:45:58.531196+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:45:59.975 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:45:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:45:58.329038+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:45:59.976 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:45:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:45:59.531+0000 7f394a283700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-12T16:45:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:45:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:45:58.329038+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:00.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 systemd[1]: Starting Ceph mon.c for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:46:01.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.877+0000 7f824f0a7880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T16:46:01.318 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.877+0000 7f824f0a7880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-12T16:46:01.318 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.877+0000 7f824f0a7880 0 pidfile_write: ignore empty --pid-file 2024-02-12T16:46:01.318 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 0 load: jerasure load: lrc load: isa 2024-02-12T16:46:01.318 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T16:46:01.318 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Compile date Feb 8 2024 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: DB SUMMARY 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: CURRENT file: CURRENT 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 566 ; 2024-02-12T16:46:01.319 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.error_if_exists: 0 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.create_if_missing: 0 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.env: 0x558c44a17080 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.fs: Posix File System 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.info_log: 0x558c458e8220 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.statistics: (nil) 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.use_fsync: 0 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T16:46:01.320 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.882+0000 7f824f0a7880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.db_log_dir: 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T16:46:01.321 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.write_buffer_manager: 0x558c46534450 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T16:46:01.322 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.unordered_write: 0 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.row_cache: None 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.wal_filter: None 2024-02-12T16:46:01.323 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.two_write_queues: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.atomic_flush: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T16:46:01.324 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_open_files: -1 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Compression algorithms supported: 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: kZSTD supported: 0 2024-02-12T16:46:01.325 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: kXpressCompression supported: 0 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: kZlibCompression supported: 1 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.merge_operator: 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_filter: None 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T16:46:01.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558c4588dd28) 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cache_index_and_filter_blocks: 1 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: pin_top_level_index_and_filter: 1 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: index_type: 0 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: data_block_index_type: 0 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: index_shortening: 1 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: data_block_hash_table_util_ratio: 0.750000 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: hash_index_allow_collision: 1 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: checksum: 1 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: no_block_cache: 0 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: block_cache: 0x558c458c2f10 2024-02-12T16:46:01.327 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: block_cache_name: BinnedLRUCache 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: block_cache_options: 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: capacity : 536870912 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: num_shard_bits : 4 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: strict_capacity_limit : 0 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: high_pri_pool_ratio: 0.000 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: block_cache_compressed: (nil) 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: persistent_cache: (nil) 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: block_size: 4096 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: block_size_deviation: 10 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: block_restart_interval: 16 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: index_block_restart_interval: 1 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: metadata_block_size: 4096 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: partition_filters: 0 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: use_delta_encoding: 1 2024-02-12T16:46:01.328 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: whole_key_filtering: 1 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: verify_compression: 0 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: read_amp_bytes_per_bit: 0 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: format_version: 2 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: enable_index_compression: 1 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: block_align: 0 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compression: NoCompression 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.num_levels: 7 2024-02-12T16:46:01.329 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T16:46:01.330 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T16:46:01.331 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T16:46:01.332 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.table_properties_collectors: 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.bloom_locality: 0 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T16:46:01.333 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.ttl: 2592000 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756360884156, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.883+0000 7f824f0a7880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.884+0000 7f824f0a7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756360884598, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1492, "table_properties": {"data_size": 578, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707756360, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.884+0000 7f824f0a7880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.334 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.884+0000 7f824f0a7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756360884978, "job": 1, "event": "recovery_finished"} 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.885+0000 7f824f0a7880 4 rocksdb: DB pointer 0x558c4598b800 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.885+0000 7f8235e98700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.885+0000 7f8235e98700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ** DB Stats ** 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ** Compaction Stats [default] ** 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:46:01.335 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: L0 1/0 1.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Sum 1/0 1.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ** Compaction Stats [default] ** 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:46:01.336 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ** Compaction Stats [default] ** 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: L0 1/0 1.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Sum 1/0 1.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ** Compaction Stats [default] ** 2024-02-12T16:46:01.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.338 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.885+0000 7f824f0a7880 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.885+0000 7f824f0a7880 0 using public_addrv v1:172.21.15.33:6790/0 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.886+0000 7f824f0a7880 0 starting mon.c rank -1 at public addrs v1:172.21.15.33:6790/0 at bind addrs v1:172.21.15.33:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.887+0000 7f824f0a7880 1 mon.c@-1(???) e0 preinit fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.907+0000 7f823869d700 0 mon.c@-1(synchronizing).mds e1 new map 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.907+0000 7f823869d700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: e1 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: legacy client fscid: -1 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: No filesystems configured 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.907+0000 7f823869d700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.907+0000 7f8233e94700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-12T16:46:01.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.907+0000 7f823dea8700 4 rocksdb: (Original Log Time 2024/02/12-16:46:00.908261) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.907+0000 7f823dea8700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.907+0000 7f823dea8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707756360908310, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1127, "num_deletes": 25, "total_data_size": 3743181, "memory_usage": 3764344, "flush_reason": "Manual Compaction"} 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.907+0000 7f823dea8700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:46:01.340 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:46:01.341 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:45:58.329038+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:01.341 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T16:46:00.908+0000 7f823869d700 1 mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-02-12T16:46:01.341 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 bash[46068]: 5a9907b544116996c937201dff740a7d499a26dc99fa653afa45c5bdce7c719d 2024-02-12T16:46:01.341 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:00 smithi033 systemd[1]: Started Ceph mon.c for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:46:04.726 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-12T16:46:04.726 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:46:06.269 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:00.917952+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-12T16:46:06.269 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:00.918557+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:46:06.269 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:00.918731+0000 mon.a (mon.0) 148 : cluster [INF] mon.a calling monitor election 2024-02-12T16:46:06.269 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:00.933810+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:46:06.269 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:00.934030+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.269 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:01.911726+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:02.329746+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:02.911800+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:03.911862+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:04.329991+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:04.912169+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:05.912065+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.933828+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.936945+0000 mon.a (mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.33:6789/0,b=v1:172.21.15.94:6789/0,c=v1:172.21.15.33:6790/0} removed_ranks: {} 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.948812+0000 mon.a (mon.0) 158 : cluster [DBG] fsmap 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.948840+0000 mon.a (mon.0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:46:06.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.949052+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 79s) 2024-02-12T16:46:06.271 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.949219+0000 mon.a (mon.0) 161 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-12T16:46:06.271 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.953472+0000 mon.a (mon.0) 162 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-12T16:46:06.271 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.953497+0000 mon.a (mon.0) 163 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-12T16:46:06.271 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:05.953514+0000 mon.a (mon.0) 164 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.33:6790/0 is down (out of quorum) 2024-02-12T16:46:06.271 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:05.956015+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:06.271 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:05.957380+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:00.917952+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:00.918557+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:00.918731+0000 mon.a (mon.0) 148 : cluster [INF] mon.a calling monitor election 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:00.933810+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:00.934030+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:01.911726+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:02.329746+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:02.911800+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:03.911862+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:04.329991+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:04.912169+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:05.912065+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.933828+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.936945+0000 mon.a (mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.33:6789/0,b=v1:172.21.15.94:6789/0,c=v1:172.21.15.33:6790/0} removed_ranks: {} 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.948812+0000 mon.a (mon.0) 158 : cluster [DBG] fsmap 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.948840+0000 mon.a (mon.0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.949052+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 79s) 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.949219+0000 mon.a (mon.0) 161 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.953472+0000 mon.a (mon.0) 162 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.953497+0000 mon.a (mon.0) 163 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:05.953514+0000 mon.a (mon.0) 164 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.33:6790/0 is down (out of quorum) 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:05.956015+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:06.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:05.957380+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-02-12T16:46:06.747 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":3,"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","modified":"2024-02-12T16:46:00.912637Z","created":"2024-02-12T16:43:47.392898Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6790","nonce":0}]},"addr":"172.21.15.33:6790/0","public_addr":"172.21.15.33:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-12T16:46:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:dumped monmap epoch 3 2024-02-12T16:46:07.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:06.330244+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:06.465925+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:06.745681+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? v1:172.21.15.94:0/3928468683' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:06.912176+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.100179+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.406592+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.407071+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.407624+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.425328+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.427424+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.429979+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.431241+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.431757+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:07.432247+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:06.330244+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:06.465925+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:06.745681+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? v1:172.21.15.94:0/3928468683' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:06.912176+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.100179+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.406592+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.407071+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.407624+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.425328+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.427424+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.429979+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:07.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.431241+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:07.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.431757+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:07.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:07.432247+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:07.816 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-12T16:46:07.816 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph config generate-minimal-conf 2024-02-12T16:46:08.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:06.330244+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:08.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:06.465925+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:08.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:06.745681+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? v1:172.21.15.94:0/3928468683' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:46:08.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:06.912176+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.100179+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.406592+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.407071+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.407624+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.425328+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.427424+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.429979+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.431241+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.431757+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:08.139 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:07.432247+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:08.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:02.913592+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-12T16:46:08.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.934854+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.935213+0000 mon.a (mon.0) 180 : cluster [INF] mon.a calling monitor election 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.935329+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.951094+0000 mon.a (mon.0) 181 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.961529+0000 mon.a (mon.0) 182 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.33:6789/0,b=v1:172.21.15.94:6789/0,c=v1:172.21.15.33:6790/0} removed_ranks: {} 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.961990+0000 mon.a (mon.0) 183 : cluster [DBG] fsmap 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.962012+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.962164+0000 mon.a (mon.0) 185 : cluster [DBG] mgrmap e13: y(active, since 81s) 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.962479+0000 mon.a (mon.0) 186 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.962497+0000 mon.a (mon.0) 187 : cluster [INF] Cluster is now healthy 2024-02-12T16:46:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:07.968294+0000 mon.a (mon.0) 188 : cluster [INF] overall HEALTH_OK 2024-02-12T16:46:08.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:02.913592+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-12T16:46:08.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.934854+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-02-12T16:46:08.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.935213+0000 mon.a (mon.0) 180 : cluster [INF] mon.a calling monitor election 2024-02-12T16:46:08.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.935329+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-02-12T16:46:08.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.951094+0000 mon.a (mon.0) 181 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-12T16:46:08.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.961529+0000 mon.a (mon.0) 182 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.33:6789/0,b=v1:172.21.15.94:6789/0,c=v1:172.21.15.33:6790/0} removed_ranks: {} 2024-02-12T16:46:08.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.961990+0000 mon.a (mon.0) 183 : cluster [DBG] fsmap 2024-02-12T16:46:08.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.962012+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.962164+0000 mon.a (mon.0) 185 : cluster [DBG] mgrmap e13: y(active, since 81s) 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.962479+0000 mon.a (mon.0) 186 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.962497+0000 mon.a (mon.0) 187 : cluster [INF] Cluster is now healthy 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:07.968294+0000 mon.a (mon.0) 188 : cluster [INF] overall HEALTH_OK 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:02.913592+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.934854+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.935213+0000 mon.a (mon.0) 180 : cluster [INF] mon.a calling monitor election 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.935329+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.951094+0000 mon.a (mon.0) 181 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.961529+0000 mon.a (mon.0) 182 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.33:6789/0,b=v1:172.21.15.94:6789/0,c=v1:172.21.15.33:6790/0} removed_ranks: {} 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.961990+0000 mon.a (mon.0) 183 : cluster [DBG] fsmap 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.962012+0000 mon.a (mon.0) 184 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.962164+0000 mon.a (mon.0) 185 : cluster [DBG] mgrmap e13: y(active, since 81s) 2024-02-12T16:46:08.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.962479+0000 mon.a (mon.0) 186 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-12T16:46:08.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.962497+0000 mon.a (mon.0) 187 : cluster [INF] Cluster is now healthy 2024-02-12T16:46:08.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:07.968294+0000 mon.a (mon.0) 188 : cluster [INF] overall HEALTH_OK 2024-02-12T16:46:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:08.330501+0000 mgr.y (mgr.14146) 52 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:08.912554+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:09.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:08.330501+0000 mgr.y (mgr.14146) 52 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:08.912554+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:08.330501+0000 mgr.y (mgr.14146) 52 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:08.912554+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:10.868 INFO:teuthology.orchestra.run.smithi033.stdout:# minimal ceph.conf for cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:46:10.869 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2024-02-12T16:46:10.869 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:46:10.869 INFO:teuthology.orchestra.run.smithi033.stdout: mon_host = 172.21.15.33:6789/0 172.21.15.94:6789/0 v1:172.21.15.33:6790/0 2024-02-12T16:46:10.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:09.638082+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:10.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:09.638576+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-12T16:46:10.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:09.638798+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:10.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:09.639393+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:10.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:09.639911+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:10.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:09.640418+0000 mgr.y (mgr.14146) 54 : cephadm [INF] Reconfiguring daemon mon.c on smithi033 2024-02-12T16:46:11.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:09.638082+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:09.638576+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:09.638798+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:09.639393+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:09.639911+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:09.640418+0000 mgr.y (mgr.14146) 54 : cephadm [INF] Reconfiguring daemon mon.c on smithi033 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:09.638082+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:09.638576+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:09.638798+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:09.639393+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:09.639911+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:09.640418+0000 mgr.y (mgr.14146) 54 : cephadm [INF] Reconfiguring daemon mon.c on smithi033 2024-02-12T16:46:11.686 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-12T16:46:11.686 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:46:11.686 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2024-02-12T16:46:11.721 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:46:11.721 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:46:11.792 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:46:11.792 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.conf 2024-02-12T16:46:11.828 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:46:11.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:46:11.900 INFO:tasks.cephadm:Adding mgr.y on smithi033 2024-02-12T16:46:11.900 INFO:tasks.cephadm:Adding mgr.x on smithi094 2024-02-12T16:46:11.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch apply mgr '2;smithi033=y;smithi094=x' 2024-02-12T16:46:11.966 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:10.330790+0000 mgr.y (mgr.14146) 55 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:11.966 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:10.867498+0000 mon.a (mon.0) 194 : audit [DBG] from='client.? v1:172.21.15.33:0/3039082307' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:12.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:10.330790+0000 mgr.y (mgr.14146) 55 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:12.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:10.867498+0000 mon.a (mon.0) 194 : audit [DBG] from='client.? v1:172.21.15.33:0/3039082307' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:12.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:10.330790+0000 mgr.y (mgr.14146) 55 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:12.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:10.867498+0000 mon.a (mon.0) 194 : audit [DBG] from='client.? v1:172.21.15.33:0/3039082307' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:12.197437+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:12.197966+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-12T16:46:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:12.198187+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:12.198802+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:12.199410+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:12.199946+0000 mgr.y (mgr.14146) 57 : cephadm [INF] Reconfiguring daemon mon.b on smithi094 2024-02-12T16:46:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:12.331025+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:13.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:12.197437+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:12.197966+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:12.198187+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:12.198802+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:12.199410+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:12.199946+0000 mgr.y (mgr.14146) 57 : cephadm [INF] Reconfiguring daemon mon.b on smithi094 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:12.331025+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:12.197437+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:12.197966+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:12.198187+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:12.198802+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:12.199410+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:12.199946+0000 mgr.y (mgr.14146) 57 : cephadm [INF] Reconfiguring daemon mon.b on smithi094 2024-02-12T16:46:13.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:12.331025+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:14.802 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled mgr update... 2024-02-12T16:46:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:13.736807+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:13.737944+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:13.739567+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:13.740338+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:14.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:13.744810+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:13.736807+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:13.737944+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:13.739567+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:13.740338+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:13.744810+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:13.736807+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:13.737944+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:13.739567+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:13.740338+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:15.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:13.744810+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:15.464 DEBUG:teuthology.orchestra.run.smithi094:mgr.x> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.x.service 2024-02-12T16:46:15.506 INFO:tasks.cephadm:Deploying OSDs... 2024-02-12T16:46:15.506 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:46:15.507 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-02-12T16:46:15.529 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-12T16:46:15.529 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-02-12T16:46:15.538 INFO:journalctl@ceph.mgr.x.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:46:15.591 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-12T16:46:15.591 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T16:46:15.591 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 158629 Links: 1 2024-02-12T16:46:15.592 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T16:46:15.592 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-02-12T16:46:15.592 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-02-12 16:46:00.403090646 +0000 2024-02-12T16:46:15.592 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-02-12 16:38:52.901532549 +0000 2024-02-12T16:46:15.592 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-02-12 16:38:52.901532549 +0000 2024-02-12T16:46:15.592 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-02-12T16:46:15.592 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-12T16:46:15.664 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-02-12T16:46:15.664 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-02-12T16:46:15.664 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000244876 s, 2.1 MB/s 2024-02-12T16:46:15.665 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-12T16:46:15.728 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-02-12T16:46:15.789 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-12T16:46:15.789 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T16:46:15.789 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 164806 Links: 1 2024-02-12T16:46:15.789 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T16:46:15.789 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-02-12T16:46:15.789 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-02-12 16:46:00.404090630 +0000 2024-02-12T16:46:15.789 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-02-12 16:38:53.252527032 +0000 2024-02-12T16:46:15.789 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-02-12 16:38:53.252527032 +0000 2024-02-12T16:46:15.790 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-02-12T16:46:15.790 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-12T16:46:15.859 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-02-12T16:46:15.860 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-02-12T16:46:15.860 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000228173 s, 2.2 MB/s 2024-02-12T16:46:15.861 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-12T16:46:15.924 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-02-12T16:46:15.984 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-12T16:46:15.984 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T16:46:15.984 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 165736 Links: 1 2024-02-12T16:46:15.984 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T16:46:15.984 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-02-12T16:46:15.984 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-02-12 16:46:00.404090630 +0000 2024-02-12T16:46:15.985 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-02-12 16:38:53.626521153 +0000 2024-02-12T16:46:15.985 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-02-12 16:38:53.626521153 +0000 2024-02-12T16:46:15.985 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-02-12T16:46:15.985 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:14.331324+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.800276+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.813318+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.814990+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.815805+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.820697+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.822142+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.824663+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.825419+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T16:46:16.024 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.826245+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:14.331324+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.800276+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.813318+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.814990+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.815805+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.820697+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.822142+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.824663+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.825419+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.826245+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:14.331324+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.800276+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.813318+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.814990+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.815805+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.820697+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.822142+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T16:46:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.824663+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-12T16:46:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.825419+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T16:46:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.826245+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-02-12T16:46:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-02-12T16:46:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000191577 s, 2.7 MB/s 2024-02-12T16:46:16.053 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-12T16:46:16.115 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 165843 Links: 1 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-02-12 16:46:00.404090630 +0000 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-02-12 16:38:53.983515541 +0000 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-02-12 16:38:53.983515541 +0000 2024-02-12T16:46:16.175 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-02-12T16:46:16.175 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-12T16:46:16.243 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-02-12T16:46:16.243 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-02-12T16:46:16.244 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000238582 s, 2.1 MB/s 2024-02-12T16:46:16.245 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-12T16:46:16.306 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:46:16.306 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-02-12T16:46:16.325 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-12T16:46:16.325 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-02-12T16:46:16.383 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-12T16:46:16.383 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T16:46:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 162572 Links: 1 2024-02-12T16:46:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T16:46:16.383 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-02-12T16:46:16.384 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-02-12 16:45:51.700567539 +0000 2024-02-12T16:46:16.384 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-02-12 16:38:45.200038885 +0000 2024-02-12T16:46:16.384 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-02-12 16:38:45.200038885 +0000 2024-02-12T16:46:16.384 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2024-02-12T16:46:16.384 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-12T16:46:16.450 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-02-12T16:46:16.450 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-02-12T16:46:16.450 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 8.5027e-05 s, 6.0 MB/s 2024-02-12T16:46:16.451 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-12T16:46:16.510 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 169332 Links: 1 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-02-12 16:45:51.700567539 +0000 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-02-12 16:38:45.535033331 +0000 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-02-12 16:38:45.535033331 +0000 2024-02-12T16:46:16.568 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2024-02-12T16:46:16.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-12T16:46:16.633 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-02-12T16:46:16.633 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-02-12T16:46:16.633 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000113716 s, 4.5 MB/s 2024-02-12T16:46:16.634 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-12T16:46:16.693 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-02-12T16:46:16.752 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-12T16:46:16.752 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T16:46:16.752 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 162714 Links: 1 2024-02-12T16:46:16.752 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T16:46:16.752 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-02-12T16:46:16.752 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-02-12 16:45:51.700567539 +0000 2024-02-12T16:46:16.752 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-02-12 16:38:45.872027744 +0000 2024-02-12T16:46:16.752 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-02-12 16:38:45.872027744 +0000 2024-02-12T16:46:16.753 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2024-02-12T16:46:16.753 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-12T16:46:16.851 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-02-12T16:46:16.851 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-02-12T16:46:16.851 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 9.3602e-05 s, 5.5 MB/s 2024-02-12T16:46:16.852 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-12T16:46:16.871 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-02-12T16:46:16.930 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-12T16:46:16.931 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T16:46:16.931 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 162775 Links: 1 2024-02-12T16:46:16.931 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T16:46:16.931 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-02-12T16:46:16.931 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-02-12 16:46:16.804144550 +0000 2024-02-12T16:46:16.931 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-02-12 16:38:46.227021858 +0000 2024-02-12T16:46:16.931 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-02-12 16:38:46.227021858 +0000 2024-02-12T16:46:16.931 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2024-02-12T16:46:16.931 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-12T16:46:16.998 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:16 smithi094 systemd[1]: Starting Ceph mgr.x for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:46:16.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:14.795205+0000 mgr.y (mgr.14146) 60 : audit [DBG] from='client.24112 v1:172.21.15.94:0/2466838847' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi033=y;smithi094=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:16.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:14.796669+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Saving service mgr spec with placement smithi033=y;smithi094=x;count:2 2024-02-12T16:46:16.998 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:14.826888+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Deploying daemon mgr.x on smithi094 2024-02-12T16:46:17.000 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-02-12T16:46:17.000 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-02-12T16:46:17.000 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 8.8286e-05 s, 5.8 MB/s 2024-02-12T16:46:17.001 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-12T16:46:17.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:14.795205+0000 mgr.y (mgr.14146) 60 : audit [DBG] from='client.24112 v1:172.21.15.94:0/2466838847' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi033=y;smithi094=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:17.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:14.796669+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Saving service mgr spec with placement smithi033=y;smithi094=x;count:2 2024-02-12T16:46:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:14.826888+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Deploying daemon mgr.x on smithi094 2024-02-12T16:46:17.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:14.795205+0000 mgr.y (mgr.14146) 60 : audit [DBG] from='client.24112 v1:172.21.15.94:0/2466838847' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi033=y;smithi094=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:17.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:14.796669+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Saving service mgr spec with placement smithi033=y;smithi094=x;count:2 2024-02-12T16:46:17.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:14.826888+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Deploying daemon mgr.x on smithi094 2024-02-12T16:46:17.061 INFO:tasks.cephadm:Deploying osd.0 on smithi033 with /dev/vg_nvme/lv_4... 2024-02-12T16:46:17.061 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-12T16:46:17.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:16.331690+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:17.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:17.653050+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:17.654619+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:17.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:17 smithi094 bash[42303]: 3fe02a013af4cbafcc3dd6828b9a4fc074523dae973862e74aa0565e7714d255 2024-02-12T16:46:17.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:17 smithi094 systemd[1]: Started Ceph mgr.x for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:46:17.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:17.626+0000 7f02bd8be680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T16:46:17.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:17.712+0000 7f02bd8be680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T16:46:18.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:16.331690+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:18.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:17.653050+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:18.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:17.654619+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:16.331690+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:17.653050+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:17.654619+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:18.490 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:18.107+0000 7f02bd8be680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T16:46:19.163 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:18.873+0000 7f02bd8be680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T16:46:19.163 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:18.956+0000 7f02bd8be680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T16:46:19.164 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:19.162+0000 7f02bd8be680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T16:46:19.798 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-12T16:46:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-12T16:46:19.799 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-02-12T16:46:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-02-12T16:46:19.799 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112191 s, 935 MB/s 2024-02-12T16:46:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-02-12T16:46:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:18.332090+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:18.654989+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:18.655753+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:18.656614+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:18.661136+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:18.662911+0000 mgr.y (mgr.14146) 65 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-12T16:46:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:18.663216+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T16:46:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:18.664100+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T16:46:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:18.664763+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:18.665368+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring daemon mgr.y on smithi033 2024-02-12T16:46:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:18.332090+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:18.654989+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:18.655753+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:18.656614+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:18.661136+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:18.662911+0000 mgr.y (mgr.14146) 65 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-12T16:46:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:18.663216+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T16:46:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:18.664100+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:18.664763+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:18.665368+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring daemon mgr.y on smithi033 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:18.332090+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:18.654989+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:18.655753+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:18.656614+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:18.661136+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:18.662911+0000 mgr.y (mgr.14146) 65 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:18.663216+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:18.664100+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:18.664763+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:18.665368+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring daemon mgr.y on smithi033 2024-02-12T16:46:20.388 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:20.096+0000 7f02bd8be680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T16:46:20.388 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:20.221+0000 7f02bd8be680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:46:20.388 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:20.298+0000 7f02bd8be680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T16:46:20.389 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:20.386+0000 7f02bd8be680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T16:46:20.473 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4 2024-02-12T16:46:20.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:20.577+0000 7f02bd8be680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T16:46:21.241 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:20.853+0000 7f02bd8be680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T16:46:21.241 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:20.950+0000 7f02bd8be680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T16:46:21.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:21.663+0000 7f02bd8be680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T16:46:21.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:21.743+0000 7f02bd8be680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T16:46:21.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:21.834+0000 7f02bd8be680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T16:46:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:20.332459+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:20.948898+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:20.949859+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:20.951124+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:20.951666+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:20.956036+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:21.470536+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:20.332459+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:20.948898+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:20.949859+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:20.951124+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:20.951666+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:20.956036+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:21.470536+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:20.332459+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:20.948898+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:20.949859+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:20.951124+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:20.951666+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:20.956036+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:21.470536+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:22.259 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:22.004+0000 7f02bd8be680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T16:46:22.518 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:22.258+0000 7f02bd8be680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T16:46:22.519 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:22.387+0000 7f02bd8be680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T16:46:22.519 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:22.517+0000 7f02bd8be680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:46:22.770 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:22.687+0000 7f02bd8be680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T16:46:23.240 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:46:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:46:22.768+0000 7f02bd8be680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T16:46:23.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:22.332687+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:22.768886+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:46:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:22.770704+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:46:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:22.771189+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:22.782649+0000 mon.a (mon.0) 231 : cluster [DBG] Standby manager daemon x started 2024-02-12T16:46:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:22.783499+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T16:46:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:22.783808+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T16:46:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:22.784601+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T16:46:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:22.784830+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T16:46:24.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:22.332687+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:24.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:22.768886+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:22.770704+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:22.771189+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:22.782649+0000 mon.a (mon.0) 231 : cluster [DBG] Standby manager daemon x started 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:22.783499+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:22.783808+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:22.784601+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:22.784830+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:22.332687+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:22.768886+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:22.770704+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:46:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:22.771189+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:24.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:22.782649+0000 mon.a (mon.0) 231 : cluster [DBG] Standby manager daemon x started 2024-02-12T16:46:24.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:22.783499+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T16:46:24.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:22.783808+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T16:46:24.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:22.784601+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T16:46:24.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:22.784830+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.94:0/994341650' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T16:46:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:22.766697+0000 mgr.y (mgr.14146) 69 : audit [DBG] from='client.34118 v1:172.21.15.33:0/2614522637' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:24.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:23.666142+0000 mon.a (mon.0) 232 : cluster [DBG] mgrmap e14: y(active, since 97s), standbys: x 2024-02-12T16:46:24.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:23.666272+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T16:46:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:22.766697+0000 mgr.y (mgr.14146) 69 : audit [DBG] from='client.34118 v1:172.21.15.33:0/2614522637' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:23.666142+0000 mon.a (mon.0) 232 : cluster [DBG] mgrmap e14: y(active, since 97s), standbys: x 2024-02-12T16:46:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:23.666272+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T16:46:25.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:22.766697+0000 mgr.y (mgr.14146) 69 : audit [DBG] from='client.34118 v1:172.21.15.33:0/2614522637' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:25.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:23.666142+0000 mon.a (mon.0) 232 : cluster [DBG] mgrmap e14: y(active, since 97s), standbys: x 2024-02-12T16:46:25.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:23.666272+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T16:46:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:24.333046+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:25.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:25.188394+0000 mon.a (mon.0) 234 : audit [INF] from='client.? v1:172.21.15.33:0/1702765307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7"}]: dispatch 2024-02-12T16:46:25.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:25.192634+0000 mon.a (mon.0) 235 : audit [INF] from='client.? v1:172.21.15.33:0/1702765307' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7"}]': finished 2024-02-12T16:46:25.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:25.192680+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-12T16:46:25.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:25.192803+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:26.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:24.333046+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:25.188394+0000 mon.a (mon.0) 234 : audit [INF] from='client.? v1:172.21.15.33:0/1702765307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7"}]: dispatch 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:25.192634+0000 mon.a (mon.0) 235 : audit [INF] from='client.? v1:172.21.15.33:0/1702765307' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7"}]': finished 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:25.192680+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:25.192803+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:24.333046+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:25.188394+0000 mon.a (mon.0) 234 : audit [INF] from='client.? v1:172.21.15.33:0/1702765307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7"}]: dispatch 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:25.192634+0000 mon.a (mon.0) 235 : audit [INF] from='client.? v1:172.21.15.33:0/1702765307' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7"}]': finished 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:25.192680+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-12T16:46:26.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:25.192803+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:26.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:25.756514+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? v1:172.21.15.33:0/3595931425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:46:27.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:25.756514+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? v1:172.21.15.33:0/3595931425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:46:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:25.756514+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? v1:172.21.15.33:0/3595931425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:46:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:26.333319+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:28.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:26.333319+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:28.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:26.333319+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:29.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:28.333682+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:30.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:28.333682+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:28.333682+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:31.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:30.334004+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:31.507701+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-12T16:46:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:31.508303+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:30.334004+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:31.507701+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-12T16:46:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:31.508303+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:30.334004+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:31.507701+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-12T16:46:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:31.508303+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:32.921 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:31.508883+0000 mgr.y (mgr.14146) 74 : cephadm [INF] Deploying daemon osd.0 on smithi033 2024-02-12T16:46:32.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:31.508883+0000 mgr.y (mgr.14146) 74 : cephadm [INF] Deploying daemon osd.0 on smithi033 2024-02-12T16:46:32.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:31.508883+0000 mgr.y (mgr.14146) 74 : cephadm [INF] Deploying daemon osd.0 on smithi033 2024-02-12T16:46:33.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:32.334280+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:34.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:32.334280+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:34.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:32.334280+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:34.334590+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:34.334590+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:34.334590+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:37.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:36.334859+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:36.735294+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:36.754550+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:36.334859+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:36.735294+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:36.754550+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:38.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:36.334859+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:38.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:36.735294+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:38.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:36.754550+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:38.335099+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:39.353726+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:39.354327+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:39.354877+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:39.358908+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:40.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:38.335099+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:40.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:39.353726+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:39.354327+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:39.354877+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:39.358908+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:40.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:38.335099+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:40.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:39.353726+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:40.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:39.354327+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:40.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:39.354877+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:40.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:39.358908+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:40.351 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 0 on host 'smithi033' 2024-02-12T16:46:41.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:39.848374+0000 mon.c (mon.2) 3 : audit [INF] from='osd.0 v1:172.21.15.33:6801/2545411201' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T16:46:41.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:39.848722+0000 mon.a (mon.0) 247 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T16:46:41.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:40.349006+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:41.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:40.376118+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:39.848374+0000 mon.c (mon.2) 3 : audit [INF] from='osd.0 v1:172.21.15.33:6801/2545411201' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T16:46:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:39.848722+0000 mon.a (mon.0) 247 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T16:46:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:40.349006+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:40.376118+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:41.052 DEBUG:teuthology.orchestra.run.smithi033:osd.0> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.0.service 2024-02-12T16:46:41.055 INFO:tasks.cephadm:Deploying osd.1 on smithi033 with /dev/vg_nvme/lv_3... 2024-02-12T16:46:41.055 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-12T16:46:41.085 INFO:journalctl@ceph.osd.0.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:46:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:39.848374+0000 mon.c (mon.2) 3 : audit [INF] from='osd.0 v1:172.21.15.33:6801/2545411201' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T16:46:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:39.848722+0000 mon.a (mon.0) 247 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T16:46:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:40.349006+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:40.376118+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:40.335335+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:40.751021+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-12T16:46:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:40.751081+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-12T16:46:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:40.751188+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:40.751718+0000 mon.c (mon.2) 4 : audit [INF] from='osd.0 v1:172.21.15.33:6801/2545411201' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:46:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:40.752025+0000 mon.a (mon.0) 253 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:46:41.779 INFO:journalctl@ceph.osd.0.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-0[49559]: debug 2024-02-12T16:46:41.756+0000 7f5ec4a0a700 -1 osd.0 0 waiting for initial osdmap 2024-02-12T16:46:41.779 INFO:journalctl@ceph.osd.0.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-0[49559]: debug 2024-02-12T16:46:41.759+0000 7f5ebe981700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:46:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:40.335335+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:40.751021+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-12T16:46:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:40.751081+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-12T16:46:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:40.751188+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:40.751718+0000 mon.c (mon.2) 4 : audit [INF] from='osd.0 v1:172.21.15.33:6801/2545411201' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:46:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:40.752025+0000 mon.a (mon.0) 253 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:46:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:40.335335+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:42.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:40.751021+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-12T16:46:42.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:40.751081+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-12T16:46:42.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:40.751188+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:42.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:40.751718+0000 mon.c (mon.2) 4 : audit [INF] from='osd.0 v1:172.21.15.33:6801/2545411201' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:46:42.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:40.752025+0000 mon.a (mon.0) 253 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:46:42.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:41.752306+0000 mon.a (mon.0) 254 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:46:42.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:41.752391+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-12T16:46:42.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:41.752722+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:42.979 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:41.757956+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:42.979 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:41.752306+0000 mon.a (mon.0) 254 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:46:42.979 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:41.752391+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-12T16:46:42.979 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:41.752722+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:42.979 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:41.757956+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:41.752306+0000 mon.a (mon.0) 254 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:46:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:41.752391+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-12T16:46:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:41.752722+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:43.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:41.757956+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:40.879185+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:40.879288+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:42.335547+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:42.758783+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:42.764032+0000 mon.a (mon.0) 259 : cluster [INF] osd.0 v1:172.21.15.33:6801/2545411201 boot 2024-02-12T16:46:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:42.764081+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-12T16:46:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:42.764378+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:40.879185+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:40.879288+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:42.335547+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:42.758783+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:42.764032+0000 mon.a (mon.0) 259 : cluster [INF] osd.0 v1:172.21.15.33:6801/2545411201 boot 2024-02-12T16:46:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:42.764081+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-12T16:46:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:42.764378+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:40.879185+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:44.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:40.879288+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:44.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:42.335547+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:46:44.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:42.758783+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:44.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:42.764032+0000 mon.a (mon.0) 259 : cluster [INF] osd.0 v1:172.21.15.33:6801/2545411201 boot 2024-02-12T16:46:44.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:42.764081+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-12T16:46:44.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:42.764378+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:46:44.391 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-12T16:46:44.392 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-12T16:46:44.392 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-02-12T16:46:44.392 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-02-12T16:46:44.392 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130334 s, 805 MB/s 2024-02-12T16:46:44.392 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-02-12T16:46:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:43.769080+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-12T16:46:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:43.769080+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-12T16:46:45.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:43.769080+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-12T16:46:45.416 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_3 2024-02-12T16:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:44.335778+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:45.035918+0000 mgr.y (mgr.14146) 82 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:45.042146+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:45.042827+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:45.043305+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 16102M 2024-02-12T16:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:45.046302+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:45.047013+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:45.047561+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:45.050927+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:44.335778+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:45.035918+0000 mgr.y (mgr.14146) 82 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:45.042146+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:45.042827+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:45.043305+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 16102M 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:45.046302+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:45.047013+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:45.047561+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:45.050927+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:44.335778+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:45.035918+0000 mgr.y (mgr.14146) 82 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:46:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:45.042146+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:45.042827+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:46:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:45.043305+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 16102M 2024-02-12T16:46:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:45.046302+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:45.047013+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:45.047561+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:45.050927+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:46:47.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:46.336159+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:47.220 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:46.336159+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:47.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:46.336159+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:48.171 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:47.659279+0000 mgr.y (mgr.14146) 85 : audit [DBG] from='client.34145 v1:172.21.15.33:0/4155768021' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:48.171 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:47.661271+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:46:48.172 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:47.663179+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:46:48.172 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:47.663690+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:48.172 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:47.659279+0000 mgr.y (mgr.14146) 85 : audit [DBG] from='client.34145 v1:172.21.15.33:0/4155768021' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:48.172 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:47.661271+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:46:48.172 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:47.663179+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:46:48.172 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:47.663690+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:47.659279+0000 mgr.y (mgr.14146) 85 : audit [DBG] from='client.34145 v1:172.21.15.33:0/4155768021' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:46:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:47.661271+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:46:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:47.663179+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:46:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:47.663690+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:48.961 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:48.336433+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:48.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:48.336433+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:48.336433+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:50.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:49.990884+0000 mon.a (mon.0) 272 : audit [INF] from='client.? v1:172.21.15.33:0/3430241968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24"}]: dispatch 2024-02-12T16:46:50.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:49.994505+0000 mon.a (mon.0) 273 : audit [INF] from='client.? v1:172.21.15.33:0/3430241968' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24"}]': finished 2024-02-12T16:46:50.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:49.994563+0000 mon.a (mon.0) 274 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-12T16:46:50.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:49.994676+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:46:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:49.990884+0000 mon.a (mon.0) 272 : audit [INF] from='client.? v1:172.21.15.33:0/3430241968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24"}]: dispatch 2024-02-12T16:46:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:49.994505+0000 mon.a (mon.0) 273 : audit [INF] from='client.? v1:172.21.15.33:0/3430241968' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24"}]': finished 2024-02-12T16:46:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:49.994563+0000 mon.a (mon.0) 274 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-12T16:46:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:49.994676+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:46:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:49.990884+0000 mon.a (mon.0) 272 : audit [INF] from='client.? v1:172.21.15.33:0/3430241968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24"}]: dispatch 2024-02-12T16:46:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:49.994505+0000 mon.a (mon.0) 273 : audit [INF] from='client.? v1:172.21.15.33:0/3430241968' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24"}]': finished 2024-02-12T16:46:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:49.994563+0000 mon.a (mon.0) 274 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-12T16:46:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:49.994676+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:46:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:50.336714+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:50.616551+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? v1:172.21.15.33:0/1217799177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:46:51.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:50.336714+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:51.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:50.616551+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? v1:172.21.15.33:0/1217799177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:46:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:50.336714+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:50.616551+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? v1:172.21.15.33:0/1217799177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:46:53.518 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:52.337110+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:53.518 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:52.337110+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:53.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:52.337110+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:55.672 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:54.337417+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:55.673 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:54.337417+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:54.337417+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:57.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:56.301822+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-12T16:46:57.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:46:56.302457+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:57.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:46:56.303013+0000 mgr.y (mgr.14146) 90 : cephadm [INF] Deploying daemon osd.1 on smithi033 2024-02-12T16:46:57.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:56.337680+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:57.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:56.301822+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-12T16:46:57.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:46:56.302457+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:57.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:46:56.303013+0000 mgr.y (mgr.14146) 90 : cephadm [INF] Deploying daemon osd.1 on smithi033 2024-02-12T16:46:57.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:56.337680+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:56.301822+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-12T16:46:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:46:56.302457+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:46:56.303013+0000 mgr.y (mgr.14146) 90 : cephadm [INF] Deploying daemon osd.1 on smithi033 2024-02-12T16:46:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:56.337680+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:46:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:46:58.337944+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:59.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:46:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:46:58.337944+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:46:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:46:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:46:58.337944+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:01.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:00.338348+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:01.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:01.311843+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:01.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:01.349867+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:01.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:00.338348+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:01.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:01.311843+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:01.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:01.349867+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:00.338348+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:01.311843+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:01.349867+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:03.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:02.338720+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:02.338720+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:02.338720+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:04.610 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 1 on host 'smithi033' 2024-02-12T16:47:05.362 DEBUG:teuthology.orchestra.run.smithi033:osd.1> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.1.service 2024-02-12T16:47:05.365 INFO:tasks.cephadm:Deploying osd.2 on smithi033 with /dev/vg_nvme/lv_2... 2024-02-12T16:47:05.365 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-12T16:47:05.402 INFO:journalctl@ceph.osd.1.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:47:05.655 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:04.339003+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:04.357896+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:04.358640+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:04.359211+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:04.363642+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:04.413293+0000 mon.a (mon.0) 284 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:04.608022+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:04.629661+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:04.339003+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:04.357896+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:04.358640+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:04.359211+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:05.657 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:04.363642+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.657 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:04.413293+0000 mon.a (mon.0) 284 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T16:47:05.657 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:04.608022+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.657 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:04.629661+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:04.339003+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:04.357896+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:04.358640+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:04.359211+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:04.363642+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:04.413293+0000 mon.a (mon.0) 284 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T16:47:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:04.608022+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:04.629661+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:06.528 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[53989]: debug 2024-02-12T16:47:06.408+0000 7f751e7c0700 -1 osd.1 0 waiting for initial osdmap 2024-02-12T16:47:06.528 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[53989]: debug 2024-02-12T16:47:06.414+0000 7f7518737700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:47:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:05.402284+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T16:47:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:05.402370+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-12T16:47:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:05.402499+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:05.402988+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:06.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:05.402284+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T16:47:06.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:05.402370+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-12T16:47:06.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:05.402499+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:06.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:05.402988+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:05.402284+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T16:47:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:05.402370+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-12T16:47:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:05.402499+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:05.402988+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:07.731 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:06.339407+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:06.405858+0000 mon.a (mon.0) 291 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:06.405958+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:06.406376+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:06.412237+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:06.339407+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:06.405858+0000 mon.a (mon.0) 291 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:06.405958+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:06.406376+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:07.732 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:06.412237+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:07.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:06.339407+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:07.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:06.405858+0000 mon.a (mon.0) 291 : audit [INF] from='osd.1 v1:172.21.15.33:6805/1723231708' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:07.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:06.405958+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-12T16:47:07.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:06.406376+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:07.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:06.412237+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:08.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:05.447511+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:08.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:05.447579+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:08.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:07.409260+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:07.413145+0000 mon.a (mon.0) 296 : cluster [INF] osd.1 v1:172.21.15.33:6805/1723231708 boot 2024-02-12T16:47:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:07.413179+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-12T16:47:08.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:07.413683+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:08.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:05.447511+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:05.447579+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:07.409260+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:07.413145+0000 mon.a (mon.0) 296 : cluster [INF] osd.1 v1:172.21.15.33:6805/1723231708 boot 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:07.413179+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:07.413683+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:05.447511+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:05.447579+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:07.409260+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:07.413145+0000 mon.a (mon.0) 296 : cluster [INF] osd.1 v1:172.21.15.33:6805/1723231708 boot 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:07.413179+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-12T16:47:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:07.413683+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:47:09.669 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-12T16:47:09.670 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-12T16:47:09.670 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-02-12T16:47:09.670 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-02-12T16:47:09.670 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108275 s, 968 MB/s 2024-02-12T16:47:09.670 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:08.339882+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:08.423190+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:08.878809+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:09.194716+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:09.195277+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:09.199328+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:09.200040+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:09.200672+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:09.204364+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:09.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:08.339882+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:09.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:08.423190+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-12T16:47:09.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:08.878809+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:09.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:09.194716+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:09.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:09.195277+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:09.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:09.199328+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:09.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:09.200040+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:09.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:09.200672+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:09.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:09.204364+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:08.339882+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T16:47:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:08.423190+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-12T16:47:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:08.878809+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:09.194716+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:09.195277+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:09.199328+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:09.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:09.200040+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:09.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:09.200672+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:09.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:09.204364+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:10.338 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_2 2024-02-12T16:47:10.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:47:08.874188+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:10.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:47:09.195657+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 8051M 2024-02-12T16:47:10.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:47:08.874188+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:10.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:47:09.195657+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 8051M 2024-02-12T16:47:10.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:47:08.874188+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:10.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:47:09.195657+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 8051M 2024-02-12T16:47:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:10.340203+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:10.340203+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:11.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:10.340203+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:13.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:12.340455+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:13.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:12.524563+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.14265 v1:172.21.15.33:0/2461144339' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:47:13.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:12.526537+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:47:13.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:12.528502+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:47:13.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:12.528988+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:13.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:12.340455+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:13.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:12.524563+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.14265 v1:172.21.15.33:0/2461144339' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:47:13.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:12.526537+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:47:13.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:12.528502+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:47:13.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:12.528988+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:13.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:12.340455+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:12.524563+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.14265 v1:172.21.15.33:0/2461144339' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:47:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:12.526537+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:47:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:12.528502+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:47:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:12.528988+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:15.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:14.340744+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:15.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:15.104280+0000 mon.a (mon.0) 310 : audit [INF] from='client.? v1:172.21.15.33:0/2148447759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c"}]: dispatch 2024-02-12T16:47:15.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:15.108585+0000 mon.a (mon.0) 311 : audit [INF] from='client.? v1:172.21.15.33:0/2148447759' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c"}]': finished 2024-02-12T16:47:15.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:15.108643+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-12T16:47:15.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:15.108745+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:15.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:14.340744+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:15.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:15.104280+0000 mon.a (mon.0) 310 : audit [INF] from='client.? v1:172.21.15.33:0/2148447759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c"}]: dispatch 2024-02-12T16:47:15.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:15.108585+0000 mon.a (mon.0) 311 : audit [INF] from='client.? v1:172.21.15.33:0/2148447759' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c"}]': finished 2024-02-12T16:47:15.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:15.108643+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-12T16:47:15.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:15.108745+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:15.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:14.340744+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:15.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:15.104280+0000 mon.a (mon.0) 310 : audit [INF] from='client.? v1:172.21.15.33:0/2148447759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c"}]: dispatch 2024-02-12T16:47:15.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:15.108585+0000 mon.a (mon.0) 311 : audit [INF] from='client.? v1:172.21.15.33:0/2148447759' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c"}]': finished 2024-02-12T16:47:15.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:15.108643+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-12T16:47:15.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:15.108745+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:16.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:15.720802+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? v1:172.21.15.33:0/2394457295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:47:16.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:15.720802+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? v1:172.21.15.33:0/2394457295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:47:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:15.720802+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? v1:172.21.15.33:0/2394457295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:47:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:16.341138+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:16.341138+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:17.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:16.341138+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:18.341459+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:19.772 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:18.341459+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:19.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:18.341459+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:20.341924+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:21.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:20.341924+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:20.341924+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:21.605426+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-12T16:47:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:21.606131+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:47:21.606728+0000 mgr.y (mgr.14146) 107 : cephadm [INF] Deploying daemon osd.2 on smithi033 2024-02-12T16:47:22.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:21.605426+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-12T16:47:22.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:21.606131+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:22.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:47:21.606728+0000 mgr.y (mgr.14146) 107 : cephadm [INF] Deploying daemon osd.2 on smithi033 2024-02-12T16:47:22.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:21.605426+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-12T16:47:22.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:21.606131+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:22.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:47:21.606728+0000 mgr.y (mgr.14146) 107 : cephadm [INF] Deploying daemon osd.2 on smithi033 2024-02-12T16:47:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:22.342397+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:23.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:22.342397+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:22.342397+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:25.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:24.342651+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:25.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:24.342651+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:24.342651+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:26.342923+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:27.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:26.342923+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:27.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:26.342923+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:26.980804+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:27.011962+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:28.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:26.980804+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:28.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:27.011962+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:28.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:26.980804+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:28.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:27.011962+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:29.223 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:28.343186+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:29.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:28.343186+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:29.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:28.343186+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:30.270 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 2 on host 'smithi033' 2024-02-12T16:47:30.865 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:29.597939+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.865 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:29.598569+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:29.599205+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:29.606591+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:30.081701+0000 mon.c (mon.2) 7 : audit [INF] from='osd.2 v1:172.21.15.33:6809/3126580659' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:30.081978+0000 mon.a (mon.0) 322 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:30.268611+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:30.273386+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:29.597939+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:29.598569+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:29.599205+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:29.606591+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.866 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:30.081701+0000 mon.c (mon.2) 7 : audit [INF] from='osd.2 v1:172.21.15.33:6809/3126580659' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T16:47:30.867 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:30.081978+0000 mon.a (mon.0) 322 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T16:47:30.867 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:30.268611+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.867 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:30.273386+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:30.895 DEBUG:teuthology.orchestra.run.smithi033:osd.2> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.2.service 2024-02-12T16:47:30.898 INFO:tasks.cephadm:Deploying osd.3 on smithi033 with /dev/vg_nvme/lv_1... 2024-02-12T16:47:30.898 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-12T16:47:30.931 INFO:journalctl@ceph.osd.2.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:47:30.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:29.597939+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:29.598569+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:30.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:29.599205+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:30.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:29.606591+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:30.081701+0000 mon.c (mon.2) 7 : audit [INF] from='osd.2 v1:172.21.15.33:6809/3126580659' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T16:47:30.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:30.081978+0000 mon.a (mon.0) 322 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T16:47:30.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:30.268611+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:30.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:30.273386+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:31.778 INFO:journalctl@ceph.osd.2.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-2[58733]: debug 2024-02-12T16:47:31.613+0000 7fbd6c846700 -1 osd.2 0 waiting for initial osdmap 2024-02-12T16:47:31.778 INFO:journalctl@ceph.osd.2.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-2[58733]: debug 2024-02-12T16:47:31.616+0000 7fbd68fc2700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:47:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:30.343432+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:30.607229+0000 mon.a (mon.0) 325 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:30.607299+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:30.607421+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:30.608367+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 v1:172.21.15.33:6809/3126580659' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:30.608653+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:30.343432+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:30.607229+0000 mon.a (mon.0) 325 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:30.607299+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:30.607421+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:30.608367+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 v1:172.21.15.33:6809/3126580659' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:31.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:30.608653+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:30.343432+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:30.607229+0000 mon.a (mon.0) 325 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-12T16:47:31.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:30.607299+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-12T16:47:31.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:30.607421+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:31.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:30.608367+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 v1:172.21.15.33:6809/3126580659' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:31.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:30.608653+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:33.167 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:31.609989+0000 mon.a (mon.0) 329 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:33.167 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:31.610071+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-12T16:47:33.167 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:31.610449+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:33.167 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:31.614001+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:33.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:31.609989+0000 mon.a (mon.0) 329 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:33.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:31.610071+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-12T16:47:33.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:31.610449+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:33.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:31.614001+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:31.609989+0000 mon.a (mon.0) 329 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:31.610071+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-12T16:47:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:31.610449+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:31.614001+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:33.740 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-12T16:47:33.740 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-12T16:47:33.740 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-02-12T16:47:33.741 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-02-12T16:47:33.741 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108895 s, 963 MB/s 2024-02-12T16:47:33.741 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:31.112343+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:31.112416+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:32.343848+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:32.616213+0000 mon.a (mon.0) 333 : cluster [INF] osd.2 v1:172.21.15.33:6809/3126580659 boot 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:32.616264+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:32.616690+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:32.617254+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:33.616157+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:33.616215+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-12T16:47:34.380 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:33.616964+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:31.112343+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:31.112416+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:32.343848+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:32.616213+0000 mon.a (mon.0) 333 : cluster [INF] osd.2 v1:172.21.15.33:6809/3126580659 boot 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:32.616264+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:32.616690+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:32.617254+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:33.616157+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:33.616215+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-12T16:47:34.381 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:33.616964+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:47:34.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:31.112343+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:31.112416+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:32.343848+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:32.616213+0000 mon.a (mon.0) 333 : cluster [INF] osd.2 v1:172.21.15.33:6809/3126580659 boot 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:32.616264+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:32.616690+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:32.617254+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:33.616157+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:33.616215+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-12T16:47:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:33.616964+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:47:34.630 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_1 2024-02-12T16:47:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:34.344237+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:34.617593+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T16:47:35.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:34.617678+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-12T16:47:35.992 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:34.344237+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:35.992 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:34.617593+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T16:47:35.992 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:34.617678+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-12T16:47:35.993 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:34.344237+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:35.993 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:34.617593+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T16:47:35.993 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:34.617678+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-12T16:47:36.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:35.623972+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-12T16:47:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:35.623972+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-12T16:47:36.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:35.623972+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-12T16:47:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:36.344702+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:36.851015+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:36.851914+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:36.852604+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:36.853194+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:36.859241+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:36.860209+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:36.860835+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:36.867282+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:37.021604+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:37.023515+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:37.023995+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:36.344702+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:36.851015+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:36.851914+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:36.852604+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:36.853194+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:36.859241+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:36.860209+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:36.860835+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:36.867282+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:37.021604+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:47:37.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:37.023515+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:47:37.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:37.023995+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:37.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:36.344702+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:36.851015+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:36.851914+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:36.852604+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:36.853194+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:36.859241+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:36.860209+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:36.860835+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:36.867282+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:37.021604+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:37.023515+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:47:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:37.023995+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:47:36.846820+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:47:36.853650+0000 mgr.y (mgr.14146) 117 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 5367M 2024-02-12T16:47:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:37.019610+0000 mgr.y (mgr.14146) 118 : audit [DBG] from='client.34199 v1:172.21.15.33:0/4012111811' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:47:38.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:47:36.846820+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:38.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:47:36.853650+0000 mgr.y (mgr.14146) 117 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 5367M 2024-02-12T16:47:38.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:37.019610+0000 mgr.y (mgr.14146) 118 : audit [DBG] from='client.34199 v1:172.21.15.33:0/4012111811' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:47:38.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:47:36.846820+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:38.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:47:36.853650+0000 mgr.y (mgr.14146) 117 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 5367M 2024-02-12T16:47:38.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:37.019610+0000 mgr.y (mgr.14146) 118 : audit [DBG] from='client.34199 v1:172.21.15.33:0/4012111811' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:47:39.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:38.345064+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:38.345064+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:40.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:38.345064+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:40.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:39.754300+0000 mon.a (mon.0) 354 : audit [INF] from='client.? v1:172.21.15.33:0/2374895595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9"}]: dispatch 2024-02-12T16:47:40.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:39.758193+0000 mon.a (mon.0) 355 : audit [INF] from='client.? v1:172.21.15.33:0/2374895595' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9"}]': finished 2024-02-12T16:47:40.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:39.758242+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-12T16:47:40.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:39.758374+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:40.361812+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.33:0/1685900488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:47:41.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:39.754300+0000 mon.a (mon.0) 354 : audit [INF] from='client.? v1:172.21.15.33:0/2374895595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9"}]: dispatch 2024-02-12T16:47:41.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:39.758193+0000 mon.a (mon.0) 355 : audit [INF] from='client.? v1:172.21.15.33:0/2374895595' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9"}]': finished 2024-02-12T16:47:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:39.758242+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-12T16:47:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:39.758374+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:40.361812+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.33:0/1685900488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:47:41.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:39.754300+0000 mon.a (mon.0) 354 : audit [INF] from='client.? v1:172.21.15.33:0/2374895595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9"}]: dispatch 2024-02-12T16:47:41.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:39.758193+0000 mon.a (mon.0) 355 : audit [INF] from='client.? v1:172.21.15.33:0/2374895595' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9"}]': finished 2024-02-12T16:47:41.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:39.758242+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-12T16:47:41.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:39.758374+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:41.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:40.361812+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.33:0/1685900488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:47:42.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:40.345344+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:40.345344+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:40.345344+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:44.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:42.345813+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:42.345813+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:42.345813+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:45.821 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:44.346170+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:45.821 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:44.346170+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:44.346170+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:46.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:46.088015+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-12T16:47:46.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:46.088728+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:46.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:47:46.089263+0000 mgr.y (mgr.14146) 123 : cephadm [INF] Deploying daemon osd.3 on smithi033 2024-02-12T16:47:46.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:46.346455+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:47.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:46.088015+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-12T16:47:47.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:46.088728+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:47.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:47:46.089263+0000 mgr.y (mgr.14146) 123 : cephadm [INF] Deploying daemon osd.3 on smithi033 2024-02-12T16:47:47.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:46.346455+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:47.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:46.088015+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-12T16:47:47.263 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:46.088728+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:47.263 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:47:46.089263+0000 mgr.y (mgr.14146) 123 : cephadm [INF] Deploying daemon osd.3 on smithi033 2024-02-12T16:47:47.263 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:46.346455+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:48.346918+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:48.346918+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:48.346918+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:51.411 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:49.929792+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:51.411 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:49.929866+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:51.411 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:50.347413+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:51.411 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:50.408819+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-12T16:47:51.411 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:50.408939+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:51.412 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:51.147660+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:51.412 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:51.159933+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:51.670 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:49.929792+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:51.670 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:49.929866+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:51.670 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:50.347413+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:51.670 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:50.408819+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-12T16:47:51.670 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:50.408939+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:51.671 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:51.147660+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:51.671 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:51.159933+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:49.929792+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:49.929866+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:50.347413+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:50.408819+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-12T16:47:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:50.408939+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:51.147660+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:51.159933+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:53.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:52.347813+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:53.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:52.347813+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:52.347813+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:54.573 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 3 on host 'smithi033' 2024-02-12T16:47:55.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:53.890132+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:53.891018+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:55.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:53.891661+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:55.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:53.896527+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:54.271609+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 v1:172.21.15.33:6813/3941314957' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:47:55.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:54.272208+0000 mon.a (mon.0) 368 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:47:55.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:54.348258+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:55.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:54.571506+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:54.613250+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:55.258 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:53.890132+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.258 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:53.891018+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:55.258 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:53.891661+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:53.896527+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:54.271609+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 v1:172.21.15.33:6813/3941314957' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:54.272208+0000 mon.a (mon.0) 368 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:54.348258+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:54.571506+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:54.613250+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:53.890132+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:53.891018+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:53.891661+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:53.896527+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:54.271609+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 v1:172.21.15.33:6813/3941314957' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:54.272208+0000 mon.a (mon.0) 368 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:47:55.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:54.348258+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:55.260 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:54.571506+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:55.260 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:54.613250+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:55.287 DEBUG:teuthology.orchestra.run.smithi033:osd.3> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.3.service 2024-02-12T16:47:55.289 INFO:tasks.cephadm:Deploying osd.4 on smithi094 with /dev/vg_nvme/lv_4... 2024-02-12T16:47:55.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-12T16:47:55.325 INFO:journalctl@ceph.osd.3.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:47:55.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:54.904062+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T16:47:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:54.904106+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-12T16:47:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:54.904219+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:54.907119+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 v1:172.21.15.33:6813/3941314957' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:54.907403+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:56.245 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-3[63188]: debug 2024-02-12T16:47:55.913+0000 7efd8e90a700 -1 osd.3 0 waiting for initial osdmap 2024-02-12T16:47:56.246 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-3[63188]: debug 2024-02-12T16:47:55.917+0000 7efd8707e700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:47:56.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:54.904062+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T16:47:56.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:54.904106+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-12T16:47:56.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:54.904219+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:56.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:54.907119+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 v1:172.21.15.33:6813/3941314957' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:56.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:54.907403+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:56.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:54.904062+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T16:47:56.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:54.904106+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-12T16:47:56.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:54.904219+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:56.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:54.907119+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 v1:172.21.15.33:6813/3941314957' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:56.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:54.907403+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T16:47:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:55.907261+0000 mon.a (mon.0) 375 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:55.907341+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-12T16:47:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:55.907664+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:55.914187+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:56.348615+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:55.907261+0000 mon.a (mon.0) 375 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:55.907341+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:55.907664+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:55.914187+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:56.348615+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:55.907261+0000 mon.a (mon.0) 375 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:55.907341+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:55.907664+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:55.914187+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:56.348615+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T16:47:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-12T16:47:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-12T16:47:57.321 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-02-12T16:47:57.321 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-02-12T16:47:57.322 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0091771 s, 1.1 GB/s 2024-02-12T16:47:57.322 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-02-12T16:47:57.915 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:55.303422+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:57.915 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:55.303474+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:57.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi094:vg_nvme/lv_4 2024-02-12T16:47:58.143 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:55.303422+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:58.143 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:55.303474+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:58.143 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:56.915298+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:58.143 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:56.923264+0000 mon.a (mon.0) 380 : cluster [INF] osd.3 v1:172.21.15.33:6813/3941314957 boot 2024-02-12T16:47:58.143 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:56.923295+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-12T16:47:58.143 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:56.923605+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:58.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:55.303422+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:58.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:55.303474+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:58.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:56.915298+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:58.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:56.923264+0000 mon.a (mon.0) 380 : cluster [INF] osd.3 v1:172.21.15.33:6813/3941314957 boot 2024-02-12T16:47:58.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:56.923295+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-12T16:47:58.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:56.923605+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:58.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:56.915298+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:58.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:56.923264+0000 mon.a (mon.0) 380 : cluster [INF] osd.3 v1:172.21.15.33:6813/3941314957 boot 2024-02-12T16:47:58.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:56.923295+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-12T16:47:58.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:56.923605+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:47:59.212 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:57.928424+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-12T16:47:59.212 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:47:58.348952+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:47:59.212 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:47:58.386692+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.390949+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.391798+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.392339+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.392857+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.393342+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:47:58.393766+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 4025M 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.396818+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.397672+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.398202+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:59.213 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:58.402533+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:57.928424+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:47:58.348952+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:47:58.386692+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.390949+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.391798+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.392339+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.392857+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.393342+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:47:58.393766+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 4025M 2024-02-12T16:47:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.396818+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.397672+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.398202+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:58.402533+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:57.928424+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:47:58.348952+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:47:58.386692+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi033 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.390949+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.391798+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.392339+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.392857+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.393342+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:47:59.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:47:58.393766+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 4025M 2024-02-12T16:47:59.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.396818+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:47:59.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.397672+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:59.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.398202+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:59.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:58.402533+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:59.871016+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:59.873424+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:00.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:47:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:59.874133+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:00.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:59.871016+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:00.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:59.873424+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:00.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:47:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:59.874133+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:59.871016+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:59.873424+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:47:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:59.874133+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:47:59.868457+0000 mgr.y (mgr.14146) 133 : audit [DBG] from='client.24232 v1:172.21.15.94:0/2727892340' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:00.349350+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:01.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:47:59.868457+0000 mgr.y (mgr.14146) 133 : audit [DBG] from='client.24232 v1:172.21.15.94:0/2727892340' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:01.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:00.349350+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:01.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:47:59.868457+0000 mgr.y (mgr.14146) 133 : audit [DBG] from='client.24232 v1:172.21.15.94:0/2727892340' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:01.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:00.349350+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:02.268268+0000 mon.b (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.94:0/1380096622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]: dispatch 2024-02-12T16:48:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:02.268741+0000 mon.a (mon.0) 396 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]: dispatch 2024-02-12T16:48:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:02.273382+0000 mon.a (mon.0) 397 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]': finished 2024-02-12T16:48:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:02.273458+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-12T16:48:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:02.273593+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:02.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:02.268268+0000 mon.b (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.94:0/1380096622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]: dispatch 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:02.268741+0000 mon.a (mon.0) 396 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]: dispatch 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:02.273382+0000 mon.a (mon.0) 397 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]': finished 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:02.273458+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:02.273593+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:02.268268+0000 mon.b (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.94:0/1380096622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]: dispatch 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:02.268741+0000 mon.a (mon.0) 396 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]: dispatch 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:02.273382+0000 mon.a (mon.0) 397 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0498129c-861c-4f7e-b8ba-219949da3130"}]': finished 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:02.273458+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-12T16:48:02.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:02.273593+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:02.349829+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:02.842276+0000 mon.b (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.94:0/1449395137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:03.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:02.349829+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:03.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:02.842276+0000 mon.b (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.94:0/1449395137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:02.349829+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:02.842276+0000 mon.b (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.94:0/1449395137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:05.643 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:04.350212+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:05.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:04.350212+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:05.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:04.350212+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:06.975 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:06.350580+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:07.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:06.350580+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:06.350580+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:07.474388+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:07.474514+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:07.929042+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-12T16:48:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:07.929208+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:08.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:08.243679+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-12T16:48:08.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:08.244489+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:08.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:48:08.245177+0000 mgr.y (mgr.14146) 138 : cephadm [INF] Deploying daemon osd.4 on smithi094 2024-02-12T16:48:08.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:08.350969+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:07.474388+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:07.474514+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:07.929042+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:07.929208+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:08.243679+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:08.244489+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:48:08.245177+0000 mgr.y (mgr.14146) 138 : cephadm [INF] Deploying daemon osd.4 on smithi094 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:08.350969+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:07.474388+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:07.474514+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:07.929042+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-12T16:48:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:07.929208+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:08.243679+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-12T16:48:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:08.244489+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:48:08.245177+0000 mgr.y (mgr.14146) 138 : cephadm [INF] Deploying daemon osd.4 on smithi094 2024-02-12T16:48:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:08.350969+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:10.351462+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:11.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:10.351462+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:11.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:10.351462+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:13.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:12.351833+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:13.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:12.924502+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:13.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:12.954176+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:13.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:12.351833+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:13.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:12.924502+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:13.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:12.954176+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:12.351833+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:12.924502+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:12.954176+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:14.352173+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:15.341221+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:15.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:14.352173+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:15.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:15.341221+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:15.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:14.352173+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:15.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:15.341221+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:16.072 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 4 on host 'smithi094' 2024-02-12T16:48:16.699 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:15.731559+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:16.699 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:15.732425+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:16.699 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:15.737650+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:16.699 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:16.016324+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 v1:172.21.15.94:6800/544126779' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T16:48:16.699 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:16.016717+0000 mon.a (mon.0) 410 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T16:48:16.699 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:16.069589+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:16.699 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:16.105183+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:16.720 DEBUG:teuthology.orchestra.run.smithi094:osd.4> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.4.service 2024-02-12T16:48:16.722 INFO:tasks.cephadm:Deploying osd.5 on smithi094 with /dev/vg_nvme/lv_3... 2024-02-12T16:48:16.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-12T16:48:16.751 INFO:journalctl@ceph.osd.4.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:48:16.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:15.731559+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:16.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:15.732425+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:15.737650+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:16.016324+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 v1:172.21.15.94:6800/544126779' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:16.016717+0000 mon.a (mon.0) 410 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:16.069589+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:16.105183+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:15.731559+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:15.732425+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:15.737650+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:16.016324+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 v1:172.21.15.94:6800/544126779' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:16.016717+0000 mon.a (mon.0) 410 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:16.069589+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:16.105183+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:16.352536+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:16.923917+0000 mon.a (mon.0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-12T16:48:17.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:16.924051+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-12T16:48:17.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:16.924198+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:17.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:16.924477+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.94:6800/544126779' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:17.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:16.924869+0000 mon.a (mon.0) 416 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:16.352536+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:16.923917+0000 mon.a (mon.0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:16.924051+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:16.924198+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:16.924477+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.94:6800/544126779' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:16.924869+0000 mon.a (mon.0) 416 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:16.352536+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:16.923917+0000 mon.a (mon.0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:16.924051+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:16.924198+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:17.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:16.924477+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.94:6800/544126779' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:17.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:16.924869+0000 mon.a (mon.0) 416 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:18.240 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 16:48:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T16:48:17.927+0000 7f2bb4b5d700 -1 osd.4 0 waiting for initial osdmap 2024-02-12T16:48:18.240 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 16:48:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T16:48:17.932+0000 7f2badad2700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:48:18.973 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:17.925551+0000 mon.a (mon.0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:48:18.973 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:17.925679+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-02-12T16:48:18.973 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:17.925845+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:18.973 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:17.928571+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:18.973 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:18.352926+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:19.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:17.925551+0000 mon.a (mon.0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:17.925679+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:17.925845+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:17.928571+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:18.352926+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:17.925551+0000 mon.a (mon.0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:17.925679+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:17.925845+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:17.928571+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:19.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:18.352926+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T16:48:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:16.993212+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:16.993263+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:18.931322+0000 mon.a (mon.0) 421 : cluster [INF] osd.4 v1:172.21.15.94:6800/544126779 boot 2024-02-12T16:48:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:18.931438+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-12T16:48:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:18.931788+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:20.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:16.993212+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:16.993263+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:18.931322+0000 mon.a (mon.0) 421 : cluster [INF] osd.4 v1:172.21.15.94:6800/544126779 boot 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:18.931438+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:18.931788+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:16.993212+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:16.993263+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:18.931322+0000 mon.a (mon.0) 421 : cluster [INF] osd.4 v1:172.21.15.94:6800/544126779 boot 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:18.931438+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-12T16:48:20.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:18.931788+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:48:21.011 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-12T16:48:21.011 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-12T16:48:21.011 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-02-12T16:48:21.011 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-02-12T16:48:21.011 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00920098 s, 1.1 GB/s 2024-02-12T16:48:21.011 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-02-12T16:48:21.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:19.933193+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-12T16:48:21.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:20.353292+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v130: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:48:20.742454+0000 mgr.y (mgr.14146) 146 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:20.747037+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:20.748023+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:48:20.748745+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 17122M 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:20.755078+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:20.756086+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:20.756920+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:20.764129+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:20.938512+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:19.933193+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:20.353292+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v130: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:48:20.742454+0000 mgr.y (mgr.14146) 146 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:20.747037+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:20.748023+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:48:20.748745+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 17122M 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:20.755078+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:20.756086+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:20.756920+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:20.764129+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:20.938512+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:19.933193+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:20.353292+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v130: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:48:20.742454+0000 mgr.y (mgr.14146) 146 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:20.747037+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:20.748023+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:48:20.748745+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 17122M 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:20.755078+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:20.756086+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:20.756920+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:20.764129+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:21.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:20.938512+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2024-02-12T16:48:21.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi094:vg_nvme/lv_3 2024-02-12T16:48:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:22.353672+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v132: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:22.353672+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v132: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:22.353672+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v132: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:24.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:23.663698+0000 mgr.y (mgr.14146) 149 : audit [DBG] from='client.24259 v1:172.21.15.94:0/2032498192' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:24.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:23.666111+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:24.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:23.668581+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:24.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:23.669270+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:24.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:23.663698+0000 mgr.y (mgr.14146) 149 : audit [DBG] from='client.24259 v1:172.21.15.94:0/2032498192' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:24.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:23.666111+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:24.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:23.668581+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:24.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:23.669270+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:23.663698+0000 mgr.y (mgr.14146) 149 : audit [DBG] from='client.24259 v1:172.21.15.94:0/2032498192' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:23.666111+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:23.668581+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:23.669270+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:25.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:24.354116+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:25.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:24.354116+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:24.354116+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:26.077428+0000 mon.b (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.94:0/144319316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]: dispatch 2024-02-12T16:48:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:26.077976+0000 mon.a (mon.0) 435 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]: dispatch 2024-02-12T16:48:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:26.082972+0000 mon.a (mon.0) 436 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]': finished 2024-02-12T16:48:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:26.083063+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-12T16:48:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:26.083248+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:26.354427+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:26.668810+0000 mon.b (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.94:0/2021305072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:26.077428+0000 mon.b (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.94:0/144319316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]: dispatch 2024-02-12T16:48:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:26.077976+0000 mon.a (mon.0) 435 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]: dispatch 2024-02-12T16:48:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:26.082972+0000 mon.a (mon.0) 436 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]': finished 2024-02-12T16:48:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:26.083063+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-12T16:48:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:26.083248+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:26.354427+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:26.668810+0000 mon.b (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.94:0/2021305072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:26.077428+0000 mon.b (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.94:0/144319316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]: dispatch 2024-02-12T16:48:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:26.077976+0000 mon.a (mon.0) 435 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]: dispatch 2024-02-12T16:48:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:26.082972+0000 mon.a (mon.0) 436 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50"}]': finished 2024-02-12T16:48:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:26.083063+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-12T16:48:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:26.083248+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:26.354427+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:26.668810+0000 mon.b (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.94:0/2021305072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:29.558 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:28.354764+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:29.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:28.354764+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:29.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:28.354764+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:30.355086+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:31.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:30.355086+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:30.355086+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:32.150318+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-12T16:48:32.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:32.151169+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:32.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:32.150318+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-12T16:48:32.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:32.151169+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:32.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:32.150318+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-12T16:48:32.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:32.151169+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:48:32.151949+0000 mgr.y (mgr.14146) 154 : cephadm [INF] Deploying daemon osd.5 on smithi094 2024-02-12T16:48:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:32.355552+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:33.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:48:32.151949+0000 mgr.y (mgr.14146) 154 : cephadm [INF] Deploying daemon osd.5 on smithi094 2024-02-12T16:48:33.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:32.355552+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:48:32.151949+0000 mgr.y (mgr.14146) 154 : cephadm [INF] Deploying daemon osd.5 on smithi094 2024-02-12T16:48:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:32.355552+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:34.356015+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:35.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:34.356015+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:35.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:34.356015+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:36.356416+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:37.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:36.356416+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:37.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:36.356416+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:37.121961+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:37.122520+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:38.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:37.121961+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:38.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:37.122520+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:37.121961+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:37.122520+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:39.360 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:38.356840+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:38.356840+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:38.356840+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:40.452 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 5 on host 'smithi094' 2024-02-12T16:48:40.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:39.726435+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:40.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:39.727295+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:40.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:39.728153+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:39.733161+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:40.263333+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 v1:172.21.15.94:6804/3229884005' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T16:48:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:40.263823+0000 mon.a (mon.0) 447 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T16:48:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:40.450022+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:40.451283+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:39.726435+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:39.727295+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:39.728153+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:39.733161+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:40.263333+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 v1:172.21.15.94:6804/3229884005' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:40.263823+0000 mon.a (mon.0) 447 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:40.450022+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:40.451283+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:41.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:39.726435+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:39.727295+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:39.728153+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:39.733161+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:40.263333+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 v1:172.21.15.94:6804/3229884005' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T16:48:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:40.263823+0000 mon.a (mon.0) 447 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T16:48:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:40.450022+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:40.451283+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:41.191 DEBUG:teuthology.orchestra.run.smithi094:osd.5> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.5.service 2024-02-12T16:48:41.193 INFO:tasks.cephadm:Deploying osd.6 on smithi094 with /dev/vg_nvme/lv_2... 2024-02-12T16:48:41.193 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-12T16:48:41.222 INFO:journalctl@ceph.osd.5.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:48:41.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 16:48:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T16:48:41.740+0000 7f941ef9a700 -1 osd.5 0 waiting for initial osdmap 2024-02-12T16:48:41.991 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 16:48:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T16:48:41.747+0000 7f9419712700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:48:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:40.357350+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:40.737027+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-12T16:48:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:40.737103+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-12T16:48:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:40.737228+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:40.737506+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.94:6804/3229884005' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:40.737891+0000 mon.a (mon.0) 453 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:42.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:40.357350+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:42.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:40.737027+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:40.737103+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:40.737228+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:40.737506+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.94:6804/3229884005' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:40.737891+0000 mon.a (mon.0) 453 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:40.357350+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:40.737027+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:40.737103+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:40.737228+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:40.737506+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.94:6804/3229884005' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:40.737891+0000 mon.a (mon.0) 453 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:48:42.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:41.739138+0000 mon.a (mon.0) 454 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:48:42.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:41.739233+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-12T16:48:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:41.739750+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:41.744127+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:41.739138+0000 mon.a (mon.0) 454 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:48:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:41.739233+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-12T16:48:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:41.739750+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:41.744127+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:41.739138+0000 mon.a (mon.0) 454 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:48:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:41.739233+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-12T16:48:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:41.739750+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:41.744127+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:43.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:41.225511+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:41.225570+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:42.357765+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:42.744722+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:42.750832+0000 mon.a (mon.0) 459 : cluster [INF] osd.5 v1:172.21.15.94:6804/3229884005 boot 2024-02-12T16:48:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:42.750863+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-12T16:48:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:42.751024+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:44.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:41.225511+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:41.225570+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:42.357765+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:42.744722+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:42.750832+0000 mon.a (mon.0) 459 : cluster [INF] osd.5 v1:172.21.15.94:6804/3229884005 boot 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:42.750863+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:42.751024+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:41.225511+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:41.225570+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:42.357765+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:42.744722+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:42.750832+0000 mon.a (mon.0) 459 : cluster [INF] osd.5 v1:172.21.15.94:6804/3229884005 boot 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:42.750863+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-12T16:48:44.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:42.751024+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:48:44.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:43.752392+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-12T16:48:44.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:44.638149+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:44.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:44.639119+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:44.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:44.640056+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:44.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:44.648555+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:44.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:44.649464+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:44.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:44.650177+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:44.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:44.657909+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:43.752392+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:44.638149+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:44.639119+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:44.640056+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:44.648555+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:44.649464+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:44.650177+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:44.657909+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:43.752392+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-12T16:48:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:44.638149+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:44.639119+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:44.640056+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:48:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:44.648555+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:44.649464+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:44.650177+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:48:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:44.657909+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:48:45.150 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-12T16:48:45.150 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-12T16:48:45.150 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-02-12T16:48:45.150 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-02-12T16:48:45.150 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00926178 s, 1.1 GB/s 2024-02-12T16:48:45.150 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-02-12T16:48:45.831 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:44.358237+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:45.831 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:48:44.633228+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:48:45.831 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:48:44.640705+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 8561M 2024-02-12T16:48:45.831 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:44.766652+0000 mon.a (mon.0) 470 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-12T16:48:45.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi094:vg_nvme/lv_2 2024-02-12T16:48:46.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:44.358237+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:48:44.633228+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:48:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:48:44.640705+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 8561M 2024-02-12T16:48:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:44.766652+0000 mon.a (mon.0) 470 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-12T16:48:46.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:44.358237+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:46.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:48:44.633228+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:48:46.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:48:44.640705+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 8561M 2024-02-12T16:48:46.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:44.766652+0000 mon.a (mon.0) 470 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-12T16:48:47.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:46.358648+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:47.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:46.358648+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:47.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:46.358648+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:48.191 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:47.709558+0000 mgr.y (mgr.14146) 165 : audit [DBG] from='client.24286 v1:172.21.15.94:0/678229361' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:48.192 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:47.711908+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:48.192 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:47.714255+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:48.192 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:47.715013+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:47.709558+0000 mgr.y (mgr.14146) 165 : audit [DBG] from='client.24286 v1:172.21.15.94:0/678229361' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:47.711908+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:47.714255+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:47.715013+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:47.709558+0000 mgr.y (mgr.14146) 165 : audit [DBG] from='client.24286 v1:172.21.15.94:0/678229361' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:48:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:47.711908+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:48:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:47.714255+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:48:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:47.715013+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:49.141 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:48.359029+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:49.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:48.359029+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:49.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:48.359029+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:50.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:50.402848+0000 mon.b (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.94:0/2888302046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]: dispatch 2024-02-12T16:48:50.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:50.403476+0000 mon.a (mon.0) 474 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]: dispatch 2024-02-12T16:48:50.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:50.408516+0000 mon.a (mon.0) 475 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]': finished 2024-02-12T16:48:50.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:50.408574+0000 mon.a (mon.0) 476 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-12T16:48:50.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:50.408707+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:50.402848+0000 mon.b (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.94:0/2888302046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]: dispatch 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:50.403476+0000 mon.a (mon.0) 474 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]: dispatch 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:50.408516+0000 mon.a (mon.0) 475 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]': finished 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:50.408574+0000 mon.a (mon.0) 476 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:50.408707+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:50.402848+0000 mon.b (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.94:0/2888302046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]: dispatch 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:50.403476+0000 mon.a (mon.0) 474 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]: dispatch 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:50.408516+0000 mon.a (mon.0) 475 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821"}]': finished 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:50.408574+0000 mon.a (mon.0) 476 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-12T16:48:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:50.408707+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:48:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:50.359417+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:50.973859+0000 mon.b (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.94:0/1282395055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:50.359417+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:50.973859+0000 mon.b (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.94:0/1282395055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:51.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:50.359417+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:51.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:50.973859+0000 mon.b (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.94:0/1282395055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:48:53.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:52.359779+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:53.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:52.359779+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:53.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:52.359779+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:55.534 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:54.168637+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:55.534 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:54.168728+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:55.534 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:54.360170+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:55.534 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:54.468133+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-12T16:48:55.534 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:54.468322+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:48:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:54.168637+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:54.168728+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:54.360170+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:54.468133+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-12T16:48:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:54.468322+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:48:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:54.168637+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:55.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:54.168728+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:55.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:54.360170+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:55.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:54.468133+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-12T16:48:55.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:54.468322+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:48:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:56.360666+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:56.587023+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-12T16:48:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:48:56.587863+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:48:56.588585+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Deploying daemon osd.6 on smithi094 2024-02-12T16:48:57.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:56.360666+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:57.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:56.587023+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-12T16:48:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:48:56.587863+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:48:56.588585+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Deploying daemon osd.6 on smithi094 2024-02-12T16:48:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:56.360666+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:56.587023+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-12T16:48:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:48:56.587863+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:48:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:48:56.588585+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Deploying daemon osd.6 on smithi094 2024-02-12T16:48:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:48:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:48:58.361087+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:48:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:48:58.361087+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:48:59.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:48:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:48:58.361087+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:00.361689+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:00.361689+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:01.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:00.361689+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:01.466877+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:01.469660+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:02.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:01.466877+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:02.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:01.469660+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:02.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:01.466877+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:02.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:01.469660+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:02.362138+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:03.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:02.362138+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:02.362138+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:04.690 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 6 on host 'smithi094' 2024-02-12T16:49:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:03.924821+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:03.925664+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:03.926529+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:03.931222+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:04.259218+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:04.262280+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:04.362659+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:04.596080+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 v1:172.21.15.94:6808/507305558' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:04.596544+0000 mon.a (mon.0) 488 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:04.687583+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:04.699033+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:05.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:03.924821+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:05.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:03.925664+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:03.926529+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:03.931222+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:04.259218+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:04.262280+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:04.362659+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:04.596080+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 v1:172.21.15.94:6808/507305558' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:04.596544+0000 mon.a (mon.0) 488 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:04.687583+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:04.699033+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:03.924821+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:03.925664+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:03.926529+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:03.931222+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:04.259218+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:04.262280+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:04.362659+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:04.596080+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 v1:172.21.15.94:6808/507305558' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:04.596544+0000 mon.a (mon.0) 488 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:04.687583+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:05.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:04.699033+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:05.399 DEBUG:teuthology.orchestra.run.smithi094:osd.6> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.6.service 2024-02-12T16:49:05.402 INFO:tasks.cephadm:Deploying osd.7 on smithi094 with /dev/vg_nvme/lv_1... 2024-02-12T16:49:05.402 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-12T16:49:05.435 INFO:journalctl@ceph.osd.6.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:49:06.240 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 16:49:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T16:49:05.969+0000 7f1b9b426700 -1 osd.6 0 waiting for initial osdmap 2024-02-12T16:49:06.241 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 16:49:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T16:49:05.975+0000 7f1b97ba2700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:49:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:04.944995+0000 mon.a (mon.0) 491 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-12T16:49:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:04.945097+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-12T16:49:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:04.945222+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:04.945547+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.94:6808/507305558' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:04.945955+0000 mon.a (mon.0) 494 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:04.944995+0000 mon.a (mon.0) 491 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:04.945097+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:04.945222+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:04.945547+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.94:6808/507305558' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:04.945955+0000 mon.a (mon.0) 494 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:04.944995+0000 mon.a (mon.0) 491 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:04.945097+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:04.945222+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:04.945547+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.94:6808/507305558' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:04.945955+0000 mon.a (mon.0) 494 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:07.142 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:05.945063+0000 mon.a (mon.0) 495 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:49:07.142 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:05.945177+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-12T16:49:07.142 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:05.945767+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:07.143 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:05.952635+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:07.143 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:06.363161+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:07.143 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:06.947134+0000 mon.a (mon.0) 499 : cluster [INF] osd.6 v1:172.21.15.94:6808/507305558 boot 2024-02-12T16:49:07.143 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:06.947199+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-12T16:49:07.143 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:06.947394+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:07.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:05.945063+0000 mon.a (mon.0) 495 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:05.945177+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:05.945767+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:05.952635+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:06.363161+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:06.947134+0000 mon.a (mon.0) 499 : cluster [INF] osd.6 v1:172.21.15.94:6808/507305558 boot 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:06.947199+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:06.947394+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:05.945063+0000 mon.a (mon.0) 495 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:49:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:05.945177+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-12T16:49:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:05.945767+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:05.952635+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:06.363161+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T16:49:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:06.947134+0000 mon.a (mon.0) 499 : cluster [INF] osd.6 v1:172.21.15.94:6808/507305558 boot 2024-02-12T16:49:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:06.947199+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-12T16:49:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:06.947394+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:49:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:05.554268+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:08.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:05.554322+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:08.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:07.947930+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-12T16:49:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:05.554268+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:05.554322+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:07.947930+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-12T16:49:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:05.554268+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:05.554322+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:07.947930+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-12T16:49:08.343 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-12T16:49:08.343 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-12T16:49:08.343 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-02-12T16:49:08.344 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-02-12T16:49:08.344 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00916125 s, 1.1 GB/s 2024-02-12T16:49:08.344 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-02-12T16:49:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:08.363645+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:08.950609+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-12T16:49:09.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:08.363645+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:09.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:08.950609+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-12T16:49:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:08.363645+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:08.950609+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-12T16:49:09.411 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi094:vg_nvme/lv_1 2024-02-12T16:49:11.327 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:10.052255+0000 mgr.y (mgr.14146) 178 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:49:11.328 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:10.057626+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.328 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:10.058675+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.328 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:10.059482+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.328 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:10.060173+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.328 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:10.060714+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 5707M 2024-02-12T16:49:11.328 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:10.064420+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.329 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:10.065353+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:11.329 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:10.066143+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:11.329 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:10.073063+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.329 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:10.364143+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:11.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:10.052255+0000 mgr.y (mgr.14146) 178 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:49:11.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:10.057626+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:10.058675+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:10.059482+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:10.060173+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:10.060714+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 5707M 2024-02-12T16:49:11.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:10.064420+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:10.065353+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:10.066143+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:10.073063+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:10.364143+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:10.052255+0000 mgr.y (mgr.14146) 178 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:10.057626+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:10.058675+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:10.059482+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:10.060173+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:10.060714+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 5707M 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:10.064420+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:10.065353+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:11.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:10.066143+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:11.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:10.073063+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:11.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:10.364143+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:12.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:11.745815+0000 mgr.y (mgr.14146) 181 : audit [DBG] from='client.24313 v1:172.21.15.94:0/1096303932' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:12.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:11.748217+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:49:12.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:11.750723+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:49:12.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:11.751407+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:12.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:11.745815+0000 mgr.y (mgr.14146) 181 : audit [DBG] from='client.24313 v1:172.21.15.94:0/1096303932' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:12.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:11.748217+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:49:12.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:11.750723+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:49:12.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:11.751407+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:11.745815+0000 mgr.y (mgr.14146) 181 : audit [DBG] from='client.24313 v1:172.21.15.94:0/1096303932' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:11.748217+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T16:49:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:11.750723+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T16:49:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:11.751407+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:13.337 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:12.364641+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:13.337 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:12.391049+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:13.337 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:12.391113+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:12.364641+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:12.391049+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:12.391113+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:12.364641+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:12.391049+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:12.391113+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:14.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:13.083324+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-12T16:49:14.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:13.083324+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-12T16:49:14.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:13.083324+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-12T16:49:15.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:13.998094+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:13.998176+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:14.084582+0000 mon.a (mon.0) 516 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2024-02-12T16:49:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:14.104030+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.94:0/714870292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]: dispatch 2024-02-12T16:49:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:14.104531+0000 mon.a (mon.0) 517 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]: dispatch 2024-02-12T16:49:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:14.114073+0000 mon.a (mon.0) 518 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]': finished 2024-02-12T16:49:15.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:14.114139+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-12T16:49:15.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:14.114296+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:15.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:14.365029+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:15.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:14.705526+0000 mon.b (mon.1) 22 : audit [DBG] from='client.? v1:172.21.15.94:0/4265358641' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:49:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:13.998094+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:13.998176+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:14.084582+0000 mon.a (mon.0) 516 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2024-02-12T16:49:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:14.104030+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.94:0/714870292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]: dispatch 2024-02-12T16:49:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:14.104531+0000 mon.a (mon.0) 517 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]: dispatch 2024-02-12T16:49:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:14.114073+0000 mon.a (mon.0) 518 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]': finished 2024-02-12T16:49:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:14.114139+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-12T16:49:15.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:14.114296+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:15.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:14.365029+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:15.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:14.705526+0000 mon.b (mon.1) 22 : audit [DBG] from='client.? v1:172.21.15.94:0/4265358641' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:49:15.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:13.998094+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:15.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:13.998176+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:15.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:14.084582+0000 mon.a (mon.0) 516 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2024-02-12T16:49:15.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:14.104030+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.94:0/714870292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]: dispatch 2024-02-12T16:49:15.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:14.104531+0000 mon.a (mon.0) 517 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]: dispatch 2024-02-12T16:49:15.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:14.114073+0000 mon.a (mon.0) 518 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66"}]': finished 2024-02-12T16:49:15.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:14.114139+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-12T16:49:15.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:14.114296+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:15.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:14.365029+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:15.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:14.705526+0000 mon.b (mon.1) 22 : audit [DBG] from='client.? v1:172.21.15.94:0/4265358641' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T16:49:17.235 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:16.365526+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:16.365526+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:16.365526+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:18.366048+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:19.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:18.366048+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:18.366048+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:20.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:20.203655+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-12T16:49:20.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:20.204450+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:20.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:20.203655+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-12T16:49:20.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:20.204450+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:20.203655+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-12T16:49:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:20.204450+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:21.648 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:20.205114+0000 mgr.y (mgr.14146) 186 : cephadm [INF] Deploying daemon osd.7 on smithi094 2024-02-12T16:49:21.648 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:20.366567+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:21.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:20.205114+0000 mgr.y (mgr.14146) 186 : cephadm [INF] Deploying daemon osd.7 on smithi094 2024-02-12T16:49:21.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:20.366567+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:21.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:20.205114+0000 mgr.y (mgr.14146) 186 : cephadm [INF] Deploying daemon osd.7 on smithi094 2024-02-12T16:49:21.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:20.366567+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:23.432 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:22.366957+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:23.432 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:22.665134+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:23.432 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:22.665198+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:23.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:22.366957+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:22.665134+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:22.665198+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:22.366957+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:22.665134+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:22.665198+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:23.435911+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-12T16:49:24.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:23.436065+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:24.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:23.435911+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-12T16:49:24.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:23.436065+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:23.435911+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-12T16:49:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:23.436065+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:25.443 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:24.367260+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:25.443 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:25.189107+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:25.443 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:25.228575+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:25.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:24.367260+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:25.189107+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:25.228575+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:25.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:24.367260+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:25.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:25.189107+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:25.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:25.228575+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:26.367767+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:26.367767+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:27.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:26.367767+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:28.723 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 7 on host 'smithi094' 2024-02-12T16:49:29.039 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:27.959868+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.040 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:28.254101+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:29.040 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:28.255049+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:29.040 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:28.260076+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.040 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:28.318397+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 v1:172.21.15.94:6812/3401977873' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:49:29.040 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:28.318789+0000 mon.a (mon.0) 531 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:49:29.040 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:28.368246+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:29.040 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:28.721690+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.040 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:28.727890+0000 mon.a (mon.0) 533 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:27.959868+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:28.254101+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:28.255049+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:28.260076+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:28.318397+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 v1:172.21.15.94:6812/3401977873' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:28.318789+0000 mon.a (mon.0) 531 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:28.368246+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:28.721690+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:28.727890+0000 mon.a (mon.0) 533 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:27.959868+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:28.254101+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:28.255049+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:28.260076+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:28.318397+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 v1:172.21.15.94:6812/3401977873' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:28.318789+0000 mon.a (mon.0) 531 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:28.368246+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:28.721690+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:28.727890+0000 mon.a (mon.0) 533 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:29.312 DEBUG:teuthology.orchestra.run.smithi094:osd.7> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.7.service 2024-02-12T16:49:29.315 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-12T16:49:29.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-12T16:49:29.346 INFO:journalctl@ceph.osd.7.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:49:30.271 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:29.264831+0000 mon.a (mon.0) 534 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T16:49:30.271 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:29.264932+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-12T16:49:30.271 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:29.265107+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:30.271 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:29.265435+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.94:6812/3401977873' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:30.271 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:29.266014+0000 mon.a (mon.0) 537 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:30.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:29.264831+0000 mon.a (mon.0) 534 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T16:49:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:29.264932+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-12T16:49:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:29.265107+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:29.265435+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.94:6812/3401977873' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:29.266014+0000 mon.a (mon.0) 537 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:30.576 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 16:49:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T16:49:30.270+0000 7f504b901700 -1 osd.7 0 waiting for initial osdmap 2024-02-12T16:49:30.576 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 16:49:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T16:49:30.277+0000 7f5044075700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:49:30.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:29.264831+0000 mon.a (mon.0) 534 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T16:49:30.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:29.264932+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-12T16:49:30.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:29.265107+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:30.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:29.265435+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.94:6812/3401977873' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:30.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:29.266014+0000 mon.a (mon.0) 537 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-02-12T16:49:31.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:30.266603+0000 mon.a (mon.0) 538 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:49:31.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:30.266678+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2024-02-12T16:49:31.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:30.267057+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:30.271286+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:30.368764+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:31.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:30.938758+0000 mon.a (mon.0) 542 : cluster [INF] osd.7 v1:172.21.15.94:6812/3401977873 boot 2024-02-12T16:49:31.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:30.938823+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-12T16:49:31.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:30.939252+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:30.266603+0000 mon.a (mon.0) 538 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:49:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:30.266678+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2024-02-12T16:49:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:30.267057+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:30.271286+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:30.368764+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:30.938758+0000 mon.a (mon.0) 542 : cluster [INF] osd.7 v1:172.21.15.94:6812/3401977873 boot 2024-02-12T16:49:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:30.938823+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-12T16:49:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:30.939252+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.557 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:30.266603+0000 mon.a (mon.0) 538 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-02-12T16:49:31.558 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:30.266678+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2024-02-12T16:49:31.558 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:30.267057+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.558 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:30.271286+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.558 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:30.368764+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T16:49:31.558 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:30.938758+0000 mon.a (mon.0) 542 : cluster [INF] osd.7 v1:172.21.15.94:6812/3401977873 boot 2024-02-12T16:49:31.558 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:30.938823+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-12T16:49:31.558 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:30.939252+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:49:31.990 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:49:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:29.328809+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:29.328869+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:31.944199+0000 mon.a (mon.0) 545 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-12T16:49:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:31.988880+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.33:0/2758036386' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-12T16:49:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:29.328809+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:29.328869+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:31.944199+0000 mon.a (mon.0) 545 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-12T16:49:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:31.988880+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.33:0/2758036386' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-12T16:49:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:29.328809+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:29.328869+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:31.944199+0000 mon.a (mon.0) 545 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-12T16:49:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:31.988880+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.33:0/2758036386' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-12T16:49:32.758 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1707756570,"num_in_osds":8,"osd_in_since":1707756554,"num_remapped_pgs":0} 2024-02-12T16:49:32.758 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi033 2024-02-12T16:49:32.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi033=foo.a' 2024-02-12T16:49:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:32.369036+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:32.444485+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:49:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.451361+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.452875+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.453520+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.454087+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.454645+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:32.455087+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 4280M 2024-02-12T16:49:33.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.461016+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.461821+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:33.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.462486+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:33.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:32.466320+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:32.952994+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:32.369036+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:32.444485+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.451361+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.452875+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.453520+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.454087+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.454645+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:32.455087+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 4280M 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.461016+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.461821+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.462486+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:32.466320+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:32.952994+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:32.369036+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:32.444485+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Detected new or changed devices on smithi094 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.451361+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.452875+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.453520+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.454087+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.454645+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:32.455087+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 4280M 2024-02-12T16:49:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.461016+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.461821+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.462486+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:32.466320+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:32.952994+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-12T16:49:34.816 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled rgw.foo.a update... 2024-02-12T16:49:35.466 DEBUG:teuthology.orchestra.run.smithi033:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@rgw.foo.a.service 2024-02-12T16:49:35.469 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi094 2024-02-12T16:49:35.469 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool create iscsi 3 3 replicated 2024-02-12T16:49:35.503 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:34.369551+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.814959+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.837973+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.839556+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.840265+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.845026+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.850109+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.850744+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-12T16:49:35.643 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.853807+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-12T16:49:35.644 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.857930+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.644 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.858669+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:35.644 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:34.369551+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:35.644 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.814959+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.644 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.837973+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:35.644 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.839556+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:35.644 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.840265+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:35.644 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.845026+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.645 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.850109+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.645 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.850744+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-12T16:49:35.645 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.853807+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-12T16:49:35.645 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.857930+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.645 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.858669+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:34.369551+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.814959+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.837973+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.839556+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.840265+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.845026+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.850109+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.850744+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.853807+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.857930+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.858669+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:36.593 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:34.809192+0000 mgr.y (mgr.14146) 197 : audit [DBG] from='client.24346 v1:172.21.15.94:0/1028816140' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi033=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:36.594 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:34.810789+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi033=foo.a;count:1 2024-02-12T16:49:36.594 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:34.847147+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi033=foo.a;count:1 2024-02-12T16:49:36.594 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:34.859331+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Deploying daemon rgw.foo.a on smithi033 2024-02-12T16:49:36.594 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:34.809192+0000 mgr.y (mgr.14146) 197 : audit [DBG] from='client.24346 v1:172.21.15.94:0/1028816140' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi033=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:36.594 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:34.810789+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi033=foo.a;count:1 2024-02-12T16:49:36.594 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:34.847147+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi033=foo.a;count:1 2024-02-12T16:49:36.594 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:34.859331+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Deploying daemon rgw.foo.a on smithi033 2024-02-12T16:49:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:34.809192+0000 mgr.y (mgr.14146) 197 : audit [DBG] from='client.24346 v1:172.21.15.94:0/1028816140' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi033=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:34.810789+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi033=foo.a;count:1 2024-02-12T16:49:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:34.847147+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi033=foo.a;count:1 2024-02-12T16:49:36.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:34.859331+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Deploying daemon rgw.foo.a on smithi033 2024-02-12T16:49:37.777 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 16:49:37 smithi033 bash[65818]: 1c72ab01d792099412635889a4765724b277d6bba56ca67dc7d8d6329fa42c75 2024-02-12T16:49:37.778 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 16:49:37 smithi033 systemd[1]: Started Ceph rgw.foo.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:49:38.095 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'iscsi' created 2024-02-12T16:49:38.361 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:36.369882+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:36.369882+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:36.369882+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:38.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool application enable iscsi rbd 2024-02-12T16:49:39.357 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:37.593993+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.94:0/1635936941' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-12T16:49:39.357 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:37.648651+0000 mon.a (mon.0) 566 : audit [INF] from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-12T16:49:39.357 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:38.081894+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:39.357 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:38.085882+0000 mon.a (mon.0) 568 : audit [INF] from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-12T16:49:39.357 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:38.085960+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-12T16:49:39.358 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:38.086418+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:39.358 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:38.091384+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-12T16:49:39.358 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:38.091704+0000 mon.a (mon.0) 571 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-12T16:49:39.358 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:38.370242+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:37.593993+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.94:0/1635936941' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:37.648651+0000 mon.a (mon.0) 566 : audit [INF] from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:38.081894+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:38.085882+0000 mon.a (mon.0) 568 : audit [INF] from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:38.085960+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:38.086418+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:38.091384+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:38.091704+0000 mon.a (mon.0) 571 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-12T16:49:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:38.370242+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:37.593993+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.94:0/1635936941' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:37.648651+0000 mon.a (mon.0) 566 : audit [INF] from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:38.081894+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:38.085882+0000 mon.a (mon.0) 568 : audit [INF] from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:38.085960+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:38.086418+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:38.091384+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:38.091704+0000 mon.a (mon.0) 571 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-12T16:49:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:38.370242+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:40.287 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:39.093945+0000 mon.a (mon.0) 572 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-12T16:49:40.287 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:39.094051+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-12T16:49:40.287 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:39.593878+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:40.287 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:39.594605+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:40.287 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:39.595222+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:40.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:39.093945+0000 mon.a (mon.0) 572 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-12T16:49:40.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:39.094051+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-12T16:49:40.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:39.593878+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:40.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:39.594605+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:40.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:39.595222+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:39.093945+0000 mon.a (mon.0) 572 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-12T16:49:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:39.094051+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-12T16:49:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:39.593878+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:39.594605+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:39.595222+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:41.183 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rbd' on pool 'iscsi' 2024-02-12T16:49:41.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:39.599150+0000 mgr.y (mgr.14146) 203 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-12T16:49:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:40.165221+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-12T16:49:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:40.168934+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:40.169449+0000 mon.a (mon.0) 578 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:40.170331+0000 mon.a (mon.0) 579 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:40.370892+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v196: 68 pgs: 19 creating+peering, 2 active+clean, 47 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:40.646846+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.94:0/219576070' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-12T16:49:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:40.647596+0000 mon.a (mon.0) 580 : audit [INF] from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-12T16:49:41.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:39.599150+0000 mgr.y (mgr.14146) 203 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:40.165221+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:40.168934+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:40.169449+0000 mon.a (mon.0) 578 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:40.170331+0000 mon.a (mon.0) 579 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:40.370892+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v196: 68 pgs: 19 creating+peering, 2 active+clean, 47 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:40.646846+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.94:0/219576070' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:40.647596+0000 mon.a (mon.0) 580 : audit [INF] from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:39.599150+0000 mgr.y (mgr.14146) 203 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:40.165221+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-12T16:49:41.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:40.168934+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:40.169449+0000 mon.a (mon.0) 578 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:40.170331+0000 mon.a (mon.0) 579 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-12T16:49:41.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:40.370892+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v196: 68 pgs: 19 creating+peering, 2 active+clean, 47 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:49:41.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:40.646846+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.94:0/219576070' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-12T16:49:41.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:40.647596+0000 mon.a (mon.0) 580 : audit [INF] from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-12T16:49:41.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch apply iscsi iscsi user password --placement '1;smithi094=iscsi.a' 2024-02-12T16:49:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:41.170960+0000 mon.a (mon.0) 581 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-12T16:49:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:41.171038+0000 mon.a (mon.0) 582 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-12T16:49:42.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:41.171136+0000 mon.a (mon.0) 583 : audit [INF] from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-12T16:49:42.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:41.171207+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-12T16:49:42.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:42.060512+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:41.170960+0000 mon.a (mon.0) 581 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:41.171038+0000 mon.a (mon.0) 582 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:41.171136+0000 mon.a (mon.0) 583 : audit [INF] from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:41.171207+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:42.060512+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:41.170960+0000 mon.a (mon.0) 581 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:41.171038+0000 mon.a (mon.0) 582 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:41.171136+0000 mon.a (mon.0) 583 : audit [INF] from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-12T16:49:42.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:41.171207+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-12T16:49:42.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:42.060512+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:43.392 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:42.180002+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-12T16:49:43.392 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:42.182136+0000 mon.b (mon.1) 29 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.393 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:42.182263+0000 mon.a (mon.0) 587 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.393 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:42.182524+0000 mon.a (mon.0) 588 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.393 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:42.371515+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v199: 100 pgs: 19 creating+peering, 16 active+clean, 65 unknown; 450 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 463 B/s wr, 1 op/s 2024-02-12T16:49:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:42.180002+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:42.182136+0000 mon.b (mon.1) 29 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:42.182263+0000 mon.a (mon.0) 587 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:42.182524+0000 mon.a (mon.0) 588 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:42.371515+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v199: 100 pgs: 19 creating+peering, 16 active+clean, 65 unknown; 450 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 463 B/s wr, 1 op/s 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:42.180002+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:42.182136+0000 mon.b (mon.1) 29 : audit [INF] from='client.? v1:172.21.15.33:0/3785705817' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:42.182263+0000 mon.a (mon.0) 587 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:42.182524+0000 mon.a (mon.0) 588 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-12T16:49:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:42.371515+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v199: 100 pgs: 19 creating+peering, 16 active+clean, 65 unknown; 450 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 463 B/s wr, 1 op/s 2024-02-12T16:49:44.125 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled iscsi.iscsi update... 2024-02-12T16:49:44.445 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:43.188861+0000 mon.a (mon.0) 589 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-12T16:49:44.445 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:43.188968+0000 mon.a (mon.0) 590 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-12T16:49:44.445 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:43.189144+0000 mon.a (mon.0) 591 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-12T16:49:44.445 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:44.123398+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:44.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:43.188861+0000 mon.a (mon.0) 589 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-12T16:49:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:43.188968+0000 mon.a (mon.0) 590 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-12T16:49:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:43.189144+0000 mon.a (mon.0) 591 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-12T16:49:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:44.123398+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:43.188861+0000 mon.a (mon.0) 589 : audit [INF] from='client.? v1:172.21.15.33:0/534103199' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-12T16:49:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:43.188968+0000 mon.a (mon.0) 590 : audit [INF] from='client.24355 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-12T16:49:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:43.189144+0000 mon.a (mon.0) 591 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-12T16:49:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:44.123398+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:44.728 DEBUG:teuthology.orchestra.run.smithi094:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@iscsi.iscsi.a.service 2024-02-12T16:49:44.731 INFO:tasks.cephadm:Adding prometheus.a on smithi094 2024-02-12T16:49:44.731 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch apply prometheus '1;smithi094=a' 2024-02-12T16:49:44.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:49:45.457 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:43.818124+0000 mgr.y (mgr.14146) 206 : audit [DBG] from='client.24409 v1:172.21.15.94:0/3536968621' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi094=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:45.457 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:43.820063+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi094=iscsi.a;count:1 2024-02-12T16:49:45.457 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:44.192646+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-12T16:49:45.457 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:44.197561+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.33:0/1589842856' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.457 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:44.198606+0000 mon.b (mon.1) 31 : audit [INF] from='client.? v1:172.21.15.33:0/3604934113' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.457 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:44.198644+0000 mon.a (mon.0) 594 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.457 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:44.199512+0000 mon.a (mon.0) 595 : audit [INF] from='client.24403 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.457 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:44.371918+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v202: 132 pgs: 4 creating+peering, 76 active+clean, 52 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-12T16:49:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:43.818124+0000 mgr.y (mgr.14146) 206 : audit [DBG] from='client.24409 v1:172.21.15.94:0/3536968621' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi094=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:43.820063+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi094=iscsi.a;count:1 2024-02-12T16:49:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:44.192646+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-12T16:49:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:44.197561+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.33:0/1589842856' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:44.198606+0000 mon.b (mon.1) 31 : audit [INF] from='client.? v1:172.21.15.33:0/3604934113' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:44.198644+0000 mon.a (mon.0) 594 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:44.199512+0000 mon.a (mon.0) 595 : audit [INF] from='client.24403 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:44.371918+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v202: 132 pgs: 4 creating+peering, 76 active+clean, 52 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-12T16:49:45.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:43.818124+0000 mgr.y (mgr.14146) 206 : audit [DBG] from='client.24409 v1:172.21.15.94:0/3536968621' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi094=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:45.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:43.820063+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi094=iscsi.a;count:1 2024-02-12T16:49:45.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:44.192646+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-12T16:49:45.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:44.197561+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.33:0/1589842856' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:44.198606+0000 mon.b (mon.1) 31 : audit [INF] from='client.? v1:172.21.15.33:0/3604934113' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:44.198644+0000 mon.a (mon.0) 594 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:44.199512+0000 mon.a (mon.0) 595 : audit [INF] from='client.24403 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-12T16:49:45.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:44.371918+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v202: 132 pgs: 4 creating+peering, 76 active+clean, 52 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-12T16:49:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:45.206665+0000 mon.a (mon.0) 596 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-12T16:49:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:45.206791+0000 mon.a (mon.0) 597 : audit [INF] from='client.24403 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-12T16:49:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:45.206850+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-12T16:49:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:45.213816+0000 mon.b (mon.1) 32 : audit [INF] from='client.? v1:172.21.15.33:0/1589842856' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:45.214056+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.33:0/3604934113' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:45.214830+0000 mon.a (mon.0) 599 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:45.215316+0000 mon.a (mon.0) 600 : audit [INF] from='client.24403 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:45.666218+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:45.666267+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:45.206665+0000 mon.a (mon.0) 596 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:45.206791+0000 mon.a (mon.0) 597 : audit [INF] from='client.24403 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:45.206850+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:45.213816+0000 mon.b (mon.1) 32 : audit [INF] from='client.? v1:172.21.15.33:0/1589842856' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:45.214056+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.33:0/3604934113' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:45.214830+0000 mon.a (mon.0) 599 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:45.215316+0000 mon.a (mon.0) 600 : audit [INF] from='client.24403 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:45.666218+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:45.666267+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:45.206665+0000 mon.a (mon.0) 596 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-12T16:49:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:45.206791+0000 mon.a (mon.0) 597 : audit [INF] from='client.24403 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-12T16:49:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:45.206850+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-12T16:49:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:45.213816+0000 mon.b (mon.1) 32 : audit [INF] from='client.? v1:172.21.15.33:0/1589842856' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:45.214056+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.33:0/3604934113' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:45.214830+0000 mon.a (mon.0) 599 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:45.215316+0000 mon.a (mon.0) 600 : audit [INF] from='client.24403 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-12T16:49:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:45.666218+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:45.666267+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:46.972 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled prometheus update... 2024-02-12T16:49:47.407 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.207249+0000 mon.a (mon.0) 601 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-12T16:49:47.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.207374+0000 mon.a (mon.0) 602 : audit [INF] from='client.24403 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-12T16:49:47.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:46.207465+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-12T16:49:47.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.364434+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.369957+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:46.372559+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v205: 132 pgs: 9 creating+peering, 99 active+clean, 24 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-12T16:49:47.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.375014+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:46.381823+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-12T16:49:47.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.391981+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-12T16:49:47.409 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.395517+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-12T16:49:47.409 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.403685+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:47.409 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:46.404430+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi094 2024-02-12T16:49:47.409 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.971005+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.207249+0000 mon.a (mon.0) 601 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.207374+0000 mon.a (mon.0) 602 : audit [INF] from='client.24403 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:46.207465+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.364434+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.369957+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:46.372559+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v205: 132 pgs: 9 creating+peering, 99 active+clean, 24 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.375014+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:46.381823+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.391981+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.395517+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-12T16:49:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.403685+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:46.404430+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi094 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.971005+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.207249+0000 mon.a (mon.0) 601 : audit [INF] from='client.24397 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.207374+0000 mon.a (mon.0) 602 : audit [INF] from='client.24403 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:46.207465+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.364434+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.369957+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:46.372559+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v205: 132 pgs: 9 creating+peering, 99 active+clean, 24 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.375014+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:46.381823+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.391981+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-12T16:49:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.395517+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-12T16:49:47.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.403685+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:47.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:46.404430+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi094 2024-02-12T16:49:47.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.971005+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:48.053 DEBUG:teuthology.orchestra.run.smithi094:prometheus.a> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@prometheus.a.service 2024-02-12T16:49:48.055 INFO:tasks.cephadm:Adding node-exporter.a on smithi033 2024-02-12T16:49:48.055 INFO:tasks.cephadm:Adding node-exporter.b on smithi094 2024-02-12T16:49:48.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch apply node-exporter '2;smithi033=a;smithi094=b' 2024-02-12T16:49:48.084 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:49:48.638 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:46.963234+0000 mgr.y (mgr.14146) 212 : audit [DBG] from='client.24433 v1:172.21.15.94:0/1619158230' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi094=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:48.638 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:46.964882+0000 mgr.y (mgr.14146) 213 : cephadm [INF] Saving service prometheus spec with placement smithi094=a;count:1 2024-02-12T16:49:48.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:46.963234+0000 mgr.y (mgr.14146) 212 : audit [DBG] from='client.24433 v1:172.21.15.94:0/1619158230' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi094=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:48.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:46.964882+0000 mgr.y (mgr.14146) 213 : cephadm [INF] Saving service prometheus spec with placement smithi094=a;count:1 2024-02-12T16:49:48.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:46.963234+0000 mgr.y (mgr.14146) 212 : audit [DBG] from='client.24433 v1:172.21.15.94:0/1619158230' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi094=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:48.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:46.964882+0000 mgr.y (mgr.14146) 213 : cephadm [INF] Saving service prometheus spec with placement smithi094=a;count:1 2024-02-12T16:49:49.641 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:48.373167+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v206: 132 pgs: 5 creating+peering, 113 active+clean, 14 unknown; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-02-12T16:49:49.641 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:48.389537+0000 mon.a (mon.0) 611 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-12T16:49:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:48.373167+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v206: 132 pgs: 5 creating+peering, 113 active+clean, 14 unknown; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-02-12T16:49:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:48.389537+0000 mon.a (mon.0) 611 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-12T16:49:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:48.373167+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v206: 132 pgs: 5 creating+peering, 113 active+clean, 14 unknown; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-02-12T16:49:49.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:48.389537+0000 mon.a (mon.0) 611 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-12T16:49:50.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:50 smithi094 bash[62287]: log file path now is '/var/log/tcmu-runner.log' 2024-02-12T16:49:50.697 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled node-exporter update... 2024-02-12T16:49:51.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:51 smithi094 bash[62483]: f8813344a0d2ec9620ddaa526d2201b685f5d794adffcb1622d63ac12d07f887 2024-02-12T16:49:51.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:51 smithi094 systemd[1]: Started Ceph iscsi.iscsi.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:49:51.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug Started the configuration object watcher 2024-02-12T16:49:51.491 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug Checking for config object changes every 1s 2024-02-12T16:49:51.491 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug Processing osd blocklist entries for this node 2024-02-12T16:49:51.766 DEBUG:teuthology.orchestra.run.smithi033:node-exporter.a> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@node-exporter.a.service 2024-02-12T16:49:51.770 DEBUG:teuthology.orchestra.run.smithi094:node-exporter.b> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@node-exporter.b.service 2024-02-12T16:49:51.772 INFO:tasks.cephadm:Adding alertmanager.a on smithi033 2024-02-12T16:49:51.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch apply alertmanager '1;smithi033=a' 2024-02-12T16:49:51.802 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:49:51.808 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:49:51.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug Reading the configuration object to update local LIO configuration 2024-02-12T16:49:51.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug Configuration does not have an entry for this host(smithi094.front.sepia.ceph.com) - nothing to define to LIO 2024-02-12T16:49:51.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-12T16:49:51.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:50.373817+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 4.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 74 KiB/s rd, 5.3 KiB/s wr, 180 op/s 2024-02-12T16:49:51.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:50.687003+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24439 v1:172.21.15.94:0/2501031900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi033=a;smithi094=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:51.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:50.688478+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service node-exporter spec with placement smithi033=a;smithi094=b;count:2 2024-02-12T16:49:51.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:50.694549+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:51.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:51.204064+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:51.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:51.205826+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:51.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:51.619303+0000 mon.b (mon.1) 34 : audit [DBG] from='client.? v1:172.21.15.94:0/3047516767' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-12T16:49:52.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:50.373817+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 4.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 74 KiB/s rd, 5.3 KiB/s wr, 180 op/s 2024-02-12T16:49:52.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:50.687003+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24439 v1:172.21.15.94:0/2501031900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi033=a;smithi094=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:52.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:50.688478+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service node-exporter spec with placement smithi033=a;smithi094=b;count:2 2024-02-12T16:49:52.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:50.694549+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:52.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:51.204064+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:52.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:51.205826+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:52.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:51.619303+0000 mon.b (mon.1) 34 : audit [DBG] from='client.? v1:172.21.15.94:0/3047516767' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-12T16:49:52.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:50.373817+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 4.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 74 KiB/s rd, 5.3 KiB/s wr, 180 op/s 2024-02-12T16:49:52.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:50.687003+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24439 v1:172.21.15.94:0/2501031900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi033=a;smithi094=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:52.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:50.688478+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service node-exporter spec with placement smithi033=a;smithi094=b;count:2 2024-02-12T16:49:52.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:50.694549+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:52.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:51.204064+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:52.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:51.205826+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:49:52.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:51.619303+0000 mon.b (mon.1) 34 : audit [DBG] from='client.? v1:172.21.15.94:0/3047516767' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-12T16:49:53.314 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:52.053593+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:53.314 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:52.053693+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:53.315 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:52.070769+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:53.315 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:52.374457+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-12T16:49:53.315 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:52.698914+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-12T16:49:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:52.053593+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:52.053693+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:52.070769+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:52.374457+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-12T16:49:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:52.698914+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-12T16:49:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:52.053593+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:52.053693+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:52.070769+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:52.374457+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-12T16:49:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:52.698914+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-12T16:49:54.339 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled alertmanager update... 2024-02-12T16:49:55.352 DEBUG:teuthology.orchestra.run.smithi033:alertmanager.a> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@alertmanager.a.service 2024-02-12T16:49:55.355 INFO:tasks.cephadm:Adding grafana.a on smithi094 2024-02-12T16:49:55.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph orch apply grafana '1;smithi094=a' 2024-02-12T16:49:55.388 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T16:49:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:54.331706+0000 mgr.y (mgr.14146) 219 : audit [DBG] from='client.24469 v1:172.21.15.94:0/758144393' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:49:54.333141+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Saving service alertmanager spec with placement smithi033=a;count:1 2024-02-12T16:49:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:54.336906+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:54.375013+0000 mgr.y (mgr.14146) 221 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 138 op/s 2024-02-12T16:49:55.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:54.331706+0000 mgr.y (mgr.14146) 219 : audit [DBG] from='client.24469 v1:172.21.15.94:0/758144393' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:49:54.333141+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Saving service alertmanager spec with placement smithi033=a;count:1 2024-02-12T16:49:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:54.336906+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:54.375013+0000 mgr.y (mgr.14146) 221 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 138 op/s 2024-02-12T16:49:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:54.331706+0000 mgr.y (mgr.14146) 219 : audit [DBG] from='client.24469 v1:172.21.15.94:0/758144393' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:49:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:49:54.333141+0000 mgr.y (mgr.14146) 220 : cephadm [INF] Saving service alertmanager spec with placement smithi033=a;count:1 2024-02-12T16:49:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:54.336906+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:54.375013+0000 mgr.y (mgr.14146) 221 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 138 op/s 2024-02-12T16:49:56.765 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ignoring --setuser ceph since I am not root 2024-02-12T16:49:56.766 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ignoring --setgroup ceph since I am not root 2024-02-12T16:49:56.766 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:56.720+0000 7f3897e9b700 1 -- 172.21.15.94:0/2082423992 <== mon.1 v1:172.21.15.94:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 2846284852 0 0) 0x55ee93a8e9c0 con 0x55ee93a65400 2024-02-12T16:49:56.766 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:55.675201+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:56.766 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:55.676253+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:56.766 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:55.677022+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:56.766 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:55.681635+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:56.766 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:55.686828+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-12T16:49:56.766 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:55.944374+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-12T16:49:56.941 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ignoring --setuser ceph since I am not root 2024-02-12T16:49:56.941 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ignoring --setgroup ceph since I am not root 2024-02-12T16:49:56.942 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:56.854+0000 7f12782df680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T16:49:56.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:55.675201+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:56.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:55.676253+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:56.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:55.677022+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:56.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:55.681635+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:56.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:55.686828+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-12T16:49:56.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:55.944374+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-12T16:49:56.942 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:55.675201+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:56.943 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:55.676253+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:56.943 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:55.677022+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:56.943 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:55.681635+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' 2024-02-12T16:49:56.943 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:55.686828+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-12T16:49:56.943 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:55.944374+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-12T16:49:57.016 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:56.846+0000 7f38a8079680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T16:49:57.017 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:56.928+0000 7f38a8079680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T16:49:57.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:56.938+0000 7f12782df680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T16:49:57.684 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:57.320+0000 7f12782df680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T16:49:57.687 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:57.300+0000 7f38a8079680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T16:49:57.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:49:56.687572+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-12T16:49:57.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:49:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:49:56.687646+0000 mon.a (mon.0) 625 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-12T16:49:58.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:49:56.687572+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-12T16:49:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:49:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:49:56.687646+0000 mon.a (mon.0) 625 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-12T16:49:58.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:49:56.687572+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14146 v1:172.21.15.33:0/1765222688' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-12T16:49:58.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:49:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:49:56.687646+0000 mon.a (mon.0) 625 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-12T16:49:58.347 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:58.050+0000 7f38a8079680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T16:49:58.347 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:58.134+0000 7f38a8079680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T16:49:58.416 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:58.110+0000 7f12782df680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T16:49:58.416 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:58.193+0000 7f12782df680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T16:49:58.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:58.344+0000 7f38a8079680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T16:49:58.777 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:58.413+0000 7f12782df680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T16:49:59.564 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:59.270+0000 7f38a8079680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T16:49:59.565 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:59.396+0000 7f38a8079680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:49:59.565 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:59.473+0000 7f38a8079680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T16:49:59.707 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:59.395+0000 7f12782df680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T16:49:59.707 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:59.530+0000 7f12782df680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:49:59.707 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:59.612+0000 7f12782df680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T16:49:59.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:59.562+0000 7f38a8079680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T16:49:59.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:49:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:49:59.754+0000 7f38a8079680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T16:50:00.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:59.705+0000 7f12782df680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T16:50:00.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:49:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:49:59.908+0000 7f12782df680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T16:50:00.307 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:00.203+0000 7f12782df680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T16:50:00.307 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:00.000128+0000 mon.a (mon.0) 626 : cluster [INF] overall HEALTH_OK 2024-02-12T16:50:00.308 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:00.000128+0000 mon.a (mon.0) 626 : cluster [INF] overall HEALTH_OK 2024-02-12T16:50:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:00.000128+0000 mon.a (mon.0) 626 : cluster [INF] overall HEALTH_OK 2024-02-12T16:50:00.490 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:00.030+0000 7f38a8079680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T16:50:00.491 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:00.128+0000 7f38a8079680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T16:50:00.777 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:00.306+0000 7f12782df680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T16:50:01.188 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:00.839+0000 7f38a8079680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T16:50:01.188 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:00.920+0000 7f38a8079680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T16:50:01.188 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:01.012+0000 7f38a8079680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T16:50:01.431 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:01.064+0000 7f12782df680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T16:50:01.431 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:01.150+0000 7f12782df680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T16:50:01.431 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:01.247+0000 7f12782df680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T16:50:01.442 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:01.184+0000 7f38a8079680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T16:50:01.703 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:01.439+0000 7f38a8079680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T16:50:01.703 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:01.569+0000 7f38a8079680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T16:50:01.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:00.944222+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-12T16:50:01.956 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:01.700+0000 7f38a8079680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:50:01.956 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:01.871+0000 7f38a8079680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T16:50:01.977 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:00.944222+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-12T16:50:01.978 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:00.944222+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-12T16:50:01.978 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:01.429+0000 7f12782df680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T16:50:01.978 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:01.697+0000 7f12782df680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T16:50:01.978 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:01.834+0000 7f12782df680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T16:50:02.240 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: debug 2024-02-12T16:50:01.953+0000 7f38a8079680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T16:50:02.240 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: [12/Feb/2024:16:50:01] ENGINE Bus STARTING 2024-02-12T16:50:02.241 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: CherryPy Checker: 2024-02-12T16:50:02.241 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: The Application mounted at '' has an empty config. 2024-02-12T16:50:02.241 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: 2024-02-12T16:50:02.241 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: [12/Feb/2024:16:50:02] ENGINE Serving on http://:::9283 2024-02-12T16:50:02.241 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: [12/Feb/2024:16:50:02] ENGINE Bus STARTED 2024-02-12T16:50:02.245 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:01.976+0000 7f12782df680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T16:50:02.246 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:02.156+0000 7f12782df680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T16:50:02.498 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T16:50:02.244+0000 7f12782df680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T16:50:02.498 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [79B blob data] 2024-02-12T16:50:02.498 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:02] ENGINE Bus STARTING 2024-02-12T16:50:02.498 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [79B blob data] 2024-02-12T16:50:02.777 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: CherryPy Checker: 2024-02-12T16:50:02.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: The Application mounted at '' has an empty config. 2024-02-12T16:50:02.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: 2024-02-12T16:50:02.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:02] ENGINE Serving on http://:::9283 2024-02-12T16:50:02.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:02] ENGINE Bus STARTED 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:01.968444+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon x restarted 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:01.968543+0000 mon.a (mon.0) 629 : cluster [DBG] Standby manager daemon x started 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:01.969933+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:01.970396+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:01.972220+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:01.973114+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:02.257917+0000 mon.a (mon.0) 630 : cluster [INF] Active manager daemon y restarted 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:02.258287+0000 mon.a (mon.0) 631 : cluster [INF] Activating manager daemon y 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:02.263810+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-12T16:50:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:02.264206+0000 mon.a (mon.0) 633 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00601364s), standbys: x 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.268435+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.268624+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.268820+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.269560+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.269764+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.270021+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.270308+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.270588+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.270886+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.271110+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.271266+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.271467+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:50:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.271620+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.271814+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.271899+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.272267+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:02.287372+0000 mon.a (mon.0) 634 : cluster [INF] Manager daemon y is now available 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.390304+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.391587+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.427275+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.438732+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.439034+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.459879+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:50:03.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:02.460367+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:50:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:01.968444+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon x restarted 2024-02-12T16:50:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:01.968543+0000 mon.a (mon.0) 629 : cluster [DBG] Standby manager daemon x started 2024-02-12T16:50:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:01.969933+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:01.970396+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:01.972220+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:01.973114+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:02.257917+0000 mon.a (mon.0) 630 : cluster [INF] Active manager daemon y restarted 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:02.258287+0000 mon.a (mon.0) 631 : cluster [INF] Activating manager daemon y 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:02.263810+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:02.264206+0000 mon.a (mon.0) 633 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00601364s), standbys: x 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.268435+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.268624+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.268820+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.269560+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.269764+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T16:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.270021+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:01.968444+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon x restarted 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:01.968543+0000 mon.a (mon.0) 629 : cluster [DBG] Standby manager daemon x started 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:01.969933+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:01.970396+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:01.972220+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:01.973114+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.? v1:172.21.15.94:0/1907385023' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:02.257917+0000 mon.a (mon.0) 630 : cluster [INF] Active manager daemon y restarted 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:02.258287+0000 mon.a (mon.0) 631 : cluster [INF] Activating manager daemon y 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:02.263810+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:02.264206+0000 mon.a (mon.0) 633 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00601364s), standbys: x 2024-02-12T16:50:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.268435+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:50:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.268624+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:50:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.268820+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:50:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.269560+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T16:50:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.269764+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T16:50:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.270021+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.270308+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.270588+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.270886+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.271110+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.271266+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.271467+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.271620+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.271814+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.271899+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.272267+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:02.287372+0000 mon.a (mon.0) 634 : cluster [INF] Manager daemon y is now available 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.390304+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:03.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.391587+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.427275+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.438732+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.439034+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.459879+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:02.460367+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.270308+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.270588+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.270886+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:50:03.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.271110+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.271266+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.271467+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.271620+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.271814+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.271899+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.272267+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:02.287372+0000 mon.a (mon.0) 634 : cluster [INF] Manager daemon y is now available 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.390304+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.391587+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.427275+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.438732+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.439034+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.459879+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:50:03.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:02.460367+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T16:50:03.285 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled grafana update... 2024-02-12T16:50:03.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:50:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:50:04.320 DEBUG:teuthology.orchestra.run.smithi094:grafana.a> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@grafana.a.service 2024-02-12T16:50:04.322 INFO:tasks.cephadm:Setting up client nodes... 2024-02-12T16:50:04.323 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-12T16:50:04.353 INFO:journalctl@ceph.grafana.a.smithi094.stdout:-- Logs begin at Mon 2024-02-12 16:32:36 UTC. -- 2024-02-12T16:50:04.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:03.268246+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e18: y(active, since 1.01005s), standbys: x 2024-02-12T16:50:04.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:03.276032+0000 mgr.y (mgr.24481) 4 : cephadm [INF] Saving service grafana spec with placement smithi094=a;count:1 2024-02-12T16:50:04.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:03.279395+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:04.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.004776+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:04.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:03.268246+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e18: y(active, since 1.01005s), standbys: x 2024-02-12T16:50:04.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:03.276032+0000 mgr.y (mgr.24481) 4 : cephadm [INF] Saving service grafana spec with placement smithi094=a;count:1 2024-02-12T16:50:04.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:03.279395+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:04.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.004776+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:03.268246+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e18: y(active, since 1.01005s), standbys: x 2024-02-12T16:50:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:03.276032+0000 mgr.y (mgr.24481) 4 : cephadm [INF] Saving service grafana spec with placement smithi094=a;count:1 2024-02-12T16:50:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:03.279395+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.004776+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:04.270878+0000 mgr.y (mgr.24481) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:50:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.289897+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.290303+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.291085+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.291374+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.292017+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.292301+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.292928+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.293207+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:04.293729+0000 mgr.y (mgr.24481) 6 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 3769M 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:04.298014+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.235190+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:04.270878+0000 mgr.y (mgr.24481) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.289897+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.290303+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.291085+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.291374+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.292017+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.292301+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.292928+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.293207+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:04.293729+0000 mgr.y (mgr.24481) 6 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 3769M 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:04.298014+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:05.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.235190+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:04.270878+0000 mgr.y (mgr.24481) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:50:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.289897+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.290303+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.291085+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.291374+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.292017+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.292301+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.292928+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.293207+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:04.293729+0000 mgr.y (mgr.24481) 6 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 3769M 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:04.298014+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.235190+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:05.279084+0000 mon.a (mon.0) 647 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.518823+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.519207+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.520040+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.520296+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.521029+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.521283+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.522012+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.522264+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:05.522868+0000 mgr.y (mgr.24481) 7 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 4024M 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.529395+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.530549+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.531143+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:05.532244+0000 mgr.y (mgr.24481) 8 : cephadm [INF] Updating smithi033:/etc/ceph/ceph.conf 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:05.532505+0000 mgr.y (mgr.24481) 9 : cephadm [INF] Updating smithi094:/etc/ceph/ceph.conf 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:05.548613+0000 mgr.y (mgr.24481) 10 : cephadm [INF] Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:05.279084+0000 mon.a (mon.0) 647 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.518823+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.519207+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.520040+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.520296+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.521029+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.521283+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.522012+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.522264+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:05.522868+0000 mgr.y (mgr.24481) 7 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 4024M 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.529395+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.530549+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.531143+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:05.532244+0000 mgr.y (mgr.24481) 8 : cephadm [INF] Updating smithi033:/etc/ceph/ceph.conf 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:05.532505+0000 mgr.y (mgr.24481) 9 : cephadm [INF] Updating smithi094:/etc/ceph/ceph.conf 2024-02-12T16:50:06.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:05.548613+0000 mgr.y (mgr.24481) 10 : cephadm [INF] Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:05.577237+0000 mgr.y (mgr.24481) 11 : cephadm [INF] Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.581268+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.633324+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.640033+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:05.650958+0000 mgr.y (mgr.24481) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi033 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:05.946003+0000 mon.a (mon.0) 656 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.947400+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.949625+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.950354+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:05.950612+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:50:06.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:05.577237+0000 mgr.y (mgr.24481) 11 : cephadm [INF] Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.581268+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.633324+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.640033+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:05.650958+0000 mgr.y (mgr.24481) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi033 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:05.946003+0000 mon.a (mon.0) 656 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.947400+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.949625+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.950354+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:50:06.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:05.950612+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:50:06.650 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2024-02-12T16:50:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQA+TMplrkBIJhAAQoNkzCwcovD/5MQjH6Q+Kg== 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:05.279084+0000 mon.a (mon.0) 647 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.518823+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.519207+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.520040+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.520296+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.521029+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.521283+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.522012+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.522264+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:05.522868+0000 mgr.y (mgr.24481) 7 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 4024M 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.529395+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.530549+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.531143+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:05.532244+0000 mgr.y (mgr.24481) 8 : cephadm [INF] Updating smithi033:/etc/ceph/ceph.conf 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:05.532505+0000 mgr.y (mgr.24481) 9 : cephadm [INF] Updating smithi094:/etc/ceph/ceph.conf 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:05.548613+0000 mgr.y (mgr.24481) 10 : cephadm [INF] Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:05.577237+0000 mgr.y (mgr.24481) 11 : cephadm [INF] Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.581268+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.633324+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.640033+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:05.650958+0000 mgr.y (mgr.24481) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi033 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:05.946003+0000 mon.a (mon.0) 656 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.947400+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.949625+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.950354+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:50:06.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:05.950612+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T16:50:07.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:06.271441+0000 mgr.y (mgr.24481) 13 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:50:07.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:06.642119+0000 mon.a (mon.0) 659 : audit [INF] from='client.? v1:172.21.15.33:0/3782763975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:07.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:06.648755+0000 mon.a (mon.0) 660 : audit [INF] from='client.? v1:172.21.15.33:0/3782763975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T16:50:07.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:06.958216+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T16:50:07.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:06.958295+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-12T16:50:07.722 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:06.271441+0000 mgr.y (mgr.24481) 13 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:50:07.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:06.642119+0000 mon.a (mon.0) 659 : audit [INF] from='client.? v1:172.21.15.33:0/3782763975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:07.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:06.648755+0000 mon.a (mon.0) 660 : audit [INF] from='client.? v1:172.21.15.33:0/3782763975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T16:50:07.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:06.958216+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T16:50:07.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:06.958295+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-12T16:50:07.741 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-02-12T16:50:07.741 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-12T16:50:07.741 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-12T16:50:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:06.271441+0000 mgr.y (mgr.24481) 13 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:50:07.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:06.642119+0000 mon.a (mon.0) 659 : audit [INF] from='client.? v1:172.21.15.33:0/3782763975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:07.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:06.648755+0000 mon.a (mon.0) 660 : audit [INF] from='client.? v1:172.21.15.33:0/3782763975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T16:50:07.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:06.958216+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T16:50:07.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:06.958295+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-12T16:50:07.784 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-12T16:50:09.400 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:08.271936+0000 mgr.y (mgr.24481) 14 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-12T16:50:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:08.271936+0000 mgr.y (mgr.24481) 14 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-12T16:50:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:08.271936+0000 mgr.y (mgr.24481) 14 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-12T16:50:09.844 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2024-02-12T16:50:09.844 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBBTMpljUILMhAAKvA3dRZFLFsyTK5SH5A7/A== 2024-02-12T16:50:10.459 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:09.838797+0000 mon.b (mon.1) 39 : audit [INF] from='client.? v1:172.21.15.94:0/4154156011' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:10.459 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:09.839393+0000 mon.a (mon.0) 663 : audit [INF] from='client.24511 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:10.459 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:09.842717+0000 mon.a (mon.0) 664 : audit [INF] from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T16:50:10.479 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-02-12T16:50:10.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-12T16:50:10.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-12T16:50:10.521 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T16:50:10.553 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:09.838797+0000 mon.b (mon.1) 39 : audit [INF] from='client.? v1:172.21.15.94:0/4154156011' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:10.553 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:09.839393+0000 mon.a (mon.0) 663 : audit [INF] from='client.24511 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:10.553 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:09.842717+0000 mon.a (mon.0) 664 : audit [INF] from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T16:50:10.553 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:09.838797+0000 mon.b (mon.1) 39 : audit [INF] from='client.? v1:172.21.15.94:0/4154156011' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:10.553 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:09.839393+0000 mon.a (mon.0) 663 : audit [INF] from='client.24511 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T16:50:10.553 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:09.842717+0000 mon.a (mon.0) 664 : audit [INF] from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T16:50:11.236 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:11.561 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:10.272632+0000 mgr.y (mgr.24481) 15 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-12T16:50:11.562 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:10.947549+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-12T16:50:11.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:10.272632+0000 mgr.y (mgr.24481) 15 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-12T16:50:11.562 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:10.947549+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-12T16:50:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:10.272632+0000 mgr.y (mgr.24481) 15 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-12T16:50:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:10.947549+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-12T16:50:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:12.272959+0000 mgr.y (mgr.24481) 16 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-02-12T16:50:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:12.272959+0000 mgr.y (mgr.24481) 16 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-02-12T16:50:13.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:50:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:50:13.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:12.272959+0000 mgr.y (mgr.24481) 16 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-02-12T16:50:14.479 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:14 smithi033 systemd[1]: Starting Ceph alertmanager.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:50:14.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:13.283055+0000 mgr.y (mgr.24481) 17 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:14.480 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:13.283055+0000 mgr.y (mgr.24481) 17 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:14.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:13.283055+0000 mgr.y (mgr.24481) 17 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:14.934 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:50:14.934 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":72,"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","created":"2024-02-12T16:43:50.282468+0000","modified":"2024-02-12T16:50:10.943492+0000","last_up_change":"2024-02-12T16:49:30.933575+0000","last_in_change":"2024-02-12T16:49:14.105093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T16:47:33.056945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-12T16:49:37.647343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-12T16:49:38.076200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-12T16:49:39.589565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-12T16:49:41.623856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-12T16:49:43.693806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"5dd172eb-ee9d-4700-9b80-60f91c5f2cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6801","nonce":2545411201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6802","nonce":2545411201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6804","nonce":2545411201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6803","nonce":2545411201}]},"public_addr":"172.21.15.33:6801/2545411201","cluster_addr":"172.21.15.33:6802/2545411201","heartbeat_back_addr":"172.21.15.33:6804/2545411201","heartbeat_front_addr":"172.21.15.33:6803/2545411201","state":["exists","up"]},{"osd":1,"uuid":"71231bc4-1716-431d-bd88-f6a3cad0bd24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6805","nonce":1723231708}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6806","nonce":1723231708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6808","nonce":1723231708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6807","nonce":1723231708}]},"public_addr":"172.21.15.33:6805/1723231708","cluster_addr":"172.21.15.33:6806/1723231708","heartbeat_back_addr":"172.21.15.33:6808/1723231708","heartbeat_front_addr":"172.21.15.33:6807/1723231708","state":["exists","up"]},{"osd":2,"uuid":"8a9c7127-d17c-4c44-b666-4211006de47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6809","nonce":3126580659}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6810","nonce":3126580659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6812","nonce":3126580659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6811","nonce":3126580659}]},"public_addr":"172.21.15.33:6809/3126580659","cluster_addr":"172.21.15.33:6810/3126580659","heartbeat_back_addr":"172.21.15.33:6812/3126580659","heartbeat_front_addr":"172.21.15.33:6811/3126580659","state":["exists","up"]},{"osd":3,"uuid":"3cc40671-c8f7-412c-a0f1-987f48a016b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6813","nonce":3941314957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6814","nonce":3941314957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6816","nonce":3941314957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6815","nonce":3941314957}]},"public_addr":"172.21.15.33:6813/3941314957","cluster_addr":"172.21.15.33:6814/3941314957","heartbeat_back_addr":"172.21.15.33:6816/3941314957","heartbeat_front_addr":"172.21.15.33:6815/3941314957","state":["exists","up"]},{"osd":4,"uuid":"0498129c-861c-4f7e-b8ba-219949da3130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":544126779}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":544126779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":544126779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":544126779}]},"public_addr":"172.21.15.94:6800/544126779","cluster_addr":"172.21.15.94:6801/544126779","heartbeat_back_addr":"172.21.15.94:6803/544126779","heartbeat_front_addr":"172.21.15.94:6802/544126779","state":["exists","up"]},{"osd":5,"uuid":"05718a1e-1164-4b54-a493-23c1cd3adb50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":3229884005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3229884005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3229884005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":3229884005}]},"public_addr":"172.21.15.94:6804/3229884005","cluster_addr":"172.21.15.94:6805/3229884005","heartbeat_back_addr":"172.21.15.94:6807/3229884005","heartbeat_front_addr":"172.21.15.94:6806/3229884005","state":["exists","up"]},{"osd":6,"uuid":"dc48640f-8f27-405c-a9a4-cf1e3e6c8821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":507305558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":507305558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":507305558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":507305558}]},"public_addr":"172.21.15.94:6808/507305558","cluster_addr":"172.21.15.94:6809/507305558","heartbeat_back_addr":"172.21.15.94:6811/507305558","heartbeat_front_addr":"172.21.15.94:6810/507305558","state":["exists","up"]},{"osd":7,"uuid":"7d55aa3f-06b6-4fc3-97fd-62b535f78b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3401977873}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":3401977873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":3401977873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":3401977873}]},"public_addr":"172.21.15.94:6812/3401977873","cluster_addr":"172.21.15.94:6813/3401977873","heartbeat_back_addr":"172.21.15.94:6815/3401977873","heartbeat_front_addr":"172.21.15.94:6814/3401977873","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:13.998181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:12.391116+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:48:54.168730+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:22.665200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:45.666269+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:52.053698+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:05.554325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:29.328871+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/2829464018":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/1835128904":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2277321750":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2116602769":"2024-02-13T16:50:02.257936+0000","172.21.15.33:6800/3967118778":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/2298361247":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/218227980":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/2070582473":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/4291575956":"2024-02-13T16:44:46.313346+0000","172.21.15.33:6800/3157562110":"2024-02-13T16:44:22.727241+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T16:50:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:14.273737+0000 mgr.y (mgr.24481) 18 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:14.930880+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? v1:172.21.15.33:0/3160857954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:15.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:14.273737+0000 mgr.y (mgr.24481) 18 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:15.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:14.930880+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? v1:172.21.15.33:0/3160857954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:15.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:14.273737+0000 mgr.y (mgr.24481) 18 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:15.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:14.930880+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? v1:172.21.15.33:0/3160857954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:15.842 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T16:47:33.056945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-12T16:49:37.647343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-12T16:49:38.076200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-12T16:49:39.589565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-12T16:49:41.623856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-12T16:49:43.693806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}] 2024-02-12T16:50:15.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-02-12T16:50:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:16.277 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:15.976Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-12T16:50:16.277 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:15.976Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-12T16:50:16.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:15.978Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.33 port=9094 2024-02-12T16:50:16.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:15.979Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-12T16:50:16.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:16.005Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-12T16:50:16.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:16.006Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-12T16:50:16.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:16.008Z caller=main.go:518 msg=Listening address=:9093 2024-02-12T16:50:16.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:16.008Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-12T16:50:16.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:16 smithi033 bash[68273]: 05a3a65e495a63d4ec65c45a3f6d9a3e5a13108c35562f364be02705a077960e 2024-02-12T16:50:16.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:16 smithi033 systemd[1]: Started Ceph alertmanager.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:50:17.028 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:16 smithi033 systemd[1]: Starting Ceph node-exporter.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:50:17.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:16.126956+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:17.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:16.130412+0000 mgr.y (mgr.24481) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi033 2024-02-12T16:50:17.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:16.274106+0000 mgr.y (mgr.24481) 20 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:50:17.467 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:16.126956+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:17.467 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:16.130412+0000 mgr.y (mgr.24481) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi033 2024-02-12T16:50:17.468 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:16.274106+0000 mgr.y (mgr.24481) 20 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:50:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:16.126956+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:16.130412+0000 mgr.y (mgr.24481) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi033 2024-02-12T16:50:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:16.274106+0000 mgr.y (mgr.24481) 20 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:50:17.982 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:17 smithi033 bash[68696]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-12T16:50:18.148 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-02-12T16:50:18.243 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:18 smithi033 bash[68696]: Getting image source signatures 2024-02-12T16:50:18.243 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:17.980Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000138173s 2024-02-12T16:50:18.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:17.432382+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:18.147385+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.33:0/2541229438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-12T16:50:18.528 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:18 smithi033 bash[68696]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-12T16:50:18.528 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:18 smithi033 bash[68696]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-12T16:50:18.528 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:18 smithi033 bash[68696]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-12T16:50:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:17.432382+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:18.147385+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.33:0/2541229438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-12T16:50:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:17.432382+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:18.147385+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.33:0/2541229438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-12T16:50:19.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool get .rgw.root pg_num 2024-02-12T16:50:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:18.274768+0000 mgr.y (mgr.24481) 21 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:50:19.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:18.274768+0000 mgr.y (mgr.24481) 21 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:50:19.778 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:19 smithi033 bash[68696]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-12T16:50:19.778 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:19 smithi033 bash[68696]: Writing manifest to image destination 2024-02-12T16:50:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:18.274768+0000 mgr.y (mgr.24481) 21 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:50:20.293 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:18.760345+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:18.760441+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:19.439965+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-12T16:50:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:20.354817+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:20.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:18.760345+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:20.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:18.760441+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:20.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:19.439965+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-12T16:50:20.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:20.354817+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:20.529 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.212Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-12T16:50:20.529 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.212Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-12T16:50:20.529 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=arp 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=edac 2024-02-12T16:50:20.530 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-12T16:50:20.531 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=os 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=stat 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.213Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-12T16:50:20.532 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:115 level=info collector=time 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:115 level=info collector=uname 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a[68838]: ts=2024-02-12T16:50:20.214Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 bash[68696]: 2afe51dacdec16aee880b56c9f15584d81901a2ddf073be4352c326999011629 2024-02-12T16:50:20.533 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 16:50:20 smithi033 systemd[1]: Started Ceph node-exporter.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:50:20.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:18.760345+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:20.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:18.760441+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:20.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:19.439965+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-12T16:50:20.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:20.354817+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:21.052 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:20 smithi094 systemd[1]: Starting Ceph node-exporter.b for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:50:21.449 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:21 smithi094 bash[64142]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-12T16:50:21.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:20.275134+0000 mgr.y (mgr.24481) 22 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:50:21.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:20.356070+0000 mgr.y (mgr.24481) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi094 2024-02-12T16:50:21.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:20.949621+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-12T16:50:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:20.275134+0000 mgr.y (mgr.24481) 22 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:50:21.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:20.356070+0000 mgr.y (mgr.24481) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi094 2024-02-12T16:50:21.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:20.949621+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-12T16:50:21.740 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:21 smithi094 bash[64142]: Getting image source signatures 2024-02-12T16:50:21.740 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:21 smithi094 bash[64142]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-12T16:50:21.740 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:21 smithi094 bash[64142]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-12T16:50:21.740 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:21 smithi094 bash[64142]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-12T16:50:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:20.275134+0000 mgr.y (mgr.24481) 22 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:50:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:20.356070+0000 mgr.y (mgr.24481) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi094 2024-02-12T16:50:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:20.949621+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-12T16:50:22.280 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-02-12T16:50:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:22.278891+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.33:0/1487096562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-12T16:50:22.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:22.278891+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.33:0/1487096562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-12T16:50:22.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:22.278891+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.33:0/1487096562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-12T16:50:23.139 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool get iscsi pg_num 2024-02-12T16:50:23.425 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:23.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:50:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:50:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:22.275474+0000 mgr.y (mgr.24481) 24 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:50:23.741 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:23 smithi094 bash[64142]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-12T16:50:23.741 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:23 smithi094 bash[64142]: Writing manifest to image destination 2024-02-12T16:50:23.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:22.275474+0000 mgr.y (mgr.24481) 24 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:50:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:22.275474+0000 mgr.y (mgr.24481) 24 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:50:24.461 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.098Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.098Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.098Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.098Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=arp 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=edac 2024-02-12T16:50:24.462 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-12T16:50:24.463 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=os 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=stat 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=time 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-12T16:50:24.464 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=uname 2024-02-12T16:50:24.465 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-12T16:50:24.465 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-12T16:50:24.465 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-12T16:50:24.465 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-12T16:50:24.465 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b[64189]: ts=2024-02-12T16:50:24.099Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-12T16:50:24.465 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 bash[64142]: c1620898fb23b6614fd23ab20339c7ab860443165c1179c6351704b2d832cf6c 2024-02-12T16:50:24.465 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 16:50:24 smithi094 systemd[1]: Started Ceph node-exporter.b for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:50:24.595 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:23.290167+0000 mgr.y (mgr.24481) 25 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:24.596 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:24.259512+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:24.596 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:23.290167+0000 mgr.y (mgr.24481) 25 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:24.596 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:24.259512+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:24.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:23.290167+0000 mgr.y (mgr.24481) 25 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:24.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:24.259512+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:25.322 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 3 2024-02-12T16:50:25.739 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:24.270837+0000 mgr.y (mgr.24481) 26 : cephadm [INF] Deploying daemon prometheus.a on smithi094 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:24.275992+0000 mgr.y (mgr.24481) 27 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:24.394720+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:24.394772+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:25.031881+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:25.034836+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:25.268912+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:25.321202+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? v1:172.21.15.33:0/1845634687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:24.270837+0000 mgr.y (mgr.24481) 26 : cephadm [INF] Deploying daemon prometheus.a on smithi094 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:24.275992+0000 mgr.y (mgr.24481) 27 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:25.740 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:24.394720+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:25.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:24.394772+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:25.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:25.031881+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:50:25.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:25.034836+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:50:25.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:25.268912+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-12T16:50:25.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:25.321202+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? v1:172.21.15.33:0/1845634687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-12T16:50:25.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:24.270837+0000 mgr.y (mgr.24481) 26 : cephadm [INF] Deploying daemon prometheus.a on smithi094 2024-02-12T16:50:25.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:24.275992+0000 mgr.y (mgr.24481) 27 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:25.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:24.394720+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:25.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:24.394772+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:25.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:25.031881+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:50:25.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:25.034836+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:50:25.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:25.268912+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-12T16:50:25.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:25.321202+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? v1:172.21.15.33:0/1845634687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-12T16:50:26.028 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:50:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:50:25.982Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002834996s 2024-02-12T16:50:26.093 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool get default.rgw.log pg_num 2024-02-12T16:50:26.373 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:26.953 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:25.951719+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-12T16:50:26.953 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:26.276320+0000 mgr.y (mgr.24481) 28 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:26.953 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:25.951719+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-12T16:50:26.954 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:26.276320+0000 mgr.y (mgr.24481) 28 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:25.951719+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-12T16:50:27.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:26.276320+0000 mgr.y (mgr.24481) 28 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:28.383 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-02-12T16:50:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:27.440162+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:28.381178+0000 mon.b (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.33:0/1370293548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-12T16:50:28.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:27.440162+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:28.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:28.381178+0000 mon.b (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.33:0/1370293548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-12T16:50:28.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:27.440162+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:28.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:28.381178+0000 mon.b (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.33:0/1370293548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-12T16:50:29.141 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool get default.rgw.control pg_num 2024-02-12T16:50:29.421 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:29.446 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:28.276893+0000 mgr.y (mgr.24481) 29 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-12T16:50:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:28.276893+0000 mgr.y (mgr.24481) 29 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-12T16:50:29.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:28.276893+0000 mgr.y (mgr.24481) 29 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-12T16:50:31.321 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-02-12T16:50:31.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:30.277475+0000 mgr.y (mgr.24481) 30 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:31.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:30.758205+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:31.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:30.758321+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:31.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:31.319845+0000 mon.a (mon.0) 677 : audit [DBG] from='client.? v1:172.21.15.33:0/3701476398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-12T16:50:31.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:30.277475+0000 mgr.y (mgr.24481) 30 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:31.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:30.758205+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:31.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:30.758321+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:31.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:31.319845+0000 mon.a (mon.0) 677 : audit [DBG] from='client.? v1:172.21.15.33:0/3701476398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-12T16:50:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:30.277475+0000 mgr.y (mgr.24481) 30 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:30.758205+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:30.758321+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:31.319845+0000 mon.a (mon.0) 677 : audit [DBG] from='client.? v1:172.21.15.33:0/3701476398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-12T16:50:32.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool get default.rgw.meta pg_num 2024-02-12T16:50:32.279 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:31.449292+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-12T16:50:32.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:31.449292+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-12T16:50:32.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:31.449292+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-12T16:50:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:32.277796+0000 mgr.y (mgr.24481) 31 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-02-12T16:50:33.491 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:50:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:50:33.724 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:32.277796+0000 mgr.y (mgr.24481) 31 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-02-12T16:50:33.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:32.277796+0000 mgr.y (mgr.24481) 31 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-02-12T16:50:33.939 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:33 smithi094 systemd[1]: Starting Ceph prometheus.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:50:34.407 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-02-12T16:50:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:33.298005+0000 mgr.y (mgr.24481) 32 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:34.237848+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:34.237897+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:34.405997+0000 mon.a (mon.0) 679 : audit [DBG] from='client.? v1:172.21.15.33:0/2598290356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-12T16:50:34.741 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.504Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-12T16:50:34.741 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.504Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-12T16:50:34.741 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.504Z caller=main.go:518 level=info host_details="(Linux 4.18.0-536.el8.x86_64 #1 SMP Thu Jan 18 15:15:29 UTC 2024 x86_64 smithi094 (none))" 2024-02-12T16:50:34.741 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.504Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-12T16:50:34.741 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.504Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-12T16:50:34.741 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.505Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-12T16:50:34.741 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.506Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-12T16:50:34.741 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.506Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.509Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.509Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.59µs 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.509Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.509Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.509Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.397µs wal_replay_duration=159.505µs total_replay_duration=193.506µs 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.509Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.509Z caller=main.go:947 level=info msg="TSDB started" 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.510Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.524Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.832018ms db_storage=581ns remote_storage=1.118µs web_handler=350ns query_engine=517ns scrape=446.783µs scrape_sd=20.63µs notify=17.559µs notify_sd=5.309µs rules=14.118613ms 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:50:34.524Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 bash[64654]: 448efb68b398ac78b34a1d2cf9ac9b701ee0887948170e56292a8ff8685b6001 2024-02-12T16:50:34.742 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:50:34 smithi094 systemd[1]: Started Ceph prometheus.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:50:34.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:33.298005+0000 mgr.y (mgr.24481) 32 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:34.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:34.237848+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:34.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:34.237897+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:34.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:34.405997+0000 mon.a (mon.0) 679 : audit [DBG] from='client.? v1:172.21.15.33:0/2598290356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-12T16:50:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:33.298005+0000 mgr.y (mgr.24481) 32 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:34.237848+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:34.237897+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:34.405997+0000 mon.a (mon.0) 679 : audit [DBG] from='client.? v1:172.21.15.33:0/2598290356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-12T16:50:35.207 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:34] ENGINE Bus STOPPING 2024-02-12T16:50:35.207 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-12T16:50:35.207 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:34] ENGINE Bus STOPPED 2024-02-12T16:50:35.207 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:34] ENGINE Bus STARTING 2024-02-12T16:50:35.208 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:34] ENGINE Serving on http://:::9283 2024-02-12T16:50:35.208 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:50:34] ENGINE Bus STARTED 2024-02-12T16:50:35.227 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-12T16:50:35.227 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-12T16:50:35.227 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph mgr dump --format=json 2024-02-12T16:50:35.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:34.278327+0000 mgr.y (mgr.24481) 33 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:35.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:34.475361+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-12T16:50:35.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:34.664665+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:34.765302+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:34.771114+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.503 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:35.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:34.278327+0000 mgr.y (mgr.24481) 33 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:34.475361+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:34.664665+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:34.765302+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:34.771114+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:34.775827+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:34.780196+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:34.830988+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:34.831039+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:34.775827+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-12T16:50:35.724 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:34.780196+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:34.830988+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:35.725 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:34.831039+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:34.278327+0000 mgr.y (mgr.24481) 33 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:34.475361+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-12T16:50:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:34.664665+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:34.765302+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:34.771114+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:34.775827+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-12T16:50:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:34.780196+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:35.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:34.830988+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:35.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:34.831039+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:36.621 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:34.776233+0000 mgr.y (mgr.24481) 34 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:34.792484+0000 mgr.y (mgr.24481) 35 : cephadm [INF] Deploying daemon grafana.a on smithi094 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:35.718331+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:35.718401+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:35.782914+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:35.954333+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:34.776233+0000 mgr.y (mgr.24481) 34 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:34.792484+0000 mgr.y (mgr.24481) 35 : cephadm [INF] Deploying daemon grafana.a on smithi094 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:35.718331+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:35.718401+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:35.782914+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-12T16:50:36.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:35.954333+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-12T16:50:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:34.776233+0000 mgr.y (mgr.24481) 34 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-12T16:50:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:34.792484+0000 mgr.y (mgr.24481) 35 : cephadm [INF] Deploying daemon grafana.a on smithi094 2024-02-12T16:50:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:35.718331+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:35.718401+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:35.782914+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-12T16:50:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:35.954333+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-12T16:50:37.384 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:50:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:36.278899+0000 mgr.y (mgr.24481) 36 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:50:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:37.377651+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.33:0/458606621' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-12T16:50:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:37.447143+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:37.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:36.278899+0000 mgr.y (mgr.24481) 36 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:50:37.764 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:37.377651+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.33:0/458606621' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-12T16:50:37.764 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:37.447143+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:37.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:36.278899+0000 mgr.y (mgr.24481) 36 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:50:37.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:37.377651+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.33:0/458606621' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-12T16:50:37.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:37.447143+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:38.168 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":19,"active_gid":24481,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6800","nonce":3049509963}]},"active_addr":"172.21.15.33:6800/3049509963","active_change":"2024-02-12T16:50:02.258182+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":24475,"name":"x","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.33:8443/","prometheus":"http://172.21.15.33:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":69,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3575926640}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":2691712036}]}]} 2024-02-12T16:50:38.172 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-12T16:50:38.172 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-12T16:50:38.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T16:50:38.457 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:39.710 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:38.279427+0000 mgr.y (mgr.24481) 37 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-12T16:50:39.711 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:38.279427+0000 mgr.y (mgr.24481) 37 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-12T16:50:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:38.279427+0000 mgr.y (mgr.24481) 37 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-12T16:50:40.407 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:50:40.407 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":80,"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","created":"2024-02-12T16:43:50.282468+0000","modified":"2024-02-12T16:50:35.949338+0000","last_up_change":"2024-02-12T16:49:30.933575+0000","last_in_change":"2024-02-12T16:49:14.105093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T16:47:33.056945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-12T16:49:37.647343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-12T16:49:38.076200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-12T16:49:39.589565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-12T16:49:41.623856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-12T16:49:43.693806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"5dd172eb-ee9d-4700-9b80-60f91c5f2cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6801","nonce":2545411201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6802","nonce":2545411201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6804","nonce":2545411201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6803","nonce":2545411201}]},"public_addr":"172.21.15.33:6801/2545411201","cluster_addr":"172.21.15.33:6802/2545411201","heartbeat_back_addr":"172.21.15.33:6804/2545411201","heartbeat_front_addr":"172.21.15.33:6803/2545411201","state":["exists","up"]},{"osd":1,"uuid":"71231bc4-1716-431d-bd88-f6a3cad0bd24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6805","nonce":1723231708}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6806","nonce":1723231708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6808","nonce":1723231708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6807","nonce":1723231708}]},"public_addr":"172.21.15.33:6805/1723231708","cluster_addr":"172.21.15.33:6806/1723231708","heartbeat_back_addr":"172.21.15.33:6808/1723231708","heartbeat_front_addr":"172.21.15.33:6807/1723231708","state":["exists","up"]},{"osd":2,"uuid":"8a9c7127-d17c-4c44-b666-4211006de47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6809","nonce":3126580659}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6810","nonce":3126580659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6812","nonce":3126580659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6811","nonce":3126580659}]},"public_addr":"172.21.15.33:6809/3126580659","cluster_addr":"172.21.15.33:6810/3126580659","heartbeat_back_addr":"172.21.15.33:6812/3126580659","heartbeat_front_addr":"172.21.15.33:6811/3126580659","state":["exists","up"]},{"osd":3,"uuid":"3cc40671-c8f7-412c-a0f1-987f48a016b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6813","nonce":3941314957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6814","nonce":3941314957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6816","nonce":3941314957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6815","nonce":3941314957}]},"public_addr":"172.21.15.33:6813/3941314957","cluster_addr":"172.21.15.33:6814/3941314957","heartbeat_back_addr":"172.21.15.33:6816/3941314957","heartbeat_front_addr":"172.21.15.33:6815/3941314957","state":["exists","up"]},{"osd":4,"uuid":"0498129c-861c-4f7e-b8ba-219949da3130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":544126779}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":544126779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":544126779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":544126779}]},"public_addr":"172.21.15.94:6800/544126779","cluster_addr":"172.21.15.94:6801/544126779","heartbeat_back_addr":"172.21.15.94:6803/544126779","heartbeat_front_addr":"172.21.15.94:6802/544126779","state":["exists","up"]},{"osd":5,"uuid":"05718a1e-1164-4b54-a493-23c1cd3adb50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":3229884005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3229884005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3229884005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":3229884005}]},"public_addr":"172.21.15.94:6804/3229884005","cluster_addr":"172.21.15.94:6805/3229884005","heartbeat_back_addr":"172.21.15.94:6807/3229884005","heartbeat_front_addr":"172.21.15.94:6806/3229884005","state":["exists","up"]},{"osd":6,"uuid":"dc48640f-8f27-405c-a9a4-cf1e3e6c8821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":507305558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":507305558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":507305558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":507305558}]},"public_addr":"172.21.15.94:6808/507305558","cluster_addr":"172.21.15.94:6809/507305558","heartbeat_back_addr":"172.21.15.94:6811/507305558","heartbeat_front_addr":"172.21.15.94:6810/507305558","state":["exists","up"]},{"osd":7,"uuid":"7d55aa3f-06b6-4fc3-97fd-62b535f78b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3401977873}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":3401977873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":3401977873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":3401977873}]},"public_addr":"172.21.15.94:6812/3401977873","cluster_addr":"172.21.15.94:6813/3401977873","heartbeat_back_addr":"172.21.15.94:6815/3401977873","heartbeat_front_addr":"172.21.15.94:6814/3401977873","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:35.718403+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:34.237899+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:24.394774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:30.758372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:45.666269+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:52.053698+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:18.760443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:34.831041+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/2829464018":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/1835128904":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2277321750":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2116602769":"2024-02-13T16:50:02.257936+0000","172.21.15.33:6800/3967118778":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/2298361247":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/218227980":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/2070582473":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/4291575956":"2024-02-13T16:44:46.313346+0000","172.21.15.33:6800/3157562110":"2024-02-13T16:44:22.727241+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T16:50:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:40.405101+0000 mon.a (mon.0) 688 : audit [DBG] from='client.? v1:172.21.15.33:0/1417090005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:40.405101+0000 mon.a (mon.0) 688 : audit [DBG] from='client.? v1:172.21.15.33:0/1417090005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:40.405101+0000 mon.a (mon.0) 688 : audit [DBG] from='client.? v1:172.21.15.33:0/1417090005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:41.099 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:50:40] "GET /metrics HTTP/1.1" 200 172112 "" "Prometheus/2.33.4" 2024-02-12T16:50:41.122 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-12T16:50:41.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T16:50:41.404 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:40.279724+0000 mgr.y (mgr.24481) 38 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:41.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:40.279724+0000 mgr.y (mgr.24481) 38 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:40.279724+0000 mgr.y (mgr.24481) 38 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:42.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:40.781986+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:50:42.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:40.782818+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:50:42.708 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:40.781986+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:50:42.708 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:40.782818+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:50:42.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:50:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:50:42.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:40.781986+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:50:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:40.782818+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:50:43.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:50:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:50:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:42.253871+0000 osd.0 (osd.0) 9 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:50:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:42.254746+0000 osd.0 (osd.0) 10 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:50:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:42.280059+0000 mgr.y (mgr.24481) 39 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-02-12T16:50:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:42.862792+0000 osd.4 (osd.4) 5 : cluster [DBG] 2.f scrub starts 2024-02-12T16:50:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:42.866258+0000 osd.4 (osd.4) 6 : cluster [DBG] 2.f scrub ok 2024-02-12T16:50:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:42.874189+0000 osd.1 (osd.1) 9 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:50:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:42.876120+0000 osd.1 (osd.1) 10 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:42.253871+0000 osd.0 (osd.0) 9 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:42.254746+0000 osd.0 (osd.0) 10 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:42.280059+0000 mgr.y (mgr.24481) 39 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:42.862792+0000 osd.4 (osd.4) 5 : cluster [DBG] 2.f scrub starts 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:42.866258+0000 osd.4 (osd.4) 6 : cluster [DBG] 2.f scrub ok 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:42.874189+0000 osd.1 (osd.1) 9 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:42.876120+0000 osd.1 (osd.1) 10 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:42.253871+0000 osd.0 (osd.0) 9 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:42.254746+0000 osd.0 (osd.0) 10 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:50:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:42.280059+0000 mgr.y (mgr.24481) 39 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-02-12T16:50:43.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:42.862792+0000 osd.4 (osd.4) 5 : cluster [DBG] 2.f scrub starts 2024-02-12T16:50:43.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:42.866258+0000 osd.4 (osd.4) 6 : cluster [DBG] 2.f scrub ok 2024-02-12T16:50:43.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:42.874189+0000 osd.1 (osd.1) 9 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:50:43.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:42.876120+0000 osd.1 (osd.1) 10 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:50:43.903 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:50:43.903 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":80,"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","created":"2024-02-12T16:43:50.282468+0000","modified":"2024-02-12T16:50:35.949338+0000","last_up_change":"2024-02-12T16:49:30.933575+0000","last_in_change":"2024-02-12T16:49:14.105093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T16:47:33.056945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-12T16:49:37.647343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-12T16:49:38.076200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-12T16:49:39.589565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-12T16:49:41.623856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-12T16:49:43.693806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"5dd172eb-ee9d-4700-9b80-60f91c5f2cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6801","nonce":2545411201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6802","nonce":2545411201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6804","nonce":2545411201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6803","nonce":2545411201}]},"public_addr":"172.21.15.33:6801/2545411201","cluster_addr":"172.21.15.33:6802/2545411201","heartbeat_back_addr":"172.21.15.33:6804/2545411201","heartbeat_front_addr":"172.21.15.33:6803/2545411201","state":["exists","up"]},{"osd":1,"uuid":"71231bc4-1716-431d-bd88-f6a3cad0bd24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6805","nonce":1723231708}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6806","nonce":1723231708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6808","nonce":1723231708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6807","nonce":1723231708}]},"public_addr":"172.21.15.33:6805/1723231708","cluster_addr":"172.21.15.33:6806/1723231708","heartbeat_back_addr":"172.21.15.33:6808/1723231708","heartbeat_front_addr":"172.21.15.33:6807/1723231708","state":["exists","up"]},{"osd":2,"uuid":"8a9c7127-d17c-4c44-b666-4211006de47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6809","nonce":3126580659}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6810","nonce":3126580659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6812","nonce":3126580659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6811","nonce":3126580659}]},"public_addr":"172.21.15.33:6809/3126580659","cluster_addr":"172.21.15.33:6810/3126580659","heartbeat_back_addr":"172.21.15.33:6812/3126580659","heartbeat_front_addr":"172.21.15.33:6811/3126580659","state":["exists","up"]},{"osd":3,"uuid":"3cc40671-c8f7-412c-a0f1-987f48a016b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6813","nonce":3941314957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6814","nonce":3941314957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6816","nonce":3941314957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6815","nonce":3941314957}]},"public_addr":"172.21.15.33:6813/3941314957","cluster_addr":"172.21.15.33:6814/3941314957","heartbeat_back_addr":"172.21.15.33:6816/3941314957","heartbeat_front_addr":"172.21.15.33:6815/3941314957","state":["exists","up"]},{"osd":4,"uuid":"0498129c-861c-4f7e-b8ba-219949da3130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":544126779}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":544126779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":544126779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":544126779}]},"public_addr":"172.21.15.94:6800/544126779","cluster_addr":"172.21.15.94:6801/544126779","heartbeat_back_addr":"172.21.15.94:6803/544126779","heartbeat_front_addr":"172.21.15.94:6802/544126779","state":["exists","up"]},{"osd":5,"uuid":"05718a1e-1164-4b54-a493-23c1cd3adb50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":3229884005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3229884005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3229884005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":3229884005}]},"public_addr":"172.21.15.94:6804/3229884005","cluster_addr":"172.21.15.94:6805/3229884005","heartbeat_back_addr":"172.21.15.94:6807/3229884005","heartbeat_front_addr":"172.21.15.94:6806/3229884005","state":["exists","up"]},{"osd":6,"uuid":"dc48640f-8f27-405c-a9a4-cf1e3e6c8821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":507305558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":507305558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":507305558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":507305558}]},"public_addr":"172.21.15.94:6808/507305558","cluster_addr":"172.21.15.94:6809/507305558","heartbeat_back_addr":"172.21.15.94:6811/507305558","heartbeat_front_addr":"172.21.15.94:6810/507305558","state":["exists","up"]},{"osd":7,"uuid":"7d55aa3f-06b6-4fc3-97fd-62b535f78b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3401977873}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":3401977873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":3401977873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":3401977873}]},"public_addr":"172.21.15.94:6812/3401977873","cluster_addr":"172.21.15.94:6813/3401977873","heartbeat_back_addr":"172.21.15.94:6815/3401977873","heartbeat_front_addr":"172.21.15.94:6814/3401977873","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:35.718403+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:34.237899+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:24.394774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:30.758372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:45.666269+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:49:52.053698+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:18.760443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:34.831041+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/2829464018":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/1835128904":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2277321750":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2116602769":"2024-02-13T16:50:02.257936+0000","172.21.15.33:6800/3967118778":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/2298361247":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/218227980":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/2070582473":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/4291575956":"2024-02-13T16:44:46.313346+0000","172.21.15.33:6800/3157562110":"2024-02-13T16:44:22.727241+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T16:50:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:43.242461+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:50:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:43.246391+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:50:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:43.268499+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.e scrub starts 2024-02-12T16:50:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:43.269293+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2024-02-12T16:50:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:43.308533+0000 mgr.y (mgr.24481) 40 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:43.900834+0000 mon.a (mon.0) 689 : audit [DBG] from='client.? v1:172.21.15.33:0/1410992034' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:44.089110+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.b scrub starts 2024-02-12T16:50:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:44.092015+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.b scrub ok 2024-02-12T16:50:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:43.242461+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:50:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:43.246391+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:50:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:43.268499+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.e scrub starts 2024-02-12T16:50:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:43.269293+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2024-02-12T16:50:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:43.308533+0000 mgr.y (mgr.24481) 40 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:43.900834+0000 mon.a (mon.0) 689 : audit [DBG] from='client.? v1:172.21.15.33:0/1410992034' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:44.089110+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.b scrub starts 2024-02-12T16:50:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:44.092015+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.b scrub ok 2024-02-12T16:50:44.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:43.242461+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:50:44.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:43.246391+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:50:44.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:43.268499+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.e scrub starts 2024-02-12T16:50:44.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:43.269293+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2024-02-12T16:50:44.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:43.308533+0000 mgr.y (mgr.24481) 40 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:44.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:43.900834+0000 mon.a (mon.0) 689 : audit [DBG] from='client.? v1:172.21.15.33:0/1410992034' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:50:44.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:44.089110+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.b scrub starts 2024-02-12T16:50:44.653 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:44.092015+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.b scrub ok 2024-02-12T16:50:44.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-12T16:50:44.942 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:45.675 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:44.209648+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:50:45.675 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:44.211353+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:50:45.676 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:44.280704+0000 mgr.y (mgr.24481) 41 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:45.676 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:45.274628+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:50:45.775 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:44.209648+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:50:45.775 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:44.211353+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:50:45.775 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:44.280704+0000 mgr.y (mgr.24481) 41 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:45.776 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:45.274628+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:50:45.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:44.209648+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:50:45.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:44.211353+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:50:45.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:44.280704+0000 mgr.y (mgr.24481) 41 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:45.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:45.274628+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:50:46.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:45.171297+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.b scrub starts 2024-02-12T16:50:46.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:45.174930+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.b scrub ok 2024-02-12T16:50:46.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:45.275461+0000 osd.5 (osd.5) 8 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:50:46.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.093017+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.1a deep-scrub starts 2024-02-12T16:50:46.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.095908+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.1a deep-scrub ok 2024-02-12T16:50:46.630 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:45.171297+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.b scrub starts 2024-02-12T16:50:46.630 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:45.174930+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.b scrub ok 2024-02-12T16:50:46.630 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:45.275461+0000 osd.5 (osd.5) 8 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:50:46.630 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.093017+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.1a deep-scrub starts 2024-02-12T16:50:46.630 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.095908+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.1a deep-scrub ok 2024-02-12T16:50:46.728 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:45.171297+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.b scrub starts 2024-02-12T16:50:46.729 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:45.174930+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.b scrub ok 2024-02-12T16:50:46.729 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:45.275461+0000 osd.5 (osd.5) 8 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:50:46.729 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.093017+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.1a deep-scrub starts 2024-02-12T16:50:46.729 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.095908+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.1a deep-scrub ok 2024-02-12T16:50:47.371 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 systemd[1]: Starting Ceph grafana.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:50:47.376 INFO:teuthology.orchestra.run.smithi033.stdout:34359738418 2024-02-12T16:50:47.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-12T16:50:47.659 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:47.759 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.142982+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e deep-scrub starts 2024-02-12T16:50:47.759 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.144004+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e deep-scrub ok 2024-02-12T16:50:47.759 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.281195+0000 mgr.y (mgr.24481) 42 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-02-12T16:50:47.759 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.746492+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:50:47.759 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.750056+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:50:47.759 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.807525+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:50:47.759 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:46.810416+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:50:47.762 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-12T16:50:47.763 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-12T16:50:47.764 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-12T16:50:47.764 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-12T16:50:47.764 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="App mode production" logger=settings 2024-02-12T16:50:47.764 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-12T16:50:47.765 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-12T16:50:47.765 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-12T16:50:47.765 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-12T16:50:47.765 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-12T16:50:47.765 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-12T16:50:47.765 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-12T16:50:47.766 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-02-12T16:50:47.767 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-02-12T16:50:47.767 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-12T16:50:47.767 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-12T16:50:47.767 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-12T16:50:47.767 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-12T16:50:47.767 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-12T16:50:47.768 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-12T16:50:47.768 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-12T16:50:47.768 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-12T16:50:47.768 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-12T16:50:47.768 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-12T16:50:47.769 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-12T16:50:47.770 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-12T16:50:47.771 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-12T16:50:47.772 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-12T16:50:47.773 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-12T16:50:47.774 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-12T16:50:47.775 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-12T16:50:47.775 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-12T16:50:47.776 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-12T16:50:47.776 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-12T16:50:47.776 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-12T16:50:47.777 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-12T16:50:47.778 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-12T16:50:47.779 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 bash[65025]: 1485a3af29549d37b501082e0828f4b2e96e0691decac51a204e0d54a054b9c9 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-12T16:50:47.780 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 systemd[1]: Started Ceph grafana.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-12T16:50:47.781 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-12T16:50:47.782 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-12T16:50:47.783 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-12T16:50:47.784 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-12T16:50:47.785 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-12T16:50:47.785 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-12T16:50:47.785 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-12T16:50:47.785 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-12T16:50:47.785 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-12T16:50:47.785 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-12T16:50:47.785 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-12T16:50:47.785 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-12T16:50:47.786 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-12T16:50:47.787 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-12T16:50:47.788 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-12T16:50:47.789 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-12T16:50:47.790 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-12T16:50:47.791 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-12T16:50:47.792 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-12T16:50:47.793 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-12T16:50:47.794 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-12T16:50:47.795 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-12T16:50:47.796 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-12T16:50:47.796 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-12T16:50:47.796 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-12T16:50:47.796 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-12T16:50:47.796 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-12T16:50:47.796 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-12T16:50:47.796 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-12T16:50:47.796 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-12T16:50:47.797 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-12T16:50:47.798 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-12T16:50:47.798 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-12T16:50:47.798 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-12T16:50:47.798 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-12T16:50:47.798 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-12T16:50:47.798 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=175.722876ms 2024-02-12T16:50:47.798 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-12T16:50:47.798 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-12T16:50:47.799 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-12T16:50:47.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.142982+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e deep-scrub starts 2024-02-12T16:50:47.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.144004+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e deep-scrub ok 2024-02-12T16:50:47.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.281195+0000 mgr.y (mgr.24481) 42 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-02-12T16:50:47.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.746492+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:50:47.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.750056+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.807525+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:46.810416+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.142982+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e deep-scrub starts 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.144004+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e deep-scrub ok 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.281195+0000 mgr.y (mgr.24481) 42 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.746492+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.750056+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.807525+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:50:47.800 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:46.810416+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:50:48.240 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-12T16:50:48.240 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-12T16:50:48.241 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-12T16:50:48.241 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-12T16:50:48.241 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-12T16:50:48.241 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-12T16:50:48.241 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-12T16:50:48.241 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 16:50:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T16:50:47+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-12T16:50:48.940 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:47.280772+0000 osd.2 (osd.2) 9 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:50:48.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:47.282705+0000 osd.2 (osd.2) 10 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:50:48.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:47.314667+0000 osd.5 (osd.5) 9 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:50:48.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:47.317729+0000 osd.5 (osd.5) 10 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:50:48.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:47.692923+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:48.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:47.694497+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:48.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:47.789816+0000 osd.6 (osd.6) 7 : cluster [DBG] 4.a scrub starts 2024-02-12T16:50:48.942 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:47.792381+0000 osd.6 (osd.6) 8 : cluster [DBG] 4.a scrub ok 2024-02-12T16:50:48.942 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:47.280772+0000 osd.2 (osd.2) 9 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:50:48.942 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:47.282705+0000 osd.2 (osd.2) 10 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:50:48.942 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:47.314667+0000 osd.5 (osd.5) 9 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:50:48.942 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:47.317729+0000 osd.5 (osd.5) 10 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:50:48.942 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:47.692923+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:48.943 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:47.694497+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:48.943 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:47.789816+0000 osd.6 (osd.6) 7 : cluster [DBG] 4.a scrub starts 2024-02-12T16:50:48.943 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:47.792381+0000 osd.6 (osd.6) 8 : cluster [DBG] 4.a scrub ok 2024-02-12T16:50:48.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:47.280772+0000 osd.2 (osd.2) 9 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:50:48.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:47.282705+0000 osd.2 (osd.2) 10 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:50:48.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:47.314667+0000 osd.5 (osd.5) 9 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:50:48.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:47.317729+0000 osd.5 (osd.5) 10 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:50:48.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:47.692923+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:48.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:47.694497+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:48.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:47.789816+0000 osd.6 (osd.6) 7 : cluster [DBG] 4.a scrub starts 2024-02-12T16:50:48.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:47.792381+0000 osd.6 (osd.6) 8 : cluster [DBG] 4.a scrub ok 2024-02-12T16:50:49.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:48.281746+0000 mgr.y (mgr.24481) 43 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:48.281746+0000 mgr.y (mgr.24481) 43 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:49.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:48.281746+0000 mgr.y (mgr.24481) 43 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:50.364 INFO:teuthology.orchestra.run.smithi033.stdout:55834574895 2024-02-12T16:50:50.365 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-12T16:50:50.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:49.212984+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.16 deep-scrub starts 2024-02-12T16:50:50.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:49.213941+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.16 deep-scrub ok 2024-02-12T16:50:50.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:49.759106+0000 osd.1 (osd.1) 13 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:50:50.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:49.760151+0000 osd.1 (osd.1) 14 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:50:50.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:49.212984+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.16 deep-scrub starts 2024-02-12T16:50:50.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:49.213941+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.16 deep-scrub ok 2024-02-12T16:50:50.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:49.759106+0000 osd.1 (osd.1) 13 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:50:50.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:49.760151+0000 osd.1 (osd.1) 14 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:50:50.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:50.249800+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:50:50.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:50.252949+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:50:50.985 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:49.212984+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.16 deep-scrub starts 2024-02-12T16:50:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:49.213941+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.16 deep-scrub ok 2024-02-12T16:50:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:49.759106+0000 osd.1 (osd.1) 13 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:50:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:49.760151+0000 osd.1 (osd.1) 14 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:50:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:50.249800+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:50:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:50.252949+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:50:51.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:50:50] "GET /metrics HTTP/1.1" 200 172113 "" "Prometheus/2.33.4" 2024-02-12T16:50:51.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:50.249800+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:50:51.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:50.252949+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:50:51.990 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:50.183086+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:50:51.990 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:50.186014+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:50:51.990 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:50.282316+0000 mgr.y (mgr.24481) 44 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:50.795920+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e deep-scrub starts 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:50.799978+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e deep-scrub ok 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:51.120169+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:51.123056+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:50.183086+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:50.186014+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:50.282316+0000 mgr.y (mgr.24481) 44 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:50.795920+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e deep-scrub starts 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:50.799978+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e deep-scrub ok 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:51.120169+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:50:51.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:51.123056+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:50:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:50.183086+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:50:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:50.186014+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:50:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:50.282316+0000 mgr.y (mgr.24481) 44 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-12T16:50:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:50.795920+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e deep-scrub starts 2024-02-12T16:50:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:50.799978+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e deep-scrub ok 2024-02-12T16:50:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:51.120169+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:50:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:51.123056+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:50:52.837 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:51.307477+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-02-12T16:50:52.837 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:51.308309+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-02-12T16:50:52.837 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:51.749994+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:51.823166+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 deep-scrub starts 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:51.823963+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 deep-scrub ok 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:51.904643+0000 osd.3 (osd.3) 7 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:51.908322+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:52.282613+0000 mgr.y (mgr.24481) 45 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:52.463068+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:51.307477+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:51.308309+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:51.749994+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:51.823166+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 deep-scrub starts 2024-02-12T16:50:52.838 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:51.823963+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 deep-scrub ok 2024-02-12T16:50:52.839 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:51.904643+0000 osd.3 (osd.3) 7 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:50:52.839 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:51.908322+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:50:52.839 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:52.282613+0000 mgr.y (mgr.24481) 45 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-12T16:50:52.839 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:52.463068+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:52.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:50:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:51.307477+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:51.308309+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:51.749994+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:51.823166+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 deep-scrub starts 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:51.823963+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 deep-scrub ok 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:51.904643+0000 osd.3 (osd.3) 7 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:51.908322+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:52.282613+0000 mgr.y (mgr.24481) 45 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-12T16:50:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:52.463068+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:53.711 INFO:teuthology.orchestra.run.smithi033.stdout:77309411368 2024-02-12T16:50:53.712 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-12T16:50:53.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:50:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:50:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:52.288801+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T16:50:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:52.289718+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T16:50:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:53.316568+0000 mgr.y (mgr.24481) 46 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:54.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:52.288801+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T16:50:54.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:52.289718+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T16:50:54.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:53.316568+0000 mgr.y (mgr.24481) 46 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:54.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:52.288801+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T16:50:54.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:52.289718+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T16:50:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:53.316568+0000 mgr.y (mgr.24481) 46 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:50:54.252 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:55.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:54.131997+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.8 deep-scrub starts 2024-02-12T16:50:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:54.135005+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.8 deep-scrub ok 2024-02-12T16:50:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:54.283414+0000 mgr.y (mgr.24481) 47 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:55.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:54.131997+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.8 deep-scrub starts 2024-02-12T16:50:55.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:54.135005+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.8 deep-scrub ok 2024-02-12T16:50:55.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:54.283414+0000 mgr.y (mgr.24481) 47 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:55.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:54.131997+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.8 deep-scrub starts 2024-02-12T16:50:55.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:54.135005+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.8 deep-scrub ok 2024-02-12T16:50:55.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:54.283414+0000 mgr.y (mgr.24481) 47 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:55.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:54.298560+0000 osd.5 (osd.5) 13 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:50:55.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:54.302055+0000 osd.5 (osd.5) 14 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:50:55.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:54.849536+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:50:55.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:54.852477+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:55.108214+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:55.111672+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:55.179016+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:55.180199+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:55.180866+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:50:55.185778+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:55.189348+0000 mgr.y (mgr.24481) 48 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T16:50:55.192190+0000 mgr.y (mgr.24481) 49 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi033 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:54.298560+0000 osd.5 (osd.5) 13 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:54.302055+0000 osd.5 (osd.5) 14 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:54.849536+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:50:55.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:54.852477+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:50:55.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:55.108214+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:50:55.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:55.111672+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:50:55.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:55.179016+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:55.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:55.180199+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:55.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:55.180866+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:55.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:50:55.185778+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:55.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:55.189348+0000 mgr.y (mgr.24481) 48 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-12T16:50:55.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T16:50:55.192190+0000 mgr.y (mgr.24481) 49 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi033 2024-02-12T16:50:56.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:54.298560+0000 osd.5 (osd.5) 13 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:50:56.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:54.302055+0000 osd.5 (osd.5) 14 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:54.849536+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:54.852477+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:55.108214+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:55.111672+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:55.179016+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:55.180199+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:55.180866+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:50:55.185778+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:55.189348+0000 mgr.y (mgr.24481) 48 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-12T16:50:56.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T16:50:55.192190+0000 mgr.y (mgr.24481) 49 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi033 2024-02-12T16:50:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:55.284751+0000 osd.2 (osd.2) 15 : cluster [DBG] 6.f scrub starts 2024-02-12T16:50:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:55.285681+0000 osd.2 (osd.2) 16 : cluster [DBG] 6.f scrub ok 2024-02-12T16:50:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:55.319500+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.1b deep-scrub starts 2024-02-12T16:50:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:55.322439+0000 osd.5 (osd.5) 16 : cluster [DBG] 5.1b deep-scrub ok 2024-02-12T16:50:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:55.940436+0000 osd.6 (osd.6) 9 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:50:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:55.941557+0000 osd.6 (osd.6) 10 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:50:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:56.146351+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:50:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:56.149235+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:50:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:56.284027+0000 mgr.y (mgr.24481) 50 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:57.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:55.284751+0000 osd.2 (osd.2) 15 : cluster [DBG] 6.f scrub starts 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:55.285681+0000 osd.2 (osd.2) 16 : cluster [DBG] 6.f scrub ok 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:55.319500+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.1b deep-scrub starts 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:55.322439+0000 osd.5 (osd.5) 16 : cluster [DBG] 5.1b deep-scrub ok 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:55.940436+0000 osd.6 (osd.6) 9 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:55.941557+0000 osd.6 (osd.6) 10 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:56.146351+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:56.149235+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:56.284027+0000 mgr.y (mgr.24481) 50 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:55.284751+0000 osd.2 (osd.2) 15 : cluster [DBG] 6.f scrub starts 2024-02-12T16:50:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:55.285681+0000 osd.2 (osd.2) 16 : cluster [DBG] 6.f scrub ok 2024-02-12T16:50:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:55.319500+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.1b deep-scrub starts 2024-02-12T16:50:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:55.322439+0000 osd.5 (osd.5) 16 : cluster [DBG] 5.1b deep-scrub ok 2024-02-12T16:50:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:55.940436+0000 osd.6 (osd.6) 9 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:50:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:55.941557+0000 osd.6 (osd.6) 10 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:50:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:56.146351+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:50:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:56.149235+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:50:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:56.284027+0000 mgr.y (mgr.24481) 50 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:50:57.971 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:56.317303+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:50:57.971 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:56.320235+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:50:57.971 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:57.181666+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:50:57.971 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:57.182672+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:50:57.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:56.317303+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:50:57.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:56.320235+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:50:57.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:57.181666+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:50:57.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:57.182672+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:50:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:56.317303+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:50:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:56.320235+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:50:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:57.181666+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:50:58.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:57.182672+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:50:58.927 INFO:teuthology.orchestra.run.smithi033.stdout:111669149733 2024-02-12T16:50:58.928 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-12T16:50:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:57.874572+0000 osd.1 (osd.1) 15 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:50:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:57.875867+0000 osd.1 (osd.1) 16 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:50:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:57.962614+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:50:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:57.964619+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:50:59.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:50:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:58.284632+0000 mgr.y (mgr.24481) 51 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:59.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:57.874572+0000 osd.1 (osd.1) 15 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:57.875867+0000 osd.1 (osd.1) 16 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:57.962614+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:57.964619+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:58.284632+0000 mgr.y (mgr.24481) 51 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:57.874572+0000 osd.1 (osd.1) 15 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:57.875867+0000 osd.1 (osd.1) 16 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:57.962614+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:57.964619+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:50:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:50:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:58.284632+0000 mgr.y (mgr.24481) 51 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:00.439 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:00 smithi033 systemd[1]: Stopping Ceph alertmanager.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:51:00.440 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:00 smithi033 bash[72092]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-12T16:51:00.440 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:59.820279+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:51:00.440 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:50:59.823892+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:51:00.440 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:59.820279+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:51:00.440 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:50:59.823892+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:51:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:59.820279+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:51:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:50:59.823892+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:51:00.749 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[68322]: level=info ts=2024-02-12T16:51:00.577Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-12T16:51:01.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:00] "GET /metrics HTTP/1.1" 200 172113 "" "Prometheus/2.33.4" 2024-02-12T16:51:01.418 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:00.236149+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.7 deep-scrub starts 2024-02-12T16:51:01.418 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:00.237114+0000 osd.5 (osd.5) 20 : cluster [DBG] 6.7 deep-scrub ok 2024-02-12T16:51:01.418 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:00.285067+0000 mgr.y (mgr.24481) 52 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:51:01.418 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:00.236149+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.7 deep-scrub starts 2024-02-12T16:51:01.418 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:00.237114+0000 osd.5 (osd.5) 20 : cluster [DBG] 6.7 deep-scrub ok 2024-02-12T16:51:01.418 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:00.285067+0000 mgr.y (mgr.24481) 52 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:51:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:00.236149+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.7 deep-scrub starts 2024-02-12T16:51:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:00.237114+0000 osd.5 (osd.5) 20 : cluster [DBG] 6.7 deep-scrub ok 2024-02-12T16:51:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:00.285067+0000 mgr.y (mgr.24481) 52 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:51:02.027 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:01 smithi033 bash[72171]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a 2024-02-12T16:51:02.437 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:00.777608+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:51:02.437 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:00.778573+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:51:02.437 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:01.024870+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:51:02.437 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:01.027835+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:51:02.437 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:01.220978+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:51:02.438 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:01.223965+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:51:02.438 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:00.777608+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:51:02.438 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:00.778573+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:51:02.438 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:01.024870+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:51:02.438 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:01.027835+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:51:02.438 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:01.220978+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:51:02.438 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:01.223965+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:51:02.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:00.777608+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:51:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:00.778573+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:51:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:01.024870+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:51:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:01.027835+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:51:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:01.220978+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:51:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:01.223965+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:51:02.734 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:02 smithi033 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-12T16:51:02.735 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:02 smithi033 systemd[1]: Stopped Ceph alertmanager.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:51:02.735 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:02 smithi033 systemd[1]: Starting Ceph alertmanager.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T16:51:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:51:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:51:03.432 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:01.755896+0000 osd.1 (osd.1) 19 : cluster [DBG] 4.12 deep-scrub starts 2024-02-12T16:51:03.432 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:01.759797+0000 osd.1 (osd.1) 20 : cluster [DBG] 4.12 deep-scrub ok 2024-02-12T16:51:03.432 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.215110+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub starts 2024-02-12T16:51:03.432 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.216187+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.d scrub ok 2024-02-12T16:51:03.432 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.285501+0000 mgr.y (mgr.24481) 53 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-12T16:51:03.432 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:02.291476+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:02.291820+0000 mon.c (mon.2) 53 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:02.291856+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:02.292120+0000 mon.c (mon.2) 54 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:02.292140+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:02.292440+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:02.292443+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:02.292708+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.853432+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.857224+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:01.755896+0000 osd.1 (osd.1) 19 : cluster [DBG] 4.12 deep-scrub starts 2024-02-12T16:51:03.433 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:01.759797+0000 osd.1 (osd.1) 20 : cluster [DBG] 4.12 deep-scrub ok 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.215110+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub starts 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.216187+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.d scrub ok 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.285501+0000 mgr.y (mgr.24481) 53 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:02.291476+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:02.291820+0000 mon.c (mon.2) 53 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:02.291856+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:02.292120+0000 mon.c (mon.2) 54 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:02.292140+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:02.292440+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:02.292443+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:02.292708+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.853432+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:51:03.434 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.857224+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:51:03.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:01.755896+0000 osd.1 (osd.1) 19 : cluster [DBG] 4.12 deep-scrub starts 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:01.759797+0000 osd.1 (osd.1) 20 : cluster [DBG] 4.12 deep-scrub ok 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.215110+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub starts 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.216187+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.d scrub ok 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.285501+0000 mgr.y (mgr.24481) 53 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:02.291476+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:02.291820+0000 mon.c (mon.2) 53 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:02.291856+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:02.292120+0000 mon.c (mon.2) 54 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:02.292140+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:02.292440+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:02.292443+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:02.292708+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-12T16:51:03.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.853432+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:51:03.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.857224+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:51:03.808 INFO:teuthology.orchestra.run.smithi033.stdout:137438953506 2024-02-12T16:51:03.808 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-12T16:51:03.990 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 16:51:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T16:51:03.506Z caller=notifier.go:526 level=error component=notifier alertmanager=http://smithi033.front.sepia.ceph.com:9093/api/v2/alerts count=2 msg="Error sending alert" err="Post \"http://smithi033.front.sepia.ceph.com:9093/api/v2/alerts\": dial tcp 172.21.15.33:9093: connect: connection refused" 2024-02-12T16:51:04.114 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:03.985Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-12T16:51:04.114 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:03.985Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-12T16:51:04.114 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:03.986Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.33 port=9094 2024-02-12T16:51:04.114 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:03.988Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-12T16:51:04.114 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:04.014Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-12T16:51:04.114 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:04.014Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-12T16:51:04.114 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:04.017Z caller=main.go:518 msg=Listening address=:9093 2024-02-12T16:51:04.114 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:04.017Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.277254+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.278328+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.784368+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.785420+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:02.998306+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:03.001306+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:03.103674+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:03.103804+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:03.103879+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-12T16:51:04.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:03.103962+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-02-12T16:51:04.116 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:03.104041+0000 mon.a (mon.0) 703 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-12T16:51:04.116 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:03.324822+0000 mgr.y (mgr.24481) 54 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:04.342 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:04.429 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:04 smithi033 bash[72398]: 619ed9992e089f615f33f309aeb52f9e0d0f2ac9b61e7fef94475e8d5c48f816 2024-02-12T16:51:04.429 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:04 smithi033 systemd[1]: Started Ceph alertmanager.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T16:51:04.429 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STOPPING 2024-02-12T16:51:04.429 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-12T16:51:04.429 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STOPPED 2024-02-12T16:51:04.429 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STARTING 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.277254+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.278328+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.784368+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.785420+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:02.998306+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:03.001306+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:03.103674+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:03.103804+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:03.103879+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:03.103962+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-02-12T16:51:04.430 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:03.104041+0000 mon.a (mon.0) 703 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-12T16:51:04.431 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:03.324822+0000 mgr.y (mgr.24481) 54 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:04.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.277254+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:51:04.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.278328+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:51:04.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.784368+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-02-12T16:51:04.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.785420+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-02-12T16:51:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:02.998306+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:51:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:03.001306+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:51:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:03.103674+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2024-02-12T16:51:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:03.103804+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-12T16:51:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:03.103879+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-12T16:51:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:03.103962+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-02-12T16:51:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:03.104041+0000 mon.a (mon.0) 703 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-12T16:51:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:03.324822+0000 mgr.y (mgr.24481) 54 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:04.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Serving on http://:::9283 2024-02-12T16:51:04.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STARTED 2024-02-12T16:51:04.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STOPPING 2024-02-12T16:51:04.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-12T16:51:04.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STOPPED 2024-02-12T16:51:04.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STARTING 2024-02-12T16:51:04.778 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Serving on http://:::9283 2024-02-12T16:51:04.779 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STARTED 2024-02-12T16:51:04.779 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STOPPING 2024-02-12T16:51:04.779 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-12T16:51:04.779 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STOPPED 2024-02-12T16:51:04.779 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STARTING 2024-02-12T16:51:04.779 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Serving on http://:::9283 2024-02-12T16:51:04.779 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:04] ENGINE Bus STARTED 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:03.234106+0000 osd.2 (osd.2) 19 : cluster [DBG] 6.14 scrub starts 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:03.235539+0000 osd.2 (osd.2) 20 : cluster [DBG] 6.14 scrub ok 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:03.958039+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:03.961496+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:04.048454+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d deep-scrub starts 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:04.051340+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d deep-scrub ok 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:04.109636+0000 mon.a (mon.0) 704 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.282398+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:04.286573+0000 mgr.y (mgr.24481) 55 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.287142+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-12T16:51:05.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.287639+0000 mgr.y (mgr.24481) 56 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.288493+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi033.front.sepia.ceph.com:9093"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.288830+0000 mgr.y (mgr.24481) 57 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi033.front.sepia.ceph.com:9093"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.293703+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:04.295862+0000 osd.4 (osd.4) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:04.295945+0000 osd.4 (osd.4) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.301075+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.301512+0000 mgr.y (mgr.24481) 58 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.303828+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi094.front.sepia.ceph.com:9095"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.304325+0000 mgr.y (mgr.24481) 59 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi094.front.sepia.ceph.com:9095"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.309788+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.315425+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.315926+0000 mgr.y (mgr.24481) 60 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.320655+0000 mon.c (mon.2) 61 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi094.front.sepia.ceph.com:3000"}]: dispatch 2024-02-12T16:51:05.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.321080+0000 mgr.y (mgr.24481) 61 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi094.front.sepia.ceph.com:3000"}]: dispatch 2024-02-12T16:51:05.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.328220+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:04.331227+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:05.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:04.878848+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.c scrub starts 2024-02-12T16:51:05.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:04.882330+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.c scrub ok 2024-02-12T16:51:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:03.234106+0000 osd.2 (osd.2) 19 : cluster [DBG] 6.14 scrub starts 2024-02-12T16:51:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:03.235539+0000 osd.2 (osd.2) 20 : cluster [DBG] 6.14 scrub ok 2024-02-12T16:51:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:03.958039+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:51:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:03.961496+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:51:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:04.048454+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d deep-scrub starts 2024-02-12T16:51:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:04.051340+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d deep-scrub ok 2024-02-12T16:51:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:04.109636+0000 mon.a (mon.0) 704 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.282398+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:04.286573+0000 mgr.y (mgr.24481) 55 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.287142+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.287639+0000 mgr.y (mgr.24481) 56 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.288493+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi033.front.sepia.ceph.com:9093"}]: dispatch 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.288830+0000 mgr.y (mgr.24481) 57 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi033.front.sepia.ceph.com:9093"}]: dispatch 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.293703+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:04.295862+0000 osd.4 (osd.4) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:04.295945+0000 osd.4 (osd.4) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.301075+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.301512+0000 mgr.y (mgr.24481) 58 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.303828+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi094.front.sepia.ceph.com:9095"}]: dispatch 2024-02-12T16:51:05.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.304325+0000 mgr.y (mgr.24481) 59 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi094.front.sepia.ceph.com:9095"}]: dispatch 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.309788+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.315425+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.315926+0000 mgr.y (mgr.24481) 60 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.320655+0000 mon.c (mon.2) 61 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi094.front.sepia.ceph.com:3000"}]: dispatch 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.321080+0000 mgr.y (mgr.24481) 61 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi094.front.sepia.ceph.com:3000"}]: dispatch 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.328220+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:04.331227+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:04.878848+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.c scrub starts 2024-02-12T16:51:05.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:04.882330+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.c scrub ok 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:03.234106+0000 osd.2 (osd.2) 19 : cluster [DBG] 6.14 scrub starts 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:03.235539+0000 osd.2 (osd.2) 20 : cluster [DBG] 6.14 scrub ok 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:03.958039+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:03.961496+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:04.048454+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d deep-scrub starts 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:04.051340+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d deep-scrub ok 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:04.109636+0000 mon.a (mon.0) 704 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.282398+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:04.286573+0000 mgr.y (mgr.24481) 55 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:51:05.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.287142+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.287639+0000 mgr.y (mgr.24481) 56 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.288493+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi033.front.sepia.ceph.com:9093"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.288830+0000 mgr.y (mgr.24481) 57 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi033.front.sepia.ceph.com:9093"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.293703+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:04.295862+0000 osd.4 (osd.4) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:04.295945+0000 osd.4 (osd.4) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.301075+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.301512+0000 mgr.y (mgr.24481) 58 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.303828+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi094.front.sepia.ceph.com:9095"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.304325+0000 mgr.y (mgr.24481) 59 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi094.front.sepia.ceph.com:9095"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.309788+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.315425+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.315926+0000 mgr.y (mgr.24481) 60 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-12T16:51:05.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.320655+0000 mon.c (mon.2) 61 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi094.front.sepia.ceph.com:3000"}]: dispatch 2024-02-12T16:51:05.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.321080+0000 mgr.y (mgr.24481) 61 : audit [DBG] from='mon.? v1:172.21.15.33:0/2820594172' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi094.front.sepia.ceph.com:3000"}]: dispatch 2024-02-12T16:51:05.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.328220+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:05.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:04.331227+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:05.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:04.878848+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.c scrub starts 2024-02-12T16:51:05.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:04.882330+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.c scrub ok 2024-02-12T16:51:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:05.044063+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:51:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:05.047017+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:51:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:05.200972+0000 osd.5 (osd.5) 25 : cluster [DBG] 5.f deep-scrub starts 2024-02-12T16:51:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:05.201743+0000 osd.5 (osd.5) 26 : cluster [DBG] 5.f deep-scrub ok 2024-02-12T16:51:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:05.295685+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-12T16:51:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:05.960926+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-12T16:51:06.278 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:05.988Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00018964s 2024-02-12T16:51:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:05.044063+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:51:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:05.047017+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:51:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:05.200972+0000 osd.5 (osd.5) 25 : cluster [DBG] 5.f deep-scrub starts 2024-02-12T16:51:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:05.201743+0000 osd.5 (osd.5) 26 : cluster [DBG] 5.f deep-scrub ok 2024-02-12T16:51:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:05.295685+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-12T16:51:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:05.960926+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-12T16:51:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:05.044063+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:51:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:05.047017+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:51:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:05.200972+0000 osd.5 (osd.5) 25 : cluster [DBG] 5.f deep-scrub starts 2024-02-12T16:51:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:05.201743+0000 osd.5 (osd.5) 26 : cluster [DBG] 5.f deep-scrub ok 2024-02-12T16:51:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:05.295685+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-12T16:51:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:05.960926+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-12T16:51:07.424 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:06.287090+0000 mgr.y (mgr.24481) 62 : cluster [DBG] pgmap v50: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:51:07.425 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:06.287090+0000 mgr.y (mgr.24481) 62 : cluster [DBG] pgmap v50: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:51:07.450 INFO:teuthology.orchestra.run.smithi033.stdout:163208757279 2024-02-12T16:51:07.451 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-12T16:51:07.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:06.287090+0000 mgr.y (mgr.24481) 62 : cluster [DBG] pgmap v50: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:51:07.786 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:08.485 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:07.017872+0000 osd.7 (osd.7) 25 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:51:08.485 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:07.021334+0000 osd.7 (osd.7) 26 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:51:08.485 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:07.238560+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:51:08.485 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:07.240231+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:51:08.485 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:07.017872+0000 osd.7 (osd.7) 25 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:51:08.485 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:07.021334+0000 osd.7 (osd.7) 26 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:51:08.485 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:07.238560+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:51:08.485 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:07.240231+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:51:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:07.017872+0000 osd.7 (osd.7) 25 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:51:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:07.021334+0000 osd.7 (osd.7) 26 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:51:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:07.238560+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:51:08.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:07.240231+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:51:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:07.672856+0000 osd.5 (osd.5) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:07.672937+0000 osd.5 (osd.5) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.051191+0000 osd.3 (osd.3) 13 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:51:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.054107+0000 osd.3 (osd.3) 14 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:51:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.071999+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:51:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.073424+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:51:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.126429+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-12T16:51:09.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.287717+0000 mgr.y (mgr.24481) 63 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 7 op/s; 26 B/s, 4 objects/s recovering 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:08.748744+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:07.672856+0000 osd.5 (osd.5) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:07.672937+0000 osd.5 (osd.5) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.051191+0000 osd.3 (osd.3) 13 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.054107+0000 osd.3 (osd.3) 14 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.071999+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.073424+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.126429+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.287717+0000 mgr.y (mgr.24481) 63 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 7 op/s; 26 B/s, 4 objects/s recovering 2024-02-12T16:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:08.748744+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:07.672856+0000 osd.5 (osd.5) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:07.672937+0000 osd.5 (osd.5) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.051191+0000 osd.3 (osd.3) 13 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:51:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.054107+0000 osd.3 (osd.3) 14 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:51:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.071999+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:51:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.073424+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:51:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.126429+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-12T16:51:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.287717+0000 mgr.y (mgr.24481) 63 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 7 op/s; 26 B/s, 4 objects/s recovering 2024-02-12T16:51:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:08.748744+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:10.213 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.885530+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:51:10.213 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.888386+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.948930+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:08.952892+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:09.090863+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:09.094060+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:09.157573+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.15 scrub starts 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:09.159065+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.15 scrub ok 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.885530+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.888386+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.948930+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:08.952892+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:09.090863+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:51:10.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:09.094060+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:51:10.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:09.157573+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.15 scrub starts 2024-02-12T16:51:10.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:09.159065+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.15 scrub ok 2024-02-12T16:51:10.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.885530+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:51:10.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.888386+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:51:10.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.948930+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:51:10.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:08.952892+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:51:10.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:09.090863+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:51:10.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:09.094060+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:51:10.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:09.157573+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.15 scrub starts 2024-02-12T16:51:10.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:09.159065+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.15 scrub ok 2024-02-12T16:51:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:193273528346 2024-02-12T16:51:10.989 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-12T16:51:11.139 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:10] "GET /metrics HTTP/1.1" 200 172128 "" "Prometheus/2.33.4" 2024-02-12T16:51:11.140 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:10.118345+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:51:11.140 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:10.119392+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:51:11.140 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:10.288145+0000 mgr.y (mgr.24481) 64 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 18 B/s, 3 objects/s recovering 2024-02-12T16:51:11.140 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:10.895659+0000 osd.4 (osd.4) 23 : cluster [DBG] 6.1e deep-scrub starts 2024-02-12T16:51:11.140 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:10.898589+0000 osd.4 (osd.4) 24 : cluster [DBG] 6.1e deep-scrub ok 2024-02-12T16:51:11.140 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:10.960908+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-12T16:51:11.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:10.118345+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:51:11.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:10.119392+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:51:11.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:10.288145+0000 mgr.y (mgr.24481) 64 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 18 B/s, 3 objects/s recovering 2024-02-12T16:51:11.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:10.895659+0000 osd.4 (osd.4) 23 : cluster [DBG] 6.1e deep-scrub starts 2024-02-12T16:51:11.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:10.898589+0000 osd.4 (osd.4) 24 : cluster [DBG] 6.1e deep-scrub ok 2024-02-12T16:51:11.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:10.960908+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-12T16:51:11.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:10.118345+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:51:11.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:10.119392+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:51:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:10.288145+0000 mgr.y (mgr.24481) 64 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 18 B/s, 3 objects/s recovering 2024-02-12T16:51:11.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:10.895659+0000 osd.4 (osd.4) 23 : cluster [DBG] 6.1e deep-scrub starts 2024-02-12T16:51:11.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:10.898589+0000 osd.4 (osd.4) 24 : cluster [DBG] 6.1e deep-scrub ok 2024-02-12T16:51:11.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:10.960908+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-12T16:51:11.609 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:12.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:11.087228+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:51:12.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:11.088404+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:51:12.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:11.134315+0000 osd.5 (osd.5) 31 : cluster [DBG] 2.16 deep-scrub starts 2024-02-12T16:51:12.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:11.135257+0000 osd.5 (osd.5) 32 : cluster [DBG] 2.16 deep-scrub ok 2024-02-12T16:51:12.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:11.202618+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.3 scrub starts 2024-02-12T16:51:12.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:11.206411+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.3 scrub ok 2024-02-12T16:51:12.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:11.488270+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:12.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:11.985616+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:12.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:11.986259+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:12.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:11.990732+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:12.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:11.997006+0000 mon.c (mon.2) 65 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:12.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:11.087228+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:11.088404+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:11.134315+0000 osd.5 (osd.5) 31 : cluster [DBG] 2.16 deep-scrub starts 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:11.135257+0000 osd.5 (osd.5) 32 : cluster [DBG] 2.16 deep-scrub ok 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:11.202618+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.3 scrub starts 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:11.206411+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.3 scrub ok 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:11.488270+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:11.985616+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:11.986259+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:11.990732+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:12.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:11.997006+0000 mon.c (mon.2) 65 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:11.087228+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:11.088404+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:11.134315+0000 osd.5 (osd.5) 31 : cluster [DBG] 2.16 deep-scrub starts 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:11.135257+0000 osd.5 (osd.5) 32 : cluster [DBG] 2.16 deep-scrub ok 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:11.202618+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.3 scrub starts 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:11.206411+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.3 scrub ok 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:11.488270+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:11.985616+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:11.986259+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:11.990732+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:12.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:11.997006+0000 mon.c (mon.2) 65 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:51:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:51:13.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:11.973659+0000 osd.1 (osd.1) 25 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:11.980565+0000 osd.1 (osd.1) 26 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:12.288541+0000 mgr.y (mgr.24481) 65 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s; 17 B/s, 3 objects/s recovering 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:12.888534+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.d scrub starts 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:12.892144+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.d scrub ok 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:11.973659+0000 osd.1 (osd.1) 25 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:11.980565+0000 osd.1 (osd.1) 26 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:12.288541+0000 mgr.y (mgr.24481) 65 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s; 17 B/s, 3 objects/s recovering 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:12.888534+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.d scrub starts 2024-02-12T16:51:13.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:12.892144+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.d scrub ok 2024-02-12T16:51:13.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:51:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:51:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:11.973659+0000 osd.1 (osd.1) 25 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T16:51:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:11.980565+0000 osd.1 (osd.1) 26 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T16:51:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:12.288541+0000 mgr.y (mgr.24481) 65 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s; 17 B/s, 3 objects/s recovering 2024-02-12T16:51:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:12.888534+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.d scrub starts 2024-02-12T16:51:13.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:12.892144+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.d scrub ok 2024-02-12T16:51:14.277 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 16:51:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T16:51:13.990Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002398914s 2024-02-12T16:51:14.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:13.010549+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:51:14.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:13.011809+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:51:14.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:13.189864+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:51:14.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:13.192838+0000 osd.0 (osd.0) 30 : cluster [DBG] 6.9 scrub ok 2024-02-12T16:51:14.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:13.332604+0000 mgr.y (mgr.24481) 66 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:13.010549+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:51:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:13.011809+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:51:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:13.189864+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:51:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:13.192838+0000 osd.0 (osd.0) 30 : cluster [DBG] 6.9 scrub ok 2024-02-12T16:51:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:13.332604+0000 mgr.y (mgr.24481) 66 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:14.421 INFO:teuthology.orchestra.run.smithi033.stdout:231928234006 2024-02-12T16:51:14.422 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-12T16:51:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:13.010549+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:51:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:13.011809+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:51:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:13.189864+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:51:14.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:13.192838+0000 osd.0 (osd.0) 30 : cluster [DBG] 6.9 scrub ok 2024-02-12T16:51:14.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:13.332604+0000 mgr.y (mgr.24481) 66 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:14.816 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:15.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:13.982639+0000 osd.1 (osd.1) 29 : cluster [DBG] 5.c scrub starts 2024-02-12T16:51:15.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:13.985608+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.c scrub ok 2024-02-12T16:51:15.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:14.040791+0000 osd.6 (osd.6) 19 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:51:15.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:14.042852+0000 osd.6 (osd.6) 20 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:51:15.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:14.289346+0000 mgr.y (mgr.24481) 67 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 13 B/s, 2 objects/s recovering 2024-02-12T16:51:15.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:14.840495+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:51:15.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:14.844110+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:51:15.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:13.982639+0000 osd.1 (osd.1) 29 : cluster [DBG] 5.c scrub starts 2024-02-12T16:51:15.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:13.985608+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.c scrub ok 2024-02-12T16:51:15.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:14.040791+0000 osd.6 (osd.6) 19 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:51:15.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:14.042852+0000 osd.6 (osd.6) 20 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:51:15.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:14.289346+0000 mgr.y (mgr.24481) 67 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 13 B/s, 2 objects/s recovering 2024-02-12T16:51:15.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:14.840495+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:51:15.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:14.844110+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:51:15.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:13.982639+0000 osd.1 (osd.1) 29 : cluster [DBG] 5.c scrub starts 2024-02-12T16:51:15.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:13.985608+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.c scrub ok 2024-02-12T16:51:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:14.040791+0000 osd.6 (osd.6) 19 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:51:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:14.042852+0000 osd.6 (osd.6) 20 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:51:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:14.289346+0000 mgr.y (mgr.24481) 67 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 13 B/s, 2 objects/s recovering 2024-02-12T16:51:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:14.840495+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:51:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:14.844110+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:51:16.728 INFO:teuthology.orchestra.run.smithi033.stdout:34359738424 2024-02-12T16:51:17.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:15.962215+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:15.965169+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.007514+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.6 scrub starts 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.010503+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.6 scrub ok 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.019539+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.020466+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.071531+0000 osd.3 (osd.3) 17 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.075047+0000 osd.3 (osd.3) 18 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.289759+0000 mgr.y (mgr.24481) 68 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s; 13 B/s, 2 objects/s recovering 2024-02-12T16:51:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:16.725675+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.33:0/2818910445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:15.962215+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:15.965169+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.007514+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.6 scrub starts 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.010503+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.6 scrub ok 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.019539+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.020466+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.071531+0000 osd.3 (osd.3) 17 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.075047+0000 osd.3 (osd.3) 18 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.289759+0000 mgr.y (mgr.24481) 68 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s; 13 B/s, 2 objects/s recovering 2024-02-12T16:51:17.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:16.725675+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.33:0/2818910445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:51:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:15.962215+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T16:51:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:15.965169+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T16:51:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.007514+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.6 scrub starts 2024-02-12T16:51:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.010503+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.6 scrub ok 2024-02-12T16:51:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.019539+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:51:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.020466+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:51:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.071531+0000 osd.3 (osd.3) 17 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:51:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.075047+0000 osd.3 (osd.3) 18 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:51:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.289759+0000 mgr.y (mgr.24481) 68 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s; 13 B/s, 2 objects/s recovering 2024-02-12T16:51:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:16.725675+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.33:0/2818910445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:51:17.974 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.143950+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.d scrub starts 2024-02-12T16:51:17.974 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:16.145565+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.d scrub ok 2024-02-12T16:51:17.974 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:17.046662+0000 osd.0 (osd.0) 31 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T16:51:17.974 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:17.050072+0000 osd.0 (osd.0) 32 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T16:51:17.974 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.143950+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.d scrub starts 2024-02-12T16:51:17.975 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:16.145565+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.d scrub ok 2024-02-12T16:51:17.975 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:17.046662+0000 osd.0 (osd.0) 31 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T16:51:17.975 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:17.050072+0000 osd.0 (osd.0) 32 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T16:51:17.999 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738424 for osd.0 2024-02-12T16:51:18.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-12T16:51:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.143950+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.d scrub starts 2024-02-12T16:51:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:16.145565+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.d scrub ok 2024-02-12T16:51:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:17.046662+0000 osd.0 (osd.0) 31 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T16:51:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:17.050072+0000 osd.0 (osd.0) 32 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T16:51:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:19.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:17.898444+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:51:19.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:17.900085+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:51:19.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:18.025751+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:51:19.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:18.026620+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:51:19.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:18.076527+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:51:19.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:18.079489+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:51:19.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:18.290336+0000 mgr.y (mgr.24481) 69 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:19.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:18.701726+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:17.898444+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:51:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:17.900085+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:51:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:18.025751+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:51:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:18.026620+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:51:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:18.076527+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:51:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:18.079489+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:51:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:18.290336+0000 mgr.y (mgr.24481) 69 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:18.701726+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:19.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:17.898444+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:51:19.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:17.900085+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:51:19.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:18.025751+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:51:19.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:18.026620+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:51:19.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:18.076527+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:51:19.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:18.079489+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:51:19.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:18.290336+0000 mgr.y (mgr.24481) 69 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:19.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:18.701726+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:18.944795+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.7 deep-scrub starts 2024-02-12T16:51:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:18.950026+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.7 deep-scrub ok 2024-02-12T16:51:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:19.026501+0000 osd.6 (osd.6) 25 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:51:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:19.027242+0000 osd.6 (osd.6) 26 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:51:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:19.035489+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.10 scrub starts 2024-02-12T16:51:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:19.038383+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.10 scrub ok 2024-02-12T16:51:20.256 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:18.944795+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.7 deep-scrub starts 2024-02-12T16:51:20.256 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:18.950026+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.7 deep-scrub ok 2024-02-12T16:51:20.256 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:19.026501+0000 osd.6 (osd.6) 25 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:51:20.256 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:19.027242+0000 osd.6 (osd.6) 26 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:51:20.257 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:19.035489+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.10 scrub starts 2024-02-12T16:51:20.257 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:19.038383+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.10 scrub ok 2024-02-12T16:51:20.257 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:18.944795+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.7 deep-scrub starts 2024-02-12T16:51:20.257 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:18.950026+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.7 deep-scrub ok 2024-02-12T16:51:20.257 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:19.026501+0000 osd.6 (osd.6) 25 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:51:20.257 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:19.027242+0000 osd.6 (osd.6) 26 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:51:20.257 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:19.035489+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.10 scrub starts 2024-02-12T16:51:20.257 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:19.038383+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.10 scrub ok 2024-02-12T16:51:20.712 INFO:teuthology.orchestra.run.smithi033.stdout:55834574901 2024-02-12T16:51:21.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:20] "GET /metrics HTTP/1.1" 200 172140 "" "Prometheus/2.33.4" 2024-02-12T16:51:21.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:20.038653+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.15 scrub starts 2024-02-12T16:51:21.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:20.041552+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.15 scrub ok 2024-02-12T16:51:21.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:20.094069+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:21.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:20.095348+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:21.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:20.096077+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:21.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:20.100388+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.12 scrub starts 2024-02-12T16:51:21.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:20.103172+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:21.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:20.103288+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.12 scrub ok 2024-02-12T16:51:21.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:20.290815+0000 mgr.y (mgr.24481) 70 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:21.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:20.710529+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.33:0/370877926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:20.038653+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.15 scrub starts 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:20.041552+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.15 scrub ok 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:20.094069+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:20.095348+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:20.096077+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:20.100388+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.12 scrub starts 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:20.103172+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:20.103288+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.12 scrub ok 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:20.290815+0000 mgr.y (mgr.24481) 70 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:21.523 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:20.710529+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.33:0/370877926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:20.038653+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.15 scrub starts 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:20.041552+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.15 scrub ok 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:20.094069+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:20.095348+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:20.096077+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:20.100388+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.12 scrub starts 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:20.103172+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:20.103288+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.12 scrub ok 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:20.290815+0000 mgr.y (mgr.24481) 70 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:21.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:20.710529+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.33:0/370877926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:51:21.544 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574895 got 55834574901 for osd.1 2024-02-12T16:51:21.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-12T16:51:21.823 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:22.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:20.968759+0000 osd.1 (osd.1) 35 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:51:22.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:20.972294+0000 osd.1 (osd.1) 36 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:51:22.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:21.575922+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:22.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:21.576014+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:22.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:21.894858+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1a scrub starts 2024-02-12T16:51:22.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:21.897744+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1a scrub ok 2024-02-12T16:51:22.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:20.968759+0000 osd.1 (osd.1) 35 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:51:22.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:20.972294+0000 osd.1 (osd.1) 36 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:51:22.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:21.575922+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:22.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:21.576014+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:22.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:21.894858+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1a scrub starts 2024-02-12T16:51:22.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:21.897744+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1a scrub ok 2024-02-12T16:51:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:20.968759+0000 osd.1 (osd.1) 35 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:51:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:20.972294+0000 osd.1 (osd.1) 36 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:51:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:21.575922+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:22.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:21.576014+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:22.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:21.894858+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1a scrub starts 2024-02-12T16:51:22.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:21.897744+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1a scrub ok 2024-02-12T16:51:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:51:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:51:23.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.017437+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-02-12T16:51:23.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.020424+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-02-12T16:51:23.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.059233+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:51:23.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.062120+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:51:23.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.069240+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:51:23.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.070052+0000 osd.5 (osd.5) 36 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.106329+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.291350+0000 mgr.y (mgr.24481) 71 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.017437+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.020424+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.059233+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.062120+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.069240+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.070052+0000 osd.5 (osd.5) 36 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.106329+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-12T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.291350+0000 mgr.y (mgr.24481) 71 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:23.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:51:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.017437+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-02-12T16:51:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.020424+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-02-12T16:51:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.059233+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:51:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.062120+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:51:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.069240+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:51:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.070052+0000 osd.5 (osd.5) 36 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:51:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.106329+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-12T16:51:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.291350+0000 mgr.y (mgr.24481) 71 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:24.237 INFO:teuthology.orchestra.run.smithi033.stdout:77309411374 2024-02-12T16:51:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.126932+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.8 deep-scrub starts 2024-02-12T16:51:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:22.127828+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.8 deep-scrub ok 2024-02-12T16:51:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:23.145085+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:51:24.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:23.148772+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:51:24.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:23.343582+0000 mgr.y (mgr.24481) 72 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:24.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.126932+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.8 deep-scrub starts 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:22.127828+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.8 deep-scrub ok 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:23.145085+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:23.148772+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:23.343582+0000 mgr.y (mgr.24481) 72 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.126932+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.8 deep-scrub starts 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:22.127828+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.8 deep-scrub ok 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:23.145085+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:23.148772+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:51:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:23.343582+0000 mgr.y (mgr.24481) 72 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:24.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411374 for osd.2 2024-02-12T16:51:24.972 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-12T16:51:25.258 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:25.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:23.964944+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:23.967843+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:24.027109+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.a scrub starts 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:24.030013+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.a scrub ok 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:24.082326+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:24.083263+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:24.236188+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? v1:172.21.15.33:0/1726210615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:24.292218+0000 mgr.y (mgr.24481) 73 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:24.894076+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:24.897144+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:51:25.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:23.964944+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:23.967843+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:24.027109+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.a scrub starts 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:24.030013+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.a scrub ok 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:24.082326+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:24.083263+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:24.236188+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? v1:172.21.15.33:0/1726210615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:24.292218+0000 mgr.y (mgr.24481) 73 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:24.894076+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:51:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:24.897144+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:51:25.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:23.964944+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:51:25.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:23.967843+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:51:25.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:24.027109+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.a scrub starts 2024-02-12T16:51:25.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:24.030013+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.a scrub ok 2024-02-12T16:51:25.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:24.082326+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:51:25.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:24.083263+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:51:25.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:24.236188+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? v1:172.21.15.33:0/1726210615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:51:25.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:24.292218+0000 mgr.y (mgr.24481) 73 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:25.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:24.894076+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:51:25.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:24.897144+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:51:26.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:25.026549+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.15 scrub starts 2024-02-12T16:51:26.337 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:25.028334+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.15 scrub ok 2024-02-12T16:51:26.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:25.026549+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.15 scrub starts 2024-02-12T16:51:26.337 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:25.028334+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.15 scrub ok 2024-02-12T16:51:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:25.026549+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.15 scrub starts 2024-02-12T16:51:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:25.028334+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.15 scrub ok 2024-02-12T16:51:27.025 INFO:teuthology.orchestra.run.smithi033.stdout:111669149739 2024-02-12T16:51:27.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:25.910209+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:51:27.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:25.913141+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:51:27.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:26.160620+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:51:27.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:26.163564+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:51:27.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:26.292600+0000 mgr.y (mgr.24481) 74 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:27.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:27.024005+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.33:0/1509698562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:51:27.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:25.910209+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:51:27.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:25.913141+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:51:27.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:26.160620+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:51:27.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:26.163564+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:51:27.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:26.292600+0000 mgr.y (mgr.24481) 74 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:27.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:27.024005+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.33:0/1509698562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:51:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:25.910209+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:51:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:25.913141+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:51:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:26.160620+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:51:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:26.163564+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:51:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:26.292600+0000 mgr.y (mgr.24481) 74 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:27.024005+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.33:0/1509698562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:51:27.763 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149733 got 111669149739 for osd.3 2024-02-12T16:51:27.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-12T16:51:28.041 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:28.099922+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.1c deep-scrub starts 2024-02-12T16:51:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:28.103964+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.1c deep-scrub ok 2024-02-12T16:51:28.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:28.104570+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.17 scrub starts 2024-02-12T16:51:28.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:28.108032+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.17 scrub ok 2024-02-12T16:51:28.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:28.099922+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.1c deep-scrub starts 2024-02-12T16:51:28.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:28.103964+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.1c deep-scrub ok 2024-02-12T16:51:28.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:28.104570+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.17 scrub starts 2024-02-12T16:51:28.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:28.108032+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.17 scrub ok 2024-02-12T16:51:28.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:28.099922+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.1c deep-scrub starts 2024-02-12T16:51:28.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:28.103964+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.1c deep-scrub ok 2024-02-12T16:51:28.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:28.104570+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.17 scrub starts 2024-02-12T16:51:28.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:28.108032+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.17 scrub ok 2024-02-12T16:51:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:28.293175+0000 mgr.y (mgr.24481) 75 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:28.883035+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.10 deep-scrub starts 2024-02-12T16:51:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:29.063090+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-02-12T16:51:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:29.066103+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-02-12T16:51:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:28.293175+0000 mgr.y (mgr.24481) 75 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:28.883035+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.10 deep-scrub starts 2024-02-12T16:51:29.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:29.063090+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-02-12T16:51:29.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:29.066103+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-02-12T16:51:29.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:28.293175+0000 mgr.y (mgr.24481) 75 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:29.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:28.883035+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.10 deep-scrub starts 2024-02-12T16:51:29.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:29.063090+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-02-12T16:51:29.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:29.066103+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-02-12T16:51:30.042 INFO:teuthology.orchestra.run.smithi033.stdout:137438953512 2024-02-12T16:51:30.359 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:28.886316+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.10 deep-scrub ok 2024-02-12T16:51:30.359 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:30.041419+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.33:0/2078654641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:51:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:28.886316+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.10 deep-scrub ok 2024-02-12T16:51:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:30.041419+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.33:0/2078654641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:51:30.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:28.886316+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.10 deep-scrub ok 2024-02-12T16:51:30.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:30.041419+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.33:0/2078654641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:51:30.799 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953506 got 137438953512 for osd.4 2024-02-12T16:51:30.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-12T16:51:31.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:30] "GET /metrics HTTP/1.1" 200 172140 "" "Prometheus/2.33.4" 2024-02-12T16:51:31.081 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:31.611 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:30.293487+0000 mgr.y (mgr.24481) 76 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:31.611 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:30.773097+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.e scrub starts 2024-02-12T16:51:31.611 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:30.776865+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.e scrub ok 2024-02-12T16:51:31.611 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:30.293487+0000 mgr.y (mgr.24481) 76 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:31.611 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:30.773097+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.e scrub starts 2024-02-12T16:51:31.611 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:30.776865+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.e scrub ok 2024-02-12T16:51:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:30.293487+0000 mgr.y (mgr.24481) 76 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:30.773097+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.e scrub starts 2024-02-12T16:51:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:30.776865+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.e scrub ok 2024-02-12T16:51:32.626 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:32.011787+0000 osd.3 (osd.3) 27 : cluster [DBG] 5.17 deep-scrub starts 2024-02-12T16:51:32.626 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:32.014768+0000 osd.3 (osd.3) 28 : cluster [DBG] 5.17 deep-scrub ok 2024-02-12T16:51:32.627 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:32.011787+0000 osd.3 (osd.3) 27 : cluster [DBG] 5.17 deep-scrub starts 2024-02-12T16:51:32.627 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:32.014768+0000 osd.3 (osd.3) 28 : cluster [DBG] 5.17 deep-scrub ok 2024-02-12T16:51:32.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:51:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:51:32.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:32.011787+0000 osd.3 (osd.3) 27 : cluster [DBG] 5.17 deep-scrub starts 2024-02-12T16:51:32.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:32.014768+0000 osd.3 (osd.3) 28 : cluster [DBG] 5.17 deep-scrub ok 2024-02-12T16:51:33.029 INFO:teuthology.orchestra.run.smithi033.stdout:163208757284 2024-02-12T16:51:33.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:51:33.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:31.886067+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.d scrub starts 2024-02-12T16:51:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:31.889817+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.d scrub ok 2024-02-12T16:51:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:32.293828+0000 mgr.y (mgr.24481) 77 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-02-12T16:51:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:32.834034+0000 osd.4 (osd.4) 37 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:51:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:32.836924+0000 osd.4 (osd.4) 38 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:51:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:33.028402+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.33:0/2928362028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:51:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:33.029742+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.b scrub starts 2024-02-12T16:51:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:33.032780+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.b scrub ok 2024-02-12T16:51:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:31.886067+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.d scrub starts 2024-02-12T16:51:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:31.889817+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.d scrub ok 2024-02-12T16:51:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:32.293828+0000 mgr.y (mgr.24481) 77 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-02-12T16:51:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:32.834034+0000 osd.4 (osd.4) 37 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:51:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:32.836924+0000 osd.4 (osd.4) 38 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:51:33.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:33.028402+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.33:0/2928362028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:51:33.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:33.029742+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.b scrub starts 2024-02-12T16:51:33.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:33.032780+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.b scrub ok 2024-02-12T16:51:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:31.886067+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.d scrub starts 2024-02-12T16:51:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:31.889817+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.d scrub ok 2024-02-12T16:51:33.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:32.293828+0000 mgr.y (mgr.24481) 77 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-02-12T16:51:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:32.834034+0000 osd.4 (osd.4) 37 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:51:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:32.836924+0000 osd.4 (osd.4) 38 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:51:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:33.028402+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.33:0/2928362028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:51:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:33.029742+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.b scrub starts 2024-02-12T16:51:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:33.032780+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.b scrub ok 2024-02-12T16:51:33.865 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757279 got 163208757284 for osd.5 2024-02-12T16:51:33.865 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-12T16:51:34.147 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:32.883601+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1 deep-scrub starts 2024-02-12T16:51:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:32.884665+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1 deep-scrub ok 2024-02-12T16:51:34.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:33.354288+0000 mgr.y (mgr.24481) 78 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:34.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:34.078603+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.c scrub starts 2024-02-12T16:51:34.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:34.081640+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.c scrub ok 2024-02-12T16:51:34.742 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:32.883601+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1 deep-scrub starts 2024-02-12T16:51:34.742 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:32.884665+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1 deep-scrub ok 2024-02-12T16:51:34.742 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:33.354288+0000 mgr.y (mgr.24481) 78 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:34.742 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:34.078603+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.c scrub starts 2024-02-12T16:51:34.742 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:34.081640+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.c scrub ok 2024-02-12T16:51:34.742 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:32.883601+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1 deep-scrub starts 2024-02-12T16:51:34.743 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:32.884665+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1 deep-scrub ok 2024-02-12T16:51:34.743 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:33.354288+0000 mgr.y (mgr.24481) 78 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:34.743 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:34.078603+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.c scrub starts 2024-02-12T16:51:34.743 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:34.081640+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.c scrub ok 2024-02-12T16:51:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:34.294741+0000 mgr.y (mgr.24481) 79 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:34.432141+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:34.432191+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:34.757261+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:51:35.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:34.760706+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:51:35.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:34.294741+0000 mgr.y (mgr.24481) 79 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:34.432141+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:34.432191+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:34.757261+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:34.760706+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:34.294741+0000 mgr.y (mgr.24481) 79 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:34.432141+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:34.432191+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:34.757261+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:51:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:34.760706+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:51:36.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:35.384327+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-12T16:51:36.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:35.746360+0000 osd.4 (osd.4) 41 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:51:36.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:35.749233+0000 osd.4 (osd.4) 42 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:51:36.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:36.022858+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:51:36.386 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:36.026564+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:51:36.485 INFO:teuthology.orchestra.run.smithi033.stdout:193273528351 2024-02-12T16:51:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:35.384327+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-12T16:51:36.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:35.746360+0000 osd.4 (osd.4) 41 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:51:36.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:35.749233+0000 osd.4 (osd.4) 42 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:51:36.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:36.022858+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:51:36.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:36.026564+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:51:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:35.384327+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-12T16:51:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:35.746360+0000 osd.4 (osd.4) 41 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:51:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:35.749233+0000 osd.4 (osd.4) 42 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:51:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:36.022858+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:51:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:36.026564+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:51:37.205 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528346 got 193273528351 for osd.6 2024-02-12T16:51:37.206 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-12T16:51:37.492 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:37.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:36.295043+0000 mgr.y (mgr.24481) 80 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:36.484074+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.33:0/3602322490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:51:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:37.181913+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:51:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:37.184902+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:51:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:36.295043+0000 mgr.y (mgr.24481) 80 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:36.484074+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.33:0/3602322490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:51:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:37.181913+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:51:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:37.184902+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:51:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:36.295043+0000 mgr.y (mgr.24481) 80 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:36.484074+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.33:0/3602322490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:51:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:37.181913+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:51:37.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:37.184902+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:51:38.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:36.839670+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:51:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:36.842641+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:51:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:38.070482+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.1e scrub starts 2024-02-12T16:51:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:38.073437+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.1e scrub ok 2024-02-12T16:51:38.773 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:36.839670+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:51:38.773 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:36.842641+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:51:38.773 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:38.070482+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.1e scrub starts 2024-02-12T16:51:38.773 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:38.073437+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.1e scrub ok 2024-02-12T16:51:38.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:36.839670+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:51:38.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:36.842641+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:51:38.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:38.070482+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.1e scrub starts 2024-02-12T16:51:38.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:38.073437+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.1e scrub ok 2024-02-12T16:51:39.509 INFO:teuthology.orchestra.run.smithi033.stdout:231928234011 2024-02-12T16:51:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:37.875304+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:51:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:37.878784+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:51:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:38.295758+0000 mgr.y (mgr.24481) 81 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:38.770642+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.e scrub starts 2024-02-12T16:51:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:38.773694+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.e scrub ok 2024-02-12T16:51:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:39.037328+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.13 scrub starts 2024-02-12T16:51:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:39.040199+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.13 scrub ok 2024-02-12T16:51:39.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:37.875304+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:37.878784+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:38.295758+0000 mgr.y (mgr.24481) 81 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:38.770642+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.e scrub starts 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:38.773694+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.e scrub ok 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:39.037328+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.13 scrub starts 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:39.040199+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.13 scrub ok 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:37.875304+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:37.878784+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:38.295758+0000 mgr.y (mgr.24481) 81 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:38.770642+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.e scrub starts 2024-02-12T16:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:38.773694+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.e scrub ok 2024-02-12T16:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:39.037328+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.13 scrub starts 2024-02-12T16:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:39.040199+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.13 scrub ok 2024-02-12T16:51:40.400 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234006 got 231928234011 for osd.7 2024-02-12T16:51:40.400 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-12T16:51:40.400 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:51:40.687 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:39.306523+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:51:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:39.307471+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:51:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:39.507440+0000 mon.a (mon.0) 723 : audit [DBG] from='client.? v1:172.21.15.33:0/3082900345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:51:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:39.306523+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:51:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:39.307471+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:51:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:39.507440+0000 mon.a (mon.0) 723 : audit [DBG] from='client.? v1:172.21.15.33:0/3082900345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:51:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:39.306523+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:51:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:39.307471+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:51:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:39.507440+0000 mon.a (mon.0) 723 : audit [DBG] from='client.? v1:172.21.15.33:0/3082900345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:51:41.146 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:40] "GET /metrics HTTP/1.1" 200 172143 "" "Prometheus/2.33.4" 2024-02-12T16:51:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:40.272038+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.a scrub starts 2024-02-12T16:51:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:40.273174+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.a scrub ok 2024-02-12T16:51:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:40.296084+0000 mgr.y (mgr.24481) 82 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:41.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:40.272038+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.a scrub starts 2024-02-12T16:51:41.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:40.273174+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.a scrub ok 2024-02-12T16:51:41.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:40.296084+0000 mgr.y (mgr.24481) 82 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:40.272038+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.a scrub starts 2024-02-12T16:51:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:40.273174+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.a scrub ok 2024-02-12T16:51:41.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:40.296084+0000 mgr.y (mgr.24481) 82 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:42.662 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:51:42.665 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:51:42.683 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:42.037826+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.11 deep-scrub starts 2024-02-12T16:51:42.683 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:42.040824+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.11 deep-scrub ok 2024-02-12T16:51:42.683 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:42.037826+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.11 deep-scrub starts 2024-02-12T16:51:42.683 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:42.040824+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.11 deep-scrub ok 2024-02-12T16:51:42.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:51:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:51:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:42.037826+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.11 deep-scrub starts 2024-02-12T16:51:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:42.040824+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.11 deep-scrub ok 2024-02-12T16:51:43.421 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-02-12T16:51:42.296337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":832,"num_read_kb":584,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":488,"ondisk_log_size":488,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381996,"kb_used_data":4948,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579220,"statfs":{"total":767960285184,"available":765521121280,"internally_reserved":0,"allocated":5066752,"data_stored":2339530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002699"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228411+0000","last_change":"2024-02-12T16:49:45.224168+0000","last_active":"2024-02-12T16:51:22.228411+0000","last_peered":"2024-02-12T16:51:22.228411+0000","last_clean":"2024-02-12T16:51:22.228411+0000","last_became_active":"2024-02-12T16:49:45.223515+0000","last_became_peered":"2024-02-12T16:49:45.223515+0000","last_unstale":"2024-02-12T16:51:22.228411+0000","last_undegraded":"2024-02-12T16:51:22.228411+0000","last_fullsized":"2024-02-12T16:51:22.228411+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:49:44.186028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105779+0000","last_change":"2024-02-12T16:50:51.824034+0000","last_active":"2024-02-12T16:51:22.105779+0000","last_peered":"2024-02-12T16:51:22.105779+0000","last_clean":"2024-02-12T16:51:22.105779+0000","last_became_active":"2024-02-12T16:49:43.195544+0000","last_became_peered":"2024-02-12T16:49:43.195544+0000","last_unstale":"2024-02-12T16:51:22.105779+0000","last_undegraded":"2024-02-12T16:51:22.105779+0000","last_fullsized":"2024-02-12T16:51:22.105779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.823967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107225+0000","last_change":"2024-02-12T16:50:42.254831+0000","last_active":"2024-02-12T16:51:22.107225+0000","last_peered":"2024-02-12T16:51:22.107225+0000","last_clean":"2024-02-12T16:51:22.107225+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:51:22.107225+0000","last_undegraded":"2024-02-12T16:51:22.107225+0000","last_fullsized":"2024-02-12T16:51:22.107225+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:42.254752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.254752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228527+0000","last_change":"2024-02-12T16:51:16.075132+0000","last_active":"2024-02-12T16:51:22.228527+0000","last_peered":"2024-02-12T16:51:22.228527+0000","last_clean":"2024-02-12T16:51:22.228527+0000","last_became_active":"2024-02-12T16:49:41.185805+0000","last_became_peered":"2024-02-12T16:49:41.185805+0000","last_unstale":"2024-02-12T16:51:22.228527+0000","last_undegraded":"2024-02-12T16:51:22.228527+0000","last_fullsized":"2024-02-12T16:51:22.228527+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:51:16.075051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.075051+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105944+0000","last_change":"2024-02-12T16:51:21.897841+0000","last_active":"2024-02-12T16:51:22.105944+0000","last_peered":"2024-02-12T16:51:22.105944+0000","last_clean":"2024-02-12T16:51:22.105944+0000","last_became_active":"2024-02-12T16:49:45.216118+0000","last_became_peered":"2024-02-12T16:49:45.216118+0000","last_unstale":"2024-02-12T16:51:22.105944+0000","last_undegraded":"2024-02-12T16:51:22.105944+0000","last_fullsized":"2024-02-12T16:51:22.105944+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:21.897749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:21.897749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105577+0000","last_change":"2024-02-12T16:51:13.011883+0000","last_active":"2024-02-12T16:51:22.105577+0000","last_peered":"2024-02-12T16:51:22.105577+0000","last_clean":"2024-02-12T16:51:22.105577+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:51:22.105577+0000","last_undegraded":"2024-02-12T16:51:22.105577+0000","last_fullsized":"2024-02-12T16:51:22.105577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.011814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.011814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:38.073505+0000","last_change":"2024-02-12T16:51:38.073505+0000","last_active":"2024-02-12T16:51:38.073505+0000","last_peered":"2024-02-12T16:51:38.073505+0000","last_clean":"2024-02-12T16:51:38.073505+0000","last_became_active":"2024-02-12T16:49:39.102172+0000","last_became_peered":"2024-02-12T16:49:39.102172+0000","last_unstale":"2024-02-12T16:51:38.073505+0000","last_undegraded":"2024-02-12T16:51:38.073505+0000","last_fullsized":"2024-02-12T16:51:38.073505+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.073441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.073441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106003+0000","last_change":"2024-02-12T16:51:17.900217+0000","last_active":"2024-02-12T16:51:22.106003+0000","last_peered":"2024-02-12T16:51:22.106003+0000","last_clean":"2024-02-12T16:51:22.106003+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:51:22.106003+0000","last_undegraded":"2024-02-12T16:51:22.106003+0000","last_fullsized":"2024-02-12T16:51:22.106003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:17.900093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:17.900093+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230508+0000","last_change":"2024-02-12T16:50:47.317845+0000","last_active":"2024-02-12T16:51:22.230508+0000","last_peered":"2024-02-12T16:51:22.230508+0000","last_clean":"2024-02-12T16:51:22.230508+0000","last_became_active":"2024-02-12T16:49:45.218184+0000","last_became_peered":"2024-02-12T16:49:45.218184+0000","last_unstale":"2024-02-12T16:51:22.230508+0000","last_undegraded":"2024-02-12T16:51:22.230508+0000","last_fullsized":"2024-02-12T16:51:22.230508+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:47.317736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.317736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229329+0000","last_change":"2024-02-12T16:50:46.095997+0000","last_active":"2024-02-12T16:51:22.229329+0000","last_peered":"2024-02-12T16:51:22.229329+0000","last_clean":"2024-02-12T16:51:22.229329+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:51:22.229329+0000","last_undegraded":"2024-02-12T16:51:22.229329+0000","last_fullsized":"2024-02-12T16:51:22.229329+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.095924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229791+0000","last_change":"2024-02-12T16:51:04.051422+0000","last_active":"2024-02-12T16:51:22.229791+0000","last_peered":"2024-02-12T16:51:22.229791+0000","last_clean":"2024-02-12T16:51:22.229791+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:51:22.229791+0000","last_undegraded":"2024-02-12T16:51:22.229791+0000","last_fullsized":"2024-02-12T16:51:22.229791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:51:04.051345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":50,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:34.760832+0000","last_change":"2024-02-12T16:51:34.760803+0000","last_active":"2024-02-12T16:51:34.760832+0000","last_peered":"2024-02-12T16:51:34.760832+0000","last_clean":"2024-02-12T16:51:34.760832+0000","last_became_active":"2024-02-12T16:49:41.178018+0000","last_became_peered":"2024-02-12T16:49:41.178018+0000","last_unstale":"2024-02-12T16:51:34.760832+0000","last_undegraded":"2024-02-12T16:51:34.760832+0000","last_fullsized":"2024-02-12T16:51:34.760832+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:51:34.760711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.760711+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107587+0000","last_change":"2024-02-12T16:51:22.020495+0000","last_active":"2024-02-12T16:51:22.107587+0000","last_peered":"2024-02-12T16:51:22.107587+0000","last_clean":"2024-02-12T16:51:22.107587+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:51:22.107587+0000","last_undegraded":"2024-02-12T16:51:22.107587+0000","last_fullsized":"2024-02-12T16:51:22.107587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.020430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.020430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229062+0000","last_change":"2024-02-12T16:50:55.322528+0000","last_active":"2024-02-12T16:51:22.229062+0000","last_peered":"2024-02-12T16:51:22.229062+0000","last_clean":"2024-02-12T16:51:22.229062+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:51:22.229062+0000","last_undegraded":"2024-02-12T16:51:22.229062+0000","last_fullsized":"2024-02-12T16:51:22.229062+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.322446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229425+0000","last_change":"2024-02-12T16:51:01.027931+0000","last_active":"2024-02-12T16:51:22.229425+0000","last_peered":"2024-02-12T16:51:22.229425+0000","last_clean":"2024-02-12T16:51:22.229425+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:51:22.229425+0000","last_undegraded":"2024-02-12T16:51:22.229425+0000","last_fullsized":"2024-02-12T16:51:22.229425+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:01.027841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.027841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106032+0000","last_change":"2024-02-12T16:51:02.857339+0000","last_active":"2024-02-12T16:51:22.106032+0000","last_peered":"2024-02-12T16:51:22.106032+0000","last_clean":"2024-02-12T16:51:22.106032+0000","last_became_active":"2024-02-12T16:49:41.181117+0000","last_became_peered":"2024-02-12T16:49:41.181117+0000","last_unstale":"2024-02-12T16:51:22.106032+0000","last_undegraded":"2024-02-12T16:51:22.106032+0000","last_fullsized":"2024-02-12T16:51:22.106032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:02.857234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.857234+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228670+0000","last_change":"2024-02-12T16:51:03.001426+0000","last_active":"2024-02-12T16:51:22.228670+0000","last_peered":"2024-02-12T16:51:22.228670+0000","last_clean":"2024-02-12T16:51:22.228670+0000","last_became_active":"2024-02-12T16:49:45.221559+0000","last_became_peered":"2024-02-12T16:49:45.221559+0000","last_unstale":"2024-02-12T16:51:22.228670+0000","last_undegraded":"2024-02-12T16:51:22.228670+0000","last_fullsized":"2024-02-12T16:51:22.228670+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:03.001318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.001318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106444+0000","last_change":"2024-02-12T16:50:54.852577+0000","last_active":"2024-02-12T16:51:22.106444+0000","last_peered":"2024-02-12T16:51:22.106444+0000","last_clean":"2024-02-12T16:51:22.106444+0000","last_became_active":"2024-02-12T16:49:43.201709+0000","last_became_peered":"2024-02-12T16:49:43.201709+0000","last_unstale":"2024-02-12T16:51:22.106444+0000","last_undegraded":"2024-02-12T16:51:22.106444+0000","last_fullsized":"2024-02-12T16:51:22.106444+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:54.852481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.852481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:29.066207+0000","last_change":"2024-02-12T16:51:29.066176+0000","last_active":"2024-02-12T16:51:29.066207+0000","last_peered":"2024-02-12T16:51:29.066207+0000","last_clean":"2024-02-12T16:51:29.066207+0000","last_became_active":"2024-02-12T16:49:39.105438+0000","last_became_peered":"2024-02-12T16:49:39.105438+0000","last_unstale":"2024-02-12T16:51:29.066207+0000","last_undegraded":"2024-02-12T16:51:29.066207+0000","last_fullsized":"2024-02-12T16:51:29.066207+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:29.066107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:29.066107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:23.148968+0000","last_change":"2024-02-12T16:51:23.148923+0000","last_active":"2024-02-12T16:51:23.148968+0000","last_peered":"2024-02-12T16:51:23.148968+0000","last_clean":"2024-02-12T16:51:23.148968+0000","last_became_active":"2024-02-12T16:49:41.181980+0000","last_became_peered":"2024-02-12T16:49:41.181980+0000","last_unstale":"2024-02-12T16:51:23.148968+0000","last_undegraded":"2024-02-12T16:51:23.148968+0000","last_fullsized":"2024-02-12T16:51:23.148968+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:51:23.148783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:23.148783+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106834+0000","last_change":"2024-02-12T16:51:10.898681+0000","last_active":"2024-02-12T16:51:22.106834+0000","last_peered":"2024-02-12T16:51:22.106834+0000","last_clean":"2024-02-12T16:51:22.106834+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:51:22.106834+0000","last_undegraded":"2024-02-12T16:51:22.106834+0000","last_fullsized":"2024-02-12T16:51:22.106834+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:51:10.898594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106469+0000","last_change":"2024-02-12T16:51:20.972423+0000","last_active":"2024-02-12T16:51:22.106469+0000","last_peered":"2024-02-12T16:51:22.106469+0000","last_clean":"2024-02-12T16:51:22.106469+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:51:22.106469+0000","last_undegraded":"2024-02-12T16:51:22.106469+0000","last_fullsized":"2024-02-12T16:51:22.106469+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.972298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.972298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230677+0000","last_change":"2024-02-12T16:50:40.782894+0000","last_active":"2024-02-12T16:51:22.230677+0000","last_peered":"2024-02-12T16:51:22.230677+0000","last_clean":"2024-02-12T16:51:22.230677+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:51:22.230677+0000","last_undegraded":"2024-02-12T16:51:22.230677+0000","last_fullsized":"2024-02-12T16:51:22.230677+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:40.782823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:40.782823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128597+0000","last_change":"2024-02-12T16:50:47.282788+0000","last_active":"2024-02-12T16:51:22.128597+0000","last_peered":"2024-02-12T16:51:22.128597+0000","last_clean":"2024-02-12T16:51:22.128597+0000","last_became_active":"2024-02-12T16:49:41.180319+0000","last_became_peered":"2024-02-12T16:49:41.180319+0000","last_unstale":"2024-02-12T16:51:22.128597+0000","last_undegraded":"2024-02-12T16:51:22.128597+0000","last_fullsized":"2024-02-12T16:51:22.128597+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:50:47.282711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.282711+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106383+0000","last_change":"2024-02-12T16:51:08.888476+0000","last_active":"2024-02-12T16:51:22.106383+0000","last_peered":"2024-02-12T16:51:22.106383+0000","last_clean":"2024-02-12T16:51:22.106383+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:51:22.106383+0000","last_undegraded":"2024-02-12T16:51:22.106383+0000","last_fullsized":"2024-02-12T16:51:22.106383+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:08.888390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.888390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107197+0000","last_change":"2024-02-12T16:50:46.144078+0000","last_active":"2024-02-12T16:51:22.107197+0000","last_peered":"2024-02-12T16:51:22.107197+0000","last_clean":"2024-02-12T16:51:22.107197+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:51:22.107197+0000","last_undegraded":"2024-02-12T16:51:22.107197+0000","last_fullsized":"2024-02-12T16:51:22.107197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.144010+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228279+0000","last_change":"2024-02-12T16:51:09.094169+0000","last_active":"2024-02-12T16:51:22.228279+0000","last_peered":"2024-02-12T16:51:22.228279+0000","last_clean":"2024-02-12T16:51:22.228279+0000","last_became_active":"2024-02-12T16:49:39.101492+0000","last_became_peered":"2024-02-12T16:49:39.101492+0000","last_unstale":"2024-02-12T16:51:22.228279+0000","last_undegraded":"2024-02-12T16:51:22.228279+0000","last_fullsized":"2024-02-12T16:51:22.228279+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:09.094067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:09.094067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229886+0000","last_change":"2024-02-12T16:51:14.042938+0000","last_active":"2024-02-12T16:51:22.229886+0000","last_peered":"2024-02-12T16:51:22.229886+0000","last_clean":"2024-02-12T16:51:22.229886+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:51:22.229886+0000","last_undegraded":"2024-02-12T16:51:22.229886+0000","last_fullsized":"2024-02-12T16:51:22.229886+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:14.042858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:14.042858+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:28.104077+0000","last_change":"2024-02-12T16:51:28.104047+0000","last_active":"2024-02-12T16:51:28.104077+0000","last_peered":"2024-02-12T16:51:28.104077+0000","last_clean":"2024-02-12T16:51:28.104077+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:51:28.104077+0000","last_undegraded":"2024-02-12T16:51:28.104077+0000","last_fullsized":"2024-02-12T16:51:28.104077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.103969+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:36.203567+0000","last_change":"2024-02-12T16:51:08.073509+0000","last_active":"2024-02-12T16:51:36.203567+0000","last_peered":"2024-02-12T16:51:36.203567+0000","last_clean":"2024-02-12T16:51:36.203567+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:51:36.203567+0000","last_undegraded":"2024-02-12T16:51:36.203567+0000","last_fullsized":"2024-02-12T16:51:36.203567+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:08.073428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.073428+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228434+0000","last_change":"2024-02-12T16:51:01.224070+0000","last_active":"2024-02-12T16:51:22.228434+0000","last_peered":"2024-02-12T16:51:22.228434+0000","last_clean":"2024-02-12T16:51:22.228434+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:51:22.228434+0000","last_undegraded":"2024-02-12T16:51:22.228434+0000","last_fullsized":"2024-02-12T16:51:22.228434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:01.223973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.223973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107152+0000","last_change":"2024-02-12T16:50:44.211440+0000","last_active":"2024-02-12T16:51:22.107152+0000","last_peered":"2024-02-12T16:51:22.107152+0000","last_clean":"2024-02-12T16:51:22.107152+0000","last_became_active":"2024-02-12T16:49:41.669589+0000","last_became_peered":"2024-02-12T16:49:41.669589+0000","last_unstale":"2024-02-12T16:51:22.107152+0000","last_undegraded":"2024-02-12T16:51:22.107152+0000","last_fullsized":"2024-02-12T16:51:22.107152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:50:44.211369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:44.211369+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228863+0000","last_change":"2024-02-12T16:49:45.222705+0000","last_active":"2024-02-12T16:51:22.228863+0000","last_peered":"2024-02-12T16:51:22.228863+0000","last_clean":"2024-02-12T16:51:22.228863+0000","last_became_active":"2024-02-12T16:49:45.222341+0000","last_became_peered":"2024-02-12T16:49:45.222341+0000","last_unstale":"2024-02-12T16:51:22.228863+0000","last_undegraded":"2024-02-12T16:51:22.228863+0000","last_fullsized":"2024-02-12T16:51:22.228863+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:49:44.186028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229028+0000","last_change":"2024-02-12T16:51:19.038453+0000","last_active":"2024-02-12T16:51:22.229028+0000","last_peered":"2024-02-12T16:51:22.229028+0000","last_clean":"2024-02-12T16:51:22.229028+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:51:22.229028+0000","last_undegraded":"2024-02-12T16:51:22.229028+0000","last_fullsized":"2024-02-12T16:51:22.229028+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:19.038388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:19.038388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230171+0000","last_change":"2024-02-12T16:51:18.026686+0000","last_active":"2024-02-12T16:51:22.230171+0000","last_peered":"2024-02-12T16:51:22.230171+0000","last_clean":"2024-02-12T16:51:22.230171+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:51:22.230171+0000","last_undegraded":"2024-02-12T16:51:22.230171+0000","last_fullsized":"2024-02-12T16:51:22.230171+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:18.026624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.026624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228467+0000","last_change":"2024-02-12T16:50:51.908417+0000","last_active":"2024-02-12T16:51:22.228467+0000","last_peered":"2024-02-12T16:51:22.228467+0000","last_clean":"2024-02-12T16:51:22.228467+0000","last_became_active":"2024-02-12T16:49:41.669704+0000","last_became_peered":"2024-02-12T16:49:41.669704+0000","last_unstale":"2024-02-12T16:51:22.228467+0000","last_undegraded":"2024-02-12T16:51:22.228467+0000","last_fullsized":"2024-02-12T16:51:22.228467+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:50:51.908326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.908326+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228893+0000","last_change":"2024-02-12T16:51:15.965316+0000","last_active":"2024-02-12T16:51:22.228893+0000","last_peered":"2024-02-12T16:51:22.228893+0000","last_clean":"2024-02-12T16:51:22.228893+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:51:22.228893+0000","last_undegraded":"2024-02-12T16:51:22.228893+0000","last_fullsized":"2024-02-12T16:51:22.228893+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:51:15.965186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228597+0000","last_change":"2024-02-12T16:50:55.941652+0000","last_active":"2024-02-12T16:51:22.228597+0000","last_peered":"2024-02-12T16:51:22.228597+0000","last_clean":"2024-02-12T16:51:22.228597+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:51:22.228597+0000","last_undegraded":"2024-02-12T16:51:22.228597+0000","last_fullsized":"2024-02-12T16:51:22.228597+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.941575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.941575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229439+0000","last_change":"2024-02-12T16:51:11.135365+0000","last_active":"2024-02-12T16:51:22.229439+0000","last_peered":"2024-02-12T16:51:22.229439+0000","last_clean":"2024-02-12T16:51:22.229439+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:51:22.229439+0000","last_undegraded":"2024-02-12T16:51:22.229439+0000","last_fullsized":"2024-02-12T16:51:22.229439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.135261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228389+0000","last_change":"2024-02-12T16:49:41.184402+0000","last_active":"2024-02-12T16:51:22.228389+0000","last_peered":"2024-02-12T16:51:22.228389+0000","last_clean":"2024-02-12T16:51:22.228389+0000","last_became_active":"2024-02-12T16:49:41.184044+0000","last_became_peered":"2024-02-12T16:49:41.184044+0000","last_unstale":"2024-02-12T16:51:22.228389+0000","last_undegraded":"2024-02-12T16:51:22.228389+0000","last_fullsized":"2024-02-12T16:51:22.228389+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:49:40.156223+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228190+0000","last_change":"2024-02-12T16:49:45.221002+0000","last_active":"2024-02-12T16:51:22.228190+0000","last_peered":"2024-02-12T16:51:22.228190+0000","last_clean":"2024-02-12T16:51:22.228190+0000","last_became_active":"2024-02-12T16:49:45.220469+0000","last_became_peered":"2024-02-12T16:49:45.220469+0000","last_unstale":"2024-02-12T16:51:22.228190+0000","last_undegraded":"2024-02-12T16:51:22.228190+0000","last_fullsized":"2024-02-12T16:51:22.228190+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:49:44.186028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106088+0000","last_change":"2024-02-12T16:50:57.875954+0000","last_active":"2024-02-12T16:51:22.106088+0000","last_peered":"2024-02-12T16:51:22.106088+0000","last_clean":"2024-02-12T16:51:22.106088+0000","last_became_active":"2024-02-12T16:49:43.204615+0000","last_became_peered":"2024-02-12T16:49:43.204615+0000","last_unstale":"2024-02-12T16:51:22.106088+0000","last_undegraded":"2024-02-12T16:51:22.106088+0000","last_fullsized":"2024-02-12T16:51:22.106088+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:57.875871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.875871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106525+0000","last_change":"2024-02-12T16:50:46.810493+0000","last_active":"2024-02-12T16:51:22.106525+0000","last_peered":"2024-02-12T16:51:22.106525+0000","last_clean":"2024-02-12T16:51:22.106525+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:51:22.106525+0000","last_undegraded":"2024-02-12T16:51:22.106525+0000","last_fullsized":"2024-02-12T16:51:22.106525+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.810420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.810420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106255+0000","last_change":"2024-02-12T16:51:14.844224+0000","last_active":"2024-02-12T16:51:22.106255+0000","last_peered":"2024-02-12T16:51:22.106255+0000","last_clean":"2024-02-12T16:51:22.106255+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:51:22.106255+0000","last_undegraded":"2024-02-12T16:51:22.106255+0000","last_fullsized":"2024-02-12T16:51:22.106255+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:14.844115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:14.844115+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:28.886428+0000","last_change":"2024-02-12T16:51:28.886399+0000","last_active":"2024-02-12T16:51:28.886428+0000","last_peered":"2024-02-12T16:51:28.886428+0000","last_clean":"2024-02-12T16:51:28.886428+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:51:28.886428+0000","last_undegraded":"2024-02-12T16:51:28.886428+0000","last_fullsized":"2024-02-12T16:51:28.886428+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.886321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228753+0000","last_change":"2024-02-12T16:49:43.208669+0000","last_active":"2024-02-12T16:51:22.228753+0000","last_peered":"2024-02-12T16:51:22.228753+0000","last_clean":"2024-02-12T16:51:22.228753+0000","last_became_active":"2024-02-12T16:49:43.208343+0000","last_became_peered":"2024-02-12T16:49:43.208343+0000","last_unstale":"2024-02-12T16:51:22.228753+0000","last_undegraded":"2024-02-12T16:51:22.228753+0000","last_fullsized":"2024-02-12T16:51:22.228753+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:49:42.168508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229419+0000","last_change":"2024-02-12T16:51:11.088535+0000","last_active":"2024-02-12T16:51:22.229419+0000","last_peered":"2024-02-12T16:51:22.229419+0000","last_clean":"2024-02-12T16:51:22.229419+0000","last_became_active":"2024-02-12T16:49:39.107471+0000","last_became_peered":"2024-02-12T16:49:39.107471+0000","last_unstale":"2024-02-12T16:51:22.229419+0000","last_undegraded":"2024-02-12T16:51:22.229419+0000","last_fullsized":"2024-02-12T16:51:22.229419+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:11.088412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.088412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106003+0000","last_change":"2024-02-12T16:51:01.759881+0000","last_active":"2024-02-12T16:51:22.106003+0000","last_peered":"2024-02-12T16:51:22.106003+0000","last_clean":"2024-02-12T16:51:22.106003+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:51:22.106003+0000","last_undegraded":"2024-02-12T16:51:22.106003+0000","last_fullsized":"2024-02-12T16:51:22.106003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.759802+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:24.897271+0000","last_change":"2024-02-12T16:51:24.897238+0000","last_active":"2024-02-12T16:51:24.897271+0000","last_peered":"2024-02-12T16:51:24.897271+0000","last_clean":"2024-02-12T16:51:24.897271+0000","last_became_active":"2024-02-12T16:49:45.217958+0000","last_became_peered":"2024-02-12T16:49:45.217958+0000","last_unstale":"2024-02-12T16:51:24.897271+0000","last_undegraded":"2024-02-12T16:51:24.897271+0000","last_fullsized":"2024-02-12T16:51:24.897271+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.897149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.897149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:35.746551+0000","last_change":"2024-02-12T16:49:43.206493+0000","last_active":"2024-02-12T16:51:35.746551+0000","last_peered":"2024-02-12T16:51:35.746551+0000","last_clean":"2024-02-12T16:51:35.746551+0000","last_became_active":"2024-02-12T16:49:43.206068+0000","last_became_peered":"2024-02-12T16:49:43.206068+0000","last_unstale":"2024-02-12T16:51:35.746551+0000","last_undegraded":"2024-02-12T16:51:35.746551+0000","last_fullsized":"2024-02-12T16:51:35.746551+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:49:42.168508+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:25.913239+0000","last_change":"2024-02-12T16:51:25.913214+0000","last_active":"2024-02-12T16:51:25.913239+0000","last_peered":"2024-02-12T16:51:25.913239+0000","last_clean":"2024-02-12T16:51:25.913239+0000","last_became_active":"2024-02-12T16:49:39.103401+0000","last_became_peered":"2024-02-12T16:49:39.103401+0000","last_unstale":"2024-02-12T16:51:25.913239+0000","last_undegraded":"2024-02-12T16:51:25.913239+0000","last_fullsized":"2024-02-12T16:51:25.913239+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:25.913145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.913145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:25.028457+0000","last_change":"2024-02-12T16:51:25.028420+0000","last_active":"2024-02-12T16:51:25.028457+0000","last_peered":"2024-02-12T16:51:25.028457+0000","last_clean":"2024-02-12T16:51:25.028457+0000","last_became_active":"2024-02-12T16:49:41.669427+0000","last_became_peered":"2024-02-12T16:49:41.669427+0000","last_unstale":"2024-02-12T16:51:25.028457+0000","last_undegraded":"2024-02-12T16:51:25.028457+0000","last_fullsized":"2024-02-12T16:51:25.028457+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:25.028339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.028339+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106992+0000","last_change":"2024-02-12T16:50:49.214010+0000","last_active":"2024-02-12T16:51:22.106992+0000","last_peered":"2024-02-12T16:51:22.106992+0000","last_clean":"2024-02-12T16:51:22.106992+0000","last_became_active":"2024-02-12T16:49:45.212245+0000","last_became_peered":"2024-02-12T16:49:45.212245+0000","last_unstale":"2024-02-12T16:51:22.106992+0000","last_undegraded":"2024-02-12T16:51:22.106992+0000","last_fullsized":"2024-02-12T16:51:22.106992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:50:49.213945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:36.186031+0000","last_change":"2024-02-12T16:51:09.159161+0000","last_active":"2024-02-12T16:51:36.186031+0000","last_peered":"2024-02-12T16:51:36.186031+0000","last_clean":"2024-02-12T16:51:36.186031+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:51:36.186031+0000","last_undegraded":"2024-02-12T16:51:36.186031+0000","last_fullsized":"2024-02-12T16:51:36.186031+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:09.159070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:09.159070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230070+0000","last_change":"2024-02-12T16:51:20.103380+0000","last_active":"2024-02-12T16:51:22.230070+0000","last_peered":"2024-02-12T16:51:22.230070+0000","last_clean":"2024-02-12T16:51:22.230070+0000","last_became_active":"2024-02-12T16:49:39.104936+0000","last_became_peered":"2024-02-12T16:49:39.104936+0000","last_unstale":"2024-02-12T16:51:22.230070+0000","last_undegraded":"2024-02-12T16:51:22.230070+0000","last_fullsized":"2024-02-12T16:51:22.230070+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.103297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.103297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:36.026655+0000","last_change":"2024-02-12T16:51:36.026655+0000","last_active":"2024-02-12T16:51:36.026655+0000","last_peered":"2024-02-12T16:51:36.026655+0000","last_clean":"2024-02-12T16:51:36.026655+0000","last_became_active":"2024-02-12T16:51:04.116639+0000","last_became_peered":"2024-02-12T16:51:04.116639+0000","last_unstale":"2024-02-12T16:51:36.026655+0000","last_undegraded":"2024-02-12T16:51:36.026655+0000","last_fullsized":"2024-02-12T16:51:36.026655+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:36.026569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:36.026569+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229707+0000","last_change":"2024-02-12T16:51:20.041621+0000","last_active":"2024-02-12T16:51:22.229707+0000","last_peered":"2024-02-12T16:51:22.229707+0000","last_clean":"2024-02-12T16:51:22.229707+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:51:22.229707+0000","last_undegraded":"2024-02-12T16:51:22.229707+0000","last_fullsized":"2024-02-12T16:51:22.229707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.041556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.041556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229369+0000","last_change":"2024-02-12T16:50:45.275564+0000","last_active":"2024-02-12T16:51:22.229369+0000","last_peered":"2024-02-12T16:51:22.229369+0000","last_clean":"2024-02-12T16:51:22.229369+0000","last_became_active":"2024-02-12T16:49:43.201595+0000","last_became_peered":"2024-02-12T16:49:43.201595+0000","last_unstale":"2024-02-12T16:51:22.229369+0000","last_undegraded":"2024-02-12T16:51:22.229369+0000","last_fullsized":"2024-02-12T16:51:22.229369+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:45.275469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:45.275469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228684+0000","last_change":"2024-02-12T16:51:19.027311+0000","last_active":"2024-02-12T16:51:22.228684+0000","last_peered":"2024-02-12T16:51:22.228684+0000","last_clean":"2024-02-12T16:51:22.228684+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:51:22.228684+0000","last_undegraded":"2024-02-12T16:51:22.228684+0000","last_fullsized":"2024-02-12T16:51:22.228684+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:19.027246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:19.027246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":50,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:28.108130+0000","last_change":"2024-02-12T16:51:28.108097+0000","last_active":"2024-02-12T16:51:28.108130+0000","last_peered":"2024-02-12T16:51:28.108130+0000","last_clean":"2024-02-12T16:51:28.108130+0000","last_became_active":"2024-02-12T16:49:41.669651+0000","last_became_peered":"2024-02-12T16:49:41.669651+0000","last_unstale":"2024-02-12T16:51:28.108130+0000","last_undegraded":"2024-02-12T16:51:28.108130+0000","last_fullsized":"2024-02-12T16:51:28.108130+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:51:28.108036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.108036+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128417+0000","last_change":"2024-02-12T16:51:03.235610+0000","last_active":"2024-02-12T16:51:22.128417+0000","last_peered":"2024-02-12T16:51:22.128417+0000","last_clean":"2024-02-12T16:51:22.128417+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:51:22.128417+0000","last_undegraded":"2024-02-12T16:51:22.128417+0000","last_fullsized":"2024-02-12T16:51:22.128417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:03.235544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.235544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:32.014878+0000","last_change":"2024-02-12T16:51:32.014849+0000","last_active":"2024-02-12T16:51:32.014878+0000","last_peered":"2024-02-12T16:51:32.014878+0000","last_clean":"2024-02-12T16:51:32.014878+0000","last_became_active":"2024-02-12T16:49:43.207454+0000","last_became_peered":"2024-02-12T16:49:43.207454+0000","last_unstale":"2024-02-12T16:51:32.014878+0000","last_undegraded":"2024-02-12T16:51:32.014878+0000","last_fullsized":"2024-02-12T16:51:32.014878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.014772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128245+0000","last_change":"2024-02-12T16:49:39.106642+0000","last_active":"2024-02-12T16:51:22.128245+0000","last_peered":"2024-02-12T16:51:22.128245+0000","last_clean":"2024-02-12T16:51:22.128245+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:51:22.128245+0000","last_undegraded":"2024-02-12T16:51:22.128245+0000","last_fullsized":"2024-02-12T16:51:22.128245+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:49:38.079400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107066+0000","last_change":"2024-02-12T16:51:17.050140+0000","last_active":"2024-02-12T16:51:22.107066+0000","last_peered":"2024-02-12T16:51:22.107066+0000","last_clean":"2024-02-12T16:51:22.107066+0000","last_became_active":"2024-02-12T16:49:41.668665+0000","last_became_peered":"2024-02-12T16:49:41.668665+0000","last_unstale":"2024-02-12T16:51:22.107066+0000","last_undegraded":"2024-02-12T16:51:22.107066+0000","last_fullsized":"2024-02-12T16:51:22.107066+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:51:17.050077+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:33.032908+0000","last_change":"2024-02-12T16:51:33.032870+0000","last_active":"2024-02-12T16:51:33.032908+0000","last_peered":"2024-02-12T16:51:33.032908+0000","last_clean":"2024-02-12T16:51:33.032908+0000","last_became_active":"2024-02-12T16:49:45.220328+0000","last_became_peered":"2024-02-12T16:49:45.220328+0000","last_unstale":"2024-02-12T16:51:33.032908+0000","last_undegraded":"2024-02-12T16:51:33.032908+0000","last_fullsized":"2024-02-12T16:51:33.032908+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:33.032784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:33.032784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"58'2","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106489+0000","last_change":"2024-02-12T16:50:42.866331+0000","last_active":"2024-02-12T16:51:22.106489+0000","last_peered":"2024-02-12T16:51:22.106489+0000","last_clean":"2024-02-12T16:51:22.106489+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:51:22.106489+0000","last_undegraded":"2024-02-12T16:51:22.106489+0000","last_fullsized":"2024-02-12T16:51:22.106489+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-02-12T16:50:42.866263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.866263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106048+0000","last_change":"2024-02-12T16:50:46.750203+0000","last_active":"2024-02-12T16:51:22.106048+0000","last_peered":"2024-02-12T16:51:22.106048+0000","last_clean":"2024-02-12T16:51:22.106048+0000","last_became_active":"2024-02-12T16:49:41.177967+0000","last_became_peered":"2024-02-12T16:49:41.177967+0000","last_unstale":"2024-02-12T16:51:22.106048+0000","last_undegraded":"2024-02-12T16:51:22.106048+0000","last_fullsized":"2024-02-12T16:51:22.106048+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:50:46.750065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.750065+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128115+0000","last_change":"2024-02-12T16:51:22.127898+0000","last_active":"2024-02-12T16:51:22.128115+0000","last_peered":"2024-02-12T16:51:22.128115+0000","last_clean":"2024-02-12T16:51:22.128115+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:51:22.128115+0000","last_undegraded":"2024-02-12T16:51:22.128115+0000","last_fullsized":"2024-02-12T16:51:22.128115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.127831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107390+0000","last_change":"2024-02-12T16:50:50.186122+0000","last_active":"2024-02-12T16:51:22.107390+0000","last_peered":"2024-02-12T16:51:22.107390+0000","last_clean":"2024-02-12T16:51:22.107390+0000","last_became_active":"2024-02-12T16:49:45.217626+0000","last_became_peered":"2024-02-12T16:49:45.217626+0000","last_unstale":"2024-02-12T16:51:22.107390+0000","last_undegraded":"2024-02-12T16:51:22.107390+0000","last_fullsized":"2024-02-12T16:51:22.107390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:50.186021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.186021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:23.967951+0000","last_change":"2024-02-12T16:51:23.967921+0000","last_active":"2024-02-12T16:51:23.967951+0000","last_peered":"2024-02-12T16:51:23.967951+0000","last_clean":"2024-02-12T16:51:23.967951+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:51:23.967951+0000","last_undegraded":"2024-02-12T16:51:23.967951+0000","last_fullsized":"2024-02-12T16:51:23.967951+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:23.967848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:23.967848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128588+0000","last_change":"2024-02-12T16:51:04.121170+0000","last_active":"2024-02-12T16:51:22.128588+0000","last_peered":"2024-02-12T16:51:22.128588+0000","last_clean":"2024-02-12T16:51:22.128588+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:51:22.128588+0000","last_undegraded":"2024-02-12T16:51:22.128588+0000","last_fullsized":"2024-02-12T16:51:22.128588+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:50:42.876140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.876140+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:35.385580+0000","last_change":"2024-02-12T16:51:11.206549+0000","last_active":"2024-02-12T16:51:35.385580+0000","last_peered":"2024-02-12T16:51:35.385580+0000","last_clean":"2024-02-12T16:51:35.385580+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:51:35.385580+0000","last_undegraded":"2024-02-12T16:51:35.385580+0000","last_fullsized":"2024-02-12T16:51:35.385580+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:11.206420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.206420+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:26.163668+0000","last_change":"2024-02-12T16:51:26.163640+0000","last_active":"2024-02-12T16:51:26.163668+0000","last_peered":"2024-02-12T16:51:26.163668+0000","last_clean":"2024-02-12T16:51:26.163668+0000","last_became_active":"2024-02-12T16:49:45.215520+0000","last_became_peered":"2024-02-12T16:49:45.215520+0000","last_unstale":"2024-02-12T16:51:26.163668+0000","last_undegraded":"2024-02-12T16:51:26.163668+0000","last_fullsized":"2024-02-12T16:51:26.163668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:26.163568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:26.163568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230141+0000","last_change":"2024-02-12T16:50:50.253026+0000","last_active":"2024-02-12T16:51:22.230141+0000","last_peered":"2024-02-12T16:51:22.230141+0000","last_clean":"2024-02-12T16:51:22.230141+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:51:22.230141+0000","last_undegraded":"2024-02-12T16:51:22.230141+0000","last_fullsized":"2024-02-12T16:51:22.230141+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:50.252952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.252952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":114,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.878869+0000","last_change":"2024-02-12T16:51:37.878869+0000","last_active":"2024-02-12T16:51:37.878869+0000","last_peered":"2024-02-12T16:51:37.878869+0000","last_clean":"2024-02-12T16:51:37.878869+0000","last_became_active":"2024-02-12T16:49:41.179610+0000","last_became_peered":"2024-02-12T16:49:41.179610+0000","last_unstale":"2024-02-12T16:51:37.878869+0000","last_undegraded":"2024-02-12T16:51:37.878869+0000","last_fullsized":"2024-02-12T16:51:37.878869+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-12T16:51:37.878788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.878788+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106869+0000","last_change":"2024-02-12T16:50:57.182750+0000","last_active":"2024-02-12T16:51:22.106869+0000","last_peered":"2024-02-12T16:51:22.106869+0000","last_clean":"2024-02-12T16:51:22.106869+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:51:22.106869+0000","last_undegraded":"2024-02-12T16:51:22.106869+0000","last_fullsized":"2024-02-12T16:51:22.106869+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:57.182679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.182679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229988+0000","last_change":"2024-02-12T16:51:22.062228+0000","last_active":"2024-02-12T16:51:22.229988+0000","last_peered":"2024-02-12T16:51:22.229988+0000","last_clean":"2024-02-12T16:51:22.229988+0000","last_became_active":"2024-02-12T16:49:45.221460+0000","last_became_peered":"2024-02-12T16:49:45.221460+0000","last_unstale":"2024-02-12T16:51:22.229988+0000","last_undegraded":"2024-02-12T16:51:22.229988+0000","last_fullsized":"2024-02-12T16:51:22.229988+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.062124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.062124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":159,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:36.516659+0000","last_change":"2024-02-12T16:50:49.760225+0000","last_active":"2024-02-12T16:51:36.516659+0000","last_peered":"2024-02-12T16:51:36.516659+0000","last_clean":"2024-02-12T16:51:36.516659+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:51:36.516659+0000","last_undegraded":"2024-02-12T16:51:36.516659+0000","last_fullsized":"2024-02-12T16:51:36.516659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:50:49.760157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:49.760157+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128334+0000","last_change":"2024-02-12T16:51:02.278460+0000","last_active":"2024-02-12T16:51:22.128334+0000","last_peered":"2024-02-12T16:51:22.128334+0000","last_clean":"2024-02-12T16:51:22.128334+0000","last_became_active":"2024-02-12T16:49:39.105685+0000","last_became_peered":"2024-02-12T16:49:39.105685+0000","last_unstale":"2024-02-12T16:51:22.128334+0000","last_undegraded":"2024-02-12T16:51:22.128334+0000","last_fullsized":"2024-02-12T16:51:22.128334+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.278337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.278337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106958+0000","last_change":"2024-02-12T16:51:18.950103+0000","last_active":"2024-02-12T16:51:22.106958+0000","last_peered":"2024-02-12T16:51:22.106958+0000","last_clean":"2024-02-12T16:51:22.106958+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:51:22.106958+0000","last_undegraded":"2024-02-12T16:51:22.106958+0000","last_fullsized":"2024-02-12T16:51:22.106958+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:51:18.950031+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:51:18.950031+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.950031+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:24.083427+0000","last_change":"2024-02-12T16:51:24.083385+0000","last_active":"2024-02-12T16:51:24.083427+0000","last_peered":"2024-02-12T16:51:24.083427+0000","last_clean":"2024-02-12T16:51:24.083427+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:51:24.083427+0000","last_undegraded":"2024-02-12T16:51:24.083427+0000","last_fullsized":"2024-02-12T16:51:24.083427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.083270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.083270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230793+0000","last_change":"2024-02-12T16:51:00.237219+0000","last_active":"2024-02-12T16:51:22.230793+0000","last_peered":"2024-02-12T16:51:22.230793+0000","last_clean":"2024-02-12T16:51:22.230793+0000","last_became_active":"2024-02-12T16:49:45.213152+0000","last_became_peered":"2024-02-12T16:49:45.213152+0000","last_unstale":"2024-02-12T16:51:22.230793+0000","last_undegraded":"2024-02-12T16:51:22.230793+0000","last_fullsized":"2024-02-12T16:51:22.230793+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:51:00.237119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228271+0000","last_change":"2024-02-12T16:51:08.054217+0000","last_active":"2024-02-12T16:51:22.228271+0000","last_peered":"2024-02-12T16:51:22.228271+0000","last_clean":"2024-02-12T16:51:22.228271+0000","last_became_active":"2024-02-12T16:49:39.105761+0000","last_became_peered":"2024-02-12T16:49:39.105761+0000","last_unstale":"2024-02-12T16:51:22.228271+0000","last_undegraded":"2024-02-12T16:51:22.228271+0000","last_fullsized":"2024-02-12T16:51:22.228271+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:08.054114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.054114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228987+0000","last_change":"2024-02-12T16:50:56.320321+0000","last_active":"2024-02-12T16:51:22.228987+0000","last_peered":"2024-02-12T16:51:22.228987+0000","last_clean":"2024-02-12T16:51:22.228987+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:51:22.228987+0000","last_undegraded":"2024-02-12T16:51:22.228987+0000","last_fullsized":"2024-02-12T16:51:22.228987+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:56.320240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:56.320240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.184984+0000","last_change":"2024-02-12T16:51:37.184984+0000","last_active":"2024-02-12T16:51:37.184984+0000","last_peered":"2024-02-12T16:51:37.184984+0000","last_clean":"2024-02-12T16:51:37.184984+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:51:37.184984+0000","last_undegraded":"2024-02-12T16:51:37.184984+0000","last_fullsized":"2024-02-12T16:51:37.184984+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:37.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.184907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230258+0000","last_change":"2024-02-12T16:50:57.964701+0000","last_active":"2024-02-12T16:51:22.230258+0000","last_peered":"2024-02-12T16:51:22.230258+0000","last_clean":"2024-02-12T16:51:22.230258+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:51:22.230258+0000","last_undegraded":"2024-02-12T16:51:22.230258+0000","last_fullsized":"2024-02-12T16:51:22.230258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:50:57.964625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.964625+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229610+0000","last_change":"2024-02-12T16:51:05.047096+0000","last_active":"2024-02-12T16:51:22.229610+0000","last_peered":"2024-02-12T16:51:22.229610+0000","last_clean":"2024-02-12T16:51:22.229610+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:51:22.229610+0000","last_undegraded":"2024-02-12T16:51:22.229610+0000","last_fullsized":"2024-02-12T16:51:22.229610+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:05.047021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:05.047021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:36.842712+0000","last_change":"2024-02-12T16:51:36.842712+0000","last_active":"2024-02-12T16:51:36.842712+0000","last_peered":"2024-02-12T16:51:36.842712+0000","last_clean":"2024-02-12T16:51:36.842712+0000","last_became_active":"2024-02-12T16:49:45.218096+0000","last_became_peered":"2024-02-12T16:49:45.218096+0000","last_unstale":"2024-02-12T16:51:36.842712+0000","last_undegraded":"2024-02-12T16:51:36.842712+0000","last_fullsized":"2024-02-12T16:51:36.842712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:36.842645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:36.842645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106219+0000","last_change":"2024-02-12T16:51:18.079605+0000","last_active":"2024-02-12T16:51:22.106219+0000","last_peered":"2024-02-12T16:51:22.106219+0000","last_clean":"2024-02-12T16:51:22.106219+0000","last_became_active":"2024-02-12T16:49:39.104017+0000","last_became_peered":"2024-02-12T16:49:39.104017+0000","last_unstale":"2024-02-12T16:51:22.106219+0000","last_undegraded":"2024-02-12T16:51:22.106219+0000","last_fullsized":"2024-02-12T16:51:22.106219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:18.079497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.079497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228728+0000","last_change":"2024-02-12T16:51:08.953016+0000","last_active":"2024-02-12T16:51:22.228728+0000","last_peered":"2024-02-12T16:51:22.228728+0000","last_clean":"2024-02-12T16:51:22.228728+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:51:22.228728+0000","last_undegraded":"2024-02-12T16:51:22.228728+0000","last_fullsized":"2024-02-12T16:51:22.228728+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:51:08.952901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.952901+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107040+0000","last_change":"2024-02-12T16:51:07.240302+0000","last_active":"2024-02-12T16:51:22.107040+0000","last_peered":"2024-02-12T16:51:22.107040+0000","last_clean":"2024-02-12T16:51:22.107040+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:51:22.107040+0000","last_undegraded":"2024-02-12T16:51:22.107040+0000","last_fullsized":"2024-02-12T16:51:22.107040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:51:07.240236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:07.240236+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230709+0000","last_change":"2024-02-12T16:51:22.070134+0000","last_active":"2024-02-12T16:51:22.230709+0000","last_peered":"2024-02-12T16:51:22.230709+0000","last_clean":"2024-02-12T16:51:22.230709+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:51:22.230709+0000","last_undegraded":"2024-02-12T16:51:22.230709+0000","last_fullsized":"2024-02-12T16:51:22.230709+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.070056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.070056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:32.884762+0000","last_change":"2024-02-12T16:51:32.884735+0000","last_active":"2024-02-12T16:51:32.884762+0000","last_peered":"2024-02-12T16:51:32.884762+0000","last_clean":"2024-02-12T16:51:32.884762+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:51:32.884762+0000","last_undegraded":"2024-02-12T16:51:32.884762+0000","last_fullsized":"2024-02-12T16:51:32.884762+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.884670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230018+0000","last_change":"2024-02-12T16:50:56.149313+0000","last_active":"2024-02-12T16:51:22.230018+0000","last_peered":"2024-02-12T16:51:22.230018+0000","last_clean":"2024-02-12T16:51:22.230018+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:51:22.230018+0000","last_undegraded":"2024-02-12T16:51:22.230018+0000","last_fullsized":"2024-02-12T16:51:22.230018+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:56.149242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:56.149242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106052+0000","last_change":"2024-02-12T16:50:43.246481+0000","last_active":"2024-02-12T16:51:22.106052+0000","last_peered":"2024-02-12T16:51:22.106052+0000","last_clean":"2024-02-12T16:51:22.106052+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:51:22.106052+0000","last_undegraded":"2024-02-12T16:51:22.106052+0000","last_fullsized":"2024-02-12T16:51:22.106052+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:50:43.246399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:43.246399+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230675+0000","last_change":"2024-02-12T16:51:10.119500+0000","last_active":"2024-02-12T16:51:22.230675+0000","last_peered":"2024-02-12T16:51:22.230675+0000","last_clean":"2024-02-12T16:51:22.230675+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:51:22.230675+0000","last_undegraded":"2024-02-12T16:51:22.230675+0000","last_fullsized":"2024-02-12T16:51:22.230675+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:10.119399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:10.119399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:32.837050+0000","last_change":"2024-02-12T16:51:32.837004+0000","last_active":"2024-02-12T16:51:32.837050+0000","last_peered":"2024-02-12T16:51:32.837050+0000","last_clean":"2024-02-12T16:51:32.837050+0000","last_became_active":"2024-02-12T16:49:45.222560+0000","last_became_peered":"2024-02-12T16:49:45.222560+0000","last_unstale":"2024-02-12T16:51:32.837050+0000","last_undegraded":"2024-02-12T16:51:32.837050+0000","last_fullsized":"2024-02-12T16:51:32.837050+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.836928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.836928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107458+0000","last_change":"2024-02-12T16:51:16.010678+0000","last_active":"2024-02-12T16:51:22.107458+0000","last_peered":"2024-02-12T16:51:22.107458+0000","last_clean":"2024-02-12T16:51:22.107458+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:51:22.107458+0000","last_undegraded":"2024-02-12T16:51:22.107458+0000","last_fullsized":"2024-02-12T16:51:22.107458+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:16.010512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.010512+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228202+0000","last_change":"2024-02-12T16:49:41.670234+0000","last_active":"2024-02-12T16:51:22.228202+0000","last_peered":"2024-02-12T16:51:22.228202+0000","last_clean":"2024-02-12T16:51:22.228202+0000","last_became_active":"2024-02-12T16:49:41.669593+0000","last_became_peered":"2024-02-12T16:49:41.669593+0000","last_unstale":"2024-02-12T16:51:22.228202+0000","last_undegraded":"2024-02-12T16:51:22.228202+0000","last_fullsized":"2024-02-12T16:51:22.228202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:49:40.156223+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:35.749314+0000","last_change":"2024-02-12T16:51:35.749314+0000","last_active":"2024-02-12T16:51:35.749314+0000","last_peered":"2024-02-12T16:51:35.749314+0000","last_clean":"2024-02-12T16:51:35.749314+0000","last_became_active":"2024-02-12T16:49:43.202312+0000","last_became_peered":"2024-02-12T16:49:43.202312+0000","last_unstale":"2024-02-12T16:51:35.749314+0000","last_undegraded":"2024-02-12T16:51:35.749314+0000","last_fullsized":"2024-02-12T16:51:35.749314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:35.749237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:35.749237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229911+0000","last_change":"2024-02-12T16:50:55.111751+0000","last_active":"2024-02-12T16:51:22.229911+0000","last_peered":"2024-02-12T16:51:22.229911+0000","last_clean":"2024-02-12T16:51:22.229911+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:51:22.229911+0000","last_undegraded":"2024-02-12T16:51:22.229911+0000","last_fullsized":"2024-02-12T16:51:22.229911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:50:55.111675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.111675+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230072+0000","last_change":"2024-02-12T16:51:16.020537+0000","last_active":"2024-02-12T16:51:22.230072+0000","last_peered":"2024-02-12T16:51:22.230072+0000","last_clean":"2024-02-12T16:51:22.230072+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:51:22.230072+0000","last_undegraded":"2024-02-12T16:51:22.230072+0000","last_fullsized":"2024-02-12T16:51:22.230072+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:16.020471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.020471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105929+0000","last_change":"2024-02-12T16:50:59.823981+0000","last_active":"2024-02-12T16:51:22.105929+0000","last_peered":"2024-02-12T16:51:22.105929+0000","last_clean":"2024-02-12T16:51:22.105929+0000","last_became_active":"2024-02-12T16:49:41.178569+0000","last_became_peered":"2024-02-12T16:49:41.178569+0000","last_unstale":"2024-02-12T16:51:22.105929+0000","last_undegraded":"2024-02-12T16:51:22.105929+0000","last_fullsized":"2024-02-12T16:51:22.105929+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:50:59.823898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:59.823898+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:35.745538+0000","last_change":"2024-02-12T16:51:03.961580+0000","last_active":"2024-02-12T16:51:35.745538+0000","last_peered":"2024-02-12T16:51:35.745538+0000","last_clean":"2024-02-12T16:51:35.745538+0000","last_became_active":"2024-02-12T16:49:43.205391+0000","last_became_peered":"2024-02-12T16:49:43.205391+0000","last_unstale":"2024-02-12T16:51:35.745538+0000","last_undegraded":"2024-02-12T16:51:35.745538+0000","last_fullsized":"2024-02-12T16:51:35.745538+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:03.961503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.961503+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:34.081749+0000","last_change":"2024-02-12T16:51:34.081725+0000","last_active":"2024-02-12T16:51:34.081749+0000","last_peered":"2024-02-12T16:51:34.081749+0000","last_clean":"2024-02-12T16:51:34.081749+0000","last_became_active":"2024-02-12T16:49:45.223659+0000","last_became_peered":"2024-02-12T16:49:45.223659+0000","last_unstale":"2024-02-12T16:51:34.081749+0000","last_undegraded":"2024-02-12T16:51:34.081749+0000","last_fullsized":"2024-02-12T16:51:34.081749+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:34.081644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.081644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230576+0000","last_change":"2024-02-12T16:51:05.201812+0000","last_active":"2024-02-12T16:51:22.230576+0000","last_peered":"2024-02-12T16:51:22.230576+0000","last_clean":"2024-02-12T16:51:22.230576+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:51:22.230576+0000","last_undegraded":"2024-02-12T16:51:22.230576+0000","last_fullsized":"2024-02-12T16:51:22.230576+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:51:05.201749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128212+0000","last_change":"2024-02-12T16:51:04.206339+0000","last_active":"2024-02-12T16:51:22.128212+0000","last_peered":"2024-02-12T16:51:22.128212+0000","last_clean":"2024-02-12T16:51:22.128212+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:51:22.128212+0000","last_undegraded":"2024-02-12T16:51:22.128212+0000","last_fullsized":"2024-02-12T16:51:22.128212+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.123059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.123059+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.123059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:30.776982+0000","last_change":"2024-02-12T16:51:30.776952+0000","last_active":"2024-02-12T16:51:30.776982+0000","last_peered":"2024-02-12T16:51:30.776982+0000","last_clean":"2024-02-12T16:51:30.776982+0000","last_became_active":"2024-02-12T16:49:41.178616+0000","last_became_peered":"2024-02-12T16:49:41.178616+0000","last_unstale":"2024-02-12T16:51:30.776982+0000","last_undegraded":"2024-02-12T16:51:30.776982+0000","last_fullsized":"2024-02-12T16:51:30.776982+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:30.776871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:30.776871+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229287+0000","last_change":"2024-02-12T16:51:02.216274+0000","last_active":"2024-02-12T16:51:22.229287+0000","last_peered":"2024-02-12T16:51:22.229287+0000","last_clean":"2024-02-12T16:51:22.229287+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:51:22.229287+0000","last_undegraded":"2024-02-12T16:51:22.229287+0000","last_fullsized":"2024-02-12T16:51:22.229287+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.216194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.216194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"80'9","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:35.384657+0000","last_change":"2024-02-12T16:50:50.800052+0000","last_active":"2024-02-12T16:51:35.384657+0000","last_peered":"2024-02-12T16:51:35.384657+0000","last_clean":"2024-02-12T16:51:35.384657+0000","last_became_active":"2024-02-12T16:49:43.197112+0000","last_became_peered":"2024-02-12T16:49:43.197112+0000","last_unstale":"2024-02-12T16:51:35.384657+0000","last_undegraded":"2024-02-12T16:51:35.384657+0000","last_fullsized":"2024-02-12T16:51:35.384657+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.799984+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105725+0000","last_change":"2024-02-12T16:51:00.778646+0000","last_active":"2024-02-12T16:51:22.105725+0000","last_peered":"2024-02-12T16:51:22.105725+0000","last_clean":"2024-02-12T16:51:22.105725+0000","last_became_active":"2024-02-12T16:49:39.095868+0000","last_became_peered":"2024-02-12T16:49:39.095868+0000","last_unstale":"2024-02-12T16:51:22.105725+0000","last_undegraded":"2024-02-12T16:51:22.105725+0000","last_fullsized":"2024-02-12T16:51:22.105725+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:00.778578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:00.778578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106912+0000","last_change":"2024-02-12T16:51:11.980641+0000","last_active":"2024-02-12T16:51:22.106912+0000","last_peered":"2024-02-12T16:51:22.106912+0000","last_clean":"2024-02-12T16:51:22.106912+0000","last_became_active":"2024-02-12T16:49:41.178544+0000","last_became_peered":"2024-02-12T16:49:41.178544+0000","last_unstale":"2024-02-12T16:51:22.106912+0000","last_undegraded":"2024-02-12T16:51:22.106912+0000","last_fullsized":"2024-02-12T16:51:22.106912+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:51:11.980577+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:51:11.980577+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.980577+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106445+0000","last_change":"2024-02-12T16:49:45.217771+0000","last_active":"2024-02-12T16:51:22.106445+0000","last_peered":"2024-02-12T16:51:22.106445+0000","last_clean":"2024-02-12T16:51:22.106445+0000","last_became_active":"2024-02-12T16:49:45.217458+0000","last_became_peered":"2024-02-12T16:49:45.217458+0000","last_unstale":"2024-02-12T16:51:22.106445+0000","last_undegraded":"2024-02-12T16:51:22.106445+0000","last_fullsized":"2024-02-12T16:51:22.106445+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:49:44.186028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:36.022479+0000","last_change":"2024-02-12T16:51:16.145692+0000","last_active":"2024-02-12T16:51:36.022479+0000","last_peered":"2024-02-12T16:51:36.022479+0000","last_clean":"2024-02-12T16:51:36.022479+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:51:36.022479+0000","last_undegraded":"2024-02-12T16:51:36.022479+0000","last_fullsized":"2024-02-12T16:51:36.022479+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:16.145576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.145576+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105910+0000","last_change":"2024-02-12T16:51:02.785509+0000","last_active":"2024-02-12T16:51:22.105910+0000","last_peered":"2024-02-12T16:51:22.105910+0000","last_clean":"2024-02-12T16:51:22.105910+0000","last_became_active":"2024-02-12T16:49:39.101478+0000","last_became_peered":"2024-02-12T16:49:39.101478+0000","last_unstale":"2024-02-12T16:51:22.105910+0000","last_undegraded":"2024-02-12T16:51:22.105910+0000","last_fullsized":"2024-02-12T16:51:22.105910+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.785426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.785426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106313+0000","last_change":"2024-02-12T16:51:04.882407+0000","last_active":"2024-02-12T16:51:22.106313+0000","last_peered":"2024-02-12T16:51:22.106313+0000","last_clean":"2024-02-12T16:51:22.106313+0000","last_became_active":"2024-02-12T16:49:41.179396+0000","last_became_peered":"2024-02-12T16:49:41.179396+0000","last_unstale":"2024-02-12T16:51:22.106313+0000","last_undegraded":"2024-02-12T16:51:22.106313+0000","last_fullsized":"2024-02-12T16:51:22.106313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:04.882334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:04.882334+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128544+0000","last_change":"2024-02-12T16:50:55.285750+0000","last_active":"2024-02-12T16:51:22.128544+0000","last_peered":"2024-02-12T16:51:22.128544+0000","last_clean":"2024-02-12T16:51:22.128544+0000","last_became_active":"2024-02-12T16:49:45.220893+0000","last_became_peered":"2024-02-12T16:49:45.220893+0000","last_unstale":"2024-02-12T16:51:22.128544+0000","last_undegraded":"2024-02-12T16:51:22.128544+0000","last_fullsized":"2024-02-12T16:51:22.128544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.285685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.285685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106984+0000","last_change":"2024-02-12T16:51:13.985683+0000","last_active":"2024-02-12T16:51:22.106984+0000","last_peered":"2024-02-12T16:51:22.106984+0000","last_clean":"2024-02-12T16:51:22.106984+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:51:22.106984+0000","last_undegraded":"2024-02-12T16:51:22.106984+0000","last_fullsized":"2024-02-12T16:51:22.106984+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.985614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.985614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229102+0000","last_change":"2024-02-12T16:50:44.092086+0000","last_active":"2024-02-12T16:51:22.229102+0000","last_peered":"2024-02-12T16:51:22.229102+0000","last_clean":"2024-02-12T16:51:22.229102+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:51:22.229102+0000","last_undegraded":"2024-02-12T16:51:22.229102+0000","last_fullsized":"2024-02-12T16:51:22.229102+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:44.092019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:44.092019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106721+0000","last_change":"2024-02-12T16:51:12.892241+0000","last_active":"2024-02-12T16:51:22.106721+0000","last_peered":"2024-02-12T16:51:22.106721+0000","last_clean":"2024-02-12T16:51:22.106721+0000","last_became_active":"2024-02-12T16:49:41.177416+0000","last_became_peered":"2024-02-12T16:49:41.177416+0000","last_unstale":"2024-02-12T16:51:22.106721+0000","last_undegraded":"2024-02-12T16:51:22.106721+0000","last_fullsized":"2024-02-12T16:51:22.106721+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:51:12.892148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:12.892148+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228631+0000","last_change":"2024-02-12T16:50:54.135100+0000","last_active":"2024-02-12T16:51:22.228631+0000","last_peered":"2024-02-12T16:51:22.228631+0000","last_clean":"2024-02-12T16:51:22.228631+0000","last_became_active":"2024-02-12T16:49:45.217589+0000","last_became_peered":"2024-02-12T16:49:45.217589+0000","last_unstale":"2024-02-12T16:51:22.228631+0000","last_undegraded":"2024-02-12T16:51:22.228631+0000","last_fullsized":"2024-02-12T16:51:22.228631+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.135009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128264+0000","last_change":"2024-02-12T16:50:52.289793+0000","last_active":"2024-02-12T16:51:22.128264+0000","last_peered":"2024-02-12T16:51:22.128264+0000","last_clean":"2024-02-12T16:51:22.128264+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:51:22.128264+0000","last_undegraded":"2024-02-12T16:51:22.128264+0000","last_fullsized":"2024-02-12T16:51:22.128264+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:52.289725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:52.289725+0000","last_clean_scrub_stamp":"2024-02-12T16:50:52.289725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128562+0000","last_change":"2024-02-12T16:50:51.308424+0000","last_active":"2024-02-12T16:51:22.128562+0000","last_peered":"2024-02-12T16:51:22.128562+0000","last_clean":"2024-02-12T16:51:22.128562+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:51:22.128562+0000","last_undegraded":"2024-02-12T16:51:22.128562+0000","last_fullsized":"2024-02-12T16:51:22.128562+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.308317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.308317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230573+0000","last_change":"2024-02-12T16:51:04.126502+0000","last_active":"2024-02-12T16:51:22.230573+0000","last_peered":"2024-02-12T16:51:22.230573+0000","last_clean":"2024-02-12T16:51:22.230573+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:51:22.230573+0000","last_undegraded":"2024-02-12T16:51:22.230573+0000","last_fullsized":"2024-02-12T16:51:22.230573+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:50:47.792388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.792388+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105882+0000","last_change":"2024-02-12T16:51:13.192907+0000","last_active":"2024-02-12T16:51:22.105882+0000","last_peered":"2024-02-12T16:51:22.105882+0000","last_clean":"2024-02-12T16:51:22.105882+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:51:22.105882+0000","last_undegraded":"2024-02-12T16:51:22.105882+0000","last_fullsized":"2024-02-12T16:51:22.105882+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.192842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.192842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128731+0000","last_change":"2024-02-12T16:49:43.202308+0000","last_active":"2024-02-12T16:51:22.128731+0000","last_peered":"2024-02-12T16:51:22.128731+0000","last_clean":"2024-02-12T16:51:22.128731+0000","last_became_active":"2024-02-12T16:49:43.201946+0000","last_became_peered":"2024-02-12T16:49:43.201946+0000","last_unstale":"2024-02-12T16:51:22.128731+0000","last_undegraded":"2024-02-12T16:51:22.128731+0000","last_fullsized":"2024-02-12T16:51:22.128731+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:49:42.168508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"58'1","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:31.889946+0000","last_change":"2024-02-12T16:51:31.889909+0000","last_active":"2024-02-12T16:51:31.889946+0000","last_peered":"2024-02-12T16:51:31.889946+0000","last_clean":"2024-02-12T16:51:31.889946+0000","last_became_active":"2024-02-12T16:49:39.099695+0000","last_became_peered":"2024-02-12T16:49:39.099695+0000","last_unstale":"2024-02-12T16:51:31.889946+0000","last_undegraded":"2024-02-12T16:51:31.889946+0000","last_fullsized":"2024-02-12T16:51:31.889946+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:51:31.889821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:31.889821+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106218+0000","last_change":"2024-02-12T16:50:45.175080+0000","last_active":"2024-02-12T16:51:22.106218+0000","last_peered":"2024-02-12T16:51:22.106218+0000","last_clean":"2024-02-12T16:51:22.106218+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:51:22.106218+0000","last_undegraded":"2024-02-12T16:51:22.106218+0000","last_fullsized":"2024-02-12T16:51:22.106218+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:50:45.174941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:45.174941+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:24.030122+0000","last_change":"2024-02-12T16:51:24.030089+0000","last_active":"2024-02-12T16:51:24.030122+0000","last_peered":"2024-02-12T16:51:24.030122+0000","last_clean":"2024-02-12T16:51:24.030122+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:51:24.030122+0000","last_undegraded":"2024-02-12T16:51:24.030122+0000","last_fullsized":"2024-02-12T16:51:24.030122+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.030017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.030017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128833+0000","last_change":"2024-02-12T16:50:43.269381+0000","last_active":"2024-02-12T16:51:22.128833+0000","last_peered":"2024-02-12T16:51:22.128833+0000","last_clean":"2024-02-12T16:51:22.128833+0000","last_became_active":"2024-02-12T16:49:39.105982+0000","last_became_peered":"2024-02-12T16:49:39.105982+0000","last_unstale":"2024-02-12T16:51:22.128833+0000","last_undegraded":"2024-02-12T16:51:22.128833+0000","last_fullsized":"2024-02-12T16:51:22.128833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:43.269296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:43.269296+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228632+0000","last_change":"2024-02-12T16:50:54.302143+0000","last_active":"2024-02-12T16:51:22.228632+0000","last_peered":"2024-02-12T16:51:22.228632+0000","last_clean":"2024-02-12T16:51:22.228632+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:51:22.228632+0000","last_undegraded":"2024-02-12T16:51:22.228632+0000","last_fullsized":"2024-02-12T16:51:22.228632+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:50:54.302060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.302060+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:35.745221+0000","last_change":"2024-02-12T16:51:07.021402+0000","last_active":"2024-02-12T16:51:35.745221+0000","last_peered":"2024-02-12T16:51:35.745221+0000","last_clean":"2024-02-12T16:51:35.745221+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:51:35.745221+0000","last_undegraded":"2024-02-12T16:51:35.745221+0000","last_fullsized":"2024-02-12T16:51:35.745221+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:07.021338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:07.021338+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234011,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297756,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447396,"statfs":{"total":95995035648,"available":95690133504,"internally_reserved":0,"allocated":634880,"data_stored":292275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69799999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.435},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.497},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80500000000000005,"5min":1.056,"15min":1.056},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65900000000000003,"5min":1.208,"15min":1.208},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56899999999999995,"15min":0.57999999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72299999999999998,"5min":1.766,"15min":1.766},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.61299999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.51100000000000001,"15min":0.52200000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67000000000000004,"5min":1.327,"15min":1.327},"last":0.35299999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68400000000000005,"5min":1.573,"15min":1.573},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53500000000000003,"15min":0.54300000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76400000000000001,"5min":1.621,"15min":1.621},"last":0.39900000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77700000000000002,"5min":1.272,"15min":1.272},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73199999999999998,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.51800000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528352,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297736,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447416,"statfs":{"total":95995035648,"available":95690153984,"internally_reserved":0,"allocated":614400,"data_stored":291625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48199999999999998,"15min":0.48299999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67000000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65700000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.747,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53200000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72799999999999998,"5min":0.754,"15min":0.754},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.50600000000000001,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64500000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.39800000000000002},"min":{"1min":0.28699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.42399999999999999,"15min":0.41799999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.318,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:51:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.56799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297792,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447360,"statfs":{"total":95995035648,"available":95690096640,"internally_reserved":0,"allocated":671744,"data_stored":294352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35399999999999998,"15min":0.34000000000000002},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.38800000000000001,"15min":0.372},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":0.25},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.40999999999999998,"15min":0.38600000000000001},"min":{"1min":0.29399999999999998,"5min":0.246,"15min":0.246},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.45600000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.42099999999999999,"15min":0.40899999999999997},"min":{"1min":0.35199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:51:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.40899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77300000000000002,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77200000000000002,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.376}]},{"osd":6,"last update":"Mon Feb 12 16:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:51:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.64800000000000002,"15min":0.65800000000000003},"min":{"1min":0.438,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79900000000000004,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.438},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83999999999999997,"5min":1.109,"15min":1.109},"last":0.38300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":630784,"data_stored":291959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"min":{"1min":0.255,"5min":0.192,"15min":0.192},"max":{"1min":0.55300000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.35299999999999998,"15min":0.317},"min":{"1min":0.27700000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.61599999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.40699999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.38},"min":{"1min":0.27400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.61499999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.378},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.57699999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.371}]},{"osd":3,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.40500000000000003,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56399999999999995,"5min":0.622,"15min":0.622},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.45800000000000002,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.433}]},{"osd":4,"last update":"Mon Feb 12 16:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.745,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67300000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.73099999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.45000000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.438},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71299999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:50:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45600000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297744,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447408,"statfs":{"total":95995035648,"available":95690145792,"internally_reserved":0,"allocated":622592,"data_stored":292183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.39800000000000002,"15min":0.38200000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.42999999999999999,"15min":0.40400000000000003},"min":{"1min":0.27100000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.47499999999999998,"15min":0.42099999999999999},"min":{"1min":0.27400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":6.8040000000000003,"5min":6.8040000000000003,"15min":6.8040000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.442,"15min":0.41299999999999998},"min":{"1min":0.29199999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.375}]},{"osd":3,"last update":"Mon Feb 12 16:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47199999999999998,"15min":0.46600000000000003},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.57599999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.503,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.74199999999999999,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":0.40899999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.70499999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.503},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.61499999999999999,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.69099999999999995,"5min":2.875,"15min":2.875},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.71599999999999997,"15min":0.74199999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82099999999999995,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54600000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90400000000000003,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":7.5629999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.48799999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":293275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.436,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52100000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.372}]},{"osd":1,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.42599999999999999,"15min":0.436},"min":{"1min":0.255,"5min":0.216,"15min":0.216},"max":{"1min":0.52300000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42899999999999999,"15min":0.436},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.503,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.442,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.436,"15min":0.44400000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.503,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58899999999999997,"15min":0.61099999999999999},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78100000000000003,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.44},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73699999999999999,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.621,"15min":0.63600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.754,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.61899999999999999,"15min":0.63100000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.69699999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76500000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.432,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.57799999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953514,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297756,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447396,"statfs":{"total":95995035648,"available":95690133504,"internally_reserved":0,"allocated":634880,"data_stored":292628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.55300000000000005},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.51800000000000002},"min":{"1min":0.30399999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.504}]},{"osd":1,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.30299999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66300000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.57199999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.67200000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53700000000000003,"15min":0.55400000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61099999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:50:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33200000000000002},"min":{"1min":0.27100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.376,"15min":0.371},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.628},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.32600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45800000000000002,"15min":0.44600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.71399999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757286,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":606208,"data_stored":291233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51600000000000001,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.72099999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72699999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.67200000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.34300000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.71299999999999997,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.501,"15min":0.502},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.624}]},{"osd":3,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52300000000000002,"15min":0.53000000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.73799999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.51000000000000001,"15min":0.51100000000000001},"min":{"1min":0.374,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70299999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.876}]},{"osd":4,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.38},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.80300000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.44500000000000001,"15min":0.439},"min":{"1min":0.35499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.57199999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:51:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.746},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:51:43.430 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:51:43.740 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:43.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:51:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:51:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:42.296521+0000 mgr.y (mgr.24481) 83 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:42.644333+0000 mgr.y (mgr.24481) 84 : audit [DBG] from='client.34658 v1:172.21.15.33:0/3806275672' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:43.069440+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.10 scrub starts 2024-02-12T16:51:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:43.072895+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.10 scrub ok 2024-02-12T16:51:43.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:42.296521+0000 mgr.y (mgr.24481) 83 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:43.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:42.644333+0000 mgr.y (mgr.24481) 84 : audit [DBG] from='client.34658 v1:172.21.15.33:0/3806275672' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:43.069440+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.10 scrub starts 2024-02-12T16:51:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:43.072895+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.10 scrub ok 2024-02-12T16:51:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:42.296521+0000 mgr.y (mgr.24481) 83 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:42.644333+0000 mgr.y (mgr.24481) 84 : audit [DBG] from='client.34658 v1:172.21.15.33:0/3806275672' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:43.069440+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.10 scrub starts 2024-02-12T16:51:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:43.072895+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.10 scrub ok 2024-02-12T16:51:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:43.365354+0000 mgr.y (mgr.24481) 85 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:44.087962+0000 osd.3 (osd.3) 43 : cluster [DBG] 5.14 scrub starts 2024-02-12T16:51:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:44.091431+0000 osd.3 (osd.3) 44 : cluster [DBG] 5.14 scrub ok 2024-02-12T16:51:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:43.365354+0000 mgr.y (mgr.24481) 85 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:44.087962+0000 osd.3 (osd.3) 43 : cluster [DBG] 5.14 scrub starts 2024-02-12T16:51:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:44.091431+0000 osd.3 (osd.3) 44 : cluster [DBG] 5.14 scrub ok 2024-02-12T16:51:44.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:43.365354+0000 mgr.y (mgr.24481) 85 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:44.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:44.087962+0000 osd.3 (osd.3) 43 : cluster [DBG] 5.14 scrub starts 2024-02-12T16:51:44.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:44.091431+0000 osd.3 (osd.3) 44 : cluster [DBG] 5.14 scrub ok 2024-02-12T16:51:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:44.297317+0000 mgr.y (mgr.24481) 86 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:44.302448+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a deep-scrub starts 2024-02-12T16:51:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:44.303374+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a deep-scrub ok 2024-02-12T16:51:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:44.623349+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:45.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:44.623414+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:45.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:45.049688+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.1b scrub starts 2024-02-12T16:51:45.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:45.052929+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.1b scrub ok 2024-02-12T16:51:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:44.297317+0000 mgr.y (mgr.24481) 86 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:44.302448+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a deep-scrub starts 2024-02-12T16:51:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:44.303374+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a deep-scrub ok 2024-02-12T16:51:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:44.623349+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:44.623414+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:45.049688+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.1b scrub starts 2024-02-12T16:51:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:45.052929+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.1b scrub ok 2024-02-12T16:51:45.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:44.297317+0000 mgr.y (mgr.24481) 86 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:45.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:44.302448+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a deep-scrub starts 2024-02-12T16:51:45.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:44.303374+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a deep-scrub ok 2024-02-12T16:51:45.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:44.623349+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:45.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:44.623414+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:45.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:45.049688+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.1b scrub starts 2024-02-12T16:51:45.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:45.052929+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.1b scrub ok 2024-02-12T16:51:45.855 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:51:45.857 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:51:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:45.415932+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-12T16:51:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:45.415932+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-12T16:51:46.635 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-02-12T16:51:45.413339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":837,"num_read_kb":589,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":488,"ondisk_log_size":488,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381796,"kb_used_data":4948,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579420,"statfs":{"total":767960285184,"available":765521326080,"internally_reserved":0,"allocated":5066752,"data_stored":2339530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.030155"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228411+0000","last_change":"2024-02-12T16:49:45.224168+0000","last_active":"2024-02-12T16:51:22.228411+0000","last_peered":"2024-02-12T16:51:22.228411+0000","last_clean":"2024-02-12T16:51:22.228411+0000","last_became_active":"2024-02-12T16:49:45.223515+0000","last_became_peered":"2024-02-12T16:49:45.223515+0000","last_unstale":"2024-02-12T16:51:22.228411+0000","last_undegraded":"2024-02-12T16:51:22.228411+0000","last_fullsized":"2024-02-12T16:51:22.228411+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:49:44.186028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105779+0000","last_change":"2024-02-12T16:50:51.824034+0000","last_active":"2024-02-12T16:51:22.105779+0000","last_peered":"2024-02-12T16:51:22.105779+0000","last_clean":"2024-02-12T16:51:22.105779+0000","last_became_active":"2024-02-12T16:49:43.195544+0000","last_became_peered":"2024-02-12T16:49:43.195544+0000","last_unstale":"2024-02-12T16:51:22.105779+0000","last_undegraded":"2024-02-12T16:51:22.105779+0000","last_fullsized":"2024-02-12T16:51:22.105779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.823967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107225+0000","last_change":"2024-02-12T16:50:42.254831+0000","last_active":"2024-02-12T16:51:22.107225+0000","last_peered":"2024-02-12T16:51:22.107225+0000","last_clean":"2024-02-12T16:51:22.107225+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:51:22.107225+0000","last_undegraded":"2024-02-12T16:51:22.107225+0000","last_fullsized":"2024-02-12T16:51:22.107225+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:42.254752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.254752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228527+0000","last_change":"2024-02-12T16:51:16.075132+0000","last_active":"2024-02-12T16:51:22.228527+0000","last_peered":"2024-02-12T16:51:22.228527+0000","last_clean":"2024-02-12T16:51:22.228527+0000","last_became_active":"2024-02-12T16:49:41.185805+0000","last_became_peered":"2024-02-12T16:49:41.185805+0000","last_unstale":"2024-02-12T16:51:22.228527+0000","last_undegraded":"2024-02-12T16:51:22.228527+0000","last_fullsized":"2024-02-12T16:51:22.228527+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:51:16.075051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.075051+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105944+0000","last_change":"2024-02-12T16:51:21.897841+0000","last_active":"2024-02-12T16:51:22.105944+0000","last_peered":"2024-02-12T16:51:22.105944+0000","last_clean":"2024-02-12T16:51:22.105944+0000","last_became_active":"2024-02-12T16:49:45.216118+0000","last_became_peered":"2024-02-12T16:49:45.216118+0000","last_unstale":"2024-02-12T16:51:22.105944+0000","last_undegraded":"2024-02-12T16:51:22.105944+0000","last_fullsized":"2024-02-12T16:51:22.105944+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:21.897749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:21.897749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105577+0000","last_change":"2024-02-12T16:51:13.011883+0000","last_active":"2024-02-12T16:51:22.105577+0000","last_peered":"2024-02-12T16:51:22.105577+0000","last_clean":"2024-02-12T16:51:22.105577+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:51:22.105577+0000","last_undegraded":"2024-02-12T16:51:22.105577+0000","last_fullsized":"2024-02-12T16:51:22.105577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.011814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.011814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:38.073505+0000","last_change":"2024-02-12T16:51:38.073505+0000","last_active":"2024-02-12T16:51:38.073505+0000","last_peered":"2024-02-12T16:51:38.073505+0000","last_clean":"2024-02-12T16:51:38.073505+0000","last_became_active":"2024-02-12T16:49:39.102172+0000","last_became_peered":"2024-02-12T16:49:39.102172+0000","last_unstale":"2024-02-12T16:51:38.073505+0000","last_undegraded":"2024-02-12T16:51:38.073505+0000","last_fullsized":"2024-02-12T16:51:38.073505+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.073441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.073441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106003+0000","last_change":"2024-02-12T16:51:17.900217+0000","last_active":"2024-02-12T16:51:22.106003+0000","last_peered":"2024-02-12T16:51:22.106003+0000","last_clean":"2024-02-12T16:51:22.106003+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:51:22.106003+0000","last_undegraded":"2024-02-12T16:51:22.106003+0000","last_fullsized":"2024-02-12T16:51:22.106003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:17.900093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:17.900093+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230508+0000","last_change":"2024-02-12T16:50:47.317845+0000","last_active":"2024-02-12T16:51:22.230508+0000","last_peered":"2024-02-12T16:51:22.230508+0000","last_clean":"2024-02-12T16:51:22.230508+0000","last_became_active":"2024-02-12T16:49:45.218184+0000","last_became_peered":"2024-02-12T16:49:45.218184+0000","last_unstale":"2024-02-12T16:51:22.230508+0000","last_undegraded":"2024-02-12T16:51:22.230508+0000","last_fullsized":"2024-02-12T16:51:22.230508+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:47.317736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.317736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229329+0000","last_change":"2024-02-12T16:50:46.095997+0000","last_active":"2024-02-12T16:51:22.229329+0000","last_peered":"2024-02-12T16:51:22.229329+0000","last_clean":"2024-02-12T16:51:22.229329+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:51:22.229329+0000","last_undegraded":"2024-02-12T16:51:22.229329+0000","last_fullsized":"2024-02-12T16:51:22.229329+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.095924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229791+0000","last_change":"2024-02-12T16:51:04.051422+0000","last_active":"2024-02-12T16:51:22.229791+0000","last_peered":"2024-02-12T16:51:22.229791+0000","last_clean":"2024-02-12T16:51:22.229791+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:51:22.229791+0000","last_undegraded":"2024-02-12T16:51:22.229791+0000","last_fullsized":"2024-02-12T16:51:22.229791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:51:04.051345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":50,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:34.760832+0000","last_change":"2024-02-12T16:51:34.760803+0000","last_active":"2024-02-12T16:51:34.760832+0000","last_peered":"2024-02-12T16:51:34.760832+0000","last_clean":"2024-02-12T16:51:34.760832+0000","last_became_active":"2024-02-12T16:49:41.178018+0000","last_became_peered":"2024-02-12T16:49:41.178018+0000","last_unstale":"2024-02-12T16:51:34.760832+0000","last_undegraded":"2024-02-12T16:51:34.760832+0000","last_fullsized":"2024-02-12T16:51:34.760832+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:51:34.760711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.760711+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107587+0000","last_change":"2024-02-12T16:51:22.020495+0000","last_active":"2024-02-12T16:51:22.107587+0000","last_peered":"2024-02-12T16:51:22.107587+0000","last_clean":"2024-02-12T16:51:22.107587+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:51:22.107587+0000","last_undegraded":"2024-02-12T16:51:22.107587+0000","last_fullsized":"2024-02-12T16:51:22.107587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.020430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.020430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229062+0000","last_change":"2024-02-12T16:50:55.322528+0000","last_active":"2024-02-12T16:51:22.229062+0000","last_peered":"2024-02-12T16:51:22.229062+0000","last_clean":"2024-02-12T16:51:22.229062+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:51:22.229062+0000","last_undegraded":"2024-02-12T16:51:22.229062+0000","last_fullsized":"2024-02-12T16:51:22.229062+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.322446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229425+0000","last_change":"2024-02-12T16:51:01.027931+0000","last_active":"2024-02-12T16:51:22.229425+0000","last_peered":"2024-02-12T16:51:22.229425+0000","last_clean":"2024-02-12T16:51:22.229425+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:51:22.229425+0000","last_undegraded":"2024-02-12T16:51:22.229425+0000","last_fullsized":"2024-02-12T16:51:22.229425+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:01.027841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.027841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106032+0000","last_change":"2024-02-12T16:51:02.857339+0000","last_active":"2024-02-12T16:51:22.106032+0000","last_peered":"2024-02-12T16:51:22.106032+0000","last_clean":"2024-02-12T16:51:22.106032+0000","last_became_active":"2024-02-12T16:49:41.181117+0000","last_became_peered":"2024-02-12T16:49:41.181117+0000","last_unstale":"2024-02-12T16:51:22.106032+0000","last_undegraded":"2024-02-12T16:51:22.106032+0000","last_fullsized":"2024-02-12T16:51:22.106032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:02.857234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.857234+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228670+0000","last_change":"2024-02-12T16:51:03.001426+0000","last_active":"2024-02-12T16:51:22.228670+0000","last_peered":"2024-02-12T16:51:22.228670+0000","last_clean":"2024-02-12T16:51:22.228670+0000","last_became_active":"2024-02-12T16:49:45.221559+0000","last_became_peered":"2024-02-12T16:49:45.221559+0000","last_unstale":"2024-02-12T16:51:22.228670+0000","last_undegraded":"2024-02-12T16:51:22.228670+0000","last_fullsized":"2024-02-12T16:51:22.228670+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:03.001318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.001318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106444+0000","last_change":"2024-02-12T16:50:54.852577+0000","last_active":"2024-02-12T16:51:22.106444+0000","last_peered":"2024-02-12T16:51:22.106444+0000","last_clean":"2024-02-12T16:51:22.106444+0000","last_became_active":"2024-02-12T16:49:43.201709+0000","last_became_peered":"2024-02-12T16:49:43.201709+0000","last_unstale":"2024-02-12T16:51:22.106444+0000","last_undegraded":"2024-02-12T16:51:22.106444+0000","last_fullsized":"2024-02-12T16:51:22.106444+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:54.852481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.852481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:29.066207+0000","last_change":"2024-02-12T16:51:29.066176+0000","last_active":"2024-02-12T16:51:29.066207+0000","last_peered":"2024-02-12T16:51:29.066207+0000","last_clean":"2024-02-12T16:51:29.066207+0000","last_became_active":"2024-02-12T16:49:39.105438+0000","last_became_peered":"2024-02-12T16:49:39.105438+0000","last_unstale":"2024-02-12T16:51:29.066207+0000","last_undegraded":"2024-02-12T16:51:29.066207+0000","last_fullsized":"2024-02-12T16:51:29.066207+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:29.066107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:29.066107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:23.148968+0000","last_change":"2024-02-12T16:51:23.148923+0000","last_active":"2024-02-12T16:51:23.148968+0000","last_peered":"2024-02-12T16:51:23.148968+0000","last_clean":"2024-02-12T16:51:23.148968+0000","last_became_active":"2024-02-12T16:49:41.181980+0000","last_became_peered":"2024-02-12T16:49:41.181980+0000","last_unstale":"2024-02-12T16:51:23.148968+0000","last_undegraded":"2024-02-12T16:51:23.148968+0000","last_fullsized":"2024-02-12T16:51:23.148968+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:51:23.148783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:23.148783+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106834+0000","last_change":"2024-02-12T16:51:10.898681+0000","last_active":"2024-02-12T16:51:22.106834+0000","last_peered":"2024-02-12T16:51:22.106834+0000","last_clean":"2024-02-12T16:51:22.106834+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:51:22.106834+0000","last_undegraded":"2024-02-12T16:51:22.106834+0000","last_fullsized":"2024-02-12T16:51:22.106834+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:51:10.898594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106469+0000","last_change":"2024-02-12T16:51:20.972423+0000","last_active":"2024-02-12T16:51:22.106469+0000","last_peered":"2024-02-12T16:51:22.106469+0000","last_clean":"2024-02-12T16:51:22.106469+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:51:22.106469+0000","last_undegraded":"2024-02-12T16:51:22.106469+0000","last_fullsized":"2024-02-12T16:51:22.106469+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.972298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.972298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230677+0000","last_change":"2024-02-12T16:50:40.782894+0000","last_active":"2024-02-12T16:51:22.230677+0000","last_peered":"2024-02-12T16:51:22.230677+0000","last_clean":"2024-02-12T16:51:22.230677+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:51:22.230677+0000","last_undegraded":"2024-02-12T16:51:22.230677+0000","last_fullsized":"2024-02-12T16:51:22.230677+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:40.782823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:40.782823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128597+0000","last_change":"2024-02-12T16:50:47.282788+0000","last_active":"2024-02-12T16:51:22.128597+0000","last_peered":"2024-02-12T16:51:22.128597+0000","last_clean":"2024-02-12T16:51:22.128597+0000","last_became_active":"2024-02-12T16:49:41.180319+0000","last_became_peered":"2024-02-12T16:49:41.180319+0000","last_unstale":"2024-02-12T16:51:22.128597+0000","last_undegraded":"2024-02-12T16:51:22.128597+0000","last_fullsized":"2024-02-12T16:51:22.128597+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:50:47.282711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.282711+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106383+0000","last_change":"2024-02-12T16:51:08.888476+0000","last_active":"2024-02-12T16:51:22.106383+0000","last_peered":"2024-02-12T16:51:22.106383+0000","last_clean":"2024-02-12T16:51:22.106383+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:51:22.106383+0000","last_undegraded":"2024-02-12T16:51:22.106383+0000","last_fullsized":"2024-02-12T16:51:22.106383+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:08.888390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.888390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107197+0000","last_change":"2024-02-12T16:50:46.144078+0000","last_active":"2024-02-12T16:51:22.107197+0000","last_peered":"2024-02-12T16:51:22.107197+0000","last_clean":"2024-02-12T16:51:22.107197+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:51:22.107197+0000","last_undegraded":"2024-02-12T16:51:22.107197+0000","last_fullsized":"2024-02-12T16:51:22.107197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.144010+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228279+0000","last_change":"2024-02-12T16:51:09.094169+0000","last_active":"2024-02-12T16:51:22.228279+0000","last_peered":"2024-02-12T16:51:22.228279+0000","last_clean":"2024-02-12T16:51:22.228279+0000","last_became_active":"2024-02-12T16:49:39.101492+0000","last_became_peered":"2024-02-12T16:49:39.101492+0000","last_unstale":"2024-02-12T16:51:22.228279+0000","last_undegraded":"2024-02-12T16:51:22.228279+0000","last_fullsized":"2024-02-12T16:51:22.228279+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:09.094067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:09.094067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229886+0000","last_change":"2024-02-12T16:51:14.042938+0000","last_active":"2024-02-12T16:51:22.229886+0000","last_peered":"2024-02-12T16:51:22.229886+0000","last_clean":"2024-02-12T16:51:22.229886+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:51:22.229886+0000","last_undegraded":"2024-02-12T16:51:22.229886+0000","last_fullsized":"2024-02-12T16:51:22.229886+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:14.042858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:14.042858+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:28.104077+0000","last_change":"2024-02-12T16:51:28.104047+0000","last_active":"2024-02-12T16:51:28.104077+0000","last_peered":"2024-02-12T16:51:28.104077+0000","last_clean":"2024-02-12T16:51:28.104077+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:51:28.104077+0000","last_undegraded":"2024-02-12T16:51:28.104077+0000","last_fullsized":"2024-02-12T16:51:28.104077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.103969+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.229873+0000","last_change":"2024-02-12T16:51:08.073509+0000","last_active":"2024-02-12T16:51:37.229873+0000","last_peered":"2024-02-12T16:51:37.229873+0000","last_clean":"2024-02-12T16:51:37.229873+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:51:37.229873+0000","last_undegraded":"2024-02-12T16:51:37.229873+0000","last_fullsized":"2024-02-12T16:51:37.229873+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:08.073428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.073428+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228434+0000","last_change":"2024-02-12T16:51:01.224070+0000","last_active":"2024-02-12T16:51:22.228434+0000","last_peered":"2024-02-12T16:51:22.228434+0000","last_clean":"2024-02-12T16:51:22.228434+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:51:22.228434+0000","last_undegraded":"2024-02-12T16:51:22.228434+0000","last_fullsized":"2024-02-12T16:51:22.228434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:01.223973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.223973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107152+0000","last_change":"2024-02-12T16:50:44.211440+0000","last_active":"2024-02-12T16:51:22.107152+0000","last_peered":"2024-02-12T16:51:22.107152+0000","last_clean":"2024-02-12T16:51:22.107152+0000","last_became_active":"2024-02-12T16:49:41.669589+0000","last_became_peered":"2024-02-12T16:49:41.669589+0000","last_unstale":"2024-02-12T16:51:22.107152+0000","last_undegraded":"2024-02-12T16:51:22.107152+0000","last_fullsized":"2024-02-12T16:51:22.107152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:50:44.211369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:44.211369+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228863+0000","last_change":"2024-02-12T16:49:45.222705+0000","last_active":"2024-02-12T16:51:22.228863+0000","last_peered":"2024-02-12T16:51:22.228863+0000","last_clean":"2024-02-12T16:51:22.228863+0000","last_became_active":"2024-02-12T16:49:45.222341+0000","last_became_peered":"2024-02-12T16:49:45.222341+0000","last_unstale":"2024-02-12T16:51:22.228863+0000","last_undegraded":"2024-02-12T16:51:22.228863+0000","last_fullsized":"2024-02-12T16:51:22.228863+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:49:44.186028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229028+0000","last_change":"2024-02-12T16:51:19.038453+0000","last_active":"2024-02-12T16:51:22.229028+0000","last_peered":"2024-02-12T16:51:22.229028+0000","last_clean":"2024-02-12T16:51:22.229028+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:51:22.229028+0000","last_undegraded":"2024-02-12T16:51:22.229028+0000","last_fullsized":"2024-02-12T16:51:22.229028+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:19.038388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:19.038388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230171+0000","last_change":"2024-02-12T16:51:18.026686+0000","last_active":"2024-02-12T16:51:22.230171+0000","last_peered":"2024-02-12T16:51:22.230171+0000","last_clean":"2024-02-12T16:51:22.230171+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:51:22.230171+0000","last_undegraded":"2024-02-12T16:51:22.230171+0000","last_fullsized":"2024-02-12T16:51:22.230171+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:18.026624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.026624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228467+0000","last_change":"2024-02-12T16:50:51.908417+0000","last_active":"2024-02-12T16:51:22.228467+0000","last_peered":"2024-02-12T16:51:22.228467+0000","last_clean":"2024-02-12T16:51:22.228467+0000","last_became_active":"2024-02-12T16:49:41.669704+0000","last_became_peered":"2024-02-12T16:49:41.669704+0000","last_unstale":"2024-02-12T16:51:22.228467+0000","last_undegraded":"2024-02-12T16:51:22.228467+0000","last_fullsized":"2024-02-12T16:51:22.228467+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:50:51.908326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.908326+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228893+0000","last_change":"2024-02-12T16:51:15.965316+0000","last_active":"2024-02-12T16:51:22.228893+0000","last_peered":"2024-02-12T16:51:22.228893+0000","last_clean":"2024-02-12T16:51:22.228893+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:51:22.228893+0000","last_undegraded":"2024-02-12T16:51:22.228893+0000","last_fullsized":"2024-02-12T16:51:22.228893+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:51:15.965186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228597+0000","last_change":"2024-02-12T16:50:55.941652+0000","last_active":"2024-02-12T16:51:22.228597+0000","last_peered":"2024-02-12T16:51:22.228597+0000","last_clean":"2024-02-12T16:51:22.228597+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:51:22.228597+0000","last_undegraded":"2024-02-12T16:51:22.228597+0000","last_fullsized":"2024-02-12T16:51:22.228597+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.941575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.941575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229439+0000","last_change":"2024-02-12T16:51:11.135365+0000","last_active":"2024-02-12T16:51:22.229439+0000","last_peered":"2024-02-12T16:51:22.229439+0000","last_clean":"2024-02-12T16:51:22.229439+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:51:22.229439+0000","last_undegraded":"2024-02-12T16:51:22.229439+0000","last_fullsized":"2024-02-12T16:51:22.229439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.135261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:43.072967+0000","last_change":"2024-02-12T16:51:43.072967+0000","last_active":"2024-02-12T16:51:43.072967+0000","last_peered":"2024-02-12T16:51:43.072967+0000","last_clean":"2024-02-12T16:51:43.072967+0000","last_became_active":"2024-02-12T16:49:41.184044+0000","last_became_peered":"2024-02-12T16:49:41.184044+0000","last_unstale":"2024-02-12T16:51:43.072967+0000","last_undegraded":"2024-02-12T16:51:43.072967+0000","last_fullsized":"2024-02-12T16:51:43.072967+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:51:43.072899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:43.072899+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:42.040901+0000","last_change":"2024-02-12T16:51:42.040901+0000","last_active":"2024-02-12T16:51:42.040901+0000","last_peered":"2024-02-12T16:51:42.040901+0000","last_clean":"2024-02-12T16:51:42.040901+0000","last_became_active":"2024-02-12T16:49:45.220469+0000","last_became_peered":"2024-02-12T16:49:45.220469+0000","last_unstale":"2024-02-12T16:51:42.040901+0000","last_undegraded":"2024-02-12T16:51:42.040901+0000","last_fullsized":"2024-02-12T16:51:42.040901+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:51:42.040828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106088+0000","last_change":"2024-02-12T16:50:57.875954+0000","last_active":"2024-02-12T16:51:22.106088+0000","last_peered":"2024-02-12T16:51:22.106088+0000","last_clean":"2024-02-12T16:51:22.106088+0000","last_became_active":"2024-02-12T16:49:43.204615+0000","last_became_peered":"2024-02-12T16:49:43.204615+0000","last_unstale":"2024-02-12T16:51:22.106088+0000","last_undegraded":"2024-02-12T16:51:22.106088+0000","last_fullsized":"2024-02-12T16:51:22.106088+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:57.875871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.875871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106525+0000","last_change":"2024-02-12T16:50:46.810493+0000","last_active":"2024-02-12T16:51:22.106525+0000","last_peered":"2024-02-12T16:51:22.106525+0000","last_clean":"2024-02-12T16:51:22.106525+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:51:22.106525+0000","last_undegraded":"2024-02-12T16:51:22.106525+0000","last_fullsized":"2024-02-12T16:51:22.106525+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.810420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.810420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106255+0000","last_change":"2024-02-12T16:51:14.844224+0000","last_active":"2024-02-12T16:51:22.106255+0000","last_peered":"2024-02-12T16:51:22.106255+0000","last_clean":"2024-02-12T16:51:22.106255+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:51:22.106255+0000","last_undegraded":"2024-02-12T16:51:22.106255+0000","last_fullsized":"2024-02-12T16:51:22.106255+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:14.844115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:14.844115+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:28.886428+0000","last_change":"2024-02-12T16:51:28.886399+0000","last_active":"2024-02-12T16:51:28.886428+0000","last_peered":"2024-02-12T16:51:28.886428+0000","last_clean":"2024-02-12T16:51:28.886428+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:51:28.886428+0000","last_undegraded":"2024-02-12T16:51:28.886428+0000","last_fullsized":"2024-02-12T16:51:28.886428+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.886321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:39.040277+0000","last_change":"2024-02-12T16:51:39.040277+0000","last_active":"2024-02-12T16:51:39.040277+0000","last_peered":"2024-02-12T16:51:39.040277+0000","last_clean":"2024-02-12T16:51:39.040277+0000","last_became_active":"2024-02-12T16:49:43.208343+0000","last_became_peered":"2024-02-12T16:49:43.208343+0000","last_unstale":"2024-02-12T16:51:39.040277+0000","last_undegraded":"2024-02-12T16:51:39.040277+0000","last_fullsized":"2024-02-12T16:51:39.040277+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.040203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.040203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229419+0000","last_change":"2024-02-12T16:51:11.088535+0000","last_active":"2024-02-12T16:51:22.229419+0000","last_peered":"2024-02-12T16:51:22.229419+0000","last_clean":"2024-02-12T16:51:22.229419+0000","last_became_active":"2024-02-12T16:49:39.107471+0000","last_became_peered":"2024-02-12T16:49:39.107471+0000","last_unstale":"2024-02-12T16:51:22.229419+0000","last_undegraded":"2024-02-12T16:51:22.229419+0000","last_fullsized":"2024-02-12T16:51:22.229419+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:11.088412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.088412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106003+0000","last_change":"2024-02-12T16:51:01.759881+0000","last_active":"2024-02-12T16:51:22.106003+0000","last_peered":"2024-02-12T16:51:22.106003+0000","last_clean":"2024-02-12T16:51:22.106003+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:51:22.106003+0000","last_undegraded":"2024-02-12T16:51:22.106003+0000","last_fullsized":"2024-02-12T16:51:22.106003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.759802+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:24.897271+0000","last_change":"2024-02-12T16:51:24.897238+0000","last_active":"2024-02-12T16:51:24.897271+0000","last_peered":"2024-02-12T16:51:24.897271+0000","last_clean":"2024-02-12T16:51:24.897271+0000","last_became_active":"2024-02-12T16:49:45.217958+0000","last_became_peered":"2024-02-12T16:49:45.217958+0000","last_unstale":"2024-02-12T16:51:24.897271+0000","last_undegraded":"2024-02-12T16:51:24.897271+0000","last_fullsized":"2024-02-12T16:51:24.897271+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.897149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.897149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.230039+0000","last_change":"2024-02-12T16:49:43.206493+0000","last_active":"2024-02-12T16:51:37.230039+0000","last_peered":"2024-02-12T16:51:37.230039+0000","last_clean":"2024-02-12T16:51:37.230039+0000","last_became_active":"2024-02-12T16:49:43.206068+0000","last_became_peered":"2024-02-12T16:49:43.206068+0000","last_unstale":"2024-02-12T16:51:37.230039+0000","last_undegraded":"2024-02-12T16:51:37.230039+0000","last_fullsized":"2024-02-12T16:51:37.230039+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:49:42.168508+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:25.913239+0000","last_change":"2024-02-12T16:51:25.913214+0000","last_active":"2024-02-12T16:51:25.913239+0000","last_peered":"2024-02-12T16:51:25.913239+0000","last_clean":"2024-02-12T16:51:25.913239+0000","last_became_active":"2024-02-12T16:49:39.103401+0000","last_became_peered":"2024-02-12T16:49:39.103401+0000","last_unstale":"2024-02-12T16:51:25.913239+0000","last_undegraded":"2024-02-12T16:51:25.913239+0000","last_fullsized":"2024-02-12T16:51:25.913239+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:25.913145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.913145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:25.028457+0000","last_change":"2024-02-12T16:51:25.028420+0000","last_active":"2024-02-12T16:51:25.028457+0000","last_peered":"2024-02-12T16:51:25.028457+0000","last_clean":"2024-02-12T16:51:25.028457+0000","last_became_active":"2024-02-12T16:49:41.669427+0000","last_became_peered":"2024-02-12T16:49:41.669427+0000","last_unstale":"2024-02-12T16:51:25.028457+0000","last_undegraded":"2024-02-12T16:51:25.028457+0000","last_fullsized":"2024-02-12T16:51:25.028457+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:25.028339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.028339+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106992+0000","last_change":"2024-02-12T16:50:49.214010+0000","last_active":"2024-02-12T16:51:22.106992+0000","last_peered":"2024-02-12T16:51:22.106992+0000","last_clean":"2024-02-12T16:51:22.106992+0000","last_became_active":"2024-02-12T16:49:45.212245+0000","last_became_peered":"2024-02-12T16:49:45.212245+0000","last_unstale":"2024-02-12T16:51:22.106992+0000","last_undegraded":"2024-02-12T16:51:22.106992+0000","last_fullsized":"2024-02-12T16:51:22.106992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:50:49.213945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.229869+0000","last_change":"2024-02-12T16:51:09.159161+0000","last_active":"2024-02-12T16:51:37.229869+0000","last_peered":"2024-02-12T16:51:37.229869+0000","last_clean":"2024-02-12T16:51:37.229869+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:51:37.229869+0000","last_undegraded":"2024-02-12T16:51:37.229869+0000","last_fullsized":"2024-02-12T16:51:37.229869+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:09.159070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:09.159070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230070+0000","last_change":"2024-02-12T16:51:20.103380+0000","last_active":"2024-02-12T16:51:22.230070+0000","last_peered":"2024-02-12T16:51:22.230070+0000","last_clean":"2024-02-12T16:51:22.230070+0000","last_became_active":"2024-02-12T16:49:39.104936+0000","last_became_peered":"2024-02-12T16:49:39.104936+0000","last_unstale":"2024-02-12T16:51:22.230070+0000","last_undegraded":"2024-02-12T16:51:22.230070+0000","last_fullsized":"2024-02-12T16:51:22.230070+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.103297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.103297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:36.026655+0000","last_change":"2024-02-12T16:51:36.026655+0000","last_active":"2024-02-12T16:51:36.026655+0000","last_peered":"2024-02-12T16:51:36.026655+0000","last_clean":"2024-02-12T16:51:36.026655+0000","last_became_active":"2024-02-12T16:51:04.116639+0000","last_became_peered":"2024-02-12T16:51:04.116639+0000","last_unstale":"2024-02-12T16:51:36.026655+0000","last_undegraded":"2024-02-12T16:51:36.026655+0000","last_fullsized":"2024-02-12T16:51:36.026655+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:36.026569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:36.026569+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229707+0000","last_change":"2024-02-12T16:51:20.041621+0000","last_active":"2024-02-12T16:51:22.229707+0000","last_peered":"2024-02-12T16:51:22.229707+0000","last_clean":"2024-02-12T16:51:22.229707+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:51:22.229707+0000","last_undegraded":"2024-02-12T16:51:22.229707+0000","last_fullsized":"2024-02-12T16:51:22.229707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.041556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.041556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229369+0000","last_change":"2024-02-12T16:50:45.275564+0000","last_active":"2024-02-12T16:51:22.229369+0000","last_peered":"2024-02-12T16:51:22.229369+0000","last_clean":"2024-02-12T16:51:22.229369+0000","last_became_active":"2024-02-12T16:49:43.201595+0000","last_became_peered":"2024-02-12T16:49:43.201595+0000","last_unstale":"2024-02-12T16:51:22.229369+0000","last_undegraded":"2024-02-12T16:51:22.229369+0000","last_fullsized":"2024-02-12T16:51:22.229369+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:45.275469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:45.275469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228684+0000","last_change":"2024-02-12T16:51:19.027311+0000","last_active":"2024-02-12T16:51:22.228684+0000","last_peered":"2024-02-12T16:51:22.228684+0000","last_clean":"2024-02-12T16:51:22.228684+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:51:22.228684+0000","last_undegraded":"2024-02-12T16:51:22.228684+0000","last_fullsized":"2024-02-12T16:51:22.228684+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:19.027246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:19.027246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":50,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:28.108130+0000","last_change":"2024-02-12T16:51:28.108097+0000","last_active":"2024-02-12T16:51:28.108130+0000","last_peered":"2024-02-12T16:51:28.108130+0000","last_clean":"2024-02-12T16:51:28.108130+0000","last_became_active":"2024-02-12T16:49:41.669651+0000","last_became_peered":"2024-02-12T16:49:41.669651+0000","last_unstale":"2024-02-12T16:51:28.108130+0000","last_undegraded":"2024-02-12T16:51:28.108130+0000","last_fullsized":"2024-02-12T16:51:28.108130+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:51:28.108036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.108036+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128417+0000","last_change":"2024-02-12T16:51:03.235610+0000","last_active":"2024-02-12T16:51:22.128417+0000","last_peered":"2024-02-12T16:51:22.128417+0000","last_clean":"2024-02-12T16:51:22.128417+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:51:22.128417+0000","last_undegraded":"2024-02-12T16:51:22.128417+0000","last_fullsized":"2024-02-12T16:51:22.128417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:03.235544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.235544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:32.014878+0000","last_change":"2024-02-12T16:51:32.014849+0000","last_active":"2024-02-12T16:51:32.014878+0000","last_peered":"2024-02-12T16:51:32.014878+0000","last_clean":"2024-02-12T16:51:32.014878+0000","last_became_active":"2024-02-12T16:49:43.207454+0000","last_became_peered":"2024-02-12T16:49:43.207454+0000","last_unstale":"2024-02-12T16:51:32.014878+0000","last_undegraded":"2024-02-12T16:51:32.014878+0000","last_fullsized":"2024-02-12T16:51:32.014878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.014772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:39.307549+0000","last_change":"2024-02-12T16:51:39.307549+0000","last_active":"2024-02-12T16:51:39.307549+0000","last_peered":"2024-02-12T16:51:39.307549+0000","last_clean":"2024-02-12T16:51:39.307549+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:51:39.307549+0000","last_undegraded":"2024-02-12T16:51:39.307549+0000","last_fullsized":"2024-02-12T16:51:39.307549+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.307475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.307475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107066+0000","last_change":"2024-02-12T16:51:17.050140+0000","last_active":"2024-02-12T16:51:22.107066+0000","last_peered":"2024-02-12T16:51:22.107066+0000","last_clean":"2024-02-12T16:51:22.107066+0000","last_became_active":"2024-02-12T16:49:41.668665+0000","last_became_peered":"2024-02-12T16:49:41.668665+0000","last_unstale":"2024-02-12T16:51:22.107066+0000","last_undegraded":"2024-02-12T16:51:22.107066+0000","last_fullsized":"2024-02-12T16:51:22.107066+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:51:17.050077+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:33.032908+0000","last_change":"2024-02-12T16:51:33.032870+0000","last_active":"2024-02-12T16:51:33.032908+0000","last_peered":"2024-02-12T16:51:33.032908+0000","last_clean":"2024-02-12T16:51:33.032908+0000","last_became_active":"2024-02-12T16:49:45.220328+0000","last_became_peered":"2024-02-12T16:49:45.220328+0000","last_unstale":"2024-02-12T16:51:33.032908+0000","last_undegraded":"2024-02-12T16:51:33.032908+0000","last_fullsized":"2024-02-12T16:51:33.032908+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:33.032784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:33.032784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"58'2","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106489+0000","last_change":"2024-02-12T16:50:42.866331+0000","last_active":"2024-02-12T16:51:22.106489+0000","last_peered":"2024-02-12T16:51:22.106489+0000","last_clean":"2024-02-12T16:51:22.106489+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:51:22.106489+0000","last_undegraded":"2024-02-12T16:51:22.106489+0000","last_fullsized":"2024-02-12T16:51:22.106489+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-02-12T16:50:42.866263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.866263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106048+0000","last_change":"2024-02-12T16:50:46.750203+0000","last_active":"2024-02-12T16:51:22.106048+0000","last_peered":"2024-02-12T16:51:22.106048+0000","last_clean":"2024-02-12T16:51:22.106048+0000","last_became_active":"2024-02-12T16:49:41.177967+0000","last_became_peered":"2024-02-12T16:49:41.177967+0000","last_unstale":"2024-02-12T16:51:22.106048+0000","last_undegraded":"2024-02-12T16:51:22.106048+0000","last_fullsized":"2024-02-12T16:51:22.106048+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:50:46.750065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.750065+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128115+0000","last_change":"2024-02-12T16:51:22.127898+0000","last_active":"2024-02-12T16:51:22.128115+0000","last_peered":"2024-02-12T16:51:22.128115+0000","last_clean":"2024-02-12T16:51:22.128115+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:51:22.128115+0000","last_undegraded":"2024-02-12T16:51:22.128115+0000","last_fullsized":"2024-02-12T16:51:22.128115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.127831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107390+0000","last_change":"2024-02-12T16:50:50.186122+0000","last_active":"2024-02-12T16:51:22.107390+0000","last_peered":"2024-02-12T16:51:22.107390+0000","last_clean":"2024-02-12T16:51:22.107390+0000","last_became_active":"2024-02-12T16:49:45.217626+0000","last_became_peered":"2024-02-12T16:49:45.217626+0000","last_unstale":"2024-02-12T16:51:22.107390+0000","last_undegraded":"2024-02-12T16:51:22.107390+0000","last_fullsized":"2024-02-12T16:51:22.107390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:50.186021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.186021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:23.967951+0000","last_change":"2024-02-12T16:51:23.967921+0000","last_active":"2024-02-12T16:51:23.967951+0000","last_peered":"2024-02-12T16:51:23.967951+0000","last_clean":"2024-02-12T16:51:23.967951+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:51:23.967951+0000","last_undegraded":"2024-02-12T16:51:23.967951+0000","last_fullsized":"2024-02-12T16:51:23.967951+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:23.967848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:23.967848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128588+0000","last_change":"2024-02-12T16:51:04.121170+0000","last_active":"2024-02-12T16:51:22.128588+0000","last_peered":"2024-02-12T16:51:22.128588+0000","last_clean":"2024-02-12T16:51:22.128588+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:51:22.128588+0000","last_undegraded":"2024-02-12T16:51:22.128588+0000","last_fullsized":"2024-02-12T16:51:22.128588+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:50:42.876140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.876140+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.229818+0000","last_change":"2024-02-12T16:51:11.206549+0000","last_active":"2024-02-12T16:51:37.229818+0000","last_peered":"2024-02-12T16:51:37.229818+0000","last_clean":"2024-02-12T16:51:37.229818+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:51:37.229818+0000","last_undegraded":"2024-02-12T16:51:37.229818+0000","last_fullsized":"2024-02-12T16:51:37.229818+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:11.206420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.206420+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:26.163668+0000","last_change":"2024-02-12T16:51:26.163640+0000","last_active":"2024-02-12T16:51:26.163668+0000","last_peered":"2024-02-12T16:51:26.163668+0000","last_clean":"2024-02-12T16:51:26.163668+0000","last_became_active":"2024-02-12T16:49:45.215520+0000","last_became_peered":"2024-02-12T16:49:45.215520+0000","last_unstale":"2024-02-12T16:51:26.163668+0000","last_undegraded":"2024-02-12T16:51:26.163668+0000","last_fullsized":"2024-02-12T16:51:26.163668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:26.163568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:26.163568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230141+0000","last_change":"2024-02-12T16:50:50.253026+0000","last_active":"2024-02-12T16:51:22.230141+0000","last_peered":"2024-02-12T16:51:22.230141+0000","last_clean":"2024-02-12T16:51:22.230141+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:51:22.230141+0000","last_undegraded":"2024-02-12T16:51:22.230141+0000","last_fullsized":"2024-02-12T16:51:22.230141+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:50.252952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.252952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":115,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.878896+0000","last_change":"2024-02-12T16:51:37.878869+0000","last_active":"2024-02-12T16:51:37.878896+0000","last_peered":"2024-02-12T16:51:37.878896+0000","last_clean":"2024-02-12T16:51:37.878896+0000","last_became_active":"2024-02-12T16:49:41.179610+0000","last_became_peered":"2024-02-12T16:49:41.179610+0000","last_unstale":"2024-02-12T16:51:37.878896+0000","last_undegraded":"2024-02-12T16:51:37.878896+0000","last_fullsized":"2024-02-12T16:51:37.878896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-12T16:51:37.878788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.878788+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106869+0000","last_change":"2024-02-12T16:50:57.182750+0000","last_active":"2024-02-12T16:51:22.106869+0000","last_peered":"2024-02-12T16:51:22.106869+0000","last_clean":"2024-02-12T16:51:22.106869+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:51:22.106869+0000","last_undegraded":"2024-02-12T16:51:22.106869+0000","last_fullsized":"2024-02-12T16:51:22.106869+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:57.182679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.182679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229988+0000","last_change":"2024-02-12T16:51:22.062228+0000","last_active":"2024-02-12T16:51:22.229988+0000","last_peered":"2024-02-12T16:51:22.229988+0000","last_clean":"2024-02-12T16:51:22.229988+0000","last_became_active":"2024-02-12T16:49:45.221460+0000","last_became_peered":"2024-02-12T16:49:45.221460+0000","last_unstale":"2024-02-12T16:51:22.229988+0000","last_undegraded":"2024-02-12T16:51:22.229988+0000","last_fullsized":"2024-02-12T16:51:22.229988+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.062124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.062124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":164,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:41.525919+0000","last_change":"2024-02-12T16:50:49.760225+0000","last_active":"2024-02-12T16:51:41.525919+0000","last_peered":"2024-02-12T16:51:41.525919+0000","last_clean":"2024-02-12T16:51:41.525919+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:51:41.525919+0000","last_undegraded":"2024-02-12T16:51:41.525919+0000","last_fullsized":"2024-02-12T16:51:41.525919+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:50:49.760157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:49.760157+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":117,"num_read_kb":112,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128334+0000","last_change":"2024-02-12T16:51:02.278460+0000","last_active":"2024-02-12T16:51:22.128334+0000","last_peered":"2024-02-12T16:51:22.128334+0000","last_clean":"2024-02-12T16:51:22.128334+0000","last_became_active":"2024-02-12T16:49:39.105685+0000","last_became_peered":"2024-02-12T16:49:39.105685+0000","last_unstale":"2024-02-12T16:51:22.128334+0000","last_undegraded":"2024-02-12T16:51:22.128334+0000","last_fullsized":"2024-02-12T16:51:22.128334+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.278337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.278337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106958+0000","last_change":"2024-02-12T16:51:18.950103+0000","last_active":"2024-02-12T16:51:22.106958+0000","last_peered":"2024-02-12T16:51:22.106958+0000","last_clean":"2024-02-12T16:51:22.106958+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:51:22.106958+0000","last_undegraded":"2024-02-12T16:51:22.106958+0000","last_fullsized":"2024-02-12T16:51:22.106958+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:51:18.950031+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:51:18.950031+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.950031+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:24.083427+0000","last_change":"2024-02-12T16:51:24.083385+0000","last_active":"2024-02-12T16:51:24.083427+0000","last_peered":"2024-02-12T16:51:24.083427+0000","last_clean":"2024-02-12T16:51:24.083427+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:51:24.083427+0000","last_undegraded":"2024-02-12T16:51:24.083427+0000","last_fullsized":"2024-02-12T16:51:24.083427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.083270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.083270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230793+0000","last_change":"2024-02-12T16:51:00.237219+0000","last_active":"2024-02-12T16:51:22.230793+0000","last_peered":"2024-02-12T16:51:22.230793+0000","last_clean":"2024-02-12T16:51:22.230793+0000","last_became_active":"2024-02-12T16:49:45.213152+0000","last_became_peered":"2024-02-12T16:49:45.213152+0000","last_unstale":"2024-02-12T16:51:22.230793+0000","last_undegraded":"2024-02-12T16:51:22.230793+0000","last_fullsized":"2024-02-12T16:51:22.230793+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:51:00.237119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228271+0000","last_change":"2024-02-12T16:51:08.054217+0000","last_active":"2024-02-12T16:51:22.228271+0000","last_peered":"2024-02-12T16:51:22.228271+0000","last_clean":"2024-02-12T16:51:22.228271+0000","last_became_active":"2024-02-12T16:49:39.105761+0000","last_became_peered":"2024-02-12T16:49:39.105761+0000","last_unstale":"2024-02-12T16:51:22.228271+0000","last_undegraded":"2024-02-12T16:51:22.228271+0000","last_fullsized":"2024-02-12T16:51:22.228271+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:08.054114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.054114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228987+0000","last_change":"2024-02-12T16:50:56.320321+0000","last_active":"2024-02-12T16:51:22.228987+0000","last_peered":"2024-02-12T16:51:22.228987+0000","last_clean":"2024-02-12T16:51:22.228987+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:51:22.228987+0000","last_undegraded":"2024-02-12T16:51:22.228987+0000","last_fullsized":"2024-02-12T16:51:22.228987+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:56.320240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:56.320240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.184984+0000","last_change":"2024-02-12T16:51:37.184984+0000","last_active":"2024-02-12T16:51:37.184984+0000","last_peered":"2024-02-12T16:51:37.184984+0000","last_clean":"2024-02-12T16:51:37.184984+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:51:37.184984+0000","last_undegraded":"2024-02-12T16:51:37.184984+0000","last_fullsized":"2024-02-12T16:51:37.184984+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:37.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.184907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230258+0000","last_change":"2024-02-12T16:50:57.964701+0000","last_active":"2024-02-12T16:51:22.230258+0000","last_peered":"2024-02-12T16:51:22.230258+0000","last_clean":"2024-02-12T16:51:22.230258+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:51:22.230258+0000","last_undegraded":"2024-02-12T16:51:22.230258+0000","last_fullsized":"2024-02-12T16:51:22.230258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:50:57.964625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.964625+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229610+0000","last_change":"2024-02-12T16:51:05.047096+0000","last_active":"2024-02-12T16:51:22.229610+0000","last_peered":"2024-02-12T16:51:22.229610+0000","last_clean":"2024-02-12T16:51:22.229610+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:51:22.229610+0000","last_undegraded":"2024-02-12T16:51:22.229610+0000","last_fullsized":"2024-02-12T16:51:22.229610+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:05.047021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:05.047021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:36.842712+0000","last_change":"2024-02-12T16:51:36.842712+0000","last_active":"2024-02-12T16:51:36.842712+0000","last_peered":"2024-02-12T16:51:36.842712+0000","last_clean":"2024-02-12T16:51:36.842712+0000","last_became_active":"2024-02-12T16:49:45.218096+0000","last_became_peered":"2024-02-12T16:49:45.218096+0000","last_unstale":"2024-02-12T16:51:36.842712+0000","last_undegraded":"2024-02-12T16:51:36.842712+0000","last_fullsized":"2024-02-12T16:51:36.842712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:36.842645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:36.842645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106219+0000","last_change":"2024-02-12T16:51:18.079605+0000","last_active":"2024-02-12T16:51:22.106219+0000","last_peered":"2024-02-12T16:51:22.106219+0000","last_clean":"2024-02-12T16:51:22.106219+0000","last_became_active":"2024-02-12T16:49:39.104017+0000","last_became_peered":"2024-02-12T16:49:39.104017+0000","last_unstale":"2024-02-12T16:51:22.106219+0000","last_undegraded":"2024-02-12T16:51:22.106219+0000","last_fullsized":"2024-02-12T16:51:22.106219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:18.079497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.079497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228728+0000","last_change":"2024-02-12T16:51:08.953016+0000","last_active":"2024-02-12T16:51:22.228728+0000","last_peered":"2024-02-12T16:51:22.228728+0000","last_clean":"2024-02-12T16:51:22.228728+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:51:22.228728+0000","last_undegraded":"2024-02-12T16:51:22.228728+0000","last_fullsized":"2024-02-12T16:51:22.228728+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:51:08.952901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.952901+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107040+0000","last_change":"2024-02-12T16:51:07.240302+0000","last_active":"2024-02-12T16:51:22.107040+0000","last_peered":"2024-02-12T16:51:22.107040+0000","last_clean":"2024-02-12T16:51:22.107040+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:51:22.107040+0000","last_undegraded":"2024-02-12T16:51:22.107040+0000","last_fullsized":"2024-02-12T16:51:22.107040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:51:07.240236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:07.240236+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230709+0000","last_change":"2024-02-12T16:51:22.070134+0000","last_active":"2024-02-12T16:51:22.230709+0000","last_peered":"2024-02-12T16:51:22.230709+0000","last_clean":"2024-02-12T16:51:22.230709+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:51:22.230709+0000","last_undegraded":"2024-02-12T16:51:22.230709+0000","last_fullsized":"2024-02-12T16:51:22.230709+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.070056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.070056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:32.884762+0000","last_change":"2024-02-12T16:51:32.884735+0000","last_active":"2024-02-12T16:51:32.884762+0000","last_peered":"2024-02-12T16:51:32.884762+0000","last_clean":"2024-02-12T16:51:32.884762+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:51:32.884762+0000","last_undegraded":"2024-02-12T16:51:32.884762+0000","last_fullsized":"2024-02-12T16:51:32.884762+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.884670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230018+0000","last_change":"2024-02-12T16:50:56.149313+0000","last_active":"2024-02-12T16:51:22.230018+0000","last_peered":"2024-02-12T16:51:22.230018+0000","last_clean":"2024-02-12T16:51:22.230018+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:51:22.230018+0000","last_undegraded":"2024-02-12T16:51:22.230018+0000","last_fullsized":"2024-02-12T16:51:22.230018+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:56.149242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:56.149242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106052+0000","last_change":"2024-02-12T16:50:43.246481+0000","last_active":"2024-02-12T16:51:22.106052+0000","last_peered":"2024-02-12T16:51:22.106052+0000","last_clean":"2024-02-12T16:51:22.106052+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:51:22.106052+0000","last_undegraded":"2024-02-12T16:51:22.106052+0000","last_fullsized":"2024-02-12T16:51:22.106052+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:50:43.246399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:43.246399+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230675+0000","last_change":"2024-02-12T16:51:10.119500+0000","last_active":"2024-02-12T16:51:22.230675+0000","last_peered":"2024-02-12T16:51:22.230675+0000","last_clean":"2024-02-12T16:51:22.230675+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:51:22.230675+0000","last_undegraded":"2024-02-12T16:51:22.230675+0000","last_fullsized":"2024-02-12T16:51:22.230675+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:10.119399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:10.119399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:32.837050+0000","last_change":"2024-02-12T16:51:32.837004+0000","last_active":"2024-02-12T16:51:32.837050+0000","last_peered":"2024-02-12T16:51:32.837050+0000","last_clean":"2024-02-12T16:51:32.837050+0000","last_became_active":"2024-02-12T16:49:45.222560+0000","last_became_peered":"2024-02-12T16:49:45.222560+0000","last_unstale":"2024-02-12T16:51:32.837050+0000","last_undegraded":"2024-02-12T16:51:32.837050+0000","last_fullsized":"2024-02-12T16:51:32.837050+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.836928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.836928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.107458+0000","last_change":"2024-02-12T16:51:16.010678+0000","last_active":"2024-02-12T16:51:22.107458+0000","last_peered":"2024-02-12T16:51:22.107458+0000","last_clean":"2024-02-12T16:51:22.107458+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:51:22.107458+0000","last_undegraded":"2024-02-12T16:51:22.107458+0000","last_fullsized":"2024-02-12T16:51:22.107458+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:16.010512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.010512+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228202+0000","last_change":"2024-02-12T16:49:41.670234+0000","last_active":"2024-02-12T16:51:22.228202+0000","last_peered":"2024-02-12T16:51:22.228202+0000","last_clean":"2024-02-12T16:51:22.228202+0000","last_became_active":"2024-02-12T16:49:41.669593+0000","last_became_peered":"2024-02-12T16:49:41.669593+0000","last_unstale":"2024-02-12T16:51:22.228202+0000","last_undegraded":"2024-02-12T16:51:22.228202+0000","last_fullsized":"2024-02-12T16:51:22.228202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:49:40.156223+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:35.749314+0000","last_change":"2024-02-12T16:51:35.749314+0000","last_active":"2024-02-12T16:51:35.749314+0000","last_peered":"2024-02-12T16:51:35.749314+0000","last_clean":"2024-02-12T16:51:35.749314+0000","last_became_active":"2024-02-12T16:49:43.202312+0000","last_became_peered":"2024-02-12T16:49:43.202312+0000","last_unstale":"2024-02-12T16:51:35.749314+0000","last_undegraded":"2024-02-12T16:51:35.749314+0000","last_fullsized":"2024-02-12T16:51:35.749314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:35.749237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:35.749237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229911+0000","last_change":"2024-02-12T16:50:55.111751+0000","last_active":"2024-02-12T16:51:22.229911+0000","last_peered":"2024-02-12T16:51:22.229911+0000","last_clean":"2024-02-12T16:51:22.229911+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:51:22.229911+0000","last_undegraded":"2024-02-12T16:51:22.229911+0000","last_fullsized":"2024-02-12T16:51:22.229911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:50:55.111675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.111675+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230072+0000","last_change":"2024-02-12T16:51:16.020537+0000","last_active":"2024-02-12T16:51:22.230072+0000","last_peered":"2024-02-12T16:51:22.230072+0000","last_clean":"2024-02-12T16:51:22.230072+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:51:22.230072+0000","last_undegraded":"2024-02-12T16:51:22.230072+0000","last_fullsized":"2024-02-12T16:51:22.230072+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:16.020471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.020471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105929+0000","last_change":"2024-02-12T16:50:59.823981+0000","last_active":"2024-02-12T16:51:22.105929+0000","last_peered":"2024-02-12T16:51:22.105929+0000","last_clean":"2024-02-12T16:51:22.105929+0000","last_became_active":"2024-02-12T16:49:41.178569+0000","last_became_peered":"2024-02-12T16:49:41.178569+0000","last_unstale":"2024-02-12T16:51:22.105929+0000","last_undegraded":"2024-02-12T16:51:22.105929+0000","last_fullsized":"2024-02-12T16:51:22.105929+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:50:59.823898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:59.823898+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.229857+0000","last_change":"2024-02-12T16:51:03.961580+0000","last_active":"2024-02-12T16:51:37.229857+0000","last_peered":"2024-02-12T16:51:37.229857+0000","last_clean":"2024-02-12T16:51:37.229857+0000","last_became_active":"2024-02-12T16:49:43.205391+0000","last_became_peered":"2024-02-12T16:49:43.205391+0000","last_unstale":"2024-02-12T16:51:37.229857+0000","last_undegraded":"2024-02-12T16:51:37.229857+0000","last_fullsized":"2024-02-12T16:51:37.229857+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:03.961503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.961503+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:34.081749+0000","last_change":"2024-02-12T16:51:34.081725+0000","last_active":"2024-02-12T16:51:34.081749+0000","last_peered":"2024-02-12T16:51:34.081749+0000","last_clean":"2024-02-12T16:51:34.081749+0000","last_became_active":"2024-02-12T16:49:45.223659+0000","last_became_peered":"2024-02-12T16:49:45.223659+0000","last_unstale":"2024-02-12T16:51:34.081749+0000","last_undegraded":"2024-02-12T16:51:34.081749+0000","last_fullsized":"2024-02-12T16:51:34.081749+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:34.081644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.081644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230576+0000","last_change":"2024-02-12T16:51:05.201812+0000","last_active":"2024-02-12T16:51:22.230576+0000","last_peered":"2024-02-12T16:51:22.230576+0000","last_clean":"2024-02-12T16:51:22.230576+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:51:22.230576+0000","last_undegraded":"2024-02-12T16:51:22.230576+0000","last_fullsized":"2024-02-12T16:51:22.230576+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:51:05.201749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128212+0000","last_change":"2024-02-12T16:51:04.206339+0000","last_active":"2024-02-12T16:51:22.128212+0000","last_peered":"2024-02-12T16:51:22.128212+0000","last_clean":"2024-02-12T16:51:22.128212+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:51:22.128212+0000","last_undegraded":"2024-02-12T16:51:22.128212+0000","last_fullsized":"2024-02-12T16:51:22.128212+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.123059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.123059+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.123059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:30.776982+0000","last_change":"2024-02-12T16:51:30.776952+0000","last_active":"2024-02-12T16:51:30.776982+0000","last_peered":"2024-02-12T16:51:30.776982+0000","last_clean":"2024-02-12T16:51:30.776982+0000","last_became_active":"2024-02-12T16:49:41.178616+0000","last_became_peered":"2024-02-12T16:49:41.178616+0000","last_unstale":"2024-02-12T16:51:30.776982+0000","last_undegraded":"2024-02-12T16:51:30.776982+0000","last_fullsized":"2024-02-12T16:51:30.776982+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:30.776871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:30.776871+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229287+0000","last_change":"2024-02-12T16:51:02.216274+0000","last_active":"2024-02-12T16:51:22.229287+0000","last_peered":"2024-02-12T16:51:22.229287+0000","last_clean":"2024-02-12T16:51:22.229287+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:51:22.229287+0000","last_undegraded":"2024-02-12T16:51:22.229287+0000","last_fullsized":"2024-02-12T16:51:22.229287+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.216194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.216194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"80'9","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.229869+0000","last_change":"2024-02-12T16:50:50.800052+0000","last_active":"2024-02-12T16:51:37.229869+0000","last_peered":"2024-02-12T16:51:37.229869+0000","last_clean":"2024-02-12T16:51:37.229869+0000","last_became_active":"2024-02-12T16:49:43.197112+0000","last_became_peered":"2024-02-12T16:49:43.197112+0000","last_unstale":"2024-02-12T16:51:37.229869+0000","last_undegraded":"2024-02-12T16:51:37.229869+0000","last_fullsized":"2024-02-12T16:51:37.229869+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.799984+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105725+0000","last_change":"2024-02-12T16:51:00.778646+0000","last_active":"2024-02-12T16:51:22.105725+0000","last_peered":"2024-02-12T16:51:22.105725+0000","last_clean":"2024-02-12T16:51:22.105725+0000","last_became_active":"2024-02-12T16:49:39.095868+0000","last_became_peered":"2024-02-12T16:49:39.095868+0000","last_unstale":"2024-02-12T16:51:22.105725+0000","last_undegraded":"2024-02-12T16:51:22.105725+0000","last_fullsized":"2024-02-12T16:51:22.105725+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:00.778578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:00.778578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106912+0000","last_change":"2024-02-12T16:51:11.980641+0000","last_active":"2024-02-12T16:51:22.106912+0000","last_peered":"2024-02-12T16:51:22.106912+0000","last_clean":"2024-02-12T16:51:22.106912+0000","last_became_active":"2024-02-12T16:49:41.178544+0000","last_became_peered":"2024-02-12T16:49:41.178544+0000","last_unstale":"2024-02-12T16:51:22.106912+0000","last_undegraded":"2024-02-12T16:51:22.106912+0000","last_fullsized":"2024-02-12T16:51:22.106912+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:51:11.980577+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:51:11.980577+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.980577+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:38.773791+0000","last_change":"2024-02-12T16:51:38.773791+0000","last_active":"2024-02-12T16:51:38.773791+0000","last_peered":"2024-02-12T16:51:38.773791+0000","last_clean":"2024-02-12T16:51:38.773791+0000","last_became_active":"2024-02-12T16:49:45.217458+0000","last_became_peered":"2024-02-12T16:49:45.217458+0000","last_unstale":"2024-02-12T16:51:38.773791+0000","last_undegraded":"2024-02-12T16:51:38.773791+0000","last_fullsized":"2024-02-12T16:51:38.773791+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.773699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.773699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.229831+0000","last_change":"2024-02-12T16:51:16.145692+0000","last_active":"2024-02-12T16:51:37.229831+0000","last_peered":"2024-02-12T16:51:37.229831+0000","last_clean":"2024-02-12T16:51:37.229831+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:51:37.229831+0000","last_undegraded":"2024-02-12T16:51:37.229831+0000","last_fullsized":"2024-02-12T16:51:37.229831+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:16.145576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.145576+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105910+0000","last_change":"2024-02-12T16:51:02.785509+0000","last_active":"2024-02-12T16:51:22.105910+0000","last_peered":"2024-02-12T16:51:22.105910+0000","last_clean":"2024-02-12T16:51:22.105910+0000","last_became_active":"2024-02-12T16:49:39.101478+0000","last_became_peered":"2024-02-12T16:49:39.101478+0000","last_unstale":"2024-02-12T16:51:22.105910+0000","last_undegraded":"2024-02-12T16:51:22.105910+0000","last_fullsized":"2024-02-12T16:51:22.105910+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.785426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.785426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106313+0000","last_change":"2024-02-12T16:51:04.882407+0000","last_active":"2024-02-12T16:51:22.106313+0000","last_peered":"2024-02-12T16:51:22.106313+0000","last_clean":"2024-02-12T16:51:22.106313+0000","last_became_active":"2024-02-12T16:49:41.179396+0000","last_became_peered":"2024-02-12T16:49:41.179396+0000","last_unstale":"2024-02-12T16:51:22.106313+0000","last_undegraded":"2024-02-12T16:51:22.106313+0000","last_fullsized":"2024-02-12T16:51:22.106313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:04.882334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:04.882334+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128544+0000","last_change":"2024-02-12T16:50:55.285750+0000","last_active":"2024-02-12T16:51:22.128544+0000","last_peered":"2024-02-12T16:51:22.128544+0000","last_clean":"2024-02-12T16:51:22.128544+0000","last_became_active":"2024-02-12T16:49:45.220893+0000","last_became_peered":"2024-02-12T16:49:45.220893+0000","last_unstale":"2024-02-12T16:51:22.128544+0000","last_undegraded":"2024-02-12T16:51:22.128544+0000","last_fullsized":"2024-02-12T16:51:22.128544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.285685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.285685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106984+0000","last_change":"2024-02-12T16:51:13.985683+0000","last_active":"2024-02-12T16:51:22.106984+0000","last_peered":"2024-02-12T16:51:22.106984+0000","last_clean":"2024-02-12T16:51:22.106984+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:51:22.106984+0000","last_undegraded":"2024-02-12T16:51:22.106984+0000","last_fullsized":"2024-02-12T16:51:22.106984+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.985614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.985614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.229102+0000","last_change":"2024-02-12T16:50:44.092086+0000","last_active":"2024-02-12T16:51:22.229102+0000","last_peered":"2024-02-12T16:51:22.229102+0000","last_clean":"2024-02-12T16:51:22.229102+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:51:22.229102+0000","last_undegraded":"2024-02-12T16:51:22.229102+0000","last_fullsized":"2024-02-12T16:51:22.229102+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:44.092019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:44.092019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106721+0000","last_change":"2024-02-12T16:51:12.892241+0000","last_active":"2024-02-12T16:51:22.106721+0000","last_peered":"2024-02-12T16:51:22.106721+0000","last_clean":"2024-02-12T16:51:22.106721+0000","last_became_active":"2024-02-12T16:49:41.177416+0000","last_became_peered":"2024-02-12T16:49:41.177416+0000","last_unstale":"2024-02-12T16:51:22.106721+0000","last_undegraded":"2024-02-12T16:51:22.106721+0000","last_fullsized":"2024-02-12T16:51:22.106721+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:51:12.892148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:12.892148+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228631+0000","last_change":"2024-02-12T16:50:54.135100+0000","last_active":"2024-02-12T16:51:22.228631+0000","last_peered":"2024-02-12T16:51:22.228631+0000","last_clean":"2024-02-12T16:51:22.228631+0000","last_became_active":"2024-02-12T16:49:45.217589+0000","last_became_peered":"2024-02-12T16:49:45.217589+0000","last_unstale":"2024-02-12T16:51:22.228631+0000","last_undegraded":"2024-02-12T16:51:22.228631+0000","last_fullsized":"2024-02-12T16:51:22.228631+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.135009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128264+0000","last_change":"2024-02-12T16:50:52.289793+0000","last_active":"2024-02-12T16:51:22.128264+0000","last_peered":"2024-02-12T16:51:22.128264+0000","last_clean":"2024-02-12T16:51:22.128264+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:51:22.128264+0000","last_undegraded":"2024-02-12T16:51:22.128264+0000","last_fullsized":"2024-02-12T16:51:22.128264+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:52.289725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:52.289725+0000","last_clean_scrub_stamp":"2024-02-12T16:50:52.289725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128562+0000","last_change":"2024-02-12T16:50:51.308424+0000","last_active":"2024-02-12T16:51:22.128562+0000","last_peered":"2024-02-12T16:51:22.128562+0000","last_clean":"2024-02-12T16:51:22.128562+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:51:22.128562+0000","last_undegraded":"2024-02-12T16:51:22.128562+0000","last_fullsized":"2024-02-12T16:51:22.128562+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.308317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.308317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.230573+0000","last_change":"2024-02-12T16:51:04.126502+0000","last_active":"2024-02-12T16:51:22.230573+0000","last_peered":"2024-02-12T16:51:22.230573+0000","last_clean":"2024-02-12T16:51:22.230573+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:51:22.230573+0000","last_undegraded":"2024-02-12T16:51:22.230573+0000","last_fullsized":"2024-02-12T16:51:22.230573+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:50:47.792388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.792388+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.105882+0000","last_change":"2024-02-12T16:51:13.192907+0000","last_active":"2024-02-12T16:51:22.105882+0000","last_peered":"2024-02-12T16:51:22.105882+0000","last_clean":"2024-02-12T16:51:22.105882+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:51:22.105882+0000","last_undegraded":"2024-02-12T16:51:22.105882+0000","last_fullsized":"2024-02-12T16:51:22.105882+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.192842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.192842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:40.273267+0000","last_change":"2024-02-12T16:51:40.273267+0000","last_active":"2024-02-12T16:51:40.273267+0000","last_peered":"2024-02-12T16:51:40.273267+0000","last_clean":"2024-02-12T16:51:40.273267+0000","last_became_active":"2024-02-12T16:49:43.201946+0000","last_became_peered":"2024-02-12T16:49:43.201946+0000","last_unstale":"2024-02-12T16:51:40.273267+0000","last_undegraded":"2024-02-12T16:51:40.273267+0000","last_fullsized":"2024-02-12T16:51:40.273267+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:40.273180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:40.273180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"58'1","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:31.889946+0000","last_change":"2024-02-12T16:51:31.889909+0000","last_active":"2024-02-12T16:51:31.889946+0000","last_peered":"2024-02-12T16:51:31.889946+0000","last_clean":"2024-02-12T16:51:31.889946+0000","last_became_active":"2024-02-12T16:49:39.099695+0000","last_became_peered":"2024-02-12T16:49:39.099695+0000","last_unstale":"2024-02-12T16:51:31.889946+0000","last_undegraded":"2024-02-12T16:51:31.889946+0000","last_fullsized":"2024-02-12T16:51:31.889946+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:51:31.889821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:31.889821+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.106218+0000","last_change":"2024-02-12T16:50:45.175080+0000","last_active":"2024-02-12T16:51:22.106218+0000","last_peered":"2024-02-12T16:51:22.106218+0000","last_clean":"2024-02-12T16:51:22.106218+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:51:22.106218+0000","last_undegraded":"2024-02-12T16:51:22.106218+0000","last_fullsized":"2024-02-12T16:51:22.106218+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:50:45.174941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:45.174941+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:24.030122+0000","last_change":"2024-02-12T16:51:24.030089+0000","last_active":"2024-02-12T16:51:24.030122+0000","last_peered":"2024-02-12T16:51:24.030122+0000","last_clean":"2024-02-12T16:51:24.030122+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:51:24.030122+0000","last_undegraded":"2024-02-12T16:51:24.030122+0000","last_fullsized":"2024-02-12T16:51:24.030122+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.030017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.030017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.128833+0000","last_change":"2024-02-12T16:50:43.269381+0000","last_active":"2024-02-12T16:51:22.128833+0000","last_peered":"2024-02-12T16:51:22.128833+0000","last_clean":"2024-02-12T16:51:22.128833+0000","last_became_active":"2024-02-12T16:49:39.105982+0000","last_became_peered":"2024-02-12T16:49:39.105982+0000","last_unstale":"2024-02-12T16:51:22.128833+0000","last_undegraded":"2024-02-12T16:51:22.128833+0000","last_fullsized":"2024-02-12T16:51:22.128833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:43.269296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:43.269296+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-02-12T16:51:22.228632+0000","last_change":"2024-02-12T16:50:54.302143+0000","last_active":"2024-02-12T16:51:22.228632+0000","last_peered":"2024-02-12T16:51:22.228632+0000","last_clean":"2024-02-12T16:51:22.228632+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:51:22.228632+0000","last_undegraded":"2024-02-12T16:51:22.228632+0000","last_fullsized":"2024-02-12T16:51:22.228632+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:50:54.302060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.302060+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-12T16:51:37.229874+0000","last_change":"2024-02-12T16:51:07.021402+0000","last_active":"2024-02-12T16:51:37.229874+0000","last_peered":"2024-02-12T16:51:37.229874+0000","last_clean":"2024-02-12T16:51:37.229874+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:51:37.229874+0000","last_undegraded":"2024-02-12T16:51:37.229874+0000","last_fullsized":"2024-02-12T16:51:37.229874+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:07.021338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:07.021338+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":117,"num_read_kb":112,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234012,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297756,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447396,"statfs":{"total":95995035648,"available":95690133504,"internally_reserved":0,"allocated":634880,"data_stored":292275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69799999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80500000000000005,"5min":1.056,"15min":1.056},"last":0.63500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65900000000000003,"5min":1.208,"15min":1.208},"last":0.69899999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56899999999999995,"15min":0.57999999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72299999999999998,"5min":1.766,"15min":1.766},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.61299999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.51100000000000001,"15min":0.52200000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67000000000000004,"5min":1.327,"15min":1.327},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68400000000000005,"5min":1.573,"15min":1.573},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53500000000000003,"15min":0.54300000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76400000000000001,"5min":1.621,"15min":1.621},"last":0.61599999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77700000000000002,"5min":1.272,"15min":1.272},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73199999999999998,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.64400000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528353,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297736,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447416,"statfs":{"total":95995035648,"available":95690153984,"internally_reserved":0,"allocated":614400,"data_stored":291625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48199999999999998,"15min":0.48299999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67000000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.53200000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65700000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.747,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.47799999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72799999999999998,"5min":0.754,"15min":0.754},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.50600000000000001,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64500000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.62}]},{"osd":3,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.39800000000000002},"min":{"1min":0.28699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.42399999999999999,"15min":0.41799999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.318,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:51:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":294352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35399999999999998,"15min":0.34000000000000002},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39100000000000001,"15min":0.373},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.74199999999999999,"5min":2.778,"15min":2.778},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41499999999999998,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.629,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.45600000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.42099999999999999,"15min":0.40899999999999997},"min":{"1min":0.35199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:51:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.40899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77300000000000002,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77200000000000002,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.96799999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.94299999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.84799999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:51:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.64800000000000002,"15min":0.65800000000000003},"min":{"1min":0.438,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79900000000000004,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83999999999999997,"5min":1.109,"15min":1.109},"last":0.81699999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":630784,"data_stored":291959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"min":{"1min":0.255,"5min":0.192,"15min":0.192},"max":{"1min":0.55300000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.251},{"interface":"front","average":{"1min":0.37,"5min":0.35299999999999998,"15min":0.317},"min":{"1min":0.27700000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.61599999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.42299999999999999,"15min":0.39100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.8130000000000002,"5min":3.8130000000000002,"15min":3.8130000000000002},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67400000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.40500000000000003,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56399999999999995,"5min":0.622,"15min":0.622},"last":0.313},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.45800000000000002,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.33500000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.745,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67300000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59099999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.73099999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.499}]},{"osd":6,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71299999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.45700000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86099999999999999,"5min":0.871,"15min":0.871},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59899999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.54400000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297704,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447448,"statfs":{"total":95995035648,"available":95690186752,"internally_reserved":0,"allocated":622592,"data_stored":292183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.38700000000000001,"15min":0.379},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46000000000000002,"5min":1.595,"15min":1.595},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.41999999999999998,"15min":0.40000000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47799999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.46300000000000002,"15min":0.41699999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.52000000000000002,"5min":6.8040000000000003,"15min":6.8040000000000003},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.437,"15min":0.41099999999999998},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51100000000000001,"5min":1.627,"15min":1.627},"last":0.40500000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47199999999999998,"15min":0.46600000000000003},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.57599999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.503,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.74199999999999999,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.70499999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.61499999999999999,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.69099999999999995,"5min":2.875,"15min":2.875},"last":0.61399999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.71599999999999997,"15min":0.74199999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82099999999999995,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90400000000000003,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":0.54300000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.89700000000000002,"15min":0.95299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70299999999999996,"5min":7.5629999999999997,"15min":7.5629999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.47699999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.69799999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.624}]}]},{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":293275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.436,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.314},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52100000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.307}]},{"osd":1,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.42599999999999999,"15min":0.436},"min":{"1min":0.255,"5min":0.216,"15min":0.216},"max":{"1min":0.52300000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42899999999999999,"15min":0.436},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.503,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.23100000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.442,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.436,"15min":0.44400000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.503,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.34599999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58899999999999997,"15min":0.61099999999999999},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78100000000000003,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.435},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73699999999999999,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.621,"15min":0.63600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.754,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.61899999999999999,"15min":0.63100000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.69699999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76500000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.442},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.432,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.60499999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953515,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":292628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.55300000000000005},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.51800000000000002},"min":{"1min":0.30399999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.505}]},{"osd":1,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.30299999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66300000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.57199999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.623},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.66200000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53700000000000003,"15min":0.55400000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:50:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33200000000000002},"min":{"1min":0.27100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.51800000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.376,"15min":0.371},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.64100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.32600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45800000000000002,"15min":0.44600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.56000000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757287,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297688,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447464,"statfs":{"total":95995035648,"available":95690203136,"internally_reserved":0,"allocated":606208,"data_stored":291233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51600000000000001,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.76000000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72699999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.78800000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.34300000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.71299999999999997,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.501,"15min":0.502},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.61499999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52300000000000002,"15min":0.53000000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.73799999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.51000000000000001,"15min":0.51100000000000001},"min":{"1min":0.374,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70299999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:50:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.38},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.629}]},{"osd":6,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.44500000000000001,"15min":0.439},"min":{"1min":0.35499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.70499999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:51:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:51:46.641 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-12T16:51:46.641 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-12T16:51:46.641 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-12T16:51:46.642 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph health --format=json 2024-02-12T16:51:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:45.415932+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-12T16:51:46.922 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:47.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:45.839350+0000 mgr.y (mgr.24481) 87 : audit [DBG] from='client.14739 v1:172.21.15.33:0/4102140965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:47.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:46.297734+0000 mgr.y (mgr.24481) 88 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:47.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:47.040911+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:51:47.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:47.043889+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:51:47.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:45.839350+0000 mgr.y (mgr.24481) 87 : audit [DBG] from='client.14739 v1:172.21.15.33:0/4102140965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:47.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:46.297734+0000 mgr.y (mgr.24481) 88 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:47.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:47.040911+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:51:47.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:47.043889+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:51:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:45.839350+0000 mgr.y (mgr.24481) 87 : audit [DBG] from='client.14739 v1:172.21.15.33:0/4102140965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:46.297734+0000 mgr.y (mgr.24481) 88 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:47.040911+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:51:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:47.043889+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:51:48.888 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:51:49.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:48.298286+0000 mgr.y (mgr.24481) 89 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:49.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:48.885928+0000 mon.a (mon.0) 725 : audit [DBG] from='client.? v1:172.21.15.33:0/3925960850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-12T16:51:49.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:48.298286+0000 mgr.y (mgr.24481) 89 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:49.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:48.885928+0000 mon.a (mon.0) 725 : audit [DBG] from='client.? v1:172.21.15.33:0/3925960850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-12T16:51:49.683 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-12T16:51:49.684 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-12T16:51:49.684 INFO:tasks.cephadm:Setup complete, yielding 2024-02-12T16:51:49.684 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-12T16:51:49.696 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-12T16:51:49.696 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri10-testing-2024-02-08-0854-pacific'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)'], 'mon_bind_msgr2': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-tracker-64343-pacific', 'sha1': '57bd6abdec7bb457ae7999d9c96682e9ac678e27'}} 2024-02-12T16:51:49.696 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-12T16:51:49.696 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-12T16:51:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:48.298286+0000 mgr.y (mgr.24481) 89 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:51:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:48.885928+0000 mon.a (mon.0) 725 : audit [DBG] from='client.? v1:172.21.15.33:0/3925960850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-12T16:51:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:49.300055+0000 osd.1 (osd.1) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:49.300128+0000 osd.1 (osd.1) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:49.429897+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-12T16:51:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:49.300055+0000 osd.1 (osd.1) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:49.300128+0000 osd.1 (osd.1) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:49.429897+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-12T16:51:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:49.300055+0000 osd.1 (osd.1) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:49.300128+0000 osd.1 (osd.1) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:49.429897+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-12T16:51:51.193 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:50] "GET /metrics HTTP/1.1" 200 172145 "" "Prometheus/2.33.4" 2024-02-12T16:51:51.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:50.298592+0000 mgr.y (mgr.24481) 90 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:51.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:51.091477+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:51:51.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:51.095120+0000 osd.3 (osd.3) 52 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:51:51.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:50.298592+0000 mgr.y (mgr.24481) 90 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:51.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:51.091477+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:51:51.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:51.095120+0000 osd.3 (osd.3) 52 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:51:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:50.298592+0000 mgr.y (mgr.24481) 90 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:51.091477+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:51:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:51.095120+0000 osd.3 (osd.3) 52 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:51:52.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:51] ENGINE Bus STOPPING 2024-02-12T16:51:52.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-12T16:51:52.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:51] ENGINE Bus STOPPED 2024-02-12T16:51:52.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:51] ENGINE Bus STARTING 2024-02-12T16:51:52.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:52] ENGINE Serving on http://:::9283 2024-02-12T16:51:52.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:16:51:52] ENGINE Bus STARTED 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:51.360531+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.e scrub starts 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:51.361533+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.e scrub ok 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:51.579879+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:51.579985+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:51.924192+0000 mon.a (mon.0) 727 : audit [INF] from='client.34676 ' entity='client.admin' 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:51.934599+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:51.935947+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:51.936537+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:52.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:51.943268+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:51.360531+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.e scrub starts 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:51.361533+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.e scrub ok 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:51.579879+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:51.579985+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:51.924192+0000 mon.a (mon.0) 727 : audit [INF] from='client.34676 ' entity='client.admin' 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:51.934599+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:51.935947+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:51.936537+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:51.943268+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:52.682 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-12T16:51:52.683 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-12T16:51:52.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:51:52.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:51:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:51:52.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:51.360531+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.e scrub starts 2024-02-12T16:51:52.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:51.361533+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.e scrub ok 2024-02-12T16:51:52.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:51.579879+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:52.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:51.579985+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:52.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:51.924192+0000 mon.a (mon.0) 727 : audit [INF] from='client.34676 ' entity='client.admin' 2024-02-12T16:51:52.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:51.934599+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:52.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:51.935947+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:52.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:51.936537+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:52.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:51.943268+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:51:52.963 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:52.299003+0000 mgr.y (mgr.24481) 91 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:51:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:52.450844+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-12T16:51:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:52.741502+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:51:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:52.745181+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:51:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:53.005081+0000 osd.7 (osd.7) 41 : cluster [DBG] 2.b scrub starts 2024-02-12T16:51:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:53.007984+0000 osd.7 (osd.7) 42 : cluster [DBG] 2.b scrub ok 2024-02-12T16:51:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:52.299003+0000 mgr.y (mgr.24481) 91 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:51:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:52.450844+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-12T16:51:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:52.741502+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:51:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:52.745181+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:51:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:53.005081+0000 osd.7 (osd.7) 41 : cluster [DBG] 2.b scrub starts 2024-02-12T16:51:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:53.007984+0000 osd.7 (osd.7) 42 : cluster [DBG] 2.b scrub ok 2024-02-12T16:51:53.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:51:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:51:53.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:52.299003+0000 mgr.y (mgr.24481) 91 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:51:53.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:52.450844+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-12T16:51:53.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:52.741502+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:51:53.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:52.745181+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:51:53.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:53.005081+0000 osd.7 (osd.7) 41 : cluster [DBG] 2.b scrub starts 2024-02-12T16:51:53.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:53.007984+0000 osd.7 (osd.7) 42 : cluster [DBG] 2.b scrub ok 2024-02-12T16:51:54.590 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:53.376284+0000 mgr.y (mgr.24481) 92 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:54.591 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:53.781891+0000 osd.4 (osd.4) 45 : cluster [DBG] 2.f deep-scrub starts 2024-02-12T16:51:54.591 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:53.786678+0000 osd.4 (osd.4) 46 : cluster [DBG] 2.f deep-scrub ok 2024-02-12T16:51:54.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:53.376284+0000 mgr.y (mgr.24481) 92 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:54.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:53.781891+0000 osd.4 (osd.4) 45 : cluster [DBG] 2.f deep-scrub starts 2024-02-12T16:51:54.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:53.786678+0000 osd.4 (osd.4) 46 : cluster [DBG] 2.f deep-scrub ok 2024-02-12T16:51:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:53.376284+0000 mgr.y (mgr.24481) 92 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:51:54.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:53.781891+0000 osd.4 (osd.4) 45 : cluster [DBG] 2.f deep-scrub starts 2024-02-12T16:51:54.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:53.786678+0000 osd.4 (osd.4) 46 : cluster [DBG] 2.f deep-scrub ok 2024-02-12T16:51:55.016 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:51:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:51:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:54.299898+0000 mgr.y (mgr.24481) 93 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:54.299898+0000 mgr.y (mgr.24481) 93 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:54.299898+0000 mgr.y (mgr.24481) 93 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:51:55.843 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-02-12T16:51:54.299183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":847,"num_read_kb":599,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":488,"ondisk_log_size":488,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381968,"kb_used_data":5200,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579248,"statfs":{"total":767960285184,"available":765521149952,"internally_reserved":0,"allocated":5324800,"data_stored":2474224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001641"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.917837+0000","last_change":"2024-02-12T16:51:45.053060+0000","last_active":"2024-02-12T16:51:45.917837+0000","last_peered":"2024-02-12T16:51:45.917837+0000","last_clean":"2024-02-12T16:51:45.917837+0000","last_became_active":"2024-02-12T16:49:45.223515+0000","last_became_peered":"2024-02-12T16:49:45.223515+0000","last_unstale":"2024-02-12T16:51:45.917837+0000","last_undegraded":"2024-02-12T16:51:45.917837+0000","last_fullsized":"2024-02-12T16:51:45.917837+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:45.052938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:45.052938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553346+0000","last_change":"2024-02-12T16:50:51.824034+0000","last_active":"2024-02-12T16:51:49.553346+0000","last_peered":"2024-02-12T16:51:49.553346+0000","last_clean":"2024-02-12T16:51:49.553346+0000","last_became_active":"2024-02-12T16:49:43.195544+0000","last_became_peered":"2024-02-12T16:49:43.195544+0000","last_unstale":"2024-02-12T16:51:49.553346+0000","last_undegraded":"2024-02-12T16:51:49.553346+0000","last_fullsized":"2024-02-12T16:51:49.553346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.823967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430751+0000","last_change":"2024-02-12T16:50:42.254831+0000","last_active":"2024-02-12T16:51:49.430751+0000","last_peered":"2024-02-12T16:51:49.430751+0000","last_clean":"2024-02-12T16:51:49.430751+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:51:49.430751+0000","last_undegraded":"2024-02-12T16:51:49.430751+0000","last_fullsized":"2024-02-12T16:51:49.430751+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:42.254752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.254752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.919999+0000","last_change":"2024-02-12T16:51:16.075132+0000","last_active":"2024-02-12T16:51:45.919999+0000","last_peered":"2024-02-12T16:51:45.919999+0000","last_clean":"2024-02-12T16:51:45.919999+0000","last_became_active":"2024-02-12T16:49:41.185805+0000","last_became_peered":"2024-02-12T16:49:41.185805+0000","last_unstale":"2024-02-12T16:51:45.919999+0000","last_undegraded":"2024-02-12T16:51:45.919999+0000","last_fullsized":"2024-02-12T16:51:45.919999+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:51:16.075051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.075051+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554742+0000","last_change":"2024-02-12T16:51:21.897841+0000","last_active":"2024-02-12T16:51:49.554742+0000","last_peered":"2024-02-12T16:51:49.554742+0000","last_clean":"2024-02-12T16:51:49.554742+0000","last_became_active":"2024-02-12T16:49:45.216118+0000","last_became_peered":"2024-02-12T16:49:45.216118+0000","last_unstale":"2024-02-12T16:51:49.554742+0000","last_undegraded":"2024-02-12T16:51:49.554742+0000","last_fullsized":"2024-02-12T16:51:49.554742+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:21.897749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:21.897749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429317+0000","last_change":"2024-02-12T16:51:13.011883+0000","last_active":"2024-02-12T16:51:49.429317+0000","last_peered":"2024-02-12T16:51:49.429317+0000","last_clean":"2024-02-12T16:51:49.429317+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:51:49.429317+0000","last_undegraded":"2024-02-12T16:51:49.429317+0000","last_fullsized":"2024-02-12T16:51:49.429317+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.011814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.011814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918407+0000","last_change":"2024-02-12T16:51:38.073505+0000","last_active":"2024-02-12T16:51:45.918407+0000","last_peered":"2024-02-12T16:51:45.918407+0000","last_clean":"2024-02-12T16:51:45.918407+0000","last_became_active":"2024-02-12T16:49:39.102172+0000","last_became_peered":"2024-02-12T16:49:39.102172+0000","last_unstale":"2024-02-12T16:51:45.918407+0000","last_undegraded":"2024-02-12T16:51:45.918407+0000","last_fullsized":"2024-02-12T16:51:45.918407+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.073441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.073441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553643+0000","last_change":"2024-02-12T16:51:17.900217+0000","last_active":"2024-02-12T16:51:49.553643+0000","last_peered":"2024-02-12T16:51:49.553643+0000","last_clean":"2024-02-12T16:51:49.553643+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:51:49.553643+0000","last_undegraded":"2024-02-12T16:51:49.553643+0000","last_fullsized":"2024-02-12T16:51:49.553643+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:17.900093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:17.900093+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429221+0000","last_change":"2024-02-12T16:50:47.317845+0000","last_active":"2024-02-12T16:51:49.429221+0000","last_peered":"2024-02-12T16:51:49.429221+0000","last_clean":"2024-02-12T16:51:49.429221+0000","last_became_active":"2024-02-12T16:49:45.218184+0000","last_became_peered":"2024-02-12T16:49:45.218184+0000","last_unstale":"2024-02-12T16:51:49.429221+0000","last_undegraded":"2024-02-12T16:51:49.429221+0000","last_fullsized":"2024-02-12T16:51:49.429221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:47.317736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.317736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554606+0000","last_change":"2024-02-12T16:50:46.095997+0000","last_active":"2024-02-12T16:51:49.554606+0000","last_peered":"2024-02-12T16:51:49.554606+0000","last_clean":"2024-02-12T16:51:49.554606+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:51:49.554606+0000","last_undegraded":"2024-02-12T16:51:49.554606+0000","last_fullsized":"2024-02-12T16:51:49.554606+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.095924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554888+0000","last_change":"2024-02-12T16:51:04.051422+0000","last_active":"2024-02-12T16:51:49.554888+0000","last_peered":"2024-02-12T16:51:49.554888+0000","last_clean":"2024-02-12T16:51:49.554888+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:51:49.554888+0000","last_undegraded":"2024-02-12T16:51:49.554888+0000","last_fullsized":"2024-02-12T16:51:49.554888+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:51:04.051345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553972+0000","last_change":"2024-02-12T16:51:34.760803+0000","last_active":"2024-02-12T16:51:49.553972+0000","last_peered":"2024-02-12T16:51:49.553972+0000","last_clean":"2024-02-12T16:51:49.553972+0000","last_became_active":"2024-02-12T16:49:41.178018+0000","last_became_peered":"2024-02-12T16:49:41.178018+0000","last_unstale":"2024-02-12T16:51:49.553972+0000","last_undegraded":"2024-02-12T16:51:49.553972+0000","last_fullsized":"2024-02-12T16:51:49.553972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:51:34.760711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.760711+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430277+0000","last_change":"2024-02-12T16:51:22.020495+0000","last_active":"2024-02-12T16:51:49.430277+0000","last_peered":"2024-02-12T16:51:49.430277+0000","last_clean":"2024-02-12T16:51:49.430277+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:51:49.430277+0000","last_undegraded":"2024-02-12T16:51:49.430277+0000","last_fullsized":"2024-02-12T16:51:49.430277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.020430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.020430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429614+0000","last_change":"2024-02-12T16:50:55.322528+0000","last_active":"2024-02-12T16:51:49.429614+0000","last_peered":"2024-02-12T16:51:49.429614+0000","last_clean":"2024-02-12T16:51:49.429614+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:51:49.429614+0000","last_undegraded":"2024-02-12T16:51:49.429614+0000","last_fullsized":"2024-02-12T16:51:49.429614+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.322446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554859+0000","last_change":"2024-02-12T16:51:01.027931+0000","last_active":"2024-02-12T16:51:49.554859+0000","last_peered":"2024-02-12T16:51:49.554859+0000","last_clean":"2024-02-12T16:51:49.554859+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:51:49.554859+0000","last_undegraded":"2024-02-12T16:51:49.554859+0000","last_fullsized":"2024-02-12T16:51:49.554859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:01.027841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.027841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554974+0000","last_change":"2024-02-12T16:51:02.857339+0000","last_active":"2024-02-12T16:51:49.554974+0000","last_peered":"2024-02-12T16:51:49.554974+0000","last_clean":"2024-02-12T16:51:49.554974+0000","last_became_active":"2024-02-12T16:49:41.181117+0000","last_became_peered":"2024-02-12T16:49:41.181117+0000","last_unstale":"2024-02-12T16:51:49.554974+0000","last_undegraded":"2024-02-12T16:51:49.554974+0000","last_fullsized":"2024-02-12T16:51:49.554974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:02.857234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.857234+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918421+0000","last_change":"2024-02-12T16:51:03.001426+0000","last_active":"2024-02-12T16:51:45.918421+0000","last_peered":"2024-02-12T16:51:45.918421+0000","last_clean":"2024-02-12T16:51:45.918421+0000","last_became_active":"2024-02-12T16:49:45.221559+0000","last_became_peered":"2024-02-12T16:49:45.221559+0000","last_unstale":"2024-02-12T16:51:45.918421+0000","last_undegraded":"2024-02-12T16:51:45.918421+0000","last_fullsized":"2024-02-12T16:51:45.918421+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:03.001318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.001318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553887+0000","last_change":"2024-02-12T16:50:54.852577+0000","last_active":"2024-02-12T16:51:49.553887+0000","last_peered":"2024-02-12T16:51:49.553887+0000","last_clean":"2024-02-12T16:51:49.553887+0000","last_became_active":"2024-02-12T16:49:43.201709+0000","last_became_peered":"2024-02-12T16:49:43.201709+0000","last_unstale":"2024-02-12T16:51:49.553887+0000","last_undegraded":"2024-02-12T16:51:49.553887+0000","last_fullsized":"2024-02-12T16:51:49.553887+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:54.852481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.852481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918061+0000","last_change":"2024-02-12T16:51:29.066176+0000","last_active":"2024-02-12T16:51:45.918061+0000","last_peered":"2024-02-12T16:51:45.918061+0000","last_clean":"2024-02-12T16:51:45.918061+0000","last_became_active":"2024-02-12T16:49:39.105438+0000","last_became_peered":"2024-02-12T16:49:39.105438+0000","last_unstale":"2024-02-12T16:51:45.918061+0000","last_undegraded":"2024-02-12T16:51:45.918061+0000","last_fullsized":"2024-02-12T16:51:45.918061+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:29.066107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:29.066107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918151+0000","last_change":"2024-02-12T16:51:23.148923+0000","last_active":"2024-02-12T16:51:45.918151+0000","last_peered":"2024-02-12T16:51:45.918151+0000","last_clean":"2024-02-12T16:51:45.918151+0000","last_became_active":"2024-02-12T16:49:41.181980+0000","last_became_peered":"2024-02-12T16:49:41.181980+0000","last_unstale":"2024-02-12T16:51:45.918151+0000","last_undegraded":"2024-02-12T16:51:45.918151+0000","last_fullsized":"2024-02-12T16:51:45.918151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:51:23.148783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:23.148783+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554774+0000","last_change":"2024-02-12T16:51:10.898681+0000","last_active":"2024-02-12T16:51:49.554774+0000","last_peered":"2024-02-12T16:51:49.554774+0000","last_clean":"2024-02-12T16:51:49.554774+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:51:49.554774+0000","last_undegraded":"2024-02-12T16:51:49.554774+0000","last_fullsized":"2024-02-12T16:51:49.554774+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:51:10.898594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430417+0000","last_change":"2024-02-12T16:51:20.972423+0000","last_active":"2024-02-12T16:51:49.430417+0000","last_peered":"2024-02-12T16:51:49.430417+0000","last_clean":"2024-02-12T16:51:49.430417+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:51:49.430417+0000","last_undegraded":"2024-02-12T16:51:49.430417+0000","last_fullsized":"2024-02-12T16:51:49.430417+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.972298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.972298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554749+0000","last_change":"2024-02-12T16:51:44.303455+0000","last_active":"2024-02-12T16:51:49.554749+0000","last_peered":"2024-02-12T16:51:49.554749+0000","last_clean":"2024-02-12T16:51:49.554749+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:51:49.554749+0000","last_undegraded":"2024-02-12T16:51:49.554749+0000","last_fullsized":"2024-02-12T16:51:49.554749+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:51:44.303379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417412+0000","last_change":"2024-02-12T16:50:47.282788+0000","last_active":"2024-02-12T16:51:45.417412+0000","last_peered":"2024-02-12T16:51:45.417412+0000","last_clean":"2024-02-12T16:51:45.417412+0000","last_became_active":"2024-02-12T16:49:41.180319+0000","last_became_peered":"2024-02-12T16:49:41.180319+0000","last_unstale":"2024-02-12T16:51:45.417412+0000","last_undegraded":"2024-02-12T16:51:45.417412+0000","last_fullsized":"2024-02-12T16:51:45.417412+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:50:47.282711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.282711+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430218+0000","last_change":"2024-02-12T16:51:08.888476+0000","last_active":"2024-02-12T16:51:49.430218+0000","last_peered":"2024-02-12T16:51:49.430218+0000","last_clean":"2024-02-12T16:51:49.430218+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:51:49.430218+0000","last_undegraded":"2024-02-12T16:51:49.430218+0000","last_fullsized":"2024-02-12T16:51:49.430218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:08.888390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.888390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431367+0000","last_change":"2024-02-12T16:50:46.144078+0000","last_active":"2024-02-12T16:51:49.431367+0000","last_peered":"2024-02-12T16:51:49.431367+0000","last_clean":"2024-02-12T16:51:49.431367+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:51:49.431367+0000","last_undegraded":"2024-02-12T16:51:49.431367+0000","last_fullsized":"2024-02-12T16:51:49.431367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.144010+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.919703+0000","last_change":"2024-02-12T16:51:09.094169+0000","last_active":"2024-02-12T16:51:45.919703+0000","last_peered":"2024-02-12T16:51:45.919703+0000","last_clean":"2024-02-12T16:51:45.919703+0000","last_became_active":"2024-02-12T16:49:39.101492+0000","last_became_peered":"2024-02-12T16:49:39.101492+0000","last_unstale":"2024-02-12T16:51:45.919703+0000","last_undegraded":"2024-02-12T16:51:45.919703+0000","last_fullsized":"2024-02-12T16:51:45.919703+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:09.094067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:09.094067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555258+0000","last_change":"2024-02-12T16:51:14.042938+0000","last_active":"2024-02-12T16:51:49.555258+0000","last_peered":"2024-02-12T16:51:49.555258+0000","last_clean":"2024-02-12T16:51:49.555258+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:51:49.555258+0000","last_undegraded":"2024-02-12T16:51:49.555258+0000","last_fullsized":"2024-02-12T16:51:49.555258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:14.042858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:14.042858+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555225+0000","last_change":"2024-02-12T16:51:28.104047+0000","last_active":"2024-02-12T16:51:49.555225+0000","last_peered":"2024-02-12T16:51:49.555225+0000","last_clean":"2024-02-12T16:51:49.555225+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:51:49.555225+0000","last_undegraded":"2024-02-12T16:51:49.555225+0000","last_fullsized":"2024-02-12T16:51:49.555225+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.103969+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233167+0000","last_change":"2024-02-12T16:51:08.073509+0000","last_active":"2024-02-12T16:51:52.233167+0000","last_peered":"2024-02-12T16:51:52.233167+0000","last_clean":"2024-02-12T16:51:52.233167+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:51:52.233167+0000","last_undegraded":"2024-02-12T16:51:52.233167+0000","last_fullsized":"2024-02-12T16:51:52.233167+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:08.073428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.073428+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429156+0000","last_change":"2024-02-12T16:51:01.224070+0000","last_active":"2024-02-12T16:51:49.429156+0000","last_peered":"2024-02-12T16:51:49.429156+0000","last_clean":"2024-02-12T16:51:49.429156+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:51:49.429156+0000","last_undegraded":"2024-02-12T16:51:49.429156+0000","last_fullsized":"2024-02-12T16:51:49.429156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:01.223973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.223973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431280+0000","last_change":"2024-02-12T16:50:44.211440+0000","last_active":"2024-02-12T16:51:49.431280+0000","last_peered":"2024-02-12T16:51:49.431280+0000","last_clean":"2024-02-12T16:51:49.431280+0000","last_became_active":"2024-02-12T16:49:41.669589+0000","last_became_peered":"2024-02-12T16:49:41.669589+0000","last_unstale":"2024-02-12T16:51:49.431280+0000","last_undegraded":"2024-02-12T16:51:49.431280+0000","last_fullsized":"2024-02-12T16:51:49.431280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:50:44.211369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:44.211369+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:47.043976+0000","last_change":"2024-02-12T16:51:47.043952+0000","last_active":"2024-02-12T16:51:47.043976+0000","last_peered":"2024-02-12T16:51:47.043976+0000","last_clean":"2024-02-12T16:51:47.043976+0000","last_became_active":"2024-02-12T16:49:45.222341+0000","last_became_peered":"2024-02-12T16:49:45.222341+0000","last_unstale":"2024-02-12T16:51:47.043976+0000","last_undegraded":"2024-02-12T16:51:47.043976+0000","last_fullsized":"2024-02-12T16:51:47.043976+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:47.043893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:47.043893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553875+0000","last_change":"2024-02-12T16:51:19.038453+0000","last_active":"2024-02-12T16:51:49.553875+0000","last_peered":"2024-02-12T16:51:49.553875+0000","last_clean":"2024-02-12T16:51:49.553875+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:51:49.553875+0000","last_undegraded":"2024-02-12T16:51:49.553875+0000","last_fullsized":"2024-02-12T16:51:49.553875+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:19.038388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:19.038388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555492+0000","last_change":"2024-02-12T16:51:18.026686+0000","last_active":"2024-02-12T16:51:49.555492+0000","last_peered":"2024-02-12T16:51:49.555492+0000","last_clean":"2024-02-12T16:51:49.555492+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:51:49.555492+0000","last_undegraded":"2024-02-12T16:51:49.555492+0000","last_fullsized":"2024-02-12T16:51:49.555492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:18.026624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.026624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.919953+0000","last_change":"2024-02-12T16:50:51.908417+0000","last_active":"2024-02-12T16:51:45.919953+0000","last_peered":"2024-02-12T16:51:45.919953+0000","last_clean":"2024-02-12T16:51:45.919953+0000","last_became_active":"2024-02-12T16:49:41.669704+0000","last_became_peered":"2024-02-12T16:49:41.669704+0000","last_unstale":"2024-02-12T16:51:45.919953+0000","last_undegraded":"2024-02-12T16:51:45.919953+0000","last_fullsized":"2024-02-12T16:51:45.919953+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:50:51.908326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.908326+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554514+0000","last_change":"2024-02-12T16:51:15.965316+0000","last_active":"2024-02-12T16:51:49.554514+0000","last_peered":"2024-02-12T16:51:49.554514+0000","last_clean":"2024-02-12T16:51:49.554514+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:51:49.554514+0000","last_undegraded":"2024-02-12T16:51:49.554514+0000","last_fullsized":"2024-02-12T16:51:49.554514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:51:15.965186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553920+0000","last_change":"2024-02-12T16:50:55.941652+0000","last_active":"2024-02-12T16:51:49.553920+0000","last_peered":"2024-02-12T16:51:49.553920+0000","last_clean":"2024-02-12T16:51:49.553920+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:51:49.553920+0000","last_undegraded":"2024-02-12T16:51:49.553920+0000","last_fullsized":"2024-02-12T16:51:49.553920+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.941575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.941575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430067+0000","last_change":"2024-02-12T16:51:11.135365+0000","last_active":"2024-02-12T16:51:49.430067+0000","last_peered":"2024-02-12T16:51:49.430067+0000","last_clean":"2024-02-12T16:51:49.430067+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:51:49.430067+0000","last_undegraded":"2024-02-12T16:51:49.430067+0000","last_fullsized":"2024-02-12T16:51:49.430067+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.135261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.917843+0000","last_change":"2024-02-12T16:51:43.072967+0000","last_active":"2024-02-12T16:51:45.917843+0000","last_peered":"2024-02-12T16:51:45.917843+0000","last_clean":"2024-02-12T16:51:45.917843+0000","last_became_active":"2024-02-12T16:49:41.184044+0000","last_became_peered":"2024-02-12T16:49:41.184044+0000","last_unstale":"2024-02-12T16:51:45.917843+0000","last_undegraded":"2024-02-12T16:51:45.917843+0000","last_fullsized":"2024-02-12T16:51:45.917843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:51:43.072899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:43.072899+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.919535+0000","last_change":"2024-02-12T16:51:42.040901+0000","last_active":"2024-02-12T16:51:45.919535+0000","last_peered":"2024-02-12T16:51:45.919535+0000","last_clean":"2024-02-12T16:51:45.919535+0000","last_became_active":"2024-02-12T16:49:45.220469+0000","last_became_peered":"2024-02-12T16:49:45.220469+0000","last_unstale":"2024-02-12T16:51:45.919535+0000","last_undegraded":"2024-02-12T16:51:45.919535+0000","last_fullsized":"2024-02-12T16:51:45.919535+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:51:42.040828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429710+0000","last_change":"2024-02-12T16:50:57.875954+0000","last_active":"2024-02-12T16:51:49.429710+0000","last_peered":"2024-02-12T16:51:49.429710+0000","last_clean":"2024-02-12T16:51:49.429710+0000","last_became_active":"2024-02-12T16:49:43.204615+0000","last_became_peered":"2024-02-12T16:49:43.204615+0000","last_unstale":"2024-02-12T16:51:49.429710+0000","last_undegraded":"2024-02-12T16:51:49.429710+0000","last_fullsized":"2024-02-12T16:51:49.429710+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:57.875871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.875871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430463+0000","last_change":"2024-02-12T16:50:46.810493+0000","last_active":"2024-02-12T16:51:49.430463+0000","last_peered":"2024-02-12T16:51:49.430463+0000","last_clean":"2024-02-12T16:51:49.430463+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:51:49.430463+0000","last_undegraded":"2024-02-12T16:51:49.430463+0000","last_fullsized":"2024-02-12T16:51:49.430463+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.810420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.810420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553826+0000","last_change":"2024-02-12T16:51:14.844224+0000","last_active":"2024-02-12T16:51:49.553826+0000","last_peered":"2024-02-12T16:51:49.553826+0000","last_clean":"2024-02-12T16:51:49.553826+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:51:49.553826+0000","last_undegraded":"2024-02-12T16:51:49.553826+0000","last_fullsized":"2024-02-12T16:51:49.553826+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:14.844115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:14.844115+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429922+0000","last_change":"2024-02-12T16:51:28.886399+0000","last_active":"2024-02-12T16:51:49.429922+0000","last_peered":"2024-02-12T16:51:49.429922+0000","last_clean":"2024-02-12T16:51:49.429922+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:51:49.429922+0000","last_undegraded":"2024-02-12T16:51:49.429922+0000","last_fullsized":"2024-02-12T16:51:49.429922+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.886321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918141+0000","last_change":"2024-02-12T16:51:39.040277+0000","last_active":"2024-02-12T16:51:45.918141+0000","last_peered":"2024-02-12T16:51:45.918141+0000","last_clean":"2024-02-12T16:51:45.918141+0000","last_became_active":"2024-02-12T16:49:43.208343+0000","last_became_peered":"2024-02-12T16:49:43.208343+0000","last_unstale":"2024-02-12T16:51:45.918141+0000","last_undegraded":"2024-02-12T16:51:45.918141+0000","last_fullsized":"2024-02-12T16:51:45.918141+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.040203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.040203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554424+0000","last_change":"2024-02-12T16:51:11.088535+0000","last_active":"2024-02-12T16:51:49.554424+0000","last_peered":"2024-02-12T16:51:49.554424+0000","last_clean":"2024-02-12T16:51:49.554424+0000","last_became_active":"2024-02-12T16:49:39.107471+0000","last_became_peered":"2024-02-12T16:49:39.107471+0000","last_unstale":"2024-02-12T16:51:49.554424+0000","last_undegraded":"2024-02-12T16:51:49.554424+0000","last_fullsized":"2024-02-12T16:51:49.554424+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:11.088412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.088412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429647+0000","last_change":"2024-02-12T16:51:01.759881+0000","last_active":"2024-02-12T16:51:49.429647+0000","last_peered":"2024-02-12T16:51:49.429647+0000","last_clean":"2024-02-12T16:51:49.429647+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:51:49.429647+0000","last_undegraded":"2024-02-12T16:51:49.429647+0000","last_fullsized":"2024-02-12T16:51:49.429647+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.759802+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554507+0000","last_change":"2024-02-12T16:51:24.897238+0000","last_active":"2024-02-12T16:51:49.554507+0000","last_peered":"2024-02-12T16:51:49.554507+0000","last_clean":"2024-02-12T16:51:49.554507+0000","last_became_active":"2024-02-12T16:49:45.217958+0000","last_became_peered":"2024-02-12T16:49:45.217958+0000","last_unstale":"2024-02-12T16:51:49.554507+0000","last_undegraded":"2024-02-12T16:51:49.554507+0000","last_fullsized":"2024-02-12T16:51:49.554507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.897149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.897149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.503680+0000","last_change":"2024-02-12T16:51:44.091513+0000","last_active":"2024-02-12T16:51:49.503680+0000","last_peered":"2024-02-12T16:51:49.503680+0000","last_clean":"2024-02-12T16:51:49.503680+0000","last_became_active":"2024-02-12T16:49:43.206068+0000","last_became_peered":"2024-02-12T16:49:43.206068+0000","last_unstale":"2024-02-12T16:51:49.503680+0000","last_undegraded":"2024-02-12T16:51:49.503680+0000","last_fullsized":"2024-02-12T16:51:49.503680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:44.091436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:44.091436+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431257+0000","last_change":"2024-02-12T16:51:25.913214+0000","last_active":"2024-02-12T16:51:49.431257+0000","last_peered":"2024-02-12T16:51:49.431257+0000","last_clean":"2024-02-12T16:51:49.431257+0000","last_became_active":"2024-02-12T16:49:39.103401+0000","last_became_peered":"2024-02-12T16:49:39.103401+0000","last_unstale":"2024-02-12T16:51:49.431257+0000","last_undegraded":"2024-02-12T16:51:49.431257+0000","last_fullsized":"2024-02-12T16:51:49.431257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:25.913145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.913145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430029+0000","last_change":"2024-02-12T16:51:25.028420+0000","last_active":"2024-02-12T16:51:49.430029+0000","last_peered":"2024-02-12T16:51:49.430029+0000","last_clean":"2024-02-12T16:51:49.430029+0000","last_became_active":"2024-02-12T16:49:41.669427+0000","last_became_peered":"2024-02-12T16:49:41.669427+0000","last_unstale":"2024-02-12T16:51:49.430029+0000","last_undegraded":"2024-02-12T16:51:49.430029+0000","last_fullsized":"2024-02-12T16:51:49.430029+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:25.028339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.028339+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430918+0000","last_change":"2024-02-12T16:50:49.214010+0000","last_active":"2024-02-12T16:51:49.430918+0000","last_peered":"2024-02-12T16:51:49.430918+0000","last_clean":"2024-02-12T16:51:49.430918+0000","last_became_active":"2024-02-12T16:49:45.212245+0000","last_became_peered":"2024-02-12T16:49:45.212245+0000","last_unstale":"2024-02-12T16:51:49.430918+0000","last_undegraded":"2024-02-12T16:51:49.430918+0000","last_fullsized":"2024-02-12T16:51:49.430918+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:50:49.213945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233168+0000","last_change":"2024-02-12T16:51:09.159161+0000","last_active":"2024-02-12T16:51:52.233168+0000","last_peered":"2024-02-12T16:51:52.233168+0000","last_clean":"2024-02-12T16:51:52.233168+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:51:52.233168+0000","last_undegraded":"2024-02-12T16:51:52.233168+0000","last_fullsized":"2024-02-12T16:51:52.233168+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:09.159070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:09.159070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429349+0000","last_change":"2024-02-12T16:51:20.103380+0000","last_active":"2024-02-12T16:51:49.429349+0000","last_peered":"2024-02-12T16:51:49.429349+0000","last_clean":"2024-02-12T16:51:49.429349+0000","last_became_active":"2024-02-12T16:49:39.104936+0000","last_became_peered":"2024-02-12T16:49:39.104936+0000","last_unstale":"2024-02-12T16:51:49.429349+0000","last_undegraded":"2024-02-12T16:51:49.429349+0000","last_fullsized":"2024-02-12T16:51:49.429349+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.103297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.103297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918580+0000","last_change":"2024-02-12T16:51:36.026655+0000","last_active":"2024-02-12T16:51:45.918580+0000","last_peered":"2024-02-12T16:51:45.918580+0000","last_clean":"2024-02-12T16:51:45.918580+0000","last_became_active":"2024-02-12T16:51:04.116639+0000","last_became_peered":"2024-02-12T16:51:04.116639+0000","last_unstale":"2024-02-12T16:51:45.918580+0000","last_undegraded":"2024-02-12T16:51:45.918580+0000","last_fullsized":"2024-02-12T16:51:45.918580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:36.026569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:36.026569+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554808+0000","last_change":"2024-02-12T16:51:20.041621+0000","last_active":"2024-02-12T16:51:49.554808+0000","last_peered":"2024-02-12T16:51:49.554808+0000","last_clean":"2024-02-12T16:51:49.554808+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:51:49.554808+0000","last_undegraded":"2024-02-12T16:51:49.554808+0000","last_fullsized":"2024-02-12T16:51:49.554808+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.041556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.041556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430008+0000","last_change":"2024-02-12T16:50:45.275564+0000","last_active":"2024-02-12T16:51:49.430008+0000","last_peered":"2024-02-12T16:51:49.430008+0000","last_clean":"2024-02-12T16:51:49.430008+0000","last_became_active":"2024-02-12T16:49:43.201595+0000","last_became_peered":"2024-02-12T16:49:43.201595+0000","last_unstale":"2024-02-12T16:51:49.430008+0000","last_undegraded":"2024-02-12T16:51:49.430008+0000","last_fullsized":"2024-02-12T16:51:49.430008+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:45.275469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:45.275469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554009+0000","last_change":"2024-02-12T16:51:19.027311+0000","last_active":"2024-02-12T16:51:49.554009+0000","last_peered":"2024-02-12T16:51:49.554009+0000","last_clean":"2024-02-12T16:51:49.554009+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:51:49.554009+0000","last_undegraded":"2024-02-12T16:51:49.554009+0000","last_fullsized":"2024-02-12T16:51:49.554009+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:19.027246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:19.027246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918231+0000","last_change":"2024-02-12T16:51:28.108097+0000","last_active":"2024-02-12T16:51:45.918231+0000","last_peered":"2024-02-12T16:51:45.918231+0000","last_clean":"2024-02-12T16:51:45.918231+0000","last_became_active":"2024-02-12T16:49:41.669651+0000","last_became_peered":"2024-02-12T16:49:41.669651+0000","last_unstale":"2024-02-12T16:51:45.918231+0000","last_undegraded":"2024-02-12T16:51:45.918231+0000","last_fullsized":"2024-02-12T16:51:45.918231+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:51:28.108036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.108036+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417173+0000","last_change":"2024-02-12T16:51:03.235610+0000","last_active":"2024-02-12T16:51:45.417173+0000","last_peered":"2024-02-12T16:51:45.417173+0000","last_clean":"2024-02-12T16:51:45.417173+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:51:45.417173+0000","last_undegraded":"2024-02-12T16:51:45.417173+0000","last_fullsized":"2024-02-12T16:51:45.417173+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:03.235544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.235544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918345+0000","last_change":"2024-02-12T16:51:32.014849+0000","last_active":"2024-02-12T16:51:45.918345+0000","last_peered":"2024-02-12T16:51:45.918345+0000","last_clean":"2024-02-12T16:51:45.918345+0000","last_became_active":"2024-02-12T16:49:43.207454+0000","last_became_peered":"2024-02-12T16:49:43.207454+0000","last_unstale":"2024-02-12T16:51:45.918345+0000","last_undegraded":"2024-02-12T16:51:45.918345+0000","last_fullsized":"2024-02-12T16:51:45.918345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.014772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416551+0000","last_change":"2024-02-12T16:51:39.307549+0000","last_active":"2024-02-12T16:51:45.416551+0000","last_peered":"2024-02-12T16:51:45.416551+0000","last_clean":"2024-02-12T16:51:45.416551+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:51:45.416551+0000","last_undegraded":"2024-02-12T16:51:45.416551+0000","last_fullsized":"2024-02-12T16:51:45.416551+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.307475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.307475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431086+0000","last_change":"2024-02-12T16:51:17.050140+0000","last_active":"2024-02-12T16:51:49.431086+0000","last_peered":"2024-02-12T16:51:49.431086+0000","last_clean":"2024-02-12T16:51:49.431086+0000","last_became_active":"2024-02-12T16:49:41.668665+0000","last_became_peered":"2024-02-12T16:49:41.668665+0000","last_unstale":"2024-02-12T16:51:49.431086+0000","last_undegraded":"2024-02-12T16:51:49.431086+0000","last_fullsized":"2024-02-12T16:51:49.431086+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:51:17.050077+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918230+0000","last_change":"2024-02-12T16:51:33.032870+0000","last_active":"2024-02-12T16:51:45.918230+0000","last_peered":"2024-02-12T16:51:45.918230+0000","last_clean":"2024-02-12T16:51:45.918230+0000","last_became_active":"2024-02-12T16:49:45.220328+0000","last_became_peered":"2024-02-12T16:49:45.220328+0000","last_unstale":"2024-02-12T16:51:45.918230+0000","last_undegraded":"2024-02-12T16:51:45.918230+0000","last_fullsized":"2024-02-12T16:51:45.918230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:33.032784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:33.032784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"58'2","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554196+0000","last_change":"2024-02-12T16:50:42.866331+0000","last_active":"2024-02-12T16:51:49.554196+0000","last_peered":"2024-02-12T16:51:49.554196+0000","last_clean":"2024-02-12T16:51:49.554196+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:51:49.554196+0000","last_undegraded":"2024-02-12T16:51:49.554196+0000","last_fullsized":"2024-02-12T16:51:49.554196+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-02-12T16:50:42.866263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.866263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555038+0000","last_change":"2024-02-12T16:50:46.750203+0000","last_active":"2024-02-12T16:51:49.555038+0000","last_peered":"2024-02-12T16:51:49.555038+0000","last_clean":"2024-02-12T16:51:49.555038+0000","last_became_active":"2024-02-12T16:49:41.177967+0000","last_became_peered":"2024-02-12T16:49:41.177967+0000","last_unstale":"2024-02-12T16:51:49.555038+0000","last_undegraded":"2024-02-12T16:51:49.555038+0000","last_fullsized":"2024-02-12T16:51:49.555038+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:50:46.750065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.750065+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416527+0000","last_change":"2024-02-12T16:51:22.127898+0000","last_active":"2024-02-12T16:51:45.416527+0000","last_peered":"2024-02-12T16:51:45.416527+0000","last_clean":"2024-02-12T16:51:45.416527+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:51:45.416527+0000","last_undegraded":"2024-02-12T16:51:45.416527+0000","last_fullsized":"2024-02-12T16:51:45.416527+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.127831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429954+0000","last_change":"2024-02-12T16:50:50.186122+0000","last_active":"2024-02-12T16:51:49.429954+0000","last_peered":"2024-02-12T16:51:49.429954+0000","last_clean":"2024-02-12T16:51:49.429954+0000","last_became_active":"2024-02-12T16:49:45.217626+0000","last_became_peered":"2024-02-12T16:49:45.217626+0000","last_unstale":"2024-02-12T16:51:49.429954+0000","last_undegraded":"2024-02-12T16:51:49.429954+0000","last_fullsized":"2024-02-12T16:51:49.429954+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:50.186021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.186021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430067+0000","last_change":"2024-02-12T16:51:23.967921+0000","last_active":"2024-02-12T16:51:49.430067+0000","last_peered":"2024-02-12T16:51:49.430067+0000","last_clean":"2024-02-12T16:51:49.430067+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:51:49.430067+0000","last_undegraded":"2024-02-12T16:51:49.430067+0000","last_fullsized":"2024-02-12T16:51:49.430067+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:23.967848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:23.967848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416757+0000","last_change":"2024-02-12T16:51:04.121170+0000","last_active":"2024-02-12T16:51:45.416757+0000","last_peered":"2024-02-12T16:51:45.416757+0000","last_clean":"2024-02-12T16:51:45.416757+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:51:45.416757+0000","last_undegraded":"2024-02-12T16:51:45.416757+0000","last_fullsized":"2024-02-12T16:51:45.416757+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:50:42.876140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.876140+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233116+0000","last_change":"2024-02-12T16:51:11.206549+0000","last_active":"2024-02-12T16:51:52.233116+0000","last_peered":"2024-02-12T16:51:52.233116+0000","last_clean":"2024-02-12T16:51:52.233116+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:51:52.233116+0000","last_undegraded":"2024-02-12T16:51:52.233116+0000","last_fullsized":"2024-02-12T16:51:52.233116+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:11.206420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.206420+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918495+0000","last_change":"2024-02-12T16:51:26.163640+0000","last_active":"2024-02-12T16:51:45.918495+0000","last_peered":"2024-02-12T16:51:45.918495+0000","last_clean":"2024-02-12T16:51:45.918495+0000","last_became_active":"2024-02-12T16:49:45.215520+0000","last_became_peered":"2024-02-12T16:49:45.215520+0000","last_unstale":"2024-02-12T16:51:45.918495+0000","last_undegraded":"2024-02-12T16:51:45.918495+0000","last_fullsized":"2024-02-12T16:51:45.918495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:26.163568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:26.163568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429346+0000","last_change":"2024-02-12T16:50:50.253026+0000","last_active":"2024-02-12T16:51:49.429346+0000","last_peered":"2024-02-12T16:51:49.429346+0000","last_clean":"2024-02-12T16:51:49.429346+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:51:49.429346+0000","last_undegraded":"2024-02-12T16:51:49.429346+0000","last_fullsized":"2024-02-12T16:51:49.429346+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:50.252952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.252952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233130+0000","last_change":"2024-02-12T16:51:37.878869+0000","last_active":"2024-02-12T16:51:52.233130+0000","last_peered":"2024-02-12T16:51:52.233130+0000","last_clean":"2024-02-12T16:51:52.233130+0000","last_became_active":"2024-02-12T16:49:41.179610+0000","last_became_peered":"2024-02-12T16:49:41.179610+0000","last_unstale":"2024-02-12T16:51:52.233130+0000","last_undegraded":"2024-02-12T16:51:52.233130+0000","last_fullsized":"2024-02-12T16:51:52.233130+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-12T16:51:37.878788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.878788+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430620+0000","last_change":"2024-02-12T16:50:57.182750+0000","last_active":"2024-02-12T16:51:49.430620+0000","last_peered":"2024-02-12T16:51:49.430620+0000","last_clean":"2024-02-12T16:51:49.430620+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:51:49.430620+0000","last_undegraded":"2024-02-12T16:51:49.430620+0000","last_fullsized":"2024-02-12T16:51:49.430620+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:57.182679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.182679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555401+0000","last_change":"2024-02-12T16:51:22.062228+0000","last_active":"2024-02-12T16:51:49.555401+0000","last_peered":"2024-02-12T16:51:49.555401+0000","last_clean":"2024-02-12T16:51:49.555401+0000","last_became_active":"2024-02-12T16:49:45.221460+0000","last_became_peered":"2024-02-12T16:49:45.221460+0000","last_unstale":"2024-02-12T16:51:49.555401+0000","last_undegraded":"2024-02-12T16:51:49.555401+0000","last_fullsized":"2024-02-12T16:51:49.555401+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.062124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.062124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":177,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.452896+0000","last_change":"2024-02-12T16:50:49.760225+0000","last_active":"2024-02-12T16:51:52.452896+0000","last_peered":"2024-02-12T16:51:52.452896+0000","last_clean":"2024-02-12T16:51:52.452896+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:51:52.452896+0000","last_undegraded":"2024-02-12T16:51:52.452896+0000","last_fullsized":"2024-02-12T16:51:52.452896+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:50:49.760157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:49.760157+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":127,"num_read_kb":122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416693+0000","last_change":"2024-02-12T16:51:02.278460+0000","last_active":"2024-02-12T16:51:45.416693+0000","last_peered":"2024-02-12T16:51:45.416693+0000","last_clean":"2024-02-12T16:51:45.416693+0000","last_became_active":"2024-02-12T16:49:39.105685+0000","last_became_peered":"2024-02-12T16:49:39.105685+0000","last_unstale":"2024-02-12T16:51:45.416693+0000","last_undegraded":"2024-02-12T16:51:45.416693+0000","last_fullsized":"2024-02-12T16:51:45.416693+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.278337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.278337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431294+0000","last_change":"2024-02-12T16:51:18.950103+0000","last_active":"2024-02-12T16:51:49.431294+0000","last_peered":"2024-02-12T16:51:49.431294+0000","last_clean":"2024-02-12T16:51:49.431294+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:51:49.431294+0000","last_undegraded":"2024-02-12T16:51:49.431294+0000","last_fullsized":"2024-02-12T16:51:49.431294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:51:18.950031+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:51:18.950031+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.950031+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417183+0000","last_change":"2024-02-12T16:51:24.083385+0000","last_active":"2024-02-12T16:51:45.417183+0000","last_peered":"2024-02-12T16:51:45.417183+0000","last_clean":"2024-02-12T16:51:45.417183+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:51:45.417183+0000","last_undegraded":"2024-02-12T16:51:45.417183+0000","last_fullsized":"2024-02-12T16:51:45.417183+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.083270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.083270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430257+0000","last_change":"2024-02-12T16:51:00.237219+0000","last_active":"2024-02-12T16:51:49.430257+0000","last_peered":"2024-02-12T16:51:49.430257+0000","last_clean":"2024-02-12T16:51:49.430257+0000","last_became_active":"2024-02-12T16:49:45.213152+0000","last_became_peered":"2024-02-12T16:49:45.213152+0000","last_unstale":"2024-02-12T16:51:49.430257+0000","last_undegraded":"2024-02-12T16:51:49.430257+0000","last_fullsized":"2024-02-12T16:51:49.430257+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:51:00.237119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.917938+0000","last_change":"2024-02-12T16:51:08.054217+0000","last_active":"2024-02-12T16:51:45.917938+0000","last_peered":"2024-02-12T16:51:45.917938+0000","last_clean":"2024-02-12T16:51:45.917938+0000","last_became_active":"2024-02-12T16:49:39.105761+0000","last_became_peered":"2024-02-12T16:49:39.105761+0000","last_unstale":"2024-02-12T16:51:45.917938+0000","last_undegraded":"2024-02-12T16:51:45.917938+0000","last_fullsized":"2024-02-12T16:51:45.917938+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:08.054114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.054114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429532+0000","last_change":"2024-02-12T16:50:56.320321+0000","last_active":"2024-02-12T16:51:49.429532+0000","last_peered":"2024-02-12T16:51:49.429532+0000","last_clean":"2024-02-12T16:51:49.429532+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:51:49.429532+0000","last_undegraded":"2024-02-12T16:51:49.429532+0000","last_fullsized":"2024-02-12T16:51:49.429532+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:56.320240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:56.320240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553670+0000","last_change":"2024-02-12T16:51:37.184984+0000","last_active":"2024-02-12T16:51:49.553670+0000","last_peered":"2024-02-12T16:51:49.553670+0000","last_clean":"2024-02-12T16:51:49.553670+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:51:49.553670+0000","last_undegraded":"2024-02-12T16:51:49.553670+0000","last_fullsized":"2024-02-12T16:51:49.553670+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:37.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.184907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554395+0000","last_change":"2024-02-12T16:50:57.964701+0000","last_active":"2024-02-12T16:51:49.554395+0000","last_peered":"2024-02-12T16:51:49.554395+0000","last_clean":"2024-02-12T16:51:49.554395+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:51:49.554395+0000","last_undegraded":"2024-02-12T16:51:49.554395+0000","last_fullsized":"2024-02-12T16:51:49.554395+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:50:57.964625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.964625+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555066+0000","last_change":"2024-02-12T16:51:05.047096+0000","last_active":"2024-02-12T16:51:49.555066+0000","last_peered":"2024-02-12T16:51:49.555066+0000","last_clean":"2024-02-12T16:51:49.555066+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:51:49.555066+0000","last_undegraded":"2024-02-12T16:51:49.555066+0000","last_fullsized":"2024-02-12T16:51:49.555066+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:05.047021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:05.047021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429796+0000","last_change":"2024-02-12T16:51:36.842712+0000","last_active":"2024-02-12T16:51:49.429796+0000","last_peered":"2024-02-12T16:51:49.429796+0000","last_clean":"2024-02-12T16:51:49.429796+0000","last_became_active":"2024-02-12T16:49:45.218096+0000","last_became_peered":"2024-02-12T16:49:45.218096+0000","last_unstale":"2024-02-12T16:51:49.429796+0000","last_undegraded":"2024-02-12T16:51:49.429796+0000","last_fullsized":"2024-02-12T16:51:49.429796+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:36.842645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:36.842645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430922+0000","last_change":"2024-02-12T16:51:18.079605+0000","last_active":"2024-02-12T16:51:49.430922+0000","last_peered":"2024-02-12T16:51:49.430922+0000","last_clean":"2024-02-12T16:51:49.430922+0000","last_became_active":"2024-02-12T16:49:39.104017+0000","last_became_peered":"2024-02-12T16:49:39.104017+0000","last_unstale":"2024-02-12T16:51:49.430922+0000","last_undegraded":"2024-02-12T16:51:49.430922+0000","last_fullsized":"2024-02-12T16:51:49.430922+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:18.079497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.079497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553584+0000","last_change":"2024-02-12T16:51:08.953016+0000","last_active":"2024-02-12T16:51:49.553584+0000","last_peered":"2024-02-12T16:51:49.553584+0000","last_clean":"2024-02-12T16:51:49.553584+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:51:49.553584+0000","last_undegraded":"2024-02-12T16:51:49.553584+0000","last_fullsized":"2024-02-12T16:51:49.553584+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:51:08.952901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.952901+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431003+0000","last_change":"2024-02-12T16:51:07.240302+0000","last_active":"2024-02-12T16:51:49.431003+0000","last_peered":"2024-02-12T16:51:49.431003+0000","last_clean":"2024-02-12T16:51:49.431003+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:51:49.431003+0000","last_undegraded":"2024-02-12T16:51:49.431003+0000","last_fullsized":"2024-02-12T16:51:49.431003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:51:07.240236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:07.240236+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430219+0000","last_change":"2024-02-12T16:51:22.070134+0000","last_active":"2024-02-12T16:51:49.430219+0000","last_peered":"2024-02-12T16:51:49.430219+0000","last_clean":"2024-02-12T16:51:49.430219+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:51:49.430219+0000","last_undegraded":"2024-02-12T16:51:49.430219+0000","last_fullsized":"2024-02-12T16:51:49.430219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.070056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.070056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429236+0000","last_change":"2024-02-12T16:51:32.884735+0000","last_active":"2024-02-12T16:51:49.429236+0000","last_peered":"2024-02-12T16:51:49.429236+0000","last_clean":"2024-02-12T16:51:49.429236+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:51:49.429236+0000","last_undegraded":"2024-02-12T16:51:49.429236+0000","last_fullsized":"2024-02-12T16:51:49.429236+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.884670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555458+0000","last_change":"2024-02-12T16:50:56.149313+0000","last_active":"2024-02-12T16:51:49.555458+0000","last_peered":"2024-02-12T16:51:49.555458+0000","last_clean":"2024-02-12T16:51:49.555458+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:51:49.555458+0000","last_undegraded":"2024-02-12T16:51:49.555458+0000","last_fullsized":"2024-02-12T16:51:49.555458+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:56.149242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:56.149242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.745302+0000","last_change":"2024-02-12T16:51:52.745302+0000","last_active":"2024-02-12T16:51:52.745302+0000","last_peered":"2024-02-12T16:51:52.745302+0000","last_clean":"2024-02-12T16:51:52.745302+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:51:52.745302+0000","last_undegraded":"2024-02-12T16:51:52.745302+0000","last_fullsized":"2024-02-12T16:51:52.745302+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:51:52.745190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:52.745190+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554673+0000","last_change":"2024-02-12T16:51:10.119500+0000","last_active":"2024-02-12T16:51:49.554673+0000","last_peered":"2024-02-12T16:51:49.554673+0000","last_clean":"2024-02-12T16:51:49.554673+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:51:49.554673+0000","last_undegraded":"2024-02-12T16:51:49.554673+0000","last_fullsized":"2024-02-12T16:51:49.554673+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:10.119399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:10.119399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555196+0000","last_change":"2024-02-12T16:51:32.837004+0000","last_active":"2024-02-12T16:51:49.555196+0000","last_peered":"2024-02-12T16:51:49.555196+0000","last_clean":"2024-02-12T16:51:49.555196+0000","last_became_active":"2024-02-12T16:49:45.222560+0000","last_became_peered":"2024-02-12T16:49:45.222560+0000","last_unstale":"2024-02-12T16:51:49.555196+0000","last_undegraded":"2024-02-12T16:51:49.555196+0000","last_fullsized":"2024-02-12T16:51:49.555196+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.836928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.836928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430867+0000","last_change":"2024-02-12T16:51:16.010678+0000","last_active":"2024-02-12T16:51:49.430867+0000","last_peered":"2024-02-12T16:51:49.430867+0000","last_clean":"2024-02-12T16:51:49.430867+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:51:49.430867+0000","last_undegraded":"2024-02-12T16:51:49.430867+0000","last_fullsized":"2024-02-12T16:51:49.430867+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:16.010512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.010512+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:51.095208+0000","last_change":"2024-02-12T16:51:51.095208+0000","last_active":"2024-02-12T16:51:51.095208+0000","last_peered":"2024-02-12T16:51:51.095208+0000","last_clean":"2024-02-12T16:51:51.095208+0000","last_became_active":"2024-02-12T16:49:41.669593+0000","last_became_peered":"2024-02-12T16:49:41.669593+0000","last_unstale":"2024-02-12T16:51:51.095208+0000","last_undegraded":"2024-02-12T16:51:51.095208+0000","last_fullsized":"2024-02-12T16:51:51.095208+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:51:51.095125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:51.095125+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555114+0000","last_change":"2024-02-12T16:51:35.749314+0000","last_active":"2024-02-12T16:51:49.555114+0000","last_peered":"2024-02-12T16:51:49.555114+0000","last_clean":"2024-02-12T16:51:49.555114+0000","last_became_active":"2024-02-12T16:49:43.202312+0000","last_became_peered":"2024-02-12T16:49:43.202312+0000","last_unstale":"2024-02-12T16:51:49.555114+0000","last_undegraded":"2024-02-12T16:51:49.555114+0000","last_fullsized":"2024-02-12T16:51:49.555114+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:35.749237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:35.749237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555339+0000","last_change":"2024-02-12T16:50:55.111751+0000","last_active":"2024-02-12T16:51:49.555339+0000","last_peered":"2024-02-12T16:51:49.555339+0000","last_clean":"2024-02-12T16:51:49.555339+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:51:49.555339+0000","last_undegraded":"2024-02-12T16:51:49.555339+0000","last_fullsized":"2024-02-12T16:51:49.555339+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:50:55.111675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.111675+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555416+0000","last_change":"2024-02-12T16:51:16.020537+0000","last_active":"2024-02-12T16:51:49.555416+0000","last_peered":"2024-02-12T16:51:49.555416+0000","last_clean":"2024-02-12T16:51:49.555416+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:51:49.555416+0000","last_undegraded":"2024-02-12T16:51:49.555416+0000","last_fullsized":"2024-02-12T16:51:49.555416+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:16.020471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.020471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554894+0000","last_change":"2024-02-12T16:50:59.823981+0000","last_active":"2024-02-12T16:51:49.554894+0000","last_peered":"2024-02-12T16:51:49.554894+0000","last_clean":"2024-02-12T16:51:49.554894+0000","last_became_active":"2024-02-12T16:49:41.178569+0000","last_became_peered":"2024-02-12T16:49:41.178569+0000","last_unstale":"2024-02-12T16:51:49.554894+0000","last_undegraded":"2024-02-12T16:51:49.554894+0000","last_fullsized":"2024-02-12T16:51:49.554894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:50:59.823898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:59.823898+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.502960+0000","last_change":"2024-02-12T16:51:03.961580+0000","last_active":"2024-02-12T16:51:49.502960+0000","last_peered":"2024-02-12T16:51:49.502960+0000","last_clean":"2024-02-12T16:51:49.502960+0000","last_became_active":"2024-02-12T16:49:43.205391+0000","last_became_peered":"2024-02-12T16:49:43.205391+0000","last_unstale":"2024-02-12T16:51:49.502960+0000","last_undegraded":"2024-02-12T16:51:49.502960+0000","last_fullsized":"2024-02-12T16:51:49.502960+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:03.961503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.961503+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918496+0000","last_change":"2024-02-12T16:51:34.081725+0000","last_active":"2024-02-12T16:51:45.918496+0000","last_peered":"2024-02-12T16:51:45.918496+0000","last_clean":"2024-02-12T16:51:45.918496+0000","last_became_active":"2024-02-12T16:49:45.223659+0000","last_became_peered":"2024-02-12T16:49:45.223659+0000","last_unstale":"2024-02-12T16:51:45.918496+0000","last_undegraded":"2024-02-12T16:51:45.918496+0000","last_fullsized":"2024-02-12T16:51:45.918496+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:34.081644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.081644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430152+0000","last_change":"2024-02-12T16:51:05.201812+0000","last_active":"2024-02-12T16:51:49.430152+0000","last_peered":"2024-02-12T16:51:49.430152+0000","last_clean":"2024-02-12T16:51:49.430152+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:51:49.430152+0000","last_undegraded":"2024-02-12T16:51:49.430152+0000","last_fullsized":"2024-02-12T16:51:49.430152+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:51:05.201749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416650+0000","last_change":"2024-02-12T16:51:04.206339+0000","last_active":"2024-02-12T16:51:45.416650+0000","last_peered":"2024-02-12T16:51:45.416650+0000","last_clean":"2024-02-12T16:51:45.416650+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:51:45.416650+0000","last_undegraded":"2024-02-12T16:51:45.416650+0000","last_fullsized":"2024-02-12T16:51:45.416650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.123059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.123059+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.123059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554864+0000","last_change":"2024-02-12T16:51:30.776952+0000","last_active":"2024-02-12T16:51:49.554864+0000","last_peered":"2024-02-12T16:51:49.554864+0000","last_clean":"2024-02-12T16:51:49.554864+0000","last_became_active":"2024-02-12T16:49:41.178616+0000","last_became_peered":"2024-02-12T16:49:41.178616+0000","last_unstale":"2024-02-12T16:51:49.554864+0000","last_undegraded":"2024-02-12T16:51:49.554864+0000","last_fullsized":"2024-02-12T16:51:49.554864+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:30.776871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:30.776871+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429794+0000","last_change":"2024-02-12T16:51:02.216274+0000","last_active":"2024-02-12T16:51:49.429794+0000","last_peered":"2024-02-12T16:51:49.429794+0000","last_clean":"2024-02-12T16:51:49.429794+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:51:49.429794+0000","last_undegraded":"2024-02-12T16:51:49.429794+0000","last_fullsized":"2024-02-12T16:51:49.429794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.216194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.216194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"80'9","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233168+0000","last_change":"2024-02-12T16:50:50.800052+0000","last_active":"2024-02-12T16:51:52.233168+0000","last_peered":"2024-02-12T16:51:52.233168+0000","last_clean":"2024-02-12T16:51:52.233168+0000","last_became_active":"2024-02-12T16:49:43.197112+0000","last_became_peered":"2024-02-12T16:49:43.197112+0000","last_unstale":"2024-02-12T16:51:52.233168+0000","last_undegraded":"2024-02-12T16:51:52.233168+0000","last_fullsized":"2024-02-12T16:51:52.233168+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.799984+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429474+0000","last_change":"2024-02-12T16:51:00.778646+0000","last_active":"2024-02-12T16:51:49.429474+0000","last_peered":"2024-02-12T16:51:49.429474+0000","last_clean":"2024-02-12T16:51:49.429474+0000","last_became_active":"2024-02-12T16:49:39.095868+0000","last_became_peered":"2024-02-12T16:49:39.095868+0000","last_unstale":"2024-02-12T16:51:49.429474+0000","last_undegraded":"2024-02-12T16:51:49.429474+0000","last_fullsized":"2024-02-12T16:51:49.429474+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:00.778578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:00.778578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431246+0000","last_change":"2024-02-12T16:51:11.980641+0000","last_active":"2024-02-12T16:51:49.431246+0000","last_peered":"2024-02-12T16:51:49.431246+0000","last_clean":"2024-02-12T16:51:49.431246+0000","last_became_active":"2024-02-12T16:49:41.178544+0000","last_became_peered":"2024-02-12T16:49:41.178544+0000","last_unstale":"2024-02-12T16:51:49.431246+0000","last_undegraded":"2024-02-12T16:51:49.431246+0000","last_fullsized":"2024-02-12T16:51:49.431246+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:51:11.980577+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:51:11.980577+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.980577+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554514+0000","last_change":"2024-02-12T16:51:38.773791+0000","last_active":"2024-02-12T16:51:49.554514+0000","last_peered":"2024-02-12T16:51:49.554514+0000","last_clean":"2024-02-12T16:51:49.554514+0000","last_became_active":"2024-02-12T16:49:45.217458+0000","last_became_peered":"2024-02-12T16:49:45.217458+0000","last_unstale":"2024-02-12T16:51:49.554514+0000","last_undegraded":"2024-02-12T16:51:49.554514+0000","last_fullsized":"2024-02-12T16:51:49.554514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.773699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.773699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553278+0000","last_change":"2024-02-12T16:51:16.145692+0000","last_active":"2024-02-12T16:51:49.553278+0000","last_peered":"2024-02-12T16:51:49.553278+0000","last_clean":"2024-02-12T16:51:49.553278+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:51:49.553278+0000","last_undegraded":"2024-02-12T16:51:49.553278+0000","last_fullsized":"2024-02-12T16:51:49.553278+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:16.145576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.145576+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429564+0000","last_change":"2024-02-12T16:51:02.785509+0000","last_active":"2024-02-12T16:51:49.429564+0000","last_peered":"2024-02-12T16:51:49.429564+0000","last_clean":"2024-02-12T16:51:49.429564+0000","last_became_active":"2024-02-12T16:49:39.101478+0000","last_became_peered":"2024-02-12T16:49:39.101478+0000","last_unstale":"2024-02-12T16:51:49.429564+0000","last_undegraded":"2024-02-12T16:51:49.429564+0000","last_fullsized":"2024-02-12T16:51:49.429564+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.785426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.785426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553729+0000","last_change":"2024-02-12T16:51:04.882407+0000","last_active":"2024-02-12T16:51:49.553729+0000","last_peered":"2024-02-12T16:51:49.553729+0000","last_clean":"2024-02-12T16:51:49.553729+0000","last_became_active":"2024-02-12T16:49:41.179396+0000","last_became_peered":"2024-02-12T16:49:41.179396+0000","last_unstale":"2024-02-12T16:51:49.553729+0000","last_undegraded":"2024-02-12T16:51:49.553729+0000","last_fullsized":"2024-02-12T16:51:49.553729+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:04.882334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:04.882334+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417404+0000","last_change":"2024-02-12T16:50:55.285750+0000","last_active":"2024-02-12T16:51:45.417404+0000","last_peered":"2024-02-12T16:51:45.417404+0000","last_clean":"2024-02-12T16:51:45.417404+0000","last_became_active":"2024-02-12T16:49:45.220893+0000","last_became_peered":"2024-02-12T16:49:45.220893+0000","last_unstale":"2024-02-12T16:51:45.417404+0000","last_undegraded":"2024-02-12T16:51:45.417404+0000","last_fullsized":"2024-02-12T16:51:45.417404+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.285685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.285685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429848+0000","last_change":"2024-02-12T16:51:13.985683+0000","last_active":"2024-02-12T16:51:49.429848+0000","last_peered":"2024-02-12T16:51:49.429848+0000","last_clean":"2024-02-12T16:51:49.429848+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:51:49.429848+0000","last_undegraded":"2024-02-12T16:51:49.429848+0000","last_fullsized":"2024-02-12T16:51:49.429848+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.985614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.985614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-12T16:51:53.008096+0000","last_change":"2024-02-12T16:51:53.008096+0000","last_active":"2024-02-12T16:51:53.008096+0000","last_peered":"2024-02-12T16:51:53.008096+0000","last_clean":"2024-02-12T16:51:53.008096+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:51:53.008096+0000","last_undegraded":"2024-02-12T16:51:53.008096+0000","last_fullsized":"2024-02-12T16:51:53.008096+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:53.007988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:53.007988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554853+0000","last_change":"2024-02-12T16:51:12.892241+0000","last_active":"2024-02-12T16:51:49.554853+0000","last_peered":"2024-02-12T16:51:49.554853+0000","last_clean":"2024-02-12T16:51:49.554853+0000","last_became_active":"2024-02-12T16:49:41.177416+0000","last_became_peered":"2024-02-12T16:49:41.177416+0000","last_unstale":"2024-02-12T16:51:49.554853+0000","last_undegraded":"2024-02-12T16:51:49.554853+0000","last_fullsized":"2024-02-12T16:51:49.554853+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:51:12.892148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:12.892148+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553491+0000","last_change":"2024-02-12T16:50:54.135100+0000","last_active":"2024-02-12T16:51:49.553491+0000","last_peered":"2024-02-12T16:51:49.553491+0000","last_clean":"2024-02-12T16:51:49.553491+0000","last_became_active":"2024-02-12T16:49:45.217589+0000","last_became_peered":"2024-02-12T16:49:45.217589+0000","last_unstale":"2024-02-12T16:51:49.553491+0000","last_undegraded":"2024-02-12T16:51:49.553491+0000","last_fullsized":"2024-02-12T16:51:49.553491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.135009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417047+0000","last_change":"2024-02-12T16:50:52.289793+0000","last_active":"2024-02-12T16:51:45.417047+0000","last_peered":"2024-02-12T16:51:45.417047+0000","last_clean":"2024-02-12T16:51:45.417047+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:51:45.417047+0000","last_undegraded":"2024-02-12T16:51:45.417047+0000","last_fullsized":"2024-02-12T16:51:45.417047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:52.289725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:52.289725+0000","last_clean_scrub_stamp":"2024-02-12T16:50:52.289725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417431+0000","last_change":"2024-02-12T16:50:51.308424+0000","last_active":"2024-02-12T16:51:45.417431+0000","last_peered":"2024-02-12T16:51:45.417431+0000","last_clean":"2024-02-12T16:51:45.417431+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:51:45.417431+0000","last_undegraded":"2024-02-12T16:51:45.417431+0000","last_fullsized":"2024-02-12T16:51:45.417431+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.308317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.308317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":104,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554590+0000","last_change":"2024-02-12T16:51:04.126502+0000","last_active":"2024-02-12T16:51:49.554590+0000","last_peered":"2024-02-12T16:51:49.554590+0000","last_clean":"2024-02-12T16:51:49.554590+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:51:49.554590+0000","last_undegraded":"2024-02-12T16:51:49.554590+0000","last_fullsized":"2024-02-12T16:51:49.554590+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:50:47.792388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.792388+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430677+0000","last_change":"2024-02-12T16:51:13.192907+0000","last_active":"2024-02-12T16:51:49.430677+0000","last_peered":"2024-02-12T16:51:49.430677+0000","last_clean":"2024-02-12T16:51:49.430677+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:51:49.430677+0000","last_undegraded":"2024-02-12T16:51:49.430677+0000","last_fullsized":"2024-02-12T16:51:49.430677+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.192842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.192842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416902+0000","last_change":"2024-02-12T16:51:40.273267+0000","last_active":"2024-02-12T16:51:45.416902+0000","last_peered":"2024-02-12T16:51:45.416902+0000","last_clean":"2024-02-12T16:51:45.416902+0000","last_became_active":"2024-02-12T16:49:43.201946+0000","last_became_peered":"2024-02-12T16:49:43.201946+0000","last_unstale":"2024-02-12T16:51:45.416902+0000","last_undegraded":"2024-02-12T16:51:45.416902+0000","last_fullsized":"2024-02-12T16:51:45.416902+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:40.273180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:40.273180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"58'1","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430591+0000","last_change":"2024-02-12T16:51:31.889909+0000","last_active":"2024-02-12T16:51:49.430591+0000","last_peered":"2024-02-12T16:51:49.430591+0000","last_clean":"2024-02-12T16:51:49.430591+0000","last_became_active":"2024-02-12T16:49:39.099695+0000","last_became_peered":"2024-02-12T16:49:39.099695+0000","last_unstale":"2024-02-12T16:51:49.430591+0000","last_undegraded":"2024-02-12T16:51:49.430591+0000","last_fullsized":"2024-02-12T16:51:49.430591+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:51:31.889821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:31.889821+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431339+0000","last_change":"2024-02-12T16:50:45.175080+0000","last_active":"2024-02-12T16:51:49.431339+0000","last_peered":"2024-02-12T16:51:49.431339+0000","last_clean":"2024-02-12T16:51:49.431339+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:51:49.431339+0000","last_undegraded":"2024-02-12T16:51:49.431339+0000","last_fullsized":"2024-02-12T16:51:49.431339+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:50:45.174941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:45.174941+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429255+0000","last_change":"2024-02-12T16:51:24.030089+0000","last_active":"2024-02-12T16:51:49.429255+0000","last_peered":"2024-02-12T16:51:49.429255+0000","last_clean":"2024-02-12T16:51:49.429255+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:51:49.429255+0000","last_undegraded":"2024-02-12T16:51:49.429255+0000","last_fullsized":"2024-02-12T16:51:49.429255+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.030017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.030017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:51.361621+0000","last_change":"2024-02-12T16:51:51.361621+0000","last_active":"2024-02-12T16:51:51.361621+0000","last_peered":"2024-02-12T16:51:51.361621+0000","last_clean":"2024-02-12T16:51:51.361621+0000","last_became_active":"2024-02-12T16:49:39.105982+0000","last_became_peered":"2024-02-12T16:49:39.105982+0000","last_unstale":"2024-02-12T16:51:51.361621+0000","last_undegraded":"2024-02-12T16:51:51.361621+0000","last_fullsized":"2024-02-12T16:51:51.361621+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:51.361540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:51.361540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429291+0000","last_change":"2024-02-12T16:50:54.302143+0000","last_active":"2024-02-12T16:51:49.429291+0000","last_peered":"2024-02-12T16:51:49.429291+0000","last_clean":"2024-02-12T16:51:49.429291+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:51:49.429291+0000","last_undegraded":"2024-02-12T16:51:49.429291+0000","last_fullsized":"2024-02-12T16:51:49.429291+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:50:54.302060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.302060+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233143+0000","last_change":"2024-02-12T16:51:07.021402+0000","last_active":"2024-02-12T16:51:52.233143+0000","last_peered":"2024-02-12T16:51:52.233143+0000","last_clean":"2024-02-12T16:51:52.233143+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:51:52.233143+0000","last_undegraded":"2024-02-12T16:51:52.233143+0000","last_fullsized":"2024-02-12T16:51:52.233143+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:07.021338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:07.021338+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":127,"num_read_kb":122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234014,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":311517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69799999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.313},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80500000000000005,"5min":1.056,"15min":1.056},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65900000000000003,"5min":1.208,"15min":1.208},"last":0.36499999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56899999999999995,"15min":0.57999999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72299999999999998,"5min":1.766,"15min":1.766},"last":0.71399999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.61299999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.374},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.51100000000000001,"15min":0.52200000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67000000000000004,"5min":1.327,"15min":1.327},"last":0.432}]},{"osd":5,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68400000000000005,"5min":1.573,"15min":1.573},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53500000000000003,"15min":0.54300000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76400000000000001,"5min":1.621,"15min":1.621},"last":0.39800000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77700000000000002,"5min":1.272,"15min":1.272},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73199999999999998,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.53700000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528355,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":304453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48199999999999998,"15min":0.48299999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67000000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.63700000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65700000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.747,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60599999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72799999999999998,"5min":0.754,"15min":0.754},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.50600000000000001,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64500000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.65700000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.39800000000000002},"min":{"1min":0.28699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.42399999999999999,"15min":0.41799999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.318,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36799999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:51:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.39200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297788,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447364,"statfs":{"total":95995035648,"available":95690100736,"internally_reserved":0,"allocated":708608,"data_stored":313594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35399999999999998,"15min":0.34000000000000002},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39100000000000001,"15min":0.373},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.74199999999999999,"5min":2.778,"15min":2.778},"last":0.376},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41499999999999998,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.629,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.48399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.45600000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.42099999999999999,"15min":0.40899999999999997},"min":{"1min":0.35199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:51:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.40899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77300000000000002,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77200000000000002,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.95099999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.86599999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:51:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.64800000000000002,"15min":0.65800000000000003},"min":{"1min":0.438,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79900000000000004,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83999999999999997,"5min":1.109,"15min":1.109},"last":0.79000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297748,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447404,"statfs":{"total":95995035648,"available":95690141696,"internally_reserved":0,"allocated":667648,"data_stored":311201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"min":{"1min":0.255,"5min":0.192,"15min":0.192},"max":{"1min":0.55300000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.35299999999999998,"15min":0.317},"min":{"1min":0.27700000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.61599999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.42299999999999999,"15min":0.39100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.8130000000000002,"5min":3.8130000000000002,"15min":3.8130000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67400000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.40500000000000003,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56399999999999995,"5min":0.622,"15min":0.622},"last":0.376},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.45800000000000002,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.745,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67300000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.84699999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83299999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.45000000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.621}]},{"osd":6,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71299999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.80100000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.45700000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86099999999999999,"5min":0.871,"15min":0.871},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59899999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":305011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.38700000000000001,"15min":0.379},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46000000000000002,"5min":1.595,"15min":1.595},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.41999999999999998,"15min":0.40000000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47799999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.46300000000000002,"15min":0.41699999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.52000000000000002,"5min":6.8040000000000003,"15min":6.8040000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.437,"15min":0.41099999999999998},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51100000000000001,"5min":1.627,"15min":1.627},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47199999999999998,"15min":0.46600000000000003},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.57599999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.503,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.74199999999999999,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.70499999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.61499999999999999,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.69099999999999995,"5min":2.875,"15min":2.875},"last":0.61399999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.67600000000000005,"15min":0.72799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.73099999999999998,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58699999999999997,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69999999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.83899999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90400000000000003,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.89700000000000002,"15min":0.95299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70299999999999996,"5min":7.5629999999999997,"15min":7.5629999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.47699999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.69799999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.86899999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297756,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447396,"statfs":{"total":95995035648,"available":95690133504,"internally_reserved":0,"allocated":675840,"data_stored":306103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.436,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52100000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.42599999999999999,"15min":0.436},"min":{"1min":0.255,"5min":0.216,"15min":0.216},"max":{"1min":0.52300000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42899999999999999,"15min":0.436},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.503,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.442,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.436,"15min":0.44400000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.503,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58899999999999997,"15min":0.61099999999999999},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78100000000000003,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73699999999999999,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.40100000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61199999999999999,"15min":0.63300000000000001},"min":{"1min":0.45000000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77500000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60399999999999998,"15min":0.626},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82199999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76500000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.434},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.496}]},{"osd":7,"last update":"Mon Feb 12 16:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.432,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.53800000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":311870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.55300000000000005},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.51800000000000002},"min":{"1min":0.30399999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.438}]},{"osd":1,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.30299999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66300000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55300000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.57199999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53200000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53700000000000003,"15min":0.55400000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.621}]},{"osd":5,"last update":"Mon Feb 12 16:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.36799999999999999,"15min":0.34699999999999998},"min":{"1min":0.25800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.36099999999999999,"15min":0.34000000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.376,"15min":0.371},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.32600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45800000000000002,"15min":0.44600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.59699999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757289,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297724,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":643072,"data_stored":310475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.443},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.496,"15min":0.49199999999999999},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76000000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56799999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78800000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.503}]},{"osd":2,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33500000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.505,"15min":0.503},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71899999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.38100000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.40799999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.38400000000000001},"min":{"1min":0.375,"5min":0.22,"15min":0.22},"max":{"1min":0.627,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.42299999999999999,"15min":0.39100000000000001},"min":{"1min":0.39100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.44500000000000001,"15min":0.439},"min":{"1min":0.35499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:51:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:51:55.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:51:56.149 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:56.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:54.999315+0000 mgr.y (mgr.24481) 94 : audit [DBG] from='client.14757 v1:172.21.15.33:0/2426360909' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:56.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:55.346012+0000 osd.2 (osd.2) 35 : cluster [DBG] 5.b scrub starts 2024-02-12T16:51:56.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:55.347013+0000 osd.2 (osd.2) 36 : cluster [DBG] 5.b scrub ok 2024-02-12T16:51:56.615 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:54.999315+0000 mgr.y (mgr.24481) 94 : audit [DBG] from='client.14757 v1:172.21.15.33:0/2426360909' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:56.615 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:55.346012+0000 osd.2 (osd.2) 35 : cluster [DBG] 5.b scrub starts 2024-02-12T16:51:56.615 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:55.347013+0000 osd.2 (osd.2) 36 : cluster [DBG] 5.b scrub ok 2024-02-12T16:51:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:54.999315+0000 mgr.y (mgr.24481) 94 : audit [DBG] from='client.14757 v1:172.21.15.33:0/2426360909' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:55.346012+0000 osd.2 (osd.2) 35 : cluster [DBG] 5.b scrub starts 2024-02-12T16:51:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:55.347013+0000 osd.2 (osd.2) 36 : cluster [DBG] 5.b scrub ok 2024-02-12T16:51:57.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:56.300406+0000 mgr.y (mgr.24481) 95 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:51:57.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:57.048408+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:51:57.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:57.052424+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:51:57.664 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:56.300406+0000 mgr.y (mgr.24481) 95 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:51:57.664 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:57.048408+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:51:57.664 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:57.052424+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:51:57.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:56.300406+0000 mgr.y (mgr.24481) 95 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:51:57.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:57.048408+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:51:57.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:57.052424+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:51:58.106 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:51:58.108 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:51:58.470 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:57.691568+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:51:58.470 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:57.693347+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:51:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:57.691568+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:51:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:57.693347+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:51:58.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:57.691568+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:51:58.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:57.693347+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:51:58.886 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-02-12T16:51:56.300143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":847,"num_read_kb":599,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":488,"ondisk_log_size":488,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381968,"kb_used_data":5200,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579248,"statfs":{"total":767960285184,"available":765521149952,"internally_reserved":0,"allocated":5324800,"data_stored":2474224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002247"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.917837+0000","last_change":"2024-02-12T16:51:45.053060+0000","last_active":"2024-02-12T16:51:45.917837+0000","last_peered":"2024-02-12T16:51:45.917837+0000","last_clean":"2024-02-12T16:51:45.917837+0000","last_became_active":"2024-02-12T16:49:45.223515+0000","last_became_peered":"2024-02-12T16:49:45.223515+0000","last_unstale":"2024-02-12T16:51:45.917837+0000","last_undegraded":"2024-02-12T16:51:45.917837+0000","last_fullsized":"2024-02-12T16:51:45.917837+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:45.052938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:45.052938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553346+0000","last_change":"2024-02-12T16:50:51.824034+0000","last_active":"2024-02-12T16:51:49.553346+0000","last_peered":"2024-02-12T16:51:49.553346+0000","last_clean":"2024-02-12T16:51:49.553346+0000","last_became_active":"2024-02-12T16:49:43.195544+0000","last_became_peered":"2024-02-12T16:49:43.195544+0000","last_unstale":"2024-02-12T16:51:49.553346+0000","last_undegraded":"2024-02-12T16:51:49.553346+0000","last_fullsized":"2024-02-12T16:51:49.553346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.823967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430751+0000","last_change":"2024-02-12T16:50:42.254831+0000","last_active":"2024-02-12T16:51:49.430751+0000","last_peered":"2024-02-12T16:51:49.430751+0000","last_clean":"2024-02-12T16:51:49.430751+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:51:49.430751+0000","last_undegraded":"2024-02-12T16:51:49.430751+0000","last_fullsized":"2024-02-12T16:51:49.430751+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:42.254752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.254752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.919999+0000","last_change":"2024-02-12T16:51:16.075132+0000","last_active":"2024-02-12T16:51:45.919999+0000","last_peered":"2024-02-12T16:51:45.919999+0000","last_clean":"2024-02-12T16:51:45.919999+0000","last_became_active":"2024-02-12T16:49:41.185805+0000","last_became_peered":"2024-02-12T16:49:41.185805+0000","last_unstale":"2024-02-12T16:51:45.919999+0000","last_undegraded":"2024-02-12T16:51:45.919999+0000","last_fullsized":"2024-02-12T16:51:45.919999+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:51:16.075051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.075051+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554742+0000","last_change":"2024-02-12T16:51:21.897841+0000","last_active":"2024-02-12T16:51:49.554742+0000","last_peered":"2024-02-12T16:51:49.554742+0000","last_clean":"2024-02-12T16:51:49.554742+0000","last_became_active":"2024-02-12T16:49:45.216118+0000","last_became_peered":"2024-02-12T16:49:45.216118+0000","last_unstale":"2024-02-12T16:51:49.554742+0000","last_undegraded":"2024-02-12T16:51:49.554742+0000","last_fullsized":"2024-02-12T16:51:49.554742+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:21.897749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:21.897749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429317+0000","last_change":"2024-02-12T16:51:13.011883+0000","last_active":"2024-02-12T16:51:49.429317+0000","last_peered":"2024-02-12T16:51:49.429317+0000","last_clean":"2024-02-12T16:51:49.429317+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:51:49.429317+0000","last_undegraded":"2024-02-12T16:51:49.429317+0000","last_fullsized":"2024-02-12T16:51:49.429317+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.011814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.011814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918407+0000","last_change":"2024-02-12T16:51:38.073505+0000","last_active":"2024-02-12T16:51:45.918407+0000","last_peered":"2024-02-12T16:51:45.918407+0000","last_clean":"2024-02-12T16:51:45.918407+0000","last_became_active":"2024-02-12T16:49:39.102172+0000","last_became_peered":"2024-02-12T16:49:39.102172+0000","last_unstale":"2024-02-12T16:51:45.918407+0000","last_undegraded":"2024-02-12T16:51:45.918407+0000","last_fullsized":"2024-02-12T16:51:45.918407+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.073441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.073441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553643+0000","last_change":"2024-02-12T16:51:17.900217+0000","last_active":"2024-02-12T16:51:49.553643+0000","last_peered":"2024-02-12T16:51:49.553643+0000","last_clean":"2024-02-12T16:51:49.553643+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:51:49.553643+0000","last_undegraded":"2024-02-12T16:51:49.553643+0000","last_fullsized":"2024-02-12T16:51:49.553643+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:17.900093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:17.900093+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429221+0000","last_change":"2024-02-12T16:50:47.317845+0000","last_active":"2024-02-12T16:51:49.429221+0000","last_peered":"2024-02-12T16:51:49.429221+0000","last_clean":"2024-02-12T16:51:49.429221+0000","last_became_active":"2024-02-12T16:49:45.218184+0000","last_became_peered":"2024-02-12T16:49:45.218184+0000","last_unstale":"2024-02-12T16:51:49.429221+0000","last_undegraded":"2024-02-12T16:51:49.429221+0000","last_fullsized":"2024-02-12T16:51:49.429221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:47.317736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.317736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554606+0000","last_change":"2024-02-12T16:50:46.095997+0000","last_active":"2024-02-12T16:51:49.554606+0000","last_peered":"2024-02-12T16:51:49.554606+0000","last_clean":"2024-02-12T16:51:49.554606+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:51:49.554606+0000","last_undegraded":"2024-02-12T16:51:49.554606+0000","last_fullsized":"2024-02-12T16:51:49.554606+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.095924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554888+0000","last_change":"2024-02-12T16:51:04.051422+0000","last_active":"2024-02-12T16:51:49.554888+0000","last_peered":"2024-02-12T16:51:49.554888+0000","last_clean":"2024-02-12T16:51:49.554888+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:51:49.554888+0000","last_undegraded":"2024-02-12T16:51:49.554888+0000","last_fullsized":"2024-02-12T16:51:49.554888+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:51:04.051345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553972+0000","last_change":"2024-02-12T16:51:34.760803+0000","last_active":"2024-02-12T16:51:49.553972+0000","last_peered":"2024-02-12T16:51:49.553972+0000","last_clean":"2024-02-12T16:51:49.553972+0000","last_became_active":"2024-02-12T16:49:41.178018+0000","last_became_peered":"2024-02-12T16:49:41.178018+0000","last_unstale":"2024-02-12T16:51:49.553972+0000","last_undegraded":"2024-02-12T16:51:49.553972+0000","last_fullsized":"2024-02-12T16:51:49.553972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:51:34.760711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.760711+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430277+0000","last_change":"2024-02-12T16:51:22.020495+0000","last_active":"2024-02-12T16:51:49.430277+0000","last_peered":"2024-02-12T16:51:49.430277+0000","last_clean":"2024-02-12T16:51:49.430277+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:51:49.430277+0000","last_undegraded":"2024-02-12T16:51:49.430277+0000","last_fullsized":"2024-02-12T16:51:49.430277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.020430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.020430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429614+0000","last_change":"2024-02-12T16:50:55.322528+0000","last_active":"2024-02-12T16:51:49.429614+0000","last_peered":"2024-02-12T16:51:49.429614+0000","last_clean":"2024-02-12T16:51:49.429614+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:51:49.429614+0000","last_undegraded":"2024-02-12T16:51:49.429614+0000","last_fullsized":"2024-02-12T16:51:49.429614+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.322446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554859+0000","last_change":"2024-02-12T16:51:01.027931+0000","last_active":"2024-02-12T16:51:49.554859+0000","last_peered":"2024-02-12T16:51:49.554859+0000","last_clean":"2024-02-12T16:51:49.554859+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:51:49.554859+0000","last_undegraded":"2024-02-12T16:51:49.554859+0000","last_fullsized":"2024-02-12T16:51:49.554859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:01.027841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.027841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554974+0000","last_change":"2024-02-12T16:51:02.857339+0000","last_active":"2024-02-12T16:51:49.554974+0000","last_peered":"2024-02-12T16:51:49.554974+0000","last_clean":"2024-02-12T16:51:49.554974+0000","last_became_active":"2024-02-12T16:49:41.181117+0000","last_became_peered":"2024-02-12T16:49:41.181117+0000","last_unstale":"2024-02-12T16:51:49.554974+0000","last_undegraded":"2024-02-12T16:51:49.554974+0000","last_fullsized":"2024-02-12T16:51:49.554974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:02.857234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.857234+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918421+0000","last_change":"2024-02-12T16:51:03.001426+0000","last_active":"2024-02-12T16:51:45.918421+0000","last_peered":"2024-02-12T16:51:45.918421+0000","last_clean":"2024-02-12T16:51:45.918421+0000","last_became_active":"2024-02-12T16:49:45.221559+0000","last_became_peered":"2024-02-12T16:49:45.221559+0000","last_unstale":"2024-02-12T16:51:45.918421+0000","last_undegraded":"2024-02-12T16:51:45.918421+0000","last_fullsized":"2024-02-12T16:51:45.918421+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:03.001318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.001318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553887+0000","last_change":"2024-02-12T16:50:54.852577+0000","last_active":"2024-02-12T16:51:49.553887+0000","last_peered":"2024-02-12T16:51:49.553887+0000","last_clean":"2024-02-12T16:51:49.553887+0000","last_became_active":"2024-02-12T16:49:43.201709+0000","last_became_peered":"2024-02-12T16:49:43.201709+0000","last_unstale":"2024-02-12T16:51:49.553887+0000","last_undegraded":"2024-02-12T16:51:49.553887+0000","last_fullsized":"2024-02-12T16:51:49.553887+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:54.852481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.852481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918061+0000","last_change":"2024-02-12T16:51:29.066176+0000","last_active":"2024-02-12T16:51:45.918061+0000","last_peered":"2024-02-12T16:51:45.918061+0000","last_clean":"2024-02-12T16:51:45.918061+0000","last_became_active":"2024-02-12T16:49:39.105438+0000","last_became_peered":"2024-02-12T16:49:39.105438+0000","last_unstale":"2024-02-12T16:51:45.918061+0000","last_undegraded":"2024-02-12T16:51:45.918061+0000","last_fullsized":"2024-02-12T16:51:45.918061+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:29.066107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:29.066107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918151+0000","last_change":"2024-02-12T16:51:23.148923+0000","last_active":"2024-02-12T16:51:45.918151+0000","last_peered":"2024-02-12T16:51:45.918151+0000","last_clean":"2024-02-12T16:51:45.918151+0000","last_became_active":"2024-02-12T16:49:41.181980+0000","last_became_peered":"2024-02-12T16:49:41.181980+0000","last_unstale":"2024-02-12T16:51:45.918151+0000","last_undegraded":"2024-02-12T16:51:45.918151+0000","last_fullsized":"2024-02-12T16:51:45.918151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:51:23.148783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:23.148783+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554774+0000","last_change":"2024-02-12T16:51:10.898681+0000","last_active":"2024-02-12T16:51:49.554774+0000","last_peered":"2024-02-12T16:51:49.554774+0000","last_clean":"2024-02-12T16:51:49.554774+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:51:49.554774+0000","last_undegraded":"2024-02-12T16:51:49.554774+0000","last_fullsized":"2024-02-12T16:51:49.554774+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:51:10.898594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430417+0000","last_change":"2024-02-12T16:51:20.972423+0000","last_active":"2024-02-12T16:51:49.430417+0000","last_peered":"2024-02-12T16:51:49.430417+0000","last_clean":"2024-02-12T16:51:49.430417+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:51:49.430417+0000","last_undegraded":"2024-02-12T16:51:49.430417+0000","last_fullsized":"2024-02-12T16:51:49.430417+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.972298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.972298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554749+0000","last_change":"2024-02-12T16:51:44.303455+0000","last_active":"2024-02-12T16:51:49.554749+0000","last_peered":"2024-02-12T16:51:49.554749+0000","last_clean":"2024-02-12T16:51:49.554749+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:51:49.554749+0000","last_undegraded":"2024-02-12T16:51:49.554749+0000","last_fullsized":"2024-02-12T16:51:49.554749+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:51:44.303379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417412+0000","last_change":"2024-02-12T16:50:47.282788+0000","last_active":"2024-02-12T16:51:45.417412+0000","last_peered":"2024-02-12T16:51:45.417412+0000","last_clean":"2024-02-12T16:51:45.417412+0000","last_became_active":"2024-02-12T16:49:41.180319+0000","last_became_peered":"2024-02-12T16:49:41.180319+0000","last_unstale":"2024-02-12T16:51:45.417412+0000","last_undegraded":"2024-02-12T16:51:45.417412+0000","last_fullsized":"2024-02-12T16:51:45.417412+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:50:47.282711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.282711+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430218+0000","last_change":"2024-02-12T16:51:08.888476+0000","last_active":"2024-02-12T16:51:49.430218+0000","last_peered":"2024-02-12T16:51:49.430218+0000","last_clean":"2024-02-12T16:51:49.430218+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:51:49.430218+0000","last_undegraded":"2024-02-12T16:51:49.430218+0000","last_fullsized":"2024-02-12T16:51:49.430218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:08.888390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.888390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431367+0000","last_change":"2024-02-12T16:50:46.144078+0000","last_active":"2024-02-12T16:51:49.431367+0000","last_peered":"2024-02-12T16:51:49.431367+0000","last_clean":"2024-02-12T16:51:49.431367+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:51:49.431367+0000","last_undegraded":"2024-02-12T16:51:49.431367+0000","last_fullsized":"2024-02-12T16:51:49.431367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.144010+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.919703+0000","last_change":"2024-02-12T16:51:09.094169+0000","last_active":"2024-02-12T16:51:45.919703+0000","last_peered":"2024-02-12T16:51:45.919703+0000","last_clean":"2024-02-12T16:51:45.919703+0000","last_became_active":"2024-02-12T16:49:39.101492+0000","last_became_peered":"2024-02-12T16:49:39.101492+0000","last_unstale":"2024-02-12T16:51:45.919703+0000","last_undegraded":"2024-02-12T16:51:45.919703+0000","last_fullsized":"2024-02-12T16:51:45.919703+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:09.094067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:09.094067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555258+0000","last_change":"2024-02-12T16:51:14.042938+0000","last_active":"2024-02-12T16:51:49.555258+0000","last_peered":"2024-02-12T16:51:49.555258+0000","last_clean":"2024-02-12T16:51:49.555258+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:51:49.555258+0000","last_undegraded":"2024-02-12T16:51:49.555258+0000","last_fullsized":"2024-02-12T16:51:49.555258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:14.042858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:14.042858+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555225+0000","last_change":"2024-02-12T16:51:28.104047+0000","last_active":"2024-02-12T16:51:49.555225+0000","last_peered":"2024-02-12T16:51:49.555225+0000","last_clean":"2024-02-12T16:51:49.555225+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:51:49.555225+0000","last_undegraded":"2024-02-12T16:51:49.555225+0000","last_fullsized":"2024-02-12T16:51:49.555225+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.103969+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233167+0000","last_change":"2024-02-12T16:51:08.073509+0000","last_active":"2024-02-12T16:51:52.233167+0000","last_peered":"2024-02-12T16:51:52.233167+0000","last_clean":"2024-02-12T16:51:52.233167+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:51:52.233167+0000","last_undegraded":"2024-02-12T16:51:52.233167+0000","last_fullsized":"2024-02-12T16:51:52.233167+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:08.073428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.073428+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429156+0000","last_change":"2024-02-12T16:51:01.224070+0000","last_active":"2024-02-12T16:51:49.429156+0000","last_peered":"2024-02-12T16:51:49.429156+0000","last_clean":"2024-02-12T16:51:49.429156+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:51:49.429156+0000","last_undegraded":"2024-02-12T16:51:49.429156+0000","last_fullsized":"2024-02-12T16:51:49.429156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:01.223973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.223973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431280+0000","last_change":"2024-02-12T16:50:44.211440+0000","last_active":"2024-02-12T16:51:49.431280+0000","last_peered":"2024-02-12T16:51:49.431280+0000","last_clean":"2024-02-12T16:51:49.431280+0000","last_became_active":"2024-02-12T16:49:41.669589+0000","last_became_peered":"2024-02-12T16:49:41.669589+0000","last_unstale":"2024-02-12T16:51:49.431280+0000","last_undegraded":"2024-02-12T16:51:49.431280+0000","last_fullsized":"2024-02-12T16:51:49.431280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:50:44.211369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:44.211369+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:47.043976+0000","last_change":"2024-02-12T16:51:47.043952+0000","last_active":"2024-02-12T16:51:47.043976+0000","last_peered":"2024-02-12T16:51:47.043976+0000","last_clean":"2024-02-12T16:51:47.043976+0000","last_became_active":"2024-02-12T16:49:45.222341+0000","last_became_peered":"2024-02-12T16:49:45.222341+0000","last_unstale":"2024-02-12T16:51:47.043976+0000","last_undegraded":"2024-02-12T16:51:47.043976+0000","last_fullsized":"2024-02-12T16:51:47.043976+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:47.043893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:47.043893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553875+0000","last_change":"2024-02-12T16:51:19.038453+0000","last_active":"2024-02-12T16:51:49.553875+0000","last_peered":"2024-02-12T16:51:49.553875+0000","last_clean":"2024-02-12T16:51:49.553875+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:51:49.553875+0000","last_undegraded":"2024-02-12T16:51:49.553875+0000","last_fullsized":"2024-02-12T16:51:49.553875+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:19.038388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:19.038388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555492+0000","last_change":"2024-02-12T16:51:18.026686+0000","last_active":"2024-02-12T16:51:49.555492+0000","last_peered":"2024-02-12T16:51:49.555492+0000","last_clean":"2024-02-12T16:51:49.555492+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:51:49.555492+0000","last_undegraded":"2024-02-12T16:51:49.555492+0000","last_fullsized":"2024-02-12T16:51:49.555492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:18.026624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.026624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.919953+0000","last_change":"2024-02-12T16:50:51.908417+0000","last_active":"2024-02-12T16:51:45.919953+0000","last_peered":"2024-02-12T16:51:45.919953+0000","last_clean":"2024-02-12T16:51:45.919953+0000","last_became_active":"2024-02-12T16:49:41.669704+0000","last_became_peered":"2024-02-12T16:49:41.669704+0000","last_unstale":"2024-02-12T16:51:45.919953+0000","last_undegraded":"2024-02-12T16:51:45.919953+0000","last_fullsized":"2024-02-12T16:51:45.919953+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:50:51.908326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.908326+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554514+0000","last_change":"2024-02-12T16:51:15.965316+0000","last_active":"2024-02-12T16:51:49.554514+0000","last_peered":"2024-02-12T16:51:49.554514+0000","last_clean":"2024-02-12T16:51:49.554514+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:51:49.554514+0000","last_undegraded":"2024-02-12T16:51:49.554514+0000","last_fullsized":"2024-02-12T16:51:49.554514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:51:15.965186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553920+0000","last_change":"2024-02-12T16:50:55.941652+0000","last_active":"2024-02-12T16:51:49.553920+0000","last_peered":"2024-02-12T16:51:49.553920+0000","last_clean":"2024-02-12T16:51:49.553920+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:51:49.553920+0000","last_undegraded":"2024-02-12T16:51:49.553920+0000","last_fullsized":"2024-02-12T16:51:49.553920+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.941575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.941575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430067+0000","last_change":"2024-02-12T16:51:11.135365+0000","last_active":"2024-02-12T16:51:49.430067+0000","last_peered":"2024-02-12T16:51:49.430067+0000","last_clean":"2024-02-12T16:51:49.430067+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:51:49.430067+0000","last_undegraded":"2024-02-12T16:51:49.430067+0000","last_fullsized":"2024-02-12T16:51:49.430067+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.135261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.917843+0000","last_change":"2024-02-12T16:51:43.072967+0000","last_active":"2024-02-12T16:51:45.917843+0000","last_peered":"2024-02-12T16:51:45.917843+0000","last_clean":"2024-02-12T16:51:45.917843+0000","last_became_active":"2024-02-12T16:49:41.184044+0000","last_became_peered":"2024-02-12T16:49:41.184044+0000","last_unstale":"2024-02-12T16:51:45.917843+0000","last_undegraded":"2024-02-12T16:51:45.917843+0000","last_fullsized":"2024-02-12T16:51:45.917843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:51:43.072899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:43.072899+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.919535+0000","last_change":"2024-02-12T16:51:42.040901+0000","last_active":"2024-02-12T16:51:45.919535+0000","last_peered":"2024-02-12T16:51:45.919535+0000","last_clean":"2024-02-12T16:51:45.919535+0000","last_became_active":"2024-02-12T16:49:45.220469+0000","last_became_peered":"2024-02-12T16:49:45.220469+0000","last_unstale":"2024-02-12T16:51:45.919535+0000","last_undegraded":"2024-02-12T16:51:45.919535+0000","last_fullsized":"2024-02-12T16:51:45.919535+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:51:42.040828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429710+0000","last_change":"2024-02-12T16:50:57.875954+0000","last_active":"2024-02-12T16:51:49.429710+0000","last_peered":"2024-02-12T16:51:49.429710+0000","last_clean":"2024-02-12T16:51:49.429710+0000","last_became_active":"2024-02-12T16:49:43.204615+0000","last_became_peered":"2024-02-12T16:49:43.204615+0000","last_unstale":"2024-02-12T16:51:49.429710+0000","last_undegraded":"2024-02-12T16:51:49.429710+0000","last_fullsized":"2024-02-12T16:51:49.429710+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:57.875871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.875871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430463+0000","last_change":"2024-02-12T16:50:46.810493+0000","last_active":"2024-02-12T16:51:49.430463+0000","last_peered":"2024-02-12T16:51:49.430463+0000","last_clean":"2024-02-12T16:51:49.430463+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:51:49.430463+0000","last_undegraded":"2024-02-12T16:51:49.430463+0000","last_fullsized":"2024-02-12T16:51:49.430463+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:46.810420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.810420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553826+0000","last_change":"2024-02-12T16:51:14.844224+0000","last_active":"2024-02-12T16:51:49.553826+0000","last_peered":"2024-02-12T16:51:49.553826+0000","last_clean":"2024-02-12T16:51:49.553826+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:51:49.553826+0000","last_undegraded":"2024-02-12T16:51:49.553826+0000","last_fullsized":"2024-02-12T16:51:49.553826+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:14.844115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:14.844115+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429922+0000","last_change":"2024-02-12T16:51:28.886399+0000","last_active":"2024-02-12T16:51:49.429922+0000","last_peered":"2024-02-12T16:51:49.429922+0000","last_clean":"2024-02-12T16:51:49.429922+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:51:49.429922+0000","last_undegraded":"2024-02-12T16:51:49.429922+0000","last_fullsized":"2024-02-12T16:51:49.429922+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.886321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918141+0000","last_change":"2024-02-12T16:51:39.040277+0000","last_active":"2024-02-12T16:51:45.918141+0000","last_peered":"2024-02-12T16:51:45.918141+0000","last_clean":"2024-02-12T16:51:45.918141+0000","last_became_active":"2024-02-12T16:49:43.208343+0000","last_became_peered":"2024-02-12T16:49:43.208343+0000","last_unstale":"2024-02-12T16:51:45.918141+0000","last_undegraded":"2024-02-12T16:51:45.918141+0000","last_fullsized":"2024-02-12T16:51:45.918141+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.040203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.040203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554424+0000","last_change":"2024-02-12T16:51:11.088535+0000","last_active":"2024-02-12T16:51:49.554424+0000","last_peered":"2024-02-12T16:51:49.554424+0000","last_clean":"2024-02-12T16:51:49.554424+0000","last_became_active":"2024-02-12T16:49:39.107471+0000","last_became_peered":"2024-02-12T16:49:39.107471+0000","last_unstale":"2024-02-12T16:51:49.554424+0000","last_undegraded":"2024-02-12T16:51:49.554424+0000","last_fullsized":"2024-02-12T16:51:49.554424+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:11.088412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.088412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429647+0000","last_change":"2024-02-12T16:51:01.759881+0000","last_active":"2024-02-12T16:51:49.429647+0000","last_peered":"2024-02-12T16:51:49.429647+0000","last_clean":"2024-02-12T16:51:49.429647+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:51:49.429647+0000","last_undegraded":"2024-02-12T16:51:49.429647+0000","last_fullsized":"2024-02-12T16:51:49.429647+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:51:01.759802+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554507+0000","last_change":"2024-02-12T16:51:24.897238+0000","last_active":"2024-02-12T16:51:49.554507+0000","last_peered":"2024-02-12T16:51:49.554507+0000","last_clean":"2024-02-12T16:51:49.554507+0000","last_became_active":"2024-02-12T16:49:45.217958+0000","last_became_peered":"2024-02-12T16:49:45.217958+0000","last_unstale":"2024-02-12T16:51:49.554507+0000","last_undegraded":"2024-02-12T16:51:49.554507+0000","last_fullsized":"2024-02-12T16:51:49.554507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.897149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.897149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.503680+0000","last_change":"2024-02-12T16:51:44.091513+0000","last_active":"2024-02-12T16:51:49.503680+0000","last_peered":"2024-02-12T16:51:49.503680+0000","last_clean":"2024-02-12T16:51:49.503680+0000","last_became_active":"2024-02-12T16:49:43.206068+0000","last_became_peered":"2024-02-12T16:49:43.206068+0000","last_unstale":"2024-02-12T16:51:49.503680+0000","last_undegraded":"2024-02-12T16:51:49.503680+0000","last_fullsized":"2024-02-12T16:51:49.503680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:44.091436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:44.091436+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431257+0000","last_change":"2024-02-12T16:51:25.913214+0000","last_active":"2024-02-12T16:51:49.431257+0000","last_peered":"2024-02-12T16:51:49.431257+0000","last_clean":"2024-02-12T16:51:49.431257+0000","last_became_active":"2024-02-12T16:49:39.103401+0000","last_became_peered":"2024-02-12T16:49:39.103401+0000","last_unstale":"2024-02-12T16:51:49.431257+0000","last_undegraded":"2024-02-12T16:51:49.431257+0000","last_fullsized":"2024-02-12T16:51:49.431257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:25.913145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.913145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430029+0000","last_change":"2024-02-12T16:51:25.028420+0000","last_active":"2024-02-12T16:51:49.430029+0000","last_peered":"2024-02-12T16:51:49.430029+0000","last_clean":"2024-02-12T16:51:49.430029+0000","last_became_active":"2024-02-12T16:49:41.669427+0000","last_became_peered":"2024-02-12T16:49:41.669427+0000","last_unstale":"2024-02-12T16:51:49.430029+0000","last_undegraded":"2024-02-12T16:51:49.430029+0000","last_fullsized":"2024-02-12T16:51:49.430029+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:25.028339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.028339+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430918+0000","last_change":"2024-02-12T16:50:49.214010+0000","last_active":"2024-02-12T16:51:49.430918+0000","last_peered":"2024-02-12T16:51:49.430918+0000","last_clean":"2024-02-12T16:51:49.430918+0000","last_became_active":"2024-02-12T16:49:45.212245+0000","last_became_peered":"2024-02-12T16:49:45.212245+0000","last_unstale":"2024-02-12T16:51:49.430918+0000","last_undegraded":"2024-02-12T16:51:49.430918+0000","last_fullsized":"2024-02-12T16:51:49.430918+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:50:49.213945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233168+0000","last_change":"2024-02-12T16:51:09.159161+0000","last_active":"2024-02-12T16:51:52.233168+0000","last_peered":"2024-02-12T16:51:52.233168+0000","last_clean":"2024-02-12T16:51:52.233168+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:51:52.233168+0000","last_undegraded":"2024-02-12T16:51:52.233168+0000","last_fullsized":"2024-02-12T16:51:52.233168+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:09.159070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:09.159070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429349+0000","last_change":"2024-02-12T16:51:20.103380+0000","last_active":"2024-02-12T16:51:49.429349+0000","last_peered":"2024-02-12T16:51:49.429349+0000","last_clean":"2024-02-12T16:51:49.429349+0000","last_became_active":"2024-02-12T16:49:39.104936+0000","last_became_peered":"2024-02-12T16:49:39.104936+0000","last_unstale":"2024-02-12T16:51:49.429349+0000","last_undegraded":"2024-02-12T16:51:49.429349+0000","last_fullsized":"2024-02-12T16:51:49.429349+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.103297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.103297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918580+0000","last_change":"2024-02-12T16:51:36.026655+0000","last_active":"2024-02-12T16:51:45.918580+0000","last_peered":"2024-02-12T16:51:45.918580+0000","last_clean":"2024-02-12T16:51:45.918580+0000","last_became_active":"2024-02-12T16:51:04.116639+0000","last_became_peered":"2024-02-12T16:51:04.116639+0000","last_unstale":"2024-02-12T16:51:45.918580+0000","last_undegraded":"2024-02-12T16:51:45.918580+0000","last_fullsized":"2024-02-12T16:51:45.918580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:36.026569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:36.026569+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554808+0000","last_change":"2024-02-12T16:51:20.041621+0000","last_active":"2024-02-12T16:51:49.554808+0000","last_peered":"2024-02-12T16:51:49.554808+0000","last_clean":"2024-02-12T16:51:49.554808+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:51:49.554808+0000","last_undegraded":"2024-02-12T16:51:49.554808+0000","last_fullsized":"2024-02-12T16:51:49.554808+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.041556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.041556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430008+0000","last_change":"2024-02-12T16:50:45.275564+0000","last_active":"2024-02-12T16:51:49.430008+0000","last_peered":"2024-02-12T16:51:49.430008+0000","last_clean":"2024-02-12T16:51:49.430008+0000","last_became_active":"2024-02-12T16:49:43.201595+0000","last_became_peered":"2024-02-12T16:49:43.201595+0000","last_unstale":"2024-02-12T16:51:49.430008+0000","last_undegraded":"2024-02-12T16:51:49.430008+0000","last_fullsized":"2024-02-12T16:51:49.430008+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:45.275469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:45.275469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554009+0000","last_change":"2024-02-12T16:51:19.027311+0000","last_active":"2024-02-12T16:51:49.554009+0000","last_peered":"2024-02-12T16:51:49.554009+0000","last_clean":"2024-02-12T16:51:49.554009+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:51:49.554009+0000","last_undegraded":"2024-02-12T16:51:49.554009+0000","last_fullsized":"2024-02-12T16:51:49.554009+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:19.027246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:19.027246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918231+0000","last_change":"2024-02-12T16:51:28.108097+0000","last_active":"2024-02-12T16:51:45.918231+0000","last_peered":"2024-02-12T16:51:45.918231+0000","last_clean":"2024-02-12T16:51:45.918231+0000","last_became_active":"2024-02-12T16:49:41.669651+0000","last_became_peered":"2024-02-12T16:49:41.669651+0000","last_unstale":"2024-02-12T16:51:45.918231+0000","last_undegraded":"2024-02-12T16:51:45.918231+0000","last_fullsized":"2024-02-12T16:51:45.918231+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:51:28.108036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.108036+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417173+0000","last_change":"2024-02-12T16:51:03.235610+0000","last_active":"2024-02-12T16:51:45.417173+0000","last_peered":"2024-02-12T16:51:45.417173+0000","last_clean":"2024-02-12T16:51:45.417173+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:51:45.417173+0000","last_undegraded":"2024-02-12T16:51:45.417173+0000","last_fullsized":"2024-02-12T16:51:45.417173+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:03.235544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.235544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918345+0000","last_change":"2024-02-12T16:51:32.014849+0000","last_active":"2024-02-12T16:51:45.918345+0000","last_peered":"2024-02-12T16:51:45.918345+0000","last_clean":"2024-02-12T16:51:45.918345+0000","last_became_active":"2024-02-12T16:49:43.207454+0000","last_became_peered":"2024-02-12T16:49:43.207454+0000","last_unstale":"2024-02-12T16:51:45.918345+0000","last_undegraded":"2024-02-12T16:51:45.918345+0000","last_fullsized":"2024-02-12T16:51:45.918345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.014772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416551+0000","last_change":"2024-02-12T16:51:39.307549+0000","last_active":"2024-02-12T16:51:45.416551+0000","last_peered":"2024-02-12T16:51:45.416551+0000","last_clean":"2024-02-12T16:51:45.416551+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:51:45.416551+0000","last_undegraded":"2024-02-12T16:51:45.416551+0000","last_fullsized":"2024-02-12T16:51:45.416551+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.307475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.307475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431086+0000","last_change":"2024-02-12T16:51:17.050140+0000","last_active":"2024-02-12T16:51:49.431086+0000","last_peered":"2024-02-12T16:51:49.431086+0000","last_clean":"2024-02-12T16:51:49.431086+0000","last_became_active":"2024-02-12T16:49:41.668665+0000","last_became_peered":"2024-02-12T16:49:41.668665+0000","last_unstale":"2024-02-12T16:51:49.431086+0000","last_undegraded":"2024-02-12T16:51:49.431086+0000","last_fullsized":"2024-02-12T16:51:49.431086+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:51:17.050077+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918230+0000","last_change":"2024-02-12T16:51:33.032870+0000","last_active":"2024-02-12T16:51:45.918230+0000","last_peered":"2024-02-12T16:51:45.918230+0000","last_clean":"2024-02-12T16:51:45.918230+0000","last_became_active":"2024-02-12T16:49:45.220328+0000","last_became_peered":"2024-02-12T16:49:45.220328+0000","last_unstale":"2024-02-12T16:51:45.918230+0000","last_undegraded":"2024-02-12T16:51:45.918230+0000","last_fullsized":"2024-02-12T16:51:45.918230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:33.032784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:33.032784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"58'2","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554196+0000","last_change":"2024-02-12T16:50:42.866331+0000","last_active":"2024-02-12T16:51:49.554196+0000","last_peered":"2024-02-12T16:51:49.554196+0000","last_clean":"2024-02-12T16:51:49.554196+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:51:49.554196+0000","last_undegraded":"2024-02-12T16:51:49.554196+0000","last_fullsized":"2024-02-12T16:51:49.554196+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-02-12T16:50:42.866263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.866263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555038+0000","last_change":"2024-02-12T16:50:46.750203+0000","last_active":"2024-02-12T16:51:49.555038+0000","last_peered":"2024-02-12T16:51:49.555038+0000","last_clean":"2024-02-12T16:51:49.555038+0000","last_became_active":"2024-02-12T16:49:41.177967+0000","last_became_peered":"2024-02-12T16:49:41.177967+0000","last_unstale":"2024-02-12T16:51:49.555038+0000","last_undegraded":"2024-02-12T16:51:49.555038+0000","last_fullsized":"2024-02-12T16:51:49.555038+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:50:46.750065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:46.750065+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416527+0000","last_change":"2024-02-12T16:51:22.127898+0000","last_active":"2024-02-12T16:51:45.416527+0000","last_peered":"2024-02-12T16:51:45.416527+0000","last_clean":"2024-02-12T16:51:45.416527+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:51:45.416527+0000","last_undegraded":"2024-02-12T16:51:45.416527+0000","last_fullsized":"2024-02-12T16:51:45.416527+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.127831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429954+0000","last_change":"2024-02-12T16:50:50.186122+0000","last_active":"2024-02-12T16:51:49.429954+0000","last_peered":"2024-02-12T16:51:49.429954+0000","last_clean":"2024-02-12T16:51:49.429954+0000","last_became_active":"2024-02-12T16:49:45.217626+0000","last_became_peered":"2024-02-12T16:49:45.217626+0000","last_unstale":"2024-02-12T16:51:49.429954+0000","last_undegraded":"2024-02-12T16:51:49.429954+0000","last_fullsized":"2024-02-12T16:51:49.429954+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:50.186021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.186021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430067+0000","last_change":"2024-02-12T16:51:23.967921+0000","last_active":"2024-02-12T16:51:49.430067+0000","last_peered":"2024-02-12T16:51:49.430067+0000","last_clean":"2024-02-12T16:51:49.430067+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:51:49.430067+0000","last_undegraded":"2024-02-12T16:51:49.430067+0000","last_fullsized":"2024-02-12T16:51:49.430067+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:23.967848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:23.967848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416757+0000","last_change":"2024-02-12T16:51:04.121170+0000","last_active":"2024-02-12T16:51:45.416757+0000","last_peered":"2024-02-12T16:51:45.416757+0000","last_clean":"2024-02-12T16:51:45.416757+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:51:45.416757+0000","last_undegraded":"2024-02-12T16:51:45.416757+0000","last_fullsized":"2024-02-12T16:51:45.416757+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:50:42.876140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:42.876140+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233116+0000","last_change":"2024-02-12T16:51:11.206549+0000","last_active":"2024-02-12T16:51:52.233116+0000","last_peered":"2024-02-12T16:51:52.233116+0000","last_clean":"2024-02-12T16:51:52.233116+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:51:52.233116+0000","last_undegraded":"2024-02-12T16:51:52.233116+0000","last_fullsized":"2024-02-12T16:51:52.233116+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:11.206420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.206420+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918495+0000","last_change":"2024-02-12T16:51:26.163640+0000","last_active":"2024-02-12T16:51:45.918495+0000","last_peered":"2024-02-12T16:51:45.918495+0000","last_clean":"2024-02-12T16:51:45.918495+0000","last_became_active":"2024-02-12T16:49:45.215520+0000","last_became_peered":"2024-02-12T16:49:45.215520+0000","last_unstale":"2024-02-12T16:51:45.918495+0000","last_undegraded":"2024-02-12T16:51:45.918495+0000","last_fullsized":"2024-02-12T16:51:45.918495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:26.163568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:26.163568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429346+0000","last_change":"2024-02-12T16:50:50.253026+0000","last_active":"2024-02-12T16:51:49.429346+0000","last_peered":"2024-02-12T16:51:49.429346+0000","last_clean":"2024-02-12T16:51:49.429346+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:51:49.429346+0000","last_undegraded":"2024-02-12T16:51:49.429346+0000","last_fullsized":"2024-02-12T16:51:49.429346+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:50.252952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.252952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233130+0000","last_change":"2024-02-12T16:51:37.878869+0000","last_active":"2024-02-12T16:51:52.233130+0000","last_peered":"2024-02-12T16:51:52.233130+0000","last_clean":"2024-02-12T16:51:52.233130+0000","last_became_active":"2024-02-12T16:49:41.179610+0000","last_became_peered":"2024-02-12T16:49:41.179610+0000","last_unstale":"2024-02-12T16:51:52.233130+0000","last_undegraded":"2024-02-12T16:51:52.233130+0000","last_fullsized":"2024-02-12T16:51:52.233130+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-12T16:51:37.878788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.878788+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430620+0000","last_change":"2024-02-12T16:50:57.182750+0000","last_active":"2024-02-12T16:51:49.430620+0000","last_peered":"2024-02-12T16:51:49.430620+0000","last_clean":"2024-02-12T16:51:49.430620+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:51:49.430620+0000","last_undegraded":"2024-02-12T16:51:49.430620+0000","last_fullsized":"2024-02-12T16:51:49.430620+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:57.182679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.182679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555401+0000","last_change":"2024-02-12T16:51:22.062228+0000","last_active":"2024-02-12T16:51:49.555401+0000","last_peered":"2024-02-12T16:51:49.555401+0000","last_clean":"2024-02-12T16:51:49.555401+0000","last_became_active":"2024-02-12T16:49:45.221460+0000","last_became_peered":"2024-02-12T16:49:45.221460+0000","last_unstale":"2024-02-12T16:51:49.555401+0000","last_undegraded":"2024-02-12T16:51:49.555401+0000","last_fullsized":"2024-02-12T16:51:49.555401+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.062124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.062124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":177,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.452896+0000","last_change":"2024-02-12T16:50:49.760225+0000","last_active":"2024-02-12T16:51:52.452896+0000","last_peered":"2024-02-12T16:51:52.452896+0000","last_clean":"2024-02-12T16:51:52.452896+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:51:52.452896+0000","last_undegraded":"2024-02-12T16:51:52.452896+0000","last_fullsized":"2024-02-12T16:51:52.452896+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:50:49.760157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:49.760157+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":127,"num_read_kb":122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416693+0000","last_change":"2024-02-12T16:51:02.278460+0000","last_active":"2024-02-12T16:51:45.416693+0000","last_peered":"2024-02-12T16:51:45.416693+0000","last_clean":"2024-02-12T16:51:45.416693+0000","last_became_active":"2024-02-12T16:49:39.105685+0000","last_became_peered":"2024-02-12T16:49:39.105685+0000","last_unstale":"2024-02-12T16:51:45.416693+0000","last_undegraded":"2024-02-12T16:51:45.416693+0000","last_fullsized":"2024-02-12T16:51:45.416693+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.278337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.278337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431294+0000","last_change":"2024-02-12T16:51:18.950103+0000","last_active":"2024-02-12T16:51:49.431294+0000","last_peered":"2024-02-12T16:51:49.431294+0000","last_clean":"2024-02-12T16:51:49.431294+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:51:49.431294+0000","last_undegraded":"2024-02-12T16:51:49.431294+0000","last_fullsized":"2024-02-12T16:51:49.431294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:51:18.950031+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:51:18.950031+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.950031+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417183+0000","last_change":"2024-02-12T16:51:24.083385+0000","last_active":"2024-02-12T16:51:45.417183+0000","last_peered":"2024-02-12T16:51:45.417183+0000","last_clean":"2024-02-12T16:51:45.417183+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:51:45.417183+0000","last_undegraded":"2024-02-12T16:51:45.417183+0000","last_fullsized":"2024-02-12T16:51:45.417183+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.083270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.083270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430257+0000","last_change":"2024-02-12T16:51:00.237219+0000","last_active":"2024-02-12T16:51:49.430257+0000","last_peered":"2024-02-12T16:51:49.430257+0000","last_clean":"2024-02-12T16:51:49.430257+0000","last_became_active":"2024-02-12T16:49:45.213152+0000","last_became_peered":"2024-02-12T16:49:45.213152+0000","last_unstale":"2024-02-12T16:51:49.430257+0000","last_undegraded":"2024-02-12T16:51:49.430257+0000","last_fullsized":"2024-02-12T16:51:49.430257+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:51:00.237119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.917938+0000","last_change":"2024-02-12T16:51:08.054217+0000","last_active":"2024-02-12T16:51:45.917938+0000","last_peered":"2024-02-12T16:51:45.917938+0000","last_clean":"2024-02-12T16:51:45.917938+0000","last_became_active":"2024-02-12T16:49:39.105761+0000","last_became_peered":"2024-02-12T16:49:39.105761+0000","last_unstale":"2024-02-12T16:51:45.917938+0000","last_undegraded":"2024-02-12T16:51:45.917938+0000","last_fullsized":"2024-02-12T16:51:45.917938+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:08.054114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.054114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429532+0000","last_change":"2024-02-12T16:50:56.320321+0000","last_active":"2024-02-12T16:51:49.429532+0000","last_peered":"2024-02-12T16:51:49.429532+0000","last_clean":"2024-02-12T16:51:49.429532+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:51:49.429532+0000","last_undegraded":"2024-02-12T16:51:49.429532+0000","last_fullsized":"2024-02-12T16:51:49.429532+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:56.320240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:56.320240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553670+0000","last_change":"2024-02-12T16:51:37.184984+0000","last_active":"2024-02-12T16:51:49.553670+0000","last_peered":"2024-02-12T16:51:49.553670+0000","last_clean":"2024-02-12T16:51:49.553670+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:51:49.553670+0000","last_undegraded":"2024-02-12T16:51:49.553670+0000","last_fullsized":"2024-02-12T16:51:49.553670+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:37.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.184907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554395+0000","last_change":"2024-02-12T16:50:57.964701+0000","last_active":"2024-02-12T16:51:49.554395+0000","last_peered":"2024-02-12T16:51:49.554395+0000","last_clean":"2024-02-12T16:51:49.554395+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:51:49.554395+0000","last_undegraded":"2024-02-12T16:51:49.554395+0000","last_fullsized":"2024-02-12T16:51:49.554395+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:50:57.964625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:57.964625+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555066+0000","last_change":"2024-02-12T16:51:05.047096+0000","last_active":"2024-02-12T16:51:49.555066+0000","last_peered":"2024-02-12T16:51:49.555066+0000","last_clean":"2024-02-12T16:51:49.555066+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:51:49.555066+0000","last_undegraded":"2024-02-12T16:51:49.555066+0000","last_fullsized":"2024-02-12T16:51:49.555066+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:05.047021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:05.047021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429796+0000","last_change":"2024-02-12T16:51:36.842712+0000","last_active":"2024-02-12T16:51:49.429796+0000","last_peered":"2024-02-12T16:51:49.429796+0000","last_clean":"2024-02-12T16:51:49.429796+0000","last_became_active":"2024-02-12T16:49:45.218096+0000","last_became_peered":"2024-02-12T16:49:45.218096+0000","last_unstale":"2024-02-12T16:51:49.429796+0000","last_undegraded":"2024-02-12T16:51:49.429796+0000","last_fullsized":"2024-02-12T16:51:49.429796+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:36.842645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:36.842645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430922+0000","last_change":"2024-02-12T16:51:18.079605+0000","last_active":"2024-02-12T16:51:49.430922+0000","last_peered":"2024-02-12T16:51:49.430922+0000","last_clean":"2024-02-12T16:51:49.430922+0000","last_became_active":"2024-02-12T16:49:39.104017+0000","last_became_peered":"2024-02-12T16:49:39.104017+0000","last_unstale":"2024-02-12T16:51:49.430922+0000","last_undegraded":"2024-02-12T16:51:49.430922+0000","last_fullsized":"2024-02-12T16:51:49.430922+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:18.079497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:18.079497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553584+0000","last_change":"2024-02-12T16:51:08.953016+0000","last_active":"2024-02-12T16:51:49.553584+0000","last_peered":"2024-02-12T16:51:49.553584+0000","last_clean":"2024-02-12T16:51:49.553584+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:51:49.553584+0000","last_undegraded":"2024-02-12T16:51:49.553584+0000","last_fullsized":"2024-02-12T16:51:49.553584+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:51:08.952901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:08.952901+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431003+0000","last_change":"2024-02-12T16:51:07.240302+0000","last_active":"2024-02-12T16:51:49.431003+0000","last_peered":"2024-02-12T16:51:49.431003+0000","last_clean":"2024-02-12T16:51:49.431003+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:51:49.431003+0000","last_undegraded":"2024-02-12T16:51:49.431003+0000","last_fullsized":"2024-02-12T16:51:49.431003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:51:07.240236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:07.240236+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430219+0000","last_change":"2024-02-12T16:51:22.070134+0000","last_active":"2024-02-12T16:51:49.430219+0000","last_peered":"2024-02-12T16:51:49.430219+0000","last_clean":"2024-02-12T16:51:49.430219+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:51:49.430219+0000","last_undegraded":"2024-02-12T16:51:49.430219+0000","last_fullsized":"2024-02-12T16:51:49.430219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.070056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.070056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429236+0000","last_change":"2024-02-12T16:51:32.884735+0000","last_active":"2024-02-12T16:51:49.429236+0000","last_peered":"2024-02-12T16:51:49.429236+0000","last_clean":"2024-02-12T16:51:49.429236+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:51:49.429236+0000","last_undegraded":"2024-02-12T16:51:49.429236+0000","last_fullsized":"2024-02-12T16:51:49.429236+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.884670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555458+0000","last_change":"2024-02-12T16:50:56.149313+0000","last_active":"2024-02-12T16:51:49.555458+0000","last_peered":"2024-02-12T16:51:49.555458+0000","last_clean":"2024-02-12T16:51:49.555458+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:51:49.555458+0000","last_undegraded":"2024-02-12T16:51:49.555458+0000","last_fullsized":"2024-02-12T16:51:49.555458+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:56.149242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:56.149242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.745302+0000","last_change":"2024-02-12T16:51:52.745302+0000","last_active":"2024-02-12T16:51:52.745302+0000","last_peered":"2024-02-12T16:51:52.745302+0000","last_clean":"2024-02-12T16:51:52.745302+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:51:52.745302+0000","last_undegraded":"2024-02-12T16:51:52.745302+0000","last_fullsized":"2024-02-12T16:51:52.745302+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:51:52.745190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:52.745190+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554673+0000","last_change":"2024-02-12T16:51:10.119500+0000","last_active":"2024-02-12T16:51:49.554673+0000","last_peered":"2024-02-12T16:51:49.554673+0000","last_clean":"2024-02-12T16:51:49.554673+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:51:49.554673+0000","last_undegraded":"2024-02-12T16:51:49.554673+0000","last_fullsized":"2024-02-12T16:51:49.554673+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:10.119399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:10.119399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555196+0000","last_change":"2024-02-12T16:51:32.837004+0000","last_active":"2024-02-12T16:51:49.555196+0000","last_peered":"2024-02-12T16:51:49.555196+0000","last_clean":"2024-02-12T16:51:49.555196+0000","last_became_active":"2024-02-12T16:49:45.222560+0000","last_became_peered":"2024-02-12T16:49:45.222560+0000","last_unstale":"2024-02-12T16:51:49.555196+0000","last_undegraded":"2024-02-12T16:51:49.555196+0000","last_fullsized":"2024-02-12T16:51:49.555196+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:32.836928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:32.836928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430867+0000","last_change":"2024-02-12T16:51:16.010678+0000","last_active":"2024-02-12T16:51:49.430867+0000","last_peered":"2024-02-12T16:51:49.430867+0000","last_clean":"2024-02-12T16:51:49.430867+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:51:49.430867+0000","last_undegraded":"2024-02-12T16:51:49.430867+0000","last_fullsized":"2024-02-12T16:51:49.430867+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:16.010512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.010512+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:51.095208+0000","last_change":"2024-02-12T16:51:51.095208+0000","last_active":"2024-02-12T16:51:51.095208+0000","last_peered":"2024-02-12T16:51:51.095208+0000","last_clean":"2024-02-12T16:51:51.095208+0000","last_became_active":"2024-02-12T16:49:41.669593+0000","last_became_peered":"2024-02-12T16:49:41.669593+0000","last_unstale":"2024-02-12T16:51:51.095208+0000","last_undegraded":"2024-02-12T16:51:51.095208+0000","last_fullsized":"2024-02-12T16:51:51.095208+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:51:51.095125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:51.095125+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555114+0000","last_change":"2024-02-12T16:51:35.749314+0000","last_active":"2024-02-12T16:51:49.555114+0000","last_peered":"2024-02-12T16:51:49.555114+0000","last_clean":"2024-02-12T16:51:49.555114+0000","last_became_active":"2024-02-12T16:49:43.202312+0000","last_became_peered":"2024-02-12T16:49:43.202312+0000","last_unstale":"2024-02-12T16:51:49.555114+0000","last_undegraded":"2024-02-12T16:51:49.555114+0000","last_fullsized":"2024-02-12T16:51:49.555114+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:35.749237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:35.749237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555339+0000","last_change":"2024-02-12T16:50:55.111751+0000","last_active":"2024-02-12T16:51:49.555339+0000","last_peered":"2024-02-12T16:51:49.555339+0000","last_clean":"2024-02-12T16:51:49.555339+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:51:49.555339+0000","last_undegraded":"2024-02-12T16:51:49.555339+0000","last_fullsized":"2024-02-12T16:51:49.555339+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:50:55.111675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.111675+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.555416+0000","last_change":"2024-02-12T16:51:16.020537+0000","last_active":"2024-02-12T16:51:49.555416+0000","last_peered":"2024-02-12T16:51:49.555416+0000","last_clean":"2024-02-12T16:51:49.555416+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:51:49.555416+0000","last_undegraded":"2024-02-12T16:51:49.555416+0000","last_fullsized":"2024-02-12T16:51:49.555416+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:16.020471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.020471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554894+0000","last_change":"2024-02-12T16:50:59.823981+0000","last_active":"2024-02-12T16:51:49.554894+0000","last_peered":"2024-02-12T16:51:49.554894+0000","last_clean":"2024-02-12T16:51:49.554894+0000","last_became_active":"2024-02-12T16:49:41.178569+0000","last_became_peered":"2024-02-12T16:49:41.178569+0000","last_unstale":"2024-02-12T16:51:49.554894+0000","last_undegraded":"2024-02-12T16:51:49.554894+0000","last_fullsized":"2024-02-12T16:51:49.554894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:50:59.823898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:59.823898+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.502960+0000","last_change":"2024-02-12T16:51:03.961580+0000","last_active":"2024-02-12T16:51:49.502960+0000","last_peered":"2024-02-12T16:51:49.502960+0000","last_clean":"2024-02-12T16:51:49.502960+0000","last_became_active":"2024-02-12T16:49:43.205391+0000","last_became_peered":"2024-02-12T16:49:43.205391+0000","last_unstale":"2024-02-12T16:51:49.502960+0000","last_undegraded":"2024-02-12T16:51:49.502960+0000","last_fullsized":"2024-02-12T16:51:49.502960+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:03.961503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:03.961503+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.918496+0000","last_change":"2024-02-12T16:51:34.081725+0000","last_active":"2024-02-12T16:51:45.918496+0000","last_peered":"2024-02-12T16:51:45.918496+0000","last_clean":"2024-02-12T16:51:45.918496+0000","last_became_active":"2024-02-12T16:49:45.223659+0000","last_became_peered":"2024-02-12T16:49:45.223659+0000","last_unstale":"2024-02-12T16:51:45.918496+0000","last_undegraded":"2024-02-12T16:51:45.918496+0000","last_fullsized":"2024-02-12T16:51:45.918496+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:34.081644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.081644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430152+0000","last_change":"2024-02-12T16:51:05.201812+0000","last_active":"2024-02-12T16:51:49.430152+0000","last_peered":"2024-02-12T16:51:49.430152+0000","last_clean":"2024-02-12T16:51:49.430152+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:51:49.430152+0000","last_undegraded":"2024-02-12T16:51:49.430152+0000","last_fullsized":"2024-02-12T16:51:49.430152+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:51:05.201749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416650+0000","last_change":"2024-02-12T16:51:04.206339+0000","last_active":"2024-02-12T16:51:45.416650+0000","last_peered":"2024-02-12T16:51:45.416650+0000","last_clean":"2024-02-12T16:51:45.416650+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:51:45.416650+0000","last_undegraded":"2024-02-12T16:51:45.416650+0000","last_fullsized":"2024-02-12T16:51:45.416650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.123059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.123059+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.123059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554864+0000","last_change":"2024-02-12T16:51:30.776952+0000","last_active":"2024-02-12T16:51:49.554864+0000","last_peered":"2024-02-12T16:51:49.554864+0000","last_clean":"2024-02-12T16:51:49.554864+0000","last_became_active":"2024-02-12T16:49:41.178616+0000","last_became_peered":"2024-02-12T16:49:41.178616+0000","last_unstale":"2024-02-12T16:51:49.554864+0000","last_undegraded":"2024-02-12T16:51:49.554864+0000","last_fullsized":"2024-02-12T16:51:49.554864+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:30.776871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:30.776871+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429794+0000","last_change":"2024-02-12T16:51:02.216274+0000","last_active":"2024-02-12T16:51:49.429794+0000","last_peered":"2024-02-12T16:51:49.429794+0000","last_clean":"2024-02-12T16:51:49.429794+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:51:49.429794+0000","last_undegraded":"2024-02-12T16:51:49.429794+0000","last_fullsized":"2024-02-12T16:51:49.429794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.216194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.216194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"80'9","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233168+0000","last_change":"2024-02-12T16:50:50.800052+0000","last_active":"2024-02-12T16:51:52.233168+0000","last_peered":"2024-02-12T16:51:52.233168+0000","last_clean":"2024-02-12T16:51:52.233168+0000","last_became_active":"2024-02-12T16:49:43.197112+0000","last_became_peered":"2024-02-12T16:49:43.197112+0000","last_unstale":"2024-02-12T16:51:52.233168+0000","last_undegraded":"2024-02-12T16:51:52.233168+0000","last_fullsized":"2024-02-12T16:51:52.233168+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:50:50.799984+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429474+0000","last_change":"2024-02-12T16:51:00.778646+0000","last_active":"2024-02-12T16:51:49.429474+0000","last_peered":"2024-02-12T16:51:49.429474+0000","last_clean":"2024-02-12T16:51:49.429474+0000","last_became_active":"2024-02-12T16:49:39.095868+0000","last_became_peered":"2024-02-12T16:49:39.095868+0000","last_unstale":"2024-02-12T16:51:49.429474+0000","last_undegraded":"2024-02-12T16:51:49.429474+0000","last_fullsized":"2024-02-12T16:51:49.429474+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:00.778578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:00.778578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431246+0000","last_change":"2024-02-12T16:51:11.980641+0000","last_active":"2024-02-12T16:51:49.431246+0000","last_peered":"2024-02-12T16:51:49.431246+0000","last_clean":"2024-02-12T16:51:49.431246+0000","last_became_active":"2024-02-12T16:49:41.178544+0000","last_became_peered":"2024-02-12T16:49:41.178544+0000","last_unstale":"2024-02-12T16:51:49.431246+0000","last_undegraded":"2024-02-12T16:51:49.431246+0000","last_fullsized":"2024-02-12T16:51:49.431246+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:51:11.980577+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:51:11.980577+0000","last_clean_scrub_stamp":"2024-02-12T16:51:11.980577+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554514+0000","last_change":"2024-02-12T16:51:38.773791+0000","last_active":"2024-02-12T16:51:49.554514+0000","last_peered":"2024-02-12T16:51:49.554514+0000","last_clean":"2024-02-12T16:51:49.554514+0000","last_became_active":"2024-02-12T16:49:45.217458+0000","last_became_peered":"2024-02-12T16:49:45.217458+0000","last_unstale":"2024-02-12T16:51:49.554514+0000","last_undegraded":"2024-02-12T16:51:49.554514+0000","last_fullsized":"2024-02-12T16:51:49.554514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.773699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.773699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553278+0000","last_change":"2024-02-12T16:51:16.145692+0000","last_active":"2024-02-12T16:51:49.553278+0000","last_peered":"2024-02-12T16:51:49.553278+0000","last_clean":"2024-02-12T16:51:49.553278+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:51:49.553278+0000","last_undegraded":"2024-02-12T16:51:49.553278+0000","last_fullsized":"2024-02-12T16:51:49.553278+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:16.145576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:16.145576+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429564+0000","last_change":"2024-02-12T16:51:02.785509+0000","last_active":"2024-02-12T16:51:49.429564+0000","last_peered":"2024-02-12T16:51:49.429564+0000","last_clean":"2024-02-12T16:51:49.429564+0000","last_became_active":"2024-02-12T16:49:39.101478+0000","last_became_peered":"2024-02-12T16:49:39.101478+0000","last_unstale":"2024-02-12T16:51:49.429564+0000","last_undegraded":"2024-02-12T16:51:49.429564+0000","last_fullsized":"2024-02-12T16:51:49.429564+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:02.785426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:02.785426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553729+0000","last_change":"2024-02-12T16:51:04.882407+0000","last_active":"2024-02-12T16:51:49.553729+0000","last_peered":"2024-02-12T16:51:49.553729+0000","last_clean":"2024-02-12T16:51:49.553729+0000","last_became_active":"2024-02-12T16:49:41.179396+0000","last_became_peered":"2024-02-12T16:49:41.179396+0000","last_unstale":"2024-02-12T16:51:49.553729+0000","last_undegraded":"2024-02-12T16:51:49.553729+0000","last_fullsized":"2024-02-12T16:51:49.553729+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:04.882334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:04.882334+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417404+0000","last_change":"2024-02-12T16:50:55.285750+0000","last_active":"2024-02-12T16:51:45.417404+0000","last_peered":"2024-02-12T16:51:45.417404+0000","last_clean":"2024-02-12T16:51:45.417404+0000","last_became_active":"2024-02-12T16:49:45.220893+0000","last_became_peered":"2024-02-12T16:49:45.220893+0000","last_unstale":"2024-02-12T16:51:45.417404+0000","last_undegraded":"2024-02-12T16:51:45.417404+0000","last_fullsized":"2024-02-12T16:51:45.417404+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:55.285685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:50:55.285685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429848+0000","last_change":"2024-02-12T16:51:13.985683+0000","last_active":"2024-02-12T16:51:49.429848+0000","last_peered":"2024-02-12T16:51:49.429848+0000","last_clean":"2024-02-12T16:51:49.429848+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:51:49.429848+0000","last_undegraded":"2024-02-12T16:51:49.429848+0000","last_fullsized":"2024-02-12T16:51:49.429848+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.985614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.985614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-12T16:51:53.008096+0000","last_change":"2024-02-12T16:51:53.008096+0000","last_active":"2024-02-12T16:51:53.008096+0000","last_peered":"2024-02-12T16:51:53.008096+0000","last_clean":"2024-02-12T16:51:53.008096+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:51:53.008096+0000","last_undegraded":"2024-02-12T16:51:53.008096+0000","last_fullsized":"2024-02-12T16:51:53.008096+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:53.007988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:53.007988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554853+0000","last_change":"2024-02-12T16:51:12.892241+0000","last_active":"2024-02-12T16:51:49.554853+0000","last_peered":"2024-02-12T16:51:49.554853+0000","last_clean":"2024-02-12T16:51:49.554853+0000","last_became_active":"2024-02-12T16:49:41.177416+0000","last_became_peered":"2024-02-12T16:49:41.177416+0000","last_unstale":"2024-02-12T16:51:49.554853+0000","last_undegraded":"2024-02-12T16:51:49.554853+0000","last_fullsized":"2024-02-12T16:51:49.554853+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:51:12.892148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:12.892148+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.553491+0000","last_change":"2024-02-12T16:50:54.135100+0000","last_active":"2024-02-12T16:51:49.553491+0000","last_peered":"2024-02-12T16:51:49.553491+0000","last_clean":"2024-02-12T16:51:49.553491+0000","last_became_active":"2024-02-12T16:49:45.217589+0000","last_became_peered":"2024-02-12T16:49:45.217589+0000","last_unstale":"2024-02-12T16:51:49.553491+0000","last_undegraded":"2024-02-12T16:51:49.553491+0000","last_fullsized":"2024-02-12T16:51:49.553491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.135009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417047+0000","last_change":"2024-02-12T16:50:52.289793+0000","last_active":"2024-02-12T16:51:45.417047+0000","last_peered":"2024-02-12T16:51:45.417047+0000","last_clean":"2024-02-12T16:51:45.417047+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:51:45.417047+0000","last_undegraded":"2024-02-12T16:51:45.417047+0000","last_fullsized":"2024-02-12T16:51:45.417047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:52.289725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:52.289725+0000","last_clean_scrub_stamp":"2024-02-12T16:50:52.289725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.417431+0000","last_change":"2024-02-12T16:50:51.308424+0000","last_active":"2024-02-12T16:51:45.417431+0000","last_peered":"2024-02-12T16:51:45.417431+0000","last_clean":"2024-02-12T16:51:45.417431+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:51:45.417431+0000","last_undegraded":"2024-02-12T16:51:45.417431+0000","last_fullsized":"2024-02-12T16:51:45.417431+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:50:51.308317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:50:51.308317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":104,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.554590+0000","last_change":"2024-02-12T16:51:04.126502+0000","last_active":"2024-02-12T16:51:49.554590+0000","last_peered":"2024-02-12T16:51:49.554590+0000","last_clean":"2024-02-12T16:51:49.554590+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:51:49.554590+0000","last_undegraded":"2024-02-12T16:51:49.554590+0000","last_fullsized":"2024-02-12T16:51:49.554590+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:50:47.792388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:47.792388+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430677+0000","last_change":"2024-02-12T16:51:13.192907+0000","last_active":"2024-02-12T16:51:49.430677+0000","last_peered":"2024-02-12T16:51:49.430677+0000","last_clean":"2024-02-12T16:51:49.430677+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:51:49.430677+0000","last_undegraded":"2024-02-12T16:51:49.430677+0000","last_fullsized":"2024-02-12T16:51:49.430677+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:13.192842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:13.192842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-12T16:51:45.416902+0000","last_change":"2024-02-12T16:51:40.273267+0000","last_active":"2024-02-12T16:51:45.416902+0000","last_peered":"2024-02-12T16:51:45.416902+0000","last_clean":"2024-02-12T16:51:45.416902+0000","last_became_active":"2024-02-12T16:49:43.201946+0000","last_became_peered":"2024-02-12T16:49:43.201946+0000","last_unstale":"2024-02-12T16:51:45.416902+0000","last_undegraded":"2024-02-12T16:51:45.416902+0000","last_fullsized":"2024-02-12T16:51:45.416902+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:40.273180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:40.273180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"58'1","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.430591+0000","last_change":"2024-02-12T16:51:31.889909+0000","last_active":"2024-02-12T16:51:49.430591+0000","last_peered":"2024-02-12T16:51:49.430591+0000","last_clean":"2024-02-12T16:51:49.430591+0000","last_became_active":"2024-02-12T16:49:39.099695+0000","last_became_peered":"2024-02-12T16:49:39.099695+0000","last_unstale":"2024-02-12T16:51:49.430591+0000","last_undegraded":"2024-02-12T16:51:49.430591+0000","last_fullsized":"2024-02-12T16:51:49.430591+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:51:31.889821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:31.889821+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.431339+0000","last_change":"2024-02-12T16:50:45.175080+0000","last_active":"2024-02-12T16:51:49.431339+0000","last_peered":"2024-02-12T16:51:49.431339+0000","last_clean":"2024-02-12T16:51:49.431339+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:51:49.431339+0000","last_undegraded":"2024-02-12T16:51:49.431339+0000","last_fullsized":"2024-02-12T16:51:49.431339+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:50:45.174941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:45.174941+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429255+0000","last_change":"2024-02-12T16:51:24.030089+0000","last_active":"2024-02-12T16:51:49.429255+0000","last_peered":"2024-02-12T16:51:49.429255+0000","last_clean":"2024-02-12T16:51:49.429255+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:51:49.429255+0000","last_undegraded":"2024-02-12T16:51:49.429255+0000","last_fullsized":"2024-02-12T16:51:49.429255+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:24.030017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:24.030017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:51.361621+0000","last_change":"2024-02-12T16:51:51.361621+0000","last_active":"2024-02-12T16:51:51.361621+0000","last_peered":"2024-02-12T16:51:51.361621+0000","last_clean":"2024-02-12T16:51:51.361621+0000","last_became_active":"2024-02-12T16:49:39.105982+0000","last_became_peered":"2024-02-12T16:49:39.105982+0000","last_unstale":"2024-02-12T16:51:51.361621+0000","last_undegraded":"2024-02-12T16:51:51.361621+0000","last_fullsized":"2024-02-12T16:51:51.361621+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:51.361540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:51.361540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:49.429291+0000","last_change":"2024-02-12T16:50:54.302143+0000","last_active":"2024-02-12T16:51:49.429291+0000","last_peered":"2024-02-12T16:51:49.429291+0000","last_clean":"2024-02-12T16:51:49.429291+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:51:49.429291+0000","last_undegraded":"2024-02-12T16:51:49.429291+0000","last_fullsized":"2024-02-12T16:51:49.429291+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:50:54.302060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:50:54.302060+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-12T16:51:52.233143+0000","last_change":"2024-02-12T16:51:07.021402+0000","last_active":"2024-02-12T16:51:52.233143+0000","last_peered":"2024-02-12T16:51:52.233143+0000","last_clean":"2024-02-12T16:51:52.233143+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:51:52.233143+0000","last_undegraded":"2024-02-12T16:51:52.233143+0000","last_fullsized":"2024-02-12T16:51:52.233143+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:07.021338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:07.021338+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":127,"num_read_kb":122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234014,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":311517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69799999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.313},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80500000000000005,"5min":1.056,"15min":1.056},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65900000000000003,"5min":1.208,"15min":1.208},"last":0.36499999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56899999999999995,"15min":0.57999999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72299999999999998,"5min":1.766,"15min":1.766},"last":0.71399999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.61299999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.374},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.51100000000000001,"15min":0.52200000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67000000000000004,"5min":1.327,"15min":1.327},"last":0.432}]},{"osd":5,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68400000000000005,"5min":1.573,"15min":1.573},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53500000000000003,"15min":0.54300000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76400000000000001,"5min":1.621,"15min":1.621},"last":0.39800000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77700000000000002,"5min":1.272,"15min":1.272},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73199999999999998,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.53700000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528355,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":304453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48199999999999998,"15min":0.48299999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67000000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.63700000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65700000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.747,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60599999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72799999999999998,"5min":0.754,"15min":0.754},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.50600000000000001,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64500000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.65700000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.39800000000000002},"min":{"1min":0.28699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.42399999999999999,"15min":0.41799999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.318,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36799999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:51:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.39200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297788,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447364,"statfs":{"total":95995035648,"available":95690100736,"internally_reserved":0,"allocated":708608,"data_stored":313594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35399999999999998,"15min":0.34000000000000002},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39100000000000001,"15min":0.373},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.74199999999999999,"5min":2.778,"15min":2.778},"last":0.376},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41499999999999998,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.629,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.48399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.45600000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.42099999999999999,"15min":0.40899999999999997},"min":{"1min":0.35199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:51:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.40899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77300000000000002,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77200000000000002,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.95099999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.86599999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:51:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.64800000000000002,"15min":0.65800000000000003},"min":{"1min":0.438,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79900000000000004,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.47199999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83999999999999997,"5min":1.109,"15min":1.109},"last":0.79000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297748,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447404,"statfs":{"total":95995035648,"available":95690141696,"internally_reserved":0,"allocated":667648,"data_stored":311201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"min":{"1min":0.255,"5min":0.192,"15min":0.192},"max":{"1min":0.55300000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.35299999999999998,"15min":0.317},"min":{"1min":0.27700000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.61599999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.42299999999999999,"15min":0.39100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.8130000000000002,"5min":3.8130000000000002,"15min":3.8130000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67400000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.40500000000000003,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56399999999999995,"5min":0.622,"15min":0.622},"last":0.376},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.45800000000000002,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.745,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67300000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.84699999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83299999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.45000000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.621}]},{"osd":6,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71299999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.80100000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.45700000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86099999999999999,"5min":0.871,"15min":0.871},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59899999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":305011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.38700000000000001,"15min":0.379},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46000000000000002,"5min":1.595,"15min":1.595},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.41999999999999998,"15min":0.40000000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47799999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.46300000000000002,"15min":0.41699999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.52000000000000002,"5min":6.8040000000000003,"15min":6.8040000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.437,"15min":0.41099999999999998},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51100000000000001,"5min":1.627,"15min":1.627},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47199999999999998,"15min":0.46600000000000003},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.57599999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.503,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.74199999999999999,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.70499999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.61499999999999999,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.69099999999999995,"5min":2.875,"15min":2.875},"last":0.61399999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.67600000000000005,"15min":0.72799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.73099999999999998,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58699999999999997,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69999999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.83899999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90400000000000003,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.89700000000000002,"15min":0.95299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70299999999999996,"5min":7.5629999999999997,"15min":7.5629999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.47699999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.69799999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.86899999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297756,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447396,"statfs":{"total":95995035648,"available":95690133504,"internally_reserved":0,"allocated":675840,"data_stored":306103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.436,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52100000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.42599999999999999,"15min":0.436},"min":{"1min":0.255,"5min":0.216,"15min":0.216},"max":{"1min":0.52300000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42899999999999999,"15min":0.436},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.503,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.442,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.436,"15min":0.44400000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.503,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58899999999999997,"15min":0.61099999999999999},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78100000000000003,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73699999999999999,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.40100000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61199999999999999,"15min":0.63300000000000001},"min":{"1min":0.45000000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77500000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60399999999999998,"15min":0.626},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82199999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76500000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.434},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.496}]},{"osd":7,"last update":"Mon Feb 12 16:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.432,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.53800000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":311870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.55300000000000005},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.51800000000000002},"min":{"1min":0.30399999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.438}]},{"osd":1,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.30299999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66300000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55300000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.57199999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53200000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53700000000000003,"15min":0.55400000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.621}]},{"osd":5,"last update":"Mon Feb 12 16:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.36799999999999999,"15min":0.34699999999999998},"min":{"1min":0.25800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.36099999999999999,"15min":0.34000000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.376,"15min":0.371},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.32600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45800000000000002,"15min":0.44600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.59699999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757289,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297724,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":643072,"data_stored":310475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.443},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.496,"15min":0.49199999999999999},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76000000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56799999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78800000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.503}]},{"osd":2,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33500000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.505,"15min":0.503},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71899999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.38100000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.40799999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.38400000000000001},"min":{"1min":0.375,"5min":0.22,"15min":0.22},"max":{"1min":0.627,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.42299999999999999,"15min":0.39100000000000001},"min":{"1min":0.39100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.44500000000000001,"15min":0.439},"min":{"1min":0.35499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:51:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:51:58.892 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-12T16:51:58.892 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump 2024-02-12T16:51:59.172 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:59.655 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:51:58.090410+0000 mgr.y (mgr.24481) 96 : audit [DBG] from='client.34688 v1:172.21.15.33:0/1996164085' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:59.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:58.300950+0000 mgr.y (mgr.24481) 97 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:51:59.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:59.041368+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:51:59.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:51:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:51:59.044304+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:51:59.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:51:58.090410+0000 mgr.y (mgr.24481) 96 : audit [DBG] from='client.34688 v1:172.21.15.33:0/1996164085' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:59.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:58.300950+0000 mgr.y (mgr.24481) 97 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:51:59.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:59.041368+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:51:59.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:51:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:51:59.044304+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:51:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:51:58.090410+0000 mgr.y (mgr.24481) 96 : audit [DBG] from='client.34688 v1:172.21.15.33:0/1996164085' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:51:59.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:58.300950+0000 mgr.y (mgr.24481) 97 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:51:59.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:59.041368+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:51:59.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:51:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:51:59.044304+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:52:01.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:52:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:00] "GET /metrics HTTP/1.1" 200 172145 "" "Prometheus/2.33.4" 2024-02-12T16:52:01.092 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 91 2024-02-12T16:52:01.092 INFO:teuthology.orchestra.run.smithi033.stdout:fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 2024-02-12T16:52:01.092 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-02-12T16:43:50.282468+0000 2024-02-12T16:52:01.092 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2024-02-12T16:51:52.434353+0000 2024-02-12T16:52:01.092 INFO:teuthology.orchestra.run.smithi033.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:crush_version 18 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:full_ratio 0.95 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:backfillfull_ratio 0.9 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:nearfull_ratio 0.85 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:require_min_compat_client luminous 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:min_compat_client luminous 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:require_osd_release pacific 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:stretch_mode_enabled false 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 59 flags hashpspool stripe_width 0 application rgw 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:pool 3 'iscsi' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rbd 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 61 flags hashpspool stripe_width 0 application rgw 2024-02-12T16:52:01.093 INFO:teuthology.orchestra.run.smithi033.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 application rgw 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:max_osd 8 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 up in weight 1 up_from 8 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.33:6801/2545411201 v1:172.21.15.33:6802/2545411201 exists,up 5dd172eb-ee9d-4700-9b80-60f91c5f2cb7 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 up in weight 1 up_from 13 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.33:6805/1723231708 v1:172.21.15.33:6806/1723231708 exists,up 71231bc4-1716-431d-bd88-f6a3cad0bd24 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) v1:172.21.15.33:6809/3126580659 v1:172.21.15.33:6810/3126580659 exists,up 8a9c7127-d17c-4c44-b666-4211006de47c 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 up in weight 1 up_from 26 up_thru 81 down_at 0 last_clean_interval [0,0) v1:172.21.15.33:6813/3941314957 v1:172.21.15.33:6814/3941314957 exists,up 3cc40671-c8f7-412c-a0f1-987f48a016b9 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 up in weight 1 up_from 32 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6800/544126779 v1:172.21.15.94:6801/544126779 exists,up 0498129c-861c-4f7e-b8ba-219949da3130 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 up in weight 1 up_from 38 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6804/3229884005 v1:172.21.15.94:6805/3229884005 exists,up 05718a1e-1164-4b54-a493-23c1cd3adb50 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 up in weight 1 up_from 45 up_thru 81 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6808/507305558 v1:172.21.15.94:6809/507305558 exists,up dc48640f-8f27-405c-a9a4-cf1e3e6c8821 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 up in weight 1 up_from 54 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6812/3401977873 v1:172.21.15.94:6813/3401977873 exists,up 7d55aa3f-06b6-4fc3-97fd-62b535f78b66 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.8 [7,2] 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.2 [1,2] 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.a [1,2] 2024-02-12T16:52:01.094 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.14 [1,2] 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/2829464018 expires 2024-02-13T16:50:02.257936+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/1835128904 expires 2024-02-13T16:50:02.257936+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2277321750 expires 2024-02-13T16:50:02.257936+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2116602769 expires 2024-02-13T16:50:02.257936+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/3967118778 expires 2024-02-13T16:44:46.313346+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2298361247 expires 2024-02-13T16:44:46.313346+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/218227980 expires 2024-02-13T16:44:22.727241+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2070582473 expires 2024-02-13T16:44:22.727241+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/4291575956 expires 2024-02-13T16:44:46.313346+0000 2024-02-12T16:52:01.095 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/3157562110 expires 2024-02-13T16:44:22.727241+0000 2024-02-12T16:52:01.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:00.301541+0000 mgr.y (mgr.24481) 98 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:01.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:01.090914+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? v1:172.21.15.33:0/3889900439' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-12T16:52:01.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:00.301541+0000 mgr.y (mgr.24481) 98 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:01.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:01.090914+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? v1:172.21.15.33:0/3889900439' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-12T16:52:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:00.301541+0000 mgr.y (mgr.24481) 98 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:01.090914+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? v1:172.21.15.33:0/3889900439' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-12T16:52:01.811 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.33:6801/2545411201 v1:172.21.15.33:6802/2545411201 exists,up 5dd172eb-ee9d-4700-9b80-60f91c5f2cb7', 'osd.1 up in weight 1 up_from 13 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.33:6805/1723231708 v1:172.21.15.33:6806/1723231708 exists,up 71231bc4-1716-431d-bd88-f6a3cad0bd24', 'osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) v1:172.21.15.33:6809/3126580659 v1:172.21.15.33:6810/3126580659 exists,up 8a9c7127-d17c-4c44-b666-4211006de47c', 'osd.3 up in weight 1 up_from 26 up_thru 81 down_at 0 last_clean_interval [0,0) v1:172.21.15.33:6813/3941314957 v1:172.21.15.33:6814/3941314957 exists,up 3cc40671-c8f7-412c-a0f1-987f48a016b9', 'osd.4 up in weight 1 up_from 32 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6800/544126779 v1:172.21.15.94:6801/544126779 exists,up 0498129c-861c-4f7e-b8ba-219949da3130', 'osd.5 up in weight 1 up_from 38 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6804/3229884005 v1:172.21.15.94:6805/3229884005 exists,up 05718a1e-1164-4b54-a493-23c1cd3adb50', 'osd.6 up in weight 1 up_from 45 up_thru 81 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6808/507305558 v1:172.21.15.94:6809/507305558 exists,up dc48640f-8f27-405c-a9a4-cf1e3e6c8821', 'osd.7 up in weight 1 up_from 54 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6812/3401977873 v1:172.21.15.94:6813/3401977873 exists,up 7d55aa3f-06b6-4fc3-97fd-62b535f78b66'] 2024-02-12T16:52:01.811 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon mon.a config show 2024-02-12T16:52:02.091 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:02.599 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:01.447378+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.f scrub starts 2024-02-12T16:52:02.599 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:01.448273+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.f scrub ok 2024-02-12T16:52:02.599 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:01.945355+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.1b scrub starts 2024-02-12T16:52:02.599 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:01.948282+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.1b scrub ok 2024-02-12T16:52:02.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:01.447378+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.f scrub starts 2024-02-12T16:52:02.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:01.448273+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.f scrub ok 2024-02-12T16:52:02.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:01.945355+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.1b scrub starts 2024-02-12T16:52:02.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:01.948282+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.1b scrub ok 2024-02-12T16:52:02.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:52:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:52:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:01.447378+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.f scrub starts 2024-02-12T16:52:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:01.448273+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.f scrub ok 2024-02-12T16:52:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:01.945355+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.1b scrub starts 2024-02-12T16:52:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:01.948282+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.1b scrub ok 2024-02-12T16:52:03.590 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:02.301904+0000 mgr.y (mgr.24481) 99 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:03.590 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:02.438630+0000 osd.2 (osd.2) 39 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:52:03.591 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:02.440624+0000 osd.2 (osd.2) 40 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:52:03.591 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:02.741607+0000 osd.1 (osd.1) 49 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:52:03.591 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:02.742968+0000 osd.1 (osd.1) 50 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:52:03.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:02.301904+0000 mgr.y (mgr.24481) 99 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:03.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:02.438630+0000 osd.2 (osd.2) 39 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:52:03.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:02.440624+0000 osd.2 (osd.2) 40 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:52:03.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:02.741607+0000 osd.1 (osd.1) 49 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:52:03.591 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:02.742968+0000 osd.1 (osd.1) 50 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:52:03.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:52:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:52:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:02.301904+0000 mgr.y (mgr.24481) 99 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:02.438630+0000 osd.2 (osd.2) 39 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:52:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:02.440624+0000 osd.2 (osd.2) 40 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:52:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:02.741607+0000 osd.1 (osd.1) 49 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:52:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:02.742968+0000 osd.1 (osd.1) 50 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:52:03.895 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:52:03.895 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mon.a", 2024-02-12T16:52:03.895 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster": "ceph", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "admin_socket_mode": "", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_client_required": "cephx, none", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_cluster_required": "cephx", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_debug": "true", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_service_required": "cephx", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_supported": "", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio": "true", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-12T16:52:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_poll_ms": "250", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_reap_max": "16", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_async_discard": "false", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_block_size": "4096", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio": "false", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_inflight_ios": "false", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_enable_discard": "false", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_flock_retry": "3", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_inject_crash": "0", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring": "false", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring_hipri": "false", 2024-02-12T16:52:03.897 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_type": "", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_alloc_size": "1048576", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_allocator": "hybrid", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_buffered_io": "true", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_for_zeros": "false", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_compact_log_sync": "false", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_max_log_runway": "4194304", 2024-02-12T16:52:03.898 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_max_prefetch": "1048576", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_min_flush_size": "524288", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_min_log_runway": "1048576", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_replay_recovery": "false", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_sync_write": "false", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_allocator": "hybrid", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-12T16:52:03.899 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_create": "true", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_create": "false", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_path": "", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_size": "0", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_path": "", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_preallocate_file": "false", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_size": "107374182400", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_create": "false", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_path": "", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_size": "100663296", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs": "true", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-12T16:52:03.900 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_autotune": "true", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size": "0", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_type": "2q", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_clone_cow": "true", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-12T16:52:03.901 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_mode": "none", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_csum_type": "crc32c", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_freelist": "false", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_bug21040": "false", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-12T16:52:03.902 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_small_allocations": "0", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_default_buffered_read": "true", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_default_buffered_write": "false", 2024-02-12T16:52:03.903 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-12T16:52:03.904 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mount": "false", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_umount": "false", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_ignore_data_csum": "false", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-12T16:52:03.905 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_op_age": "5.000000", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_alloc_size": "0", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size": "0", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_deferred_txc": "32", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size": "0", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_nid_prealloc": "1024", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-12T16:52:03.906 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_retry_disk_reads": "3", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_cf": "true", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_mem": "512", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-12T16:52:03.907 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_tracing": "false", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-12T16:52:03.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_cluster_require_signatures": "false", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_cluster_require_version": "2", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_require_signatures": "false", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_require_version": "2", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_service_require_signatures": "false", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_service_require_version": "2", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_sign_messages": "true", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "chdir": "", 2024-02-12T16:52:03.909 INFO:teuthology.orchestra.run.smithi033.stdout: "client_acl_type": "", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_asio_thread_count": "2", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_cache_mid": "0.750000", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_cache_size": "16384", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_caps_release_delay": "5", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_check_pool_perm": "true", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_force_sync_read": "false", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_getattr_caps": "false", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_inject_tick_delay": "0", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_die_on_failed_remount": "false", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_dirsize_rbytes": "true", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_force_lazyio": "false", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_fs": "", 2024-02-12T16:52:03.910 INFO:teuthology.orchestra.run.smithi033.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_inject_release_failure": "false", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_max_inline_size": "4096", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mds_namespace": "", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_metadata": "", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_gid": "-1", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_timeout": "300", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_uid": "-1", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mountpoint": "/", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_notify_timeout": "10", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc": "true", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_dirty": "104857600", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_objects": "1000", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_size": "209715200", 2024-02-12T16:52:03.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_target_dirty": "8388608", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_permissions": "true", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_quota": "true", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_quota_df": "true", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_max_bytes": "0", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_max_periods": "4", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_min": "131072", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_reconnect_stale": "false", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_shutdown_timeout": "30", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_snapdir": ".snap", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_tick_interval": "1", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_trace": "", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_try_dentry_invalidate": "false", 2024-02-12T16:52:03.912 INFO:teuthology.orchestra.run.smithi033.stdout: "client_use_faked_inos": "false", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "client_use_random_mds": "false", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog": "false", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog_port": "12201", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_monitors": "default=true", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog": "false", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog_level": "info", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "-", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_network": "", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_network_interface": "", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_isal": "false", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_level": "5", 2024-02-12T16:52:03.913 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_winsize": "-15", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zstd_level": "1", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_osd_obc_lru_size": "10", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location": "", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location_hook": "", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location_hook_timeout": "10", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "daemonize": "false", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_allow_any_pool_priority": "false", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asok": "1/5", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asok_assert_abort": "false", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asserts_on_shutdown": "true", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_auth": "1/5", 2024-02-12T16:52:03.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bdev": "1/3", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bluefs": "1/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bluestore": "1/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_buffer": "0/1", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_cephfs_mirror": "0/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_cephsqlite": "0/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_civetweb": "1/10", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_client": "0/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_compressor": "1/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_context": "0/1", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_crush": "1/1", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_crypto": "1/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_deliberately_leak_memory": "false", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_disable_randomized_ping": "false", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_dpdk": "1/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_eventtrace": "1/5", 2024-02-12T16:52:03.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_filer": "0/1", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_filestore": "1/3", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_finisher": "1/1", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_fuse": "1/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_heartbeat_testing_span": "0", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_heartbeatmap": "1/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_javaclient": "1/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_journal": "1/3", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_journaler": "0/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_kstore": "1/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_leveldb": "4/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_lockdep": "0/1", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds": "1/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_balancer": "1/5", 2024-02-12T16:52:03.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_locker": "1/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_log": "1/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_log_expire": "1/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_migrator": "1/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_memdb": "4/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mgr": "2/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mgrc": "1/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mon": "20/20", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_monc": "0/10", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_ms": "1/1", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_none": "0/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objclass": "0/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objectcacher": "0/5", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objecter": "0/1", 2024-02-12T16:52:03.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_optracker": "0/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_osd": "1/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_paxos": "20/20", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_perfcounter": "1/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_prioritycache": "1/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rados": "0/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd": "0/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_mirror": "0/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_pwl": "0/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_replay": "0/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_refs": "0/0", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_reserver": "1/1", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw": "1/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_sync": "1/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rocksdb": "4/5", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_striper": "0/1", 2024-02-12T16:52:03.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_test": "0/5", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_throttle": "1/1", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_timer": "0/1", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_tp": "0/5", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "device_failure_prediction_mode": "none", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_graylog": "false", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_stderr": "true", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_syslog": "false", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event_tracing": "false", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "fake_statfs_for_testing": "0", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "fatal_signal_handlers": "true", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "filer_max_purge_ops": "10", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "filer_max_truncate_ops": "128", 2024-02-12T16:52:03.919 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_apply_finisher_threads": "1", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_blackhole": "false", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_btrfs_clone_range": "true", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_btrfs_snap": "true", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_caller_concurrency": "10", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_collect_device_partition_information": "true", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_commit_timeout": "600.000000", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_inject_read_err": "false", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_omap_check": "false", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_verify_split": "false", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_dump_file": "", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-12T16:52:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fadvise": "true", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fail_eio": "true", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fd_cache_shards": "16", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fd_cache_size": "128", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fiemap": "false", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fiemap_threshold": "4096", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_inject_stall": "0", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_parallel": "false", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_trailing": "false", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_writeahead": "false", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_kill_at": "0", 2024-02-12T16:52:03.921 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs": "0", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size": "0", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_merge_threshold": "-10", 2024-02-12T16:52:03.922 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_odsync_write": "false", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_backend": "rocksdb", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_backend_path": "", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_thread_timeout": "60", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_threads": "2", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_punch_hole": "false", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-12T16:52:03.923 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_ops": "50", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_seek_data_hole": "false", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_sloppy_crc": "false", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_splice": "false", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_split_multiple": "2", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_split_rand_factor": "20", 2024-02-12T16:52:03.924 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_update_to": "1000", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_enable": "true", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_xfs_extsize": "false", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_zfs_snap": "false", 2024-02-12T16:52:03.925 INFO:teuthology.orchestra.run.smithi033.stdout: "fio_dir": "/tmp/fio", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_allow_other": "true", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_atomic_o_trunc": "true", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_big_writes": "true", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_debug": "false", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_default_permissions": "false", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_disable_pagecache": "false", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_max_write": "0", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_multithreaded": "true", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_require_active_mds": "true", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_set_user_groups": "true", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_move": "true", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_read": "true", 2024-02-12T16:52:03.926 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_write": "true", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_use_invalidate_cb": "true", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "gss_target_name": "ceph", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_file": "", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_inject_failure": "0", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_interval": "5", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "host": "", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-12T16:52:03.927 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "inject_early_sigterm": "false", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_aio": "true", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_align_min_size": "65536", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_block_align": "true", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_block_size": "4096", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_dio": "true", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_discard": "false", 2024-02-12T16:52:03.928 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_force_aio": "false", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_ignore_corruption": "false", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_max_write_bytes": "10485760", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_max_write_entries": "100", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_replay_from": "0", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_write_header_frequency": "0", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_zero_on_create": "false", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_prefetch_periods": "10", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_prezero_periods": "5", 2024-02-12T16:52:03.929 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_write_head_interval": "15", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "key": "", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "keyfile": "", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_backend": "rocksdb", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_default_stripe_size": "65536", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_fsck_on_mount": "false", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_max_bytes": "67108864", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_max_ops": "512", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_nid_prealloc": "1024", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_onode_map_size": "1024", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_sync_submit_transaction": "false", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_sync_transaction": "false", 2024-02-12T16:52:03.930 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_block_size": "65536", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_bloom_size": "0", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_cache_size": "536870912", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_compact_on_mount": "false", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_compression": "false", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_log": "", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_log_to_ceph_log": "true", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_max_open_files": "0", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_paranoid": "false", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "leveldb_write_buffer_size": "33554432", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "librados_thread_count": "2", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "lockdep": "false", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "lockdep_force_backtrace": "false", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "log_coarse_timestamps": "true", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "log_flush_on_exit": "false", 2024-02-12T16:52:03.931 INFO:teuthology.orchestra.run.smithi033.stdout: "log_graylog_host": "127.0.0.1", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_graylog_port": "12201", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_max_new": "1000", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_max_recent": "10000", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_stderr_prefix": "debug ", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_stop_at_utilization": "0.970000", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": "true", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_graylog": "false", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_stderr": "false", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_syslog": "false", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "max_rotating_auth_attempts": "10", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_action_on_write_error": "1", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_alternate_name_max": "8192", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_asio_thread_count": "2", 2024-02-12T16:52:03.932 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_export_pin": "true", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_dirs": "true", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_interval": "5", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_interval": "10", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_max": "-1", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_max_until": "-1", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_merge_size": "50", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_midchunk": "0.300000", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_min_start": "0.200000", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_minchunk": "0.001000", 2024-02-12T16:52:03.933 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_mode": "0", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_need_max": "1.200000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_need_min": "0.800000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_bits": "3", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_size": "10000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_target_decay": "10.000000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_grace": "15.000000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_interval": "4.000000", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-12T16:52:03.934 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_mid": "0.700000", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_release_free_interval": "10", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_reservation": "0.050000", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_interval": "1", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_threshold": "262144", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_prealloc_inos": "1000", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_connect_bootstrapping": "false", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_damage_table_max_entries": "10000", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-12T16:52:03.935 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_auth_pins": "false", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_frag": "false", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_scatterstat": "false", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_subtrees": "false", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_decay_halflife": "5.000000", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_default_dir_hash": "2", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_defer_session_stale": "true", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_deny_all_reconnect": "false", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_keys_per_op": "16384", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_max_commit_size": "10", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_max_entries": "0", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-12T16:52:03.936 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_on_map": "false", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_early_reply": "true", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_enable_op_tracker": "true", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_enforce_unique_name": "true", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_random": "true", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-12T16:52:03.937 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_health_summarize_threshold": "10", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_health_dummy": "false", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_migrator_session_race": "false", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_join_fs": "", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_journal_format": "1", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_create_at": "0", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_export_at": "0", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_import_at": "0", 2024-02-12T16:52:03.938 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_at": "0", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_expire_at": "0", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_replay_at": "0", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_link_at": "0", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_mdstable_at": "0", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_openc_at": "0", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_rename_at": "0", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_events_per_segment": "1024", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_max_events": "-1", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_max_segments": "128", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_pause": "false", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_segment_size": "0", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_warn_factor": "2.000000", 2024-02-12T16:52:03.939 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_caps_per_client": "1048576", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_completed_flushes": "100000", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_completed_requests": "100000", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_export_size": "20971520", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_file_recover": "32", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_files": "64", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_ops": "8192", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_snaps_per_dir": "100", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_xattr_pairs_size": "65536", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_metrics_update_interval": "2", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_min_caps_per_client": "100", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_min_caps_working_set": "10000", 2024-02-12T16:52:03.940 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_numa_node": "-1", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_complaint_time": "30.000000", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_duration": "600", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_size": "20", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_log_threshold": "5", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_ping_grace": "15", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_ping_interval": "5", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_caps": "30000", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_warning_threshold": "262144", 2024-02-12T16:52:03.941 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_replay_interval": "1.000000", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_root_ino_gid": "0", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_root_ino_uid": "0", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-12T16:52:03.942 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_shutdown_check": "0", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_skip_ino": "0", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_max_uid": "4294967294", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_min_uid": "0", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_rstat": "false", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_standby_replay_damaged": "false", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_thrash_exports": "0", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_thrash_fragments": "0", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_tick_interval": "5.000000", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_valgrind_exit": "false", 2024-02-12T16:52:03.943 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_verify_backtrace": "1", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_verify_scatter": "false", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_wipe_sessions": "false", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool_debug": "false", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_device_bytes": "1073741824", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_page_set": "false", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_page_size": "65536", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_bytes": "134217728", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_messages": "512", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-12T16:52:03.944 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_disabled_modules": "", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_max_pg_num_change": "128", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mds_bytes": "134217728", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mds_messages": "128", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mon_bytes": "134217728", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mon_messages": "128", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_osd_bytes": "536870912", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_osd_messages": "8192", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_standby_modules": "true", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_stats_period": "5", 2024-02-12T16:52:03.945 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_stats_threshold": "5", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_tick_period": "2", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_allow_pool_delete": "true", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_allow_pool_size_one": "false", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_auth_validate_all_caps": "true", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_bytes": "104857600", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_directed_command_retry": "2", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-12T16:52:03.946 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_parallel": "3", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_log_interval": "1.000000", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_ping_interval": "10.000000", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_file_level": "debug", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_file": "true", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_stderr": "true", 2024-02-12T16:52:03.947 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_bootstrap": "false", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_start": "false", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_trim": "true", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cpu_threads": "4", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_crush_min_required_version": "hammer", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_daemon_bytes": "419430400", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-12T16:52:03.948 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_avail_crit": "5", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_avail_warn": "5", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_size_warn": "16106127360", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_json": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_transactions": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_extra_checks": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_octopus": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_pacific": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-12T16:52:03.949 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_down_mkfs_grace": "60", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_election_default_strategy": "1", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_election_timeout": "5.000000", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ping_divisor": "2", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_enable_op_tracker": "true", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_fake_pool_delete": "false", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_force_quorum_join": "false", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_globalid_prealloc": "10000", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_detail_to_clog": "true", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_log_update_period": "5", 2024-02-12T16:52:03.950 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_max_detail": "50", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog": "true", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog_interval": "600", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_host": "172.21.15.33:6789/0", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_host_override": "", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_initial_members": "", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease": "5.000000", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-12T16:52:03.951 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_log_max_summary": "50", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_log_epochs": "500", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_osd": "10000", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_pg_per_osd": "10000", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_pool_pg_num": "65536", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_force_trim_to": "0", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_skip_sanity": "false", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_memory_autotune": "true", 2024-02-12T16:52:03.952 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_memory_target": "2147483648", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_beacon_grace": "30", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_digest_period": "5", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_inactive_grace": "60", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_min_osdmap_epochs": "50", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_complaint_time": "30", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_duration": "600", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_size": "20", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_slow_op_size": "20", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_log_threshold": "5", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-12T16:52:03.953 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_in": "false", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_cache_size": "500", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_down_out_interval": "600", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_force_trim_to": "0", 2024-02-12T16:52:03.954 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_down_reporters": "2", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-12T16:52:03.955 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_report_timeout": "900", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_stuck_threshold": "60", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-12T16:52:03.956 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_probe_timeout": "2.000000", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_max_change": "0.050000", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_max_osds": "4", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_interval": "86400", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_max_keys": "100", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_timeout": "300", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_session_timeout": "300", 2024-02-12T16:52:03.957 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_smart_report_timeout": "5", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stat_smooth_intervals": "6", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_pool_min_size": "2", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_pool_size": "4", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_debug": "false", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_provider_kill_at": "0", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_requester_kill_at": "0", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_timeout": "60.000000", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_target_pg_per_osd": "100", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_tick_interval": "5", 2024-02-12T16:52:03.958 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_timecheck_interval": "300.000000", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_older_version_delay": "604800", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_misplaced": "false", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_older_version": "true", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_no_app": "true", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-12T16:52:03.959 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "monmap": "", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_max_op_threads": "5", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_op_threads": "3", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_cm": "false", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_device_name": "", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_dscp": "96", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-12T16:52:03.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_local_gid": "", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_port_num": "1", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_sl": "3", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_support_srq": "true", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_type": "ib", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_reap_threshold": "5", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_before_connect": "false", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_ipv4": "true", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_ipv6": "false", 2024-02-12T16:52:03.961 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_msgr1": "true", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_msgr2": "false", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_port_max": "7300", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_port_min": "6800", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_retry_count": "3", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_retry_delay": "5", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_client": "false", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mds": "false", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mgr": "false", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mon": "false", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_osd": "false", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_client_mode": "crc secure", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_cluster_mode": "crc secure", 2024-02-12T16:52:03.962 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_cluster_type": "", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_connection_idle_timeout": "900", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_connection_ready_timeout": "10", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_crc_data": "true", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_crc_header": "true", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_bad_msg": "false", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_bug": "true", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_old_message": "true", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_skipped_message": "false", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_coremask": "0xF", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hugepages": "", 2024-02-12T16:52:03.963 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_lro": "true", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_memory_channel": "4", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_pmd": "", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_port_id": "0", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dump_on_send": "false", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_initial_backoff": "0.200000", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_max": "1.000000", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-12T16:52:03.964 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_type": "", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_socket_failures": "0", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_learn_addr_from_peer": "true", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_max_accept_failures": "4", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_max_backoff": "15.000000", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_client_mode": "secure crc", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_service_mode": "secure crc", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_pq_min_cost": "65536", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_public_type": "", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_service_mode": "crc secure", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_listen_backlog": "512", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_nodelay": "true", 2024-02-12T16:52:03.965 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_rcvbuf": "0", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_type": "async+posix", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "no_config_file": "false", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_completion_locks_per_session": "32", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inflight_ops": "1024", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_tick_interval": "5.000000", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_timeout": "10.000000", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "objectstore_blackhole": "false", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "openssl_engine_opts": "", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_delay_time": "5.000000", 2024-02-12T16:52:03.966 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_hist_halflife": "1000", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_max_low_ops": "2", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_max_ops": "4", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_slop": "0.020000", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_asio_thread_count": "2", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_async_recovery_min_cost": "100", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_scan_max": "512", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_scan_min": "64", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_degraded": "false", 2024-02-12T16:52:03.967 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_peering": "false", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_unfound": "true", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_beacon_report_interval": "300", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_duration": "30", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_max_block_size": "67108864", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_blkin_trace_all": "false", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_check_for_log_corruption": "false", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-12T16:52:03.968 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_update_on_start": "true", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_message_cap": "256", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_message_size_cap": "524288000", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_op_priority": "63", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_watch_timeout": "30", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_command_max_records": "256", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_command_thread_suicide_timeout": "900", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_command_thread_timeout": "600", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_compact_on_start": "false", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_update_on_start": "true", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_update_weight_set": "true", 2024-02-12T16:52:03.969 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_feed_pullee": "-1", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_misdirected_ops": "false", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_no_acting_change": "false", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_no_purge_strays": "false", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_op_order": "false", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-12T16:52:03.970 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_shutdown": "false", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_snaps": "false", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_keys": "1024", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-12T16:52:03.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_stride": "524288", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_default_notify_timeout": "30", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep": "0.000000", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_discard_disconnected_ops": "true", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_enable_op_tracker": "true", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-12T16:52:03.972 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_info": "true", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_shutdown": "true", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_function_tracing": "false", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_grace": "20", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_interval": "6", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_peers": "10", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_size": "2000", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_stale": "600", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_max_size": "100000", 2024-02-12T16:52:03.973 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_min_size": "1000", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal_size": "5120", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_kill_backfill_at": "0", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_cache_size": "50", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_dedup": "true", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_message_max": "40", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_share_max_epochs": "40", 2024-02-12T16:52:03.974 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_attr_name_len": "100", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_attr_size": "0", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_backfills": "1", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_markdown_count": "5", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_markdown_period": "600", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_name_len": "2048", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_namespace_len": "256", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_size": "134217728", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_blocked_by": "16", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_log_entries": "10000", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pgls": "1024", 2024-02-12T16:52:03.975 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_push_cost": "8388608", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_push_objects": "10", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_scrubs": "1", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_trimming_pgs": "2", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_write_op_reply_len": "64", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_write_size": "90", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2024-02-12T16:52:03.976 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_profile": "high_client_ops", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_lim": "999999", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_res": "1", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-12T16:52:03.977 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_base": "805306368", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_cache_min": "134217728", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target": "4294967296", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target_autotune": "false", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_min_pg_log_entries": "250", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_min_recovery_priority": "0", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_report_interval": "5", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-12T16:52:03.978 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_num_cache_shards": "32", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_num_op_tracker_shard": "32", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_auto_affinity": "true", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_node": "-1", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_prefer_iface": "true", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objecter_finishers": "1", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore": "bluestore", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore_fuse": "false", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore_tracing": "false", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_complaint_time": "30.000000", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_duration": "600", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_size": "20", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_slow_op_size": "20", 2024-02-12T16:52:03.979 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_log_threshold": "5", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards": "0", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards_hdd": "5", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards_ssd": "8", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_pq_min_cost": "65536", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_queue": "wpq", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_queue_cut_off": "high", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_thread_timeout": "15", 2024-02-12T16:52:03.980 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_open_classes_on_start": "true", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_os_flags": "0", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_peering_op_priority": "255", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_delete_cost": "1048576", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_delete_priority": "5", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_trim_max": "10000", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_trim_min": "100", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_stat_report_interval_max": "500", 2024-02-12T16:52:03.981 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flags": "0", 2024-02-12T16:52:03.982 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_min_size": "0", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pg_num": "32", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pgp_num": "0", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_size": "3", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_type": "replicated", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_push_per_object_cost": "1000", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recover_clone_overlap": "true", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-12T16:52:03.983 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_cost": "20971520", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active": "0", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_single_start": "1", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_op_priority": "3", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_priority": "5", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep": "0.000000", 2024-02-12T16:52:03.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_repair_during_recovery": "false", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_requested_scrub_priority": "120", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_auto_repair": "false", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_begin_hour": "0", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_begin_week_day": "0", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_chunk_max": "25", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_chunk_min": "5", 2024-02-12T16:52:03.985 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_cost": "52428800", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_during_recovery": "false", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_end_hour": "0", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_end_week_day": "0", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_invalid_stats": "true", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_max_preemptions": "5", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_priority": "5", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_sleep": "0.000000", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-12T16:52:03.986 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_skip_data_digest": "false", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_smart_report_timeout": "5", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_cost": "1048576", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_priority": "5", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_target_transaction_size": "30", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-12T16:52:03.987 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tracing": "false", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_use_stale_snap": "false", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "osdc_blkin_trace_all": "false", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_kill_at": "0", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_max_join_drift": "10", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_min": "500", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_min_wait": "0.050000", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_propose_interval": "1.000000", 2024-02-12T16:52:03.988 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_max": "500", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_min": "10", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_stash_full_interval": "25", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_trim_max": "500", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_trim_min": "250", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "perf": "true", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "pid_file": "", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-12T16:52:03.989 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "-", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrv": "", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "public_bind_addr": "-", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "public_network": "", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "public_network_interface": "", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "qat_compressor_enabled": "false", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_mon_op_timeout": "0", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_osd_op_timeout": "0", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_tracing": "false", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_atime_update_interval": "60", 2024-02-12T16:52:03.992 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_balance_parent_reads": "false", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_balance_snap_reads": "false", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blkin_trace_all": "false", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache": "true", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_policy": "writearound", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_size": "33554432", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-12T16:52:03.993 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_clone_copy_on_read": "false", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_compression_hint": "none", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_concurrent_management_ops": "10", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_clone_format": "auto", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_data_pool": "", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_features": "61", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_format": "2", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_map_options": "", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_order": "22", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_pool": "rbd", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_stripe_count": "0", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_stripe_unit": "0", 2024-02-12T16:52:03.994 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_enable_alloc_hint": "true", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_io_scheduler": "simple", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-12T16:52:03.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_order": "24", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_pool": "", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_splay_width": "4", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_localize_parent_reads": "false", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_localize_snap_reads": "false", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-12T16:52:03.996 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-12T16:52:03.997 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mtime_update_interval": "60", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_non_blocking_aio": "true", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_op_thread_timeout": "60", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_op_threads": "1", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_parent_cache_enabled": "false", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-12T16:52:03.998 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_plugins": "", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_burst": "0", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_limit": "0", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_burst": "0", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_limit": "0", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-12T16:52:03.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_read_from_replica_policy": "default", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-12T16:52:04.000 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_skip_partial_discard": "true", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_tracing": "false", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_validate_names": "true", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_validate_pool": "true", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "restapi_base_url": "", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "restapi_log_level": "", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_acl_grants_max_num": "100", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_admin_entry": "admin", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_barbican_url": "", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-12T16:52:04.001 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_enabled": "true", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_expiry_interval": "900", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_lru_size": "10000", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_content_length_compat": "false", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_copy_obj_progress": "true", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cors_rules_max_num": "100", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cross_domain_policy": "", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_addr": "", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-12T16:52:04.002 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_password": "", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_username": "", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_require_ssl": "true", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_addr": "", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_auth": "token", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_namespace": "", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_prefix": "", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-12T16:52:04.003 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_token_file": "", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_buffersize": "524288", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_low_speed_time": "300", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_changes_size": "1000", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_num_shards": "128", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_window": "30", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_notify_interval_msec": "200", 2024-02-12T16:52:04.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-12T16:52:04.005 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dns_name": "", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dns_s3website_name": "", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dynamic_resharding": "true", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_gc_threads": "true", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_lc_threads": "true", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_ops_log": "false", 2024-02-12T16:52:04.006 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_quota_threads": "true", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_static_website": "false", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_usage_log": "false", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enforce_swift_acls": "true", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_exit_timeout_secs": "120", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_expose_bucket": "false", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_extended_http_attrs": "", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_fcgi_socket_backlog": "1024", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_frontends": "beast port=7480", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_deferred": "50", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_objs": "32", 2024-02-12T16:52:04.007 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_processor_period": "3600", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_host": "", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_init_timeout": "300", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-12T16:52:04.008 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_domain": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_password": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_password_path": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_project": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_tenant": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_token": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_token_path": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_user": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_api_version": "2", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_domain": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_password": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_project": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_user": "", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-12T16:52:04.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_url": "", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_debug_interval": "-1", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_lock_max_time": "90", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_objs": "32", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_rules": "1000", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_worker": "3", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_thread_delay": "0", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_dnattr": "uid", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_searchfilter": "", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-12T16:52:04.010 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_http_headers": "", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_object_name_utc": "false", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attr_name_len": "0", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attr_size": "0", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_chunk_size": "4194304", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-12T16:52:04.011 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_listing_results": "1000", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_notify_retries": "10", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_put_param_size": "1048576", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_put_size": "5368709120", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_slo_entries": "1000", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_md_log_max_shards": "64", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_mp_lock_max_time": "600", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-12T16:52:04.012 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_max_gc": "300", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_num_async_rados_threads": "32", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_num_control_oids": "8", 2024-02-12T16:52:04.013 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_num_rados_handles": "1", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_numa_node": "-1", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_chunk_size": "100", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_gc_interval": "600", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_thread_timeout": "600", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_tracing": "false", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_token": "", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_url": "", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_verify_ssl": "true", 2024-02-12T16:52:04.014 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_file_path": "", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_rados": "true", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_socket_path": "", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_push_interval": "2.000000", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_port": "", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_print_continue": "true", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-12T16:52:04.015 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_tracing": "false", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_realm": "", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_region": "", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_request_uri": "", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_batch_size": "64", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_max_aio": "128", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_num_logs": "16", 2024-02-12T16:52:04.016 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_thread_interval": "600", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_resolve_cname": "false", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_run_sync_thread": "true", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_scheduler_type": "throttler", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_script_uri": "", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_service_provider_name": "", 2024-02-12T16:52:04.017 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_socket_path": "", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_client_id": "", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_client_secret": "", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_entry": "sts", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_key": "sts", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_min_session_duration": "900", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_token_introspection_url": "", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_account_in_url": "false", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_auth_entry": "auth", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_auth_url": "", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_custom_header": "", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_need_stats": "true", 2024-02-12T16:52:04.018 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_tenant_name": "", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_token_expiration": "86400", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_url": "", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_url_prefix": "swift", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_lease_period": "120", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-12T16:52:04.019 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_thread_pool_size": "512", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_comment": "", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_createby": "", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_encoding": "", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_flag": "false", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_origin": "", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_tracker": "", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_trust_forwarded_https": "false", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-12T16:52:04.020 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_max_shards": "32", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_max_user_shards": "1", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_use_opa_authz": "false", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_max_buckets": "1000", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_unique_email": "true", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_verify_ssl": "true", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zone": "", 2024-02-12T16:52:04.021 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zonegroup": "", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_block_size": "4096", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_size": "536870912", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion": "false", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-12T16:52:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_index_type": "binary_search", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_partition_filters": "false", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_perf": "false", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "run_dir": "/var/run/ceph", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "setgroup": "ceph", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "setuser": "ceph", 2024-02-12T16:52:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: "setuser_match_path": "", 2024-02-12T16:52:04.024 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-12T16:52:04.024 INFO:teuthology.orchestra.run.smithi033.stdout: "thp": "false", 2024-02-12T16:52:04.024 INFO:teuthology.orchestra.run.smithi033.stdout: "threadpool_default_timeout": "60", 2024-02-12T16:52:04.024 INFO:teuthology.orchestra.run.smithi033.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-12T16:52:04.024 INFO:teuthology.orchestra.run.smithi033.stdout: "throttler_perf_counter": "true" 2024-02-12T16:52:04.024 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:52:04.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:03.308631+0000 osd.6 (osd.6) 31 : cluster [DBG] 4.a scrub starts 2024-02-12T16:52:04.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:03.310765+0000 osd.6 (osd.6) 32 : cluster [DBG] 4.a scrub ok 2024-02-12T16:52:04.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:03.387532+0000 mgr.y (mgr.24481) 100 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:04.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:03.308631+0000 osd.6 (osd.6) 31 : cluster [DBG] 4.a scrub starts 2024-02-12T16:52:04.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:03.310765+0000 osd.6 (osd.6) 32 : cluster [DBG] 4.a scrub ok 2024-02-12T16:52:04.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:03.387532+0000 mgr.y (mgr.24481) 100 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:04.711 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-12T16:52:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:03.308631+0000 osd.6 (osd.6) 31 : cluster [DBG] 4.a scrub starts 2024-02-12T16:52:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:03.310765+0000 osd.6 (osd.6) 32 : cluster [DBG] 4.a scrub ok 2024-02-12T16:52:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:03.387532+0000 mgr.y (mgr.24481) 100 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:04.991 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:04.302537+0000 mgr.y (mgr.24481) 101 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-12T16:52:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:04.631322+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:05.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:04.631430+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:05.675 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:04.302537+0000 mgr.y (mgr.24481) 101 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-12T16:52:05.675 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:04.631322+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:05.675 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:04.631430+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:04.302537+0000 mgr.y (mgr.24481) 101 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-12T16:52:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:04.631322+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:04.631430+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:06.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:05.493705+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:05.494763+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.c scrub starts 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:05.495859+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.c scrub ok 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:05.729825+0000 osd.1 (osd.1) 51 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:05.730616+0000 osd.1 (osd.1) 52 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:05.493705+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:05.494763+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.c scrub starts 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:05.495859+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.c scrub ok 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:05.729825+0000 osd.1 (osd.1) 51 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:52:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:05.730616+0000 osd.1 (osd.1) 52 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:52:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:05.493705+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-12T16:52:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:05.494763+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.c scrub starts 2024-02-12T16:52:06.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:05.495859+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.c scrub ok 2024-02-12T16:52:06.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:05.729825+0000 osd.1 (osd.1) 51 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:52:06.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:05.730616+0000 osd.1 (osd.1) 52 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:52:07.108 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a: {} 2024-02-12T16:52:07.117 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b: {} 2024-02-12T16:52:07.126 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c: {} 2024-02-12T16:52:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-12T16:52:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-12T16:52:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:06.302948+0000 mgr.y (mgr.24481) 102 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:06.720597+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.12 scrub starts 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:06.722338+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.12 scrub ok 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:06.904472+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:06.907492+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:07.095638+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? v1:172.21.15.33:0/2300430599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:06.302948+0000 mgr.y (mgr.24481) 102 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:06.720597+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.12 scrub starts 2024-02-12T16:52:07.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:06.722338+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.12 scrub ok 2024-02-12T16:52:07.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:06.904472+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:52:07.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:06.907492+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:52:07.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:07.095638+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? v1:172.21.15.33:0/2300430599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:52:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:06.302948+0000 mgr.y (mgr.24481) 102 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:06.720597+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.12 scrub starts 2024-02-12T16:52:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:06.722338+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.12 scrub ok 2024-02-12T16:52:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:06.904472+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:52:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:06.907492+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:52:07.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:07.095638+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? v1:172.21.15.33:0/2300430599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:52:08.000 INFO:teuthology.run_tasks:Running task full_sequential... 2024-02-12T16:52:08.010 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:52:08.012 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:52:08.012 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-02-12T16:52:08.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-02-12T16:52:08.012 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-12T16:52:08.013 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:52:08.013 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:52:08.013 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-12T16:52:08.013 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.435532 2024-02-12T16:52:08.013 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 4 0.435532006817144 2024-02-12T16:52:08.013 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-12T16:52:08.014 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:08.014 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-12T16:52:08.014 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:52:08.014 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-12T16:52:08.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:52:08.015 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-12T16:52:08.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:52:08.060 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:08.093 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:08.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:08.202 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:08.244 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:08.287 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:08.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:08.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:08.402 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:08.402 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:08.406 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:08.430 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:08.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:08.555 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:08.583 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:08.602 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:08.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:08.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:08.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:07.494720+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.8 scrub starts 2024-02-12T16:52:08.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:07.495625+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.8 scrub ok 2024-02-12T16:52:08.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:07.749604+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:52:08.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:07.752646+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:52:08.742 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:08.750 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:07.494720+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.8 scrub starts 2024-02-12T16:52:08.750 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:07.495625+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.8 scrub ok 2024-02-12T16:52:08.751 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:07.749604+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:52:08.751 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:07.752646+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:52:08.751 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:07.494720+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.8 scrub starts 2024-02-12T16:52:08.751 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:07.495625+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.8 scrub ok 2024-02-12T16:52:08.751 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:07.749604+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:52:08.751 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:07.752646+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:52:08.762 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:08.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:08.866 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:08.896 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:08.970 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:08.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:09.074 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:52:09.102 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:09.200 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:52:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:09.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:52:09.305 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:09.335 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:09.502 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:08.304054+0000 mgr.y (mgr.24481) 103 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:09.502 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:08.565516+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.b scrub starts 2024-02-12T16:52:09.502 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:08.569239+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.b scrub ok 2024-02-12T16:52:09.503 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:08.304054+0000 mgr.y (mgr.24481) 103 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:09.503 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:08.565516+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.b scrub starts 2024-02-12T16:52:09.503 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:08.569239+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.b scrub ok 2024-02-12T16:52:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:08.304054+0000 mgr.y (mgr.24481) 103 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:08.565516+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.b scrub starts 2024-02-12T16:52:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:08.569239+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.b scrub ok 2024-02-12T16:52:10.543 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:10.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:10.217387+0000 osd.7 (osd.7) 47 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:52:10.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:10.220352+0000 osd.7 (osd.7) 48 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:52:10.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:10.217387+0000 osd.7 (osd.7) 47 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:52:10.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:10.220352+0000 osd.7 (osd.7) 48 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:52:10.965 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:10.986 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:10.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:10.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:10.217387+0000 osd.7 (osd.7) 47 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:52:10.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:10.220352+0000 osd.7 (osd.7) 48 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:52:11.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:11.125 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:11.147 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:11.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:11.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:11.236 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:52:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:10] "GET /metrics HTTP/1.1" 200 172155 "" "Prometheus/2.33.4" 2024-02-12T16:52:11.249 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:11.431 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:11.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:11.535 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:11.567 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:11.589 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:11.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:11.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:11.722 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:11.742 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:11.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:11.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:10.304635+0000 mgr.y (mgr.24481) 104 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:11.230244+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1d scrub starts 2024-02-12T16:52:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:11.233132+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1d scrub ok 2024-02-12T16:52:11.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:10.304635+0000 mgr.y (mgr.24481) 104 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:11.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:11.230244+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1d scrub starts 2024-02-12T16:52:11.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:11.233132+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1d scrub ok 2024-02-12T16:52:11.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:11.875 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:10.304635+0000 mgr.y (mgr.24481) 104 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:11.230244+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1d scrub starts 2024-02-12T16:52:11.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:11.233132+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1d scrub ok 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:11.417301+0000 osd.2 (osd.2) 45 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:11.419406+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:11.922000+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:11.925470+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:12.197455+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:12.200348+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:11.417301+0000 osd.2 (osd.2) 45 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:11.419406+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:11.922000+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:52:12.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:11.925470+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:52:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:12.197455+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:52:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:12.200348+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:52:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:52:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:52:12.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:11.417301+0000 osd.2 (osd.2) 45 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:52:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:11.419406+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:52:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:11.922000+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:52:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:11.925470+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:52:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:12.197455+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:52:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:12.200348+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:52:13.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:52:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:52:13.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:12.305031+0000 mgr.y (mgr.24481) 105 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:13.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:12.823304+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.e scrub starts 2024-02-12T16:52:13.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:12.826755+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.e scrub ok 2024-02-12T16:52:13.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:12.305031+0000 mgr.y (mgr.24481) 105 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:13.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:12.823304+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.e scrub starts 2024-02-12T16:52:13.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:12.826755+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.e scrub ok 2024-02-12T16:52:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:12.305031+0000 mgr.y (mgr.24481) 105 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:12.823304+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.e scrub starts 2024-02-12T16:52:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:12.826755+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.e scrub ok 2024-02-12T16:52:13.928 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:13.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:14.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:14.066 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:14.087 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:14.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:14.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:14.221 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:14.242 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:14.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:14.346 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:14.374 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:14.393 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:14.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:14.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:14.529 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:14.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:13.299141+0000 osd.6 (osd.6) 33 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:52:14.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:13.300025+0000 osd.6 (osd.6) 34 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:52:14.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:13.388531+0000 mgr.y (mgr.24481) 106 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:14.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:13.299141+0000 osd.6 (osd.6) 33 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:52:14.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:13.300025+0000 osd.6 (osd.6) 34 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:52:14.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:13.388531+0000 mgr.y (mgr.24481) 106 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:14.767 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:13.299141+0000 osd.6 (osd.6) 33 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:52:14.767 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:13.300025+0000 osd.6 (osd.6) 34 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:52:14.767 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:13.388531+0000 mgr.y (mgr.24481) 106 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:14.982 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:14.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:15.089 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:15.121 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:15.220 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:15.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:15.324 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:15.353 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:15.451 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:15.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:15.545 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_0' created 2024-02-12T16:52:15.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:15.586 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:15.606 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:15.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:15.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:15.741 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:15.761 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:15.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:15.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:14.305702+0000 mgr.y (mgr.24481) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:15.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:14.640504+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.33:0/2384533019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-12T16:52:15.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:14.305702+0000 mgr.y (mgr.24481) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:15.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:14.640504+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.33:0/2384533019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-12T16:52:15.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:15.891 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:14.305702+0000 mgr.y (mgr.24481) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:15.891 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:14.640504+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.33:0/2384533019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-12T16:52:15.893 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:15.913 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:15.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:15.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:15.930 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:52:15.930 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:52:16.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:16.044 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:16.063 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:16.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:16.169 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:16.200 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:16.555 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:52:16.841 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:15.534578+0000 mon.a (mon.0) 734 : audit [INF] from='client.? v1:172.21.15.33:0/2384533019' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-12T16:52:16.841 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:15.534631+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-12T16:52:16.841 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:15.672563+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? v1:172.21.15.33:0/1369091050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:16.841 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:15.772739+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:15.773804+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:15.786328+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:15.786397+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:15.843683+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:15.844483+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:16.140850+0000 mon.a (mon.0) 736 : audit [INF] from='client.? v1:172.21.15.33:0/2204961607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:16.182692+0000 osd.7 (osd.7) 53 : cluster [DBG] 5.1a scrub starts 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:16.185578+0000 osd.7 (osd.7) 54 : cluster [DBG] 5.1a scrub ok 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:15.534578+0000 mon.a (mon.0) 734 : audit [INF] from='client.? v1:172.21.15.33:0/2384533019' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:15.534631+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-12T16:52:16.842 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:15.672563+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? v1:172.21.15.33:0/1369091050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:15.772739+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:15.773804+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:15.786328+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:15.786397+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:15.843683+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:15.844483+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:16.140850+0000 mon.a (mon.0) 736 : audit [INF] from='client.? v1:172.21.15.33:0/2204961607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:16.182692+0000 osd.7 (osd.7) 53 : cluster [DBG] 5.1a scrub starts 2024-02-12T16:52:16.843 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:16.185578+0000 osd.7 (osd.7) 54 : cluster [DBG] 5.1a scrub ok 2024-02-12T16:52:16.851 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:16.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:16.862 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-02-12T16:52:16.955 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:16.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:52:16.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:52:16.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:52:16.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:52:16.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:52:16.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:52:16.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:52:16.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:52:16.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:52:16.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:52:16.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:52:16.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:52:16.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:52:16.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:52:16.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:52:16.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:16.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:15.534578+0000 mon.a (mon.0) 734 : audit [INF] from='client.? v1:172.21.15.33:0/2384533019' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-12T16:52:16.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:15.534631+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-12T16:52:16.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:15.672563+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? v1:172.21.15.33:0/1369091050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:15.772739+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:15.773804+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:15.786328+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:15.786397+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:15.843683+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:15.844483+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:16.140850+0000 mon.a (mon.0) 736 : audit [INF] from='client.? v1:172.21.15.33:0/2204961607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:16.182692+0000 osd.7 (osd.7) 53 : cluster [DBG] 5.1a scrub starts 2024-02-12T16:52:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:16.185578+0000 osd.7 (osd.7) 54 : cluster [DBG] 5.1a scrub ok 2024-02-12T16:52:17.337 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:17.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:17.441 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:17.473 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:17.495 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:17.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:17.597 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:17.628 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:17.647 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:17.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:17.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:16.306340+0000 mgr.y (mgr.24481) 108 : cluster [DBG] pgmap v94: 148 pgs: 16 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:16.533513+0000 mon.a (mon.0) 737 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:16.544102+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.33:0/2204961607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:52:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:16.544166+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-12T16:52:17.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:17.436238+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:52:17.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:16.306340+0000 mgr.y (mgr.24481) 108 : cluster [DBG] pgmap v94: 148 pgs: 16 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:17.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:16.533513+0000 mon.a (mon.0) 737 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:17.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:16.544102+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.33:0/2204961607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:52:17.747 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:16.544166+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-12T16:52:17.747 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:17.436238+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:52:17.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:16.306340+0000 mgr.y (mgr.24481) 108 : cluster [DBG] pgmap v94: 148 pgs: 16 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T16:52:17.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:16.533513+0000 mon.a (mon.0) 737 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:17.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:16.544102+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.33:0/2204961607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:52:17.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:16.544166+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-12T16:52:17.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:17.436238+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:52:17.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:17.767 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:52:17.780 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:18.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:17.548897+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-12T16:52:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:17.548897+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-12T16:52:18.912 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:18.912 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:18.932 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:18.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:18.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:17.548897+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-12T16:52:19.035 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:19.067 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:19.088 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:19.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:19.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:19.222 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:19.242 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:19.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:19.346 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:19.375 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:19.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:19.498 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:19.529 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:19.568 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.4 primary-affinity to 0.435532 (8285432) 2024-02-12T16:52:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:18.237939+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:18.238025+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:18.306978+0000 mgr.y (mgr.24481) 109 : cluster [DBG] pgmap v97: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-12T16:52:19.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:18.553387+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-12T16:52:19.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:18.590390+0000 mon.a (mon.0) 743 : audit [INF] from='client.? v1:172.21.15.33:0/2332751893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.435532006817144}]: dispatch 2024-02-12T16:52:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:18.237939+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:18.238025+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:18.306978+0000 mgr.y (mgr.24481) 109 : cluster [DBG] pgmap v97: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-12T16:52:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:18.553387+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-12T16:52:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:18.590390+0000 mon.a (mon.0) 743 : audit [INF] from='client.? v1:172.21.15.33:0/2332751893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.435532006817144}]: dispatch 2024-02-12T16:52:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:18.237939+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:18.238025+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:19.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:18.306978+0000 mgr.y (mgr.24481) 109 : cluster [DBG] pgmap v97: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-12T16:52:19.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:18.553387+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-12T16:52:19.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:18.590390+0000 mon.a (mon.0) 743 : audit [INF] from='client.? v1:172.21.15.33:0/2332751893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.435532006817144}]: dispatch 2024-02-12T16:52:20.190 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:52:20.478 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:20.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:20.582 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:20.610 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:20.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:19.555451+0000 mon.a (mon.0) 744 : audit [INF] from='client.? v1:172.21.15.33:0/2332751893' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.435532006817144}]': finished 2024-02-12T16:52:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:19.555515+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-12T16:52:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:20.555986+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-12T16:52:20.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:19.555451+0000 mon.a (mon.0) 744 : audit [INF] from='client.? v1:172.21.15.33:0/2332751893' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.435532006817144}]': finished 2024-02-12T16:52:20.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:19.555515+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-12T16:52:20.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:20.555986+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-12T16:52:20.898 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:52:20.958 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:20.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:19.555451+0000 mon.a (mon.0) 744 : audit [INF] from='client.? v1:172.21.15.33:0/2332751893' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.435532006817144}]': finished 2024-02-12T16:52:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:19.555515+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-12T16:52:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:20.555986+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-12T16:52:21.063 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:21.090 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:21.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:52:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:20] "GET /metrics HTTP/1.1" 200 174635 "" "Prometheus/2.33.4" 2024-02-12T16:52:21.854 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:20.307332+0000 mgr.y (mgr.24481) 110 : cluster [DBG] pgmap v100: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:52:21.854 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:20.307332+0000 mgr.y (mgr.24481) 110 : cluster [DBG] pgmap v100: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:52:21.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:20.307332+0000 mgr.y (mgr.24481) 110 : cluster [DBG] pgmap v100: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T16:52:22.394 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:52:22.394 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:52:22.397 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:52:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:52:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:22.307771+0000 mgr.y (mgr.24481) 111 : cluster [DBG] pgmap v102: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:52:23.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:52:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:52:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:22.307771+0000 mgr.y (mgr.24481) 111 : cluster [DBG] pgmap v102: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:52:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:22.307771+0000 mgr.y (mgr.24481) 111 : cluster [DBG] pgmap v102: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:52:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:23.399417+0000 mgr.y (mgr.24481) 112 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:25.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:23.399417+0000 mgr.y (mgr.24481) 112 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:23.399417+0000 mgr.y (mgr.24481) 112 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:25.470 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:52:25.470 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T16:52:25.587 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:52:25.746 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:52:25.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:24.308375+0000 mgr.y (mgr.24481) 113 : cluster [DBG] pgmap v103: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:52:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:24.597466+0000 mon.a (mon.0) 747 : audit [INF] from='client.? v1:172.21.15.33:0/2032902753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:25.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:24.308375+0000 mgr.y (mgr.24481) 113 : cluster [DBG] pgmap v103: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:52:25.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:24.597466+0000 mon.a (mon.0) 747 : audit [INF] from='client.? v1:172.21.15.33:0/2032902753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:24.308375+0000 mgr.y (mgr.24481) 113 : cluster [DBG] pgmap v103: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:52:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:24.597466+0000 mon.a (mon.0) 747 : audit [INF] from='client.? v1:172.21.15.33:0/2032902753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:26.044 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:26.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:26.149 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:26.179 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:26.302 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:26.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:26.405 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:26.434 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:26.938 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:52:26.956 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:52:26.976 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:52:26.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:25.585859+0000 mon.a (mon.0) 748 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:26.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:25.595769+0000 mon.a (mon.0) 749 : audit [INF] from='client.? v1:172.21.15.33:0/2032902753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:52:26.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:25.595833+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-12T16:52:26.991 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:26.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:25.585859+0000 mon.a (mon.0) 748 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:26.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:25.595769+0000 mon.a (mon.0) 749 : audit [INF] from='client.? v1:172.21.15.33:0/2032902753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:52:26.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:25.595833+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-12T16:52:26.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:25.585859+0000 mon.a (mon.0) 748 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:26.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:25.595769+0000 mon.a (mon.0) 749 : audit [INF] from='client.? v1:172.21.15.33:0/2032902753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:52:26.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:25.595833+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-12T16:52:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:26.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:26.998 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:52:27.017 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:52:27.035 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:52:27.052 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:52:27.070 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:52:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:52:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:52:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:52:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:52:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:52:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:52:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:52:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:52:27.097 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:27.129 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:27.150 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:27.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:27.253 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:27.282 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:27.481 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:27.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:27.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:27.619 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:27.640 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:27.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:27.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:27.744 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:27.773 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:26.308691+0000 mgr.y (mgr.24481) 114 : cluster [DBG] pgmap v105: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:52:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:26.919203+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.33:0/2377506543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:28.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:26.308691+0000 mgr.y (mgr.24481) 114 : cluster [DBG] pgmap v105: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:52:28.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:26.919203+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.33:0/2377506543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:28.009 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:26.308691+0000 mgr.y (mgr.24481) 114 : cluster [DBG] pgmap v105: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T16:52:28.010 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:26.919203+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.33:0/2377506543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:28.023 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:28.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:28.063 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:28.063 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:28.128 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:28.181 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:28.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:28.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:28.314 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:28.333 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:28.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:28.436 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:28.464 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:28.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:28.587 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:28.606 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-02-12T16:52:28.624 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:28.439182+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.33:0/580039184' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:52:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:28.439567+0000 mon.a (mon.0) 751 : audit [INF] from='client.34748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:52:28.744 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:52:28.772 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:28.439182+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.33:0/580039184' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:52:28.772 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:28.439567+0000 mon.a (mon.0) 751 : audit [INF] from='client.34748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:52:28.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:28.439182+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.33:0/580039184' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:52:28.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:28.439567+0000 mon.a (mon.0) 751 : audit [INF] from='client.34748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:52:29.948 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:28.309249+0000 mgr.y (mgr.24481) 115 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-12T16:52:29.948 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:28.598225+0000 mon.a (mon.0) 752 : audit [INF] from='client.34748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:52:29.948 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:28.598301+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-12T16:52:29.949 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:28.309249+0000 mgr.y (mgr.24481) 115 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-12T16:52:29.949 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:28.598225+0000 mon.a (mon.0) 752 : audit [INF] from='client.34748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:52:29.949 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:28.598301+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-12T16:52:29.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:28.309249+0000 mgr.y (mgr.24481) 115 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-12T16:52:29.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:28.598225+0000 mon.a (mon.0) 752 : audit [INF] from='client.34748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:52:29.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:28.598301+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-12T16:52:30.029 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:52:30.704 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:30.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:30.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:30.842 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:30.863 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:30.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:30.967 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:30.997 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:31.016 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:31.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:31.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:52:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:30] "GET /metrics HTTP/1.1" 200 174635 "" "Prometheus/2.33.4" 2024-02-12T16:52:31.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:31.149 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:31.168 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:31.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:31.272 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:31.306 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:31.755 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:31.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:31.859 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:31.886 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:30.309775+0000 mgr.y (mgr.24481) 116 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-12T16:52:31.886 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:31.561415+0000 mon.a (mon.0) 754 : audit [INF] from='client.? v1:172.21.15.33:0/4275951014' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:52:31.886 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:30.309775+0000 mgr.y (mgr.24481) 116 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-12T16:52:31.886 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:31.561415+0000 mon.a (mon.0) 754 : audit [INF] from='client.? v1:172.21.15.33:0/4275951014' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:52:31.888 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:31.988 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T16:52:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:30.309775+0000 mgr.y (mgr.24481) 116 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-12T16:52:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:31.561415+0000 mon.a (mon.0) 754 : audit [INF] from='client.? v1:172.21.15.33:0/4275951014' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:52:32.013 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:32.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:32.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:52:32.147 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:32.220 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:52:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:32.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:52:32.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:32.358 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:32.380 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:32.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:32.465 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:52:32.465 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T16:52:32.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T16:52:32.465 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14451.0:64 5.3 5:c609908c:::notify.5:head [watch watch cookie 94295834920176] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:43.215713+0000", 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 169.245428011, 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017499690000000001, 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14451", 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/534103199", 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 64 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.215713+0000", 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.215713+0000", 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.215712+0000", 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6720000000000004e-06 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.215719+0000", 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.288e-05 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.215732+0000", 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6820000000000003e-06 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.215740+0000", 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033506399999999999 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.216075+0000", 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012123100000000001 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.216196+0000", 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041483699999999999 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.216611+0000", 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5289000000000005e-05 2024-02-12T16:52:32.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.216676+0000", 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015308800000000001 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.216829+0000", 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036004599999999999 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.217189+0000", 2024-02-12T16:52:32.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016166999999999999 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.217351+0000", 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2252999999999997e-05 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.217393+0000", 2024-02-12T16:52:32.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9825999999999999e-05 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.217423+0000", 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0136999999999999e-05 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:43.217463+0000", 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017499690000000001 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.472 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.24397.0:2 4.19 e64/59 4:9a6dd8c0:gc::gc.0:head v 64'1, mlcod=0'0)", 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.210318+0000", 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.250823306, 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019964520000000001, 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.210318+0000", 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.210318+0000", 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.210315+0000", 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6515999999999999e-05 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.210331+0000", 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0133000000000001e-05 2024-02-12T16:52:32.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.210362+0000", 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0489000000000001e-05 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.210372+0000", 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7260999999999999e-05 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.210429+0000", 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3292e-05 2024-02-12T16:52:32.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.210453+0000", 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018191279999999999 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.212272+0000", 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2632000000000001e-05 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.212314+0000", 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019964520000000001 2024-02-12T16:52:32.476 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:4 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.216861+0000", 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.244280248, 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053136260000000001, 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 4 2024-02-12T16:52:32.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.216861+0000", 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.216861+0000", 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.216858+0000", 2024-02-12T16:52:32.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2159999999999995e-06 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.216867+0000", 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2408e-05 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.216880+0000", 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4060000000000005e-06 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.216888+0000", 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1555000000000002e-05 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.216960+0000", 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9380000000000003e-06 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-02-12T16:52:32.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.216967+0000", 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039163890000000002 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.220883+0000", 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014082600000000001 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.221024+0000", 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020997799999999999 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.221234+0000", 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00044763700000000002 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.221682+0000", 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022707700000000001 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.221909+0000", 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018921300000000001 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.222098+0000", 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6483000000000003e-05 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.222134+0000", 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0336999999999997e-05 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.222175+0000", 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053136260000000001 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14451.0:218 5.3 5:c609908c:::notify.5:head [watch unwatch cookie 94295834920176] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.223099+0000", 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.238042385, 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00131613, 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.484 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14451", 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/534103199", 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 218 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223099+0000", 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223099+0000", 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223097+0000", 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6389999999999992e-06 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223104+0000", 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1671000000000001e-05 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223116+0000", 2024-02-12T16:52:32.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0129999999999996e-06 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223123+0000", 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5905000000000001e-05 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223179+0000", 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8158999999999994e-05 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223267+0000", 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000172428 2024-02-12T16:52:32.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223439+0000", 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042096800000000003 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.223860+0000", 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031220100000000001 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.224173+0000", 2024-02-12T16:52:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015633500000000001 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.224329+0000", 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3971999999999998e-05 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.224373+0000", 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2104999999999998e-05 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.224415+0000", 2024-02-12T16:52:32.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00131613 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.24397.0:34 4.7 e65/59 4:e07a6548:gc::gc.15:head v 65'4, mlcod=65'2)", 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.264170+0000", 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.19697125499999, 2024-02-12T16:52:32.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042622429999999998, 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.264170+0000", 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.264170+0000", 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.264168+0000", 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9930000000000007e-06 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.264178+0000", 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3575e-05 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.264192+0000", 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.477e-06 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.264198+0000", 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4545000000000001e-05 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.264243+0000", 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3383999999999998e-05 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.264266+0000", 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041277429999999997 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.268394+0000", 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8340999999999998e-05 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.268432+0000", 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042622429999999998 2024-02-12T16:52:32.494 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:60 4.3 4:c623b173:gc::gc.16:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.274971+0000", 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.18617041100001, 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014748389999999999, 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 60 2024-02-12T16:52:32.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.274971+0000", 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.274971+0000", 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.274970+0000", 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0780000000000003e-06 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.274975+0000", 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1189999999999997e-06 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.274983+0000", 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.746e-06 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.274988+0000", 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8067000000000001e-05 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.275016+0000", 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9727000000000001e-05 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.275055+0000", 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5591e-05 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:52:32.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.275151+0000", 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029617199999999998 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.275447+0000", 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000486517 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.275934+0000", 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00045972500000000002 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276393+0000", 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9916e-05 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276423+0000", 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2317e-05 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276446+0000", 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014748389999999999 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.500 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:62 4.16 4:6849fd06:gc::gc.17:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.276651+0000", 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.184490181, 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001646015, 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 62 2024-02-12T16:52:32.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276651+0000", 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276651+0000", 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276649+0000", 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5939999999999994e-06 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276658+0000", 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3148e-05 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276671+0000", 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2540000000000003e-06 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276678+0000", 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6677999999999997e-05 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276715+0000", 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1673000000000003e-05 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276787+0000", 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021028 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.276997+0000", 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00052981600000000001 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.277527+0000", 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00045282900000000002 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.277979+0000", 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023237 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.278212+0000", 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7746e-05 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.278260+0000", 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7521000000000002e-05 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.278297+0000", 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001646015 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.506 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:77 4.3 4:c3a682d4:reshard::reshard.0000000004:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.283741+0000", 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.17740051999999, 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014836529999999999, 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 77 2024-02-12T16:52:32.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.283741+0000", 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.283741+0000", 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.283739+0000", 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.156e-06 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.283746+0000", 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3135999999999999e-05 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.283759+0000", 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5860000000000004e-06 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.283764+0000", 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6427999999999995e-05 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.283831+0000", 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6898999999999997e-05 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.283908+0000", 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000194942 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.284103+0000", 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00045605899999999998 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.284559+0000", 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00054267200000000001 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285101+0000", 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9821999999999999e-05 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285141+0000", 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7370000000000002e-05 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285188+0000", 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5948999999999998e-05 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285224+0000", 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014836529999999999 2024-02-12T16:52:32.512 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:79 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.285433+0000", 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.17570866299999, 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022908009999999999, 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.513 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285433+0000", 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285433+0000", 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285431+0000", 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4620000000000001e-06 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285437+0000", 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6239999999999997e-06 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285447+0000", 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5549999999999999e-06 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285454+0000", 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3306999999999997e-05 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285497+0000", 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7467999999999998e-05 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285574+0000", 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013359099999999999 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.285708+0000", 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041344 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.286121+0000", 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000356755 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.286478+0000", 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001146411 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.287625+0000", 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9960999999999998e-05 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.287675+0000", 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8813999999999997e-05 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.287723+0000", 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022908009999999999 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:82 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.288490+0000", 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.172651234, 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019502160000000001, 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288490+0000", 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288490+0000", 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288489+0000", 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2529999999999998e-06 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288496+0000", 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8509999999999994e-06 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288506+0000", 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0730000000000001e-06 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288512+0000", 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2213999999999999e-05 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288554+0000", 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1329000000000003e-05 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288605+0000", 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117778 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.288723+0000", 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000365204 2024-02-12T16:52:32.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.289088+0000", 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061952199999999995 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.289708+0000", 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061442199999999999 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290322+0000", 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3094000000000002e-05 2024-02-12T16:52:32.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290395+0000", 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4799999999999998e-05 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290440+0000", 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019502160000000001 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.524 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:88 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.290607+0000", 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.170534304, 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001576685, 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 88 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290607+0000", 2024-02-12T16:52:32.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290607+0000", 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290605+0000", 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8499999999999994e-06 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290613+0000", 2024-02-12T16:52:32.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4735999999999999e-05 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290627+0000", 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8059999999999997e-06 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290634+0000", 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4533999999999997e-05 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290679+0000", 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013975000000000001 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.290818+0000", 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020270999999999999 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:52:32.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.291021+0000", 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074547000000000003 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.291767+0000", 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018610799999999999 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.291953+0000", 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013403699999999999 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292087+0000", 2024-02-12T16:52:32.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0544000000000003e-05 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292137+0000", 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6414e-05 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292184+0000", 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001576685 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:91 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.292895+0000", 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.16824603399999, 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003355715, 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 91 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292895+0000", 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292895+0000", 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292894+0000", 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7400000000000004e-06 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292901+0000", 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5577e-05 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292917+0000", 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4309999999999997e-06 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292924+0000", 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7954000000000003e-05 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.292962+0000", 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5606999999999996e-05 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.293038+0000", 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018544899999999999 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.293223+0000", 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036811899999999998 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.293592+0000", 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053358400000000003 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.294125+0000", 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020198600000000001 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296145+0000", 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9293999999999998e-05 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296204+0000", 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6697999999999999e-05 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296251+0000", 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003355715 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.536 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:95 4.16 4:6a988a9b:reshard::reshard.0000000007:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.296558+0000", 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.164583029, 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015394619999999999, 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 95 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296558+0000", 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296558+0000", 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296556+0000", 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6809999999999998e-06 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296564+0000", 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2296999999999999e-05 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296576+0000", 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0330000000000003e-06 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296583+0000", 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2104999999999998e-05 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296626+0000", 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012320899999999999 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296749+0000", 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023964 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.296988+0000", 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041424000000000002 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.297403+0000", 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00037360399999999998 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.297776+0000", 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022615499999999999 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.298002+0000", 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2911000000000002e-05 2024-02-12T16:52:32.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.298055+0000", 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2450999999999997e-05 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.298098+0000", 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015394619999999999 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:132 4.6 4:64760185:::obj_delete_at_hint.0000000010:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.309321+0000", 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.15181980700001, 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014629689999999999, 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 132 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309321+0000", 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309321+0000", 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309320+0000", 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7540000000000006e-06 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309327+0000", 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4158999999999999e-05 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309342+0000", 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7379999999999994e-06 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309349+0000", 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2272999999999999e-05 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309402+0000", 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6390999999999993e-05 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309488+0000", 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000209342 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.309697+0000", 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000725967 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.310423+0000", 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000211781 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.310635+0000", 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2789999999999998e-05 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.310728+0000", 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5055000000000002e-05 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.310763+0000", 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1498999999999999e-05 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.310784+0000", 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014629689999999999 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:156 4.3 4:c57bc89b:::obj_delete_at_hint.0000000013:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.548 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.316962+0000", 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.144178794, 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001432067, 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 156 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.316962+0000", 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.316962+0000", 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.316961+0000", 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5970000000000003e-06 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.316968+0000", 2024-02-12T16:52:32.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5441999999999999e-05 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.316984+0000", 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1089999999999998e-06 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.316992+0000", 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5661999999999998e-05 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.317038+0000", 2024-02-12T16:52:32.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9667999999999996e-05 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.317127+0000", 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000129455 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.317257+0000", 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041670100000000001 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.317673+0000", 2024-02-12T16:52:32.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047534999999999998 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.318149+0000", 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018497500000000001 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.318334+0000", 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3696999999999999e-05 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.318367+0000", 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7180999999999999e-05 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.318395+0000", 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001432067 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:225 4.b 4:d201668e:::obj_delete_at_hint.0000000023:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.346944+0000", 2024-02-12T16:52:32.554 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.11419706000001, 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013579849999999999, 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 225 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.346944+0000", 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.346944+0000", 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.346942+0000", 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0626e-05 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.346952+0000", 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1884e-05 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.346964+0000", 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4620000000000001e-06 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.346970+0000", 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6526000000000001e-05 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.347017+0000", 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9151000000000002e-05 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.347096+0000", 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000158202 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.347254+0000", 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034211199999999999 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.347596+0000", 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6514999999999993e-05 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.347693+0000", 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00056174399999999998 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.348255+0000", 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6461999999999998e-05 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.348281+0000", 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1001000000000001e-05 2024-02-12T16:52:32.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.348302+0000", 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013579849999999999 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:260 4.b 4:d242335b:gc::gc.2:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.360884+0000", 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 166.100257129, 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016735789999999999, 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.560 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 260 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.360884+0000", 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.360884+0000", 2024-02-12T16:52:32.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.360883+0000", 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5699999999999998e-06 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.360889+0000", 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.257e-05 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.360902+0000", 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9850000000000003e-06 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.360908+0000", 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2064000000000002e-05 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.360950+0000", 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7819000000000007e-05 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.361048+0000", 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019523199999999999 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.361243+0000", 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034625600000000003 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.361589+0000", 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031774899999999998 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.361907+0000", 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060383699999999997 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.362511+0000", 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5925e-05 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.362537+0000", 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1084e-05 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.362558+0000", 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016735789999999999 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:463 4.3 4:c0550c60:::obj_delete_at_hint.0000000094:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.512554+0000", 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 165.94858727900001, 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001776875, 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.566 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 463 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512554+0000", 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512554+0000", 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:52:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512553+0000", 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8409999999999996e-06 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512558+0000", 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2990000000000002e-06 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512568+0000", 2024-02-12T16:52:32.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7380000000000002e-06 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512572+0000", 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6071000000000001e-05 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512609+0000", 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7506e-05 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512686+0000", 2024-02-12T16:52:32.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012953199999999999 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.512816+0000", 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026905199999999998 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.513085+0000", 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00039697499999999999 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.513482+0000", 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000799419 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514281+0000", 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8612999999999999e-05 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514310+0000", 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1257e-05 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514331+0000", 2024-02-12T16:52:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001776875 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24397.0:464 4.3 4:c0550c60:::obj_delete_at_hint.0000000094:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:49:46.514649+0000", 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 165.94649186000001, 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014056050000000001, 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24397", 2024-02-12T16:52:32.572 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 464 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514649+0000", 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514649+0000", 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514648+0000", 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5709999999999997e-06 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514654+0000", 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7620000000000001e-06 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514662+0000", 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5079999999999999e-06 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514667+0000", 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9025000000000001e-05 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514696+0000", 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7144000000000001e-05 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514743+0000", 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104172 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.514847+0000", 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026871399999999998 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.515116+0000", 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034987700000000001 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.515466+0000", 2024-02-12T16:52:32.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00046552900000000001 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.515931+0000", 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9018999999999998e-05 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.516000+0000", 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4645000000000003e-05 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:49:46.516055+0000", 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014056050000000001 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24493.0:2 3.1 3:90ab8ff9:::rbd_task:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected e69)", 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:50:02.452986+0000", 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 150.00815565100001, 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001642293, 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24493", 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3575926640", 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 2 2024-02-12T16:52:32.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.452986+0000", 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.452986+0000", 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999967 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.452982+0000", 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0229999999999998e-06 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.452991+0000", 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.363e-06 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.453001+0000", 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6949999999999997e-06 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.453006+0000", 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000126752 2024-02-12T16:52:32.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.453133+0000", 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5209999999999997e-06 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.453141+0000", 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013825969999999999 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.454523+0000", 2024-02-12T16:52:32.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104676 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:50:02.454628+0000", 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001642293 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:52:32.582 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:52:32.582 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:32.612 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:52:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:52:32.614 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:32.637 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:32.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:32.728 DEBUG:teuthology.orchestra.run.smithi033:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-02-12T16:52:32.731 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:52:32.731 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:52:32.740 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:52:32.768 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:32.975 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:hints = 1 2024-02-12T16:52:32.975 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:52:32.975 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object prefix: benchmark_data_smithi033_81060 2024-02-12T16:52:32.975 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:52:32.975 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:52:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:31.983072+0000 mon.a (mon.0) 755 : audit [INF] from='client.? v1:172.21.15.33:0/4275951014' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:52:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:31.983139+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-12T16:52:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:32.310334+0000 mgr.y (mgr.24481) 117 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-02-12T16:52:33.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:31.983072+0000 mon.a (mon.0) 755 : audit [INF] from='client.? v1:172.21.15.33:0/4275951014' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:52:33.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:31.983139+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-12T16:52:33.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:32.310334+0000 mgr.y (mgr.24481) 117 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-02-12T16:52:33.318 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:52:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:33.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:52:33.354 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:52:33.410 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:31.983072+0000 mon.a (mon.0) 755 : audit [INF] from='client.? v1:172.21.15.33:0/4275951014' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:52:33.411 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:31.983139+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-12T16:52:33.411 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:32.310334+0000 mgr.y (mgr.24481) 117 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-02-12T16:52:33.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:33.458 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:33.482 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:33.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:33.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:33.622 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:33.646 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:33.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:33.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:52:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:52:33.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:33.782 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:33.783 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:33.802 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:33.802 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:33.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:33.815 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-12T16:52:33.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:33.939 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:33.963 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:33.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:33.975 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 1 16 2900 2884 180.24 180.25 0.0072662 0.00553179 2024-02-12T16:52:34.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:33.409540+0000 mgr.y (mgr.24481) 118 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:34.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:33.409540+0000 mgr.y (mgr.24481) 118 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:34.067 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:34.099 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:33.409540+0000 mgr.y (mgr.24481) 118 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:34.102 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:34.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:34.229 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:52:34.276 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:34.539 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:52:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:34.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:52:34.547 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:34.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:34.678 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:34.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:34.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:34.807 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:34.842 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:34.867 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:34.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:34.971 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:34.976 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 2 16 5586 5570 173.986 167.875 0.00901486 0.00573402 2024-02-12T16:52:35.022 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:35.152 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:35.155 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:34.311088+0000 mgr.y (mgr.24481) 119 : cluster [DBG] pgmap v111: 148 pgs: 148 active+clean; 65 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.1 MiB/s wr, 131 op/s 2024-02-12T16:52:35.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:34.311088+0000 mgr.y (mgr.24481) 119 : cluster [DBG] pgmap v111: 148 pgs: 148 active+clean; 65 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.1 MiB/s wr, 131 op/s 2024-02-12T16:52:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:35.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:35.257 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:35.297 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:35.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:34.311088+0000 mgr.y (mgr.24481) 119 : cluster [DBG] pgmap v111: 148 pgs: 148 active+clean; 65 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.1 MiB/s wr, 131 op/s 2024-02-12T16:52:35.976 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 3 15 8385 8370 174.319 175 0.00870393 0.00572579 2024-02-12T16:52:36.058 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:36.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:36.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:36.224 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:36.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:36.328 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:36.364 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:36.388 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:36.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:36.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:36.526 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:36.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:36.654 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:36.688 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:36.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:36.816 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:52:36.885 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:36.977 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 4 16 10741 10725 167.489 147.188 0.00647567 0.00595706 2024-02-12T16:52:36.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:35.911408+0000 osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:36.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:35.911471+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:36.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:35.987687+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-12T16:52:36.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:36.311625+0000 mgr.y (mgr.24481) 120 : cluster [DBG] pgmap v113: 148 pgs: 148 active+clean; 65 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 8.4 MiB/s wr, 135 op/s 2024-02-12T16:52:36.996 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:35.911408+0000 osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:36.996 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:35.911471+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:37.168 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:52:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:37.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:52:37.271 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:37.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:35.987687+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-12T16:52:37.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:36.311625+0000 mgr.y (mgr.24481) 120 : cluster [DBG] pgmap v113: 148 pgs: 148 active+clean; 65 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 8.4 MiB/s wr, 135 op/s 2024-02-12T16:52:37.302 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:35.911408+0000 osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:37.302 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:35.911471+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:37.302 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:35.987687+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-12T16:52:37.303 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:36.311625+0000 mgr.y (mgr.24481) 120 : cluster [DBG] pgmap v113: 148 pgs: 148 active+clean; 65 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 8.4 MiB/s wr, 135 op/s 2024-02-12T16:52:37.305 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:37.331 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:37.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:37.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:37.483 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:37.594 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:37.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:37.701 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:37.735 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:37.757 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:37.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:37.862 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:37.917 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:37.979 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 5 15 13203 13188 164.776 153.938 0.00489881 0.00606131 2024-02-12T16:52:38.041 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:52:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:36.277510+0000 osd.2 (osd.2) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:36.277569+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:37.012031+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-12T16:52:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:37.146871+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.33:0/3285934647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:52:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:37.147487+0000 mon.a (mon.0) 759 : audit [INF] from='client.24784 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:52:38.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:36.277510+0000 osd.2 (osd.2) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:38.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:36.277569+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:38.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:37.012031+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-12T16:52:38.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:37.146871+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.33:0/3285934647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:52:38.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:37.147487+0000 mon.a (mon.0) 759 : audit [INF] from='client.24784 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:52:38.381 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:38.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:38.390 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:38.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:38.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:36.277510+0000 osd.2 (osd.2) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:38.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:36.277569+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:38.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:37.012031+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-12T16:52:38.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:37.146871+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.33:0/3285934647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:52:38.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:37.147487+0000 mon.a (mon.0) 759 : audit [INF] from='client.24784 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:52:38.523 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:38.551 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:38.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:38.655 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:38.712 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:38.977 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 6 16 15501 15485 161.24 143.562 0.0030747 0.00619501 2024-02-12T16:52:39.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:38.012283+0000 mon.a (mon.0) 760 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:39.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:38.026137+0000 mon.a (mon.0) 761 : audit [INF] from='client.24784 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:52:39.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:38.026200+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-12T16:52:39.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:38.315512+0000 mgr.y (mgr.24481) 121 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-02-12T16:52:39.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:38.540608+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:52:39.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:38.542713+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:52:39.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:38.012283+0000 mon.a (mon.0) 760 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:39.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:38.026137+0000 mon.a (mon.0) 761 : audit [INF] from='client.24784 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:52:39.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:38.026200+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-12T16:52:39.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:38.315512+0000 mgr.y (mgr.24481) 121 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-02-12T16:52:39.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:38.540608+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:52:39.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:38.542713+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:52:39.286 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:39.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:39.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:39.428 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:38.012283+0000 mon.a (mon.0) 760 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:52:39.428 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:38.026137+0000 mon.a (mon.0) 761 : audit [INF] from='client.24784 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:52:39.429 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:38.026200+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-12T16:52:39.429 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:38.315512+0000 mgr.y (mgr.24481) 121 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-02-12T16:52:39.429 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:38.540608+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:52:39.429 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:38.542713+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:52:39.430 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:39.454 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:39.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:39.557 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:39.589 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:39.611 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:39.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:39.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:39.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:39.774 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:39.814 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:39.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:39.920 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:39.949 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:39.970 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:39.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:39.978 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 7 16 17612 17596 157.038 131.938 0.00891348 0.00635917 2024-02-12T16:52:40.074 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:40.100 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:52:40.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:38.892895+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:52:40.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:38.896863+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:52:40.133 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:38.892895+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:52:40.133 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:38.896863+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:52:40.138 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:40.148 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:52:40.179 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:52:40.216 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:52:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:38.892895+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:52:40.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:38.896863+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:52:40.252 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:52:40.280 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:40.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:40.284 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:52:40.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:52:40.358 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:52:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:52:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:52:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:52:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:52:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:52:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:52:40.370 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:52:40.371 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:52:40.385 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:40.448 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:40.894 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:52:40.894 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:52:40.978 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 8 15 19550 19535 152.556 121.188 0.00632839 0.0065494 2024-02-12T16:52:41.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:52:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:40] "GET /metrics HTTP/1.1" 200 174631 "" "Prometheus/2.33.4" 2024-02-12T16:52:41.117 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:41.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:41.222 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:41.259 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:41.285 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:41.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:41.317 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:39.277215+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.14 scrub starts 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:39.278588+0000 osd.2 (osd.2) 50 : cluster [DBG] 6.14 scrub ok 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.019247+0000 osd.3 (osd.3) 57 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.022587+0000 osd.3 (osd.3) 58 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:40.066948+0000 mon.b (mon.1) 42 : audit [DBG] from='client.? v1:172.21.15.33:0/2882009401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.087337+0000 osd.5 (osd.5) 49 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.090904+0000 osd.5 (osd.5) 50 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.315967+0000 mgr.y (mgr.24481) 122 : cluster [DBG] pgmap v117: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.554765+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:52:41.365 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.557053+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.986086+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:39.277215+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.14 scrub starts 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:39.278588+0000 osd.2 (osd.2) 50 : cluster [DBG] 6.14 scrub ok 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.019247+0000 osd.3 (osd.3) 57 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.022587+0000 osd.3 (osd.3) 58 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:40.066948+0000 mon.b (mon.1) 42 : audit [DBG] from='client.? v1:172.21.15.33:0/2882009401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.087337+0000 osd.5 (osd.5) 49 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.090904+0000 osd.5 (osd.5) 50 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.315967+0000 mgr.y (mgr.24481) 122 : cluster [DBG] pgmap v117: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.554765+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.557053+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:52:41.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.986086+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-12T16:52:41.390 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:41.421 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:39.277215+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.14 scrub starts 2024-02-12T16:52:41.421 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:39.278588+0000 osd.2 (osd.2) 50 : cluster [DBG] 6.14 scrub ok 2024-02-12T16:52:41.421 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.019247+0000 osd.3 (osd.3) 57 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:52:41.421 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.022587+0000 osd.3 (osd.3) 58 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:52:41.421 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:40.066948+0000 mon.b (mon.1) 42 : audit [DBG] from='client.? v1:172.21.15.33:0/2882009401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:41.421 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.087337+0000 osd.5 (osd.5) 49 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:52:41.421 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.090904+0000 osd.5 (osd.5) 50 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:52:41.422 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.315967+0000 mgr.y (mgr.24481) 122 : cluster [DBG] pgmap v117: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-12T16:52:41.422 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.554765+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:52:41.422 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.557053+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:52:41.422 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.986086+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-12T16:52:41.423 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:41.445 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:41.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:41.549 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:41.613 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:41.978 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 9 16 21356 21340 148.14 112.812 0.00205885 0.00668832 2024-02-12T16:52:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.266640+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:52:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:40.271185+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:52:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:41.144810+0000 osd.7 (osd.7) 55 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:52:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:41.149376+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:52:42.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:41.544676+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:52:42.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:41.547312+0000 osd.6 (osd.6) 42 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:52:42.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.266640+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:52:42.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:40.271185+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:52:42.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:41.144810+0000 osd.7 (osd.7) 55 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:52:42.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:41.149376+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:52:42.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:41.544676+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:52:42.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:41.547312+0000 osd.6 (osd.6) 42 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:52:42.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.266640+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:52:42.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:40.271185+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:52:42.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:41.144810+0000 osd.7 (osd.7) 55 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:52:42.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:41.149376+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:52:42.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:41.544676+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:52:42.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:41.547312+0000 osd.6 (osd.6) 42 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:52:42.978 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 10 16 23208 23192 144.9 115.75 0.0128469 0.00689275 2024-02-12T16:52:42.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:52:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:52:43.422 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:41.986234+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:52:43.422 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:41.989910+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:52:43.422 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:42.316396+0000 mgr.y (mgr.24481) 123 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-12T16:52:43.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:52:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:52:43.492 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:41.986234+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:52:43.492 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:41.989910+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:52:43.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:42.316396+0000 mgr.y (mgr.24481) 123 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-12T16:52:43.493 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:41.986234+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:52:43.493 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:41.989910+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:52:43.493 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:42.316396+0000 mgr.y (mgr.24481) 123 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-12T16:52:43.509 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:43.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:43.616 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:43.651 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:43.676 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:43.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:43.780 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:43.815 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:43.840 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:43.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:43.945 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:43.976 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:43.978 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 11 16 25271 25255 143.448 128.938 0.00938175 0.00696448 2024-02-12T16:52:43.997 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:44.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:44.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:52:44.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:44.177 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:44.213 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:44.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:43.057809+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.c scrub starts 2024-02-12T16:52:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:43.063824+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.c scrub ok 2024-02-12T16:52:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:43.083992+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:52:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:43.088997+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:52:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:43.419638+0000 mgr.y (mgr.24481) 124 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:43.573920+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:52:44.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:43.579808+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:52:44.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:43.057809+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.c scrub starts 2024-02-12T16:52:44.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:43.063824+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.c scrub ok 2024-02-12T16:52:44.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:43.083992+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:52:44.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:43.088997+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:52:44.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:43.419638+0000 mgr.y (mgr.24481) 124 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:44.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:43.573920+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:52:44.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:43.579808+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:52:44.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:43.057809+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.c scrub starts 2024-02-12T16:52:44.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:43.063824+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.c scrub ok 2024-02-12T16:52:44.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:43.083992+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:52:44.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:43.088997+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:52:44.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:43.419638+0000 mgr.y (mgr.24481) 124 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:44.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:43.573920+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:52:44.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:43.579808+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:52:44.317 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:44.347 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:44.369 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:44.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:44.474 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:44.507 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:44.531 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:44.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:44.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:44.704 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:44.980 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 12 16 26940 26924 140.182 104.312 0.00749254 0.00712777 2024-02-12T16:52:45.092 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.1 to 1 (10000) 2024-02-12T16:52:45.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.320286+0000 mgr.y (mgr.24481) 125 : cluster [DBG] pgmap v120: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 195 MiB/s wr, 3.12k op/s 2024-02-12T16:52:45.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.600805+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:52:45.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.602650+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:52:45.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.882437+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:52:45.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.888652+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:52:45.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:45.030738+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.33:0/1826527844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:52:45.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:45.031642+0000 mon.a (mon.0) 764 : audit [INF] from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:52:45.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.320286+0000 mgr.y (mgr.24481) 125 : cluster [DBG] pgmap v120: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 195 MiB/s wr, 3.12k op/s 2024-02-12T16:52:45.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.600805+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:52:45.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.602650+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:52:45.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.882437+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:52:45.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.888652+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:52:45.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:45.030738+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.33:0/1826527844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:52:45.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:45.031642+0000 mon.a (mon.0) 764 : audit [INF] from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:52:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.320286+0000 mgr.y (mgr.24481) 125 : cluster [DBG] pgmap v120: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 195 MiB/s wr, 3.12k op/s 2024-02-12T16:52:45.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.600805+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:52:45.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.602650+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:52:45.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.882437+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:52:45.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.888652+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:52:45.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:45.030738+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.33:0/1826527844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:52:45.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:45.031642+0000 mon.a (mon.0) 764 : audit [INF] from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:52:45.967 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:45.970 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T16:52:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:45.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:45.979 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 13 16 28846 28830 138.561 119.125 0.0179482 0.0072116 2024-02-12T16:52:46.021 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:46.030 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:46.076 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:46.106 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:46.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:46.232 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:46.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.277257+0000 osd.2 (osd.2) 53 : cluster [DBG] 4.d scrub starts 2024-02-12T16:52:46.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.280839+0000 osd.2 (osd.2) 54 : cluster [DBG] 4.d scrub ok 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.864865+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:44.868430+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:45.075026+0000 mon.a (mon.0) 765 : audit [INF] from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:45.075085+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:45.832753+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:45.835657+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:45.992568+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.277257+0000 osd.2 (osd.2) 53 : cluster [DBG] 4.d scrub starts 2024-02-12T16:52:46.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.280839+0000 osd.2 (osd.2) 54 : cluster [DBG] 4.d scrub ok 2024-02-12T16:52:46.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.864865+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:52:46.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:44.868430+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:52:46.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:45.075026+0000 mon.a (mon.0) 765 : audit [INF] from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:52:46.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:45.075085+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-12T16:52:46.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:45.832753+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:52:46.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:45.835657+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:52:46.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:45.992568+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-12T16:52:46.292 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:46.471 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:46.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.277257+0000 osd.2 (osd.2) 53 : cluster [DBG] 4.d scrub starts 2024-02-12T16:52:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.280839+0000 osd.2 (osd.2) 54 : cluster [DBG] 4.d scrub ok 2024-02-12T16:52:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.864865+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:52:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:44.868430+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:52:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:45.075026+0000 mon.a (mon.0) 765 : audit [INF] from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:52:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:45.075085+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-12T16:52:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:45.832753+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:52:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:45.835657+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:52:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:45.992568+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-12T16:52:46.581 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:46.633 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:46.717 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:46.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:46.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:46.852 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:46.877 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:46.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:46.979 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 14 16 30383 30367 135.525 96.0625 0.00449063 0.007372 2024-02-12T16:52:46.981 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:47.030 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:47.067 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:47.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:47.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:47.206 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:47.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:45.312891+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:52:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:45.314631+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:52:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:46.320916+0000 mgr.y (mgr.24481) 126 : cluster [DBG] pgmap v123: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-12T16:52:47.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:45.312891+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:52:47.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:45.314631+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:52:47.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:46.320916+0000 mgr.y (mgr.24481) 126 : cluster [DBG] pgmap v123: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-12T16:52:47.337 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:47.371 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:47.397 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:45.312891+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:52:47.397 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:45.314631+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:52:47.397 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:46.320916+0000 mgr.y (mgr.24481) 126 : cluster [DBG] pgmap v123: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-12T16:52:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:47.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:47.431 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:52:47.499 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:47.529 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:47.553 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:47.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:47.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:47.688 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:47.710 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:47.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:47.813 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:47.845 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:47.868 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:47.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:47.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:47.980 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 15 16 32069 32053 133.508 105.375 0.0078427 0.00748369 2024-02-12T16:52:48.007 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:48.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:48.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:52:48.206 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:46.977565+0000 osd.5 (osd.5) 53 : cluster [DBG] 6.d scrub starts 2024-02-12T16:52:48.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:46.978983+0000 osd.5 (osd.5) 54 : cluster [DBG] 6.d scrub ok 2024-02-12T16:52:48.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:47.015661+0000 osd.3 (osd.3) 61 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:52:48.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:47.024535+0000 osd.3 (osd.3) 62 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:52:48.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:47.557411+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.e scrub starts 2024-02-12T16:52:48.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:47.559655+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.e scrub ok 2024-02-12T16:52:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:46.977565+0000 osd.5 (osd.5) 53 : cluster [DBG] 6.d scrub starts 2024-02-12T16:52:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:46.978983+0000 osd.5 (osd.5) 54 : cluster [DBG] 6.d scrub ok 2024-02-12T16:52:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:47.015661+0000 osd.3 (osd.3) 61 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:52:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:47.024535+0000 osd.3 (osd.3) 62 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:52:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:47.557411+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.e scrub starts 2024-02-12T16:52:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:47.559655+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.e scrub ok 2024-02-12T16:52:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:46.977565+0000 osd.5 (osd.5) 53 : cluster [DBG] 6.d scrub starts 2024-02-12T16:52:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:46.978983+0000 osd.5 (osd.5) 54 : cluster [DBG] 6.d scrub ok 2024-02-12T16:52:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:47.015661+0000 osd.3 (osd.3) 61 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:52:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:47.024535+0000 osd.3 (osd.3) 62 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:52:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:47.557411+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.e scrub starts 2024-02-12T16:52:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:47.559655+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.e scrub ok 2024-02-12T16:52:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:48.981 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 16 16 33710 33694 131.573 102.562 0.0137721 0.00759515 2024-02-12T16:52:49.167 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:47.994791+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:47.997700+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.013069+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.029040+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.115036+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.8 scrub starts 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.128697+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.8 scrub ok 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.321789+0000 mgr.y (mgr.24481) 127 : cluster [DBG] pgmap v124: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.586160+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:52:49.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.590057+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:47.994791+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:47.997700+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.013069+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.029040+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.115036+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.8 scrub starts 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.128697+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.8 scrub ok 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.321789+0000 mgr.y (mgr.24481) 127 : cluster [DBG] pgmap v124: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.586160+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:52:49.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.590057+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:52:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:49.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:52:49.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:49.334 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:49.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:49.439 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:49.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:47.994791+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:52:49.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:47.997700+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:52:49.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.013069+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:52:49.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.029040+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:52:49.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.115036+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.8 scrub starts 2024-02-12T16:52:49.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.128697+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.8 scrub ok 2024-02-12T16:52:49.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.321789+0000 mgr.y (mgr.24481) 127 : cluster [DBG] pgmap v124: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-02-12T16:52:49.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.586160+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:52:49.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.590057+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:52:49.512 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:49.736 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:49.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:49.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:52:49.876 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:49.899 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:52:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:49.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:52:49.982 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 17 16 35272 35256 129.573 97.625 0.00899918 0.00771237 2024-02-12T16:52:50.003 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:50.034 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:50.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:50.140 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.401539+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:52:50.140 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:48.411002+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:52:50.140 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:49.579446+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:52:50.140 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:49.581854+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:52:50.163 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:50.232 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:50.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:50.372 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:52:50.373 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:52:50.402 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.401539+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:52:50.403 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:48.411002+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:52:50.403 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:49.579446+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:52:50.403 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:49.581854+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:52:50.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.401539+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:52:50.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:48.411002+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:52:50.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:49.579446+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:52:50.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:49.581854+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:52:50.430 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:52:50.480 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:50.507 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:52:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:50.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:52:50.630 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:50.664 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:50.688 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:50.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:50.793 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:50.831 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:50.856 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:50.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:50.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:52:50.981 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 18 16 36861 36845 127.891 99.3125 0.00529858 0.00781436 2024-02-12T16:52:50.994 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:51.019 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:52:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:51.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:52:51.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:51.131 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:52:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:50] "GET /metrics HTTP/1.1" 200 175191 "" "Prometheus/2.33.4" 2024-02-12T16:52:51.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:49.983426+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.16 scrub starts 2024-02-12T16:52:51.175 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:51.184 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T16:52:51.185 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:49.983426+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.16 scrub starts 2024-02-12T16:52:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:49.986395+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.16 scrub ok 2024-02-12T16:52:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:50.201285+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.33:0/3651894484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:50.201938+0000 mon.a (mon.0) 768 : audit [INF] from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:50.322187+0000 mgr.y (mgr.24481) 128 : cluster [DBG] pgmap v125: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.33k op/s 2024-02-12T16:52:51.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:49.986395+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.16 scrub ok 2024-02-12T16:52:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:50.201285+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.33:0/3651894484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:50.201938+0000 mon.a (mon.0) 768 : audit [INF] from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:50.322187+0000 mgr.y (mgr.24481) 128 : cluster [DBG] pgmap v125: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.33k op/s 2024-02-12T16:52:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:49.983426+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.16 scrub starts 2024-02-12T16:52:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:49.986395+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.16 scrub ok 2024-02-12T16:52:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:50.201285+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.33:0/3651894484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:50.201938+0000 mon.a (mon.0) 768 : audit [INF] from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:52:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:50.322187+0000 mgr.y (mgr.24481) 128 : cluster [DBG] pgmap v125: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.33k op/s 2024-02-12T16:52:51.654 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:51.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:51.682 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:52:51.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:51.823 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:51.872 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:51.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:51.978 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:51.987 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 19 16 38342 38326 126.013 92.5625 0.00728616 0.00792921 2024-02-12T16:52:52.076 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.019782+0000 osd.5 (osd.5) 59 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:52:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.025147+0000 osd.5 (osd.5) 60 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.137136+0000 mon.a (mon.0) 769 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.137173+0000 mon.a (mon.0) 770 : cluster [INF] Cluster is now healthy 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:51.146204+0000 mon.a (mon.0) 771 : audit [INF] from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.146270+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:51.950588+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.019782+0000 osd.5 (osd.5) 59 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.025147+0000 osd.5 (osd.5) 60 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.137136+0000 mon.a (mon.0) 769 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.137173+0000 mon.a (mon.0) 770 : cluster [INF] Cluster is now healthy 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:51.146204+0000 mon.a (mon.0) 771 : audit [INF] from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.146270+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-12T16:52:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:51.950588+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:52:52.444 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:52.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:52.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.019782+0000 osd.5 (osd.5) 59 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:52:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.025147+0000 osd.5 (osd.5) 60 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:52:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.137136+0000 mon.a (mon.0) 769 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:52:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.137173+0000 mon.a (mon.0) 770 : cluster [INF] Cluster is now healthy 2024-02-12T16:52:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:51.146204+0000 mon.a (mon.0) 771 : audit [INF] from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:52:52.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.146270+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-12T16:52:52.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:51.950588+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:52:52.549 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:52:52.581 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:52.602 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:52:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:52.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:52:52.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:52.759 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:52.794 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:52.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:52.901 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:52.946 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:52.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T16:52:52.984229+0000 min lat: 0.00122014 max lat: 0.115951 avg lat: 0.0080581 2024-02-12T16:52:52.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:52:52.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 20 16 39720 39704 124.015 86.125 0.00817389 0.0080581 2024-02-12T16:52:52.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:52:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:52:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:52:53.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:52:53.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.631191+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:52:53.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.634275+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.834393+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.834445+0000 osd.3 (osd.3) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.849078+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:51.863996+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:52.180508+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:52.322519+0000 mgr.y (mgr.24481) 129 : cluster [DBG] pgmap v128: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 809 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:52.362146+0000 osd.2 (osd.2) 59 : cluster [DBG] 5.d scrub starts 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:52.366505+0000 osd.2 (osd.2) 60 : cluster [DBG] 5.d scrub ok 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:52.600752+0000 osd.6 (osd.6) 55 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:52.604102+0000 osd.6 (osd.6) 56 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:52.612937+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:52.617550+0000 mon.c (mon.2) 82 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:52:53.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:52.634587+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.631191+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.634275+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.834393+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.834445+0000 osd.3 (osd.3) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.849078+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:51.863996+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:52.180508+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:52.322519+0000 mgr.y (mgr.24481) 129 : cluster [DBG] pgmap v128: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 809 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-12T16:52:53.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:52.362146+0000 osd.2 (osd.2) 59 : cluster [DBG] 5.d scrub starts 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:52.366505+0000 osd.2 (osd.2) 60 : cluster [DBG] 5.d scrub ok 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:52.600752+0000 osd.6 (osd.6) 55 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:52.604102+0000 osd.6 (osd.6) 56 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:52.612937+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:52.617550+0000 mon.c (mon.2) 82 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:52.634587+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.631191+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.634275+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.834393+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.834445+0000 osd.3 (osd.3) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.849078+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T16:52:53.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:51.863996+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:52.180508+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:52.322519+0000 mgr.y (mgr.24481) 129 : cluster [DBG] pgmap v128: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 809 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:52.362146+0000 osd.2 (osd.2) 59 : cluster [DBG] 5.d scrub starts 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:52.366505+0000 osd.2 (osd.2) 60 : cluster [DBG] 5.d scrub ok 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:52.600752+0000 osd.6 (osd.6) 55 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:52.604102+0000 osd.6 (osd.6) 56 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:52.612937+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:52.617550+0000 mon.c (mon.2) 82 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:52:53.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:52.634587+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:52:53.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 21 16 41321 41305 122.874 100.062 0.0110852 0.00813308 2024-02-12T16:52:54.476 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:52:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:52.982047+0000 osd.5 (osd.5) 61 : cluster [DBG] 5.f scrub starts 2024-02-12T16:52:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:52.982990+0000 osd.5 (osd.5) 62 : cluster [DBG] 5.f scrub ok 2024-02-12T16:52:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:53.317318+0000 osd.2 (osd.2) 61 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:52:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:53.330186+0000 osd.2 (osd.2) 62 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:52:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:53.431030+0000 mgr.y (mgr.24481) 130 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:54.507 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:52.982047+0000 osd.5 (osd.5) 61 : cluster [DBG] 5.f scrub starts 2024-02-12T16:52:54.507 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:52.982990+0000 osd.5 (osd.5) 62 : cluster [DBG] 5.f scrub ok 2024-02-12T16:52:54.507 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:53.317318+0000 osd.2 (osd.2) 61 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:52:54.507 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:53.330186+0000 osd.2 (osd.2) 62 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:52:54.507 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:53.431030+0000 mgr.y (mgr.24481) 130 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:54.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:52.982047+0000 osd.5 (osd.5) 61 : cluster [DBG] 5.f scrub starts 2024-02-12T16:52:54.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:52.982990+0000 osd.5 (osd.5) 62 : cluster [DBG] 5.f scrub ok 2024-02-12T16:52:54.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:53.317318+0000 osd.2 (osd.2) 61 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:52:54.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:53.330186+0000 osd.2 (osd.2) 62 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:52:54.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:53.431030+0000 mgr.y (mgr.24481) 130 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:52:54.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 22 16 42626 42610 120.997 81.5625 0.0156754 0.00825739 2024-02-12T16:52:55.123 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:55.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:55.235 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:52:55.252 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T16:52:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:55.294 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:52:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:55.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:52:55.398 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:52:55.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:53.855097+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.a scrub starts 2024-02-12T16:52:55.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:53.862472+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.a scrub ok 2024-02-12T16:52:55.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:54.324032+0000 mgr.y (mgr.24481) 131 : cluster [DBG] pgmap v129: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-02-12T16:52:55.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:55.083546+0000 mon.a (mon.0) 775 : audit [INF] from='client.? v1:172.21.15.33:0/2213783392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:52:55.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:53.855097+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.a scrub starts 2024-02-12T16:52:55.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:53.862472+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.a scrub ok 2024-02-12T16:52:55.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:54.324032+0000 mgr.y (mgr.24481) 131 : cluster [DBG] pgmap v129: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-02-12T16:52:55.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:55.083546+0000 mon.a (mon.0) 775 : audit [INF] from='client.? v1:172.21.15.33:0/2213783392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:52:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:55.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:53.855097+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.a scrub starts 2024-02-12T16:52:55.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:53.862472+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.a scrub ok 2024-02-12T16:52:55.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:54.324032+0000 mgr.y (mgr.24481) 131 : cluster [DBG] pgmap v129: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-02-12T16:52:55.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:55.083546+0000 mon.a (mon.0) 775 : audit [INF] from='client.? v1:172.21.15.33:0/2213783392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:52:55.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 23 16 44061 44045 119.635 89.6875 0.0145909 0.00835201 2024-02-12T16:52:56.327 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:52:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:56.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:52:56.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:52:56.490 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:56.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:55.096793+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:55.105533+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:55.220234+0000 mon.a (mon.0) 776 : audit [INF] from='client.? v1:172.21.15.33:0/2213783392' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:55.220290+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:55.238595+0000 osd.7 (osd.7) 59 : cluster [DBG] 6.12 scrub starts 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:55.241613+0000 osd.7 (osd.7) 60 : cluster [DBG] 6.12 scrub ok 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:55.996862+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:55.096793+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:55.105533+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:55.220234+0000 mon.a (mon.0) 776 : audit [INF] from='client.? v1:172.21.15.33:0/2213783392' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:52:56.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:55.220290+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-12T16:52:56.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:55.238595+0000 osd.7 (osd.7) 59 : cluster [DBG] 6.12 scrub starts 2024-02-12T16:52:56.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:55.241613+0000 osd.7 (osd.7) 60 : cluster [DBG] 6.12 scrub ok 2024-02-12T16:52:56.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:55.996862+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-12T16:52:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:52:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:56.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:52:56.655 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:52:56.689 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:52:56.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:56.725 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:52:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:55.096793+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:52:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:55.105533+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:52:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:55.220234+0000 mon.a (mon.0) 776 : audit [INF] from='client.? v1:172.21.15.33:0/2213783392' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:52:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:55.220290+0000 mon.a (mon.0) 777 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-12T16:52:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:55.238595+0000 osd.7 (osd.7) 59 : cluster [DBG] 6.12 scrub starts 2024-02-12T16:52:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:55.241613+0000 osd.7 (osd.7) 60 : cluster [DBG] 6.12 scrub ok 2024-02-12T16:52:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:55.996862+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-12T16:52:56.752 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:56.766 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:52:56.793 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:56.799 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:52:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:56.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:56.831 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:52:56.861 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:52:56.900 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:52:56.903 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:56.930 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:52:56.930 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:52:56.930 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:52:56.930 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:52:56.930 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:52:56.930 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:52:56.930 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:52:56.930 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:52:56.954 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:56.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 24 16 45329 45313 117.953 79.25 0.0128568 0.0084717 2024-02-12T16:52:57.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 3 1 2024-02-12T16:52:57.409 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:57.410 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:57.420 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:52:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:57.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:52:57.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:55.961374+0000 osd.3 (osd.3) 67 : cluster [DBG] 3.2 deep-scrub starts 2024-02-12T16:52:57.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:55.967331+0000 osd.3 (osd.3) 68 : cluster [DBG] 3.2 deep-scrub ok 2024-02-12T16:52:57.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:56.326453+0000 mgr.y (mgr.24481) 132 : cluster [DBG] pgmap v132: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 988 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-12T16:52:57.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:52:56.607346+0000 mon.a (mon.0) 779 : audit [DBG] from='client.? v1:172.21.15.33:0/1589992998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:57.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:56.820276+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:52:57.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:56.823541+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:52:57.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:55.961374+0000 osd.3 (osd.3) 67 : cluster [DBG] 3.2 deep-scrub starts 2024-02-12T16:52:57.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:55.967331+0000 osd.3 (osd.3) 68 : cluster [DBG] 3.2 deep-scrub ok 2024-02-12T16:52:57.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:56.326453+0000 mgr.y (mgr.24481) 132 : cluster [DBG] pgmap v132: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 988 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-12T16:52:57.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:52:56.607346+0000 mon.a (mon.0) 779 : audit [DBG] from='client.? v1:172.21.15.33:0/1589992998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:57.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:56.820276+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:52:57.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:56.823541+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:52:57.533 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:52:57.613 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:57.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:55.961374+0000 osd.3 (osd.3) 67 : cluster [DBG] 3.2 deep-scrub starts 2024-02-12T16:52:57.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:55.967331+0000 osd.3 (osd.3) 68 : cluster [DBG] 3.2 deep-scrub ok 2024-02-12T16:52:57.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:56.326453+0000 mgr.y (mgr.24481) 132 : cluster [DBG] pgmap v132: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 988 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-12T16:52:57.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:52:56.607346+0000 mon.a (mon.0) 779 : audit [DBG] from='client.? v1:172.21.15.33:0/1589992998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:52:57.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:56.820276+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:52:57.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:56.823541+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:52:57.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 25 16 46661 46645 116.565 83.25 0.00631142 0.00857299 2024-02-12T16:52:58.594 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:57.982671+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:52:58.594 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:57.985576+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:52:58.595 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.162316+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:52:58.595 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.167535+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:52:58.595 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:57.982671+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:52:58.595 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:57.985576+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:52:58.595 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.162316+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:52:58.595 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.167535+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:52:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:57.982671+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:52:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:57.985576+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:52:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.162316+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:52:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.167535+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:52:58.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 26 16 48002 47986 115.305 83.8125 0.0134412 0.00866687 2024-02-12T16:52:59.392 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:52:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:52:59.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:52:59.400 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.328396+0000 mgr.y (mgr.24481) 133 : cluster [DBG] pgmap v133: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-12T16:52:59.400 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.950414+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:52:59.400 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.964051+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:52:59.400 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:59.180919+0000 osd.7 (osd.7) 63 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:52:59.400 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:59.185003+0000 osd.7 (osd.7) 64 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:52:59.500 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:52:59.556 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:52:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.328396+0000 mgr.y (mgr.24481) 133 : cluster [DBG] pgmap v133: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-12T16:52:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.950414+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:52:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.964051+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:52:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:59.180919+0000 osd.7 (osd.7) 63 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:52:59.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:52:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:59.185003+0000 osd.7 (osd.7) 64 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:52:59.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.328396+0000 mgr.y (mgr.24481) 133 : cluster [DBG] pgmap v133: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-12T16:52:59.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.950414+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:52:59.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.964051+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:52:59.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:59.180919+0000 osd.7 (osd.7) 63 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:52:59.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:52:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:59.185003+0000 osd.7 (osd.7) 64 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:52:59.985 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 27 16 49344 49328 114.141 83.875 0.0103197 0.00874615 2024-02-12T16:53:00.048 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:53:00.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.690359+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:53:00.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.691229+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:53:00.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.814433+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:53:00.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:58.816864+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:53:00.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:59.291751+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:53:00.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:59.296005+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:53:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.690359+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:53:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.691229+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:53:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.814433+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:53:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:58.816864+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:53:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:59.291751+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:53:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:59.296005+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:53:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.690359+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:53:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.691229+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:53:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.814433+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:53:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:58.816864+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:53:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:59.291751+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:53:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:59.296005+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:53:00.986 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 28 16 50643 50627 112.964 81.1875 0.0100927 0.00884705 2024-02-12T16:53:01.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:53:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:00] "GET /metrics HTTP/1.1" 200 175191 "" "Prometheus/2.33.4" 2024-02-12T16:53:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:59.828581+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:53:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:52:59.833476+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:53:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:00.291122+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.e scrub starts 2024-02-12T16:53:01.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:00.294629+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.e scrub ok 2024-02-12T16:53:01.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:00.328815+0000 mgr.y (mgr.24481) 134 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-12T16:53:01.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:59.828581+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:52:59.833476+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:00.291122+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.e scrub starts 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:00.294629+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.e scrub ok 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:00.328815+0000 mgr.y (mgr.24481) 134 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:59.828581+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:52:59.833476+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:00.291122+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.e scrub starts 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:00.294629+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.e scrub ok 2024-02-12T16:53:01.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:00.328815+0000 mgr.y (mgr.24481) 134 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-12T16:53:01.841 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:01.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:01.851 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:01.952 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:01.986 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 29 15 51906 51891 111.793 79 0.00932248 0.00894051 2024-02-12T16:53:02.003 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:02.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:53:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:53:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:01.102334+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:53:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:01.103697+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:53:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:01.284297+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-02-12T16:53:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:01.285950+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-02-12T16:53:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:01.962438+0000 osd.3 (osd.3) 71 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:53:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:01.970724+0000 osd.3 (osd.3) 72 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:53:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:01.102334+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:53:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:01.103697+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:01.284297+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:01.285950+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:01.962438+0000 osd.3 (osd.3) 71 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:01.970724+0000 osd.3 (osd.3) 72 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:01.102334+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:01.103697+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:01.284297+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:01.285950+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:01.962438+0000 osd.3 (osd.3) 71 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:53:02.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:01.970724+0000 osd.3 (osd.3) 72 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:53:02.986 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 30 16 53184 53168 110.727 79.8125 0.00808971 0.00902553 2024-02-12T16:53:03.569 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:03.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:03.605 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:03.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:03.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:02.329218+0000 mgr.y (mgr.24481) 135 : cluster [DBG] pgmap v135: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 855 op/s 2024-02-12T16:53:03.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:02.467447+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:03.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:02.329218+0000 mgr.y (mgr.24481) 135 : cluster [DBG] pgmap v135: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 855 op/s 2024-02-12T16:53:03.729 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:02.467447+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:03.737 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:03.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:53:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:53:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:02.329218+0000 mgr.y (mgr.24481) 135 : cluster [DBG] pgmap v135: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 855 op/s 2024-02-12T16:53:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:02.467447+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:03.986 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 31 16 54422 54406 109.652 77.375 0.0154929 0.00911537 2024-02-12T16:53:04.407 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:04.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:04.513 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:04.569 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:04.608 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:04.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:04.729 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:03.394006+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:03.394070+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:03.441946+0000 mgr.y (mgr.24481) 136 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:03.478959+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-12T16:53:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:03.942051+0000 osd.3 (osd.3) 73 : cluster [DBG] 2.1b scrub starts 2024-02-12T16:53:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:03.947779+0000 osd.3 (osd.3) 74 : cluster [DBG] 2.1b scrub ok 2024-02-12T16:53:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.165367+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.1c scrub starts 2024-02-12T16:53:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.169885+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.1c scrub ok 2024-02-12T16:53:04.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:03.394006+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:04.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:03.394070+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:04.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:03.441946+0000 mgr.y (mgr.24481) 136 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:04.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:03.478959+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-12T16:53:04.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:03.942051+0000 osd.3 (osd.3) 73 : cluster [DBG] 2.1b scrub starts 2024-02-12T16:53:04.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:03.947779+0000 osd.3 (osd.3) 74 : cluster [DBG] 2.1b scrub ok 2024-02-12T16:53:04.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.165367+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.1c scrub starts 2024-02-12T16:53:04.773 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.169885+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.1c scrub ok 2024-02-12T16:53:04.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:03.394006+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:04.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:03.394070+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:04.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:03.441946+0000 mgr.y (mgr.24481) 136 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:04.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:03.478959+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-12T16:53:04.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:03.942051+0000 osd.3 (osd.3) 73 : cluster [DBG] 2.1b scrub starts 2024-02-12T16:53:04.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:03.947779+0000 osd.3 (osd.3) 74 : cluster [DBG] 2.1b scrub ok 2024-02-12T16:53:04.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.165367+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.1c scrub starts 2024-02-12T16:53:04.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.169885+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.1c scrub ok 2024-02-12T16:53:04.786 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:04.988 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 32 15 55551 55536 108.425 70.625 0.00791749 0.00921835 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:8451 7.4 e102/93 7:2f046c57:::benchmark_data_smithi033_81060_object8450:head v 102'495, mlcod=102'493)", 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:36.018108+0000", 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 29.249196949000002, 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030872303, 2024-02-12T16:53:05.274 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.018108+0000", 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.018108+0000", 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.018106+0000", 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8852000000000001e-05 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.018135+0000", 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0579000000000001e-05 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.018155+0000", 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6580000000000008e-06 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.018164+0000", 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5519000000000002e-05 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.018210+0000", 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0699e-05 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.018230+0000", 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030721953 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.048952+0000", 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8481000000000001e-05 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.048981+0000", 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030872303 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:8467 7.0 e102/93 7:0f834ff0:::benchmark_data_smithi033_81060_object8466:head v 102'532, mlcod=101'530)", 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:36.024374+0000", 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 29.242931311, 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024765256999999999, 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024374+0000", 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024374+0000", 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024373+0000", 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5094999999999999e-05 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024438+0000", 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4623e-05 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024452+0000", 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2429999999999999e-06 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024460+0000", 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8792000000000002e-05 2024-02-12T16:53:05.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024519+0000", 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4065000000000001e-05 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024543+0000", 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024563538999999999 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.049106+0000", 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3130999999999998e-05 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.049139+0000", 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024765256999999999 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:8466 7.d e102/93 7:b39d7bf0:::benchmark_data_smithi033_81060_object8465:head v 102'553, mlcod=102'549)", 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:36.024435+0000", 2024-02-12T16:53:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 29.242870207999999, 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024692623, 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024435+0000", 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024435+0000", 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:53:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024434+0000", 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012915100000000001 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024563+0000", 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0100000000000001e-06 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024572+0000", 2024-02-12T16:53:05.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4639999999999997e-06 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024577+0000", 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9295000000000002e-05 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024617+0000", 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0281000000000001e-05 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.024637+0000", 2024-02-12T16:53:05.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024458245999999999 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.049095+0000", 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2558000000000002e-05 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:36.049128+0000", 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024692623 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:15788 7.b 7:d9e8c0f0:::benchmark_data_smithi033_81060_object15787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:39.096974+0000", 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.170331655999998, 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051344805, 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.288 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 15788 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.096974+0000", 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.096974+0000", 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.096972+0000", 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4181000000000001e-05 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.096996+0000", 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.132e-05 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097007+0000", 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5610000000000004e-06 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097014+0000", 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8415999999999999e-05 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097062+0000", 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000121964 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097184+0000", 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000138486 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097323+0000", 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00095304999999999999 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098276+0000", 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013995699999999999 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098416+0000", 2024-02-12T16:53:05.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049831065000000001 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.148247+0000", 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2997999999999997e-05 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.148290+0000", 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.887e-05 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.148318+0000", 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051344805 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:15796 7.b 7:dfc59b0b:::benchmark_data_smithi033_81060_object15795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:39.098972+0000", 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.168332977999999, 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051079531999999997, 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.294 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 15796 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098972+0000", 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098972+0000", 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:53:05.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098970+0000", 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4624999999999999e-05 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098994+0000", 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1532000000000001e-05 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099006+0000", 2024-02-12T16:53:05.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7200000000000006e-06 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099014+0000", 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7688999999999997e-05 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099061+0000", 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00035035300000000003 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099412+0000", 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014778 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099559+0000", 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014316909999999999 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.100991+0000", 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101308 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.101092+0000", 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048877324 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.149970+0000", 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7119000000000001e-05 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150017+0000", 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4996000000000002e-05 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150052+0000", 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051079531999999997 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:15805 7.b 7:d5a74deb:::benchmark_data_smithi033_81060_object15804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:39.103214+0000", 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.164091634999998, 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046956321000000002, 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 15805 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103214+0000", 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103214+0000", 2024-02-12T16:53:05.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103211+0000", 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6619999999999999e-05 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103238+0000", 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6104999999999999e-05 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103254+0000", 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6499999999999996e-06 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103262+0000", 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5193999999999998e-05 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103347+0000", 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012991899999999999 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103477+0000", 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000140488 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103617+0000", 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016354049999999999 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105253+0000", 2024-02-12T16:53:05.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0477999999999999e-05 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105313+0000", 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044772326000000001 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150085+0000", 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4217e-05 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150130+0000", 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0429000000000002e-05 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150170+0000", 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046956321000000002 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:15809 7.b 7:d1293166:::benchmark_data_smithi033_81060_object15808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:39.105586+0000", 2024-02-12T16:53:05.306 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.161719396999999, 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044938673999999998, 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 15809 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105586+0000", 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105586+0000", 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105584+0000", 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4451999999999999e-05 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105608+0000", 2024-02-12T16:53:05.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6249999999999999e-05 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105624+0000", 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2609999999999996e-06 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105632+0000", 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9830000000000002e-05 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105681+0000", 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8424999999999994e-05 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105770+0000", 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7069000000000002e-05 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.105867+0000", 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00082133599999999998 2024-02-12T16:53:05.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.106688+0000", 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026076700000000002 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.106949+0000", 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043487141999999999 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150436+0000", 2024-02-12T16:53:05.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7642999999999998e-05 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150484+0000", 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0723999999999998e-05 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150525+0000", 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044938673999999998 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:15822 7.b 7:d2fc6be1:::benchmark_data_smithi033_81060_object15821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:39.112262+0000", 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.155043299999999, 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038389826000000002, 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 15822 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.313 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112262+0000", 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112262+0000", 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112260+0000", 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5247000000000001e-05 2024-02-12T16:53:05.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112285+0000", 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1409e-05 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112297+0000", 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1900000000000003e-06 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112302+0000", 2024-02-12T16:53:05.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7662e-05 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112339+0000", 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000121369 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112461+0000", 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011309199999999999 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.112574+0000", 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00067219299999999997 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.113246+0000", 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00043019500000000002 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.113676+0000", 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036886685000000002 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150563+0000", 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8093000000000002e-05 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150611+0000", 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0725000000000001e-05 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150652+0000", 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038389826000000002 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.318 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:45894 7.9 e111/93 7:9c60461b:::benchmark_data_smithi033_81060_object45893:head v 111'2799, mlcod=111'2796)", 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:57.399444+0000", 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.8678608260000003, 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018684600999999999, 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399444+0000", 2024-02-12T16:53:05.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399444+0000", 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399444+0000", 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7594e-05 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399471+0000", 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7340000000000008e-06 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399479+0000", 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8119999999999998e-06 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399484+0000", 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003425207 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.402909+0000", 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2175e-05 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.402931+0000", 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015159662000000001 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.418091+0000", 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8114999999999997e-05 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.418129+0000", 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018684600999999999 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:45899 7.0 e111/93 7:0b543cb9:::benchmark_data_smithi033_81060_object45898:head v 111'2900, mlcod=111'2897)", 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:57.399636+0000", 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.8676697070000001, 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024616442999999998, 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.323 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399636+0000", 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399636+0000", 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399635+0000", 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3225999999999997e-05 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399668+0000", 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3409999999999997e-06 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399675+0000", 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1780000000000002e-06 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.399680+0000", 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001838339 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.401519+0000", 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1047e-05 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.401540+0000", 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022678375000000001 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.424218+0000", 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3815000000000001e-05 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.424252+0000", 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024616442999999998 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:45914 7.b 7:dfa81098:::benchmark_data_smithi033_81060_object45913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:53:05.327 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:57.416894+0000", 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.8504118030000001, 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020118945999999999, 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 45914 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.416894+0000", 2024-02-12T16:53:05.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.416894+0000", 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.416891+0000", 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7104000000000003e-05 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.416928+0000", 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9593e-05 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.416947+0000", 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8599999999999998e-06 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.416953+0000", 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001293188 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.418246+0000", 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013738300000000001 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.418384+0000", 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000149839 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.418533+0000", 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056151910000000003 2024-02-12T16:53:05.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.424149+0000", 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1291000000000001e-05 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.424200+0000", 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012721147 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.436921+0000", 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0974999999999999e-05 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.436972+0000", 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0349000000000003e-05 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.437012+0000", 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020118945999999999 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.333 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:45915 7.d e111/93 7:b3b30d90:::benchmark_data_smithi033_81060_object45914:head v 111'2924, mlcod=111'2921)", 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:57.419202+0000", 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.8481034169999999, 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017635629999999999, 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.419202+0000", 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.419202+0000", 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.419201+0000", 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6235000000000001e-05 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.419237+0000", 2024-02-12T16:53:05.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5450000000000003e-06 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.419246+0000", 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2310000000000004e-06 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.419251+0000", 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074260130000000004 2024-02-12T16:53:05.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.426677+0000", 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1983000000000002e-05 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.426709+0000", 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010095824999999999 2024-02-12T16:53:05.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.436805+0000", 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2907999999999997e-05 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:57.436838+0000", 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017635629999999999 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:49556 7.d e111/93 7:b00114e6:::benchmark_data_smithi033_81060_object49555:head v 111'3146, mlcod=111'3143)", 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:00.170833+0000", 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.0964723689999998, 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021263843000000001, 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.170833+0000", 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.170833+0000", 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.170832+0000", 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2525999999999999e-05 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.170874+0000", 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0733999999999999e-05 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.170885+0000", 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7700000000000004e-06 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.170892+0000", 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078320649999999992 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.178724+0000", 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3169999999999999e-05 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.178737+0000", 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013321324000000001 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.192058+0000", 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8529000000000003e-05 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.192097+0000", 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021263843000000001 2024-02-12T16:53:05.342 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:51784 7.b 7:d9946692:::benchmark_data_smithi033_81060_object51783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:01.871101+0000", 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.3962045390000002, 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018375494999999999, 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.343 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 51784 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.871101+0000", 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.871101+0000", 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.871100+0000", 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3802999999999999e-05 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.871124+0000", 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2118e-05 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.871136+0000", 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6500000000000001e-06 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.871141+0000", 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0027704269999999998 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.873912+0000", 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012182 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.874034+0000", 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015517300000000001 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.874189+0000", 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015147674 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.889336+0000", 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7619999999999998e-06 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.889343+0000", 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3708999999999999e-05 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.889377+0000", 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.626e-05 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.889433+0000", 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3132e-05 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.889476+0000", 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018375494999999999 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:53341 7.4 e111/93 7:29052be9:::benchmark_data_smithi033_81060_object53340:head v 111'3313, mlcod=111'3310)", 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:03.100649+0000", 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 2.1666567730000001, 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049599338999999999, 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100649+0000", 2024-02-12T16:53:05.349 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100649+0000", 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100648+0000", 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9190000000000001e-05 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.350 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100667+0000", 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3700000000000003e-06 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100672+0000", 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1353e-05 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100684+0000", 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9512000000000001e-05 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.351 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100713+0000", 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3009e-05 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100726+0000", 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0065784140000000003 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.107305+0000", 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042943331000000001 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150248+0000", 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049599338999999999 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:53345 7.4 e111/93 7:2b03f256:::benchmark_data_smithi033_81060_object53344:head v 111'3314, mlcod=111'3310)", 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:03.108641+0000", 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 2.1586646919999999, 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046152662999999997, 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.108641+0000", 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.108641+0000", 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.108638+0000", 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9371999999999999e-05 2024-02-12T16:53:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.108668+0000", 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041555100999999997 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150223+0000", 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3180000000000007e-06 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150231+0000", 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6519e-05 2024-02-12T16:53:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150278+0000", 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3272e-05 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150301+0000", 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044652429999999998 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.154766+0000", 2024-02-12T16:53:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6962e-05 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.154793+0000", 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046152662999999997 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:53361 7.b 7:dda7b2e2:::benchmark_data_smithi033_81060_object53360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:03.110431+0000", 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 2.1568745009999999, 2024-02-12T16:53:05.357 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043967814000000001, 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 53361 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.110431+0000", 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.110431+0000", 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.110428+0000", 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4862e-05 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.110453+0000", 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039759022999999998 2024-02-12T16:53:05.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150212+0000", 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5778999999999999e-05 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150228+0000", 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8432e-05 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150276+0000", 2024-02-12T16:53:05.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000158216 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150434+0000", 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017224699999999999 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150607+0000", 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036568569999999999 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.154264+0000", 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5579999999999997e-06 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.154270+0000", 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3098000000000001e-05 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.154303+0000", 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7442000000000003e-05 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.362 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.154361+0000", 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7969000000000002e-05 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.154399+0000", 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043967814000000001 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:53369 7.0 e111/93 7:096de9f1:::benchmark_data_smithi033_81060_object53368:head v 111'3350, mlcod=111'3348)", 2024-02-12T16:53:05.363 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:03.115136+0000", 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 2.152169003, 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040148920999999997, 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.115136+0000", 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.115136+0000", 2024-02-12T16:53:05.364 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.115134+0000", 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3118999999999998e-05 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.115177+0000", 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.035052673999999999 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150230+0000", 2024-02-12T16:53:05.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0040999999999999e-05 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150240+0000", 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1779000000000004e-05 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150322+0000", 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1163999999999999e-05 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150343+0000", 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004901084 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.155244+0000", 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1056000000000001e-05 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.155285+0000", 2024-02-12T16:53:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040148920999999997 2024-02-12T16:53:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.368 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:55260 7.b 7:dc4b4e4b:::benchmark_data_smithi033_81060_object55259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:04.725839+0000", 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.54146631099999998, 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021570011, 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 55260 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.369 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.725839+0000", 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.725839+0000", 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.725836+0000", 2024-02-12T16:53:05.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0902999999999999e-05 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.725867+0000", 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7280000000000001e-05 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.725884+0000", 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5930000000000006e-06 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.725893+0000", 2024-02-12T16:53:05.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002564743 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.728458+0000", 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014741799999999999 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.728605+0000", 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000188448 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.728794+0000", 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036795479999999999 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.732473+0000", 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00052564799999999998 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.732999+0000", 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014302604999999999 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.747301+0000", 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6006999999999993e-05 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.747367+0000", 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1641999999999999e-05 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.747409+0000", 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021570011 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.374 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:55266 7.b 7:d596d889:::benchmark_data_smithi033_81060_object55265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:04.741730+0000", 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.52557564499999998, 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.02149993, 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 55266 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.375 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.741730+0000", 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.741730+0000", 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:53:05.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.741728+0000", 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8546000000000001e-05 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.741756+0000", 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0013000000000001e-05 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.741766+0000", 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.759e-06 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:53:05.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.741773+0000", 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056673940000000001 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.747441+0000", 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013841799999999999 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.747579+0000", 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017376199999999999 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:53:05.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.747753+0000", 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044433019999999997 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.752196+0000", 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053343300000000004 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.752729+0000", 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010387323 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:53:05.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.763117+0000", 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.016e-05 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.763177+0000", 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2670999999999998e-05 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:04.763230+0000", 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.02149993 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.380 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:53:05.381 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:53:05.381 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:53:05.558 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.076576+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.085840+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.330683+0000 mgr.y (mgr.24481) 137 : cluster [DBG] pgmap v137: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.61k op/s 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:04.897386+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.33:0/2053132654' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:04.897933+0000 mon.a (mon.0) 782 : audit [INF] from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.915957+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.7 deep-scrub starts 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.923367+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.7 deep-scrub ok 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:05.137932+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.10 scrub starts 2024-02-12T16:53:05.586 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:05.141241+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.10 scrub ok 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.076576+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.085840+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.330683+0000 mgr.y (mgr.24481) 137 : cluster [DBG] pgmap v137: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.61k op/s 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:04.897386+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.33:0/2053132654' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:04.897933+0000 mon.a (mon.0) 782 : audit [INF] from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.915957+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.7 deep-scrub starts 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.923367+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.7 deep-scrub ok 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:05.137932+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.10 scrub starts 2024-02-12T16:53:05.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:05.141241+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.10 scrub ok 2024-02-12T16:53:05.988 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 33 16 56738 56722 107.385 74.125 0.0154676 0.00930685 2024-02-12T16:53:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.076576+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-02-12T16:53:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.085840+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-02-12T16:53:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.330683+0000 mgr.y (mgr.24481) 137 : cluster [DBG] pgmap v137: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.61k op/s 2024-02-12T16:53:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:04.897386+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.33:0/2053132654' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:53:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:04.897933+0000 mon.a (mon.0) 782 : audit [INF] from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:53:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.915957+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.7 deep-scrub starts 2024-02-12T16:53:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.923367+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.7 deep-scrub ok 2024-02-12T16:53:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:05.137932+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.10 scrub starts 2024-02-12T16:53:05.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:05.141241+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.10 scrub ok 2024-02-12T16:53:06.636 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.917189+0000 osd.0 (osd.0) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:06.636 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:04.917231+0000 osd.0 (osd.0) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:06.636 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:05.509542+0000 mon.a (mon.0) 783 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:06.637 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:05.521421+0000 mon.a (mon.0) 784 : audit [INF] from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:53:06.637 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:05.521479+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-12T16:53:06.641 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.917189+0000 osd.0 (osd.0) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:06.641 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:04.917231+0000 osd.0 (osd.0) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:06.641 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:05.509542+0000 mon.a (mon.0) 783 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:06.641 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:05.521421+0000 mon.a (mon.0) 784 : audit [INF] from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:53:06.641 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:05.521479+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-12T16:53:06.656 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:06.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:06.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:53:06.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:06.824 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:06.988 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 34 16 57942 57926 106.439 75.25 0.0131503 0.00939101 2024-02-12T16:53:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.917189+0000 osd.0 (osd.0) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:04.917231+0000 osd.0 (osd.0) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:06.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:05.509542+0000 mon.a (mon.0) 783 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:06.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:05.521421+0000 mon.a (mon.0) 784 : audit [INF] from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:53:06.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:05.521479+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-12T16:53:07.989 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 35 16 59141 59125 105.539 74.9375 0.00708545 0.00946951 2024-02-12T16:53:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:06.331104+0000 mgr.y (mgr.24481) 138 : cluster [DBG] pgmap v139: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 817 op/s 2024-02-12T16:53:08.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:06.331104+0000 mgr.y (mgr.24481) 138 : cluster [DBG] pgmap v139: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 817 op/s 2024-02-12T16:53:08.033 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:06.331104+0000 mgr.y (mgr.24481) 138 : cluster [DBG] pgmap v139: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 817 op/s 2024-02-12T16:53:08.562 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.3 to 1 (10000) 2024-02-12T16:53:08.989 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 36 16 60268 60252 104.564 70.4375 0.0205704 0.00955609 2024-02-12T16:53:08.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:08.435641+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.33:0/1486982765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:53:08.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:08.436232+0000 mon.a (mon.0) 786 : audit [INF] from='client.34796 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:53:09.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:08.435641+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.33:0/1486982765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:53:09.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:08.436232+0000 mon.a (mon.0) 786 : audit [INF] from='client.34796 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:53:09.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:08.435641+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.33:0/1486982765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:53:09.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:08.436232+0000 mon.a (mon.0) 786 : audit [INF] from='client.34796 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:53:09.121 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:09.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:09.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:09.173 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:53:09.209 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:53:09.230 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:09.248 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:53:09.291 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:09.296 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:53:09.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:53:09.360 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:09.369 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:53:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:09.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:09.400 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:53:09.428 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:53:09.455 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:53:09.455 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:53:09.455 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:53:09.455 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:53:09.456 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:53:09.456 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:53:09.456 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:53:09.456 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:53:09.480 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:09.547 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:09.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:08.331954+0000 mgr.y (mgr.24481) 139 : cluster [DBG] pgmap v140: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-12T16:53:09.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:08.550050+0000 mon.a (mon.0) 787 : audit [INF] from='client.34796 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:53:09.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:08.550113+0000 mon.a (mon.0) 788 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-12T16:53:09.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:09.128737+0000 mon.b (mon.1) 45 : audit [DBG] from='client.? v1:172.21.15.33:0/18614382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:09.724 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:08.331954+0000 mgr.y (mgr.24481) 139 : cluster [DBG] pgmap v140: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-12T16:53:09.724 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:08.550050+0000 mon.a (mon.0) 787 : audit [INF] from='client.34796 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:53:09.724 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:08.550113+0000 mon.a (mon.0) 788 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-12T16:53:09.724 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:09.128737+0000 mon.b (mon.1) 45 : audit [DBG] from='client.? v1:172.21.15.33:0/18614382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:09.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:09.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:09.865 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:08.331954+0000 mgr.y (mgr.24481) 139 : cluster [DBG] pgmap v140: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-12T16:53:09.865 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:08.550050+0000 mon.a (mon.0) 787 : audit [INF] from='client.34796 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:53:09.865 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:08.550113+0000 mon.a (mon.0) 788 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-12T16:53:09.865 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:09.128737+0000 mon.b (mon.1) 45 : audit [DBG] from='client.? v1:172.21.15.33:0/18614382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:09.868 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:09.894 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:09.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:09.989 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 37 16 61370 61354 103.6 68.875 0.0101265 0.00964659 2024-02-12T16:53:09.998 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:10.033 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:10.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:10.162 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:10.198 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:10.224 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:10.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:10.328 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:10.401 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:10.448 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:10.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:10.554 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:10.608 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:10.862 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:10.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:10.895 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T16:53:10.987 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:11.004 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 38 16 62562 62546 102.799 74.5 0.0134698 0.00972235 2024-02-12T16:53:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:11.141 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:11.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:11.148 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:53:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:10] "GET /metrics HTTP/1.1" 200 175282 "" "Prometheus/2.33.4" 2024-02-12T16:53:11.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:11.280 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:11.303 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:11.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:11.406 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:11.440 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:11.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:11.569 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:11.628 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:11.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:53:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:10.332505+0000 mgr.y (mgr.24481) 140 : cluster [DBG] pgmap v142: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 747 B/s rd, 53 MiB/s wr, 855 op/s 2024-02-12T16:53:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:11.006332+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-12T16:53:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:10.332505+0000 mgr.y (mgr.24481) 140 : cluster [DBG] pgmap v142: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 747 B/s rd, 53 MiB/s wr, 855 op/s 2024-02-12T16:53:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:11.006332+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-12T16:53:11.788 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:11.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:11.830 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:11.901 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:11.961 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:10.332505+0000 mgr.y (mgr.24481) 140 : cluster [DBG] pgmap v142: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 747 B/s rd, 53 MiB/s wr, 855 op/s 2024-02-12T16:53:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:11.006332+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-12T16:53:12.003 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 39 16 63562 63546 101.766 62.5 0.0131448 0.00981984 2024-02-12T16:53:12.487 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:53:12.632 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:12.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:12.737 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:12.766 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:53:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:53:12.767 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:12.792 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:12.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:12.897 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:12.928 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:12.952 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:12.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:13.005 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T16:53:13.003555+0000 min lat: 0.00122014 max lat: 0.115951 avg lat: 0.00991053 2024-02-12T16:53:13.005 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:53:13.005 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 40 16 64611 64595 100.856 65.5625 0.00681896 0.00991053 2024-02-12T16:53:13.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:13.087 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:13.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:13.215 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:13.274 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:13.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:13.379 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:13.409 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:13.432 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:13.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:13.536 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:13.596 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:13.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:12.332918+0000 mgr.y (mgr.24481) 141 : cluster [DBG] pgmap v144: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 751 B/s rd, 54 MiB/s wr, 859 op/s 2024-02-12T16:53:13.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:53:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:53:13.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:12.332918+0000 mgr.y (mgr.24481) 141 : cluster [DBG] pgmap v144: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 751 B/s rd, 54 MiB/s wr, 859 op/s 2024-02-12T16:53:13.991 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:12.332918+0000 mgr.y (mgr.24481) 141 : cluster [DBG] pgmap v144: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 751 B/s rd, 54 MiB/s wr, 859 op/s 2024-02-12T16:53:14.004 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 41 16 65725 65709 100.095 69.625 0.00874602 0.00998534 2024-02-12T16:53:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:13.453380+0000 mgr.y (mgr.24481) 142 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:15.004 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 42 16 66802 66786 99.3148 67.3125 0.0065968 0.0100634 2024-02-12T16:53:15.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:13.453380+0000 mgr.y (mgr.24481) 142 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:13.453380+0000 mgr.y (mgr.24481) 142 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:15.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:14.333699+0000 mgr.y (mgr.24481) 143 : cluster [DBG] pgmap v145: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T16:53:16.007 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 43 16 67894 67878 98.5883 68.25 0.00803065 0.0101379 2024-02-12T16:53:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:14.333699+0000 mgr.y (mgr.24481) 143 : cluster [DBG] pgmap v145: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T16:53:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:14.333699+0000 mgr.y (mgr.24481) 143 : cluster [DBG] pgmap v145: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T16:53:16.670 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:53:16.670 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:53:16.713 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:16.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:16.816 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:16.852 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:16.876 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:16.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:16.967 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:16.981 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:17.006 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 44 16 68915 68899 97.7983 63.8125 0.0381063 0.0102201 2024-02-12T16:53:17.012 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:17.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:17.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:17.144 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:17.213 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:16.335418+0000 mgr.y (mgr.24481) 144 : cluster [DBG] pgmap v146: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 685 op/s 2024-02-12T16:53:17.410 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:16.335418+0000 mgr.y (mgr.24481) 144 : cluster [DBG] pgmap v146: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 685 op/s 2024-02-12T16:53:17.411 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:16.335418+0000 mgr.y (mgr.24481) 144 : cluster [DBG] pgmap v146: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 685 op/s 2024-02-12T16:53:17.422 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:17.434 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:17.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:17.446 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:53:17.548 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:17.577 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:17.598 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:17.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:17.701 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:17.773 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:17.989 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:17.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:18.006 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 45 16 69890 69874 96.9796 60.9375 0.0166259 0.0103046 2024-02-12T16:53:18.092 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:18.178 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:18.239 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:18.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:18.343 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:18.401 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:18.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:18.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:17.453192+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:18.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:18.447381+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.33:0/2646297308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:53:18.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:18.449244+0000 mon.a (mon.0) 791 : audit [INF] from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:53:18.490 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T16:53:18.507 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:18.539 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:18.565 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:18.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:18.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:18.697 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:17.453192+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:18.698 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:18.447381+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.33:0/2646297308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:53:18.698 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:18.449244+0000 mon.a (mon.0) 791 : audit [INF] from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:53:18.700 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:18.723 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:18.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:18.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:17.453192+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:18.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:18.447381+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.33:0/2646297308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:53:18.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:18.449244+0000 mon.a (mon.0) 791 : audit [INF] from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:53:18.827 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:18.890 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:19.006 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 46 15 70867 70852 96.2005 61.125 0.024334 0.0103904 2024-02-12T16:53:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:19.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:19.158 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:19.227 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:19.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:18.338114+0000 mgr.y (mgr.24481) 145 : cluster [DBG] pgmap v147: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-12T16:53:19.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:18.475745+0000 mon.a (mon.0) 792 : audit [INF] from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:53:19.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:18.475802+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-12T16:53:19.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:18.338114+0000 mgr.y (mgr.24481) 145 : cluster [DBG] pgmap v147: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-12T16:53:19.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:18.475745+0000 mon.a (mon.0) 792 : audit [INF] from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:53:19.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:18.475802+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-12T16:53:19.506 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:19.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:19.536 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T16:53:19.611 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:19.643 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:19.664 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:19.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:18.338114+0000 mgr.y (mgr.24481) 145 : cluster [DBG] pgmap v147: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-12T16:53:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:18.475745+0000 mon.a (mon.0) 792 : audit [INF] from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:53:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:18.475802+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-12T16:53:19.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:19.991 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:19.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:20.006 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 47 16 71860 71844 95.473 62 0.0137927 0.0104683 2024-02-12T16:53:20.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:20.123 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:20.151 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:20.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:20.256 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:20.317 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:20.585 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:20.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:20.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:20.723 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:20.750 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:20.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:20.856 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:20.881 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:53:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:20] "GET /metrics HTTP/1.1" 200 175323 "" "Prometheus/2.33.4" 2024-02-12T16:53:20.889 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:20.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:20.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:21.007 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 48 16 72932 72916 94.88 67 0.0113728 0.0105349 2024-02-12T16:53:21.014 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:21.071 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:21.087 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:21.249 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:21.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:21.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:21.385 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:21.408 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:21.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:21.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:20.338599+0000 mgr.y (mgr.24481) 146 : cluster [DBG] pgmap v149: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-12T16:53:21.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:21.019500+0000 mon.a (mon.0) 794 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-12T16:53:21.510 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:20.338599+0000 mgr.y (mgr.24481) 146 : cluster [DBG] pgmap v149: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-12T16:53:21.510 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:21.019500+0000 mon.a (mon.0) 794 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-12T16:53:21.513 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:21.664 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:21.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:20.338599+0000 mgr.y (mgr.24481) 146 : cluster [DBG] pgmap v149: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-12T16:53:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:21.019500+0000 mon.a (mon.0) 794 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-12T16:53:21.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:21.799 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:21.821 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:21.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:21.914 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:21.919 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:53:21.925 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:21.969 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:53:21.997 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:53:22.008 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 49 16 73850 73834 94.1119 57.375 0.0110363 0.0106206 2024-02-12T16:53:22.014 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:22.053 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:53:22.110 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:53:22.154 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:53:22.186 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:53:22.240 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:53:22.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:53:22.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:53:22.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:53:22.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:53:22.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:53:22.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:53:22.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:53:22.256 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:53:22.522 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:21.853398+0000 mon.b (mon.1) 47 : audit [DBG] from='client.? v1:172.21.15.33:0/128607701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:22.525 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:21.853398+0000 mon.b (mon.1) 47 : audit [DBG] from='client.? v1:172.21.15.33:0/128607701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:22.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:21.853398+0000 mon.b (mon.1) 47 : audit [DBG] from='client.? v1:172.21.15.33:0/128607701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:22.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:53:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:53:23.008 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 50 16 74830 74814 93.4549 61.25 0.0191161 0.0106947 2024-02-12T16:53:23.561 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:53:23.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:53:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:53:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:22.338931+0000 mgr.y (mgr.24481) 147 : cluster [DBG] pgmap v151: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-02-12T16:53:23.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:23.359916+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.33:0/2765849054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:23.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:23.360948+0000 mon.a (mon.0) 795 : audit [INF] from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:23.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:22.338931+0000 mgr.y (mgr.24481) 147 : cluster [DBG] pgmap v151: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-02-12T16:53:23.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:23.359916+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.33:0/2765849054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:23.360948+0000 mon.a (mon.0) 795 : audit [INF] from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:22.338931+0000 mgr.y (mgr.24481) 147 : cluster [DBG] pgmap v151: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-02-12T16:53:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:23.359916+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.33:0/2765849054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:23.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:23.360948+0000 mon.a (mon.0) 795 : audit [INF] from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:24.009 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 51 16 75813 75797 92.8259 61.4375 0.0176916 0.0107675 2024-02-12T16:53:24.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:23.463491+0000 mgr.y (mgr.24481) 148 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:24.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:23.517968+0000 mon.a (mon.0) 796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:24.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:23.527328+0000 mon.a (mon.0) 797 : audit [INF] from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:53:24.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:23.527401+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-12T16:53:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:23.463491+0000 mgr.y (mgr.24481) 148 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:23.517968+0000 mon.a (mon.0) 796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:23.527328+0000 mon.a (mon.0) 797 : audit [INF] from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:53:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:23.527401+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-12T16:53:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:23.463491+0000 mgr.y (mgr.24481) 148 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:23.517968+0000 mon.a (mon.0) 796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:23.527328+0000 mon.a (mon.0) 797 : audit [INF] from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:53:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:23.527401+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-12T16:53:25.011 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 52 16 76814 76798 92.2419 62.5625 0.0187298 0.0108351 2024-02-12T16:53:25.215 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:25.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:25.326 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:25.357 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:25.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:25.449 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:53:25.486 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:25.518 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:25.521 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:53:25.522 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T16:53:25.522 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T16:53:25.522 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T16:53:25.522 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:53:25.540 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:25.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:25.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:25.673 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:24.340675+0000 mgr.y (mgr.24481) 149 : cluster [DBG] pgmap v153: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-02-12T16:53:25.676 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:25.699 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:25.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:25.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:24.340675+0000 mgr.y (mgr.24481) 149 : cluster [DBG] pgmap v153: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-02-12T16:53:25.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:24.340675+0000 mgr.y (mgr.24481) 149 : cluster [DBG] pgmap v153: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-02-12T16:53:25.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:25.878 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:26.011 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 53 16 77791 77775 91.6539 61.0625 0.0136132 0.0109055 2024-02-12T16:53:26.645 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:26.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:26.674 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:53:26.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:27.011 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 54 16 78775 78759 91.0948 61.5 0.0135089 0.010972 2024-02-12T16:53:27.080 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:53:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:25.799647+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:25.799703+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:26.020997+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-12T16:53:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:26.341031+0000 mgr.y (mgr.24481) 150 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-02-12T16:53:27.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:25.799647+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:27.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:25.799703+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:27.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:26.020997+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-12T16:53:27.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:26.341031+0000 mgr.y (mgr.24481) 150 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-02-12T16:53:27.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:25.799647+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:27.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:25.799703+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:27.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:26.020997+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-12T16:53:27.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:26.341031+0000 mgr.y (mgr.24481) 150 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-02-12T16:53:28.015 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 55 16 79674 79658 90.4579 56.1875 0.0182295 0.0110508 2024-02-12T16:53:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:26.481082+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:26.481120+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:27.073424+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-12T16:53:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:26.481082+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:28.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:26.481120+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:28.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:27.073424+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-12T16:53:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:26.481082+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:26.481120+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:27.073424+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-12T16:53:29.014 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 56 16 80669 80653 89.9523 62.1875 0.0133062 0.0111127 2024-02-12T16:53:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:28.341871+0000 mgr.y (mgr.24481) 151 : cluster [DBG] pgmap v157: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-12T16:53:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:28.341871+0000 mgr.y (mgr.24481) 151 : cluster [DBG] pgmap v157: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-12T16:53:29.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:28.341871+0000 mgr.y (mgr.24481) 151 : cluster [DBG] pgmap v157: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-12T16:53:30.014 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 57 16 81673 81657 89.4753 62.75 0.0222534 0.0111716 2024-02-12T16:53:31.015 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 58 16 82698 82682 89.0356 64.0625 0.0191767 0.0112263 2024-02-12T16:53:31.090 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:31.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:31.136 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:31.196 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:31.256 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:53:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:30] "GET /metrics HTTP/1.1" 200 175323 "" "Prometheus/2.33.4" 2024-02-12T16:53:31.259 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:31.428 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:31.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:31.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:30.344494+0000 mgr.y (mgr.24481) 152 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 751 B/s rd, 45 MiB/s wr, 713 op/s 2024-02-12T16:53:31.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:31.022479+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-12T16:53:31.515 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:30.344494+0000 mgr.y (mgr.24481) 152 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 751 B/s rd, 45 MiB/s wr, 713 op/s 2024-02-12T16:53:31.515 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:31.022479+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-12T16:53:31.541 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:31.582 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:31.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:31.708 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:30.344494+0000 mgr.y (mgr.24481) 152 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 751 B/s rd, 45 MiB/s wr, 713 op/s 2024-02-12T16:53:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:31.022479+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-12T16:53:31.770 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:32.015 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 59 16 83552 83536 88.4314 53.375 0.0164829 0.0113037 2024-02-12T16:53:32.357 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:32.362 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:32.245108+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.33:0/1690303478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:53:32.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:32.245456+0000 mon.a (mon.0) 802 : audit [INF] from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:53:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T16:53:32.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:53:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:53:32.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:32.245108+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.33:0/1690303478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:53:32.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:32.245456+0000 mon.a (mon.0) 802 : audit [INF] from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:53:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:32.245108+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.33:0/1690303478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:53:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:32.245456+0000 mon.a (mon.0) 802 : audit [INF] from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:53:33.016 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T16:53:33.015187+0000 min lat: 0.00122014 max lat: 0.115951 avg lat: 0.0113713 2024-02-12T16:53:33.016 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:53:33.016 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 60 16 84457 84441 87.8998 56.5625 0.0155327 0.0113713 2024-02-12T16:53:33.299 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:33.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:33.405 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:33.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:32.344904+0000 mgr.y (mgr.24481) 153 : cluster [DBG] pgmap v160: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 809 B/s rd, 48 MiB/s wr, 769 op/s 2024-02-12T16:53:33.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:32.413714+0000 mon.a (mon.0) 803 : audit [INF] from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:53:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:32.413778+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-12T16:53:33.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:53:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:53:33.760 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:33.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:33.765 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:32.344904+0000 mgr.y (mgr.24481) 153 : cluster [DBG] pgmap v160: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 809 B/s rd, 48 MiB/s wr, 769 op/s 2024-02-12T16:53:33.765 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:32.413714+0000 mon.a (mon.0) 803 : audit [INF] from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:53:33.765 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:32.413778+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-12T16:53:33.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:32.344904+0000 mgr.y (mgr.24481) 153 : cluster [DBG] pgmap v160: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 809 B/s rd, 48 MiB/s wr, 769 op/s 2024-02-12T16:53:33.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:32.413714+0000 mon.a (mon.0) 803 : audit [INF] from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:53:33.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:32.413778+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-12T16:53:33.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:33.894 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:33.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:34.017 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 61 16 85320 85304 87.3418 53.9375 0.014701 0.0114439 2024-02-12T16:53:34.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:34.052 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:34.073 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:34.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:34.180 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:34.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:34.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:34.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:53:34.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:53:34.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:53:34.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:53:34.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:53:34.335 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:34.363 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:34.387 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:34.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:34.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:34.526 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:34.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:33.161089+0000 osd.4 (osd.4) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:34.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:33.161129+0000 osd.4 (osd.4) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:34.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:33.443704+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-12T16:53:34.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:33.475541+0000 mgr.y (mgr.24481) 154 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:34.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:33.929479+0000 mon.b (mon.1) 49 : audit [DBG] from='client.? v1:172.21.15.33:0/1373634618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:33.161089+0000 osd.4 (osd.4) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:33.161129+0000 osd.4 (osd.4) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:33.443704+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-12T16:53:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:33.475541+0000 mgr.y (mgr.24481) 154 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:33.929479+0000 mon.b (mon.1) 49 : audit [DBG] from='client.? v1:172.21.15.33:0/1373634618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:34.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:34.659 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:34.722 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:33.161089+0000 osd.4 (osd.4) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:34.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:33.161129+0000 osd.4 (osd.4) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:34.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:33.443704+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-12T16:53:34.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:33.475541+0000 mgr.y (mgr.24481) 154 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:34.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:33.929479+0000 mon.b (mon.1) 49 : audit [DBG] from='client.? v1:172.21.15.33:0/1373634618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:35.017 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 62 16 86286 86270 86.9069 60.375 0.016453 0.0115022 2024-02-12T16:53:35.597 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:35.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:35.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:35.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:34.345849+0000 mgr.y (mgr.24481) 155 : cluster [DBG] pgmap v163: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T16:53:35.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:34.345849+0000 mgr.y (mgr.24481) 155 : cluster [DBG] pgmap v163: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T16:53:35.733 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:34.345849+0000 mgr.y (mgr.24481) 155 : cluster [DBG] pgmap v163: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T16:53:35.736 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:35.763 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:35.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:35.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:35.871 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:35.926 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:36.018 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 63 16 87266 87250 86.4979 61.25 0.00869551 0.0115565 2024-02-12T16:53:36.092 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:36.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:36.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:36.232 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:36.253 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:36.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:36.360 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:36.395 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:36.420 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:36.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:36.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:36.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:36.573 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:37.022 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 64 16 88090 88074 85.9464 51.5 0.0153801 0.0116301 2024-02-12T16:53:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:36.347753+0000 mgr.y (mgr.24481) 156 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T16:53:37.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:36.347753+0000 mgr.y (mgr.24481) 156 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T16:53:37.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:36.347753+0000 mgr.y (mgr.24481) 156 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T16:53:38.022 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 65 16 89058 89042 85.5551 60.5 0.0266094 0.0116833 2024-02-12T16:53:39.022 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 66 16 90014 89998 85.1643 59.75 0.0166808 0.0117364 2024-02-12T16:53:39.649 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-12T16:53:39.655 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:39.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:39.701 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:38.351027+0000 mgr.y (mgr.24481) 157 : cluster [DBG] pgmap v165: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-12T16:53:39.702 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:38.351027+0000 mgr.y (mgr.24481) 157 : cluster [DBG] pgmap v165: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-12T16:53:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:38.351027+0000 mgr.y (mgr.24481) 157 : cluster [DBG] pgmap v165: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-12T16:53:39.763 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:39.793 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:39.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:39.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:39.949 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:39.976 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:39.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:40.024 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 67 16 90896 90880 84.7138 55.125 0.0186513 0.0118007 2024-02-12T16:53:40.080 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:40.137 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:40.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:40.244 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:40.310 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:40.348 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:40.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:40.452 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:53:40.454 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:39.886335+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.33:0/3675276245' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:53:40.454 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:39.897882+0000 mon.a (mon.0) 806 : audit [INF] from='client.24841 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:53:40.456 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:40.516 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:40.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:39.886335+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.33:0/3675276245' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:53:40.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:39.897882+0000 mon.a (mon.0) 806 : audit [INF] from='client.24841 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:53:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:39.886335+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.33:0/3675276245' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:53:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:39.897882+0000 mon.a (mon.0) 806 : audit [INF] from='client.24841 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:53:40.932 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:53:40.932 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:53:41.024 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 68 16 91749 91733 84.2523 53.3125 0.0165046 0.0118644 2024-02-12T16:53:41.115 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:41.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:41.121 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:53:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:40] "GET /metrics HTTP/1.1" 200 175344 "" "Prometheus/2.33.4" 2024-02-12T16:53:41.124 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:41.222 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:41.255 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:41.282 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:41.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:41.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:41.416 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:41.439 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:41.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:41.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:41.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:40.351464+0000 mgr.y (mgr.24481) 158 : cluster [DBG] pgmap v166: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-12T16:53:41.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:40.424835+0000 mon.a (mon.0) 807 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:41.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:40.436687+0000 mon.a (mon.0) 808 : audit [INF] from='client.24841 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:53:41.577 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:40.436864+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-12T16:53:41.579 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:41.602 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:41.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:41.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:41.739 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:41.761 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:41.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:41.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:40.351464+0000 mgr.y (mgr.24481) 158 : cluster [DBG] pgmap v166: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-12T16:53:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:40.424835+0000 mon.a (mon.0) 807 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:40.436687+0000 mon.a (mon.0) 808 : audit [INF] from='client.24841 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:53:41.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:40.436864+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-12T16:53:41.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:40.351464+0000 mgr.y (mgr.24481) 158 : cluster [DBG] pgmap v166: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-12T16:53:41.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:40.424835+0000 mon.a (mon.0) 807 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:53:41.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:40.436687+0000 mon.a (mon.0) 808 : audit [INF] from='client.24841 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:53:41.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:40.436864+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-12T16:53:41.866 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:41.875 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:53:41.960 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:42.025 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 69 16 92647 92631 83.8447 56.125 0.0223052 0.011921 2024-02-12T16:53:42.071 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:53:42.732 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:42.741 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:53:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:53:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:40.841195+0000 osd.6 (osd.6) 59 : cluster [DBG] 4.a scrub starts 2024-02-12T16:53:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:40.847533+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.a scrub ok 2024-02-12T16:53:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:40.904853+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:53:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:40.916827+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:53:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:41.736103+0000 osd.3 (osd.3) 75 : cluster [DBG] 5.17 scrub starts 2024-02-12T16:53:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:41.740885+0000 osd.3 (osd.3) 76 : cluster [DBG] 5.17 scrub ok 2024-02-12T16:53:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:42.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:42.743 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:40.841195+0000 osd.6 (osd.6) 59 : cluster [DBG] 4.a scrub starts 2024-02-12T16:53:42.743 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:40.847533+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.a scrub ok 2024-02-12T16:53:42.743 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:40.904853+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:53:42.743 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:40.916827+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:53:42.743 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:41.736103+0000 osd.3 (osd.3) 75 : cluster [DBG] 5.17 scrub starts 2024-02-12T16:53:42.743 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:41.740885+0000 osd.3 (osd.3) 76 : cluster [DBG] 5.17 scrub ok 2024-02-12T16:53:42.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:40.841195+0000 osd.6 (osd.6) 59 : cluster [DBG] 4.a scrub starts 2024-02-12T16:53:42.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:40.847533+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.a scrub ok 2024-02-12T16:53:42.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:40.904853+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:53:42.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:40.916827+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:53:42.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:41.736103+0000 osd.3 (osd.3) 75 : cluster [DBG] 5.17 scrub starts 2024-02-12T16:53:42.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:41.740885+0000 osd.3 (osd.3) 76 : cluster [DBG] 5.17 scrub ok 2024-02-12T16:53:42.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:42.874 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:42.902 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:42.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:43.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:43.025 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 70 16 93531 93515 83.4358 55.25 0.00928111 0.0119806 2024-02-12T16:53:43.045 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:43.068 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:43.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:43.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:43.203 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:43.226 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:43.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:43.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:43.287 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:43.329 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:43.403 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:43.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:42.234727+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:53:43.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:42.248963+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:53:43.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:42.351864+0000 mgr.y (mgr.24481) 159 : cluster [DBG] pgmap v168: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 575 B/s rd, 32 MiB/s wr, 518 op/s 2024-02-12T16:53:43.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:42.944385+0000 osd.5 (osd.5) 67 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:53:43.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:42.946792+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:53:43.612 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:43.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:43.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:42.234727+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:53:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:42.248963+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:53:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:42.351864+0000 mgr.y (mgr.24481) 159 : cluster [DBG] pgmap v168: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 575 B/s rd, 32 MiB/s wr, 518 op/s 2024-02-12T16:53:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:42.944385+0000 osd.5 (osd.5) 67 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:53:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:42.946792+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:53:43.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:53:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:53:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:43.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:42.234727+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:53:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:42.248963+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:53:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:42.351864+0000 mgr.y (mgr.24481) 159 : cluster [DBG] pgmap v168: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 575 B/s rd, 32 MiB/s wr, 518 op/s 2024-02-12T16:53:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:42.944385+0000 osd.5 (osd.5) 67 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:53:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:42.946792+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:53:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:43.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:43.897 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:43.960 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:44.028 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 71 16 94388 94372 83.0124 53.5625 0.00877608 0.0120414 2024-02-12T16:53:44.047 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:44.051 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:44.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:44.156 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:44.219 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:44.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:44.324 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:44.357 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:44.379 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:44.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:44.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:44.516 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:44.540 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:44.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:44.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:44.681 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:44.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:44.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:44.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:53:44.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:43.225066+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:53:44.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:43.229614+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:53:44.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:43.492525+0000 mgr.y (mgr.24481) 160 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:44.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:43.501218+0000 osd.6 (osd.6) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:44.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:43.501276+0000 osd.6 (osd.6) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:44.784 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:43.225066+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:53:44.784 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:43.229614+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:53:44.784 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:43.492525+0000 mgr.y (mgr.24481) 160 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:44.784 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:43.501218+0000 osd.6 (osd.6) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:44.784 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:43.501276+0000 osd.6 (osd.6) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:44.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:44.852 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:43.225066+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:53:44.853 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:43.229614+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:53:44.853 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:43.492525+0000 mgr.y (mgr.24481) 160 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:44.853 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:43.501218+0000 osd.6 (osd.6) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:44.853 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:43.501276+0000 osd.6 (osd.6) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:44.855 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:44.882 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:44.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:44.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:45.020 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:45.028 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 72 16 95194 95178 82.5594 50.375 0.013354 0.0121069 2024-02-12T16:53:45.045 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:45.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:45.152 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:45.241 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:45.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:43.744819+0000 osd.6 (osd.6) 63 : cluster [DBG] 7.c scrub starts 2024-02-12T16:53:45.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:43.847660+0000 osd.1 (osd.1) 75 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:53:45.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:43.855853+0000 osd.1 (osd.1) 76 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:53:45.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:44.352654+0000 mgr.y (mgr.24481) 161 : cluster [DBG] pgmap v169: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 904 op/s 2024-02-12T16:53:45.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:44.482872+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.33:0/1933039888' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:53:45.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:44.488975+0000 mon.a (mon.0) 810 : audit [INF] from='client.24847 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:53:45.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:44.500318+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-12T16:53:45.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:44.942109+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.1a scrub starts 2024-02-12T16:53:45.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:44.946078+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.1a scrub ok 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:43.744819+0000 osd.6 (osd.6) 63 : cluster [DBG] 7.c scrub starts 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:43.847660+0000 osd.1 (osd.1) 75 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:43.855853+0000 osd.1 (osd.1) 76 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:44.352654+0000 mgr.y (mgr.24481) 161 : cluster [DBG] pgmap v169: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 904 op/s 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:44.482872+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.33:0/1933039888' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:44.488975+0000 mon.a (mon.0) 810 : audit [INF] from='client.24847 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:44.500318+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:44.942109+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.1a scrub starts 2024-02-12T16:53:45.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:44.946078+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.1a scrub ok 2024-02-12T16:53:45.554 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.6 to 1 (10000) 2024-02-12T16:53:45.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:43.744819+0000 osd.6 (osd.6) 63 : cluster [DBG] 7.c scrub starts 2024-02-12T16:53:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:43.847660+0000 osd.1 (osd.1) 75 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:53:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:43.855853+0000 osd.1 (osd.1) 76 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:53:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:44.352654+0000 mgr.y (mgr.24481) 161 : cluster [DBG] pgmap v169: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 904 op/s 2024-02-12T16:53:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:44.482872+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.33:0/1933039888' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:53:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:44.488975+0000 mon.a (mon.0) 810 : audit [INF] from='client.24847 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:53:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:44.500318+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-12T16:53:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:44.942109+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.1a scrub starts 2024-02-12T16:53:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:44.946078+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.1a scrub ok 2024-02-12T16:53:46.029 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 73 16 96085 96069 82.1903 55.6875 0.0118968 0.0121624 2024-02-12T16:53:46.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:45.409172+0000 osd.6 (osd.6) 64 : cluster [DBG] 7.c scrub ok 2024-02-12T16:53:46.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:45.507541+0000 mon.a (mon.0) 812 : audit [INF] from='client.24847 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:53:46.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:45.507604+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-12T16:53:46.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:45.643586+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:53:46.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:45.658281+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:53:46.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:45.409172+0000 osd.6 (osd.6) 64 : cluster [DBG] 7.c scrub ok 2024-02-12T16:53:46.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:45.507541+0000 mon.a (mon.0) 812 : audit [INF] from='client.24847 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:53:46.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:45.507604+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-12T16:53:46.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:45.643586+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:53:46.799 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:45.658281+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:53:46.831 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:46.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:46.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:46.969 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:45.409172+0000 osd.6 (osd.6) 64 : cluster [DBG] 7.c scrub ok 2024-02-12T16:53:46.969 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:45.507541+0000 mon.a (mon.0) 812 : audit [INF] from='client.24847 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:53:46.969 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:45.507604+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-12T16:53:46.969 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:45.643586+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:53:46.969 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:45.658281+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:53:46.971 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:47.000 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:47.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:47.030 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 74 16 96975 96959 81.8299 55.625 0.0143958 0.0122158 2024-02-12T16:53:47.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:47.146 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:47.171 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:47.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:47.275 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:47.369 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:47.462 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:53:47.463 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:53:47.463 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:47.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:47.567 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:47.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:45.793893+0000 osd.6 (osd.6) 65 : cluster [DBG] 7.d scrub starts 2024-02-12T16:53:47.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:46.353040+0000 mgr.y (mgr.24481) 162 : cluster [DBG] pgmap v172: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 739 op/s 2024-02-12T16:53:47.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:47.442268+0000 mon.a (mon.0) 814 : audit [DBG] from='client.? v1:172.21.15.33:0/4134961937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:47.618 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:45.793893+0000 osd.6 (osd.6) 65 : cluster [DBG] 7.d scrub starts 2024-02-12T16:53:47.618 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:46.353040+0000 mgr.y (mgr.24481) 162 : cluster [DBG] pgmap v172: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 739 op/s 2024-02-12T16:53:47.618 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:47.442268+0000 mon.a (mon.0) 814 : audit [DBG] from='client.? v1:172.21.15.33:0/4134961937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:47.627 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:47.671 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:47.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:47.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:47.871 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:47.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:45.793893+0000 osd.6 (osd.6) 65 : cluster [DBG] 7.d scrub starts 2024-02-12T16:53:47.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:46.353040+0000 mgr.y (mgr.24481) 162 : cluster [DBG] pgmap v172: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 739 op/s 2024-02-12T16:53:47.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:47.442268+0000 mon.a (mon.0) 814 : audit [DBG] from='client.? v1:172.21.15.33:0/4134961937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:53:48.031 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 75 16 97728 97712 81.3662 47.0625 0.00553965 0.0122849 2024-02-12T16:53:48.340 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T16:53:48.426 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:53:48.426 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:53:48.426 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:53:48.426 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:53:48.426 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:53:48.427 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:53:48.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:53:48.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:53:48.644 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:48.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:48.754 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:48.805 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:49.033 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 76 16 98594 98578 81.0055 54.125 0.00992895 0.0123401 2024-02-12T16:53:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:49.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:49.275 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:49.348 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:49.540 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:49.566 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:49.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:49.576 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:53:49.670 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:49.723 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.150845+0000 osd.6 (osd.6) 66 : cluster [DBG] 7.d scrub ok 2024-02-12T16:53:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.353823+0000 mgr.y (mgr.24481) 163 : cluster [DBG] pgmap v173: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 962 op/s 2024-02-12T16:53:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.463917+0000 osd.2 (osd.2) 69 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:53:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.468543+0000 osd.2 (osd.2) 70 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:53:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.810759+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.6 scrub starts 2024-02-12T16:53:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.828627+0000 osd.1 (osd.1) 78 : cluster [DBG] 2.6 scrub ok 2024-02-12T16:53:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.917235+0000 osd.4 (osd.4) 61 : cluster [DBG] 2.f scrub starts 2024-02-12T16:53:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.920968+0000 osd.4 (osd.4) 62 : cluster [DBG] 2.f scrub ok 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.150845+0000 osd.6 (osd.6) 66 : cluster [DBG] 7.d scrub ok 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.353823+0000 mgr.y (mgr.24481) 163 : cluster [DBG] pgmap v173: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 962 op/s 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.463917+0000 osd.2 (osd.2) 69 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.468543+0000 osd.2 (osd.2) 70 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.810759+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.6 scrub starts 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.828627+0000 osd.1 (osd.1) 78 : cluster [DBG] 2.6 scrub ok 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.917235+0000 osd.4 (osd.4) 61 : cluster [DBG] 2.f scrub starts 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.920968+0000 osd.4 (osd.4) 62 : cluster [DBG] 2.f scrub ok 2024-02-12T16:53:50.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.150845+0000 osd.6 (osd.6) 66 : cluster [DBG] 7.d scrub ok 2024-02-12T16:53:50.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.353823+0000 mgr.y (mgr.24481) 163 : cluster [DBG] pgmap v173: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 962 op/s 2024-02-12T16:53:50.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.463917+0000 osd.2 (osd.2) 69 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:53:50.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.468543+0000 osd.2 (osd.2) 70 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:53:50.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.810759+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.6 scrub starts 2024-02-12T16:53:50.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.828627+0000 osd.1 (osd.1) 78 : cluster [DBG] 2.6 scrub ok 2024-02-12T16:53:50.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.917235+0000 osd.4 (osd.4) 61 : cluster [DBG] 2.f scrub starts 2024-02-12T16:53:50.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.920968+0000 osd.4 (osd.4) 62 : cluster [DBG] 2.f scrub ok 2024-02-12T16:53:50.033 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 77 16 99444 99428 80.6436 53.125 0.0234909 0.0123952 2024-02-12T16:53:50.207 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:50.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:50.271 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:50.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:50.346 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:50.371 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:50.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:50.476 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:50.532 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:50.573 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:50.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:50.680 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:50.756 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.718582+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:53:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:48.719340+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:53:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:49.619760+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:53:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:49.624061+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:53:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:49.828543+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.c scrub starts 2024-02-12T16:53:50.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:49.832789+0000 osd.1 (osd.1) 80 : cluster [DBG] 5.c scrub ok 2024-02-12T16:53:50.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:49.923123+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:53:50.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:49.924634+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:53:50.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.718582+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:53:50.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:48.719340+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:53:50.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:49.619760+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:53:50.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:49.624061+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:53:50.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:49.828543+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.c scrub starts 2024-02-12T16:53:50.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:49.832789+0000 osd.1 (osd.1) 80 : cluster [DBG] 5.c scrub ok 2024-02-12T16:53:50.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:49.923123+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:53:50.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:49.924634+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:53:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.718582+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.11 scrub starts 2024-02-12T16:53:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:48.719340+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.11 scrub ok 2024-02-12T16:53:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:49.619760+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:53:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:49.624061+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:53:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:49.828543+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.c scrub starts 2024-02-12T16:53:50.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:49.832789+0000 osd.1 (osd.1) 80 : cluster [DBG] 5.c scrub ok 2024-02-12T16:53:50.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:49.923123+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:53:50.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:49.924634+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:53:51.028 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:51.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:51.038 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 78 16 100313 100297 80.3014 54.3125 0.0169169 0.0124466 2024-02-12T16:53:51.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:51.147 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:53:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:50] "GET /metrics HTTP/1.1" 200 175356 "" "Prometheus/2.33.4" 2024-02-12T16:53:51.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:53:51.162 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:51.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:51.290 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:51.388 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:51.474 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:51.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:51.581 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:51.650 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:51.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:49.711781+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:53:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:49.715805+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:53:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:50.358801+0000 mgr.y (mgr.24481) 164 : cluster [DBG] pgmap v174: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-12T16:53:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:50.835201+0000 osd.1 (osd.1) 81 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:53:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:50.836399+0000 osd.1 (osd.1) 82 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:53:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:49.711781+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:53:51.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:49.715805+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:53:51.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:50.358801+0000 mgr.y (mgr.24481) 164 : cluster [DBG] pgmap v174: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-12T16:53:51.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:50.835201+0000 osd.1 (osd.1) 81 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:53:51.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:50.836399+0000 osd.1 (osd.1) 82 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:53:51.985 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:49.711781+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:53:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:49.715805+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:53:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:50.358801+0000 mgr.y (mgr.24481) 164 : cluster [DBG] pgmap v174: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-12T16:53:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:50.835201+0000 osd.1 (osd.1) 81 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:53:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:50.836399+0000 osd.1 (osd.1) 82 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:53:52.020 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:52.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:52.038 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 79 16 101158 101142 79.9537 52.8125 0.0281008 0.0125026 2024-02-12T16:53:52.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:52.155 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:52.183 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:52.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:52.288 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:52.364 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:52.620 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:50.676532+0000 osd.6 (osd.6) 71 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:50.681136+0000 osd.6 (osd.6) 72 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:51.197038+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.18 scrub starts 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:51.204269+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.18 scrub ok 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:51.529650+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.15 scrub starts 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:51.532549+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.15 scrub ok 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:50.676532+0000 osd.6 (osd.6) 71 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:50.681136+0000 osd.6 (osd.6) 72 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:51.197038+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.18 scrub starts 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:51.204269+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.18 scrub ok 2024-02-12T16:53:52.621 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:51.529650+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.15 scrub starts 2024-02-12T16:53:52.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:51.532549+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.15 scrub ok 2024-02-12T16:53:52.824 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:52.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:52.930 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:52.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:53:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:53:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:53:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:50.676532+0000 osd.6 (osd.6) 71 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:53:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:50.681136+0000 osd.6 (osd.6) 72 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:53:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:51.197038+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.18 scrub starts 2024-02-12T16:53:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:51.204269+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.18 scrub ok 2024-02-12T16:53:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:51.529650+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.15 scrub starts 2024-02-12T16:53:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:51.532549+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.15 scrub ok 2024-02-12T16:53:53.012 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:53.039 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T16:53:53.038589+0000 min lat: 0.00122014 max lat: 0.115951 avg lat: 0.0125573 2024-02-12T16:53:53.039 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:53:53.039 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 80 16 101989 101973 79.6026 51.9375 0.0237533 0.0125573 2024-02-12T16:53:53.372 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:53.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:53.478 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:53:53.556 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:53.632 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:52.363265+0000 mgr.y (mgr.24481) 165 : cluster [DBG] pgmap v175: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 527 op/s 2024-02-12T16:53:53.633 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:52.652164+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:53:53.633 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:52.363265+0000 mgr.y (mgr.24481) 165 : cluster [DBG] pgmap v175: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 527 op/s 2024-02-12T16:53:53.633 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:52.652164+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:53:53.663 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:53:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:53.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:53:53.777 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:53.808 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:52.363265+0000 mgr.y (mgr.24481) 165 : cluster [DBG] pgmap v175: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 527 op/s 2024-02-12T16:53:53.808 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:52.652164+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:53:53.808 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:53:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:53:53.814 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:53.842 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:53.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:53.949 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:54.004 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:54.044 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 81 16 102822 102806 79.258 52.0625 0.0277001 0.0126112 2024-02-12T16:53:54.398 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:54.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:54.431 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:54.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:53:54.538 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:54.560 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:53:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:54.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:53:54.633 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:52.674355+0000 osd.6 (osd.6) 73 : cluster [DBG] 7.7 scrub starts 2024-02-12T16:53:54.633 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:53.505032+0000 mgr.y (mgr.24481) 166 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:54.633 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:53.761687+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.33:0/3421873068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:54.633 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:53.762185+0000 mon.a (mon.0) 815 : audit [INF] from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:54.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:54.191716+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:53:54.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:54.196808+0000 mon.c (mon.2) 87 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:53:54.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:54.203172+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:54.644 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T16:53:54.665 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:54.753 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:54.914 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:52.674355+0000 osd.6 (osd.6) 73 : cluster [DBG] 7.7 scrub starts 2024-02-12T16:53:54.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:53.505032+0000 mgr.y (mgr.24481) 166 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:54.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:53.761687+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.33:0/3421873068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:54.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:53.762185+0000 mon.a (mon.0) 815 : audit [INF] from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:54.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:54.191716+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:53:54.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:54.196808+0000 mon.c (mon.2) 87 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:53:54.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:54.203172+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:54.922 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:54.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:54.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:52.674355+0000 osd.6 (osd.6) 73 : cluster [DBG] 7.7 scrub starts 2024-02-12T16:53:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:53.505032+0000 mgr.y (mgr.24481) 166 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:53:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:53.761687+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.33:0/3421873068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:53.762185+0000 mon.a (mon.0) 815 : audit [INF] from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:53:54.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:54.191716+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:53:54.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:54.196808+0000 mon.c (mon.2) 87 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:53:54.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:54.203172+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:53:55.028 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:53:55.045 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 82 16 103614 103598 78.8953 49.5 0.0206316 0.0126696 2024-02-12T16:53:55.059 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:53:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:55.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:53:55.187 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:55.241 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:55.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:55.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:55.416 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:54.369153+0000 mgr.y (mgr.24481) 167 : cluster [DBG] pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 858 op/s 2024-02-12T16:53:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:54.620054+0000 mon.a (mon.0) 817 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:53:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:54.620081+0000 mon.a (mon.0) 818 : cluster [INF] Cluster is now healthy 2024-02-12T16:53:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:54.627071+0000 mon.a (mon.0) 819 : audit [INF] from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:53:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:54.627248+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-12T16:53:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:55.498393+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.33:0/3425934099' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:53:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:55.499501+0000 mon.a (mon.0) 821 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:53:56.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:54.369153+0000 mgr.y (mgr.24481) 167 : cluster [DBG] pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 858 op/s 2024-02-12T16:53:56.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:54.620054+0000 mon.a (mon.0) 817 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:53:56.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:54.620081+0000 mon.a (mon.0) 818 : cluster [INF] Cluster is now healthy 2024-02-12T16:53:56.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:54.627071+0000 mon.a (mon.0) 819 : audit [INF] from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:53:56.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:54.627248+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-12T16:53:56.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:55.498393+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.33:0/3425934099' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:53:56.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:55.499501+0000 mon.a (mon.0) 821 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:53:56.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:54.369153+0000 mgr.y (mgr.24481) 167 : cluster [DBG] pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 858 op/s 2024-02-12T16:53:56.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:54.620054+0000 mon.a (mon.0) 817 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:53:56.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:54.620081+0000 mon.a (mon.0) 818 : cluster [INF] Cluster is now healthy 2024-02-12T16:53:56.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:54.627071+0000 mon.a (mon.0) 819 : audit [INF] from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:53:56.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:54.627248+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-12T16:53:56.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:55.498393+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.33:0/3425934099' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:53:56.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:55.499501+0000 mon.a (mon.0) 821 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:53:56.044 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 83 16 104478 104462 78.5955 54 0.00474521 0.0127193 2024-02-12T16:53:56.240 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T16:53:57.011 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:57.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:57.045 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 84 16 105328 105312 78.2924 53.125 0.0186662 0.0127679 2024-02-12T16:53:57.118 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:53:57.153 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:57.176 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:53:57.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:57.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:53:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:55.684316+0000 osd.6 (osd.6) 74 : cluster [DBG] 7.7 scrub ok 2024-02-12T16:53:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:56.213872+0000 mon.a (mon.0) 822 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:53:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:56.213922+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-12T16:53:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:56.369562+0000 mgr.y (mgr.24481) 168 : cluster [DBG] pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 638 B/s rd, 41 MiB/s wr, 660 op/s 2024-02-12T16:53:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:55.684316+0000 osd.6 (osd.6) 74 : cluster [DBG] 7.7 scrub ok 2024-02-12T16:53:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:56.213872+0000 mon.a (mon.0) 822 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:53:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:56.213922+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-12T16:53:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:56.369562+0000 mgr.y (mgr.24481) 168 : cluster [DBG] pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 638 B/s rd, 41 MiB/s wr, 660 op/s 2024-02-12T16:53:57.279 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:57.356 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:55.684316+0000 osd.6 (osd.6) 74 : cluster [DBG] 7.7 scrub ok 2024-02-12T16:53:57.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:56.213872+0000 mon.a (mon.0) 822 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:53:57.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:56.213922+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-12T16:53:57.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:56.369562+0000 mgr.y (mgr.24481) 168 : cluster [DBG] pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 638 B/s rd, 41 MiB/s wr, 660 op/s 2024-02-12T16:53:57.496 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:57.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:57.602 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:57.658 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:53:57.658 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T16:53:57.658 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T16:53:57.658 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T16:53:57.659 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:53:57.669 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:58.047 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 85 16 106248 106232 78.0459 57.5 0.00820922 0.0128082 2024-02-12T16:53:58.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:53:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:56.620749+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:53:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:56.623915+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:53:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:56.742177+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-02-12T16:53:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:56.761439+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-02-12T16:53:58.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:56.620749+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:53:58.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:56.623915+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:53:58.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:56.742177+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-02-12T16:53:58.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:56.761439+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-02-12T16:53:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:56.620749+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:53:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:56.623915+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:53:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:56.742177+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-02-12T16:53:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:56.761439+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-02-12T16:53:58.758 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:53:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:58.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:53:58.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:53:58.805 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T16:53:58.805 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:53:58.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:53:58.904 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:58.925 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:53:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:58.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:53:59.029 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:53:59.047 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 86 16 107072 107056 77.737 51.5 0.0636501 0.0128586 2024-02-12T16:53:59.118 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:59.150 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:53:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:59.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:53:59.255 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:53:59.311 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:59.451 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:57.636027+0000 osd.3 (osd.3) 83 : cluster [DBG] 4.17 scrub starts 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:57.641382+0000 osd.3 (osd.3) 84 : cluster [DBG] 4.17 scrub ok 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:57.734697+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.e scrub starts 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:57.738238+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.e scrub ok 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:58.370222+0000 mgr.y (mgr.24481) 169 : cluster [DBG] pgmap v180: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:58.519484+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:58.524717+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.035288+0000 osd.2 (osd.2) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:59.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.035338+0000 osd.2 (osd.2) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:53:59.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:57.636027+0000 osd.3 (osd.3) 83 : cluster [DBG] 4.17 scrub starts 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:57.641382+0000 osd.3 (osd.3) 84 : cluster [DBG] 4.17 scrub ok 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:57.734697+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.e scrub starts 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:57.738238+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.e scrub ok 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:58.370222+0000 mgr.y (mgr.24481) 169 : cluster [DBG] pgmap v180: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:58.519484+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:58.524717+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.035288+0000 osd.2 (osd.2) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:59.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:53:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.035338+0000 osd.2 (osd.2) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:59.466 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:59.532 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:59.532 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:59.556 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:53:59.558 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:53:59.599 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:53:59.636 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:53:59.637 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:53:59.684 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:53:59.719 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:53:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:57.636027+0000 osd.3 (osd.3) 83 : cluster [DBG] 4.17 scrub starts 2024-02-12T16:53:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:57.641382+0000 osd.3 (osd.3) 84 : cluster [DBG] 4.17 scrub ok 2024-02-12T16:53:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:57.734697+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.e scrub starts 2024-02-12T16:53:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:57.738238+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.e scrub ok 2024-02-12T16:53:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:58.370222+0000 mgr.y (mgr.24481) 169 : cluster [DBG] pgmap v180: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-12T16:53:59.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:58.519484+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T16:53:59.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:58.524717+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T16:53:59.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.035288+0000 osd.2 (osd.2) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:59.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:53:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.035338+0000 osd.2 (osd.2) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:59.784 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:53:59.813 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:53:59.858 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:53:59.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:53:59.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:53:59.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:53:59.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:53:59.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:53:59.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:53:59.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:53:59.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:54:00.047 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 87 16 107758 107742 77.3366 42.875 0.0101293 0.0129247 2024-02-12T16:54:00.355 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:58.738358+0000 osd.6 (osd.6) 75 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:54:00.355 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:58.740117+0000 osd.6 (osd.6) 76 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:54:00.355 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.168675+0000 osd.0 (osd.0) 65 : cluster [DBG] 5.3 scrub starts 2024-02-12T16:54:00.355 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.172998+0000 osd.0 (osd.0) 66 : cluster [DBG] 5.3 scrub ok 2024-02-12T16:54:00.356 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.272722+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-12T16:54:00.356 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:53:59.493021+0000 mon.a (mon.0) 825 : audit [DBG] from='client.? v1:172.21.15.33:0/575925965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:00.356 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.562438+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:54:00.356 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.566722+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:54:00.356 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:58.738358+0000 osd.6 (osd.6) 75 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:54:00.356 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:58.740117+0000 osd.6 (osd.6) 76 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:54:00.356 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.168675+0000 osd.0 (osd.0) 65 : cluster [DBG] 5.3 scrub starts 2024-02-12T16:54:00.356 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.172998+0000 osd.0 (osd.0) 66 : cluster [DBG] 5.3 scrub ok 2024-02-12T16:54:00.357 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.272722+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-12T16:54:00.357 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:53:59.493021+0000 mon.a (mon.0) 825 : audit [DBG] from='client.? v1:172.21.15.33:0/575925965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:00.357 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.562438+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:54:00.357 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.566722+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:54:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:58.738358+0000 osd.6 (osd.6) 75 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:54:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:58.740117+0000 osd.6 (osd.6) 76 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:54:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.168675+0000 osd.0 (osd.0) 65 : cluster [DBG] 5.3 scrub starts 2024-02-12T16:54:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.172998+0000 osd.0 (osd.0) 66 : cluster [DBG] 5.3 scrub ok 2024-02-12T16:54:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.272722+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-12T16:54:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:53:59.493021+0000 mon.a (mon.0) 825 : audit [DBG] from='client.? v1:172.21.15.33:0/575925965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.562438+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:54:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.566722+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:54:01.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:54:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:00] "GET /metrics HTTP/1.1" 200 175356 "" "Prometheus/2.33.4" 2024-02-12T16:54:01.047 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 88 16 108698 108682 77.1255 58.75 0.0227609 0.0129613 2024-02-12T16:54:01.482 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:01.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.736283+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:54:01.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:53:59.751026+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:54:01.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.103597+0000 osd.5 (osd.5) 71 : cluster [DBG] 5.15 scrub starts 2024-02-12T16:54:01.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.112852+0000 osd.5 (osd.5) 72 : cluster [DBG] 5.15 scrub ok 2024-02-12T16:54:01.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.178691+0000 osd.0 (osd.0) 67 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:54:01.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.186639+0000 osd.0 (osd.0) 68 : cluster [DBG] 6.9 scrub ok 2024-02-12T16:54:01.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.370611+0000 mgr.y (mgr.24481) 170 : cluster [DBG] pgmap v182: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 44 MiB/s wr, 712 op/s 2024-02-12T16:54:01.486 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.736283+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:54:01.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:53:59.751026+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:54:01.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.103597+0000 osd.5 (osd.5) 71 : cluster [DBG] 5.15 scrub starts 2024-02-12T16:54:01.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.112852+0000 osd.5 (osd.5) 72 : cluster [DBG] 5.15 scrub ok 2024-02-12T16:54:01.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.178691+0000 osd.0 (osd.0) 67 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:54:01.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.186639+0000 osd.0 (osd.0) 68 : cluster [DBG] 6.9 scrub ok 2024-02-12T16:54:01.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.370611+0000 mgr.y (mgr.24481) 170 : cluster [DBG] pgmap v182: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 44 MiB/s wr, 712 op/s 2024-02-12T16:54:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:01.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:01.589 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:01.653 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.736283+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:54:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:53:59.751026+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:54:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.103597+0000 osd.5 (osd.5) 71 : cluster [DBG] 5.15 scrub starts 2024-02-12T16:54:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.112852+0000 osd.5 (osd.5) 72 : cluster [DBG] 5.15 scrub ok 2024-02-12T16:54:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.178691+0000 osd.0 (osd.0) 67 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:54:01.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.186639+0000 osd.0 (osd.0) 68 : cluster [DBG] 6.9 scrub ok 2024-02-12T16:54:01.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.370611+0000 mgr.y (mgr.24481) 170 : cluster [DBG] pgmap v182: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 44 MiB/s wr, 712 op/s 2024-02-12T16:54:01.925 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:01.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:02.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:02.049 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 89 16 109631 109615 76.913 58.3125 0.0127087 0.0129971 2024-02-12T16:54:02.076 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:02.100 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:02.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:02.205 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:02.285 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:02.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.547382+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:54:02.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.551033+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:54:02.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.655037+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:54:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.665243+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:54:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.777117+0000 osd.1 (osd.1) 89 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:54:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:00.780842+0000 osd.1 (osd.1) 90 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:54:02.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.547382+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:54:02.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.551033+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:54:02.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.655037+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:54:02.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.665243+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:54:02.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.777117+0000 osd.1 (osd.1) 89 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:54:02.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:00.780842+0000 osd.1 (osd.1) 90 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:54:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.547382+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:54:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.551033+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:54:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.655037+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:54:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.665243+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:54:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.777117+0000 osd.1 (osd.1) 89 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:54:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:00.780842+0000 osd.1 (osd.1) 90 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:54:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:54:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:54:03.049 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total time run: 90.0069 2024-02-12T16:54:03.049 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total writes made: 110449 2024-02-12T16:54:03.049 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Write size: 65536 2024-02-12T16:54:03.049 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object size: 65536 2024-02-12T16:54:03.049 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Bandwidth (MB/sec): 76.6949 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Bandwidth: 30.4349 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max bandwidth (MB/sec): 180.25 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min bandwidth (MB/sec): 42.875 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average IOPS: 1227 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev IOPS: 486.958 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max IOPS: 2884 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min IOPS: 686 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average Latency(s): 0.013035 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Latency(s): 0.00779527 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max latency(s): 0.115951 2024-02-12T16:54:03.050 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min latency(s): 0.00122014 2024-02-12T16:54:03.053 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:54:03.379 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:03.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:03.487 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:03.559 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:01.558429+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:54:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:01.572657+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:54:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:01.661761+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.10 scrub starts 2024-02-12T16:54:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:01.670100+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.10 scrub ok 2024-02-12T16:54:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:01.761409+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:54:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:01.773034+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:54:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:02.170812+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.16 scrub starts 2024-02-12T16:54:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:02.181098+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.16 scrub ok 2024-02-12T16:54:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:02.371016+0000 mgr.y (mgr.24481) 171 : cluster [DBG] pgmap v183: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 661 B/s rd, 34 MiB/s wr, 552 op/s 2024-02-12T16:54:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:02.453595+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:03.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:54:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:54:03.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:01.558429+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:54:03.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:01.572657+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:54:03.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:01.661761+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.10 scrub starts 2024-02-12T16:54:03.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:01.670100+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.10 scrub ok 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:01.761409+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:01.773034+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:02.170812+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.16 scrub starts 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:02.181098+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.16 scrub ok 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:02.371016+0000 mgr.y (mgr.24481) 171 : cluster [DBG] pgmap v183: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 661 B/s rd, 34 MiB/s wr, 552 op/s 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:02.453595+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:01.558429+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:01.572657+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:01.661761+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.10 scrub starts 2024-02-12T16:54:03.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:01.670100+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.10 scrub ok 2024-02-12T16:54:03.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:01.761409+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.15 scrub starts 2024-02-12T16:54:03.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:01.773034+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.15 scrub ok 2024-02-12T16:54:03.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:02.170812+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.16 scrub starts 2024-02-12T16:54:03.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:02.181098+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.16 scrub ok 2024-02-12T16:54:03.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:02.371016+0000 mgr.y (mgr.24481) 171 : cluster [DBG] pgmap v183: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 661 B/s rd, 34 MiB/s wr, 552 op/s 2024-02-12T16:54:03.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:02.453595+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:03.940 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:54:04.131 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:04.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:04.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:04.235 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:04.317 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:04.354 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:54:04.365 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:04.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:04.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:04.507 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:04.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:04.636 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:04.687 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:02.619799+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:54:04.687 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:02.624856+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:54:04.687 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:02.784460+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:54:04.687 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:02.785470+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:54:04.687 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:03.515497+0000 mgr.y (mgr.24481) 172 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:04.687 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:03.986911+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.a scrub starts 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:03.996121+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.a scrub ok 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:04.205100+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.33:0/2569911574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:04.208333+0000 mon.a (mon.0) 827 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:02.619799+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:02.624856+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:02.784460+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:02.785470+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:03.515497+0000 mgr.y (mgr.24481) 172 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:03.986911+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.a scrub starts 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:03.996121+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.a scrub ok 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:04.205100+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.33:0/2569911574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:04.208333+0000 mon.a (mon.0) 827 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:04.697 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:02.619799+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:54:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:02.624856+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:54:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:02.784460+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:54:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:02.785470+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:54:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:03.515497+0000 mgr.y (mgr.24481) 172 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:03.986911+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.a scrub starts 2024-02-12T16:54:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:03.996121+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.a scrub ok 2024-02-12T16:54:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:04.205100+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.33:0/2569911574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:04.208333+0000 mon.a (mon.0) 827 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:05.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:05.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:05.269 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:05.302 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:05.327 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:05.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:05.356 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:05.356 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:05.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:05.465 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:05.495 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:05.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:05.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:03.606624+0000 osd.3 (osd.3) 91 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:03.616709+0000 osd.3 (osd.3) 92 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:04.239931+0000 osd.0 (osd.0) 71 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:04.254676+0000 osd.0 (osd.0) 72 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:04.322972+0000 mon.a (mon.0) 828 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:04.328383+0000 mon.a (mon.0) 829 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:04.328444+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:04.372663+0000 mgr.y (mgr.24481) 173 : cluster [DBG] pgmap v185: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 64 MiB/s wr, 1.06k op/s 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:03.606624+0000 osd.3 (osd.3) 91 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:54:05.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:03.616709+0000 osd.3 (osd.3) 92 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:54:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:04.239931+0000 osd.0 (osd.0) 71 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:54:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:04.254676+0000 osd.0 (osd.0) 72 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:54:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:04.322972+0000 mon.a (mon.0) 828 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:04.328383+0000 mon.a (mon.0) 829 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:54:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:04.328444+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-12T16:54:05.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:04.372663+0000 mgr.y (mgr.24481) 173 : cluster [DBG] pgmap v185: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 64 MiB/s wr, 1.06k op/s 2024-02-12T16:54:05.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:05.685 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:03.606624+0000 osd.3 (osd.3) 91 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:54:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:03.616709+0000 osd.3 (osd.3) 92 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:54:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:04.239931+0000 osd.0 (osd.0) 71 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:54:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:04.254676+0000 osd.0 (osd.0) 72 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:54:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:04.322972+0000 mon.a (mon.0) 828 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:04.328383+0000 mon.a (mon.0) 829 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:54:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:04.328444+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-12T16:54:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:04.372663+0000 mgr.y (mgr.24481) 173 : cluster [DBG] pgmap v185: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 64 MiB/s wr, 1.06k op/s 2024-02-12T16:54:06.947 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:06.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:06.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:07.052 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:06.373113+0000 mgr.y (mgr.24481) 174 : cluster [DBG] pgmap v186: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 530 op/s 2024-02-12T16:54:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:06.373113+0000 mgr.y (mgr.24481) 174 : cluster [DBG] pgmap v186: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 530 op/s 2024-02-12T16:54:07.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:06.373113+0000 mgr.y (mgr.24481) 174 : cluster [DBG] pgmap v186: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 530 op/s 2024-02-12T16:54:08.940 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:54:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:07.541121+0000 osd.3 (osd.3) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:07.541172+0000 osd.3 (osd.3) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:09.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:08.094007+0000 mon.a (mon.0) 831 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-12T16:54:09.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:08.373934+0000 mgr.y (mgr.24481) 175 : cluster [DBG] pgmap v188: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.12k op/s 2024-02-12T16:54:09.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:07.541121+0000 osd.3 (osd.3) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:09.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:07.541172+0000 osd.3 (osd.3) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:09.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:08.094007+0000 mon.a (mon.0) 831 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-12T16:54:09.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:08.373934+0000 mgr.y (mgr.24481) 175 : cluster [DBG] pgmap v188: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.12k op/s 2024-02-12T16:54:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:07.541121+0000 osd.3 (osd.3) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:07.541172+0000 osd.3 (osd.3) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:08.094007+0000 mon.a (mon.0) 831 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-12T16:54:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:08.373934+0000 mgr.y (mgr.24481) 175 : cluster [DBG] pgmap v188: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.12k op/s 2024-02-12T16:54:11.279 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:54:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:10] "GET /metrics HTTP/1.1" 200 175373 "" "Prometheus/2.33.4" 2024-02-12T16:54:11.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:10.375079+0000 mgr.y (mgr.24481) 176 : cluster [DBG] pgmap v189: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.12k op/s 2024-02-12T16:54:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:10.375079+0000 mgr.y (mgr.24481) 176 : cluster [DBG] pgmap v189: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.12k op/s 2024-02-12T16:54:11.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:10.375079+0000 mgr.y (mgr.24481) 176 : cluster [DBG] pgmap v189: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 1.12k op/s 2024-02-12T16:54:12.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:54:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:54:12.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:11.372673+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:12.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:11.372727+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:12.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:12.062115+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-12T16:54:12.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:11.372673+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:12.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:11.372727+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:12.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:12.062115+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-12T16:54:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:11.372673+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:11.372727+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:12.062115+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-12T16:54:13.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:12.375456+0000 mgr.y (mgr.24481) 177 : cluster [DBG] pgmap v191: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s 2024-02-12T16:54:13.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:54:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:54:13.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:12.375456+0000 mgr.y (mgr.24481) 177 : cluster [DBG] pgmap v191: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s 2024-02-12T16:54:13.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:12.375456+0000 mgr.y (mgr.24481) 177 : cluster [DBG] pgmap v191: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s 2024-02-12T16:54:14.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:13.215917+0000 osd.0 (osd.0) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:14.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:13.215970+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:14.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:13.525525+0000 mgr.y (mgr.24481) 178 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:14.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:14.065098+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-12T16:54:14.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:13.215917+0000 osd.0 (osd.0) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:14.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:13.215970+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:14.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:13.525525+0000 mgr.y (mgr.24481) 178 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:14.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:14.065098+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-12T16:54:14.771 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:13.215917+0000 osd.0 (osd.0) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:14.771 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:13.215970+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:14.772 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:13.525525+0000 mgr.y (mgr.24481) 178 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:14.772 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:14.065098+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-12T16:54:14.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:54:15.012 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:15.047 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:15.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:15.116 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:15.155 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:15.375 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:54:15.391 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:15.395 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:54:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:15.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:15.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:15.561 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:14.375841+0000 mgr.y (mgr.24481) 179 : cluster [DBG] pgmap v193: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-02-12T16:54:15.561 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:14.375841+0000 mgr.y (mgr.24481) 179 : cluster [DBG] pgmap v193: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-02-12T16:54:15.572 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:14.375841+0000 mgr.y (mgr.24481) 179 : cluster [DBG] pgmap v193: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-02-12T16:54:15.787 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:15.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:15.893 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:15.957 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:16.172 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:16.172 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:16.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:16.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:16.318 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:16.382 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:16.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:16.493 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:16.537 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:15.349011+0000 mgr.y (mgr.24481) 180 : audit [DBG] from='client.14940 v1:172.21.15.33:0/3778317916' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:54:16.537 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:15.349011+0000 mgr.y (mgr.24481) 180 : audit [DBG] from='client.14940 v1:172.21.15.33:0/3778317916' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:54:16.542 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:16.636 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:16.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:16.729 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:15.349011+0000 mgr.y (mgr.24481) 180 : audit [DBG] from='client.14940 v1:172.21.15.33:0/3778317916' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:54:16.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:16.814 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:16.872 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:16.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:16.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:17.031 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:17.046 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:17.046 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:17.490 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:16.376261+0000 mgr.y (mgr.24481) 181 : cluster [DBG] pgmap v194: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s 2024-02-12T16:54:17.490 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:17.453233+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:17.576 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:54:17.609 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:17.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:17.619 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:54:17.649 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:54:17.679 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:54:17.716 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:54:17.716 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:54:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:16.376261+0000 mgr.y (mgr.24481) 181 : cluster [DBG] pgmap v194: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s 2024-02-12T16:54:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:17.453233+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:17.777 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:54:17.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:16.376261+0000 mgr.y (mgr.24481) 181 : cluster [DBG] pgmap v194: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s 2024-02-12T16:54:17.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:17.453233+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:17.782 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:17.816 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:54:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:17.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:54:17.831 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:54:17.858 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:54:17.889 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:54:17.890 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:54:17.890 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:54:17.890 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:54:17.890 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:54:17.890 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:54:17.890 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:54:17.890 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:54:17.920 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:17.979 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:18.618 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:17.533346+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.33:0/644921743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:18.623 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:17.533346+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.33:0/644921743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:18.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:17.533346+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.33:0/644921743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:19.154 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:19.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:19.261 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:19.329 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:19.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:18.377039+0000 mgr.y (mgr.24481) 182 : cluster [DBG] pgmap v195: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-02-12T16:54:19.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:54:19.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:18.377039+0000 mgr.y (mgr.24481) 182 : cluster [DBG] pgmap v195: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-02-12T16:54:19.939 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-02-12T16:54:14.375612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6603478716,"num_objects":100954,"num_object_clones":0,"num_object_copies":302862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100954,"num_whiteouts":0,"num_read":981,"num_read_kb":733,"num_write":120566,"num_write_kb":7068778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88830,"ondisk_log_size":88830,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21871864,"kb_used_data":19465208,"kb_used_omap":0,"kb_used_meta":2406400,"kb_avail":728089352,"statfs":{"total":767960285184,"available":745563496448,"internally_reserved":0,"allocated":19932372992,"data_stored":19929111060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2464153600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-615907328,"num_objects":-9398,"num_object_clones":0,"num_object_copies":-28194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9398,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9398,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.278589"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":81,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102108+0000","last_change":"2024-02-12T16:51:45.053060+0000","last_active":"2024-02-12T16:54:08.102108+0000","last_peered":"2024-02-12T16:54:08.102108+0000","last_clean":"2024-02-12T16:54:08.102108+0000","last_became_active":"2024-02-12T16:49:45.223515+0000","last_became_peered":"2024-02-12T16:49:45.223515+0000","last_unstale":"2024-02-12T16:54:08.102108+0000","last_undegraded":"2024-02-12T16:54:08.102108+0000","last_fullsized":"2024-02-12T16:54:08.102108+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:45.052938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:45.052938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":56,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.111897+0000","last_change":"2024-02-12T16:53:49.715871+0000","last_active":"2024-02-12T16:54:08.111897+0000","last_peered":"2024-02-12T16:54:08.111897+0000","last_clean":"2024-02-12T16:54:08.111897+0000","last_became_active":"2024-02-12T16:52:20.563961+0000","last_became_peered":"2024-02-12T16:52:20.563961+0000","last_unstale":"2024-02-12T16:54:08.111897+0000","last_undegraded":"2024-02-12T16:54:08.111897+0000","last_fullsized":"2024-02-12T16:54:08.111897+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:49.715809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.715809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":91,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.106100+0000","last_change":"2024-02-12T16:52:41.990018+0000","last_active":"2024-02-12T16:54:08.106100+0000","last_peered":"2024-02-12T16:54:08.106100+0000","last_clean":"2024-02-12T16:54:08.106100+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:54:08.106100+0000","last_undegraded":"2024-02-12T16:54:08.106100+0000","last_fullsized":"2024-02-12T16:54:08.106100+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:41.989918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.989918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":115,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102170+0000","last_change":"2024-02-12T16:52:47.024635+0000","last_active":"2024-02-12T16:54:08.102170+0000","last_peered":"2024-02-12T16:54:08.102170+0000","last_clean":"2024-02-12T16:54:08.102170+0000","last_became_active":"2024-02-12T16:49:41.185805+0000","last_became_peered":"2024-02-12T16:49:41.185805+0000","last_unstale":"2024-02-12T16:54:08.102170+0000","last_undegraded":"2024-02-12T16:54:08.102170+0000","last_fullsized":"2024-02-12T16:54:08.102170+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:52:47.024543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.024543+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102405+0000","last_change":"2024-02-12T16:53:44.946159+0000","last_active":"2024-02-12T16:54:08.102405+0000","last_peered":"2024-02-12T16:54:08.102405+0000","last_clean":"2024-02-12T16:54:08.102405+0000","last_became_active":"2024-02-12T16:52:20.561617+0000","last_became_peered":"2024-02-12T16:52:20.561617+0000","last_unstale":"2024-02-12T16:54:08.102405+0000","last_undegraded":"2024-02-12T16:54:08.102405+0000","last_fullsized":"2024-02-12T16:54:08.102405+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099037+0000","last_change":"2024-02-12T16:52:58.817066+0000","last_active":"2024-02-12T16:54:08.099037+0000","last_peered":"2024-02-12T16:54:08.099037+0000","last_clean":"2024-02-12T16:54:08.099037+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:54:08.099037+0000","last_undegraded":"2024-02-12T16:54:08.099037+0000","last_fullsized":"2024-02-12T16:54:08.099037+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:58.816871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.816871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.101314+0000","last_change":"2024-02-12T16:51:38.073505+0000","last_active":"2024-02-12T16:54:08.101314+0000","last_peered":"2024-02-12T16:54:08.101314+0000","last_clean":"2024-02-12T16:54:08.101314+0000","last_became_active":"2024-02-12T16:49:39.102172+0000","last_became_peered":"2024-02-12T16:49:39.102172+0000","last_unstale":"2024-02-12T16:54:08.101314+0000","last_undegraded":"2024-02-12T16:54:08.101314+0000","last_fullsized":"2024-02-12T16:54:08.101314+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.073441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.073441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":109,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.093449+0000","last_change":"2024-02-12T16:53:49.924705+0000","last_active":"2024-02-12T16:54:08.093449+0000","last_peered":"2024-02-12T16:54:08.093449+0000","last_clean":"2024-02-12T16:54:08.093449+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:54:08.093449+0000","last_undegraded":"2024-02-12T16:54:08.093449+0000","last_fullsized":"2024-02-12T16:54:08.093449+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:53:49.924639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.924639+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102321+0000","last_change":"2024-02-12T16:52:43.089101+0000","last_active":"2024-02-12T16:54:08.102321+0000","last_peered":"2024-02-12T16:54:08.102321+0000","last_clean":"2024-02-12T16:54:08.102321+0000","last_became_active":"2024-02-12T16:49:45.218184+0000","last_became_peered":"2024-02-12T16:49:45.218184+0000","last_unstale":"2024-02-12T16:54:08.102321+0000","last_undegraded":"2024-02-12T16:54:08.102321+0000","last_fullsized":"2024-02-12T16:54:08.102321+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:43.089004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.089004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.111451+0000","last_change":"2024-02-12T16:52:16.185645+0000","last_active":"2024-02-12T16:54:08.111451+0000","last_peered":"2024-02-12T16:54:08.111451+0000","last_clean":"2024-02-12T16:54:08.111451+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:54:08.111451+0000","last_undegraded":"2024-02-12T16:54:08.111451+0000","last_fullsized":"2024-02-12T16:54:08.111451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:16.185582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:52:16.185582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":91,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112689+0000","last_change":"2024-02-12T16:52:11.233227+0000","last_active":"2024-02-12T16:54:08.112689+0000","last_peered":"2024-02-12T16:54:08.112689+0000","last_clean":"2024-02-12T16:54:08.112689+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:54:08.112689+0000","last_undegraded":"2024-02-12T16:54:08.112689+0000","last_fullsized":"2024-02-12T16:54:08.112689+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:11.233135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:52:11.233135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":73,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102247+0000","last_change":"2024-02-12T16:53:45.658415+0000","last_active":"2024-02-12T16:54:08.102247+0000","last_peered":"2024-02-12T16:54:08.102247+0000","last_clean":"2024-02-12T16:54:08.102247+0000","last_became_active":"2024-02-12T16:52:20.563447+0000","last_became_peered":"2024-02-12T16:52:20.563447+0000","last_unstale":"2024-02-12T16:54:08.102247+0000","last_undegraded":"2024-02-12T16:54:08.102247+0000","last_fullsized":"2024-02-12T16:54:08.102247+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:53:45.658292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.658292+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.104493+0000","last_change":"2024-02-12T16:53:51.204401+0000","last_active":"2024-02-12T16:54:08.104493+0000","last_peered":"2024-02-12T16:54:08.104493+0000","last_clean":"2024-02-12T16:54:08.104493+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:54:08.104493+0000","last_undegraded":"2024-02-12T16:54:08.104493+0000","last_fullsized":"2024-02-12T16:54:08.104493+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103275+0000","last_change":"2024-02-12T16:52:01.948362+0000","last_active":"2024-02-12T16:54:08.103275+0000","last_peered":"2024-02-12T16:54:08.103275+0000","last_clean":"2024-02-12T16:54:08.103275+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:54:08.103275+0000","last_undegraded":"2024-02-12T16:54:08.103275+0000","last_fullsized":"2024-02-12T16:54:08.103275+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:01.948286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:52:01.948286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":91,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112350+0000","last_change":"2024-02-12T16:52:10.220443+0000","last_active":"2024-02-12T16:54:08.112350+0000","last_peered":"2024-02-12T16:54:08.112350+0000","last_clean":"2024-02-12T16:54:08.112350+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:54:08.112350+0000","last_undegraded":"2024-02-12T16:54:08.112350+0000","last_fullsized":"2024-02-12T16:54:08.112350+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:10.220358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:10.220358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":79,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.101387+0000","last_change":"2024-02-12T16:52:48.029448+0000","last_active":"2024-02-12T16:54:08.101387+0000","last_peered":"2024-02-12T16:54:08.101387+0000","last_clean":"2024-02-12T16:54:08.101387+0000","last_became_active":"2024-02-12T16:52:20.561538+0000","last_became_peered":"2024-02-12T16:52:20.561538+0000","last_unstale":"2024-02-12T16:54:08.101387+0000","last_undegraded":"2024-02-12T16:54:08.101387+0000","last_fullsized":"2024-02-12T16:54:08.101387+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:48.029049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.029049+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":86,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103201+0000","last_change":"2024-02-12T16:52:58.964154+0000","last_active":"2024-02-12T16:54:08.103201+0000","last_peered":"2024-02-12T16:54:08.103201+0000","last_clean":"2024-02-12T16:54:08.103201+0000","last_became_active":"2024-02-12T16:49:45.221559+0000","last_became_peered":"2024-02-12T16:49:45.221559+0000","last_unstale":"2024-02-12T16:54:08.103201+0000","last_undegraded":"2024-02-12T16:54:08.103201+0000","last_fullsized":"2024-02-12T16:54:08.103201+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:58.964060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.964060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":51,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.100433+0000","last_change":"2024-02-12T16:52:40.022799+0000","last_active":"2024-02-12T16:54:08.100433+0000","last_peered":"2024-02-12T16:54:08.100433+0000","last_clean":"2024-02-12T16:54:08.100433+0000","last_became_active":"2024-02-12T16:52:20.565295+0000","last_became_peered":"2024-02-12T16:52:20.565295+0000","last_unstale":"2024-02-12T16:54:08.100433+0000","last_undegraded":"2024-02-12T16:54:08.100433+0000","last_fullsized":"2024-02-12T16:54:08.100433+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:40.022594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.022594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":92,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102323+0000","last_change":"2024-02-12T16:53:03.947856+0000","last_active":"2024-02-12T16:54:08.102323+0000","last_peered":"2024-02-12T16:54:08.102323+0000","last_clean":"2024-02-12T16:54:08.102323+0000","last_became_active":"2024-02-12T16:49:39.105438+0000","last_became_peered":"2024-02-12T16:49:39.105438+0000","last_unstale":"2024-02-12T16:54:08.102323+0000","last_undegraded":"2024-02-12T16:54:08.102323+0000","last_fullsized":"2024-02-12T16:54:08.102323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:03.947783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:03.947783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":113,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.100913+0000","last_change":"2024-02-12T16:53:49.624190+0000","last_active":"2024-02-12T16:54:08.100913+0000","last_peered":"2024-02-12T16:54:08.100913+0000","last_clean":"2024-02-12T16:54:08.100913+0000","last_became_active":"2024-02-12T16:49:41.181980+0000","last_became_peered":"2024-02-12T16:49:41.181980+0000","last_unstale":"2024-02-12T16:54:08.100913+0000","last_undegraded":"2024-02-12T16:54:08.100913+0000","last_fullsized":"2024-02-12T16:54:08.100913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:49.624070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.624070+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":84,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.094912+0000","last_change":"2024-02-12T16:52:45.835759+0000","last_active":"2024-02-12T16:54:08.094912+0000","last_peered":"2024-02-12T16:54:08.094912+0000","last_clean":"2024-02-12T16:54:08.094912+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:54:08.094912+0000","last_undegraded":"2024-02-12T16:54:08.094912+0000","last_fullsized":"2024-02-12T16:54:08.094912+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:45.835664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:52:45.835664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099075+0000","last_change":"2024-02-12T16:53:43.855961+0000","last_active":"2024-02-12T16:54:08.099075+0000","last_peered":"2024-02-12T16:54:08.099075+0000","last_clean":"2024-02-12T16:54:08.099075+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:54:08.099075+0000","last_undegraded":"2024-02-12T16:54:08.099075+0000","last_fullsized":"2024-02-12T16:54:08.099075+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:43.855860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.855860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":97,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112329+0000","last_change":"2024-02-12T16:52:58.691294+0000","last_active":"2024-02-12T16:54:08.112329+0000","last_peered":"2024-02-12T16:54:08.112329+0000","last_clean":"2024-02-12T16:54:08.112329+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:54:08.112329+0000","last_undegraded":"2024-02-12T16:54:08.112329+0000","last_fullsized":"2024-02-12T16:54:08.112329+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:58.691233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.691233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":125,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.109151+0000","last_change":"2024-02-12T16:53:48.468620+0000","last_active":"2024-02-12T16:54:08.109151+0000","last_peered":"2024-02-12T16:54:08.109151+0000","last_clean":"2024-02-12T16:54:08.109151+0000","last_became_active":"2024-02-12T16:49:41.180319+0000","last_became_peered":"2024-02-12T16:49:41.180319+0000","last_unstale":"2024-02-12T16:54:08.109151+0000","last_undegraded":"2024-02-12T16:54:08.109151+0000","last_fullsized":"2024-02-12T16:54:08.109151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:53:48.468547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.468547+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.098871+0000","last_change":"2024-02-12T16:52:59.833591+0000","last_active":"2024-02-12T16:54:08.098871+0000","last_peered":"2024-02-12T16:54:08.098871+0000","last_clean":"2024-02-12T16:54:08.098871+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:54:08.098871+0000","last_undegraded":"2024-02-12T16:54:08.098871+0000","last_fullsized":"2024-02-12T16:54:08.098871+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:59.833484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.833484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.104683+0000","last_change":"2024-02-12T16:53:04.085929+0000","last_active":"2024-02-12T16:54:08.104683+0000","last_peered":"2024-02-12T16:54:08.104683+0000","last_clean":"2024-02-12T16:54:08.104683+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:54:08.104683+0000","last_undegraded":"2024-02-12T16:54:08.104683+0000","last_fullsized":"2024-02-12T16:54:08.104683+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:04.085854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.085854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":92,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.101152+0000","last_change":"2024-02-12T16:53:01.970821+0000","last_active":"2024-02-12T16:54:08.101152+0000","last_peered":"2024-02-12T16:54:08.101152+0000","last_clean":"2024-02-12T16:54:08.101152+0000","last_became_active":"2024-02-12T16:49:39.101492+0000","last_became_peered":"2024-02-12T16:49:39.101492+0000","last_unstale":"2024-02-12T16:54:08.101152+0000","last_undegraded":"2024-02-12T16:54:08.101152+0000","last_fullsized":"2024-02-12T16:54:08.101152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.970730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.970730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":109,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.113073+0000","last_change":"2024-02-12T16:52:52.604196+0000","last_active":"2024-02-12T16:54:08.113073+0000","last_peered":"2024-02-12T16:54:08.113073+0000","last_clean":"2024-02-12T16:54:08.113073+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:54:08.113073+0000","last_undegraded":"2024-02-12T16:54:08.113073+0000","last_fullsized":"2024-02-12T16:54:08.113073+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:52:52.604107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.604107+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112623+0000","last_change":"2024-02-12T16:53:04.169987+0000","last_active":"2024-02-12T16:54:08.112623+0000","last_peered":"2024-02-12T16:54:08.112623+0000","last_clean":"2024-02-12T16:54:08.112623+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:54:08.112623+0000","last_undegraded":"2024-02-12T16:54:08.112623+0000","last_fullsized":"2024-02-12T16:54:08.112623+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:53:04.169892+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.169892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090476+0000","last_change":"2024-02-12T16:52:44.602751+0000","last_active":"2024-02-12T16:54:12.090476+0000","last_peered":"2024-02-12T16:54:12.090476+0000","last_clean":"2024-02-12T16:54:12.090476+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:54:12.090476+0000","last_undegraded":"2024-02-12T16:54:12.090476+0000","last_fullsized":"2024-02-12T16:54:12.090476+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:44.602656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.602656+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102375+0000","last_change":"2024-02-12T16:52:47.997796+0000","last_active":"2024-02-12T16:54:08.102375+0000","last_peered":"2024-02-12T16:54:08.102375+0000","last_clean":"2024-02-12T16:54:08.102375+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:54:08.102375+0000","last_undegraded":"2024-02-12T16:54:08.102375+0000","last_fullsized":"2024-02-12T16:54:08.102375+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:47.997708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.997708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":104,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.104637+0000","last_change":"2024-02-12T16:51:57.693428+0000","last_active":"2024-02-12T16:54:08.104637+0000","last_peered":"2024-02-12T16:54:08.104637+0000","last_clean":"2024-02-12T16:54:08.104637+0000","last_became_active":"2024-02-12T16:49:41.669589+0000","last_became_peered":"2024-02-12T16:49:41.669589+0000","last_unstale":"2024-02-12T16:54:08.104637+0000","last_undegraded":"2024-02-12T16:54:08.104637+0000","last_fullsized":"2024-02-12T16:54:08.104637+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:57.693352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:57.693352+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":86,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102649+0000","last_change":"2024-02-12T16:54:00.665342+0000","last_active":"2024-02-12T16:54:08.102649+0000","last_peered":"2024-02-12T16:54:08.102649+0000","last_clean":"2024-02-12T16:54:08.102649+0000","last_became_active":"2024-02-12T16:49:45.222341+0000","last_became_peered":"2024-02-12T16:49:45.222341+0000","last_unstale":"2024-02-12T16:54:08.102649+0000","last_undegraded":"2024-02-12T16:54:08.102649+0000","last_fullsized":"2024-02-12T16:54:08.102649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.665249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.665249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.113643+0000","last_change":"2024-02-12T16:53:05.141324+0000","last_active":"2024-02-12T16:54:08.113643+0000","last_peered":"2024-02-12T16:54:08.113643+0000","last_clean":"2024-02-12T16:54:08.113643+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:54:08.113643+0000","last_undegraded":"2024-02-12T16:54:08.113643+0000","last_fullsized":"2024-02-12T16:54:08.113643+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":92,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112848+0000","last_change":"2024-02-12T16:52:51.634372+0000","last_active":"2024-02-12T16:54:08.112848+0000","last_peered":"2024-02-12T16:54:08.112848+0000","last_clean":"2024-02-12T16:54:08.112848+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:54:08.112848+0000","last_undegraded":"2024-02-12T16:54:08.112848+0000","last_fullsized":"2024-02-12T16:54:08.112848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:51.634281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.634281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":109,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.101472+0000","last_change":"2024-02-12T16:51:57.052497+0000","last_active":"2024-02-12T16:54:08.101472+0000","last_peered":"2024-02-12T16:54:08.101472+0000","last_clean":"2024-02-12T16:54:08.101472+0000","last_became_active":"2024-02-12T16:49:41.669704+0000","last_became_peered":"2024-02-12T16:49:41.669704+0000","last_unstale":"2024-02-12T16:54:08.101472+0000","last_undegraded":"2024-02-12T16:54:08.101472+0000","last_fullsized":"2024-02-12T16:54:08.101472+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:57.052428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:57.052428+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":85,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.111323+0000","last_change":"2024-02-12T16:52:55.241717+0000","last_active":"2024-02-12T16:54:08.111323+0000","last_peered":"2024-02-12T16:54:08.111323+0000","last_clean":"2024-02-12T16:54:08.111323+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:54:08.111323+0000","last_undegraded":"2024-02-12T16:54:08.111323+0000","last_fullsized":"2024-02-12T16:54:08.111323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.241622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.241622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.111642+0000","last_change":"2024-02-12T16:53:48.719406+0000","last_active":"2024-02-12T16:54:08.111642+0000","last_peered":"2024-02-12T16:54:08.111642+0000","last_clean":"2024-02-12T16:54:08.111642+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:54:08.111642+0000","last_undegraded":"2024-02-12T16:54:08.111642+0000","last_fullsized":"2024-02-12T16:54:08.111642+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.719344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.719344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103610+0000","last_change":"2024-02-12T16:52:49.986507+0000","last_active":"2024-02-12T16:54:08.103610+0000","last_peered":"2024-02-12T16:54:08.103610+0000","last_clean":"2024-02-12T16:54:08.103610+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:54:08.103610+0000","last_undegraded":"2024-02-12T16:54:08.103610+0000","last_fullsized":"2024-02-12T16:54:08.103610+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.986403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.986403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":96,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103389+0000","last_change":"2024-02-12T16:54:01.670202+0000","last_active":"2024-02-12T16:54:08.103389+0000","last_peered":"2024-02-12T16:54:08.103389+0000","last_clean":"2024-02-12T16:54:08.103389+0000","last_became_active":"2024-02-12T16:49:41.184044+0000","last_became_peered":"2024-02-12T16:49:41.184044+0000","last_unstale":"2024-02-12T16:54:08.103389+0000","last_undegraded":"2024-02-12T16:54:08.103389+0000","last_fullsized":"2024-02-12T16:54:08.103389+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":81,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.101018+0000","last_change":"2024-02-12T16:51:42.040901+0000","last_active":"2024-02-12T16:54:08.101018+0000","last_peered":"2024-02-12T16:54:08.101018+0000","last_clean":"2024-02-12T16:54:08.101018+0000","last_became_active":"2024-02-12T16:49:45.220469+0000","last_became_peered":"2024-02-12T16:49:45.220469+0000","last_unstale":"2024-02-12T16:54:08.101018+0000","last_undegraded":"2024-02-12T16:54:08.101018+0000","last_fullsized":"2024-02-12T16:54:08.101018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:51:42.040828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":94,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099384+0000","last_change":"2024-02-12T16:54:02.785572+0000","last_active":"2024-02-12T16:54:08.099384+0000","last_peered":"2024-02-12T16:54:08.099384+0000","last_clean":"2024-02-12T16:54:08.099384+0000","last_became_active":"2024-02-12T16:49:43.204615+0000","last_became_peered":"2024-02-12T16:49:43.204615+0000","last_unstale":"2024-02-12T16:54:08.099384+0000","last_undegraded":"2024-02-12T16:54:08.099384+0000","last_fullsized":"2024-02-12T16:54:08.099384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.785477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.785477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":98,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099122+0000","last_change":"2024-02-12T16:54:01.773110+0000","last_active":"2024-02-12T16:54:08.099122+0000","last_peered":"2024-02-12T16:54:08.099122+0000","last_clean":"2024-02-12T16:54:08.099122+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:54:08.099122+0000","last_undegraded":"2024-02-12T16:54:08.099122+0000","last_fullsized":"2024-02-12T16:54:08.099122+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":109,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.094345+0000","last_change":"2024-02-12T16:53:53.872376+0000","last_active":"2024-02-12T16:54:08.094345+0000","last_peered":"2024-02-12T16:54:08.094345+0000","last_clean":"2024-02-12T16:54:08.094345+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:54:08.094345+0000","last_undegraded":"2024-02-12T16:54:08.094345+0000","last_fullsized":"2024-02-12T16:54:08.094345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:52:38.896872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:38.896872+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":80,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103983+0000","last_change":"2024-02-12T16:51:28.886399+0000","last_active":"2024-02-12T16:54:08.103983+0000","last_peered":"2024-02-12T16:54:08.103983+0000","last_clean":"2024-02-12T16:54:08.103983+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:54:08.103983+0000","last_undegraded":"2024-02-12T16:54:08.103983+0000","last_fullsized":"2024-02-12T16:54:08.103983+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.886321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":83,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102409+0000","last_change":"2024-02-12T16:51:39.040277+0000","last_active":"2024-02-12T16:54:08.102409+0000","last_peered":"2024-02-12T16:54:08.102409+0000","last_clean":"2024-02-12T16:54:08.102409+0000","last_became_active":"2024-02-12T16:49:43.208343+0000","last_became_peered":"2024-02-12T16:49:43.208343+0000","last_unstale":"2024-02-12T16:54:08.102409+0000","last_undegraded":"2024-02-12T16:54:08.102409+0000","last_fullsized":"2024-02-12T16:54:08.102409+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.040203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.040203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":92,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.111759+0000","last_change":"2024-02-12T16:52:48.590123+0000","last_active":"2024-02-12T16:54:08.111759+0000","last_peered":"2024-02-12T16:54:08.111759+0000","last_clean":"2024-02-12T16:54:08.111759+0000","last_became_active":"2024-02-12T16:49:39.107471+0000","last_became_peered":"2024-02-12T16:49:39.107471+0000","last_unstale":"2024-02-12T16:54:08.111759+0000","last_undegraded":"2024-02-12T16:54:08.111759+0000","last_fullsized":"2024-02-12T16:54:08.111759+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.590060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.590060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":111,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099292+0000","last_change":"2024-02-12T16:52:06.722431+0000","last_active":"2024-02-12T16:54:08.099292+0000","last_peered":"2024-02-12T16:54:08.099292+0000","last_clean":"2024-02-12T16:54:08.099292+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:54:08.099292+0000","last_undegraded":"2024-02-12T16:54:08.099292+0000","last_fullsized":"2024-02-12T16:54:08.099292+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:52:06.722345+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:52:06.722345+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.110100+0000","last_change":"2024-02-12T16:52:59.296102+0000","last_active":"2024-02-12T16:54:08.110100+0000","last_peered":"2024-02-12T16:54:08.110100+0000","last_clean":"2024-02-12T16:54:08.110100+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T16:54:08.110100+0000","last_undegraded":"2024-02-12T16:54:08.110100+0000","last_fullsized":"2024-02-12T16:54:08.110100+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:59.296023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.296023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091827+0000","last_change":"2024-02-12T16:51:44.091513+0000","last_active":"2024-02-12T16:54:12.091827+0000","last_peered":"2024-02-12T16:54:12.091827+0000","last_clean":"2024-02-12T16:54:12.091827+0000","last_became_active":"2024-02-12T16:49:43.206068+0000","last_became_peered":"2024-02-12T16:49:43.206068+0000","last_unstale":"2024-02-12T16:54:12.091827+0000","last_undegraded":"2024-02-12T16:54:12.091827+0000","last_fullsized":"2024-02-12T16:54:12.091827+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:44.091436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:44.091436+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":86,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.105732+0000","last_change":"2024-02-12T16:51:25.913214+0000","last_active":"2024-02-12T16:54:08.105732+0000","last_peered":"2024-02-12T16:54:08.105732+0000","last_clean":"2024-02-12T16:54:08.105732+0000","last_became_active":"2024-02-12T16:49:39.103401+0000","last_became_peered":"2024-02-12T16:49:39.103401+0000","last_unstale":"2024-02-12T16:54:08.105732+0000","last_undegraded":"2024-02-12T16:54:08.105732+0000","last_fullsized":"2024-02-12T16:54:08.105732+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:25.913145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.913145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":102,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103616+0000","last_change":"2024-02-12T16:51:25.028420+0000","last_active":"2024-02-12T16:54:08.103616+0000","last_peered":"2024-02-12T16:54:08.103616+0000","last_clean":"2024-02-12T16:54:08.103616+0000","last_became_active":"2024-02-12T16:49:41.669427+0000","last_became_peered":"2024-02-12T16:49:41.669427+0000","last_unstale":"2024-02-12T16:54:08.103616+0000","last_undegraded":"2024-02-12T16:54:08.103616+0000","last_fullsized":"2024-02-12T16:54:08.103616+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:25.028339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.028339+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":85,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.104300+0000","last_change":"2024-02-12T16:52:55.105623+0000","last_active":"2024-02-12T16:54:08.104300+0000","last_peered":"2024-02-12T16:54:08.104300+0000","last_clean":"2024-02-12T16:54:08.104300+0000","last_became_active":"2024-02-12T16:49:45.212245+0000","last_became_peered":"2024-02-12T16:49:45.212245+0000","last_unstale":"2024-02-12T16:54:08.104300+0000","last_undegraded":"2024-02-12T16:54:08.104300+0000","last_fullsized":"2024-02-12T16:54:08.104300+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.105539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.105539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091887+0000","last_change":"2024-02-12T16:54:00.112924+0000","last_active":"2024-02-12T16:54:12.091887+0000","last_peered":"2024-02-12T16:54:12.091887+0000","last_clean":"2024-02-12T16:54:12.091887+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:54:12.091887+0000","last_undegraded":"2024-02-12T16:54:12.091887+0000","last_fullsized":"2024-02-12T16:54:12.091887+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":88,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102490+0000","last_change":"2024-02-12T16:51:20.103380+0000","last_active":"2024-02-12T16:54:08.102490+0000","last_peered":"2024-02-12T16:54:08.102490+0000","last_clean":"2024-02-12T16:54:08.102490+0000","last_became_active":"2024-02-12T16:49:39.104936+0000","last_became_peered":"2024-02-12T16:49:39.104936+0000","last_unstale":"2024-02-12T16:54:08.102490+0000","last_undegraded":"2024-02-12T16:54:08.102490+0000","last_fullsized":"2024-02-12T16:54:08.102490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.103297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.103297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":126,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.100302+0000","last_change":"2024-02-12T16:54:03.616828+0000","last_active":"2024-02-12T16:54:08.100302+0000","last_peered":"2024-02-12T16:54:08.100302+0000","last_clean":"2024-02-12T16:54:08.100302+0000","last_became_active":"2024-02-12T16:51:04.116639+0000","last_became_peered":"2024-02-12T16:51:04.116639+0000","last_unstale":"2024-02-12T16:54:08.100302+0000","last_undegraded":"2024-02-12T16:54:08.100302+0000","last_fullsized":"2024-02-12T16:54:08.100302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:03.616717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.616717+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112624+0000","last_change":"2024-02-12T16:53:51.532654+0000","last_active":"2024-02-12T16:54:08.112624+0000","last_peered":"2024-02-12T16:54:08.112624+0000","last_clean":"2024-02-12T16:54:08.112624+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:54:08.112624+0000","last_undegraded":"2024-02-12T16:54:08.112624+0000","last_fullsized":"2024-02-12T16:54:08.112624+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103499+0000","last_change":"2024-02-12T16:52:15.844557+0000","last_active":"2024-02-12T16:54:08.103499+0000","last_peered":"2024-02-12T16:54:08.103499+0000","last_clean":"2024-02-12T16:54:08.103499+0000","last_became_active":"2024-02-12T16:49:43.201595+0000","last_became_peered":"2024-02-12T16:49:43.201595+0000","last_unstale":"2024-02-12T16:54:08.103499+0000","last_undegraded":"2024-02-12T16:54:08.103499+0000","last_fullsized":"2024-02-12T16:54:08.103499+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.844487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.844487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":92,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.111713+0000","last_change":"2024-02-12T16:52:43.579914+0000","last_active":"2024-02-12T16:54:08.111713+0000","last_peered":"2024-02-12T16:54:08.111713+0000","last_clean":"2024-02-12T16:54:08.111713+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:54:08.111713+0000","last_undegraded":"2024-02-12T16:54:08.111713+0000","last_fullsized":"2024-02-12T16:54:08.111713+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:43.579811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.579811+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":99,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103277+0000","last_change":"2024-02-12T16:53:57.641513+0000","last_active":"2024-02-12T16:54:08.103277+0000","last_peered":"2024-02-12T16:54:08.103277+0000","last_clean":"2024-02-12T16:54:08.103277+0000","last_became_active":"2024-02-12T16:49:41.669651+0000","last_became_peered":"2024-02-12T16:49:41.669651+0000","last_unstale":"2024-02-12T16:54:08.103277+0000","last_undegraded":"2024-02-12T16:54:08.103277+0000","last_fullsized":"2024-02-12T16:54:08.103277+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.108987+0000","last_change":"2024-02-12T16:52:39.278662+0000","last_active":"2024-02-12T16:54:08.108987+0000","last_peered":"2024-02-12T16:54:08.108987+0000","last_clean":"2024-02-12T16:54:08.108987+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:54:08.108987+0000","last_undegraded":"2024-02-12T16:54:08.108987+0000","last_fullsized":"2024-02-12T16:54:08.108987+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:39.278592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:39.278592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":90,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103349+0000","last_change":"2024-02-12T16:53:41.740985+0000","last_active":"2024-02-12T16:54:08.103349+0000","last_peered":"2024-02-12T16:54:08.103349+0000","last_clean":"2024-02-12T16:54:08.103349+0000","last_became_active":"2024-02-12T16:49:43.207454+0000","last_became_peered":"2024-02-12T16:49:43.207454+0000","last_unstale":"2024-02-12T16:54:08.103349+0000","last_undegraded":"2024-02-12T16:54:08.103349+0000","last_fullsized":"2024-02-12T16:54:08.103349+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:41.740892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:53:41.740892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.108742+0000","last_change":"2024-02-12T16:52:53.330290+0000","last_active":"2024-02-12T16:54:08.108742+0000","last_peered":"2024-02-12T16:54:08.108742+0000","last_clean":"2024-02-12T16:54:08.108742+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:54:08.108742+0000","last_undegraded":"2024-02-12T16:54:08.108742+0000","last_fullsized":"2024-02-12T16:54:08.108742+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:53.330194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:53.330194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":109,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.104536+0000","last_change":"2024-02-12T16:54:02.181218+0000","last_active":"2024-02-12T16:54:08.104536+0000","last_peered":"2024-02-12T16:54:08.104536+0000","last_clean":"2024-02-12T16:54:08.104536+0000","last_became_active":"2024-02-12T16:49:41.668665+0000","last_became_peered":"2024-02-12T16:49:41.668665+0000","last_unstale":"2024-02-12T16:54:08.104536+0000","last_undegraded":"2024-02-12T16:54:08.104536+0000","last_fullsized":"2024-02-12T16:54:08.104536+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"132'7505","reported_seq":7552,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.315312+0000","last_change":"2024-02-12T16:52:16.552919+0000","last_active":"2024-02-12T16:54:13.315312+0000","last_peered":"2024-02-12T16:54:13.315312+0000","last_clean":"2024-02-12T16:54:13.315312+0000","last_became_active":"2024-02-12T16:52:16.552631+0000","last_became_peered":"2024-02-12T16:52:16.552631+0000","last_unstale":"2024-02-12T16:54:13.315312+0000","last_undegraded":"2024-02-12T16:54:13.315312+0000","last_fullsized":"2024-02-12T16:54:13.315312+0000","mapping_epoch":93,"log_start":"111'2800","ondisk_log_start":"111'2800","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4705,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":412811264,"num_objects":6299,"num_object_clones":0,"num_object_copies":18897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":441728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":81,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102485+0000","last_change":"2024-02-12T16:51:33.032870+0000","last_active":"2024-02-12T16:54:08.102485+0000","last_peered":"2024-02-12T16:54:08.102485+0000","last_clean":"2024-02-12T16:54:08.102485+0000","last_became_active":"2024-02-12T16:49:45.220328+0000","last_became_peered":"2024-02-12T16:49:45.220328+0000","last_unstale":"2024-02-12T16:54:08.102485+0000","last_undegraded":"2024-02-12T16:54:08.102485+0000","last_fullsized":"2024-02-12T16:54:08.102485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:33.032784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:33.032784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":114,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.095371+0000","last_change":"2024-02-12T16:53:48.921052+0000","last_active":"2024-02-12T16:54:08.095371+0000","last_peered":"2024-02-12T16:54:08.095371+0000","last_clean":"2024-02-12T16:54:08.095371+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:54:08.095371+0000","last_undegraded":"2024-02-12T16:54:08.095371+0000","last_fullsized":"2024-02-12T16:54:08.095371+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:53:48.920976+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.920976+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":86,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.098871+0000","last_change":"2024-02-12T16:52:44.868547+0000","last_active":"2024-02-12T16:54:08.098871+0000","last_peered":"2024-02-12T16:54:08.098871+0000","last_clean":"2024-02-12T16:54:08.098871+0000","last_became_active":"2024-02-12T16:52:20.560975+0000","last_became_peered":"2024-02-12T16:52:20.560975+0000","last_unstale":"2024-02-12T16:54:08.098871+0000","last_undegraded":"2024-02-12T16:54:08.098871+0000","last_fullsized":"2024-02-12T16:54:08.098871+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:52:44.868438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.868438+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.108832+0000","last_change":"2024-02-12T16:52:48.411110+0000","last_active":"2024-02-12T16:54:08.108832+0000","last_peered":"2024-02-12T16:54:08.108832+0000","last_clean":"2024-02-12T16:54:08.108832+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:54:08.108832+0000","last_undegraded":"2024-02-12T16:54:08.108832+0000","last_fullsized":"2024-02-12T16:54:08.108832+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.411008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.411008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"132'7469","reported_seq":7515,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.354219+0000","last_change":"2024-02-12T16:52:20.564464+0000","last_active":"2024-02-12T16:54:13.354219+0000","last_peered":"2024-02-12T16:54:13.354219+0000","last_clean":"2024-02-12T16:54:13.354219+0000","last_became_active":"2024-02-12T16:52:20.564223+0000","last_became_peered":"2024-02-12T16:52:20.564223+0000","last_unstale":"2024-02-12T16:54:13.354219+0000","last_undegraded":"2024-02-12T16:54:13.354219+0000","last_fullsized":"2024-02-12T16:54:13.354219+0000","mapping_epoch":97,"log_start":"110'2700","ondisk_log_start":"110'2700","created":93,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4769,"ondisk_log_size":4769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":409010176,"num_objects":6241,"num_object_clones":0,"num_object_copies":18723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7469,"num_write_kb":438720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":85,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.104068+0000","last_change":"2024-02-12T16:53:42.249070+0000","last_active":"2024-02-12T16:54:08.104068+0000","last_peered":"2024-02-12T16:54:08.104068+0000","last_clean":"2024-02-12T16:54:08.104068+0000","last_became_active":"2024-02-12T16:49:45.217626+0000","last_became_peered":"2024-02-12T16:49:45.217626+0000","last_unstale":"2024-02-12T16:54:08.104068+0000","last_undegraded":"2024-02-12T16:54:08.104068+0000","last_fullsized":"2024-02-12T16:54:08.104068+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.248970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.248970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103755+0000","last_change":"2024-02-12T16:53:40.916900+0000","last_active":"2024-02-12T16:54:08.103755+0000","last_peered":"2024-02-12T16:54:08.103755+0000","last_clean":"2024-02-12T16:54:08.103755+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:54:08.103755+0000","last_undegraded":"2024-02-12T16:54:08.103755+0000","last_fullsized":"2024-02-12T16:54:08.103755+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:40.916831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:40.916831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":77,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.110977+0000","last_change":"2024-02-12T16:52:11.419489+0000","last_active":"2024-02-12T16:54:08.110977+0000","last_peered":"2024-02-12T16:54:08.110977+0000","last_clean":"2024-02-12T16:54:08.110977+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:54:08.110977+0000","last_undegraded":"2024-02-12T16:54:08.110977+0000","last_fullsized":"2024-02-12T16:54:08.110977+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:52:11.419411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:11.419411+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.084128+0000","last_change":"2024-02-12T16:53:59.173126+0000","last_active":"2024-02-12T16:54:12.084128+0000","last_peered":"2024-02-12T16:54:12.084128+0000","last_clean":"2024-02-12T16:54:12.084128+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:54:12.084128+0000","last_undegraded":"2024-02-12T16:54:12.084128+0000","last_fullsized":"2024-02-12T16:54:12.084128+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"132'7626","reported_seq":7678,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.281908+0000","last_change":"2024-02-12T16:53:55.684404+0000","last_active":"2024-02-12T16:54:13.281908+0000","last_peered":"2024-02-12T16:54:13.281908+0000","last_clean":"2024-02-12T16:54:13.281908+0000","last_became_active":"2024-02-12T16:52:16.557444+0000","last_became_peered":"2024-02-12T16:52:16.557444+0000","last_unstale":"2024-02-12T16:54:13.281908+0000","last_undegraded":"2024-02-12T16:54:13.281908+0000","last_fullsized":"2024-02-12T16:54:13.281908+0000","mapping_epoch":93,"log_start":"105'1500","ondisk_log_start":"105'1500","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'6594","last_scrub_stamp":"2024-02-12T16:53:55.684320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:53:55.684320+0000","log_size":6126,"ondisk_log_size":6126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":417988608,"num_objects":6378,"num_object_clones":0,"num_object_copies":19134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7626,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":86,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.101410+0000","last_change":"2024-02-12T16:53:56.624041+0000","last_active":"2024-02-12T16:54:08.101410+0000","last_peered":"2024-02-12T16:54:08.101410+0000","last_clean":"2024-02-12T16:54:08.101410+0000","last_became_active":"2024-02-12T16:49:45.215520+0000","last_became_peered":"2024-02-12T16:49:45.215520+0000","last_unstale":"2024-02-12T16:54:08.101410+0000","last_undegraded":"2024-02-12T16:54:08.101410+0000","last_fullsized":"2024-02-12T16:54:08.101410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:56.623922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.623922+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102527+0000","last_change":"2024-02-12T16:52:06.907569+0000","last_active":"2024-02-12T16:54:08.102527+0000","last_peered":"2024-02-12T16:54:08.102527+0000","last_clean":"2024-02-12T16:54:08.102527+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:54:08.102527+0000","last_undegraded":"2024-02-12T16:54:08.102527+0000","last_fullsized":"2024-02-12T16:54:08.102527+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:06.907496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:06.907496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":194,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078656+0000","last_change":"2024-02-12T16:53:59.751101+0000","last_active":"2024-02-12T16:54:12.078656+0000","last_peered":"2024-02-12T16:54:12.078656+0000","last_clean":"2024-02-12T16:54:12.078656+0000","last_became_active":"2024-02-12T16:49:41.179610+0000","last_became_peered":"2024-02-12T16:49:41.179610+0000","last_unstale":"2024-02-12T16:54:12.078656+0000","last_undegraded":"2024-02-12T16:54:12.078656+0000","last_fullsized":"2024-02-12T16:54:12.078656+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-12T16:53:59.751031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.751031+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.107112+0000","last_change":"2024-02-12T16:53:01.103805+0000","last_active":"2024-02-12T16:54:08.107112+0000","last_peered":"2024-02-12T16:54:08.107112+0000","last_clean":"2024-02-12T16:54:08.107112+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:54:08.107112+0000","last_undegraded":"2024-02-12T16:54:08.107112+0000","last_fullsized":"2024-02-12T16:54:08.107112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.103703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.103703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"132'7434","reported_seq":7480,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.328039+0000","last_change":"2024-02-12T16:52:16.558183+0000","last_active":"2024-02-12T16:54:13.328039+0000","last_peered":"2024-02-12T16:54:13.328039+0000","last_clean":"2024-02-12T16:54:13.328039+0000","last_became_active":"2024-02-12T16:52:16.557890+0000","last_became_peered":"2024-02-12T16:52:16.557890+0000","last_unstale":"2024-02-12T16:54:13.328039+0000","last_undegraded":"2024-02-12T16:54:13.328039+0000","last_fullsized":"2024-02-12T16:54:13.328039+0000","mapping_epoch":93,"log_start":"107'1900","ondisk_log_start":"107'1900","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5534,"ondisk_log_size":5534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":413401088,"num_objects":6308,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7434,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112802+0000","last_change":"2024-02-12T16:54:00.551098+0000","last_active":"2024-02-12T16:54:08.112802+0000","last_peered":"2024-02-12T16:54:08.112802+0000","last_clean":"2024-02-12T16:54:08.112802+0000","last_became_active":"2024-02-12T16:49:45.221460+0000","last_became_peered":"2024-02-12T16:49:45.221460+0000","last_unstale":"2024-02-12T16:54:08.112802+0000","last_undegraded":"2024-02-12T16:54:08.112802+0000","last_fullsized":"2024-02-12T16:54:08.112802+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":363,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.079716+0000","last_change":"2024-02-12T16:52:02.743093+0000","last_active":"2024-02-12T16:54:12.079716+0000","last_peered":"2024-02-12T16:54:12.079716+0000","last_clean":"2024-02-12T16:54:12.079716+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:54:12.079716+0000","last_undegraded":"2024-02-12T16:54:12.079716+0000","last_fullsized":"2024-02-12T16:54:12.079716+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:52:02.742977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:02.742977+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":267,"num_read_kb":262,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.108716+0000","last_change":"2024-02-12T16:52:40.271277+0000","last_active":"2024-02-12T16:54:08.108716+0000","last_peered":"2024-02-12T16:54:08.108716+0000","last_clean":"2024-02-12T16:54:08.108716+0000","last_became_active":"2024-02-12T16:49:39.105685+0000","last_became_peered":"2024-02-12T16:49:39.105685+0000","last_unstale":"2024-02-12T16:54:08.108716+0000","last_undegraded":"2024-02-12T16:54:08.108716+0000","last_fullsized":"2024-02-12T16:54:08.108716+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:40.271192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.271192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":123,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099345+0000","last_change":"2024-02-12T16:53:04.923477+0000","last_active":"2024-02-12T16:54:08.099345+0000","last_peered":"2024-02-12T16:54:08.099345+0000","last_clean":"2024-02-12T16:54:08.099345+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:54:08.099345+0000","last_undegraded":"2024-02-12T16:54:08.099345+0000","last_fullsized":"2024-02-12T16:54:08.099345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.923375+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.110523+0000","last_change":"2024-02-12T16:52:45.314737+0000","last_active":"2024-02-12T16:54:08.110523+0000","last_peered":"2024-02-12T16:54:08.110523+0000","last_clean":"2024-02-12T16:54:08.110523+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:54:08.110523+0000","last_undegraded":"2024-02-12T16:54:08.110523+0000","last_fullsized":"2024-02-12T16:54:08.110523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:45.314637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:45.314637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"132'7522","reported_seq":7568,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.334078+0000","last_change":"2024-02-12T16:52:16.560931+0000","last_active":"2024-02-12T16:54:13.334078+0000","last_peered":"2024-02-12T16:54:13.334078+0000","last_clean":"2024-02-12T16:54:13.334078+0000","last_became_active":"2024-02-12T16:52:16.560633+0000","last_became_peered":"2024-02-12T16:52:16.560633+0000","last_unstale":"2024-02-12T16:54:13.334078+0000","last_undegraded":"2024-02-12T16:54:13.334078+0000","last_fullsized":"2024-02-12T16:54:13.334078+0000","mapping_epoch":93,"log_start":"107'2000","ondisk_log_start":"107'2000","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5522,"ondisk_log_size":5522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":417202176,"num_objects":6366,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7522,"num_write_kb":444416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103855+0000","last_change":"2024-02-12T16:53:42.946856+0000","last_active":"2024-02-12T16:54:08.103855+0000","last_peered":"2024-02-12T16:54:08.103855+0000","last_clean":"2024-02-12T16:54:08.103855+0000","last_became_active":"2024-02-12T16:49:45.213152+0000","last_became_peered":"2024-02-12T16:49:45.213152+0000","last_unstale":"2024-02-12T16:54:08.103855+0000","last_undegraded":"2024-02-12T16:54:08.103855+0000","last_fullsized":"2024-02-12T16:54:08.103855+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":92,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.101054+0000","last_change":"2024-02-12T16:52:55.967424+0000","last_active":"2024-02-12T16:54:08.101054+0000","last_peered":"2024-02-12T16:54:08.101054+0000","last_clean":"2024-02-12T16:54:08.101054+0000","last_became_active":"2024-02-12T16:49:39.105761+0000","last_became_peered":"2024-02-12T16:49:39.105761+0000","last_unstale":"2024-02-12T16:54:08.101054+0000","last_undegraded":"2024-02-12T16:54:08.101054+0000","last_fullsized":"2024-02-12T16:54:08.101054+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.967335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103303+0000","last_change":"2024-02-12T16:52:57.985677+0000","last_active":"2024-02-12T16:54:08.103303+0000","last_peered":"2024-02-12T16:54:08.103303+0000","last_clean":"2024-02-12T16:54:08.103303+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:54:08.103303+0000","last_undegraded":"2024-02-12T16:54:08.103303+0000","last_fullsized":"2024-02-12T16:54:08.103303+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:57.985584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:57.985584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":94,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.113521+0000","last_change":"2024-02-12T16:51:37.184984+0000","last_active":"2024-02-12T16:54:08.113521+0000","last_peered":"2024-02-12T16:54:08.113521+0000","last_clean":"2024-02-12T16:54:08.113521+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:54:08.113521+0000","last_undegraded":"2024-02-12T16:54:08.113521+0000","last_fullsized":"2024-02-12T16:54:08.113521+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:37.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.184907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":117,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112238+0000","last_change":"2024-02-12T16:53:58.740196+0000","last_active":"2024-02-12T16:54:08.112238+0000","last_peered":"2024-02-12T16:54:08.112238+0000","last_clean":"2024-02-12T16:54:08.112238+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:54:08.112238+0000","last_undegraded":"2024-02-12T16:54:08.112238+0000","last_fullsized":"2024-02-12T16:54:08.112238+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:53:58.740122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.740122+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112553+0000","last_change":"2024-02-12T16:52:12.200439+0000","last_active":"2024-02-12T16:54:08.112553+0000","last_peered":"2024-02-12T16:54:08.112553+0000","last_clean":"2024-02-12T16:54:08.112553+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:54:08.112553+0000","last_undegraded":"2024-02-12T16:54:08.112553+0000","last_fullsized":"2024-02-12T16:54:08.112553+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:12.200355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:12.200355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"132'7607","reported_seq":7654,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.281510+0000","last_change":"2024-02-12T16:52:16.559957+0000","last_active":"2024-02-12T16:54:13.281510+0000","last_peered":"2024-02-12T16:54:13.281510+0000","last_clean":"2024-02-12T16:54:13.281510+0000","last_became_active":"2024-02-12T16:52:16.559699+0000","last_became_peered":"2024-02-12T16:52:16.559699+0000","last_unstale":"2024-02-12T16:54:13.281510+0000","last_undegraded":"2024-02-12T16:54:13.281510+0000","last_fullsized":"2024-02-12T16:54:13.281510+0000","mapping_epoch":93,"log_start":"109'2600","ondisk_log_start":"109'2600","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5007,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":418054144,"num_objects":6379,"num_object_clones":0,"num_object_copies":19137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":447552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.100550+0000","last_change":"2024-02-12T16:54:00.780908+0000","last_active":"2024-02-12T16:54:08.100550+0000","last_peered":"2024-02-12T16:54:08.100550+0000","last_clean":"2024-02-12T16:54:08.100550+0000","last_became_active":"2024-02-12T16:49:45.218096+0000","last_became_peered":"2024-02-12T16:49:45.218096+0000","last_unstale":"2024-02-12T16:54:08.100550+0000","last_undegraded":"2024-02-12T16:54:08.100550+0000","last_fullsized":"2024-02-12T16:54:08.100550+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.780846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.780846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":91,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.104640+0000","last_change":"2024-02-12T16:54:04.254755+0000","last_active":"2024-02-12T16:54:08.104640+0000","last_peered":"2024-02-12T16:54:08.104640+0000","last_clean":"2024-02-12T16:54:08.104640+0000","last_became_active":"2024-02-12T16:49:39.104017+0000","last_became_peered":"2024-02-12T16:49:39.104017+0000","last_unstale":"2024-02-12T16:54:08.104640+0000","last_undegraded":"2024-02-12T16:54:08.104640+0000","last_fullsized":"2024-02-12T16:54:08.104640+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":97,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.113446+0000","last_change":"2024-02-12T16:52:58.167642+0000","last_active":"2024-02-12T16:54:08.113446+0000","last_peered":"2024-02-12T16:54:08.113446+0000","last_clean":"2024-02-12T16:54:08.113446+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:54:08.113446+0000","last_undegraded":"2024-02-12T16:54:08.113446+0000","last_fullsized":"2024-02-12T16:54:08.113446+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:52:58.167543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.167543+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":107,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.104319+0000","last_change":"2024-02-12T16:53:43.229740+0000","last_active":"2024-02-12T16:54:08.104319+0000","last_peered":"2024-02-12T16:54:08.104319+0000","last_clean":"2024-02-12T16:54:08.104319+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:54:08.104319+0000","last_undegraded":"2024-02-12T16:54:08.104319+0000","last_fullsized":"2024-02-12T16:54:08.104319+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:43.229622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.229622+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":84,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103816+0000","last_change":"2024-02-12T16:51:22.070134+0000","last_active":"2024-02-12T16:54:08.103816+0000","last_peered":"2024-02-12T16:54:08.103816+0000","last_clean":"2024-02-12T16:54:08.103816+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:54:08.103816+0000","last_undegraded":"2024-02-12T16:54:08.103816+0000","last_fullsized":"2024-02-12T16:54:08.103816+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.070056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.070056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"132'7430","reported_seq":7477,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.296764+0000","last_change":"2024-02-12T16:52:16.554135+0000","last_active":"2024-02-12T16:54:13.296764+0000","last_peered":"2024-02-12T16:54:13.296764+0000","last_clean":"2024-02-12T16:54:13.296764+0000","last_became_active":"2024-02-12T16:52:16.553900+0000","last_became_peered":"2024-02-12T16:52:16.553900+0000","last_unstale":"2024-02-12T16:54:13.296764+0000","last_undegraded":"2024-02-12T16:54:13.296764+0000","last_fullsized":"2024-02-12T16:54:13.296764+0000","mapping_epoch":93,"log_start":"105'1600","ondisk_log_start":"105'1600","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5830,"ondisk_log_size":5830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":406716416,"num_objects":6206,"num_object_clones":0,"num_object_copies":18618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7430,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099123+0000","last_change":"2024-02-12T16:53:50.836514+0000","last_active":"2024-02-12T16:54:08.099123+0000","last_peered":"2024-02-12T16:54:08.099123+0000","last_clean":"2024-02-12T16:54:08.099123+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:54:08.099123+0000","last_undegraded":"2024-02-12T16:54:08.099123+0000","last_fullsized":"2024-02-12T16:54:08.099123+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.836406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.836406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":96,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112864+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T16:54:08.112864+0000","last_peered":"2024-02-12T16:54:08.112864+0000","last_clean":"2024-02-12T16:54:08.112864+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:54:08.112864+0000","last_undegraded":"2024-02-12T16:54:08.112864+0000","last_fullsized":"2024-02-12T16:54:08.112864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":125,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.105663+0000","last_change":"2024-02-12T16:51:52.745302+0000","last_active":"2024-02-12T16:54:08.105663+0000","last_peered":"2024-02-12T16:54:08.105663+0000","last_clean":"2024-02-12T16:54:08.105663+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:54:08.105663+0000","last_undegraded":"2024-02-12T16:54:08.105663+0000","last_fullsized":"2024-02-12T16:54:08.105663+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:51:52.745190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:52.745190+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112259+0000","last_change":"2024-02-12T16:53:50.681255+0000","last_active":"2024-02-12T16:54:08.112259+0000","last_peered":"2024-02-12T16:54:08.112259+0000","last_clean":"2024-02-12T16:54:08.112259+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:54:08.112259+0000","last_undegraded":"2024-02-12T16:54:08.112259+0000","last_fullsized":"2024-02-12T16:54:08.112259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.681142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.681142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"132'7476","reported_seq":7523,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.309070+0000","last_change":"2024-02-12T16:52:16.557948+0000","last_active":"2024-02-12T16:54:13.309070+0000","last_peered":"2024-02-12T16:54:13.309070+0000","last_clean":"2024-02-12T16:54:13.309070+0000","last_became_active":"2024-02-12T16:52:16.557659+0000","last_became_peered":"2024-02-12T16:52:16.557659+0000","last_unstale":"2024-02-12T16:54:13.309070+0000","last_undegraded":"2024-02-12T16:54:13.309070+0000","last_fullsized":"2024-02-12T16:54:13.309070+0000","mapping_epoch":93,"log_start":"111'2700","ondisk_log_start":"111'2700","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4776,"ondisk_log_size":4776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":407371776,"num_objects":6216,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7476,"num_write_kb":438144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":51,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.100974+0000","last_change":"2024-02-12T16:54:02.624972+0000","last_active":"2024-02-12T16:54:08.100974+0000","last_peered":"2024-02-12T16:54:08.100974+0000","last_clean":"2024-02-12T16:54:08.100974+0000","last_became_active":"2024-02-12T16:52:20.563167+0000","last_became_peered":"2024-02-12T16:52:20.563167+0000","last_unstale":"2024-02-12T16:54:08.100974+0000","last_undegraded":"2024-02-12T16:54:08.100974+0000","last_fullsized":"2024-02-12T16:54:08.100974+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.624865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.624865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.098776+0000","last_change":"2024-02-12T16:53:48.828698+0000","last_active":"2024-02-12T16:54:08.098776+0000","last_peered":"2024-02-12T16:54:08.098776+0000","last_clean":"2024-02-12T16:54:08.098776+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:54:08.098776+0000","last_undegraded":"2024-02-12T16:54:08.098776+0000","last_fullsized":"2024-02-12T16:54:08.098776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":117,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103201+0000","last_change":"2024-02-12T16:51:51.095208+0000","last_active":"2024-02-12T16:54:08.103201+0000","last_peered":"2024-02-12T16:54:08.103201+0000","last_clean":"2024-02-12T16:54:08.103201+0000","last_became_active":"2024-02-12T16:49:41.669593+0000","last_became_peered":"2024-02-12T16:49:41.669593+0000","last_unstale":"2024-02-12T16:54:08.103201+0000","last_undegraded":"2024-02-12T16:54:08.103201+0000","last_fullsized":"2024-02-12T16:54:08.103201+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:51:51.095125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:51.095125+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":86,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.094185+0000","last_change":"2024-02-12T16:52:56.823627+0000","last_active":"2024-02-12T16:54:08.094185+0000","last_peered":"2024-02-12T16:54:08.094185+0000","last_clean":"2024-02-12T16:54:08.094185+0000","last_became_active":"2024-02-12T16:49:43.202312+0000","last_became_peered":"2024-02-12T16:49:43.202312+0000","last_unstale":"2024-02-12T16:54:08.094185+0000","last_undegraded":"2024-02-12T16:54:08.094185+0000","last_fullsized":"2024-02-12T16:54:08.094185+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:56.823548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:56.823548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"132'7439","reported_seq":7486,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.231734+0000","last_change":"2024-02-12T16:52:16.557971+0000","last_active":"2024-02-12T16:54:13.231734+0000","last_peered":"2024-02-12T16:54:13.231734+0000","last_clean":"2024-02-12T16:54:13.231734+0000","last_became_active":"2024-02-12T16:52:16.557700+0000","last_became_peered":"2024-02-12T16:52:16.557700+0000","last_unstale":"2024-02-12T16:54:13.231734+0000","last_undegraded":"2024-02-12T16:54:13.231734+0000","last_fullsized":"2024-02-12T16:54:13.231734+0000","mapping_epoch":93,"log_start":"110'2700","ondisk_log_start":"110'2700","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4739,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":408354816,"num_objects":6231,"num_object_clones":0,"num_object_copies":18693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":86,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112081+0000","last_change":"2024-02-12T16:52:41.149470+0000","last_active":"2024-02-12T16:54:08.112081+0000","last_peered":"2024-02-12T16:54:08.112081+0000","last_clean":"2024-02-12T16:54:08.112081+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:54:08.112081+0000","last_undegraded":"2024-02-12T16:54:08.112081+0000","last_fullsized":"2024-02-12T16:54:08.112081+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:52:41.149382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.149382+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":92,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112922+0000","last_change":"2024-02-12T16:52:49.581959+0000","last_active":"2024-02-12T16:54:08.112922+0000","last_peered":"2024-02-12T16:54:08.112922+0000","last_clean":"2024-02-12T16:54:08.112922+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:54:08.112922+0000","last_undegraded":"2024-02-12T16:54:08.112922+0000","last_fullsized":"2024-02-12T16:54:08.112922+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.581862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.581862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":84,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102397+0000","last_change":"2024-02-12T16:52:51.025242+0000","last_active":"2024-02-12T16:54:08.102397+0000","last_peered":"2024-02-12T16:54:08.102397+0000","last_clean":"2024-02-12T16:54:08.102397+0000","last_became_active":"2024-02-12T16:52:20.563744+0000","last_became_peered":"2024-02-12T16:52:20.563744+0000","last_unstale":"2024-02-12T16:54:08.102397+0000","last_undegraded":"2024-02-12T16:54:08.102397+0000","last_fullsized":"2024-02-12T16:54:08.102397+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:52:51.025152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.025152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":151,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089214+0000","last_change":"2024-02-12T16:52:11.925541+0000","last_active":"2024-02-12T16:54:12.089214+0000","last_peered":"2024-02-12T16:54:12.089214+0000","last_clean":"2024-02-12T16:54:12.089214+0000","last_became_active":"2024-02-12T16:49:43.205391+0000","last_became_peered":"2024-02-12T16:49:43.205391+0000","last_unstale":"2024-02-12T16:54:12.089214+0000","last_undegraded":"2024-02-12T16:54:12.089214+0000","last_fullsized":"2024-02-12T16:54:12.089214+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:11.925474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:11.925474+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"132'7615","reported_seq":7668,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.256618+0000","last_change":"2024-02-12T16:53:48.150960+0000","last_active":"2024-02-12T16:54:13.256618+0000","last_peered":"2024-02-12T16:54:13.256618+0000","last_clean":"2024-02-12T16:54:13.256618+0000","last_became_active":"2024-02-12T16:52:16.560135+0000","last_became_peered":"2024-02-12T16:52:16.560135+0000","last_unstale":"2024-02-12T16:54:13.256618+0000","last_undegraded":"2024-02-12T16:54:13.256618+0000","last_fullsized":"2024-02-12T16:54:13.256618+0000","mapping_epoch":93,"log_start":"105'1400","ondisk_log_start":"105'1400","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'6180","last_scrub_stamp":"2024-02-12T16:53:48.150854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.150854+0000","log_size":6215,"ondisk_log_size":6215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":417726488,"num_objects":6375,"num_object_clones":0,"num_object_copies":19125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7615,"num_write_kb":447617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":81,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.100226+0000","last_change":"2024-02-12T16:51:34.081725+0000","last_active":"2024-02-12T16:54:08.100226+0000","last_peered":"2024-02-12T16:54:08.100226+0000","last_clean":"2024-02-12T16:54:08.100226+0000","last_became_active":"2024-02-12T16:49:45.223659+0000","last_became_peered":"2024-02-12T16:49:45.223659+0000","last_unstale":"2024-02-12T16:54:08.100226+0000","last_undegraded":"2024-02-12T16:54:08.100226+0000","last_fullsized":"2024-02-12T16:54:08.100226+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:34.081644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.081644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103740+0000","last_change":"2024-02-12T16:52:52.983089+0000","last_active":"2024-02-12T16:54:08.103740+0000","last_peered":"2024-02-12T16:54:08.103740+0000","last_clean":"2024-02-12T16:54:08.103740+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:54:08.103740+0000","last_undegraded":"2024-02-12T16:54:08.103740+0000","last_fullsized":"2024-02-12T16:54:08.103740+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:52.982996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.982996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":70,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.108804+0000","last_change":"2024-02-12T16:53:59.566836+0000","last_active":"2024-02-12T16:54:08.108804+0000","last_peered":"2024-02-12T16:54:08.108804+0000","last_clean":"2024-02-12T16:54:08.108804+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:54:08.108804+0000","last_undegraded":"2024-02-12T16:54:08.108804+0000","last_fullsized":"2024-02-12T16:54:08.108804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.566730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":82,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112574+0000","last_change":"2024-02-12T16:52:47.559724+0000","last_active":"2024-02-12T16:54:08.112574+0000","last_peered":"2024-02-12T16:54:08.112574+0000","last_clean":"2024-02-12T16:54:08.112574+0000","last_became_active":"2024-02-12T16:52:20.561707+0000","last_became_peered":"2024-02-12T16:52:20.561707+0000","last_unstale":"2024-02-12T16:54:08.112574+0000","last_undegraded":"2024-02-12T16:54:08.112574+0000","last_fullsized":"2024-02-12T16:54:08.112574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:52:47.559659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.559659+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"132'7464","reported_seq":7516,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.298749+0000","last_change":"2024-02-12T16:53:45.409281+0000","last_active":"2024-02-12T16:54:13.298749+0000","last_peered":"2024-02-12T16:54:13.298749+0000","last_clean":"2024-02-12T16:54:13.298749+0000","last_became_active":"2024-02-12T16:52:16.556464+0000","last_became_peered":"2024-02-12T16:52:16.556464+0000","last_unstale":"2024-02-12T16:54:13.298749+0000","last_undegraded":"2024-02-12T16:54:13.298749+0000","last_fullsized":"2024-02-12T16:54:13.298749+0000","mapping_epoch":93,"log_start":"105'1300","ondisk_log_start":"105'1300","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'5931","last_scrub_stamp":"2024-02-12T16:53:45.409189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.409189+0000","log_size":6164,"ondisk_log_size":6164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":408551424,"num_objects":6234,"num_object_clones":0,"num_object_copies":18702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7464,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.103467+0000","last_change":"2024-02-12T16:52:46.979058+0000","last_active":"2024-02-12T16:54:08.103467+0000","last_peered":"2024-02-12T16:54:08.103467+0000","last_clean":"2024-02-12T16:54:08.103467+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:54:08.103467+0000","last_undegraded":"2024-02-12T16:54:08.103467+0000","last_fullsized":"2024-02-12T16:54:08.103467+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:46.978987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:46.978987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"98'10","reported_seq":120,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.092005+0000","last_change":"2024-02-12T16:52:20.561940+0000","last_active":"2024-02-12T16:54:12.092005+0000","last_peered":"2024-02-12T16:54:12.092005+0000","last_clean":"2024-02-12T16:54:12.092005+0000","last_became_active":"2024-02-12T16:52:20.561534+0000","last_became_peered":"2024-02-12T16:52:20.561534+0000","last_unstale":"2024-02-12T16:54:12.092005+0000","last_undegraded":"2024-02-12T16:54:12.092005+0000","last_fullsized":"2024-02-12T16:54:12.092005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-02-12T16:52:12.826762+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:52:12.826762+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.098954+0000","last_change":"2024-02-12T16:52:15.773872+0000","last_active":"2024-02-12T16:54:08.098954+0000","last_peered":"2024-02-12T16:54:08.098954+0000","last_clean":"2024-02-12T16:54:08.098954+0000","last_became_active":"2024-02-12T16:49:39.095868+0000","last_became_peered":"2024-02-12T16:49:39.095868+0000","last_unstale":"2024-02-12T16:54:08.098954+0000","last_undegraded":"2024-02-12T16:54:08.098954+0000","last_fullsized":"2024-02-12T16:54:08.098954+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.773807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.773807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":123,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099260+0000","last_change":"2024-02-12T16:52:51.864105+0000","last_active":"2024-02-12T16:54:08.099260+0000","last_peered":"2024-02-12T16:54:08.099260+0000","last_clean":"2024-02-12T16:54:08.099260+0000","last_became_active":"2024-02-12T16:49:41.178544+0000","last_became_peered":"2024-02-12T16:49:41.178544+0000","last_unstale":"2024-02-12T16:54:08.099260+0000","last_undegraded":"2024-02-12T16:54:08.099260+0000","last_fullsized":"2024-02-12T16:54:08.099260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.864004+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"132'7584","reported_seq":7630,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.358405+0000","last_change":"2024-02-12T16:52:16.557015+0000","last_active":"2024-02-12T16:54:13.358405+0000","last_peered":"2024-02-12T16:54:13.358405+0000","last_clean":"2024-02-12T16:54:13.358405+0000","last_became_active":"2024-02-12T16:52:16.556667+0000","last_became_peered":"2024-02-12T16:52:16.556667+0000","last_unstale":"2024-02-12T16:54:13.358405+0000","last_undegraded":"2024-02-12T16:54:13.358405+0000","last_fullsized":"2024-02-12T16:54:13.358405+0000","mapping_epoch":93,"log_start":"107'2100","ondisk_log_start":"107'2100","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5484,"ondisk_log_size":5484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":420478976,"num_objects":6416,"num_object_clones":0,"num_object_copies":19248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":448000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.098539+0000","last_change":"2024-02-12T16:53:57.738352+0000","last_active":"2024-02-12T16:54:08.098539+0000","last_peered":"2024-02-12T16:54:08.098539+0000","last_clean":"2024-02-12T16:54:08.098539+0000","last_became_active":"2024-02-12T16:52:20.560085+0000","last_became_peered":"2024-02-12T16:52:20.560085+0000","last_unstale":"2024-02-12T16:54:08.098539+0000","last_undegraded":"2024-02-12T16:54:08.098539+0000","last_fullsized":"2024-02-12T16:54:08.098539+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:57.738249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.738249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":152,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.080589+0000","last_change":"2024-02-12T16:52:52.366599+0000","last_active":"2024-02-12T16:54:12.080589+0000","last_peered":"2024-02-12T16:54:12.080589+0000","last_clean":"2024-02-12T16:54:12.080589+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:54:12.080589+0000","last_undegraded":"2024-02-12T16:54:12.080589+0000","last_fullsized":"2024-02-12T16:54:12.080589+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:52.366513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.366513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099205+0000","last_change":"2024-02-12T16:52:53.862672+0000","last_active":"2024-02-12T16:54:08.099205+0000","last_peered":"2024-02-12T16:54:08.099205+0000","last_clean":"2024-02-12T16:54:08.099205+0000","last_became_active":"2024-02-12T16:49:39.101478+0000","last_became_peered":"2024-02-12T16:49:39.101478+0000","last_unstale":"2024-02-12T16:54:08.099205+0000","last_undegraded":"2024-02-12T16:54:08.099205+0000","last_fullsized":"2024-02-12T16:54:08.099205+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:53.862480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:53.862480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":78,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.100510+0000","last_change":"2024-02-12T16:52:43.063907+0000","last_active":"2024-02-12T16:54:08.100510+0000","last_peered":"2024-02-12T16:54:08.100510+0000","last_clean":"2024-02-12T16:54:08.100510+0000","last_became_active":"2024-02-12T16:52:20.564894+0000","last_became_peered":"2024-02-12T16:52:20.564894+0000","last_unstale":"2024-02-12T16:54:08.100510+0000","last_undegraded":"2024-02-12T16:54:08.100510+0000","last_fullsized":"2024-02-12T16:54:08.100510+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:52:43.063831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.063831+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"132'7489","reported_seq":7536,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.331344+0000","last_change":"2024-02-12T16:52:16.562286+0000","last_active":"2024-02-12T16:54:13.331344+0000","last_peered":"2024-02-12T16:54:13.331344+0000","last_clean":"2024-02-12T16:54:13.331344+0000","last_became_active":"2024-02-12T16:52:16.561976+0000","last_became_peered":"2024-02-12T16:52:16.561976+0000","last_unstale":"2024-02-12T16:54:13.331344+0000","last_undegraded":"2024-02-12T16:54:13.331344+0000","last_fullsized":"2024-02-12T16:54:13.331344+0000","mapping_epoch":93,"log_start":"104'1100","ondisk_log_start":"104'1100","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":6389,"ondisk_log_size":6389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":413597696,"num_objects":6311,"num_object_clones":0,"num_object_copies":18933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7489,"num_write_kb":441600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.110164+0000","last_change":"2024-02-12T16:52:01.448343+0000","last_active":"2024-02-12T16:54:08.110164+0000","last_peered":"2024-02-12T16:54:08.110164+0000","last_clean":"2024-02-12T16:54:08.110164+0000","last_became_active":"2024-02-12T16:49:45.220893+0000","last_became_peered":"2024-02-12T16:49:45.220893+0000","last_unstale":"2024-02-12T16:54:08.110164+0000","last_undegraded":"2024-02-12T16:54:08.110164+0000","last_fullsized":"2024-02-12T16:54:08.110164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:01.448278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:01.448278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.100621+0000","last_change":"2024-02-12T16:53:49.832912+0000","last_active":"2024-02-12T16:54:08.100621+0000","last_peered":"2024-02-12T16:54:08.100621+0000","last_clean":"2024-02-12T16:54:08.100621+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:54:08.100621+0000","last_undegraded":"2024-02-12T16:54:08.100621+0000","last_fullsized":"2024-02-12T16:54:08.100621+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:49.832798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.832798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":91,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.111383+0000","last_change":"2024-02-12T16:51:53.008096+0000","last_active":"2024-02-12T16:54:08.111383+0000","last_peered":"2024-02-12T16:54:08.111383+0000","last_clean":"2024-02-12T16:54:08.111383+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:54:08.111383+0000","last_undegraded":"2024-02-12T16:54:08.111383+0000","last_fullsized":"2024-02-12T16:54:08.111383+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:53.007988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:53.007988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":91,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.109305+0000","last_change":"2024-02-12T16:52:44.280962+0000","last_active":"2024-02-12T16:54:08.109305+0000","last_peered":"2024-02-12T16:54:08.109305+0000","last_clean":"2024-02-12T16:54:08.109305+0000","last_became_active":"2024-02-12T16:52:20.565139+0000","last_became_peered":"2024-02-12T16:52:20.565139+0000","last_unstale":"2024-02-12T16:54:08.109305+0000","last_undegraded":"2024-02-12T16:54:08.109305+0000","last_fullsized":"2024-02-12T16:54:08.109305+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:52:44.280844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.280844+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"132'7555","reported_seq":7601,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.336880+0000","last_change":"2024-02-12T16:52:16.557129+0000","last_active":"2024-02-12T16:54:13.336880+0000","last_peered":"2024-02-12T16:54:13.336880+0000","last_clean":"2024-02-12T16:54:13.336880+0000","last_became_active":"2024-02-12T16:52:16.556854+0000","last_became_peered":"2024-02-12T16:52:16.556854+0000","last_unstale":"2024-02-12T16:54:13.336880+0000","last_undegraded":"2024-02-12T16:54:13.336880+0000","last_fullsized":"2024-02-12T16:54:13.336880+0000","mapping_epoch":93,"log_start":"107'2100","ondisk_log_start":"107'2100","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5455,"ondisk_log_size":5455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":416743424,"num_objects":6359,"num_object_clones":0,"num_object_copies":19077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":445248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":85,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.113503+0000","last_change":"2024-02-12T16:52:48.128798+0000","last_active":"2024-02-12T16:54:08.113503+0000","last_peered":"2024-02-12T16:54:08.113503+0000","last_clean":"2024-02-12T16:54:08.113503+0000","last_became_active":"2024-02-12T16:49:45.217589+0000","last_became_peered":"2024-02-12T16:49:45.217589+0000","last_unstale":"2024-02-12T16:54:08.113503+0000","last_undegraded":"2024-02-12T16:54:08.113503+0000","last_fullsized":"2024-02-12T16:54:08.113503+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.128703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.128703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":94,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.109498+0000","last_change":"2024-02-12T16:53:58.524837+0000","last_active":"2024-02-12T16:54:08.109498+0000","last_peered":"2024-02-12T16:54:08.109498+0000","last_clean":"2024-02-12T16:54:08.109498+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:54:08.109498+0000","last_undegraded":"2024-02-12T16:54:08.109498+0000","last_fullsized":"2024-02-12T16:54:08.109498+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.524736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":93,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.109109+0000","last_change":"2024-02-12T16:52:05.495937+0000","last_active":"2024-02-12T16:54:08.109109+0000","last_peered":"2024-02-12T16:54:08.109109+0000","last_clean":"2024-02-12T16:54:08.109109+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:54:08.109109+0000","last_undegraded":"2024-02-12T16:54:08.109109+0000","last_fullsized":"2024-02-12T16:54:08.109109+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:05.495862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:05.495862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":155,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.112164+0000","last_change":"2024-02-12T16:53:40.847643+0000","last_active":"2024-02-12T16:54:08.112164+0000","last_peered":"2024-02-12T16:54:08.112164+0000","last_clean":"2024-02-12T16:54:08.112164+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:54:08.112164+0000","last_undegraded":"2024-02-12T16:54:08.112164+0000","last_fullsized":"2024-02-12T16:54:08.112164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:53:40.847540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:40.847540+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"132'7466","reported_seq":7513,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.224461+0000","last_change":"2024-02-12T16:52:16.551619+0000","last_active":"2024-02-12T16:54:13.224461+0000","last_peered":"2024-02-12T16:54:13.224461+0000","last_clean":"2024-02-12T16:54:13.224461+0000","last_became_active":"2024-02-12T16:52:16.551312+0000","last_became_peered":"2024-02-12T16:52:16.551312+0000","last_unstale":"2024-02-12T16:54:13.224461+0000","last_undegraded":"2024-02-12T16:54:13.224461+0000","last_fullsized":"2024-02-12T16:54:13.224461+0000","mapping_epoch":93,"log_start":"104'1000","ondisk_log_start":"104'1000","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":6466,"ondisk_log_size":6466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":409862144,"num_objects":6254,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7466,"num_write_kb":439040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":85,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.107095+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T16:54:08.107095+0000","last_peered":"2024-02-12T16:54:08.107095+0000","last_clean":"2024-02-12T16:54:08.107095+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:54:08.107095+0000","last_undegraded":"2024-02-12T16:54:08.107095+0000","last_fullsized":"2024-02-12T16:54:08.107095+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":89,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.110742+0000","last_change":"2024-02-12T16:53:01.286057+0000","last_active":"2024-02-12T16:54:08.110742+0000","last_peered":"2024-02-12T16:54:08.110742+0000","last_clean":"2024-02-12T16:54:08.110742+0000","last_became_active":"2024-02-12T16:49:43.201946+0000","last_became_peered":"2024-02-12T16:49:43.201946+0000","last_unstale":"2024-02-12T16:54:08.110742+0000","last_undegraded":"2024-02-12T16:54:08.110742+0000","last_fullsized":"2024-02-12T16:54:08.110742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.285961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.285961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":103,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.099424+0000","last_change":"2024-02-12T16:53:56.761528+0000","last_active":"2024-02-12T16:54:08.099424+0000","last_peered":"2024-02-12T16:54:08.099424+0000","last_clean":"2024-02-12T16:54:08.099424+0000","last_became_active":"2024-02-12T16:49:39.099695+0000","last_became_peered":"2024-02-12T16:49:39.099695+0000","last_unstale":"2024-02-12T16:54:08.099424+0000","last_undegraded":"2024-02-12T16:54:08.099424+0000","last_fullsized":"2024-02-12T16:54:08.099424+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":105,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.105802+0000","last_change":"2024-02-12T16:52:08.569339+0000","last_active":"2024-02-12T16:54:08.105802+0000","last_peered":"2024-02-12T16:54:08.105802+0000","last_clean":"2024-02-12T16:54:08.105802+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:54:08.105802+0000","last_undegraded":"2024-02-12T16:54:08.105802+0000","last_fullsized":"2024-02-12T16:54:08.105802+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:08.569246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:08.569246+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"132'7457","reported_seq":7504,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:13.310651+0000","last_change":"2024-02-12T16:52:16.550877+0000","last_active":"2024-02-12T16:54:13.310651+0000","last_peered":"2024-02-12T16:54:13.310651+0000","last_clean":"2024-02-12T16:54:13.310651+0000","last_became_active":"2024-02-12T16:52:16.550574+0000","last_became_peered":"2024-02-12T16:52:16.550574+0000","last_unstale":"2024-02-12T16:54:13.310651+0000","last_undegraded":"2024-02-12T16:54:13.310651+0000","last_fullsized":"2024-02-12T16:54:13.310651+0000","mapping_epoch":93,"log_start":"107'2300","ondisk_log_start":"107'2300","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5157,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":405602304,"num_objects":6189,"num_object_clones":0,"num_object_copies":18567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7457,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":87,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102450+0000","last_change":"2024-02-12T16:54:03.996224+0000","last_active":"2024-02-12T16:54:08.102450+0000","last_peered":"2024-02-12T16:54:08.102450+0000","last_clean":"2024-02-12T16:54:08.102450+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:54:08.102450+0000","last_undegraded":"2024-02-12T16:54:08.102450+0000","last_fullsized":"2024-02-12T16:54:08.102450+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":98,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.110428+0000","last_change":"2024-02-12T16:53:00.294697+0000","last_active":"2024-02-12T16:54:08.110428+0000","last_peered":"2024-02-12T16:54:08.110428+0000","last_clean":"2024-02-12T16:54:08.110428+0000","last_became_active":"2024-02-12T16:49:39.105982+0000","last_became_peered":"2024-02-12T16:49:39.105982+0000","last_unstale":"2024-02-12T16:54:08.110428+0000","last_undegraded":"2024-02-12T16:54:08.110428+0000","last_fullsized":"2024-02-12T16:54:08.110428+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:00.294633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:00.294633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":116,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-02-12T16:54:08.102414+0000","last_change":"2024-02-12T16:52:40.091071+0000","last_active":"2024-02-12T16:54:08.102414+0000","last_peered":"2024-02-12T16:54:08.102414+0000","last_clean":"2024-02-12T16:54:08.102414+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:54:08.102414+0000","last_undegraded":"2024-02-12T16:54:08.102414+0000","last_fullsized":"2024-02-12T16:54:08.102414+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:52:40.090931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.090931+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.085813+0000","last_change":"2024-02-12T16:52:59.185115+0000","last_active":"2024-02-12T16:54:12.085813+0000","last_peered":"2024-02-12T16:54:12.085813+0000","last_clean":"2024-02-12T16:54:12.085813+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:54:12.085813+0000","last_undegraded":"2024-02-12T16:54:12.085813+0000","last_fullsized":"2024-02-12T16:54:12.085813+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6603472920,"num_objects":100762,"num_object_clones":0,"num_object_copies":302286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120138,"num_write_kb":7068737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19808923648,"data_stored":19808911432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88338,"ondisk_log_size":88338,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":267,"num_read_kb":262,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":691,"num_read_kb":448,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234042,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2340192,"kb_used_data":2042968,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":91404960,"statfs":{"total":95995035648,"available":93598679040,"internally_reserved":0,"allocated":2091999232,"data_stored":2091590370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":0.75600000000000001,"15min":0.56699999999999995},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":4.9290000000000003,"5min":4.9290000000000003,"15min":4.9290000000000003},"last":36.439},{"interface":"front","average":{"1min":1.3879999999999999,"5min":0.69199999999999995,"15min":0.57699999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.2190000000000003,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":36.490000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.58899999999999997,"15min":0.55700000000000005},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.59799999999999998,"15min":0.56399999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":0.65300000000000002,"15min":0.53900000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":2.7509999999999999,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.59599999999999997,"15min":0.54500000000000004},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":20.338000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":0.88800000000000001,"15min":0.68600000000000005},"min":{"1min":0.34699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.423,"5min":5.423,"15min":5.423},"last":20.433},{"interface":"front","average":{"1min":1.893,"5min":0.82099999999999995,"15min":0.65100000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.5979999999999999,"5min":5.5979999999999999,"15min":5.5979999999999999},"last":20.393000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.51400000000000001,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.2759999999999998,"5min":3.2759999999999998,"15min":3.2759999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.46600000000000003,"15min":0.50700000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.58499999999999996,"5min":1.327,"15min":1.327},"last":0.29299999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.401,"5min":1.573,"15min":1.573},"last":0.372},{"interface":"front","average":{"1min":0.378,"5min":0.49399999999999999,"15min":0.53000000000000003},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54500000000000004,"5min":1.621,"15min":1.621},"last":0.34799999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.32800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.59499999999999997,"15min":0.57599999999999996},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.49199999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528383,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2319312,"kb_used_data":2022088,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":91425840,"statfs":{"total":95995035648,"available":93620060160,"internally_reserved":0,"allocated":2070618112,"data_stored":2070225008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":0.80800000000000005,"15min":0.59299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.7119999999999997,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":1.901},{"interface":"front","average":{"1min":1.7470000000000001,"5min":0.86399999999999999,"15min":0.60999999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":10.56,"5min":10.56,"15min":10.56},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":0.88200000000000001,"15min":0.64300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":5.0839999999999996,"5min":15.987,"15min":15.987},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":0.88500000000000001,"15min":0.65900000000000003},"min":{"1min":0.28000000000000003,"5min":0.25,"15min":0.25},"max":{"1min":5.1580000000000004,"5min":15.997,"15min":15.997},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":0.77100000000000002,"15min":0.59499999999999997},"min":{"1min":0.26400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":4.5910000000000002,"5min":4.5910000000000002,"15min":4.5910000000000002},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":0.67500000000000004,"15min":0.56699999999999995},"min":{"1min":0.29899999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.468,"5min":4.468,"15min":4.468},"last":1.7569999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":0.91100000000000003,"15min":0.63600000000000001},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":11.212,"5min":11.212,"15min":11.212},"last":1.794},{"interface":"front","average":{"1min":2.4889999999999999,"5min":1.071,"15min":0.70499999999999996},"min":{"1min":0.312,"5min":0.249,"15min":0.249},"max":{"1min":10.875999999999999,"5min":10.875999999999999,"15min":10.875999999999999},"last":3.0350000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.46300000000000002,"15min":0.40699999999999997},"min":{"1min":0.254,"5min":0.217,"15min":0.217},"max":{"1min":2.5920000000000001,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":1.659},{"interface":"front","average":{"1min":0.435,"5min":0.442,"15min":0.40999999999999998},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.39900000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44600000000000001,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.51400000000000001,"15min":0.45800000000000002},"min":{"1min":0.313,"5min":0.247,"15min":0.247},"max":{"1min":2.6320000000000001,"5min":2.6320000000000001,"15min":2.6320000000000001},"last":1.702}]},{"osd":7,"last update":"Mon Feb 12 16:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.502,"15min":0.46999999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.49299999999999999,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":1.732}]}]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2728260,"kb_used_data":2422844,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91016892,"statfs":{"total":95995035648,"available":93201297408,"internally_reserved":0,"allocated":2480992256,"data_stored":2480548316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:23 2024","interfaces":[{"interface":"back","average":{"1min":2.3180000000000001,"5min":0.79200000000000004,"15min":0.48599999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":10.935,"5min":10.935,"15min":10.935},"last":7.0289999999999999},{"interface":"front","average":{"1min":2.9609999999999999,"5min":0.94599999999999995,"15min":0.54000000000000004},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":12.544,"5min":12.544,"15min":12.544},"last":3.4969999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:53:46 2024","interfaces":[{"interface":"back","average":{"1min":2.605,"5min":0.84999999999999998,"15min":0.52600000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.4410000000000007,"5min":8.4410000000000007,"15min":8.4410000000000007},"last":0.622},{"interface":"front","average":{"1min":5.0579999999999998,"5min":1.4139999999999999,"15min":0.72099999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":26.693999999999999,"5min":26.693999999999999,"15min":26.693999999999999},"last":3.5609999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:54:08 2024","interfaces":[{"interface":"back","average":{"1min":5.0300000000000002,"5min":1.681,"15min":0.86299999999999999},"min":{"1min":0.35499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":26.649999999999999,"5min":26.649999999999999,"15min":26.649999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.097,"5min":1.26,"15min":0.68799999999999994},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":13.654999999999999,"5min":13.654999999999999,"15min":13.654999999999999},"last":6.9870000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:53:26 2024","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":0.79000000000000004,"15min":0.63600000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.1440000000000001,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":2.2040000000000002,"5min":0.94899999999999995,"15min":0.71199999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":12.366,"5min":12.366,"15min":12.366},"last":0.65300000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:54:00 2024","interfaces":[{"interface":"back","average":{"1min":1.855,"5min":0.876,"15min":0.66200000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":7.2309999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":4.1310000000000002,"5min":1.4670000000000001,"15min":0.88700000000000001},"min":{"1min":0.44800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":26.722999999999999,"5min":26.722999999999999,"15min":26.722999999999999},"last":3.5859999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:53:15 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":0.83399999999999996,"15min":0.66800000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.1959999999999997,"5min":7.1959999999999997,"15min":7.1959999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.446,"5min":0.80200000000000005,"15min":0.68899999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":12.429,"5min":12.429,"15min":12.429},"last":3.528}]},{"osd":7,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":2.036,"5min":0.91500000000000004,"15min":0.747},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":2.387,"5min":1.022,"15min":0.77400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":7.2629999999999999,"5min":7.2629999999999999,"15min":7.2629999999999999},"last":6.9379999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3559264,"kb_used_data":3261912,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":90185888,"statfs":{"total":95995035648,"available":92350349312,"internally_reserved":0,"allocated":3340197888,"data_stored":3339788734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:53:18 2024","interfaces":[{"interface":"back","average":{"1min":1.9039999999999999,"5min":0.66900000000000004,"15min":0.433},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.192},"max":{"1min":9.0579999999999998,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.97,"5min":0.69399999999999995,"15min":0.42999999999999999},"min":{"1min":0.27000000000000002,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.3689999999999998,"5min":5.3689999999999998,"15min":5.3689999999999998},"last":1.6499999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:53:45 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":1.097,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.2739999999999991,"5min":9.2739999999999991,"15min":9.2739999999999991},"last":0.20499999999999999},{"interface":"front","average":{"1min":2.706,"5min":0.91200000000000003,"15min":0.56599999999999995},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":1.226,"15min":0.68600000000000005},"min":{"1min":0.28399999999999997,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.596,"5min":10.596,"15min":10.596},"last":0.51800000000000002},{"interface":"front","average":{"1min":3.4399999999999999,"5min":1.367,"15min":0.76000000000000001},"min":{"1min":0.28100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":11.193,"5min":11.193,"15min":11.193},"last":0.70899999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:53:24 2024","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":0.80300000000000005,"15min":0.64800000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":5.6689999999999996,"5min":5.6689999999999996,"15min":5.6689999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.304,"5min":0.72899999999999998,"15min":0.629},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.4290000000000003,"5min":5.4290000000000003,"15min":5.4290000000000003},"last":0.48399999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:53:56 2024","interfaces":[{"interface":"back","average":{"1min":2.0790000000000002,"5min":0.93500000000000005,"15min":0.69199999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.4340000000000002,"5min":7.4340000000000002,"15min":7.4340000000000002},"last":0.495},{"interface":"front","average":{"1min":1.9990000000000001,"5min":0.94499999999999995,"15min":0.70999999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.5410000000000004,"5min":6.5410000000000004,"15min":6.5410000000000004},"last":0.81100000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:53:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":0.72099999999999997,"15min":0.623},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.4509999999999996,"5min":5.4509999999999996,"15min":5.4509999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.651,"5min":0.79500000000000004,"15min":0.65100000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.45200000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:53:45 2024","interfaces":[{"interface":"back","average":{"1min":2.0539999999999998,"5min":0.95599999999999996,"15min":0.70999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.675,"5min":0.86699999999999999,"15min":0.68600000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":5.4059999999999997,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":0.51200000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2738192,"kb_used_data":2436808,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91006960,"statfs":{"total":95995035648,"available":93191127040,"internally_reserved":0,"allocated":2495291392,"data_stored":2494898507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":2.702,"5min":0.871,"15min":0.54000000000000004},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":8.5779999999999994,"5min":8.5779999999999994,"15min":8.5779999999999994},"last":1.341},{"interface":"front","average":{"1min":2.2970000000000002,"5min":0.82299999999999995,"15min":0.53500000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":10.176,"5min":10.176,"15min":10.176},"last":1.873}]},{"osd":1,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":3.206,"5min":1.0669999999999999,"15min":0.61799999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":11.502000000000001,"5min":11.502000000000001,"15min":11.502000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":2.7370000000000001,"5min":0.93100000000000005,"15min":0.57599999999999996},"min":{"1min":0.27300000000000002,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":8.5559999999999992,"5min":8.5559999999999992,"15min":8.5559999999999992},"last":0.58999999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":6.5990000000000002,"5min":1.958,"15min":0.96099999999999997},"min":{"1min":0.505,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":51.579999999999998,"5min":51.579999999999998,"15min":51.579999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":5.3970000000000002,"5min":1.5489999999999999,"15min":0.82999999999999996},"min":{"1min":0.311,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":48.988999999999997,"5min":48.988999999999997,"15min":48.988999999999997},"last":2.0670000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":0.76200000000000001,"15min":0.67800000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.68400000000000005,"15min":0.63100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":9.2119999999999997,"5min":9.2119999999999997,"15min":9.2119999999999997},"last":1.9750000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:53:51 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":0.998,"15min":0.83599999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.4789999999999992,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":0.54000000000000004},{"interface":"front","average":{"1min":3.016,"5min":1.101,"15min":0.76700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.023999999999999,"5min":11.023999999999999,"15min":11.023999999999999},"last":0.94499999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":3.7069999999999999,"5min":1.3220000000000001,"15min":0.84299999999999997},"min":{"1min":0.38100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":10.981999999999999,"5min":10.981999999999999,"15min":10.981999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.976,"5min":0.94999999999999996,"15min":0.80400000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":9.2840000000000007,"5min":9.2840000000000007,"15min":9.2840000000000007},"last":1.925}]},{"osd":7,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":0.95999999999999996,"15min":0.97399999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":9.2550000000000008,"5min":9.2550000000000008,"15min":9.2550000000000008},"last":2.028},{"interface":"front","average":{"1min":2.8919999999999999,"5min":1.081,"15min":0.78100000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":11.058999999999999,"5min":11.058999999999999,"15min":11.058999999999999},"last":0.99199999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3959468,"kb_used_data":3649956,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":89785684,"statfs":{"total":95995035648,"available":91940540416,"internally_reserved":0,"allocated":3737554944,"data_stored":3737134479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:11 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":0.70799999999999996,"15min":0.495},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":7.8559999999999999,"5min":7.8559999999999999,"15min":7.8559999999999999},"last":6.2569999999999997},{"interface":"front","average":{"1min":1.262,"5min":0.56200000000000006,"15min":0.46300000000000002},"min":{"1min":0.28299999999999997,"5min":0.221,"15min":0.221},"max":{"1min":4.0860000000000003,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":9.2040000000000006}]},{"osd":1,"last update":"Mon Feb 12 16:53:11 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":0.53900000000000003,"15min":0.47399999999999998},"min":{"1min":0.28199999999999997,"5min":0.216,"15min":0.216},"max":{"1min":5.242,"5min":5.242,"15min":5.242},"last":9.1790000000000003},{"interface":"front","average":{"1min":1.905,"5min":0.70499999999999996,"15min":0.52800000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":8.1739999999999995,"5min":8.1739999999999995,"15min":8.1739999999999995},"last":3.5350000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:53:11 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":0.64500000000000002,"15min":0.51000000000000001},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.141,"5min":8.141,"15min":8.141},"last":3.5569999999999999},{"interface":"front","average":{"1min":1.629,"5min":0.65500000000000003,"15min":0.51700000000000002},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.252},"max":{"1min":6.3449999999999998,"5min":6.3449999999999998,"15min":6.3449999999999998},"last":6.2839999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:53:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":0.66700000000000004,"15min":0.63700000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.427,"5min":3.427,"15min":3.427},"last":9.0500000000000007},{"interface":"front","average":{"1min":1.6459999999999999,"5min":0.78400000000000003,"15min":0.67600000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.3420000000000005,"5min":8.3420000000000005,"15min":8.3420000000000005},"last":0.39500000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:53:54 2024","interfaces":[{"interface":"back","average":{"1min":3.1280000000000001,"5min":1.139,"15min":0.80900000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":19.882999999999999,"5min":19.882999999999999,"15min":19.882999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.806,"5min":0.88500000000000001,"15min":0.71899999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":12.861000000000001,"5min":12.861000000000001,"15min":12.861000000000001},"last":9.1489999999999991}]},{"osd":6,"last update":"Mon Feb 12 16:53:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":0.78900000000000003,"15min":0.68300000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":8.093,"5min":8.093,"15min":8.093},"last":3.452},{"interface":"front","average":{"1min":1.736,"5min":0.84299999999999997,"15min":0.70399999999999996},"min":{"1min":0.379,"5min":0.377,"15min":0.377},"max":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"last":0.70699999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":0.73799999999999999,"15min":0.64600000000000002},"min":{"1min":0.39300000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.4820000000000002,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":9.1210000000000004},{"interface":"front","average":{"1min":2.2149999999999999,"5min":0.93000000000000005,"15min":0.72199999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":9.2349999999999994,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.72599999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953545,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2317792,"kb_used_data":2020568,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":91427360,"statfs":{"total":95995035648,"available":93621616640,"internally_reserved":0,"allocated":2069061632,"data_stored":2068653123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:28 2024","interfaces":[{"interface":"back","average":{"1min":3.956,"5min":1.1839999999999999,"15min":0.77400000000000002},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":38.179000000000002,"5min":38.179000000000002,"15min":38.179000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":1.226,"15min":0.75900000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":38.151000000000003,"5min":38.151000000000003,"15min":38.151000000000003},"last":2.3090000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:53:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.60399999999999998,"15min":0.55800000000000005},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":1.889},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.58899999999999997,"15min":0.56599999999999995},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.7629999999999999,"5min":5.7629999999999999,"15min":5.7629999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.7189999999999999,"5min":2.7189999999999999,"15min":2.7189999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.60499999999999998,"15min":0.58599999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.7309999999999999,"5min":2.7309999999999999,"15min":2.7309999999999999},"last":0.39200000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:53:28 2024","interfaces":[{"interface":"back","average":{"1min":4.4000000000000004,"5min":1.2929999999999999,"15min":0.81000000000000005},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":38.170999999999999,"5min":38.170999999999999,"15min":38.170999999999999},"last":9.7100000000000009},{"interface":"front","average":{"1min":4.2329999999999997,"5min":1.264,"15min":0.79600000000000004},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":38.311999999999998,"5min":38.311999999999998,"15min":38.311999999999998},"last":2.3010000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.36499999999999999},"min":{"1min":0.255,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.5579999999999998,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.9279999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.41399999999999998,"15min":0.35799999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.32600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38300000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.42799999999999999,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":2.0800000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46800000000000003,"15min":0.433},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.5,"15min":0.46000000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.42299999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1909384,"kb_used_data":1608064,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91835768,"statfs":{"total":95995035648,"available":94039826432,"internally_reserved":0,"allocated":1646657536,"data_stored":1646272523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":3.4910000000000001,"5min":1.107,"15min":0.72099999999999997},"min":{"1min":0.216,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":36.880000000000003,"5min":36.880000000000003,"15min":36.880000000000003},"last":4.2930000000000001},{"interface":"front","average":{"1min":3.9319999999999999,"5min":1.1819999999999999,"15min":0.72099999999999997},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":36.954000000000001,"5min":36.954000000000001,"15min":36.954000000000001},"last":0.22900000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.63,"15min":0.55500000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.8159999999999998,"5min":3.8159999999999998,"15min":3.8159999999999998},"last":0.245},{"interface":"front","average":{"1min":2.1720000000000002,"5min":0.84799999999999998,"15min":0.62},"min":{"1min":0.29099999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.7130000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"last":3.2629999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5219999999999998,"5min":1.0069999999999999,"15min":0.69299999999999995},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":18.218,"5min":18.218,"15min":18.218},"last":2.1680000000000001},{"interface":"front","average":{"1min":2.7050000000000001,"5min":0.95199999999999996,"15min":0.65200000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":18.327000000000002,"5min":18.327000000000002,"15min":18.327000000000002},"last":2.7400000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":3.7120000000000002,"5min":1.1890000000000001,"15min":0.752},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":18.251000000000001,"5min":18.251000000000001,"15min":18.251000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":3.0110000000000001,"5min":1.0409999999999999,"15min":0.68799999999999994},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":20.263000000000002,"5min":20.263000000000002,"15min":20.263000000000002},"last":2.601}]},{"osd":4,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.52200000000000002,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.22},"max":{"1min":3.581,"5min":3.581,"15min":3.581},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.48599999999999999,"15min":0.41199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":0.28100000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:53:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46000000000000002,"15min":0.44},"min":{"1min":0.29799999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47899999999999998,"15min":0.45100000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.307}]},{"osd":7,"last update":"Mon Feb 12 16:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.53600000000000003,"15min":0.49399999999999999},"min":{"1min":0.28499999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.2639999999999998,"5min":3.2639999999999998,"15min":3.2639999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.59499999999999997,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.6099999999999999,"5min":3.6099999999999999,"15min":3.6099999999999999},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3333558272,"data_stored":3333554200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2468483072,"data_stored":2468479000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2469724160,"data_stored":2469724160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3707305984,"data_stored":3707305984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2056978432,"data_stored":2056978432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1639055360,"data_stored":1639055360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2059014144,"data_stored":2059010072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2074804224,"data_stored":2074804224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:54:19.947 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:54:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:18.377039+0000 mgr.y (mgr.24481) 182 : cluster [DBG] pgmap v195: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-02-12T16:54:19.995 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:20.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:20.045 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:20.104 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:20.138 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:20.292 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:20.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:20.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:20.430 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:20.433 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:20.558 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:54:20.559 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:20.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:20.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:20.693 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:19.926301+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.33:0/2505788041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:20.694 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:19.936415+0000 mon.a (mon.0) 835 : audit [INF] from='client.24892 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:20.696 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:20.829 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:19.926301+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.33:0/2505788041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:20.829 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:19.936415+0000 mon.a (mon.0) 835 : audit [INF] from='client.24892 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:20.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:19.926301+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.33:0/2505788041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:20.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:19.936415+0000 mon.a (mon.0) 835 : audit [INF] from='client.24892 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:20.940 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:54:21.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:54:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:20] "GET /metrics HTTP/1.1" 200 175411 "" "Prometheus/2.33.4" 2024-02-12T16:54:21.449 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:21.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:21.553 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:21.586 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:21.713 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:20.377438+0000 mgr.y (mgr.24481) 183 : cluster [DBG] pgmap v196: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-12T16:54:21.713 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:20.517718+0000 mon.a (mon.0) 836 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:21.713 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:20.529100+0000 mon.a (mon.0) 837 : audit [INF] from='client.24892 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:54:21.713 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:20.529166+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-12T16:54:21.720 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:21.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:21.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:21.859 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:21.874 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:20.377438+0000 mgr.y (mgr.24481) 183 : cluster [DBG] pgmap v196: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-12T16:54:21.875 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:20.517718+0000 mon.a (mon.0) 836 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:21.875 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:20.529100+0000 mon.a (mon.0) 837 : audit [INF] from='client.24892 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:54:21.875 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:20.529166+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-12T16:54:21.875 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:20.377438+0000 mgr.y (mgr.24481) 183 : cluster [DBG] pgmap v196: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-12T16:54:21.875 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:20.517718+0000 mon.a (mon.0) 836 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:21.875 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:20.529100+0000 mon.a (mon.0) 837 : audit [INF] from='client.24892 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:54:21.875 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:20.529166+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-12T16:54:21.985 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:21.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:22.025 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:54:22.025 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T16:54:22.025 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T16:54:22.025 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T16:54:22.025 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:54:22.090 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:22.124 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:22.279 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:22.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:22.382 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:22.485 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:22.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:54:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:54:22.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:54:22.972 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:22.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:23.081 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:23.146 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:23.158 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:23.260 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:23.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:23.368 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:23.433 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:23.524 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:23.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:23.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:23.661 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:22.378278+0000 mgr.y (mgr.24481) 184 : cluster [DBG] pgmap v198: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 615 B/s rd, 0 B/s wr, 542 op/s 2024-02-12T16:54:23.661 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:54:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:54:23.664 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:23.900 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:54:23.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:22.378278+0000 mgr.y (mgr.24481) 184 : cluster [DBG] pgmap v198: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 615 B/s rd, 0 B/s wr, 542 op/s 2024-02-12T16:54:23.941 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:22.378278+0000 mgr.y (mgr.24481) 184 : cluster [DBG] pgmap v198: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 615 B/s rd, 0 B/s wr, 542 op/s 2024-02-12T16:54:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:24.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:24.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:24.153 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:24.177 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:24.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:24.214 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:54:24.237 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:54:24.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:24.314 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:24.585 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:23.535482+0000 mgr.y (mgr.24481) 185 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:24.585 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:23.565143+0000 mgr.y (mgr.24481) 186 : audit [DBG] from='client.34859 v1:172.21.15.33:0/1045454234' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:54:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:23.535482+0000 mgr.y (mgr.24481) 185 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:25.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:23.565143+0000 mgr.y (mgr.24481) 186 : audit [DBG] from='client.34859 v1:172.21.15.33:0/1045454234' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:54:25.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:23.535482+0000 mgr.y (mgr.24481) 185 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:25.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:23.565143+0000 mgr.y (mgr.24481) 186 : audit [DBG] from='client.34859 v1:172.21.15.33:0/1045454234' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:54:25.297 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-02-12T16:54:22.378038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6308042428,"num_objects":96446,"num_object_clones":0,"num_object_copies":289338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96446,"num_whiteouts":0,"num_read":986,"num_read_kb":738,"num_write":125074,"num_write_kb":7068778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88738,"ondisk_log_size":88738,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21019992,"kb_used_data":18584024,"kb_used_omap":0,"kb_used_meta":2435072,"kb_avail":728941224,"statfs":{"total":767960285184,"available":746435813376,"internally_reserved":0,"allocated":19030040576,"data_stored":19026734828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2493513728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-295436288,"num_objects":-4508,"num_object_clones":0,"num_object_copies":-13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4508,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4508,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.314409"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":82,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090520+0000","last_change":"2024-02-12T16:51:45.053060+0000","last_active":"2024-02-12T16:54:12.090520+0000","last_peered":"2024-02-12T16:54:12.090520+0000","last_clean":"2024-02-12T16:54:12.090520+0000","last_became_active":"2024-02-12T16:49:45.223515+0000","last_became_peered":"2024-02-12T16:49:45.223515+0000","last_unstale":"2024-02-12T16:54:12.090520+0000","last_undegraded":"2024-02-12T16:54:12.090520+0000","last_fullsized":"2024-02-12T16:54:12.090520+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:45.052938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:45.052938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":57,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089671+0000","last_change":"2024-02-12T16:53:49.715871+0000","last_active":"2024-02-12T16:54:12.089671+0000","last_peered":"2024-02-12T16:54:12.089671+0000","last_clean":"2024-02-12T16:54:12.089671+0000","last_became_active":"2024-02-12T16:52:20.563961+0000","last_became_peered":"2024-02-12T16:52:20.563961+0000","last_unstale":"2024-02-12T16:54:12.089671+0000","last_undegraded":"2024-02-12T16:54:12.089671+0000","last_fullsized":"2024-02-12T16:54:12.089671+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:49.715809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.715809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":92,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.088693+0000","last_change":"2024-02-12T16:52:41.990018+0000","last_active":"2024-02-12T16:54:12.088693+0000","last_peered":"2024-02-12T16:54:12.088693+0000","last_clean":"2024-02-12T16:54:12.088693+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:54:12.088693+0000","last_undegraded":"2024-02-12T16:54:12.088693+0000","last_fullsized":"2024-02-12T16:54:12.088693+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:41.989918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.989918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":116,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089407+0000","last_change":"2024-02-12T16:52:47.024635+0000","last_active":"2024-02-12T16:54:12.089407+0000","last_peered":"2024-02-12T16:54:12.089407+0000","last_clean":"2024-02-12T16:54:12.089407+0000","last_became_active":"2024-02-12T16:49:41.185805+0000","last_became_peered":"2024-02-12T16:49:41.185805+0000","last_unstale":"2024-02-12T16:54:12.089407+0000","last_undegraded":"2024-02-12T16:54:12.089407+0000","last_fullsized":"2024-02-12T16:54:12.089407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:52:47.024543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.024543+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":52,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091390+0000","last_change":"2024-02-12T16:53:44.946159+0000","last_active":"2024-02-12T16:54:12.091390+0000","last_peered":"2024-02-12T16:54:12.091390+0000","last_clean":"2024-02-12T16:54:12.091390+0000","last_became_active":"2024-02-12T16:52:20.561617+0000","last_became_peered":"2024-02-12T16:52:20.561617+0000","last_unstale":"2024-02-12T16:54:12.091390+0000","last_undegraded":"2024-02-12T16:54:12.091390+0000","last_fullsized":"2024-02-12T16:54:12.091390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078020+0000","last_change":"2024-02-12T16:52:58.817066+0000","last_active":"2024-02-12T16:54:12.078020+0000","last_peered":"2024-02-12T16:54:12.078020+0000","last_clean":"2024-02-12T16:54:12.078020+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:54:12.078020+0000","last_undegraded":"2024-02-12T16:54:12.078020+0000","last_fullsized":"2024-02-12T16:54:12.078020+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:58.816871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.816871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089298+0000","last_change":"2024-02-12T16:51:38.073505+0000","last_active":"2024-02-12T16:54:12.089298+0000","last_peered":"2024-02-12T16:54:12.089298+0000","last_clean":"2024-02-12T16:54:12.089298+0000","last_became_active":"2024-02-12T16:49:39.102172+0000","last_became_peered":"2024-02-12T16:49:39.102172+0000","last_unstale":"2024-02-12T16:54:12.089298+0000","last_undegraded":"2024-02-12T16:54:12.089298+0000","last_fullsized":"2024-02-12T16:54:12.089298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.073441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.073441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":110,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.055746+0000","last_change":"2024-02-12T16:53:49.924705+0000","last_active":"2024-02-12T16:54:12.055746+0000","last_peered":"2024-02-12T16:54:12.055746+0000","last_clean":"2024-02-12T16:54:12.055746+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:54:12.055746+0000","last_undegraded":"2024-02-12T16:54:12.055746+0000","last_fullsized":"2024-02-12T16:54:12.055746+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:53:49.924639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.924639+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091288+0000","last_change":"2024-02-12T16:52:43.089101+0000","last_active":"2024-02-12T16:54:12.091288+0000","last_peered":"2024-02-12T16:54:12.091288+0000","last_clean":"2024-02-12T16:54:12.091288+0000","last_became_active":"2024-02-12T16:49:45.218184+0000","last_became_peered":"2024-02-12T16:49:45.218184+0000","last_unstale":"2024-02-12T16:54:12.091288+0000","last_undegraded":"2024-02-12T16:54:12.091288+0000","last_fullsized":"2024-02-12T16:54:12.091288+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:43.089004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.089004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.085890+0000","last_change":"2024-02-12T16:52:16.185645+0000","last_active":"2024-02-12T16:54:12.085890+0000","last_peered":"2024-02-12T16:54:12.085890+0000","last_clean":"2024-02-12T16:54:12.085890+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:54:12.085890+0000","last_undegraded":"2024-02-12T16:54:12.085890+0000","last_fullsized":"2024-02-12T16:54:12.085890+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:16.185582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:52:16.185582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":92,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.085889+0000","last_change":"2024-02-12T16:52:11.233227+0000","last_active":"2024-02-12T16:54:12.085889+0000","last_peered":"2024-02-12T16:54:12.085889+0000","last_clean":"2024-02-12T16:54:12.085889+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:54:12.085889+0000","last_undegraded":"2024-02-12T16:54:12.085889+0000","last_fullsized":"2024-02-12T16:54:12.085889+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:11.233135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:52:11.233135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":74,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090642+0000","last_change":"2024-02-12T16:53:45.658415+0000","last_active":"2024-02-12T16:54:12.090642+0000","last_peered":"2024-02-12T16:54:12.090642+0000","last_clean":"2024-02-12T16:54:12.090642+0000","last_became_active":"2024-02-12T16:52:20.563447+0000","last_became_peered":"2024-02-12T16:52:20.563447+0000","last_unstale":"2024-02-12T16:54:12.090642+0000","last_undegraded":"2024-02-12T16:54:12.090642+0000","last_fullsized":"2024-02-12T16:54:12.090642+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:53:45.658292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.658292+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.084870+0000","last_change":"2024-02-12T16:53:51.204401+0000","last_active":"2024-02-12T16:54:12.084870+0000","last_peered":"2024-02-12T16:54:12.084870+0000","last_clean":"2024-02-12T16:54:12.084870+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:54:12.084870+0000","last_undegraded":"2024-02-12T16:54:12.084870+0000","last_fullsized":"2024-02-12T16:54:12.084870+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091673+0000","last_change":"2024-02-12T16:52:01.948362+0000","last_active":"2024-02-12T16:54:12.091673+0000","last_peered":"2024-02-12T16:54:12.091673+0000","last_clean":"2024-02-12T16:54:12.091673+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:54:12.091673+0000","last_undegraded":"2024-02-12T16:54:12.091673+0000","last_fullsized":"2024-02-12T16:54:12.091673+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:01.948286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:52:01.948286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":92,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.085825+0000","last_change":"2024-02-12T16:52:10.220443+0000","last_active":"2024-02-12T16:54:12.085825+0000","last_peered":"2024-02-12T16:54:12.085825+0000","last_clean":"2024-02-12T16:54:12.085825+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:54:12.085825+0000","last_undegraded":"2024-02-12T16:54:12.085825+0000","last_fullsized":"2024-02-12T16:54:12.085825+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:10.220358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:10.220358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":80,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090424+0000","last_change":"2024-02-12T16:52:48.029448+0000","last_active":"2024-02-12T16:54:12.090424+0000","last_peered":"2024-02-12T16:54:12.090424+0000","last_clean":"2024-02-12T16:54:12.090424+0000","last_became_active":"2024-02-12T16:52:20.561538+0000","last_became_peered":"2024-02-12T16:52:20.561538+0000","last_unstale":"2024-02-12T16:54:12.090424+0000","last_undegraded":"2024-02-12T16:54:12.090424+0000","last_fullsized":"2024-02-12T16:54:12.090424+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:48.029049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.029049+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089710+0000","last_change":"2024-02-12T16:52:58.964154+0000","last_active":"2024-02-12T16:54:12.089710+0000","last_peered":"2024-02-12T16:54:12.089710+0000","last_clean":"2024-02-12T16:54:12.089710+0000","last_became_active":"2024-02-12T16:49:45.221559+0000","last_became_peered":"2024-02-12T16:49:45.221559+0000","last_unstale":"2024-02-12T16:54:12.089710+0000","last_undegraded":"2024-02-12T16:54:12.089710+0000","last_fullsized":"2024-02-12T16:54:12.089710+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:58.964060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.964060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":52,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091685+0000","last_change":"2024-02-12T16:52:40.022799+0000","last_active":"2024-02-12T16:54:12.091685+0000","last_peered":"2024-02-12T16:54:12.091685+0000","last_clean":"2024-02-12T16:54:12.091685+0000","last_became_active":"2024-02-12T16:52:20.565295+0000","last_became_peered":"2024-02-12T16:52:20.565295+0000","last_unstale":"2024-02-12T16:54:12.091685+0000","last_undegraded":"2024-02-12T16:54:12.091685+0000","last_fullsized":"2024-02-12T16:54:12.091685+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:40.022594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.022594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090709+0000","last_change":"2024-02-12T16:53:03.947856+0000","last_active":"2024-02-12T16:54:12.090709+0000","last_peered":"2024-02-12T16:54:12.090709+0000","last_clean":"2024-02-12T16:54:12.090709+0000","last_became_active":"2024-02-12T16:49:39.105438+0000","last_became_peered":"2024-02-12T16:49:39.105438+0000","last_unstale":"2024-02-12T16:54:12.090709+0000","last_undegraded":"2024-02-12T16:54:12.090709+0000","last_fullsized":"2024-02-12T16:54:12.090709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:03.947783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:03.947783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":114,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.088966+0000","last_change":"2024-02-12T16:53:49.624190+0000","last_active":"2024-02-12T16:54:12.088966+0000","last_peered":"2024-02-12T16:54:12.088966+0000","last_clean":"2024-02-12T16:54:12.088966+0000","last_became_active":"2024-02-12T16:49:41.181980+0000","last_became_peered":"2024-02-12T16:49:41.181980+0000","last_unstale":"2024-02-12T16:54:12.088966+0000","last_undegraded":"2024-02-12T16:54:12.088966+0000","last_fullsized":"2024-02-12T16:54:12.088966+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:49.624070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.624070+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":85,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.056325+0000","last_change":"2024-02-12T16:52:45.835759+0000","last_active":"2024-02-12T16:54:12.056325+0000","last_peered":"2024-02-12T16:54:12.056325+0000","last_clean":"2024-02-12T16:54:12.056325+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:54:12.056325+0000","last_undegraded":"2024-02-12T16:54:12.056325+0000","last_fullsized":"2024-02-12T16:54:12.056325+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:45.835664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:52:45.835664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.079793+0000","last_change":"2024-02-12T16:53:43.855961+0000","last_active":"2024-02-12T16:54:12.079793+0000","last_peered":"2024-02-12T16:54:12.079793+0000","last_clean":"2024-02-12T16:54:12.079793+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:54:12.079793+0000","last_undegraded":"2024-02-12T16:54:12.079793+0000","last_fullsized":"2024-02-12T16:54:12.079793+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:43.855860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.855860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091509+0000","last_change":"2024-02-12T16:52:58.691294+0000","last_active":"2024-02-12T16:54:12.091509+0000","last_peered":"2024-02-12T16:54:12.091509+0000","last_clean":"2024-02-12T16:54:12.091509+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:54:12.091509+0000","last_undegraded":"2024-02-12T16:54:12.091509+0000","last_fullsized":"2024-02-12T16:54:12.091509+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:58.691233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.691233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":126,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078032+0000","last_change":"2024-02-12T16:53:48.468620+0000","last_active":"2024-02-12T16:54:12.078032+0000","last_peered":"2024-02-12T16:54:12.078032+0000","last_clean":"2024-02-12T16:54:12.078032+0000","last_became_active":"2024-02-12T16:49:41.180319+0000","last_became_peered":"2024-02-12T16:49:41.180319+0000","last_unstale":"2024-02-12T16:54:12.078032+0000","last_undegraded":"2024-02-12T16:54:12.078032+0000","last_fullsized":"2024-02-12T16:54:12.078032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:53:48.468547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.468547+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.079612+0000","last_change":"2024-02-12T16:52:59.833591+0000","last_active":"2024-02-12T16:54:12.079612+0000","last_peered":"2024-02-12T16:54:12.079612+0000","last_clean":"2024-02-12T16:54:12.079612+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:54:12.079612+0000","last_undegraded":"2024-02-12T16:54:12.079612+0000","last_fullsized":"2024-02-12T16:54:12.079612+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:59.833484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.833484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.088432+0000","last_change":"2024-02-12T16:53:04.085929+0000","last_active":"2024-02-12T16:54:12.088432+0000","last_peered":"2024-02-12T16:54:12.088432+0000","last_clean":"2024-02-12T16:54:12.088432+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:54:12.088432+0000","last_undegraded":"2024-02-12T16:54:12.088432+0000","last_fullsized":"2024-02-12T16:54:12.088432+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:04.085854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.085854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.088919+0000","last_change":"2024-02-12T16:53:01.970821+0000","last_active":"2024-02-12T16:54:12.088919+0000","last_peered":"2024-02-12T16:54:12.088919+0000","last_clean":"2024-02-12T16:54:12.088919+0000","last_became_active":"2024-02-12T16:49:39.101492+0000","last_became_peered":"2024-02-12T16:49:39.101492+0000","last_unstale":"2024-02-12T16:54:12.088919+0000","last_undegraded":"2024-02-12T16:54:12.088919+0000","last_fullsized":"2024-02-12T16:54:12.088919+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.970730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.970730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":110,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090523+0000","last_change":"2024-02-12T16:52:52.604196+0000","last_active":"2024-02-12T16:54:12.090523+0000","last_peered":"2024-02-12T16:54:12.090523+0000","last_clean":"2024-02-12T16:54:12.090523+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:54:12.090523+0000","last_undegraded":"2024-02-12T16:54:12.090523+0000","last_fullsized":"2024-02-12T16:54:12.090523+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:52:52.604107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.604107+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086148+0000","last_change":"2024-02-12T16:53:04.169987+0000","last_active":"2024-02-12T16:54:12.086148+0000","last_peered":"2024-02-12T16:54:12.086148+0000","last_clean":"2024-02-12T16:54:12.086148+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:54:12.086148+0000","last_undegraded":"2024-02-12T16:54:12.086148+0000","last_fullsized":"2024-02-12T16:54:12.086148+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:53:04.169892+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.169892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":156,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.079415+0000","last_change":"2024-02-12T16:52:44.602751+0000","last_active":"2024-02-12T16:54:14.079415+0000","last_peered":"2024-02-12T16:54:14.079415+0000","last_clean":"2024-02-12T16:54:14.079415+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:54:14.079415+0000","last_undegraded":"2024-02-12T16:54:14.079415+0000","last_fullsized":"2024-02-12T16:54:14.079415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:44.602656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.602656+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091278+0000","last_change":"2024-02-12T16:52:47.997796+0000","last_active":"2024-02-12T16:54:12.091278+0000","last_peered":"2024-02-12T16:54:12.091278+0000","last_clean":"2024-02-12T16:54:12.091278+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:54:12.091278+0000","last_undegraded":"2024-02-12T16:54:12.091278+0000","last_fullsized":"2024-02-12T16:54:12.091278+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:47.997708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.997708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":105,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.088334+0000","last_change":"2024-02-12T16:51:57.693428+0000","last_active":"2024-02-12T16:54:12.088334+0000","last_peered":"2024-02-12T16:54:12.088334+0000","last_clean":"2024-02-12T16:54:12.088334+0000","last_became_active":"2024-02-12T16:49:41.669589+0000","last_became_peered":"2024-02-12T16:49:41.669589+0000","last_unstale":"2024-02-12T16:54:12.088334+0000","last_undegraded":"2024-02-12T16:54:12.088334+0000","last_fullsized":"2024-02-12T16:54:12.088334+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:57.693352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:57.693352+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090983+0000","last_change":"2024-02-12T16:54:00.665342+0000","last_active":"2024-02-12T16:54:12.090983+0000","last_peered":"2024-02-12T16:54:12.090983+0000","last_clean":"2024-02-12T16:54:12.090983+0000","last_became_active":"2024-02-12T16:49:45.222341+0000","last_became_peered":"2024-02-12T16:49:45.222341+0000","last_unstale":"2024-02-12T16:54:12.090983+0000","last_undegraded":"2024-02-12T16:54:12.090983+0000","last_fullsized":"2024-02-12T16:54:12.090983+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.665249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.665249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086879+0000","last_change":"2024-02-12T16:53:05.141324+0000","last_active":"2024-02-12T16:54:12.086879+0000","last_peered":"2024-02-12T16:54:12.086879+0000","last_clean":"2024-02-12T16:54:12.086879+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:54:12.086879+0000","last_undegraded":"2024-02-12T16:54:12.086879+0000","last_fullsized":"2024-02-12T16:54:12.086879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090415+0000","last_change":"2024-02-12T16:52:51.634372+0000","last_active":"2024-02-12T16:54:12.090415+0000","last_peered":"2024-02-12T16:54:12.090415+0000","last_clean":"2024-02-12T16:54:12.090415+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:54:12.090415+0000","last_undegraded":"2024-02-12T16:54:12.090415+0000","last_fullsized":"2024-02-12T16:54:12.090415+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:51.634281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.634281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":110,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089263+0000","last_change":"2024-02-12T16:51:57.052497+0000","last_active":"2024-02-12T16:54:12.089263+0000","last_peered":"2024-02-12T16:54:12.089263+0000","last_clean":"2024-02-12T16:54:12.089263+0000","last_became_active":"2024-02-12T16:49:41.669704+0000","last_became_peered":"2024-02-12T16:49:41.669704+0000","last_unstale":"2024-02-12T16:54:12.089263+0000","last_undegraded":"2024-02-12T16:54:12.089263+0000","last_fullsized":"2024-02-12T16:54:12.089263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:57.052428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:57.052428+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":86,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.085759+0000","last_change":"2024-02-12T16:52:55.241717+0000","last_active":"2024-02-12T16:54:12.085759+0000","last_peered":"2024-02-12T16:54:12.085759+0000","last_clean":"2024-02-12T16:54:12.085759+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:54:12.085759+0000","last_undegraded":"2024-02-12T16:54:12.085759+0000","last_fullsized":"2024-02-12T16:54:12.085759+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.241622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.241622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089907+0000","last_change":"2024-02-12T16:53:48.719406+0000","last_active":"2024-02-12T16:54:12.089907+0000","last_peered":"2024-02-12T16:54:12.089907+0000","last_clean":"2024-02-12T16:54:12.089907+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:54:12.089907+0000","last_undegraded":"2024-02-12T16:54:12.089907+0000","last_fullsized":"2024-02-12T16:54:12.089907+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.719344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.719344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.092098+0000","last_change":"2024-02-12T16:52:49.986507+0000","last_active":"2024-02-12T16:54:12.092098+0000","last_peered":"2024-02-12T16:54:12.092098+0000","last_clean":"2024-02-12T16:54:12.092098+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:54:12.092098+0000","last_undegraded":"2024-02-12T16:54:12.092098+0000","last_fullsized":"2024-02-12T16:54:12.092098+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.986403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.986403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089451+0000","last_change":"2024-02-12T16:54:01.670202+0000","last_active":"2024-02-12T16:54:12.089451+0000","last_peered":"2024-02-12T16:54:12.089451+0000","last_clean":"2024-02-12T16:54:12.089451+0000","last_became_active":"2024-02-12T16:49:41.184044+0000","last_became_peered":"2024-02-12T16:49:41.184044+0000","last_unstale":"2024-02-12T16:54:12.089451+0000","last_undegraded":"2024-02-12T16:54:12.089451+0000","last_fullsized":"2024-02-12T16:54:12.089451+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":82,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.088757+0000","last_change":"2024-02-12T16:51:42.040901+0000","last_active":"2024-02-12T16:54:12.088757+0000","last_peered":"2024-02-12T16:54:12.088757+0000","last_clean":"2024-02-12T16:54:12.088757+0000","last_became_active":"2024-02-12T16:49:45.220469+0000","last_became_peered":"2024-02-12T16:49:45.220469+0000","last_unstale":"2024-02-12T16:54:12.088757+0000","last_undegraded":"2024-02-12T16:54:12.088757+0000","last_fullsized":"2024-02-12T16:54:12.088757+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:51:42.040828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":95,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.079334+0000","last_change":"2024-02-12T16:54:02.785572+0000","last_active":"2024-02-12T16:54:12.079334+0000","last_peered":"2024-02-12T16:54:12.079334+0000","last_clean":"2024-02-12T16:54:12.079334+0000","last_became_active":"2024-02-12T16:49:43.204615+0000","last_became_peered":"2024-02-12T16:49:43.204615+0000","last_unstale":"2024-02-12T16:54:12.079334+0000","last_undegraded":"2024-02-12T16:54:12.079334+0000","last_fullsized":"2024-02-12T16:54:12.079334+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.785477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.785477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.079868+0000","last_change":"2024-02-12T16:54:01.773110+0000","last_active":"2024-02-12T16:54:12.079868+0000","last_peered":"2024-02-12T16:54:12.079868+0000","last_clean":"2024-02-12T16:54:12.079868+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:54:12.079868+0000","last_undegraded":"2024-02-12T16:54:12.079868+0000","last_fullsized":"2024-02-12T16:54:12.079868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":110,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.055880+0000","last_change":"2024-02-12T16:53:53.872376+0000","last_active":"2024-02-12T16:54:12.055880+0000","last_peered":"2024-02-12T16:54:12.055880+0000","last_clean":"2024-02-12T16:54:12.055880+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:54:12.055880+0000","last_undegraded":"2024-02-12T16:54:12.055880+0000","last_fullsized":"2024-02-12T16:54:12.055880+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:52:38.896872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:38.896872+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":81,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.084482+0000","last_change":"2024-02-12T16:51:28.886399+0000","last_active":"2024-02-12T16:54:12.084482+0000","last_peered":"2024-02-12T16:54:12.084482+0000","last_clean":"2024-02-12T16:54:12.084482+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:54:12.084482+0000","last_undegraded":"2024-02-12T16:54:12.084482+0000","last_fullsized":"2024-02-12T16:54:12.084482+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:51:28.886321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":84,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090774+0000","last_change":"2024-02-12T16:51:39.040277+0000","last_active":"2024-02-12T16:54:12.090774+0000","last_peered":"2024-02-12T16:54:12.090774+0000","last_clean":"2024-02-12T16:54:12.090774+0000","last_became_active":"2024-02-12T16:49:43.208343+0000","last_became_peered":"2024-02-12T16:49:43.208343+0000","last_unstale":"2024-02-12T16:54:12.090774+0000","last_undegraded":"2024-02-12T16:54:12.090774+0000","last_fullsized":"2024-02-12T16:54:12.090774+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.040203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.040203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090086+0000","last_change":"2024-02-12T16:52:48.590123+0000","last_active":"2024-02-12T16:54:12.090086+0000","last_peered":"2024-02-12T16:54:12.090086+0000","last_clean":"2024-02-12T16:54:12.090086+0000","last_became_active":"2024-02-12T16:49:39.107471+0000","last_became_peered":"2024-02-12T16:49:39.107471+0000","last_unstale":"2024-02-12T16:54:12.090086+0000","last_undegraded":"2024-02-12T16:54:12.090086+0000","last_fullsized":"2024-02-12T16:54:12.090086+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.590060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.590060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":112,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.079259+0000","last_change":"2024-02-12T16:52:06.722431+0000","last_active":"2024-02-12T16:54:12.079259+0000","last_peered":"2024-02-12T16:54:12.079259+0000","last_clean":"2024-02-12T16:54:12.079259+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:54:12.079259+0000","last_undegraded":"2024-02-12T16:54:12.079259+0000","last_fullsized":"2024-02-12T16:54:12.079259+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:52:06.722345+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:52:06.722345+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":52,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078241+0000","last_change":"2024-02-12T16:52:59.296102+0000","last_active":"2024-02-12T16:54:12.078241+0000","last_peered":"2024-02-12T16:54:12.078241+0000","last_clean":"2024-02-12T16:54:12.078241+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T16:54:12.078241+0000","last_undegraded":"2024-02-12T16:54:12.078241+0000","last_fullsized":"2024-02-12T16:54:12.078241+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:59.296023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.296023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.069380+0000","last_change":"2024-02-12T16:51:44.091513+0000","last_active":"2024-02-12T16:54:14.069380+0000","last_peered":"2024-02-12T16:54:14.069380+0000","last_clean":"2024-02-12T16:54:14.069380+0000","last_became_active":"2024-02-12T16:49:43.206068+0000","last_became_peered":"2024-02-12T16:49:43.206068+0000","last_unstale":"2024-02-12T16:54:14.069380+0000","last_undegraded":"2024-02-12T16:54:14.069380+0000","last_fullsized":"2024-02-12T16:54:14.069380+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:51:44.091436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:44.091436+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.083976+0000","last_change":"2024-02-12T16:51:25.913214+0000","last_active":"2024-02-12T16:54:12.083976+0000","last_peered":"2024-02-12T16:54:12.083976+0000","last_clean":"2024-02-12T16:54:12.083976+0000","last_became_active":"2024-02-12T16:49:39.103401+0000","last_became_peered":"2024-02-12T16:49:39.103401+0000","last_unstale":"2024-02-12T16:54:12.083976+0000","last_undegraded":"2024-02-12T16:54:12.083976+0000","last_fullsized":"2024-02-12T16:54:12.083976+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:25.913145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.913145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":103,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091983+0000","last_change":"2024-02-12T16:51:25.028420+0000","last_active":"2024-02-12T16:54:12.091983+0000","last_peered":"2024-02-12T16:54:12.091983+0000","last_clean":"2024-02-12T16:54:12.091983+0000","last_became_active":"2024-02-12T16:49:41.669427+0000","last_became_peered":"2024-02-12T16:49:41.669427+0000","last_unstale":"2024-02-12T16:54:12.091983+0000","last_undegraded":"2024-02-12T16:54:12.091983+0000","last_fullsized":"2024-02-12T16:54:12.091983+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:51:25.028339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:25.028339+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":86,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.087841+0000","last_change":"2024-02-12T16:52:55.105623+0000","last_active":"2024-02-12T16:54:12.087841+0000","last_peered":"2024-02-12T16:54:12.087841+0000","last_clean":"2024-02-12T16:54:12.087841+0000","last_became_active":"2024-02-12T16:49:45.212245+0000","last_became_peered":"2024-02-12T16:49:45.212245+0000","last_unstale":"2024-02-12T16:54:12.087841+0000","last_undegraded":"2024-02-12T16:54:12.087841+0000","last_fullsized":"2024-02-12T16:54:12.087841+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.105539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.105539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":156,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.069965+0000","last_change":"2024-02-12T16:54:00.112924+0000","last_active":"2024-02-12T16:54:14.069965+0000","last_peered":"2024-02-12T16:54:14.069965+0000","last_clean":"2024-02-12T16:54:14.069965+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:54:14.069965+0000","last_undegraded":"2024-02-12T16:54:14.069965+0000","last_fullsized":"2024-02-12T16:54:14.069965+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":89,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091470+0000","last_change":"2024-02-12T16:51:20.103380+0000","last_active":"2024-02-12T16:54:12.091470+0000","last_peered":"2024-02-12T16:54:12.091470+0000","last_clean":"2024-02-12T16:54:12.091470+0000","last_became_active":"2024-02-12T16:49:39.104936+0000","last_became_peered":"2024-02-12T16:49:39.104936+0000","last_unstale":"2024-02-12T16:54:12.091470+0000","last_undegraded":"2024-02-12T16:54:12.091470+0000","last_fullsized":"2024-02-12T16:54:12.091470+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:20.103297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:20.103297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091614+0000","last_change":"2024-02-12T16:54:03.616828+0000","last_active":"2024-02-12T16:54:12.091614+0000","last_peered":"2024-02-12T16:54:12.091614+0000","last_clean":"2024-02-12T16:54:12.091614+0000","last_became_active":"2024-02-12T16:51:04.116639+0000","last_became_peered":"2024-02-12T16:51:04.116639+0000","last_unstale":"2024-02-12T16:54:12.091614+0000","last_undegraded":"2024-02-12T16:54:12.091614+0000","last_fullsized":"2024-02-12T16:54:12.091614+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:03.616717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.616717+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.085808+0000","last_change":"2024-02-12T16:53:51.532654+0000","last_active":"2024-02-12T16:54:12.085808+0000","last_peered":"2024-02-12T16:54:12.085808+0000","last_clean":"2024-02-12T16:54:12.085808+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:54:12.085808+0000","last_undegraded":"2024-02-12T16:54:12.085808+0000","last_fullsized":"2024-02-12T16:54:12.085808+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091855+0000","last_change":"2024-02-12T16:52:15.844557+0000","last_active":"2024-02-12T16:54:12.091855+0000","last_peered":"2024-02-12T16:54:12.091855+0000","last_clean":"2024-02-12T16:54:12.091855+0000","last_became_active":"2024-02-12T16:49:43.201595+0000","last_became_peered":"2024-02-12T16:49:43.201595+0000","last_unstale":"2024-02-12T16:54:12.091855+0000","last_undegraded":"2024-02-12T16:54:12.091855+0000","last_fullsized":"2024-02-12T16:54:12.091855+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.844487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.844487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089983+0000","last_change":"2024-02-12T16:52:43.579914+0000","last_active":"2024-02-12T16:54:12.089983+0000","last_peered":"2024-02-12T16:54:12.089983+0000","last_clean":"2024-02-12T16:54:12.089983+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:54:12.089983+0000","last_undegraded":"2024-02-12T16:54:12.089983+0000","last_fullsized":"2024-02-12T16:54:12.089983+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:43.579811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.579811+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":100,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089792+0000","last_change":"2024-02-12T16:53:57.641513+0000","last_active":"2024-02-12T16:54:12.089792+0000","last_peered":"2024-02-12T16:54:12.089792+0000","last_clean":"2024-02-12T16:54:12.089792+0000","last_became_active":"2024-02-12T16:49:41.669651+0000","last_became_peered":"2024-02-12T16:49:41.669651+0000","last_unstale":"2024-02-12T16:54:12.089792+0000","last_undegraded":"2024-02-12T16:54:12.089792+0000","last_fullsized":"2024-02-12T16:54:12.089792+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.077697+0000","last_change":"2024-02-12T16:52:39.278662+0000","last_active":"2024-02-12T16:54:12.077697+0000","last_peered":"2024-02-12T16:54:12.077697+0000","last_clean":"2024-02-12T16:54:12.077697+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:54:12.077697+0000","last_undegraded":"2024-02-12T16:54:12.077697+0000","last_fullsized":"2024-02-12T16:54:12.077697+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:39.278592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:39.278592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":91,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089866+0000","last_change":"2024-02-12T16:53:41.740985+0000","last_active":"2024-02-12T16:54:12.089866+0000","last_peered":"2024-02-12T16:54:12.089866+0000","last_clean":"2024-02-12T16:54:12.089866+0000","last_became_active":"2024-02-12T16:49:43.207454+0000","last_became_peered":"2024-02-12T16:49:43.207454+0000","last_unstale":"2024-02-12T16:54:12.089866+0000","last_undegraded":"2024-02-12T16:54:12.089866+0000","last_fullsized":"2024-02-12T16:54:12.089866+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:41.740892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:53:41.740892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.061120+0000","last_change":"2024-02-12T16:52:53.330290+0000","last_active":"2024-02-12T16:54:12.061120+0000","last_peered":"2024-02-12T16:54:12.061120+0000","last_clean":"2024-02-12T16:54:12.061120+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:54:12.061120+0000","last_undegraded":"2024-02-12T16:54:12.061120+0000","last_fullsized":"2024-02-12T16:54:12.061120+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:53.330194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:53.330194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":110,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.088086+0000","last_change":"2024-02-12T16:54:02.181218+0000","last_active":"2024-02-12T16:54:12.088086+0000","last_peered":"2024-02-12T16:54:12.088086+0000","last_clean":"2024-02-12T16:54:12.088086+0000","last_became_active":"2024-02-12T16:49:41.668665+0000","last_became_peered":"2024-02-12T16:49:41.668665+0000","last_unstale":"2024-02-12T16:54:12.088086+0000","last_undegraded":"2024-02-12T16:54:12.088086+0000","last_fullsized":"2024-02-12T16:54:12.088086+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"133'7801","reported_seq":7849,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.351537+0000","last_change":"2024-02-12T16:52:16.552919+0000","last_active":"2024-02-12T16:54:18.351537+0000","last_peered":"2024-02-12T16:54:18.351537+0000","last_clean":"2024-02-12T16:54:18.351537+0000","last_became_active":"2024-02-12T16:52:16.552631+0000","last_became_peered":"2024-02-12T16:52:16.552631+0000","last_unstale":"2024-02-12T16:54:18.351537+0000","last_undegraded":"2024-02-12T16:54:18.351537+0000","last_fullsized":"2024-02-12T16:54:18.351537+0000","mapping_epoch":93,"log_start":"111'3100","ondisk_log_start":"111'3100","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4701,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":393412608,"num_objects":6003,"num_object_clones":0,"num_object_copies":18009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7801,"num_write_kb":441728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":82,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090841+0000","last_change":"2024-02-12T16:51:33.032870+0000","last_active":"2024-02-12T16:54:12.090841+0000","last_peered":"2024-02-12T16:54:12.090841+0000","last_clean":"2024-02-12T16:54:12.090841+0000","last_became_active":"2024-02-12T16:49:45.220328+0000","last_became_peered":"2024-02-12T16:49:45.220328+0000","last_unstale":"2024-02-12T16:54:12.090841+0000","last_undegraded":"2024-02-12T16:54:12.090841+0000","last_fullsized":"2024-02-12T16:54:12.090841+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:33.032784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:33.032784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":115,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.056353+0000","last_change":"2024-02-12T16:53:48.921052+0000","last_active":"2024-02-12T16:54:12.056353+0000","last_peered":"2024-02-12T16:54:12.056353+0000","last_clean":"2024-02-12T16:54:12.056353+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:54:12.056353+0000","last_undegraded":"2024-02-12T16:54:12.056353+0000","last_fullsized":"2024-02-12T16:54:12.056353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:53:48.920976+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.920976+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078257+0000","last_change":"2024-02-12T16:52:44.868547+0000","last_active":"2024-02-12T16:54:12.078257+0000","last_peered":"2024-02-12T16:54:12.078257+0000","last_clean":"2024-02-12T16:54:12.078257+0000","last_became_active":"2024-02-12T16:52:20.560975+0000","last_became_peered":"2024-02-12T16:52:20.560975+0000","last_unstale":"2024-02-12T16:54:12.078257+0000","last_undegraded":"2024-02-12T16:54:12.078257+0000","last_fullsized":"2024-02-12T16:54:12.078257+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:52:44.868438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.868438+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.061201+0000","last_change":"2024-02-12T16:52:48.411110+0000","last_active":"2024-02-12T16:54:12.061201+0000","last_peered":"2024-02-12T16:54:12.061201+0000","last_clean":"2024-02-12T16:54:12.061201+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:54:12.061201+0000","last_undegraded":"2024-02-12T16:54:12.061201+0000","last_fullsized":"2024-02-12T16:54:12.061201+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.411008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.411008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"133'7734","reported_seq":7781,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.317854+0000","last_change":"2024-02-12T16:52:20.564464+0000","last_active":"2024-02-12T16:54:18.317854+0000","last_peered":"2024-02-12T16:54:18.317854+0000","last_clean":"2024-02-12T16:54:18.317854+0000","last_became_active":"2024-02-12T16:52:20.564223+0000","last_became_peered":"2024-02-12T16:52:20.564223+0000","last_unstale":"2024-02-12T16:54:18.317854+0000","last_undegraded":"2024-02-12T16:54:18.317854+0000","last_fullsized":"2024-02-12T16:54:18.317854+0000","mapping_epoch":97,"log_start":"111'3000","ondisk_log_start":"111'3000","created":93,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4734,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391643136,"num_objects":5976,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7734,"num_write_kb":438720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":86,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.084557+0000","last_change":"2024-02-12T16:53:42.249070+0000","last_active":"2024-02-12T16:54:12.084557+0000","last_peered":"2024-02-12T16:54:12.084557+0000","last_clean":"2024-02-12T16:54:12.084557+0000","last_became_active":"2024-02-12T16:49:45.217626+0000","last_became_peered":"2024-02-12T16:49:45.217626+0000","last_unstale":"2024-02-12T16:54:12.084557+0000","last_undegraded":"2024-02-12T16:54:12.084557+0000","last_fullsized":"2024-02-12T16:54:12.084557+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.248970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.248970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078731+0000","last_change":"2024-02-12T16:53:40.916900+0000","last_active":"2024-02-12T16:54:12.078731+0000","last_peered":"2024-02-12T16:54:12.078731+0000","last_clean":"2024-02-12T16:54:12.078731+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:54:12.078731+0000","last_undegraded":"2024-02-12T16:54:12.078731+0000","last_fullsized":"2024-02-12T16:54:12.078731+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:40.916831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:40.916831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":78,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.076859+0000","last_change":"2024-02-12T16:52:11.419489+0000","last_active":"2024-02-12T16:54:12.076859+0000","last_peered":"2024-02-12T16:54:12.076859+0000","last_clean":"2024-02-12T16:54:12.076859+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:54:12.076859+0000","last_undegraded":"2024-02-12T16:54:12.076859+0000","last_fullsized":"2024-02-12T16:54:12.076859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:52:11.419411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:11.419411+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.080564+0000","last_change":"2024-02-12T16:53:59.173126+0000","last_active":"2024-02-12T16:54:14.080564+0000","last_peered":"2024-02-12T16:54:14.080564+0000","last_clean":"2024-02-12T16:54:14.080564+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:54:14.080564+0000","last_undegraded":"2024-02-12T16:54:14.080564+0000","last_fullsized":"2024-02-12T16:54:14.080564+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"133'7925","reported_seq":7978,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.261552+0000","last_change":"2024-02-12T16:53:55.684404+0000","last_active":"2024-02-12T16:54:18.261552+0000","last_peered":"2024-02-12T16:54:18.261552+0000","last_clean":"2024-02-12T16:54:18.261552+0000","last_became_active":"2024-02-12T16:52:16.557444+0000","last_became_peered":"2024-02-12T16:52:16.557444+0000","last_unstale":"2024-02-12T16:54:18.261552+0000","last_undegraded":"2024-02-12T16:54:18.261552+0000","last_fullsized":"2024-02-12T16:54:18.261552+0000","mapping_epoch":93,"log_start":"106'1800","ondisk_log_start":"106'1800","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'6594","last_scrub_stamp":"2024-02-12T16:53:55.684320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:53:55.684320+0000","log_size":6125,"ondisk_log_size":6125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":398393344,"num_objects":6079,"num_object_clones":0,"num_object_copies":18237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7925,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089387+0000","last_change":"2024-02-12T16:53:56.624041+0000","last_active":"2024-02-12T16:54:12.089387+0000","last_peered":"2024-02-12T16:54:12.089387+0000","last_clean":"2024-02-12T16:54:12.089387+0000","last_became_active":"2024-02-12T16:49:45.215520+0000","last_became_peered":"2024-02-12T16:49:45.215520+0000","last_unstale":"2024-02-12T16:54:12.089387+0000","last_undegraded":"2024-02-12T16:54:12.089387+0000","last_fullsized":"2024-02-12T16:54:12.089387+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:56.623922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.623922+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091516+0000","last_change":"2024-02-12T16:52:06.907569+0000","last_active":"2024-02-12T16:54:12.091516+0000","last_peered":"2024-02-12T16:54:12.091516+0000","last_clean":"2024-02-12T16:54:12.091516+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:54:12.091516+0000","last_undegraded":"2024-02-12T16:54:12.091516+0000","last_fullsized":"2024-02-12T16:54:12.091516+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:06.907496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:06.907496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":196,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.068201+0000","last_change":"2024-02-12T16:53:59.751101+0000","last_active":"2024-02-12T16:54:14.068201+0000","last_peered":"2024-02-12T16:54:14.068201+0000","last_clean":"2024-02-12T16:54:14.068201+0000","last_became_active":"2024-02-12T16:49:41.179610+0000","last_became_peered":"2024-02-12T16:49:41.179610+0000","last_unstale":"2024-02-12T16:54:14.068201+0000","last_undegraded":"2024-02-12T16:54:14.068201+0000","last_fullsized":"2024-02-12T16:54:14.068201+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-12T16:53:59.751031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.751031+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.088052+0000","last_change":"2024-02-12T16:53:01.103805+0000","last_active":"2024-02-12T16:54:12.088052+0000","last_peered":"2024-02-12T16:54:12.088052+0000","last_clean":"2024-02-12T16:54:12.088052+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:54:12.088052+0000","last_undegraded":"2024-02-12T16:54:12.088052+0000","last_fullsized":"2024-02-12T16:54:12.088052+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.103703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.103703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"133'7735","reported_seq":7782,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.287622+0000","last_change":"2024-02-12T16:52:16.558183+0000","last_active":"2024-02-12T16:54:18.287622+0000","last_peered":"2024-02-12T16:54:18.287622+0000","last_clean":"2024-02-12T16:54:18.287622+0000","last_became_active":"2024-02-12T16:52:16.557890+0000","last_became_peered":"2024-02-12T16:52:16.557890+0000","last_unstale":"2024-02-12T16:54:18.287622+0000","last_undegraded":"2024-02-12T16:54:18.287622+0000","last_fullsized":"2024-02-12T16:54:18.287622+0000","mapping_epoch":93,"log_start":"107'2200","ondisk_log_start":"107'2200","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5535,"ondisk_log_size":5535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":393674752,"num_objects":6007,"num_object_clones":0,"num_object_copies":18021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7735,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086034+0000","last_change":"2024-02-12T16:54:00.551098+0000","last_active":"2024-02-12T16:54:12.086034+0000","last_peered":"2024-02-12T16:54:12.086034+0000","last_clean":"2024-02-12T16:54:12.086034+0000","last_became_active":"2024-02-12T16:49:45.221460+0000","last_became_peered":"2024-02-12T16:49:45.221460+0000","last_unstale":"2024-02-12T16:54:12.086034+0000","last_undegraded":"2024-02-12T16:54:12.086034+0000","last_fullsized":"2024-02-12T16:54:12.086034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":369,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:16.985013+0000","last_change":"2024-02-12T16:52:02.743093+0000","last_active":"2024-02-12T16:54:16.985013+0000","last_peered":"2024-02-12T16:54:16.985013+0000","last_clean":"2024-02-12T16:54:16.985013+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:54:16.985013+0000","last_undegraded":"2024-02-12T16:54:16.985013+0000","last_fullsized":"2024-02-12T16:54:16.985013+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:52:02.742977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:02.742977+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":272,"num_read_kb":267,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.061133+0000","last_change":"2024-02-12T16:52:40.271277+0000","last_active":"2024-02-12T16:54:12.061133+0000","last_peered":"2024-02-12T16:54:12.061133+0000","last_clean":"2024-02-12T16:54:12.061133+0000","last_became_active":"2024-02-12T16:49:39.105685+0000","last_became_peered":"2024-02-12T16:49:39.105685+0000","last_unstale":"2024-02-12T16:54:12.061133+0000","last_undegraded":"2024-02-12T16:54:12.061133+0000","last_fullsized":"2024-02-12T16:54:12.061133+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:40.271192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.271192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":124,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.077719+0000","last_change":"2024-02-12T16:53:04.923477+0000","last_active":"2024-02-12T16:54:12.077719+0000","last_peered":"2024-02-12T16:54:12.077719+0000","last_clean":"2024-02-12T16:54:12.077719+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:54:12.077719+0000","last_undegraded":"2024-02-12T16:54:12.077719+0000","last_fullsized":"2024-02-12T16:54:12.077719+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.923375+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.080441+0000","last_change":"2024-02-12T16:52:45.314737+0000","last_active":"2024-02-12T16:54:12.080441+0000","last_peered":"2024-02-12T16:54:12.080441+0000","last_clean":"2024-02-12T16:54:12.080441+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:54:12.080441+0000","last_undegraded":"2024-02-12T16:54:12.080441+0000","last_fullsized":"2024-02-12T16:54:12.080441+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:45.314637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:45.314637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"133'7796","reported_seq":7843,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.360574+0000","last_change":"2024-02-12T16:52:16.560931+0000","last_active":"2024-02-12T16:54:18.360574+0000","last_peered":"2024-02-12T16:54:18.360574+0000","last_clean":"2024-02-12T16:54:18.360574+0000","last_became_active":"2024-02-12T16:52:16.560633+0000","last_became_peered":"2024-02-12T16:52:16.560633+0000","last_unstale":"2024-02-12T16:54:18.360574+0000","last_undegraded":"2024-02-12T16:54:18.360574+0000","last_fullsized":"2024-02-12T16:54:18.360574+0000","mapping_epoch":93,"log_start":"108'2300","ondisk_log_start":"108'2300","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5496,"ondisk_log_size":5496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":399245312,"num_objects":6092,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":444416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.092496+0000","last_change":"2024-02-12T16:53:42.946856+0000","last_active":"2024-02-12T16:54:12.092496+0000","last_peered":"2024-02-12T16:54:12.092496+0000","last_clean":"2024-02-12T16:54:12.092496+0000","last_became_active":"2024-02-12T16:49:45.213152+0000","last_became_peered":"2024-02-12T16:49:45.213152+0000","last_unstale":"2024-02-12T16:54:12.092496+0000","last_undegraded":"2024-02-12T16:54:12.092496+0000","last_fullsized":"2024-02-12T16:54:12.092496+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089171+0000","last_change":"2024-02-12T16:52:55.967424+0000","last_active":"2024-02-12T16:54:12.089171+0000","last_peered":"2024-02-12T16:54:12.089171+0000","last_clean":"2024-02-12T16:54:12.089171+0000","last_became_active":"2024-02-12T16:49:39.105761+0000","last_became_peered":"2024-02-12T16:49:39.105761+0000","last_unstale":"2024-02-12T16:54:12.089171+0000","last_undegraded":"2024-02-12T16:54:12.089171+0000","last_fullsized":"2024-02-12T16:54:12.089171+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.967335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.092316+0000","last_change":"2024-02-12T16:52:57.985677+0000","last_active":"2024-02-12T16:54:12.092316+0000","last_peered":"2024-02-12T16:54:12.092316+0000","last_clean":"2024-02-12T16:54:12.092316+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:54:12.092316+0000","last_undegraded":"2024-02-12T16:54:12.092316+0000","last_fullsized":"2024-02-12T16:54:12.092316+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:57.985584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:57.985584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":95,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086800+0000","last_change":"2024-02-12T16:51:37.184984+0000","last_active":"2024-02-12T16:54:12.086800+0000","last_peered":"2024-02-12T16:54:12.086800+0000","last_clean":"2024-02-12T16:54:12.086800+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:54:12.086800+0000","last_undegraded":"2024-02-12T16:54:12.086800+0000","last_fullsized":"2024-02-12T16:54:12.086800+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:37.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:51:37.184907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":118,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090366+0000","last_change":"2024-02-12T16:53:58.740196+0000","last_active":"2024-02-12T16:54:12.090366+0000","last_peered":"2024-02-12T16:54:12.090366+0000","last_clean":"2024-02-12T16:54:12.090366+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:54:12.090366+0000","last_undegraded":"2024-02-12T16:54:12.090366+0000","last_fullsized":"2024-02-12T16:54:12.090366+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:53:58.740122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.740122+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086015+0000","last_change":"2024-02-12T16:52:12.200439+0000","last_active":"2024-02-12T16:54:12.086015+0000","last_peered":"2024-02-12T16:54:12.086015+0000","last_clean":"2024-02-12T16:54:12.086015+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:54:12.086015+0000","last_undegraded":"2024-02-12T16:54:12.086015+0000","last_fullsized":"2024-02-12T16:54:12.086015+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:12.200355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:12.200355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"133'7883","reported_seq":7931,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.261691+0000","last_change":"2024-02-12T16:52:16.559957+0000","last_active":"2024-02-12T16:54:18.261691+0000","last_peered":"2024-02-12T16:54:18.261691+0000","last_clean":"2024-02-12T16:54:18.261691+0000","last_became_active":"2024-02-12T16:52:16.559699+0000","last_became_peered":"2024-02-12T16:52:16.559699+0000","last_unstale":"2024-02-12T16:54:18.261691+0000","last_undegraded":"2024-02-12T16:54:18.261691+0000","last_fullsized":"2024-02-12T16:54:18.261691+0000","mapping_epoch":93,"log_start":"111'2900","ondisk_log_start":"111'2900","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4983,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":399966208,"num_objects":6103,"num_object_clones":0,"num_object_copies":18309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":447552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.077064+0000","last_change":"2024-02-12T16:54:00.780908+0000","last_active":"2024-02-12T16:54:12.077064+0000","last_peered":"2024-02-12T16:54:12.077064+0000","last_clean":"2024-02-12T16:54:12.077064+0000","last_became_active":"2024-02-12T16:49:45.218096+0000","last_became_peered":"2024-02-12T16:49:45.218096+0000","last_unstale":"2024-02-12T16:54:12.077064+0000","last_undegraded":"2024-02-12T16:54:12.077064+0000","last_fullsized":"2024-02-12T16:54:12.077064+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.780846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.780846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":92,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.083877+0000","last_change":"2024-02-12T16:54:04.254755+0000","last_active":"2024-02-12T16:54:12.083877+0000","last_peered":"2024-02-12T16:54:12.083877+0000","last_clean":"2024-02-12T16:54:12.083877+0000","last_became_active":"2024-02-12T16:49:39.104017+0000","last_became_peered":"2024-02-12T16:49:39.104017+0000","last_unstale":"2024-02-12T16:54:12.083877+0000","last_undegraded":"2024-02-12T16:54:12.083877+0000","last_fullsized":"2024-02-12T16:54:12.083877+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086744+0000","last_change":"2024-02-12T16:52:58.167642+0000","last_active":"2024-02-12T16:54:12.086744+0000","last_peered":"2024-02-12T16:54:12.086744+0000","last_clean":"2024-02-12T16:54:12.086744+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:54:12.086744+0000","last_undegraded":"2024-02-12T16:54:12.086744+0000","last_fullsized":"2024-02-12T16:54:12.086744+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:52:58.167543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.167543+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.087927+0000","last_change":"2024-02-12T16:53:43.229740+0000","last_active":"2024-02-12T16:54:12.087927+0000","last_peered":"2024-02-12T16:54:12.087927+0000","last_clean":"2024-02-12T16:54:12.087927+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:54:12.087927+0000","last_undegraded":"2024-02-12T16:54:12.087927+0000","last_fullsized":"2024-02-12T16:54:12.087927+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:43.229622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.229622+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":85,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.092453+0000","last_change":"2024-02-12T16:51:22.070134+0000","last_active":"2024-02-12T16:54:12.092453+0000","last_peered":"2024-02-12T16:54:12.092453+0000","last_clean":"2024-02-12T16:54:12.092453+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:54:12.092453+0000","last_undegraded":"2024-02-12T16:54:12.092453+0000","last_fullsized":"2024-02-12T16:54:12.092453+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:22.070056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:22.070056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"133'7728","reported_seq":7776,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.279075+0000","last_change":"2024-02-12T16:52:16.554135+0000","last_active":"2024-02-12T16:54:18.279075+0000","last_peered":"2024-02-12T16:54:18.279075+0000","last_clean":"2024-02-12T16:54:18.279075+0000","last_became_active":"2024-02-12T16:52:16.553900+0000","last_became_peered":"2024-02-12T16:52:16.553900+0000","last_unstale":"2024-02-12T16:54:18.279075+0000","last_undegraded":"2024-02-12T16:54:18.279075+0000","last_fullsized":"2024-02-12T16:54:18.279075+0000","mapping_epoch":93,"log_start":"107'1900","ondisk_log_start":"107'1900","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5828,"ondisk_log_size":5828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":387186688,"num_objects":5908,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7728,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078098+0000","last_change":"2024-02-12T16:53:50.836514+0000","last_active":"2024-02-12T16:54:12.078098+0000","last_peered":"2024-02-12T16:54:12.078098+0000","last_clean":"2024-02-12T16:54:12.078098+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:54:12.078098+0000","last_undegraded":"2024-02-12T16:54:12.078098+0000","last_fullsized":"2024-02-12T16:54:12.078098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.836406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.836406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":97,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086118+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T16:54:12.086118+0000","last_peered":"2024-02-12T16:54:12.086118+0000","last_clean":"2024-02-12T16:54:12.086118+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:54:12.086118+0000","last_undegraded":"2024-02-12T16:54:12.086118+0000","last_fullsized":"2024-02-12T16:54:12.086118+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":126,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.083866+0000","last_change":"2024-02-12T16:51:52.745302+0000","last_active":"2024-02-12T16:54:12.083866+0000","last_peered":"2024-02-12T16:54:12.083866+0000","last_clean":"2024-02-12T16:54:12.083866+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:54:12.083866+0000","last_undegraded":"2024-02-12T16:54:12.083866+0000","last_fullsized":"2024-02-12T16:54:12.083866+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:51:52.745190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:52.745190+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091445+0000","last_change":"2024-02-12T16:53:50.681255+0000","last_active":"2024-02-12T16:54:12.091445+0000","last_peered":"2024-02-12T16:54:12.091445+0000","last_clean":"2024-02-12T16:54:12.091445+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:54:12.091445+0000","last_undegraded":"2024-02-12T16:54:12.091445+0000","last_fullsized":"2024-02-12T16:54:12.091445+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.681142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.681142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"133'7739","reported_seq":7787,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.293305+0000","last_change":"2024-02-12T16:52:16.557948+0000","last_active":"2024-02-12T16:54:18.293305+0000","last_peered":"2024-02-12T16:54:18.293305+0000","last_clean":"2024-02-12T16:54:18.293305+0000","last_became_active":"2024-02-12T16:52:16.557659+0000","last_became_peered":"2024-02-12T16:52:16.557659+0000","last_unstale":"2024-02-12T16:54:18.293305+0000","last_undegraded":"2024-02-12T16:54:18.293305+0000","last_fullsized":"2024-02-12T16:54:18.293305+0000","mapping_epoch":93,"log_start":"111'3000","ondisk_log_start":"111'3000","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4739,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":390135808,"num_objects":5953,"num_object_clones":0,"num_object_copies":17859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7739,"num_write_kb":438144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":52,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089084+0000","last_change":"2024-02-12T16:54:02.624972+0000","last_active":"2024-02-12T16:54:12.089084+0000","last_peered":"2024-02-12T16:54:12.089084+0000","last_clean":"2024-02-12T16:54:12.089084+0000","last_became_active":"2024-02-12T16:52:20.563167+0000","last_became_peered":"2024-02-12T16:52:20.563167+0000","last_unstale":"2024-02-12T16:54:12.089084+0000","last_undegraded":"2024-02-12T16:54:12.089084+0000","last_fullsized":"2024-02-12T16:54:12.089084+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.624865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.624865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":96,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.077175+0000","last_change":"2024-02-12T16:53:48.828698+0000","last_active":"2024-02-12T16:54:12.077175+0000","last_peered":"2024-02-12T16:54:12.077175+0000","last_clean":"2024-02-12T16:54:12.077175+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:54:12.077175+0000","last_undegraded":"2024-02-12T16:54:12.077175+0000","last_fullsized":"2024-02-12T16:54:12.077175+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":118,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.089298+0000","last_change":"2024-02-12T16:51:51.095208+0000","last_active":"2024-02-12T16:54:12.089298+0000","last_peered":"2024-02-12T16:54:12.089298+0000","last_clean":"2024-02-12T16:54:12.089298+0000","last_became_active":"2024-02-12T16:49:41.669593+0000","last_became_peered":"2024-02-12T16:49:41.669593+0000","last_unstale":"2024-02-12T16:54:12.089298+0000","last_undegraded":"2024-02-12T16:54:12.089298+0000","last_fullsized":"2024-02-12T16:54:12.089298+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:51:51.095125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:51.095125+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.055993+0000","last_change":"2024-02-12T16:52:56.823627+0000","last_active":"2024-02-12T16:54:12.055993+0000","last_peered":"2024-02-12T16:54:12.055993+0000","last_clean":"2024-02-12T16:54:12.055993+0000","last_became_active":"2024-02-12T16:49:43.202312+0000","last_became_peered":"2024-02-12T16:49:43.202312+0000","last_unstale":"2024-02-12T16:54:12.055993+0000","last_undegraded":"2024-02-12T16:54:12.055993+0000","last_fullsized":"2024-02-12T16:54:12.055993+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:56.823548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:56.823548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"133'7725","reported_seq":7773,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.340535+0000","last_change":"2024-02-12T16:52:16.557971+0000","last_active":"2024-02-12T16:54:18.340535+0000","last_peered":"2024-02-12T16:54:18.340535+0000","last_clean":"2024-02-12T16:54:18.340535+0000","last_became_active":"2024-02-12T16:52:16.557700+0000","last_became_peered":"2024-02-12T16:52:16.557700+0000","last_unstale":"2024-02-12T16:54:18.340535+0000","last_undegraded":"2024-02-12T16:54:18.340535+0000","last_fullsized":"2024-02-12T16:54:18.340535+0000","mapping_epoch":93,"log_start":"111'3000","ondisk_log_start":"111'3000","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":4725,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389611520,"num_objects":5945,"num_object_clones":0,"num_object_copies":17835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":87,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086649+0000","last_change":"2024-02-12T16:52:41.149470+0000","last_active":"2024-02-12T16:54:12.086649+0000","last_peered":"2024-02-12T16:54:12.086649+0000","last_clean":"2024-02-12T16:54:12.086649+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:54:12.086649+0000","last_undegraded":"2024-02-12T16:54:12.086649+0000","last_fullsized":"2024-02-12T16:54:12.086649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:52:41.149382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.149382+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":93,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090453+0000","last_change":"2024-02-12T16:52:49.581959+0000","last_active":"2024-02-12T16:54:12.090453+0000","last_peered":"2024-02-12T16:54:12.090453+0000","last_clean":"2024-02-12T16:54:12.090453+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:54:12.090453+0000","last_undegraded":"2024-02-12T16:54:12.090453+0000","last_fullsized":"2024-02-12T16:54:12.090453+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.581862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.581862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":85,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091320+0000","last_change":"2024-02-12T16:52:51.025242+0000","last_active":"2024-02-12T16:54:12.091320+0000","last_peered":"2024-02-12T16:54:12.091320+0000","last_clean":"2024-02-12T16:54:12.091320+0000","last_became_active":"2024-02-12T16:52:20.563744+0000","last_became_peered":"2024-02-12T16:52:20.563744+0000","last_unstale":"2024-02-12T16:54:12.091320+0000","last_undegraded":"2024-02-12T16:54:12.091320+0000","last_fullsized":"2024-02-12T16:54:12.091320+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:52:51.025152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.025152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":153,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.068381+0000","last_change":"2024-02-12T16:52:11.925541+0000","last_active":"2024-02-12T16:54:14.068381+0000","last_peered":"2024-02-12T16:54:14.068381+0000","last_clean":"2024-02-12T16:54:14.068381+0000","last_became_active":"2024-02-12T16:49:43.205391+0000","last_became_peered":"2024-02-12T16:49:43.205391+0000","last_unstale":"2024-02-12T16:54:14.068381+0000","last_undegraded":"2024-02-12T16:54:14.068381+0000","last_fullsized":"2024-02-12T16:54:14.068381+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:11.925474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:11.925474+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"133'7891","reported_seq":7945,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.296007+0000","last_change":"2024-02-12T16:53:48.150960+0000","last_active":"2024-02-12T16:54:18.296007+0000","last_peered":"2024-02-12T16:54:18.296007+0000","last_clean":"2024-02-12T16:54:18.296007+0000","last_became_active":"2024-02-12T16:52:16.560135+0000","last_became_peered":"2024-02-12T16:52:16.560135+0000","last_unstale":"2024-02-12T16:54:18.296007+0000","last_undegraded":"2024-02-12T16:54:18.296007+0000","last_fullsized":"2024-02-12T16:54:18.296007+0000","mapping_epoch":93,"log_start":"106'1700","ondisk_log_start":"106'1700","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'6180","last_scrub_stamp":"2024-02-12T16:53:48.150854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.150854+0000","log_size":6191,"ondisk_log_size":6191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":399638552,"num_objects":6099,"num_object_clones":0,"num_object_copies":18297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7891,"num_write_kb":447617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":82,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091591+0000","last_change":"2024-02-12T16:51:34.081725+0000","last_active":"2024-02-12T16:54:12.091591+0000","last_peered":"2024-02-12T16:54:12.091591+0000","last_clean":"2024-02-12T16:54:12.091591+0000","last_became_active":"2024-02-12T16:49:45.223659+0000","last_became_peered":"2024-02-12T16:49:45.223659+0000","last_unstale":"2024-02-12T16:54:12.091591+0000","last_undegraded":"2024-02-12T16:54:12.091591+0000","last_fullsized":"2024-02-12T16:54:12.091591+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:34.081644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:34.081644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.092377+0000","last_change":"2024-02-12T16:52:52.983089+0000","last_active":"2024-02-12T16:54:12.092377+0000","last_peered":"2024-02-12T16:54:12.092377+0000","last_clean":"2024-02-12T16:54:12.092377+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:54:12.092377+0000","last_undegraded":"2024-02-12T16:54:12.092377+0000","last_fullsized":"2024-02-12T16:54:12.092377+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:52.982996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.982996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":71,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.077729+0000","last_change":"2024-02-12T16:53:59.566836+0000","last_active":"2024-02-12T16:54:12.077729+0000","last_peered":"2024-02-12T16:54:12.077729+0000","last_clean":"2024-02-12T16:54:12.077729+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:54:12.077729+0000","last_undegraded":"2024-02-12T16:54:12.077729+0000","last_fullsized":"2024-02-12T16:54:12.077729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.566730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":83,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.090271+0000","last_change":"2024-02-12T16:52:47.559724+0000","last_active":"2024-02-12T16:54:12.090271+0000","last_peered":"2024-02-12T16:54:12.090271+0000","last_clean":"2024-02-12T16:54:12.090271+0000","last_became_active":"2024-02-12T16:52:20.561707+0000","last_became_peered":"2024-02-12T16:52:20.561707+0000","last_unstale":"2024-02-12T16:54:12.090271+0000","last_undegraded":"2024-02-12T16:54:12.090271+0000","last_fullsized":"2024-02-12T16:54:12.090271+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:52:47.559659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.559659+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"133'7748","reported_seq":7801,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.274083+0000","last_change":"2024-02-12T16:53:45.409281+0000","last_active":"2024-02-12T16:54:18.274083+0000","last_peered":"2024-02-12T16:54:18.274083+0000","last_clean":"2024-02-12T16:54:18.274083+0000","last_became_active":"2024-02-12T16:52:16.556464+0000","last_became_peered":"2024-02-12T16:52:16.556464+0000","last_unstale":"2024-02-12T16:54:18.274083+0000","last_undegraded":"2024-02-12T16:54:18.274083+0000","last_fullsized":"2024-02-12T16:54:18.274083+0000","mapping_epoch":93,"log_start":"105'1600","ondisk_log_start":"105'1600","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'5931","last_scrub_stamp":"2024-02-12T16:53:45.409189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.409189+0000","log_size":6148,"ondisk_log_size":6148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389939200,"num_objects":5950,"num_object_clones":0,"num_object_copies":17850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091744+0000","last_change":"2024-02-12T16:52:46.979058+0000","last_active":"2024-02-12T16:54:12.091744+0000","last_peered":"2024-02-12T16:54:12.091744+0000","last_clean":"2024-02-12T16:54:12.091744+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:54:12.091744+0000","last_undegraded":"2024-02-12T16:54:12.091744+0000","last_fullsized":"2024-02-12T16:54:12.091744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:46.978987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:46.978987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"98'10","reported_seq":122,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.070541+0000","last_change":"2024-02-12T16:52:20.561940+0000","last_active":"2024-02-12T16:54:14.070541+0000","last_peered":"2024-02-12T16:54:14.070541+0000","last_clean":"2024-02-12T16:54:14.070541+0000","last_became_active":"2024-02-12T16:52:20.561534+0000","last_became_peered":"2024-02-12T16:52:20.561534+0000","last_unstale":"2024-02-12T16:54:14.070541+0000","last_undegraded":"2024-02-12T16:54:14.070541+0000","last_fullsized":"2024-02-12T16:54:14.070541+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-02-12T16:52:12.826762+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:52:12.826762+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.077938+0000","last_change":"2024-02-12T16:52:15.773872+0000","last_active":"2024-02-12T16:54:12.077938+0000","last_peered":"2024-02-12T16:54:12.077938+0000","last_clean":"2024-02-12T16:54:12.077938+0000","last_became_active":"2024-02-12T16:49:39.095868+0000","last_became_peered":"2024-02-12T16:49:39.095868+0000","last_unstale":"2024-02-12T16:54:12.077938+0000","last_undegraded":"2024-02-12T16:54:12.077938+0000","last_fullsized":"2024-02-12T16:54:12.077938+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.773807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.773807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":124,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.077643+0000","last_change":"2024-02-12T16:52:51.864105+0000","last_active":"2024-02-12T16:54:12.077643+0000","last_peered":"2024-02-12T16:54:12.077643+0000","last_clean":"2024-02-12T16:54:12.077643+0000","last_became_active":"2024-02-12T16:49:41.178544+0000","last_became_peered":"2024-02-12T16:49:41.178544+0000","last_unstale":"2024-02-12T16:54:12.077643+0000","last_undegraded":"2024-02-12T16:54:12.077643+0000","last_fullsized":"2024-02-12T16:54:12.077643+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.864004+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"133'7882","reported_seq":7929,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.360510+0000","last_change":"2024-02-12T16:52:16.557015+0000","last_active":"2024-02-12T16:54:18.360510+0000","last_peered":"2024-02-12T16:54:18.360510+0000","last_clean":"2024-02-12T16:54:18.360510+0000","last_became_active":"2024-02-12T16:52:16.556667+0000","last_became_peered":"2024-02-12T16:52:16.556667+0000","last_unstale":"2024-02-12T16:54:18.360510+0000","last_undegraded":"2024-02-12T16:54:18.360510+0000","last_fullsized":"2024-02-12T16:54:18.360510+0000","mapping_epoch":93,"log_start":"108'2400","ondisk_log_start":"108'2400","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5482,"ondisk_log_size":5482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":400949248,"num_objects":6118,"num_object_clones":0,"num_object_copies":18354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7882,"num_write_kb":448000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":52,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078364+0000","last_change":"2024-02-12T16:53:57.738352+0000","last_active":"2024-02-12T16:54:12.078364+0000","last_peered":"2024-02-12T16:54:12.078364+0000","last_clean":"2024-02-12T16:54:12.078364+0000","last_became_active":"2024-02-12T16:52:20.560085+0000","last_became_peered":"2024-02-12T16:52:20.560085+0000","last_unstale":"2024-02-12T16:54:12.078364+0000","last_undegraded":"2024-02-12T16:54:12.078364+0000","last_fullsized":"2024-02-12T16:54:12.078364+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:57.738249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.738249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":154,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.082169+0000","last_change":"2024-02-12T16:52:52.366599+0000","last_active":"2024-02-12T16:54:14.082169+0000","last_peered":"2024-02-12T16:54:14.082169+0000","last_clean":"2024-02-12T16:54:14.082169+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:54:14.082169+0000","last_undegraded":"2024-02-12T16:54:14.082169+0000","last_fullsized":"2024-02-12T16:54:14.082169+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:52.366513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.366513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.079116+0000","last_change":"2024-02-12T16:52:53.862672+0000","last_active":"2024-02-12T16:54:12.079116+0000","last_peered":"2024-02-12T16:54:12.079116+0000","last_clean":"2024-02-12T16:54:12.079116+0000","last_became_active":"2024-02-12T16:49:39.101478+0000","last_became_peered":"2024-02-12T16:49:39.101478+0000","last_unstale":"2024-02-12T16:54:12.079116+0000","last_undegraded":"2024-02-12T16:54:12.079116+0000","last_fullsized":"2024-02-12T16:54:12.079116+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:53.862480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:53.862480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":79,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091725+0000","last_change":"2024-02-12T16:52:43.063907+0000","last_active":"2024-02-12T16:54:12.091725+0000","last_peered":"2024-02-12T16:54:12.091725+0000","last_clean":"2024-02-12T16:54:12.091725+0000","last_became_active":"2024-02-12T16:52:20.564894+0000","last_became_peered":"2024-02-12T16:52:20.564894+0000","last_unstale":"2024-02-12T16:54:12.091725+0000","last_undegraded":"2024-02-12T16:54:12.091725+0000","last_fullsized":"2024-02-12T16:54:12.091725+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:52:43.063831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.063831+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"133'7774","reported_seq":7820,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.320980+0000","last_change":"2024-02-12T16:52:16.562286+0000","last_active":"2024-02-12T16:54:18.320980+0000","last_peered":"2024-02-12T16:54:18.320980+0000","last_clean":"2024-02-12T16:54:18.320980+0000","last_became_active":"2024-02-12T16:52:16.561976+0000","last_became_peered":"2024-02-12T16:52:16.561976+0000","last_unstale":"2024-02-12T16:54:18.320980+0000","last_undegraded":"2024-02-12T16:54:18.320980+0000","last_fullsized":"2024-02-12T16:54:18.320980+0000","mapping_epoch":93,"log_start":"105'1300","ondisk_log_start":"105'1300","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":6474,"ondisk_log_size":6474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":394919936,"num_objects":6026,"num_object_clones":0,"num_object_copies":18078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":441600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078311+0000","last_change":"2024-02-12T16:52:01.448343+0000","last_active":"2024-02-12T16:54:12.078311+0000","last_peered":"2024-02-12T16:54:12.078311+0000","last_clean":"2024-02-12T16:54:12.078311+0000","last_became_active":"2024-02-12T16:49:45.220893+0000","last_became_peered":"2024-02-12T16:49:45.220893+0000","last_unstale":"2024-02-12T16:54:12.078311+0000","last_undegraded":"2024-02-12T16:54:12.078311+0000","last_fullsized":"2024-02-12T16:54:12.078311+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:01.448278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:01.448278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.078454+0000","last_change":"2024-02-12T16:53:49.832912+0000","last_active":"2024-02-12T16:54:12.078454+0000","last_peered":"2024-02-12T16:54:12.078454+0000","last_clean":"2024-02-12T16:54:12.078454+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:54:12.078454+0000","last_undegraded":"2024-02-12T16:54:12.078454+0000","last_fullsized":"2024-02-12T16:54:12.078454+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:49.832798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.832798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":92,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086207+0000","last_change":"2024-02-12T16:51:53.008096+0000","last_active":"2024-02-12T16:54:12.086207+0000","last_peered":"2024-02-12T16:54:12.086207+0000","last_clean":"2024-02-12T16:54:12.086207+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:54:12.086207+0000","last_undegraded":"2024-02-12T16:54:12.086207+0000","last_fullsized":"2024-02-12T16:54:12.086207+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:53.007988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:53.007988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":92,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.080508+0000","last_change":"2024-02-12T16:52:44.280962+0000","last_active":"2024-02-12T16:54:12.080508+0000","last_peered":"2024-02-12T16:54:12.080508+0000","last_clean":"2024-02-12T16:54:12.080508+0000","last_became_active":"2024-02-12T16:52:20.565139+0000","last_became_peered":"2024-02-12T16:52:20.565139+0000","last_unstale":"2024-02-12T16:54:12.080508+0000","last_undegraded":"2024-02-12T16:54:12.080508+0000","last_fullsized":"2024-02-12T16:54:12.080508+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:52:44.280844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.280844+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"133'7810","reported_seq":7857,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.366635+0000","last_change":"2024-02-12T16:52:16.557129+0000","last_active":"2024-02-12T16:54:18.366635+0000","last_peered":"2024-02-12T16:54:18.366635+0000","last_clean":"2024-02-12T16:54:18.366635+0000","last_became_active":"2024-02-12T16:52:16.556854+0000","last_became_peered":"2024-02-12T16:52:16.556854+0000","last_unstale":"2024-02-12T16:54:18.366635+0000","last_undegraded":"2024-02-12T16:54:18.366635+0000","last_fullsized":"2024-02-12T16:54:18.366635+0000","mapping_epoch":93,"log_start":"108'2300","ondisk_log_start":"108'2300","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5510,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":400031744,"num_objects":6104,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7810,"num_write_kb":445248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":86,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.086728+0000","last_change":"2024-02-12T16:52:48.128798+0000","last_active":"2024-02-12T16:54:12.086728+0000","last_peered":"2024-02-12T16:54:12.086728+0000","last_clean":"2024-02-12T16:54:12.086728+0000","last_became_active":"2024-02-12T16:49:45.217589+0000","last_became_peered":"2024-02-12T16:49:45.217589+0000","last_unstale":"2024-02-12T16:54:12.086728+0000","last_undegraded":"2024-02-12T16:54:12.086728+0000","last_fullsized":"2024-02-12T16:54:12.086728+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.128703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.128703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":95,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.080144+0000","last_change":"2024-02-12T16:53:58.524837+0000","last_active":"2024-02-12T16:54:12.080144+0000","last_peered":"2024-02-12T16:54:12.080144+0000","last_clean":"2024-02-12T16:54:12.080144+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:54:12.080144+0000","last_undegraded":"2024-02-12T16:54:12.080144+0000","last_fullsized":"2024-02-12T16:54:12.080144+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.524736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":94,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.077906+0000","last_change":"2024-02-12T16:52:05.495937+0000","last_active":"2024-02-12T16:54:12.077906+0000","last_peered":"2024-02-12T16:54:12.077906+0000","last_clean":"2024-02-12T16:54:12.077906+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:54:12.077906+0000","last_undegraded":"2024-02-12T16:54:12.077906+0000","last_fullsized":"2024-02-12T16:54:12.077906+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:05.495862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:05.495862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091378+0000","last_change":"2024-02-12T16:53:40.847643+0000","last_active":"2024-02-12T16:54:12.091378+0000","last_peered":"2024-02-12T16:54:12.091378+0000","last_clean":"2024-02-12T16:54:12.091378+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:54:12.091378+0000","last_undegraded":"2024-02-12T16:54:12.091378+0000","last_fullsized":"2024-02-12T16:54:12.091378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:53:40.847540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:40.847540+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"133'7744","reported_seq":7792,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.286528+0000","last_change":"2024-02-12T16:52:16.551619+0000","last_active":"2024-02-12T16:54:18.286528+0000","last_peered":"2024-02-12T16:54:18.286528+0000","last_clean":"2024-02-12T16:54:18.286528+0000","last_became_active":"2024-02-12T16:52:16.551312+0000","last_became_peered":"2024-02-12T16:52:16.551312+0000","last_unstale":"2024-02-12T16:54:18.286528+0000","last_undegraded":"2024-02-12T16:54:18.286528+0000","last_fullsized":"2024-02-12T16:54:18.286528+0000","mapping_epoch":93,"log_start":"105'1300","ondisk_log_start":"105'1300","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":6444,"ondisk_log_size":6444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391643136,"num_objects":5976,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":439040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":86,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.084165+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T16:54:12.084165+0000","last_peered":"2024-02-12T16:54:12.084165+0000","last_clean":"2024-02-12T16:54:12.084165+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:54:12.084165+0000","last_undegraded":"2024-02-12T16:54:12.084165+0000","last_fullsized":"2024-02-12T16:54:12.084165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.061193+0000","last_change":"2024-02-12T16:53:01.286057+0000","last_active":"2024-02-12T16:54:12.061193+0000","last_peered":"2024-02-12T16:54:12.061193+0000","last_clean":"2024-02-12T16:54:12.061193+0000","last_became_active":"2024-02-12T16:49:43.201946+0000","last_became_peered":"2024-02-12T16:49:43.201946+0000","last_unstale":"2024-02-12T16:54:12.061193+0000","last_undegraded":"2024-02-12T16:54:12.061193+0000","last_fullsized":"2024-02-12T16:54:12.061193+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.285961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.285961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":104,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.080009+0000","last_change":"2024-02-12T16:53:56.761528+0000","last_active":"2024-02-12T16:54:12.080009+0000","last_peered":"2024-02-12T16:54:12.080009+0000","last_clean":"2024-02-12T16:54:12.080009+0000","last_became_active":"2024-02-12T16:49:39.099695+0000","last_became_peered":"2024-02-12T16:49:39.099695+0000","last_unstale":"2024-02-12T16:54:12.080009+0000","last_undegraded":"2024-02-12T16:54:12.080009+0000","last_fullsized":"2024-02-12T16:54:12.080009+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":106,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.084049+0000","last_change":"2024-02-12T16:52:08.569339+0000","last_active":"2024-02-12T16:54:12.084049+0000","last_peered":"2024-02-12T16:54:12.084049+0000","last_clean":"2024-02-12T16:54:12.084049+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:54:12.084049+0000","last_undegraded":"2024-02-12T16:54:12.084049+0000","last_fullsized":"2024-02-12T16:54:12.084049+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:08.569246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:08.569246+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"133'7731","reported_seq":7779,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:18.283455+0000","last_change":"2024-02-12T16:52:16.550877+0000","last_active":"2024-02-12T16:54:18.283455+0000","last_peered":"2024-02-12T16:54:18.283455+0000","last_clean":"2024-02-12T16:54:18.283455+0000","last_became_active":"2024-02-12T16:52:16.550574+0000","last_became_peered":"2024-02-12T16:52:16.550574+0000","last_unstale":"2024-02-12T16:54:18.283455+0000","last_undegraded":"2024-02-12T16:54:18.283455+0000","last_fullsized":"2024-02-12T16:54:18.283455+0000","mapping_epoch":93,"log_start":"109'2600","ondisk_log_start":"109'2600","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:15.525157+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.525157+0000","log_size":5131,"ondisk_log_size":5131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":387645440,"num_objects":5915,"num_object_clones":0,"num_object_copies":17745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":88,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091428+0000","last_change":"2024-02-12T16:54:03.996224+0000","last_active":"2024-02-12T16:54:12.091428+0000","last_peered":"2024-02-12T16:54:12.091428+0000","last_clean":"2024-02-12T16:54:12.091428+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:54:12.091428+0000","last_undegraded":"2024-02-12T16:54:12.091428+0000","last_fullsized":"2024-02-12T16:54:12.091428+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":99,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.080334+0000","last_change":"2024-02-12T16:53:00.294697+0000","last_active":"2024-02-12T16:54:12.080334+0000","last_peered":"2024-02-12T16:54:12.080334+0000","last_clean":"2024-02-12T16:54:12.080334+0000","last_became_active":"2024-02-12T16:49:39.105982+0000","last_became_peered":"2024-02-12T16:49:39.105982+0000","last_unstale":"2024-02-12T16:54:12.080334+0000","last_undegraded":"2024-02-12T16:54:12.080334+0000","last_fullsized":"2024-02-12T16:54:12.080334+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:00.294633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:00.294633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":117,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-02-12T16:54:12.091357+0000","last_change":"2024-02-12T16:52:40.091071+0000","last_active":"2024-02-12T16:54:12.091357+0000","last_peered":"2024-02-12T16:54:12.091357+0000","last_clean":"2024-02-12T16:54:12.091357+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:54:12.091357+0000","last_undegraded":"2024-02-12T16:54:12.091357+0000","last_fullsized":"2024-02-12T16:54:12.091357+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:52:40.090931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.090931+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-02-12T16:54:14.082632+0000","last_change":"2024-02-12T16:52:59.185115+0000","last_active":"2024-02-12T16:54:14.082632+0000","last_peered":"2024-02-12T16:54:14.082632+0000","last_clean":"2024-02-12T16:54:14.082632+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:54:14.082632+0000","last_undegraded":"2024-02-12T16:54:14.082632+0000","last_fullsized":"2024-02-12T16:54:14.082632+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6308036632,"num_objects":96254,"num_object_clones":0,"num_object_copies":288762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124646,"num_write_kb":7068737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18924777472,"data_stored":18924765256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88246,"ondisk_log_size":88246,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":272,"num_read_kb":267,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":691,"num_read_kb":448,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234043,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2254444,"kb_used_data":1949028,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91490708,"statfs":{"total":95995035648,"available":93686484992,"internally_reserved":0,"allocated":1995804672,"data_stored":1995390333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":0.75600000000000001,"15min":0.56699999999999995},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":4.9290000000000003,"5min":4.9290000000000003,"15min":4.9290000000000003},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":0.69199999999999995,"15min":0.57699999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.2190000000000003,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":1.498}]},{"osd":1,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.58899999999999997,"15min":0.55700000000000005},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.59799999999999998,"15min":0.56399999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.4399999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":0.65300000000000002,"15min":0.53900000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":2.7509999999999999,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.59599999999999997,"15min":0.54500000000000004},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.38}]},{"osd":3,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":0.88800000000000001,"15min":0.68600000000000005},"min":{"1min":0.34699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.423,"5min":5.423,"15min":5.423},"last":1.984},{"interface":"front","average":{"1min":1.893,"5min":0.82099999999999995,"15min":0.65100000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.5979999999999999,"5min":5.5979999999999999,"15min":5.5979999999999999},"last":0.35899999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.51400000000000001,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.2759999999999998,"5min":3.2759999999999998,"15min":3.2759999999999998},"last":1.2549999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.46600000000000003,"15min":0.50700000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.58499999999999996,"5min":1.327,"15min":1.327},"last":1.276}]},{"osd":5,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.401,"5min":1.573,"15min":1.573},"last":0.316},{"interface":"front","average":{"1min":0.378,"5min":0.49399999999999999,"15min":0.53000000000000003},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54500000000000004,"5min":1.621,"15min":1.621},"last":0.29499999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.32800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":1.47},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.59499999999999997,"15min":0.57599999999999996},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.36399999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528384,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2232988,"kb_used_data":1931668,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91512164,"statfs":{"total":95995035648,"available":93708455936,"internally_reserved":0,"allocated":1978028032,"data_stored":1977629451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":0.80800000000000005,"15min":0.59299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.7119999999999997,"5min":5.7119999999999997,"15min":5.7119999999999997},"last":0.318},{"interface":"front","average":{"1min":1.7470000000000001,"5min":0.86399999999999999,"15min":0.60999999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":10.56,"5min":10.56,"15min":10.56},"last":0.22800000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":0.88200000000000001,"15min":0.64300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":5.0839999999999996,"5min":15.987,"15min":15.987},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":0.88500000000000001,"15min":0.65900000000000003},"min":{"1min":0.28000000000000003,"5min":0.25,"15min":0.25},"max":{"1min":5.1580000000000004,"5min":15.997,"15min":15.997},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":0.77100000000000002,"15min":0.59499999999999997},"min":{"1min":0.26400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":4.5910000000000002,"5min":4.5910000000000002,"15min":4.5910000000000002},"last":5.6959999999999997},{"interface":"front","average":{"1min":1.3129999999999999,"5min":0.67500000000000004,"15min":0.56699999999999995},"min":{"1min":0.29899999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.468,"5min":4.468,"15min":4.468},"last":3.161}]},{"osd":3,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":0.91100000000000003,"15min":0.63600000000000001},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":11.212,"5min":11.212,"15min":11.212},"last":0.72999999999999998},{"interface":"front","average":{"1min":2.4889999999999999,"5min":1.071,"15min":0.70499999999999996},"min":{"1min":0.312,"5min":0.249,"15min":0.249},"max":{"1min":10.875999999999999,"5min":10.875999999999999,"15min":10.875999999999999},"last":0.64300000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.46300000000000002,"15min":0.40699999999999997},"min":{"1min":0.254,"5min":0.217,"15min":0.217},"max":{"1min":2.5920000000000001,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.442,"15min":0.40999999999999998},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.30399999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44600000000000001,"15min":0.42599999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.51400000000000001,"15min":0.45800000000000002},"min":{"1min":0.313,"5min":0.247,"15min":0.247},"max":{"1min":2.6320000000000001,"5min":2.6320000000000001,"15min":2.6320000000000001},"last":0.438}]},{"osd":7,"last update":"Mon Feb 12 16:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.502,"15min":0.46999999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":2.4399999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.49299999999999999,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.51100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2626512,"kb_used_data":2316936,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91118640,"statfs":{"total":95995035648,"available":93305487360,"internally_reserved":0,"allocated":2372542464,"data_stored":2372093047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:23 2024","interfaces":[{"interface":"back","average":{"1min":2.3180000000000001,"5min":0.79200000000000004,"15min":0.48599999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":10.935,"5min":10.935,"15min":10.935},"last":0.496},{"interface":"front","average":{"1min":2.9609999999999999,"5min":0.94599999999999995,"15min":0.54000000000000004},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":12.544,"5min":12.544,"15min":12.544},"last":5.2199999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:53:46 2024","interfaces":[{"interface":"back","average":{"1min":2.605,"5min":0.84999999999999998,"15min":0.52600000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.4410000000000007,"5min":8.4410000000000007,"15min":8.4410000000000007},"last":0.59999999999999998},{"interface":"front","average":{"1min":5.0579999999999998,"5min":1.4139999999999999,"15min":0.72099999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":26.693999999999999,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:54:08 2024","interfaces":[{"interface":"back","average":{"1min":5.0300000000000002,"5min":1.681,"15min":0.86299999999999999},"min":{"1min":0.35499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":26.649999999999999,"5min":26.649999999999999,"15min":26.649999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":3.097,"5min":1.26,"15min":0.68799999999999994},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":13.654999999999999,"5min":13.654999999999999,"15min":13.654999999999999},"last":0.86099999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:53:26 2024","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":0.79000000000000004,"15min":0.63600000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.1440000000000001,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.2040000000000002,"5min":0.94899999999999995,"15min":0.71199999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":12.366,"5min":12.366,"15min":12.366},"last":0.373}]},{"osd":5,"last update":"Mon Feb 12 16:54:00 2024","interfaces":[{"interface":"back","average":{"1min":1.855,"5min":0.876,"15min":0.66200000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":7.2309999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":4.1310000000000002,"5min":1.4670000000000001,"15min":0.88700000000000001},"min":{"1min":0.44800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":26.722999999999999,"5min":26.722999999999999,"15min":26.722999999999999},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:53:15 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":0.83399999999999996,"15min":0.66800000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.1959999999999997,"5min":7.1959999999999997,"15min":7.1959999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.446,"5min":0.80200000000000005,"15min":0.68899999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":12.429,"5min":12.429,"15min":12.429},"last":0.438}]},{"osd":7,"last update":"Mon Feb 12 16:53:40 2024","interfaces":[{"interface":"back","average":{"1min":2.036,"5min":0.91500000000000004,"15min":0.747},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.387,"5min":1.022,"15min":0.77400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":7.2629999999999999,"5min":7.2629999999999999,"15min":7.2629999999999999},"last":1.5920000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3418860,"kb_used_data":3113124,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":90326292,"statfs":{"total":95995035648,"available":92494123008,"internally_reserved":0,"allocated":3187838976,"data_stored":3187424345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:53:18 2024","interfaces":[{"interface":"back","average":{"1min":1.9039999999999999,"5min":0.66900000000000004,"15min":0.433},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.192},"max":{"1min":9.0579999999999998,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.97,"5min":0.69399999999999995,"15min":0.42999999999999999},"min":{"1min":0.27000000000000002,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.3689999999999998,"5min":5.3689999999999998,"15min":5.3689999999999998},"last":0.21199999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:53:45 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":1.097,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.2739999999999991,"5min":9.2739999999999991,"15min":9.2739999999999991},"last":6.867},{"interface":"front","average":{"1min":2.706,"5min":0.91200000000000003,"15min":0.56599999999999995},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.28199999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":1.226,"15min":0.68600000000000005},"min":{"1min":0.28399999999999997,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.596,"5min":10.596,"15min":10.596},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.4399999999999999,"5min":1.367,"15min":0.76000000000000001},"min":{"1min":0.28100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":11.193,"5min":11.193,"15min":11.193},"last":0.376}]},{"osd":4,"last update":"Mon Feb 12 16:53:24 2024","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":0.80300000000000005,"15min":0.64800000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":5.6689999999999996,"5min":5.6689999999999996,"15min":5.6689999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.304,"5min":0.72899999999999998,"15min":0.629},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.4290000000000003,"5min":5.4290000000000003,"15min":5.4290000000000003},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:53:56 2024","interfaces":[{"interface":"back","average":{"1min":2.0790000000000002,"5min":0.93500000000000005,"15min":0.69199999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.4340000000000002,"5min":7.4340000000000002,"15min":7.4340000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.9990000000000001,"5min":0.94499999999999995,"15min":0.70999999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.5410000000000004,"5min":6.5410000000000004,"15min":6.5410000000000004},"last":2.6459999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:53:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":0.72099999999999997,"15min":0.623},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.4509999999999996,"5min":5.4509999999999996,"15min":5.4509999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.651,"5min":0.79500000000000004,"15min":0.65100000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":4.0979999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:53:45 2024","interfaces":[{"interface":"back","average":{"1min":2.0539999999999998,"5min":0.95599999999999996,"15min":0.70999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.675,"5min":0.86699999999999999,"15min":0.68600000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":5.4059999999999997,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":0.53100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2630492,"kb_used_data":2325012,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91114660,"statfs":{"total":95995035648,"available":93301411840,"internally_reserved":0,"allocated":2380812288,"data_stored":2380413926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":2.702,"5min":0.871,"15min":0.54000000000000004},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":8.5779999999999994,"5min":8.5779999999999994,"15min":8.5779999999999994},"last":9.4939999999999998},{"interface":"front","average":{"1min":2.2970000000000002,"5min":0.82299999999999995,"15min":0.53500000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":10.176,"5min":10.176,"15min":10.176},"last":3.9849999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":3.206,"5min":1.0669999999999999,"15min":0.61799999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":11.502000000000001,"5min":11.502000000000001,"15min":11.502000000000001},"last":9.4079999999999995},{"interface":"front","average":{"1min":2.7370000000000001,"5min":0.93100000000000005,"15min":0.57599999999999996},"min":{"1min":0.27300000000000002,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":8.5559999999999992,"5min":8.5559999999999992,"15min":8.5559999999999992},"last":9.5619999999999994}]},{"osd":3,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":6.5990000000000002,"5min":1.958,"15min":0.96099999999999997},"min":{"1min":0.505,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":51.579999999999998,"5min":51.579999999999998,"15min":51.579999999999998},"last":9.7400000000000002},{"interface":"front","average":{"1min":5.3970000000000002,"5min":1.5489999999999999,"15min":0.82999999999999996},"min":{"1min":0.311,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":48.988999999999997,"5min":48.988999999999997,"15min":48.988999999999997},"last":8.6189999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":0.76200000000000001,"15min":0.67800000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"last":5.79},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.68400000000000005,"15min":0.63100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":9.2119999999999997,"5min":9.2119999999999997,"15min":9.2119999999999997},"last":4.0069999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:53:51 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":0.998,"15min":0.83599999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.4789999999999992,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":5.7629999999999999},{"interface":"front","average":{"1min":3.016,"5min":1.101,"15min":0.76700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.023999999999999,"5min":11.023999999999999,"15min":11.023999999999999},"last":5.657}]},{"osd":6,"last update":"Mon Feb 12 16:54:10 2024","interfaces":[{"interface":"back","average":{"1min":3.7069999999999999,"5min":1.3220000000000001,"15min":0.84299999999999997},"min":{"1min":0.38100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":10.981999999999999,"5min":10.981999999999999,"15min":10.981999999999999},"last":5.6879999999999997},{"interface":"front","average":{"1min":1.976,"5min":0.94999999999999996,"15min":0.80400000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":9.2840000000000007,"5min":9.2840000000000007,"15min":9.2840000000000007},"last":4.0220000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":0.95999999999999996,"15min":0.97399999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":9.2550000000000008,"5min":9.2550000000000008,"15min":9.2550000000000008},"last":4.1219999999999999},{"interface":"front","average":{"1min":2.8919999999999999,"5min":1.081,"15min":0.78100000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":11.058999999999999,"5min":11.058999999999999,"15min":11.058999999999999},"last":5.726}]}]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3784440,"kb_used_data":3482800,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":89960712,"statfs":{"total":95995035648,"available":92119769088,"internally_reserved":0,"allocated":3566387200,"data_stored":3565961258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:54:13 2024","interfaces":[{"interface":"back","average":{"1min":5.6539999999999999,"5min":1.7609999999999999,"15min":0.84599999999999997},"min":{"1min":0.35999999999999999,"5min":0.187,"15min":0.187},"max":{"1min":30.257999999999999,"5min":30.257999999999999,"15min":30.257999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.0700000000000003,"5min":1.2929999999999999,"15min":0.70699999999999996},"min":{"1min":0.35499999999999998,"5min":0.221,"15min":0.221},"max":{"1min":12.787000000000001,"5min":12.787000000000001,"15min":12.787000000000001},"last":3.5979999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:54:13 2024","interfaces":[{"interface":"back","average":{"1min":3.944,"5min":1.24,"15min":0.70799999999999996},"min":{"1min":0.22800000000000001,"5min":0.216,"15min":0.216},"max":{"1min":12.989000000000001,"5min":12.989000000000001,"15min":12.989000000000001},"last":11.917},{"interface":"front","average":{"1min":3.774,"5min":1.3720000000000001,"15min":0.75},"min":{"1min":0.41899999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.499000000000001,"5min":11.499000000000001,"15min":11.499000000000001},"last":4.0499999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:54:13 2024","interfaces":[{"interface":"back","average":{"1min":3.3290000000000002,"5min":1.222,"15min":0.70199999999999996},"min":{"1min":0.42799999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":11.638,"5min":11.638,"15min":11.638},"last":3.9940000000000002},{"interface":"front","average":{"1min":5.125,"5min":1.5900000000000001,"15min":0.82899999999999996},"min":{"1min":0.31,"5min":0.252,"15min":0.252},"max":{"1min":25.465,"5min":25.465,"15min":25.465},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:53:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":0.66700000000000004,"15min":0.63700000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.427,"5min":3.427,"15min":3.427},"last":3.8079999999999998},{"interface":"front","average":{"1min":1.6459999999999999,"5min":0.78400000000000003,"15min":0.67600000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.3420000000000005,"5min":8.3420000000000005,"15min":8.3420000000000005},"last":3.9249999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:53:54 2024","interfaces":[{"interface":"back","average":{"1min":3.1280000000000001,"5min":1.139,"15min":0.80900000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":19.882999999999999,"5min":19.882999999999999,"15min":19.882999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.806,"5min":0.88500000000000001,"15min":0.71899999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":12.861000000000001,"5min":12.861000000000001,"15min":12.861000000000001},"last":3.8319999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:54:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3940000000000001,"5min":1.1419999999999999,"15min":0.80000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":6.5750000000000002,"5min":8.093,"15min":8.093},"last":3.9609999999999999},{"interface":"front","average":{"1min":2.9780000000000002,"5min":1.3120000000000001,"15min":0.85999999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":19.925999999999998,"5min":19.925999999999998,"15min":19.925999999999998},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":0.73799999999999999,"15min":0.64600000000000002},"min":{"1min":0.39300000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.4820000000000002,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":3.867},{"interface":"front","average":{"1min":2.2149999999999999,"5min":0.93000000000000005,"15min":0.72199999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":9.2349999999999994,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.38800000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953546,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2234412,"kb_used_data":1933028,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91510740,"statfs":{"total":95995035648,"available":93706997760,"internally_reserved":0,"allocated":1979420672,"data_stored":1979006686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:28 2024","interfaces":[{"interface":"back","average":{"1min":3.956,"5min":1.1839999999999999,"15min":0.77400000000000002},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":38.179000000000002,"5min":38.179000000000002,"15min":38.179000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":4.1790000000000003,"5min":1.226,"15min":0.75900000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":38.151000000000003,"5min":38.151000000000003,"15min":38.151000000000003},"last":4.657}]},{"osd":1,"last update":"Mon Feb 12 16:53:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.60399999999999998,"15min":0.55800000000000005},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.58899999999999997,"15min":0.56599999999999995},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.7629999999999999,"5min":5.7629999999999999,"15min":5.7629999999999999},"last":4.7279999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.7189999999999999,"5min":2.7189999999999999,"15min":2.7189999999999999},"last":2.4609999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.60499999999999998,"15min":0.58599999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.7309999999999999,"5min":2.7309999999999999,"15min":2.7309999999999999},"last":2.4700000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:53:28 2024","interfaces":[{"interface":"back","average":{"1min":4.4000000000000004,"5min":1.2929999999999999,"15min":0.81000000000000005},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":38.170999999999999,"5min":38.170999999999999,"15min":38.170999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":4.2329999999999997,"5min":1.264,"15min":0.79600000000000004},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":38.311999999999998,"5min":38.311999999999998,"15min":38.311999999999998},"last":0.34499999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.36499999999999999},"min":{"1min":0.255,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.5579999999999998,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":0.313},{"interface":"front","average":{"1min":0.438,"5min":0.41399999999999998,"15min":0.35799999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.35899999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38300000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.42799999999999999,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.376}]},{"osd":7,"last update":"Mon Feb 12 16:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46800000000000003,"15min":0.433},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.5,"15min":0.46000000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.44400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1837844,"kb_used_data":1532428,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91907308,"statfs":{"total":95995035648,"available":94113083392,"internally_reserved":0,"allocated":1569206272,"data_stored":1568815782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":3.4910000000000001,"5min":1.107,"15min":0.72099999999999997},"min":{"1min":0.216,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":36.880000000000003,"5min":36.880000000000003,"15min":36.880000000000003},"last":3.0129999999999999},{"interface":"front","average":{"1min":3.9319999999999999,"5min":1.1819999999999999,"15min":0.72099999999999997},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":36.954000000000001,"5min":36.954000000000001,"15min":36.954000000000001},"last":3.1269999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.63,"15min":0.55500000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.8159999999999998,"5min":3.8159999999999998,"15min":3.8159999999999998},"last":2.3900000000000001},{"interface":"front","average":{"1min":2.1720000000000002,"5min":0.84799999999999998,"15min":0.62},"min":{"1min":0.29099999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.7130000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"last":2.218}]},{"osd":2,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5219999999999998,"5min":1.0069999999999999,"15min":0.69299999999999995},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":18.218,"5min":18.218,"15min":18.218},"last":3.9990000000000001},{"interface":"front","average":{"1min":2.7050000000000001,"5min":0.95199999999999996,"15min":0.65200000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":18.327000000000002,"5min":18.327000000000002,"15min":18.327000000000002},"last":3.0059999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":3.7120000000000002,"5min":1.1890000000000001,"15min":0.752},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":18.251000000000001,"5min":18.251000000000001,"15min":18.251000000000001},"last":2.0910000000000002},{"interface":"front","average":{"1min":3.0110000000000001,"5min":1.0409999999999999,"15min":0.68799999999999994},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":20.263000000000002,"5min":20.263000000000002,"15min":20.263000000000002},"last":2.734}]},{"osd":4,"last update":"Mon Feb 12 16:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.52200000000000002,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.22},"max":{"1min":3.581,"5min":3.581,"15min":3.581},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.48599999999999999,"15min":0.41199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:54:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":0.622,"15min":0.49399999999999999},"min":{"1min":0.29899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.54600000000000004,"15min":0.47299999999999998},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.827,"5min":2.827,"15min":2.827},"last":0.46200000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.53600000000000003,"15min":0.49399999999999999},"min":{"1min":0.28499999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.2639999999999998,"5min":3.2639999999999998,"15min":3.2639999999999998},"last":0.379},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.59499999999999997,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.6099999999999999,"5min":3.6099999999999999,"15min":3.6099999999999999},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3186036736,"data_stored":3186032664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2358185984,"data_stored":2358181912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2359623680,"data_stored":2359623680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3539533824,"data_stored":3539533824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1967980544,"data_stored":1967980544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1563623424,"data_stored":1563623424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1967394816,"data_stored":1967390744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1982398464,"data_stored":1982398464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:54:25.304 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T16:54:25.305 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:54:25.305 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-12T16:54:25.305 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:54:25.305 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:54:25.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-12T16:54:25.353 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:54:25.354 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:54:25.354 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:54:25.354 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:54:25.354 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:54:25.354 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:54:25.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:54:25.418 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:54:25.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:54:25.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:54:25.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:54:25.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:54:25.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:54:25.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:54:25.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:54:25.429 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:54:25.429 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:25.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:25.675 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:54:25.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:24.379145+0000 mgr.y (mgr.24481) 187 : cluster [DBG] pgmap v199: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-02-12T16:54:25.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:25.076733+0000 mon.c (mon.2) 89 : audit [DBG] from='client.? v1:172.21.15.33:0/3586156049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:25.744 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:25.783 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:54:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:25.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:54:25.891 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:25.947 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:24.379145+0000 mgr.y (mgr.24481) 187 : cluster [DBG] pgmap v199: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-02-12T16:54:25.947 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:25.076733+0000 mon.c (mon.2) 89 : audit [DBG] from='client.? v1:172.21.15.33:0/3586156049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:25.950 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:24.379145+0000 mgr.y (mgr.24481) 187 : cluster [DBG] pgmap v199: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-02-12T16:54:25.950 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:25.076733+0000 mon.c (mon.2) 89 : audit [DBG] from='client.? v1:172.21.15.33:0/3586156049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:25.953 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:25.987 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:25.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:26.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:26.103 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:26.103 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:26.136 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:26.162 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:26.551 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:26.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:26.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:54:26.713 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:26.799 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:54:26.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:26.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:54:26.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:26.941 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:26.380667+0000 mgr.y (mgr.24481) 188 : cluster [DBG] pgmap v200: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-02-12T16:54:27.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:26.380667+0000 mgr.y (mgr.24481) 188 : cluster [DBG] pgmap v200: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-02-12T16:54:27.489 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:26.380667+0000 mgr.y (mgr.24481) 188 : cluster [DBG] pgmap v200: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-02-12T16:54:27.578 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:54:27.596 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:27.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:27.700 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:27.837 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:27.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:27.942 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:27.990 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:28.056 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:28.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:28.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:28.251 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:28.353 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:28.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:28.458 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:28.525 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:28.556 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:54:28.560 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:28.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:28.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:28.665 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:54:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T16:54:28.665 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:54:28.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:28.673 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:28.718 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:28.761 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:28.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:28.872 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:28.929 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:28.995 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:28.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:29.022 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:29.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:54:29.155 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:29.182 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:54:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:29.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:54:29.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:29.331 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:29.468 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:54:29.723 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:28.381303+0000 mgr.y (mgr.24481) 189 : cluster [DBG] pgmap v201: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 800 op/s 2024-02-12T16:54:29.723 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:28.569882+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.33:0/649367575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:54:29.723 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:28.575015+0000 mon.a (mon.0) 839 : audit [INF] from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:54:29.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:28.381303+0000 mgr.y (mgr.24481) 189 : cluster [DBG] pgmap v201: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 800 op/s 2024-02-12T16:54:29.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:28.569882+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.33:0/649367575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:54:29.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:28.575015+0000 mon.a (mon.0) 839 : audit [INF] from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:54:29.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:28.381303+0000 mgr.y (mgr.24481) 189 : cluster [DBG] pgmap v201: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 800 op/s 2024-02-12T16:54:29.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:28.569882+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.33:0/649367575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:54:29.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:28.575015+0000 mon.a (mon.0) 839 : audit [INF] from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:54:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:30.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:30.420 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:54:30.472 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:29.435802+0000 mon.a (mon.0) 840 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:29.442773+0000 mon.a (mon.0) 841 : audit [INF] from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:54:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:29.442830+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-12T16:54:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:29.635669+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:54:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:29.644628+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:29.435802+0000 mon.a (mon.0) 840 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:29.442773+0000 mon.a (mon.0) 841 : audit [INF] from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:29.442830+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:29.635669+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:29.644628+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:29.435802+0000 mon.a (mon.0) 840 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:29.442773+0000 mon.a (mon.0) 841 : audit [INF] from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:29.442830+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:29.635669+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:54:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:29.644628+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:54:31.233 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:54:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:30] "GET /metrics HTTP/1.1" 200 175411 "" "Prometheus/2.33.4" 2024-02-12T16:54:31.250 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:54:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:31.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:54:31.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:31.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:30.383764+0000 mgr.y (mgr.24481) 190 : cluster [DBG] pgmap v203: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 930 op/s 2024-02-12T16:54:31.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:30.649246+0000 osd.4 (osd.4) 67 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:54:31.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:30.654815+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:54:31.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:30.765016+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:54:31.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:30.769955+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:54:31.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:30.383764+0000 mgr.y (mgr.24481) 190 : cluster [DBG] pgmap v203: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 930 op/s 2024-02-12T16:54:31.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:30.649246+0000 osd.4 (osd.4) 67 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:54:31.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:30.654815+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:54:31.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:30.765016+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:54:31.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:30.769955+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:54:31.578 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:30.383764+0000 mgr.y (mgr.24481) 190 : cluster [DBG] pgmap v203: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 930 op/s 2024-02-12T16:54:31.578 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:30.649246+0000 osd.4 (osd.4) 67 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:54:31.578 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:30.654815+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:54:31.578 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:30.765016+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:54:31.579 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:30.769955+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:54:31.597 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:31.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:31.701 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:31.758 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:31.796 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:31.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:31.882 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:31.882 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:31.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:31.935 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:32.023 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:54:32.063 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:54:32.094 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:54:32.128 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:54:32.139 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:32.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:32.162 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:54:32.189 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:54:32.218 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:54:32.242 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:54:32.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:54:32.246 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:54:32.247 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:54:32.247 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:54:32.247 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:54:32.247 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:54:32.247 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:54:32.247 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:54:32.247 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:54:32.285 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:32.408 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:54:32.645 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:30.969489+0000 osd.1 (osd.1) 97 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:54:32.645 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:30.972445+0000 osd.1 (osd.1) 98 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:54:32.645 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:31.383314+0000 osd.2 (osd.2) 77 : cluster [DBG] 6.f scrub starts 2024-02-12T16:54:32.645 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:31.387136+0000 osd.2 (osd.2) 78 : cluster [DBG] 6.f scrub ok 2024-02-12T16:54:32.645 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:31.780307+0000 osd.7 (osd.7) 79 : cluster [DBG] 2.b scrub starts 2024-02-12T16:54:32.645 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:31.784616+0000 osd.7 (osd.7) 80 : cluster [DBG] 2.b scrub ok 2024-02-12T16:54:32.645 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:31.983096+0000 mon.a (mon.0) 843 : audit [DBG] from='client.? v1:172.21.15.33:0/600957478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:32.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:30.969489+0000 osd.1 (osd.1) 97 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:54:32.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:30.972445+0000 osd.1 (osd.1) 98 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:54:32.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:31.383314+0000 osd.2 (osd.2) 77 : cluster [DBG] 6.f scrub starts 2024-02-12T16:54:32.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:31.387136+0000 osd.2 (osd.2) 78 : cluster [DBG] 6.f scrub ok 2024-02-12T16:54:32.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:31.780307+0000 osd.7 (osd.7) 79 : cluster [DBG] 2.b scrub starts 2024-02-12T16:54:32.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:31.784616+0000 osd.7 (osd.7) 80 : cluster [DBG] 2.b scrub ok 2024-02-12T16:54:32.646 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:31.983096+0000 mon.a (mon.0) 843 : audit [DBG] from='client.? v1:172.21.15.33:0/600957478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:32.682 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:30.969489+0000 osd.1 (osd.1) 97 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:54:32.682 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:30.972445+0000 osd.1 (osd.1) 98 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:54:32.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:31.383314+0000 osd.2 (osd.2) 77 : cluster [DBG] 6.f scrub starts 2024-02-12T16:54:32.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:31.387136+0000 osd.2 (osd.2) 78 : cluster [DBG] 6.f scrub ok 2024-02-12T16:54:32.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:31.780307+0000 osd.7 (osd.7) 79 : cluster [DBG] 2.b scrub starts 2024-02-12T16:54:32.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:31.784616+0000 osd.7 (osd.7) 80 : cluster [DBG] 2.b scrub ok 2024-02-12T16:54:32.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:31.983096+0000 mon.a (mon.0) 843 : audit [DBG] from='client.? v1:172.21.15.33:0/600957478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:32.683 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:54:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:54:32.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:54:32.744 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:54:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:32.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:54:32.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:32.887 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:32.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:33.035 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:33.087 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:33.107 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:33.110 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:33.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:33.211 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:33.213 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:33.243 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:33.370 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:33.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:33.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:33.491 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:31.952459+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.12 scrub starts 2024-02-12T16:54:33.491 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:31.953991+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.12 scrub ok 2024-02-12T16:54:33.491 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:32.384161+0000 mgr.y (mgr.24481) 191 : cluster [DBG] pgmap v204: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 916 op/s 2024-02-12T16:54:33.491 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:32.679323+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.c scrub starts 2024-02-12T16:54:33.491 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:32.693792+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.c scrub ok 2024-02-12T16:54:33.491 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:32.888580+0000 osd.5 (osd.5) 75 : cluster [DBG] 4.15 scrub starts 2024-02-12T16:54:33.491 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:32.895394+0000 osd.5 (osd.5) 76 : cluster [DBG] 4.15 scrub ok 2024-02-12T16:54:33.513 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:31.952459+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.12 scrub starts 2024-02-12T16:54:33.514 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:31.953991+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.12 scrub ok 2024-02-12T16:54:33.514 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:32.384161+0000 mgr.y (mgr.24481) 191 : cluster [DBG] pgmap v204: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 916 op/s 2024-02-12T16:54:33.514 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:32.679323+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.c scrub starts 2024-02-12T16:54:33.514 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:32.693792+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.c scrub ok 2024-02-12T16:54:33.514 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:32.888580+0000 osd.5 (osd.5) 75 : cluster [DBG] 4.15 scrub starts 2024-02-12T16:54:33.514 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:32.895394+0000 osd.5 (osd.5) 76 : cluster [DBG] 4.15 scrub ok 2024-02-12T16:54:33.518 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:33.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:31.952459+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.12 scrub starts 2024-02-12T16:54:33.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:31.953991+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.12 scrub ok 2024-02-12T16:54:33.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:32.384161+0000 mgr.y (mgr.24481) 191 : cluster [DBG] pgmap v204: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 916 op/s 2024-02-12T16:54:33.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:32.679323+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.c scrub starts 2024-02-12T16:54:33.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:32.693792+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.c scrub ok 2024-02-12T16:54:33.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:32.888580+0000 osd.5 (osd.5) 75 : cluster [DBG] 4.15 scrub starts 2024-02-12T16:54:33.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:32.895394+0000 osd.5 (osd.5) 76 : cluster [DBG] 4.15 scrub ok 2024-02-12T16:54:33.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:54:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:54:34.472 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:34.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:34.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:54:34.614 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:34.636 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:32.954483+0000 osd.1 (osd.1) 101 : cluster [DBG] 7.b scrub starts 2024-02-12T16:54:34.636 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:33.547420+0000 mgr.y (mgr.24481) 192 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:34.637 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:32.954483+0000 osd.1 (osd.1) 101 : cluster [DBG] 7.b scrub starts 2024-02-12T16:54:34.637 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:33.547420+0000 mgr.y (mgr.24481) 192 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:34.707 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:32.954483+0000 osd.1 (osd.1) 101 : cluster [DBG] 7.b scrub starts 2024-02-12T16:54:34.707 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:33.547420+0000 mgr.y (mgr.24481) 192 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:34.717 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:54:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:34.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:54:34.745 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:54:34.745 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:54:34.745 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:54:34.745 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd out 3 2024-02-12T16:54:34.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:34.859 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:34.988 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:34.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:35.003 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:54:35.003 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T16:54:35.003 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T16:54:35.003 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T16:54:35.003 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:54:35.092 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:54:35.123 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:54:35.149 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:35.180 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:54:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:35.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:54:35.286 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:35.349 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:35.379 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:35.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:35.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:35.515 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:35.607 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:35.680 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:34.387149+0000 mgr.y (mgr.24481) 193 : cluster [DBG] pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-02-12T16:54:35.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:54:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:35.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:35.819 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:35.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:34.387149+0000 mgr.y (mgr.24481) 193 : cluster [DBG] pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-02-12T16:54:35.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:34.387149+0000 mgr.y (mgr.24481) 193 : cluster [DBG] pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-02-12T16:54:35.888 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:35.926 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:35.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:36.031 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:54:36.044 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:36.101 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:36.207 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:54:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:36.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:54:36.316 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:36.392 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:36.505 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:36.509 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:36.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:36.577 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T16:54:36.615 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:36.648 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:34.968878+0000 osd.1 (osd.1) 102 : cluster [DBG] 7.b scrub ok 2024-02-12T16:54:36.648 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:35.171541+0000 osd.0 (osd.0) 75 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:54:36.648 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:35.191034+0000 osd.0 (osd.0) 76 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:54:36.648 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:35.409479+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.e scrub starts 2024-02-12T16:54:36.649 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:35.871279+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.33:0/2199006016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:36.649 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:35.878817+0000 mon.a (mon.0) 844 : audit [INF] from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:36.654 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:34.968878+0000 osd.1 (osd.1) 102 : cluster [DBG] 7.b scrub ok 2024-02-12T16:54:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:35.171541+0000 osd.0 (osd.0) 75 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:54:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:35.191034+0000 osd.0 (osd.0) 76 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:54:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:35.409479+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.e scrub starts 2024-02-12T16:54:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:35.871279+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.33:0/2199006016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:36.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:35.878817+0000 mon.a (mon.0) 844 : audit [INF] from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:34.968878+0000 osd.1 (osd.1) 102 : cluster [DBG] 7.b scrub ok 2024-02-12T16:54:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:35.171541+0000 osd.0 (osd.0) 75 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:54:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:35.191034+0000 osd.0 (osd.0) 76 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:54:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:35.409479+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.e scrub starts 2024-02-12T16:54:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:35.871279+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.33:0/2199006016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:35.878817+0000 mon.a (mon.0) 844 : audit [INF] from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:54:36.863 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:36.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:36.969 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:37.017 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:37.113 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:37.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:37.218 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:37.375 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:37.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:37.480 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:54:37.535 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:37.978 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:54:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T16:54:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T16:54:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:15778 7.0 7:0537be68:::benchmark_data_smithi033_81060_object15777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:52:39.093511+0000", 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.87998268699999, 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054388443000000002, 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 15778 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093511+0000", 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093511+0000", 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093509+0000", 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6107000000000003e-05 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093565+0000", 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8630000000000003e-06 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093575+0000", 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4399999999999998e-06 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093579+0000", 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9889999999999998e-05 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093629+0000", 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120563 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093750+0000", 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160741 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.093911+0000", 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094067499999999997 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.094851+0000", 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002873324 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097725+0000", 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050079897999999998 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147805+0000", 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4348000000000002e-05 2024-02-12T16:54:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147859+0000", 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0401999999999997e-05 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147899+0000", 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054388443000000002 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.14838.0:15788 7.b e104/93 7:d9e8c0f0:::benchmark_data_smithi033_81060_object15787:head v 104'1005, mlcod=104'1003)", 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:52:39.097682+0000", 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.87581143, 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050172663999999999, 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097682+0000", 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097682+0000", 2024-02-12T16:54:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097682+0000", 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7243999999999999e-05 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097709+0000", 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2089999999999999e-06 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097713+0000", 2024-02-12T16:54:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2399999999999996e-06 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097722+0000", 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0874000000000003e-05 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097753+0000", 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2306e-05 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097766+0000", 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050057852999999999 2024-02-12T16:54:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147823+0000", 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.129e-05 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147855+0000", 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050172663999999999 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.14838.0:15778 7.0 e104/93 ondisk, result = 0)", 2024-02-12T16:54:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:52:39.097845+0000", 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.875648955, 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050062191999999998, 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097845+0000", 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097845+0000", 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:54:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097844+0000", 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3840000000000001e-06 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097847+0000", 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4769999999999997e-06 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097852+0000", 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4030000000000001e-06 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097854+0000", 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3728000000000001e-05 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097878+0000", 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1030000000000002e-06 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.097881+0000", 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050025766999999999 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147907+0000", 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050062191999999998 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.14838.0:15792 7.a e104/93 7:50b6c5b0:::benchmark_data_smithi033_81060_object15791:head v 104'993, mlcod=104'991)", 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:52:39.099089+0000", 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.87440485899999, 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.049677820999999997, 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.099089+0000", 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.099089+0000", 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.099087+0000", 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0790000000000005e-05 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.099158+0000", 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048637967999999997 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147796+0000", 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4382999999999999e-05 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147811+0000", 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9225000000000003e-05 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147870+0000", 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6123999999999999e-05 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147896+0000", 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084242500000000005 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.148738+0000", 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8209e-05 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.148767+0000", 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.049677820999999997 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:15793 7.0 7:0f702cff:::benchmark_data_smithi033_81060_object15792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:52:39.099418+0000", 2024-02-12T16:54:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.87407532, 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051642194000000002, 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 15793 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.099418+0000", 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.099418+0000", 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.099416+0000", 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1008999999999999e-05 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.099467+0000", 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048363544000000001 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147831+0000", 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8079999999999999e-06 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147840+0000", 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7853000000000001e-05 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147928+0000", 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000140457 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.148069+0000", 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100413 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.148169+0000", 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027346380000000002 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.150904+0000", 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8021e-05 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.150952+0000", 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6087000000000001e-05 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.150998+0000", 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9301999999999997e-05 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.151037+0000", 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3198e-05 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.151060+0000", 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051642194000000002 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.14838.0:15796 7.b e104/93 7:dfc59b0b:::benchmark_data_smithi033_81060_object15795:head v 104'1006, mlcod=104'1003)", 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:52:39.101859+0000", 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.871635052, 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047827351999999997, 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.101859+0000", 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.101859+0000", 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.101857+0000", 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5739000000000003e-05 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.101913+0000", 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045885816000000003 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147799+0000", 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1542999999999999e-05 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147810+0000", 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9424000000000005e-05 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147880+0000", 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0171999999999999e-05 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.147900+0000", 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001763337 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.149663+0000", 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2804000000000001e-05 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:39.149686+0000", 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047827351999999997 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:49340 7.0 7:076dc461:::benchmark_data_smithi033_81060_object49339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:52:59.952970+0000", 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.020523776999994, 2024-02-12T16:54:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063100206000000006, 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49340 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.952970+0000", 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.952970+0000", 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.952967+0000", 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6159999999999998e-05 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.953023+0000", 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0064000000000001e-05 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.953033+0000", 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0000000000000004e-06 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.953038+0000", 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8012000000000001e-05 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.953076+0000", 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.433e-05 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.953171+0000", 2024-02-12T16:54:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125289 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.953296+0000", 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014413920000000001 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.954737+0000", 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026046100000000003 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:59.954998+0000", 2024-02-12T16:54:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.060986855 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:00.015985+0000", 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2459e-05 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:00.016037+0000", 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2789000000000001e-05 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:00.016070+0000", 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063100206000000006 2024-02-12T16:54:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:50735 7.0 7:09bbb28d:::benchmark_data_smithi033_81060_object50734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:01.048218+0000", 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.925275959000004, 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054035544999999997, 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50735 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048218+0000", 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048218+0000", 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048216+0000", 2024-02-12T16:54:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4264000000000003e-05 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048250+0000", 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0648999999999999e-05 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048260+0000", 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5139999999999998e-06 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048267+0000", 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0373999999999997e-05 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048307+0000", 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2769999999999994e-05 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048400+0000", 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9739000000000002e-05 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.048500+0000", 2024-02-12T16:54:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015391409999999999 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.050039+0000", 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011622259999999999 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.051201+0000", 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050959942000000001 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.102161+0000", 2024-02-12T16:54:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3888000000000003e-05 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.102215+0000", 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8098e-05 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.102253+0000", 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054035544999999997 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:52164 7.0 7:075eaf1c:::benchmark_data_smithi033_81060_object52163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:54:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:02.179985+0000", 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.793508322999998, 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.03898886, 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52164 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.179985+0000", 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.179985+0000", 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.179983+0000", 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2874999999999995e-05 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180066+0000", 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3389e-05 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180079+0000", 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7910000000000005e-06 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180087+0000", 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4301999999999997e-05 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180131+0000", 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000158658 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180290+0000", 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000146837 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180437+0000", 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001765377 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.182202+0000", 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1560000000000003e-06 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.182207+0000", 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.036705596 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.218913+0000", 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5988999999999997e-05 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.218949+0000", 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5123e-05 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.218974+0000", 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.03898886 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:53369 7.0 7:096de9f1:::benchmark_data_smithi033_81060_object53368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:03.114516+0000", 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.858977926999998, 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042260856999999999, 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 53369 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114516+0000", 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114516+0000", 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114512+0000", 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8978000000000001e-05 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114601+0000", 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8338999999999998e-05 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114620+0000", 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2679999999999994e-06 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114628+0000", 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8573999999999999e-05 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114687+0000", 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000129324 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114816+0000", 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015455699999999999 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.114970+0000", 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018745120000000001 2024-02-12T16:54:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.116845+0000", 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001239883 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.118085+0000", 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038594134000000002 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.156679+0000", 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3847000000000001e-05 2024-02-12T16:54:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.156733+0000", 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3662000000000003e-05 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:03.156776+0000", 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042260856999999999 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:79618 7.0 7:0dd80f24:::benchmark_data_smithi033_81060_object79617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:54:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:27.940613+0000", 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.032880442000007, 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050248339000000003, 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 79618 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940613+0000", 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940613+0000", 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940610+0000", 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5663000000000001e-05 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940646+0000", 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7419e-05 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940663+0000", 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9449999999999993e-06 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940671+0000", 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3578999999999997e-05 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940715+0000", 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0630999999999993e-05 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940805+0000", 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014329100000000001 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.940949+0000", 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061096910000000004 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.947058+0000", 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025482510000000001 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.949607+0000", 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.041159251000000001 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.990766+0000", 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7812999999999997e-05 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.990824+0000", 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7871999999999999e-05 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.990861+0000", 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050248339000000003 2024-02-12T16:54:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:79623 7.0 7:0ad86157:::benchmark_data_smithi033_81060_object79622:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:27.944242+0000", 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.029251160000001, 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046773281999999999, 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 79623 2024-02-12T16:54:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944242+0000", 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944242+0000", 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944240+0000", 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2126999999999997e-05 2024-02-12T16:54:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944293+0000", 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1302e-05 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944304+0000", 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6589999999999998e-06 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944309+0000", 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3519000000000001e-05 2024-02-12T16:54:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944352+0000", 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015274200000000001 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944505+0000", 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016434500000000001 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.944669+0000", 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025940170000000001 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.947263+0000", 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002455128 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.949718+0000", 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.041219653000000002 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.990938+0000", 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4894000000000001e-05 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.990983+0000", 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2873999999999997e-05 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:27.991016+0000", 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046773281999999999 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:80528 7.0 7:0895d338:::benchmark_data_smithi033_81060_object80527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:28.835577+0000", 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.137916110999996, 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059976527000000002, 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 80528 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835577+0000", 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835577+0000", 2024-02-12T16:54:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835575+0000", 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4158999999999993e-05 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835639+0000", 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4221e-05 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835653+0000", 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2400000000000004e-06 2024-02-12T16:54:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835659+0000", 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3189999999999996e-05 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835722+0000", 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000132445 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835855+0000", 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138166 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.835993+0000", 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047472920000000002 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.840740+0000", 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7655999999999998e-05 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.840758+0000", 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.05470237 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.895460+0000", 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0739000000000003e-05 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.895511+0000", 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2865999999999999e-05 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.895554+0000", 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059976527000000002 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:80530 7.0 7:090df791:::benchmark_data_smithi033_81060_object80529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:28.838056+0000", 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.135437207999999, 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057595737000000001, 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 80530 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838056+0000", 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838056+0000", 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838054+0000", 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5322999999999997e-05 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838130+0000", 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0754000000000001e-05 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838140+0000", 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.694e-06 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838146+0000", 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113675 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838260+0000", 2024-02-12T16:54:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012404899999999999 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838384+0000", 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015996900000000001 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.838544+0000", 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022893750000000002 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.840833+0000", 2024-02-12T16:54:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027520650000000002 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.843585+0000", 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052009649999999998 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.895595+0000", 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3652999999999998e-05 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.895629+0000", 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3499999999999999e-05 2024-02-12T16:54:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:28.895652+0000", 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057595737000000001 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:100739 7.0 7:03d378c8:::benchmark_data_smithi033_81060_object100738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:51.562993+0000", 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.410500474999999, 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033724213000000003, 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 100739 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.562993+0000", 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.562993+0000", 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:54:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.562992+0000", 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2444999999999997e-05 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.563044+0000", 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4890000000000001e-06 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.563054+0000", 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1539999999999999e-06 2024-02-12T16:54:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.563060+0000", 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020212559999999999 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.565081+0000", 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8101000000000002e-05 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.565169+0000", 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011303600000000001 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.565282+0000", 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0039443489999999998 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.569226+0000", 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0078040089999999998 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.577030+0000", 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019617622000000001 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.596648+0000", 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2991000000000003e-05 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.596681+0000", 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6186999999999998e-05 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:51.596717+0000", 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033724213000000003 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:103714 7.0 7:0887e734:::benchmark_data_smithi033_81060_object103713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:55.169787+0000", 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.803706196, 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.039667332999999999, 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 103714 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.169787+0000", 2024-02-12T16:54:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.169787+0000", 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.169786+0000", 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0572000000000001e-05 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.169856+0000", 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2041e-05 2024-02-12T16:54:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.169868+0000", 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4320000000000004e-06 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.169875+0000", 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014577710000000001 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.171333+0000", 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125279 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.171458+0000", 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162151 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.171620+0000", 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024671820000000001 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.174087+0000", 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047456590000000002 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.178833+0000", 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030512060000000001 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.209345+0000", 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1525999999999993e-05 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.209407+0000", 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8152999999999998e-05 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.209455+0000", 2024-02-12T16:54:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.039667332999999999 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:103864 7.0 7:0439fbe5:::benchmark_data_smithi033_81060_object103863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:55.370192+0000", 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.603301944999998, 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037601264000000002, 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 103864 2024-02-12T16:54:38.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370192+0000", 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370192+0000", 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370189+0000", 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8140999999999999e-05 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370237+0000", 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.363e-05 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370251+0000", 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3740000000000003e-06 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370257+0000", 2024-02-12T16:54:38.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1978000000000003e-05 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370309+0000", 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015935600000000001 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370469+0000", 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000178421 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.370647+0000", 2024-02-12T16:54:38.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031189019999999998 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.373766+0000", 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0085786939999999996 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.382345+0000", 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025355174000000001 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.407700+0000", 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4138000000000002e-05 2024-02-12T16:54:38.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.407754+0000", 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8897999999999999e-05 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.407793+0000", 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037601264000000002 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:107330 7.0 7:03a36eda:::benchmark_data_smithi033_81060_object107329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e129)", 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:59.441082+0000", 2024-02-12T16:54:38.066 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.532411895999999, 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034284203999999999, 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 107330 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441082+0000", 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441082+0000", 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441080+0000", 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6758999999999997e-05 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441126+0000", 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.541e-06 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441135+0000", 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9080000000000003e-06 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441140+0000", 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3482000000000001e-05 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441193+0000", 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013550199999999999 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441329+0000", 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000182253 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.441511+0000", 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002358359 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.443869+0000", 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018924982999999999 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.462794+0000", 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012505435000000001 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.475300+0000", 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7901999999999997e-05 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.475338+0000", 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8056e-05 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.475366+0000", 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034284203999999999 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:140615 7.0 7:0f16c713:::benchmark_data_smithi033_81060_object30163:head [delete] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:54:36.059081+0000", 2024-02-12T16:54:38.073 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.9144124499999999, 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037060989000000003, 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 140615 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059081+0000", 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059081+0000", 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059079+0000", 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1879999999999999e-06 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059086+0000", 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3880000000000006e-06 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059095+0000", 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8690000000000003e-06 2024-02-12T16:54:38.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059100+0000", 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0153000000000001e-05 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059130+0000", 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013901500000000001 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059269+0000", 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000155272 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.059424+0000", 2024-02-12T16:54:38.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052080069999999997 2024-02-12T16:54:38.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:36.387565+0000 mgr.y (mgr.24481) 194 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-02-12T16:54:38.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:36.541139+0000 mon.a (mon.0) 845 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:54:38.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:36.541181+0000 mon.a (mon.0) 846 : cluster [INF] Cluster is now healthy 2024-02-12T16:54:38.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:36.551793+0000 mon.a (mon.0) 847 : audit [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:54:38.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:36.551860+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.064632+0000", 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029892206000000001 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.094524+0000", 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015442559999999999 2024-02-12T16:54:38.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096069+0000", 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9107999999999999e-05 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096108+0000", 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4277999999999999e-05 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096142+0000", 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037060989000000003 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:140640 7.0 7:0d4b507b:::benchmark_data_smithi033_81060_object30188:head [delete] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:54:36.096870+0000", 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.876623503, 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046846827000000001, 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:38.080 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 140640 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096870+0000", 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096870+0000", 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096868+0000", 2024-02-12T16:54:38.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1879999999999996e-06 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096875+0000", 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0158e-05 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096885+0000", 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4870000000000004e-06 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096890+0000", 2024-02-12T16:54:38.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7465999999999997e-05 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.096927+0000", 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7991999999999994e-05 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.097015+0000", 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9017999999999994e-05 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,3", 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.097104+0000", 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003159317 2024-02-12T16:54:38.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.100263+0000", 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035985542000000002 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.136249+0000", 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0073832409999999996 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.143632+0000", 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0302000000000001e-05 2024-02-12T16:54:38.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.143673+0000", 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4375000000000001e-05 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:36.143717+0000", 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046846827000000001 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:38.085 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:36.387565+0000 mgr.y (mgr.24481) 194 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:36.541139+0000 mon.a (mon.0) 845 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:36.541181+0000 mon.a (mon.0) 846 : cluster [INF] Cluster is now healthy 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:36.551793+0000 mon.a (mon.0) 847 : audit [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:36.551860+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:36.387565+0000 mgr.y (mgr.24481) 194 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:36.541139+0000 mon.a (mon.0) 845 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:36.541181+0000 mon.a (mon.0) 846 : cluster [INF] Cluster is now healthy 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:36.551793+0000 mon.a (mon.0) 847 : audit [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:54:38.086 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:36.551860+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-12T16:54:38.775 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:54:38.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:36.749668+0000 osd.6 (osd.6) 77 : cluster [DBG] 4.e scrub starts 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:36.753317+0000 osd.6 (osd.6) 78 : cluster [DBG] 4.e scrub ok 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:37.036839+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:37.041314+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:37.494601+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:37.497502+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.031391+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.e scrub ok 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:36.749668+0000 osd.6 (osd.6) 77 : cluster [DBG] 4.e scrub starts 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:36.753317+0000 osd.6 (osd.6) 78 : cluster [DBG] 4.e scrub ok 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:37.036839+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:54:38.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:37.041314+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:54:38.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:37.494601+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:54:38.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:37.497502+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:54:38.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.031391+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.e scrub ok 2024-02-12T16:54:38.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:36.749668+0000 osd.6 (osd.6) 77 : cluster [DBG] 4.e scrub starts 2024-02-12T16:54:38.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:36.753317+0000 osd.6 (osd.6) 78 : cluster [DBG] 4.e scrub ok 2024-02-12T16:54:38.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:37.036839+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.9 scrub starts 2024-02-12T16:54:38.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:37.041314+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.9 scrub ok 2024-02-12T16:54:38.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:37.494601+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:54:38.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:37.497502+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:54:38.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.031391+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.e scrub ok 2024-02-12T16:54:39.120 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:39.141 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:54:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:39.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:54:39.246 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:39.286 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:39.396 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:39.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:39.504 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:39.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:54:39.089+0000 7fd96a9de700 -1 mon.a@0(leader).osd e136 definitely_dead 0 2024-02-12T16:54:39.562 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:39.633 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:39.633 INFO:teuthology.orchestra.run.smithi033.stderr:marked out osd.3. 2024-02-12T16:54:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:39.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:39.740 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:39.768 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.011432+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.f scrub starts 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.016462+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.f scrub ok 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.385274+0000 osd.2 (osd.2) 81 : cluster [DBG] 2.c scrub starts 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.389035+0000 mgr.y (mgr.24481) 195 : cluster [DBG] pgmap v208: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 863 op/s 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.398791+0000 osd.2 (osd.2) 82 : cluster [DBG] 2.c scrub ok 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.707610+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.b scrub starts 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.715644+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.b scrub ok 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.778059+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.1d scrub starts 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.784153+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.1d scrub ok 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.872651+0000 osd.5 (osd.5) 77 : cluster [DBG] 2.12 scrub starts 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:38.875598+0000 osd.5 (osd.5) 78 : cluster [DBG] 2.12 scrub ok 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:39.003648+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.a scrub starts 2024-02-12T16:54:39.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:39.006475+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.a scrub ok 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:39.085342+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.33:0/2981709765' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:39.090773+0000 mon.a (mon.0) 849 : audit [INF] from='client.24916 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:39.090891+0000 mon.a (mon.0) 850 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.011432+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.f scrub starts 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.016462+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.f scrub ok 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.385274+0000 osd.2 (osd.2) 81 : cluster [DBG] 2.c scrub starts 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.389035+0000 mgr.y (mgr.24481) 195 : cluster [DBG] pgmap v208: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 863 op/s 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.398791+0000 osd.2 (osd.2) 82 : cluster [DBG] 2.c scrub ok 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.707610+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.b scrub starts 2024-02-12T16:54:39.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.715644+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.b scrub ok 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.778059+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.1d scrub starts 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.784153+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.1d scrub ok 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.872651+0000 osd.5 (osd.5) 77 : cluster [DBG] 2.12 scrub starts 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:38.875598+0000 osd.5 (osd.5) 78 : cluster [DBG] 2.12 scrub ok 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:39.003648+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.a scrub starts 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:39.006475+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.a scrub ok 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:39.085342+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.33:0/2981709765' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:39.090773+0000 mon.a (mon.0) 849 : audit [INF] from='client.24916 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-12T16:54:39.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:39.090891+0000 mon.a (mon.0) 850 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-12T16:54:39.918 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:39.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:39.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.011432+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.f scrub starts 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.016462+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.f scrub ok 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.385274+0000 osd.2 (osd.2) 81 : cluster [DBG] 2.c scrub starts 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.389035+0000 mgr.y (mgr.24481) 195 : cluster [DBG] pgmap v208: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 863 op/s 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.398791+0000 osd.2 (osd.2) 82 : cluster [DBG] 2.c scrub ok 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.707610+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.b scrub starts 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.715644+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.b scrub ok 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.778059+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.1d scrub starts 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.784153+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.1d scrub ok 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.872651+0000 osd.5 (osd.5) 77 : cluster [DBG] 2.12 scrub starts 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:38.875598+0000 osd.5 (osd.5) 78 : cluster [DBG] 2.12 scrub ok 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:39.003648+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.a scrub starts 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:39.006475+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.a scrub ok 2024-02-12T16:54:39.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:39.085342+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.33:0/2981709765' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-12T16:54:39.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:39.090773+0000 mon.a (mon.0) 849 : audit [INF] from='client.24916 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-12T16:54:39.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:39.090891+0000 mon.a (mon.0) 850 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-12T16:54:40.022 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:40.051 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:40.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:54:40.369 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:54:40.388 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:54:40.405 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:54:40.423 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:54:40.441 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:54:40.458 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:54:40.482 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:54:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:54:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:54:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:54:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:54:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:54:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:54:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:54:40.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:54:40.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:39.414040+0000 osd.2 (osd.2) 83 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:54:40.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:39.421166+0000 osd.2 (osd.2) 84 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:54:40.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:39.604797+0000 mon.a (mon.0) 851 : audit [INF] from='client.24916 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-12T16:54:40.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:39.604856+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e137: 8 total, 8 up, 7 in 2024-02-12T16:54:40.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:40.332870+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.33:0/3819499009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:40.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:39.414040+0000 osd.2 (osd.2) 83 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:39.421166+0000 osd.2 (osd.2) 84 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:39.604797+0000 mon.a (mon.0) 851 : audit [INF] from='client.24916 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:39.604856+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e137: 8 total, 8 up, 7 in 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:40.332870+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.33:0/3819499009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:39.414040+0000 osd.2 (osd.2) 83 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:39.421166+0000 osd.2 (osd.2) 84 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:39.604797+0000 mon.a (mon.0) 851 : audit [INF] from='client.24916 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:39.604856+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e137: 8 total, 8 up, 7 in 2024-02-12T16:54:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:40.332870+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.33:0/3819499009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:40.969 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:40.969 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:41.108 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:54:41.139 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:54:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:40] "GET /metrics HTTP/1.1" 200 175422 "" "Prometheus/2.33.4" 2024-02-12T16:54:41.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:39.664464+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:54:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:39.665370+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:54:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:40.191064+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.10 scrub starts 2024-02-12T16:54:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:40.193958+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.10 scrub ok 2024-02-12T16:54:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:40.389352+0000 mgr.y (mgr.24481) 196 : cluster [DBG] pgmap v210: 148 pgs: 148 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-02-12T16:54:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:40.435678+0000 osd.2 (osd.2) 85 : cluster [DBG] 7.8 scrub starts 2024-02-12T16:54:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:40.610392+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e138: 8 total, 8 up, 7 in 2024-02-12T16:54:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.172535+0000 osd.2 (osd.2) 86 : cluster [DBG] 7.8 scrub ok 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:39.664464+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:39.665370+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:40.191064+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.10 scrub starts 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:40.193958+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.10 scrub ok 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:40.389352+0000 mgr.y (mgr.24481) 196 : cluster [DBG] pgmap v210: 148 pgs: 148 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:40.435678+0000 osd.2 (osd.2) 85 : cluster [DBG] 7.8 scrub starts 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:40.610392+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e138: 8 total, 8 up, 7 in 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.172535+0000 osd.2 (osd.2) 86 : cluster [DBG] 7.8 scrub ok 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:39.664464+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:54:42.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:39.665370+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:54:42.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:40.191064+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.10 scrub starts 2024-02-12T16:54:42.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:40.193958+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.10 scrub ok 2024-02-12T16:54:42.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:40.389352+0000 mgr.y (mgr.24481) 196 : cluster [DBG] pgmap v210: 148 pgs: 148 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-02-12T16:54:42.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:40.435678+0000 osd.2 (osd.2) 85 : cluster [DBG] 7.8 scrub starts 2024-02-12T16:54:42.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:40.610392+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e138: 8 total, 8 up, 7 in 2024-02-12T16:54:42.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.172535+0000 osd.2 (osd.2) 86 : cluster [DBG] 7.8 scrub ok 2024-02-12T16:54:42.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.610521+0000 osd.6 (osd.6) 81 : cluster [DBG] 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 136'8942 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.611221+0000 osd.6 (osd.6) 82 : cluster [DBG] 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 136'8973 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.611269+0000 osd.6 (osd.6) 83 : cluster [DBG] 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 136'9157 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.611937+0000 osd.2 (osd.2) 87 : cluster [DBG] 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 136'8980 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.612679+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e139: 8 total, 8 up, 7 in 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.614346+0000 osd.5 (osd.5) 79 : cluster [DBG] 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8912 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.617289+0000 osd.1 (osd.1) 109 : cluster [DBG] 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 136'8907 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.619711+0000 osd.0 (osd.0) 79 : cluster [DBG] 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 136'9109 2024-02-12T16:54:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.622735+0000 osd.0 (osd.0) 80 : cluster [DBG] 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 136'8980 2024-02-12T16:54:42.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:41.623464+0000 osd.2 (osd.2) 88 : cluster [DBG] 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8850 2024-02-12T16:54:42.992 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:54:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.610521+0000 osd.6 (osd.6) 81 : cluster [DBG] 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 136'8942 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.611221+0000 osd.6 (osd.6) 82 : cluster [DBG] 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 136'8973 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.611269+0000 osd.6 (osd.6) 83 : cluster [DBG] 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 136'9157 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.611937+0000 osd.2 (osd.2) 87 : cluster [DBG] 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 136'8980 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.612679+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e139: 8 total, 8 up, 7 in 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.614346+0000 osd.5 (osd.5) 79 : cluster [DBG] 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8912 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.617289+0000 osd.1 (osd.1) 109 : cluster [DBG] 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 136'8907 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.619711+0000 osd.0 (osd.0) 79 : cluster [DBG] 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 136'9109 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.622735+0000 osd.0 (osd.0) 80 : cluster [DBG] 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 136'8980 2024-02-12T16:54:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:41.623464+0000 osd.2 (osd.2) 88 : cluster [DBG] 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8850 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.610521+0000 osd.6 (osd.6) 81 : cluster [DBG] 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 136'8942 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.611221+0000 osd.6 (osd.6) 82 : cluster [DBG] 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 136'8973 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.611269+0000 osd.6 (osd.6) 83 : cluster [DBG] 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 136'9157 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.611937+0000 osd.2 (osd.2) 87 : cluster [DBG] 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 136'8980 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.612679+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e139: 8 total, 8 up, 7 in 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.614346+0000 osd.5 (osd.5) 79 : cluster [DBG] 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8912 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.617289+0000 osd.1 (osd.1) 109 : cluster [DBG] 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 136'8907 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.619711+0000 osd.0 (osd.0) 79 : cluster [DBG] 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 136'9109 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.622735+0000 osd.0 (osd.0) 80 : cluster [DBG] 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 136'8980 2024-02-12T16:54:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:41.623464+0000 osd.2 (osd.2) 88 : cluster [DBG] 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8850 2024-02-12T16:54:43.052 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:43.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:54:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:54:43.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:42.389757+0000 mgr.y (mgr.24481) 197 : cluster [DBG] pgmap v213: 148 pgs: 148 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 722 op/s 2024-02-12T16:54:44.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:42.389757+0000 mgr.y (mgr.24481) 197 : cluster [DBG] pgmap v213: 148 pgs: 148 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 722 op/s 2024-02-12T16:54:44.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:42.389757+0000 mgr.y (mgr.24481) 197 : cluster [DBG] pgmap v213: 148 pgs: 148 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 722 op/s 2024-02-12T16:54:44.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:43.375719+0000 osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:44.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:43.375774+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:44.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:43.560461+0000 mgr.y (mgr.24481) 198 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:44.683 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:43.646635+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e140: 8 total, 8 up, 7 in 2024-02-12T16:54:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:43.375719+0000 osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:43.375774+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:43.560461+0000 mgr.y (mgr.24481) 198 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:43.646635+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e140: 8 total, 8 up, 7 in 2024-02-12T16:54:45.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:43.375719+0000 osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:45.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:43.375774+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:45.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:43.560461+0000 mgr.y (mgr.24481) 198 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:45.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:43.646635+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e140: 8 total, 8 up, 7 in 2024-02-12T16:54:45.059 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:45.059 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:45.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:54:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:44.390219+0000 mgr.y (mgr.24481) 199 : cluster [DBG] pgmap v215: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 466 op/s; 41631/228813 objects misplaced (18.194%); 14 MiB/s, 237 objects/s recovering 2024-02-12T16:54:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:44.664129+0000 mon.a (mon.0) 856 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:54:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:44.974504+0000 osd.5 (osd.5) 80 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:45.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:44.974556+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:44.390219+0000 mgr.y (mgr.24481) 199 : cluster [DBG] pgmap v215: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 466 op/s; 41631/228813 objects misplaced (18.194%); 14 MiB/s, 237 objects/s recovering 2024-02-12T16:54:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:44.664129+0000 mon.a (mon.0) 856 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:54:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:44.974504+0000 osd.5 (osd.5) 80 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:44.974556+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:44.390219+0000 mgr.y (mgr.24481) 199 : cluster [DBG] pgmap v215: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 466 op/s; 41631/228813 objects misplaced (18.194%); 14 MiB/s, 237 objects/s recovering 2024-02-12T16:54:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:44.664129+0000 mon.a (mon.0) 856 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:54:46.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:44.974504+0000 osd.5 (osd.5) 80 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:46.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:44.974556+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:46.230 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:46.832 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:46.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:46.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:46.986 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:45.717491+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e141: 8 total, 8 up, 7 in 2024-02-12T16:54:46.988 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:47.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:45.717491+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e141: 8 total, 8 up, 7 in 2024-02-12T16:54:47.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:45.717491+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e141: 8 total, 8 up, 7 in 2024-02-12T16:54:47.123 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:47.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:47.231 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:47.316 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:47.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:46.390699+0000 mgr.y (mgr.24481) 200 : cluster [DBG] pgmap v217: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 41631/228813 objects misplaced (18.194%); 11 MiB/s, 196 objects/s recovering 2024-02-12T16:54:47.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:47.451915+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:48.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:46.390699+0000 mgr.y (mgr.24481) 200 : cluster [DBG] pgmap v217: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 41631/228813 objects misplaced (18.194%); 11 MiB/s, 196 objects/s recovering 2024-02-12T16:54:48.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:47.451915+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:48.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:46.390699+0000 mgr.y (mgr.24481) 200 : cluster [DBG] pgmap v217: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 41631/228813 objects misplaced (18.194%); 11 MiB/s, 196 objects/s recovering 2024-02-12T16:54:48.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:47.451915+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:48.347 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:48.347 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:49.050 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:54:49.424 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:48.392830+0000 mgr.y (mgr.24481) 201 : cluster [DBG] pgmap v218: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 603 B/s rd, 0 B/s wr, 756 op/s; 36962/220131 objects misplaced (16.791%); 39 MiB/s, 633 objects/s recovering 2024-02-12T16:54:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:48.392830+0000 mgr.y (mgr.24481) 201 : cluster [DBG] pgmap v218: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 603 B/s rd, 0 B/s wr, 756 op/s; 36962/220131 objects misplaced (16.791%); 39 MiB/s, 633 objects/s recovering 2024-02-12T16:54:50.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:48.392830+0000 mgr.y (mgr.24481) 201 : cluster [DBG] pgmap v218: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 603 B/s rd, 0 B/s wr, 756 op/s; 36962/220131 objects misplaced (16.791%); 39 MiB/s, 633 objects/s recovering 2024-02-12T16:54:50.200 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:50.203 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:50.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:50.308 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:50.367 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:50.419 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:50.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:50.527 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:50.612 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:50.660 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:50.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:50.767 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:54:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:50.838 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:54:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:50.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:54:50.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:51.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:51.210 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:51.260 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:54:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:50] "GET /metrics HTTP/1.1" 200 175581 "" "Prometheus/2.33.4" 2024-02-12T16:54:51.265 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:51.438 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:51.438 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:51.742 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:51.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:51.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:51.860 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:54:51.860 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T16:54:51.898 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:52.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:50.393492+0000 mgr.y (mgr.24481) 202 : cluster [DBG] pgmap v219: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 759 op/s; 35907/217278 objects misplaced (16.526%); 38 MiB/s, 612 objects/s recovering 2024-02-12T16:54:52.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:50.393492+0000 mgr.y (mgr.24481) 202 : cluster [DBG] pgmap v219: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 759 op/s; 35907/217278 objects misplaced (16.526%); 38 MiB/s, 612 objects/s recovering 2024-02-12T16:54:52.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:54:52.197 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:50.393492+0000 mgr.y (mgr.24481) 202 : cluster [DBG] pgmap v219: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 759 op/s; 35907/217278 objects misplaced (16.526%); 38 MiB/s, 612 objects/s recovering 2024-02-12T16:54:52.219 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:54:52.256 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:52.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:52.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:52.408 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:52.445 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:52.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:52.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:52.502 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:52.550 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:52.578 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:52.592 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:52.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:52.737 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:52.776 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:54:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:54:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:54:52.777 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:53.812 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:52.394729+0000 mgr.y (mgr.24481) 203 : cluster [DBG] pgmap v220: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 585 B/s rd, 0 B/s wr, 440 op/s; 35907/217278 objects misplaced (16.526%); 27 MiB/s, 430 objects/s recovering 2024-02-12T16:54:53.812 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:52.805310+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.33:0/924137794' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:53.812 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:52.806121+0000 mon.a (mon.0) 859 : audit [INF] from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:53.827 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:54:53.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:52.394729+0000 mgr.y (mgr.24481) 203 : cluster [DBG] pgmap v220: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 585 B/s rd, 0 B/s wr, 440 op/s; 35907/217278 objects misplaced (16.526%); 27 MiB/s, 430 objects/s recovering 2024-02-12T16:54:53.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:52.805310+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.33:0/924137794' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:53.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:52.806121+0000 mon.a (mon.0) 859 : audit [INF] from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:53.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:54:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:54:54.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:52.394729+0000 mgr.y (mgr.24481) 203 : cluster [DBG] pgmap v220: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 585 B/s rd, 0 B/s wr, 440 op/s; 35907/217278 objects misplaced (16.526%); 27 MiB/s, 430 objects/s recovering 2024-02-12T16:54:54.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:52.805310+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.33:0/924137794' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:54.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:52.806121+0000 mon.a (mon.0) 859 : audit [INF] from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:54:55.036 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:55.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:55.105 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:53.571960+0000 mgr.y (mgr.24481) 204 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:55.105 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:53.778266+0000 mon.a (mon.0) 860 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:55.105 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:53.795757+0000 mon.a (mon.0) 861 : audit [INF] from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:54:55.105 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:53.795818+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e142: 8 total, 8 up, 7 in 2024-02-12T16:54:55.105 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:54.224847+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:54:55.142 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:55.195 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:55.248 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:53.571960+0000 mgr.y (mgr.24481) 204 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:55.248 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:53.778266+0000 mon.a (mon.0) 860 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:55.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:53.795757+0000 mon.a (mon.0) 861 : audit [INF] from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:54:55.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:53.795818+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e142: 8 total, 8 up, 7 in 2024-02-12T16:54:55.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:54.224847+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:54:55.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:53.571960+0000 mgr.y (mgr.24481) 204 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:54:55.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:53.778266+0000 mon.a (mon.0) 860 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:54:55.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:53.795757+0000 mon.a (mon.0) 861 : audit [INF] from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:54:55.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:53.795818+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e142: 8 total, 8 up, 7 in 2024-02-12T16:54:55.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:54.224847+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:54:55.258 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:55.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:55.737 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:55.807 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:55.839 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T16:54:56.196 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:56.202 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:54.395731+0000 mgr.y (mgr.24481) 205 : cluster [DBG] pgmap v222: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 859 op/s; 30353/206478 objects misplaced (14.700%); 55 MiB/s, 874 objects/s recovering 2024-02-12T16:54:56.202 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:54.811414+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:56.202 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:55.243505+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.33:0/2350195005' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:54:56.202 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:55.376008+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:54:56.202 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:55.388828+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:54:56.202 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:55.433563+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:56.202 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:54.395731+0000 mgr.y (mgr.24481) 205 : cluster [DBG] pgmap v222: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 859 op/s; 30353/206478 objects misplaced (14.700%); 55 MiB/s, 874 objects/s recovering 2024-02-12T16:54:56.203 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:54.811414+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:56.203 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:55.243505+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.33:0/2350195005' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:54:56.203 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:55.376008+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:54:56.203 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:55.388828+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:54:56.203 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:55.433563+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:56.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:56.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:54.395731+0000 mgr.y (mgr.24481) 205 : cluster [DBG] pgmap v222: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 859 op/s; 30353/206478 objects misplaced (14.700%); 55 MiB/s, 874 objects/s recovering 2024-02-12T16:54:56.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:54.811414+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:56.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:55.243505+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.33:0/2350195005' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:54:56.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:55.376008+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:54:56.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:55.388828+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:54:56.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:55.433563+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:56.304 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:54:56.360 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:56.532 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:54:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:56.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:54:56.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:54:56.692 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:56.813 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:54:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:56.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:54:56.860 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:54:56.861 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:54:56.920 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:54:56.987 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:55.827211+0000 mon.a (mon.0) 866 : audit [INF] from='client.? v1:172.21.15.33:0/2350195005' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:54:56.987 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:55.827277+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e143: 8 total, 8 up, 7 in 2024-02-12T16:54:56.987 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:55.827211+0000 mon.a (mon.0) 866 : audit [INF] from='client.? v1:172.21.15.33:0/2350195005' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:54:56.987 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:55.827277+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e143: 8 total, 8 up, 7 in 2024-02-12T16:54:56.991 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:57.059 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:54:57.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:55.827211+0000 mon.a (mon.0) 866 : audit [INF] from='client.? v1:172.21.15.33:0/2350195005' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:54:57.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:55.827277+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e143: 8 total, 8 up, 7 in 2024-02-12T16:54:57.656 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-12T16:54:57.660 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:54:57.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:56.396108+0000 mgr.y (mgr.24481) 206 : cluster [DBG] pgmap v224: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 569 op/s; 30353/206478 objects misplaced (14.700%); 35 MiB/s, 553 objects/s recovering 2024-02-12T16:54:57.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:57.619841+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:57.887 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:54:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:57.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:54:57.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:56.396108+0000 mgr.y (mgr.24481) 206 : cluster [DBG] pgmap v224: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 569 op/s; 30353/206478 objects misplaced (14.700%); 35 MiB/s, 553 objects/s recovering 2024-02-12T16:54:57.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:57.619841+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:58.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:54:58.001 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:54:58.027 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:58.045 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:54:58.072 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:54:58.093 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:54:58.128 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:58.128 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:54:58.171 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:54:58.201 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:54:58.214 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:54:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:58.219 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:56.396108+0000 mgr.y (mgr.24481) 206 : cluster [DBG] pgmap v224: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 569 op/s; 30353/206478 objects misplaced (14.700%); 35 MiB/s, 553 objects/s recovering 2024-02-12T16:54:58.219 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:57.619841+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:54:58.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:54:58.238 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:54:58.285 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:54:58.308 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:54:58.308 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:54:58.308 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:54:58.308 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:54:58.308 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:54:58.308 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:54:58.308 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:54:58.308 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:54:58.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:58.403 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:59.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:54:57.950071+0000 mon.a (mon.0) 869 : audit [DBG] from='client.? v1:172.21.15.33:0/1755243766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:59.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:54:57.950071+0000 mon.a (mon.0) 869 : audit [DBG] from='client.? v1:172.21.15.33:0/1755243766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:59.070 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:59.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:59.178 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:54:57.950071+0000 mon.a (mon.0) 869 : audit [DBG] from='client.? v1:172.21.15.33:0/1755243766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:54:59.264 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:59.577 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:59.636 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:59.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:59.750 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:54:59.857 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:54:59.893 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:54:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:54:58.396893+0000 mgr.y (mgr.24481) 207 : cluster [DBG] pgmap v225: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 26184/198456 objects misplaced (13.194%); 54 MiB/s, 863 objects/s recovering 2024-02-12T16:54:59.933 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:54:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:54:59.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:8462 7.5 7:a12ef8ff:::benchmark_data_smithi033_81060_object8461:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e102)", 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:52:36.015171+0000", 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 143.94653392999999, 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034273441000000002, 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:59.967 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 8462 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015171+0000", 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015171+0000", 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015169+0000", 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1118999999999995e-05 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015230+0000", 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5852999999999999e-05 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015245+0000", 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.137e-05 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015257+0000", 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9304000000000002e-05 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015296+0000", 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012370999999999999 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015420+0000", 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015557200000000001 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.015575+0000", 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0032099350000000001 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.018785+0000", 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037864349999999999 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.022572+0000", 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.026814353999999999 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.049386+0000", 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4649e-05 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.049421+0000", 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3269e-05 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:52:36.049444+0000", 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034273441000000002 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:51757 7.5 7:adf333aa:::benchmark_data_smithi033_81060_object51756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:01.835079+0000", 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.126625381, 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034272945999999999, 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:59.973 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51757 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.835079+0000", 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.835079+0000", 2024-02-12T16:54:59.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.835078+0000", 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0392e-05 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.835138+0000", 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3981999999999999e-05 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.835152+0000", 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.917e-06 2024-02-12T16:54:59.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.835159+0000", 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019393629999999999 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.837098+0000", 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011917300000000001 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.837218+0000", 2024-02-12T16:54:59.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156978 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.837374+0000", 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012509750000000001 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.838625+0000", 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013367996 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.851993+0000", 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017255124 2024-02-12T16:54:59.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.869249+0000", 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8606000000000001e-05 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.869307+0000", 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4962000000000001e-05 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:01.869352+0000", 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034272945999999999 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.978 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:52165 7.5 7:a87ced8b:::benchmark_data_smithi033_81060_object52164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:02.180637+0000", 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.781067549, 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040851483000000001, 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52165 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.979 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180637+0000", 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180637+0000", 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180635+0000", 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1456000000000002e-05 2024-02-12T16:54:59.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180696+0000", 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4055e-05 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180710+0000", 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3990000000000001e-06 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180716+0000", 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3244999999999997e-05 2024-02-12T16:54:59.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180760+0000", 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011835700000000001 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180878+0000", 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014653499999999999 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.181025+0000", 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001759942 2024-02-12T16:54:59.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.182785+0000", 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024822687 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.207607+0000", 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013803553 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.221411+0000", 2024-02-12T16:54:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0213999999999998e-05 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.221451+0000", 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7537000000000003e-05 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.221489+0000", 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040851483000000001 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:52933 7.5 7:a3163bed:::benchmark_data_smithi033_81060_object52932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:02.776319+0000", 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.185385685, 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045632915000000003, 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52933 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776319+0000", 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776319+0000", 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776317+0000", 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2389000000000003e-05 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776369+0000", 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2075e-05 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776381+0000", 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9429999999999999e-06 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776386+0000", 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3195999999999999e-05 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776419+0000", 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9982999999999995e-05 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776509+0000", 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000101128 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776611+0000", 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015528060000000001 2024-02-12T16:54:59.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.778163+0000", 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012289829999999999 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779392+0000", 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042494756000000002 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.821887+0000", 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5698999999999998e-05 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.821923+0000", 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9062000000000001e-05 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.821952+0000", 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045632915000000003 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:52947 7.5 7:aefd2641:::benchmark_data_smithi033_81060_object52946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:02.779519+0000", 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.182185799, 2024-02-12T16:54:59.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043052356, 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52947 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779519+0000", 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779519+0000", 2024-02-12T16:54:59.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779517+0000", 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0080000000000001e-05 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779577+0000", 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1793999999999999e-05 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:59.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779589+0000", 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5640000000000002e-06 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779595+0000", 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5726000000000003e-05 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779631+0000", 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2948000000000006e-05 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779714+0000", 2024-02-12T16:54:59.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9182e-05 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779813+0000", 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013416700000000001 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.781155+0000", 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022305290000000002 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.783386+0000", 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.039099086999999998 2024-02-12T16:54:59.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.822485+0000", 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8486000000000003e-05 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.822533+0000", 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8021000000000001e-05 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.822571+0000", 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043052356 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.995 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:80708 7.5 7:a793f08c:::benchmark_data_smithi033_81060_object80707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:29.050600+0000", 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.911104527000006, 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057351045000000003, 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 80708 2024-02-12T16:54:59.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050600+0000", 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050600+0000", 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050598+0000", 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3399000000000002e-05 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050651+0000", 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2693e-05 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050664+0000", 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8950000000000001e-06 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050671+0000", 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6180999999999997e-05 2024-02-12T16:54:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050717+0000", 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0001003 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050817+0000", 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112668 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050930+0000", 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021345280000000001 2024-02-12T16:54:59.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053065+0000", 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024767410000000002 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.055541+0000", 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052323124999999998 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107864+0000", 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3448000000000001e-05 2024-02-12T16:55:00.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107918+0000", 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3254999999999999e-05 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107951+0000", 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057351045000000003 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:80710 7.5 7:a5b76bb0:::benchmark_data_smithi033_81060_object80709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:00.001 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:29.053581+0000", 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.908123302000007, 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062171000999999997, 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 80710 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053581+0000", 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053581+0000", 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053580+0000", 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9789000000000001e-05 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053629+0000", 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6770000000000001e-06 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053638+0000", 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5739999999999997e-06 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053643+0000", 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3254999999999998e-05 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053686+0000", 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013029299999999999 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053816+0000", 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018151500000000001 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053998+0000", 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016970679999999999 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.055695+0000", 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053556299999999997 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.056230+0000", 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059463378999999997 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115694+0000", 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5593999999999998e-05 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115729+0000", 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2915999999999999e-05 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115752+0000", 2024-02-12T16:55:00.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062171000999999997 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:85257 7.5 7:a5a71cf9:::benchmark_data_smithi033_81060_object85256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e123)", 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:33.930540+0000", 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.0311643, 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034374290000000002, 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85257 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.007 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930540+0000", 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930540+0000", 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930538+0000", 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1198999999999995e-05 2024-02-12T16:55:00.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930599+0000", 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9100000000000005e-06 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930607+0000", 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8700000000000002e-06 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930611+0000", 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2428000000000001e-05 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930654+0000", 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4400000000000005e-05 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930738+0000", 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000101783 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.930840+0000", 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022056699999999999 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.933045+0000", 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047140180000000004 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.937759+0000", 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.027094186999999999 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.964854+0000", 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5828000000000003e-05 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.964889+0000", 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5094999999999999e-05 2024-02-12T16:55:00.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:33.964915+0000", 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034374290000000002 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:98227 7.5 7:a781f56f:::benchmark_data_smithi033_81060_object98226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:48.613807+0000", 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.347897387000003, 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033487647000000002, 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.012 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 98227 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.613807+0000", 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.613807+0000", 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999989 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.613796+0000", 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3942000000000005e-05 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.613860+0000", 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9533e-05 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.613890+0000", 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3193000000000001e-05 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.613903+0000", 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012439499999999999 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.614027+0000", 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021753000000000001 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.614245+0000", 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015536 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.614400+0000", 2024-02-12T16:55:00.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002135592 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.616536+0000", 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082183640000000006 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.624754+0000", 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022485753000000001 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.647240+0000", 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2502999999999997e-05 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.647272+0000", 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2651999999999999e-05 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:48.647295+0000", 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033487647000000002 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.017 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:99072 7.5 7:a97cf8b5:::benchmark_data_smithi033_81060_object99071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:49.556027+0000", 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.405677796999996, 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038903828000000001, 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 99072 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556027+0000", 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556027+0000", 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556024+0000", 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5201999999999999e-05 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556069+0000", 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9437999999999999e-05 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556088+0000", 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1380000000000007e-06 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556096+0000", 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6246999999999999e-05 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556153+0000", 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138104 2024-02-12T16:55:00.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556291+0000", 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000154594 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556445+0000", 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029249580000000001 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.559370+0000", 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01541178 2024-02-12T16:55:00.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.574782+0000", 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020090665000000001 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.594873+0000", 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4709000000000003e-05 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.594907+0000", 2024-02-12T16:55:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3195999999999999e-05 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.594931+0000", 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038903828000000001 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:103099 7.5 7:a0897817:::benchmark_data_smithi033_81060_object103098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:54.386758+0000", 2024-02-12T16:55:00.023 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.574946972999996, 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035363839000000001, 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 103099 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386758+0000", 2024-02-12T16:55:00.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386758+0000", 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386755+0000", 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9220000000000004e-05 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386834+0000", 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5722999999999998e-05 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386850+0000", 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0270000000000001e-06 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386856+0000", 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2657999999999997e-05 2024-02-12T16:55:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386898+0000", 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000139603 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.387038+0000", 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016862800000000001 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.387207+0000", 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042973589999999997 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.391504+0000", 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010051008 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.401555+0000", 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020485152999999999 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.422040+0000", 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.684e-05 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.422077+0000", 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4579999999999997e-05 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.422121+0000", 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035363839000000001 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:103103 7.5 7:aa3fd586:::benchmark_data_smithi033_81060_object103102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:54.386869+0000", 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.574835453000006, 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035351646, 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 103103 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386869+0000", 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386869+0000", 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386868+0000", 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4792000000000002e-05 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386903+0000", 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0950000000000001e-06 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386909+0000", 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9269999999999998e-06 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.386913+0000", 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002647052 2024-02-12T16:55:00.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.389560+0000", 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113051 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.389673+0000", 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018144 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.389855+0000", 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001800867 2024-02-12T16:55:00.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.391655+0000", 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016395989999999999 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.408051+0000", 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014103361999999999 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.422155+0000", 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3377999999999997e-05 2024-02-12T16:55:00.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.422198+0000", 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2588000000000001e-05 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:54.422221+0000", 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035351646 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:103903 7.5 7:a7f7e120:::benchmark_data_smithi033_81060_object103902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-12T16:55:00.035 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:55.427673+0000", 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.534031673000001, 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033022337999999998, 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 103903 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.427673+0000", 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.427673+0000", 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999933 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.427666+0000", 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8291000000000003e-05 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.427705+0000", 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6549e-05 2024-02-12T16:55:00.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.427721+0000", 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1879999999999996e-06 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.427728+0000", 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5133000000000003e-05 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.427773+0000", 2024-02-12T16:55:00.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013077799999999999 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.427904+0000", 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000182707 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.428087+0000", 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028051259999999998 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.430892+0000", 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056459350000000004 2024-02-12T16:55:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.436538+0000", 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024060345 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.460598+0000", 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4228999999999999e-05 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.460653+0000", 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2713999999999998e-05 2024-02-12T16:55:00.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:55.460695+0000", 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033022337999999998 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:104495 7.5 7:a1d02bf9:::benchmark_data_smithi033_81060_object104494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:56.075999+0000", 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.885705891000001, 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042203978000000003, 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.041 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 104495 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.075999+0000", 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.075999+0000", 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999919 2024-02-12T16:55:00.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.075991+0000", 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3384000000000003e-05 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076044+0000", 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2148e-05 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076056+0000", 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8499999999999999e-06 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076062+0000", 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1137999999999997e-05 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076113+0000", 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014489 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076258+0000", 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017179599999999999 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076430+0000", 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003172008 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.079602+0000", 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067976299999999998 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.086399+0000", 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031726231000000001 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.118126+0000", 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7032000000000001e-05 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.118163+0000", 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0058999999999997e-05 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.118203+0000", 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042203978000000003 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:107212 7.5 7:ae1c802b:::benchmark_data_smithi033_81060_object107211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:59.278400+0000", 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.683304397999997, 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043589808000000001, 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 107212 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278400+0000", 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278400+0000", 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:00.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278398+0000", 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1880999999999998e-05 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278480+0000", 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0026000000000001e-05 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278500+0000", 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6369999999999999e-06 2024-02-12T16:55:00.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278508+0000", 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4691999999999998e-05 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278552+0000", 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013292800000000001 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278685+0000", 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019186600000000001 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278877+0000", 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0046846479999999996 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.283562+0000", 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012374098 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.295936+0000", 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025991435 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.321927+0000", 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6848999999999999e-05 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.321964+0000", 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5956e-05 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.321990+0000", 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043589808000000001 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:107213 7.5 7:ab05a132:::benchmark_data_smithi033_81060_object107212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:59.278791+0000", 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.682913831999997, 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043341330999999997, 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 107213 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278791+0000", 2024-02-12T16:55:00.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278791+0000", 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278789+0000", 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2889000000000002e-05 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278832+0000", 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-02-12T16:55:00.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278843+0000", 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6170000000000003e-06 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278850+0000", 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025485339999999999 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.281399+0000", 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000127766 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.281526+0000", 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016311200000000001 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.281689+0000", 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020537590000000001 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.283743+0000", 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012221242 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.295964+0000", 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.026105811999999999 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.322070+0000", 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7021999999999999e-05 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.322107+0000", 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4808999999999999e-05 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.322132+0000", 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043341330999999997 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:125758 7.5 7:a349291b:::benchmark_data_smithi033_81060_object15306:head [delete] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:54:19.491237+0000", 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.470467530999997, 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035382597000000002, 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 125758 2024-02-12T16:55:00.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491237+0000", 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491237+0000", 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491235+0000", 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5133000000000001e-05 2024-02-12T16:55:00.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491250+0000", 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2163e-05 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491262+0000", 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5019999999999998e-06 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491272+0000", 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4200000000000003e-05 2024-02-12T16:55:00.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491326+0000", 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6761000000000006e-05 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491403+0000", 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0767999999999996e-05 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.491493+0000", 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020907790000000001 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.493584+0000", 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065873699999999995 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.494243+0000", 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032281098000000001 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.526524+0000", 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5988000000000001e-05 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.526570+0000", 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9617000000000003e-05 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:19.526620+0000", 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035382597000000002 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:138857 7.5 7:aeb8dc79:::benchmark_data_smithi033_81060_object28405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:54:33.887585+0000", 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.074119355000001, 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034881357000000002, 2024-02-12T16:55:00.063 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 138857 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887585+0000", 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887585+0000", 2024-02-12T16:55:00.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887583+0000", 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.759e-06 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887590+0000", 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9029999999999999e-06 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887600+0000", 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7300000000000002e-06 2024-02-12T16:55:00.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887606+0000", 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6515999999999997e-05 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887682+0000", 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1940999999999994e-05 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887764+0000", 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5588999999999997e-05 2024-02-12T16:55:00.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.887850+0000", 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020394440000000001 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.889889+0000", 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069761679999999996 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.896865+0000", 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025511393 2024-02-12T16:55:00.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.922377+0000", 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0133000000000003e-05 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.922427+0000", 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9727000000000001e-05 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:33.922467+0000", 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034881357000000002 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.068 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:139529 7.5 7:ab30e6e7:::benchmark_data_smithi033_81060_object29077:head [delete] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:54:34.617003+0000", 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.344701564000001, 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033272871000000002, 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 139529 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.617003+0000", 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.617003+0000", 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.617003+0000", 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.187e-06 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.617005+0000", 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.194e-06 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.617009+0000", 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5949999999999999e-06 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.617013+0000", 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047717289999999997 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.621784+0000", 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012981599999999999 2024-02-12T16:55:00.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.621914+0000", 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014027399999999999 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.622054+0000", 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002295732 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.624350+0000", 2024-02-12T16:55:00.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062633369999999999 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.630613+0000", 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019568985000000001 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.650182+0000", 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8906999999999998e-05 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.650231+0000", 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4535e-05 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:34.650276+0000", 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033272871000000002 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:143742 7.5 7:afb72632:::benchmark_data_smithi033_81060_object33290:head [delete] snapc 0=[] ondisk+write+known_if_redirected e136)", 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:54:39.605709+0000", 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.355996050999998, 2024-02-12T16:55:00.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037265803, 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 143742 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605709+0000", 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:55:00.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605709+0000", 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605707+0000", 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7029999999999995e-06 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605716+0000", 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5427999999999999e-05 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605731+0000", 2024-02-12T16:55:00.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1840000000000004e-06 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605737+0000", 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0085558779999999994 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.614293+0000", 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000139746 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.614433+0000", 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000147718 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.614581+0000", 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0032810280000000001 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.617862+0000", 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022027228999999999 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.639889+0000", 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030113649999999998 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.642900+0000", 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5810999999999999e-05 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.642946+0000", 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8206000000000001e-05 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.642974+0000", 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037265803 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:55:00.079 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:55:00.080 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:55:00.080 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:00.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:54:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:54:58.396893+0000 mgr.y (mgr.24481) 207 : cluster [DBG] pgmap v225: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 26184/198456 objects misplaced (13.194%); 54 MiB/s, 863 objects/s recovering 2024-02-12T16:55:00.138 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:54:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:54:58.396893+0000 mgr.y (mgr.24481) 207 : cluster [DBG] pgmap v225: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 26184/198456 objects misplaced (13.194%); 54 MiB/s, 863 objects/s recovering 2024-02-12T16:55:00.144 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:00.524 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:00.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:00.648 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:00.737 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:01.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:55:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:00] "GET /metrics HTTP/1.1" 200 175581 "" "Prometheus/2.33.4" 2024-02-12T16:55:01.427 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:01.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:01.540 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:55:01.595 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:55:02.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:00.226230+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:02.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:00.226274+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:02.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:00.403831+0000 mgr.y (mgr.24481) 208 : cluster [DBG] pgmap v226: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 892 op/s; 25184/195855 objects misplaced (12.858%); 59 MiB/s, 942 objects/s recovering 2024-02-12T16:55:02.004 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:00.906276+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e144: 8 total, 8 up, 7 in 2024-02-12T16:55:02.004 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:00.950298+0000 mon.b (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.33:0/1924903028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:02.004 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:00.950876+0000 mon.a (mon.0) 871 : audit [INF] from='client.24937 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:02.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:00.226230+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:02.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:00.226274+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:02.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:00.403831+0000 mgr.y (mgr.24481) 208 : cluster [DBG] pgmap v226: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 892 op/s; 25184/195855 objects misplaced (12.858%); 59 MiB/s, 942 objects/s recovering 2024-02-12T16:55:02.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:00.906276+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e144: 8 total, 8 up, 7 in 2024-02-12T16:55:02.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:00.950298+0000 mon.b (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.33:0/1924903028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:02.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:00.950876+0000 mon.a (mon.0) 871 : audit [INF] from='client.24937 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:00.226230+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:00.226274+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:00.403831+0000 mgr.y (mgr.24481) 208 : cluster [DBG] pgmap v226: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 892 op/s; 25184/195855 objects misplaced (12.858%); 59 MiB/s, 942 objects/s recovering 2024-02-12T16:55:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:00.906276+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e144: 8 total, 8 up, 7 in 2024-02-12T16:55:02.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:00.950298+0000 mon.b (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.33:0/1924903028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:02.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:00.950876+0000 mon.a (mon.0) 871 : audit [INF] from='client.24937 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:02.337 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:55:02.365 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:55:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:02.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:55:02.470 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:02.577 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:02.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:55:02.884 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:02.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:02.991 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:55:03.001 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.1 to 1 (10000) 2024-02-12T16:55:03.074 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:03.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:01.911814+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:03.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:01.911851+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:01.913920+0000 mon.a (mon.0) 872 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:01.946049+0000 mon.a (mon.0) 873 : audit [INF] from='client.24937 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:55:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:01.946180+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e145: 8 total, 8 up, 7 in 2024-02-12T16:55:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:02.404282+0000 mgr.y (mgr.24481) 209 : cluster [DBG] pgmap v229: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 539 op/s; 25184/195855 objects misplaced (12.858%); 35 MiB/s, 567 objects/s recovering 2024-02-12T16:55:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:02.475416+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:02.911811+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.33:0/2262296347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:55:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:02.913420+0000 mon.a (mon.0) 876 : audit [INF] from='client.34904 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:55:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:01.911814+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:01.911851+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:01.913920+0000 mon.a (mon.0) 872 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:01.946049+0000 mon.a (mon.0) 873 : audit [INF] from='client.24937 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:01.946180+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e145: 8 total, 8 up, 7 in 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:02.404282+0000 mgr.y (mgr.24481) 209 : cluster [DBG] pgmap v229: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 539 op/s; 25184/195855 objects misplaced (12.858%); 35 MiB/s, 567 objects/s recovering 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:02.475416+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:02.911811+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.33:0/2262296347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:02.913420+0000 mon.a (mon.0) 876 : audit [INF] from='client.34904 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:01.911814+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:01.911851+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:01.913920+0000 mon.a (mon.0) 872 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:01.946049+0000 mon.a (mon.0) 873 : audit [INF] from='client.24937 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:55:03.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:01.946180+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e145: 8 total, 8 up, 7 in 2024-02-12T16:55:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:02.404282+0000 mgr.y (mgr.24481) 209 : cluster [DBG] pgmap v229: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 539 op/s; 25184/195855 objects misplaced (12.858%); 35 MiB/s, 567 objects/s recovering 2024-02-12T16:55:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:02.475416+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:02.911811+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.33:0/2262296347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:55:03.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:02.913420+0000 mon.a (mon.0) 876 : audit [INF] from='client.34904 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:55:03.345 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:55:03.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:03.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:55:03.376 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:55:03.451 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:03.491 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:03.518 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:03.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:03.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:03.712 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:03.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:55:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:55:03.845 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:03.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:03.856 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:03.896 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T16:55:03.954 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:03.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:02.983176+0000 mon.a (mon.0) 877 : audit [INF] from='client.34904 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:55:03.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:02.983246+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e146: 8 total, 8 up, 7 in 2024-02-12T16:55:03.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:03.588070+0000 mgr.y (mgr.24481) 210 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:03.986 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:02.983176+0000 mon.a (mon.0) 877 : audit [INF] from='client.34904 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:55:03.986 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:02.983246+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e146: 8 total, 8 up, 7 in 2024-02-12T16:55:03.986 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:03.588070+0000 mgr.y (mgr.24481) 210 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:04.063 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:04.186 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:04.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:02.983176+0000 mon.a (mon.0) 877 : audit [INF] from='client.34904 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:55:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:02.983246+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e146: 8 total, 8 up, 7 in 2024-02-12T16:55:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:03.588070+0000 mgr.y (mgr.24481) 210 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:04.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:04.328 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:04.355 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:04.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:04.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:55:04.495 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:04.524 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:55:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:04.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:55:04.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:04.674 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:04.703 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:04.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:04.807 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:04.838 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:04.900 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:05.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:55:05.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:04.405104+0000 mgr.y (mgr.24481) 211 : cluster [DBG] pgmap v231: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 748 op/s; 20053/184983 objects misplaced (10.840%); 46 MiB/s, 734 objects/s recovering 2024-02-12T16:55:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:04.405104+0000 mgr.y (mgr.24481) 211 : cluster [DBG] pgmap v231: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 748 op/s; 20053/184983 objects misplaced (10.840%); 46 MiB/s, 734 objects/s recovering 2024-02-12T16:55:05.476 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:05.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:05.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:04.405104+0000 mgr.y (mgr.24481) 211 : cluster [DBG] pgmap v231: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 748 op/s; 20053/184983 objects misplaced (10.840%); 46 MiB/s, 734 objects/s recovering 2024-02-12T16:55:05.585 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:05.648 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:06.672 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:55:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:55:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161747 7.6 7:64497997:::benchmark_data_smithi033_81060_object51295:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.646209+0000", 2024-02-12T16:55:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.020896385, 2024-02-12T16:55:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020927247, 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161747 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.646209+0000", 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.646209+0000", 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.646207+0000", 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1500000000000002e-06 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.646214+0000", 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0650000000000005e-06 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.646223+0000", 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1159999999999999e-06 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.646231+0000", 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001386204 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.647617+0000", 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000113309 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.647731+0000", 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014637299999999999 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,7", 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.647877+0000", 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003088089 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.650965+0000", 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00311327 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.654078+0000", 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078694490000000006 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161753 7.f 7:f31489b8:::benchmark_data_smithi033_81060_object51301:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.650788+0000", 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.016317497, 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016449135, 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161753 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.650788+0000", 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.650788+0000", 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:55:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.650785+0000", 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0399e-05 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.650795+0000", 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2741e-05 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.650808+0000", 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.204e-06 2024-02-12T16:55:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.650815+0000", 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3498000000000002e-05 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.650869+0000", 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000140185 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.651009+0000", 2024-02-12T16:55:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015551599999999999 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,7", 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.651165+0000", 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0032658169999999999 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.654430+0000", 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7748999999999999e-05 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.654448+0000", 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035446179999999998 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.657993+0000", 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0072049719999999996 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161756 7.9 7:9b8c3f9c:::benchmark_data_smithi033_81060_object51304:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.656475+0000", 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.010630502, 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010859841, 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161756 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.656475+0000", 2024-02-12T16:55:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.656475+0000", 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.656472+0000", 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7299999999999994e-06 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.656481+0000", 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2945999999999999e-05 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.656494+0000", 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7109999999999995e-06 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.656502+0000", 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093377180000000001 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665839+0000", 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105245 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665945+0000", 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012395700000000001 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666069+0000", 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095937460000000002 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.14838.0:161747 7.6 e146/138)", 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.659428+0000", 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.007677237, 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079895120000000007, 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659428+0000", 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659428+0000", 2024-02-12T16:55:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659427+0000", 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2810000000000003e-06 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659434+0000", 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5174000000000001e-05 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659449+0000", 2024-02-12T16:55:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1859999999999998e-06 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659458+0000", 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0148999999999999e-05 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.14838.0:161747 7.6 e146/138)", 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.659480+0000", 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0076258100000000002, 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079870659999999993, 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659480+0000", 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659480+0000", 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659478+0000", 2024-02-12T16:55:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2220000000000002e-06 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659487+0000", 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2840000000000001e-06 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659494+0000", 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.746e-06 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659499+0000", 2024-02-12T16:55:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9236e-05 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPushReply(7.6 146/138 [PushReplyOp(7:6b6f35a3:::benchmark_data_smithi033_81060_object107726:head)])", 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.659527+0000", 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0075784210000000001, 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079903700000000001, 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659527+0000", 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659527+0000", 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659527+0000", 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7570000000000002e-06 2024-02-12T16:55:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659530+0000", 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7749999999999998e-06 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659536+0000", 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3290000000000004e-06 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659540+0000", 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3475000000000001e-05 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161758 7.4 7:2f447dcf:::benchmark_data_smithi033_81060_object51306:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.659599+0000", 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0075066719999999998, 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079716479999999996, 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161758 2024-02-12T16:55:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659599+0000", 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659599+0000", 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659597+0000", 2024-02-12T16:55:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3499999999999999e-06 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659604+0000", 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0025e-05 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659614+0000", 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6509999999999998e-06 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659620+0000", 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9234000000000006e-05 2024-02-12T16:55:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659719+0000", 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013311700000000001 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659852+0000", 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000141884 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659994+0000", 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057147109999999999 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665709+0000", 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4104000000000006e-05 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665803+0000", 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0062042859999999998 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:161757 7.d e146/93 7:b116d924:::benchmark_data_smithi033_81060_object51305:head v 146'10258, mlcod=146'10256)", 2024-02-12T16:55:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.659644+0000", 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.007461489, 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080177359999999993, 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659644+0000", 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659644+0000", 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659643+0000", 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7050000000000002e-06 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659650+0000", 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4969999999999998e-06 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659656+0000", 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0840000000000002e-06 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.659660+0000", 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041600530000000004 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.663820+0000", 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0231999999999999e-05 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.663841+0000", 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041968470000000001 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161759 7.4 7:2c7a9eb9:::benchmark_data_smithi033_81060_object51307:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.661999+0000", 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0051060669999999997, 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057267990000000003, 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161759 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.661999+0000", 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.661999+0000", 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.661996+0000", 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1606e-05 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662008+0000", 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.483e-05 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662023+0000", 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7919999999999993e-06 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662031+0000", 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093015100000000005 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662961+0000", 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7812999999999993e-05 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.663059+0000", 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116521 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.663175+0000", 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025645210000000002 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665740+0000", 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000305865 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666045+0000", 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004046203 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161760 7.f 7:f9302c36:::benchmark_data_smithi033_81060_object51308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.662040+0000", 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0050653750000000004, 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057748490000000003, 2024-02-12T16:55:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161760 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662040+0000", 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662040+0000", 2024-02-12T16:55:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662039+0000", 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5030000000000002e-06 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662043+0000", 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7979999999999996e-06 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662048+0000", 2024-02-12T16:55:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5310000000000002e-06 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662051+0000", 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00052293999999999995 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662574+0000", 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7051000000000003e-05 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662671+0000", 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106146 2024-02-12T16:55:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,7", 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.662777+0000", 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026160659999999998 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665393+0000", 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00044083700000000002 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665834+0000", 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037941490000000001 2024-02-12T16:55:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161761 7.f 7:f168612b:::benchmark_data_smithi033_81060_object51309:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.665979+0000", 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0011261750000000001, 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001925204, 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161761 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665979+0000", 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665979+0000", 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665977+0000", 2024-02-12T16:55:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0949999999999998e-06 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665984+0000", 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0567000000000001e-05 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.665995+0000", 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7910000000000001e-06 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666001+0000", 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7750999999999997e-05 2024-02-12T16:55:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666048+0000", 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000107192 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666156+0000", 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015878299999999999 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,7", 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666314+0000", 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033524999999999999 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161765 7.f 7.2bc2fe7f (undecoded) ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.666065+0000", 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.001039983, 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019139129999999999, 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161765 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666065+0000", 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666065+0000", 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666064+0000", 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1120000000000004e-06 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666069+0000", 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5390000000000002e-06 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666077+0000", 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.429e-06 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666082+0000", 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6730000000000001e-05 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161766 7.4 7:2f0401f7:::benchmark_data_smithi033_81060_object51314:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.666087+0000", 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.001018742, 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019515170000000001, 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161766 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666087+0000", 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666087+0000", 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666086+0000", 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5579999999999999e-06 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666089+0000", 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5120000000000002e-06 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666093+0000", 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3380000000000001e-06 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666096+0000", 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7620999999999997e-05 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666144+0000", 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104481 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666249+0000", 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000162566 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.666411+0000", 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032452499999999999 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 13 2024-02-12T16:55:06.722 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:55:07.044 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:07.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:07.152 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:07.237 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:07.424 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:07.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:06.405910+0000 mgr.y (mgr.24481) 212 : cluster [DBG] pgmap v232: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 604 op/s; 20053/184983 objects misplaced (10.840%); 39 MiB/s, 629 objects/s recovering 2024-02-12T16:55:07.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:06.405910+0000 mgr.y (mgr.24481) 212 : cluster [DBG] pgmap v232: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 604 op/s; 20053/184983 objects misplaced (10.840%); 39 MiB/s, 629 objects/s recovering 2024-02-12T16:55:07.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:06.405910+0000 mgr.y (mgr.24481) 212 : cluster [DBG] pgmap v232: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 604 op/s; 20053/184983 objects misplaced (10.840%); 39 MiB/s, 629 objects/s recovering 2024-02-12T16:55:08.311 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:55:09.083 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:55:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:09.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:09.171 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:55:09.172 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:09.197 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:55:09.248 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:55:09.273 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:09.286 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:55:09.319 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:55:09.354 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:55:09.357 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:09.387 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:55:09.399 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:55:09.399 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:55:09.399 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:55:09.399 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:55:09.399 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:55:09.399 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:55:09.400 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:55:09.400 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:55:09.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:08.410890+0000 mgr.y (mgr.24481) 213 : cluster [DBG] pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 841 op/s; 16048/176976 objects misplaced (9.068%); 58 MiB/s, 935 objects/s recovering 2024-02-12T16:55:09.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:09.037766+0000 mon.a (mon.0) 879 : audit [DBG] from='client.? v1:172.21.15.33:0/3549400877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:09.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:08.410890+0000 mgr.y (mgr.24481) 213 : cluster [DBG] pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 841 op/s; 16048/176976 objects misplaced (9.068%); 58 MiB/s, 935 objects/s recovering 2024-02-12T16:55:09.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:09.037766+0000 mon.a (mon.0) 879 : audit [DBG] from='client.? v1:172.21.15.33:0/3549400877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:09.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:08.410890+0000 mgr.y (mgr.24481) 213 : cluster [DBG] pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 841 op/s; 16048/176976 objects misplaced (9.068%); 58 MiB/s, 935 objects/s recovering 2024-02-12T16:55:09.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:09.037766+0000 mon.a (mon.0) 879 : audit [DBG] from='client.? v1:172.21.15.33:0/3549400877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:10.134 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:10.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:10.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:55:10.283 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:10.309 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:55:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:10.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:55:10.414 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:10.455 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:10.485 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:10.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:10.591 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:55:10.629 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:10.655 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:55:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:10.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:55:10.760 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:10.796 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:10.828 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:10.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:10.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:10.977 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:11.003 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:11.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:11.029 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:55:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:10] "GET /metrics HTTP/1.1" 200 175611 "" "Prometheus/2.33.4" 2024-02-12T16:55:11.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:55:11.149 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:11.173 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:55:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:11.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:55:11.278 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:11.343 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:11.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:10.413765+0000 mgr.y (mgr.24481) 214 : cluster [DBG] pgmap v234: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s; 15069/174039 objects misplaced (8.658%); 57 MiB/s, 908 objects/s recovering 2024-02-12T16:55:11.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:11.462582+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.33:0/1656244380' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:55:11.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:11.463230+0000 mon.a (mon.0) 880 : audit [INF] from='client.24949 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:55:11.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:10.413765+0000 mgr.y (mgr.24481) 214 : cluster [DBG] pgmap v234: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s; 15069/174039 objects misplaced (8.658%); 57 MiB/s, 908 objects/s recovering 2024-02-12T16:55:11.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:11.462582+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.33:0/1656244380' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:55:11.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:11.463230+0000 mon.a (mon.0) 880 : audit [INF] from='client.24949 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:55:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:10.413765+0000 mgr.y (mgr.24481) 214 : cluster [DBG] pgmap v234: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s; 15069/174039 objects misplaced (8.658%); 57 MiB/s, 908 objects/s recovering 2024-02-12T16:55:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:11.462582+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.33:0/1656244380' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:55:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:11.463230+0000 mon.a (mon.0) 880 : audit [INF] from='client.24949 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:55:12.139 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T16:55:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:55:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:55:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:12.099801+0000 mon.a (mon.0) 881 : audit [INF] from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:55:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:12.099867+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e147: 8 total, 8 up, 7 in 2024-02-12T16:55:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:12.154233+0000 osd.2 (osd.2) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:12.154269+0000 osd.2 (osd.2) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:12.414221+0000 mgr.y (mgr.24481) 215 : cluster [DBG] pgmap v236: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 772 op/s; 15069/174039 objects misplaced (8.658%); 51 MiB/s, 814 objects/s recovering 2024-02-12T16:55:13.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:12.099801+0000 mon.a (mon.0) 881 : audit [INF] from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:55:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:12.099867+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e147: 8 total, 8 up, 7 in 2024-02-12T16:55:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:12.154233+0000 osd.2 (osd.2) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:12.154269+0000 osd.2 (osd.2) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:12.414221+0000 mgr.y (mgr.24481) 215 : cluster [DBG] pgmap v236: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 772 op/s; 15069/174039 objects misplaced (8.658%); 51 MiB/s, 814 objects/s recovering 2024-02-12T16:55:13.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:12.099801+0000 mon.a (mon.0) 881 : audit [INF] from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:55:13.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:12.099867+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e147: 8 total, 8 up, 7 in 2024-02-12T16:55:13.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:12.154233+0000 osd.2 (osd.2) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:13.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:12.154269+0000 osd.2 (osd.2) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:13.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:12.414221+0000 mgr.y (mgr.24481) 215 : cluster [DBG] pgmap v236: 148 pgs: 1 active+remapped+backfill_toofull, 139 active+clean, 8 active+remapped+backfilling; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 772 op/s; 15069/174039 objects misplaced (8.658%); 51 MiB/s, 814 objects/s recovering 2024-02-12T16:55:13.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:55:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:55:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:13.115253+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e148: 8 total, 8 up, 7 in 2024-02-12T16:55:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:13.598491+0000 mgr.y (mgr.24481) 216 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:14.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:13.115253+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e148: 8 total, 8 up, 7 in 2024-02-12T16:55:14.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:13.598491+0000 mgr.y (mgr.24481) 216 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:14.536 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:13.115253+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e148: 8 total, 8 up, 7 in 2024-02-12T16:55:14.536 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:13.598491+0000 mgr.y (mgr.24481) 216 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:15.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:14.416922+0000 mgr.y (mgr.24481) 217 : cluster [DBG] pgmap v238: 148 pgs: 139 active+clean, 9 active+remapped+backfilling; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 10073/162885 objects misplaced (6.184%); 63 MiB/s, 1.01k objects/s recovering 2024-02-12T16:55:15.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:14.416922+0000 mgr.y (mgr.24481) 217 : cluster [DBG] pgmap v238: 148 pgs: 139 active+clean, 9 active+remapped+backfilling; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 10073/162885 objects misplaced (6.184%); 63 MiB/s, 1.01k objects/s recovering 2024-02-12T16:55:15.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:14.416922+0000 mgr.y (mgr.24481) 217 : cluster [DBG] pgmap v238: 148 pgs: 139 active+clean, 9 active+remapped+backfilling; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 10073/162885 objects misplaced (6.184%); 63 MiB/s, 1.01k objects/s recovering 2024-02-12T16:55:15.544 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:15.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:15.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:55:15.679 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:15.697 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:55:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:15.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:55:15.800 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:15.828 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:15.850 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:15.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:15.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:15.982 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:16.005 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:16.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:16.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:16.137 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:16.156 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:16.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:15.117795+0000 mon.a (mon.0) 884 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:55:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:15.145517+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e149: 8 total, 8 up, 7 in 2024-02-12T16:55:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:16.085342+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e150: 8 total, 8 up, 7 in 2024-02-12T16:55:16.260 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:16.294 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:16.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:15.117795+0000 mon.a (mon.0) 884 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:55:16.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:15.145517+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e149: 8 total, 8 up, 7 in 2024-02-12T16:55:16.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:16.085342+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e150: 8 total, 8 up, 7 in 2024-02-12T16:55:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:15.117795+0000 mon.a (mon.0) 884 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:55:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:15.145517+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e149: 8 total, 8 up, 7 in 2024-02-12T16:55:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:16.085342+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e150: 8 total, 8 up, 7 in 2024-02-12T16:55:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:16.417275+0000 mgr.y (mgr.24481) 218 : cluster [DBG] pgmap v241: 148 pgs: 139 active+clean, 9 active+remapped+backfilling; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 10073/162885 objects misplaced (6.184%); 60 MiB/s, 964 objects/s recovering 2024-02-12T16:55:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:17.096457+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e151: 8 total, 8 up, 7 in 2024-02-12T16:55:17.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:16.417275+0000 mgr.y (mgr.24481) 218 : cluster [DBG] pgmap v241: 148 pgs: 139 active+clean, 9 active+remapped+backfilling; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 10073/162885 objects misplaced (6.184%); 60 MiB/s, 964 objects/s recovering 2024-02-12T16:55:17.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:17.096457+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e151: 8 total, 8 up, 7 in 2024-02-12T16:55:17.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:16.417275+0000 mgr.y (mgr.24481) 218 : cluster [DBG] pgmap v241: 148 pgs: 139 active+clean, 9 active+remapped+backfilling; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 10073/162885 objects misplaced (6.184%); 60 MiB/s, 964 objects/s recovering 2024-02-12T16:55:17.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:17.096457+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e151: 8 total, 8 up, 7 in 2024-02-12T16:55:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:17.463407+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:18.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:17.463407+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:18.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:17.463407+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:19.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:18.420314+0000 mgr.y (mgr.24481) 219 : cluster [DBG] pgmap v243: 148 pgs: 143 active+clean, 5 active+remapped+backfilling; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 965 B/s rd, 0 B/s wr, 1.01k op/s; 6796/158037 objects misplaced (4.300%); 85 MiB/s, 1.35k objects/s recovering 2024-02-12T16:55:19.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:18.420314+0000 mgr.y (mgr.24481) 219 : cluster [DBG] pgmap v243: 148 pgs: 143 active+clean, 5 active+remapped+backfilling; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 965 B/s rd, 0 B/s wr, 1.01k op/s; 6796/158037 objects misplaced (4.300%); 85 MiB/s, 1.35k objects/s recovering 2024-02-12T16:55:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:18.420314+0000 mgr.y (mgr.24481) 219 : cluster [DBG] pgmap v243: 148 pgs: 143 active+clean, 5 active+remapped+backfilling; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 965 B/s rd, 0 B/s wr, 1.01k op/s; 6796/158037 objects misplaced (4.300%); 85 MiB/s, 1.35k objects/s recovering 2024-02-12T16:55:21.281 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:55:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:20] "GET /metrics HTTP/1.1" 200 175629 "" "Prometheus/2.33.4" 2024-02-12T16:55:21.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:20.421296+0000 mgr.y (mgr.24481) 220 : cluster [DBG] pgmap v244: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 445 op/s; 5226/154884 objects misplaced (3.374%); 46 MiB/s, 741 objects/s recovering 2024-02-12T16:55:21.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:20.421296+0000 mgr.y (mgr.24481) 220 : cluster [DBG] pgmap v244: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 445 op/s; 5226/154884 objects misplaced (3.374%); 46 MiB/s, 741 objects/s recovering 2024-02-12T16:55:21.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:20.421296+0000 mgr.y (mgr.24481) 220 : cluster [DBG] pgmap v244: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 445 op/s; 5226/154884 objects misplaced (3.374%); 46 MiB/s, 741 objects/s recovering 2024-02-12T16:55:22.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T16:55:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:55:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:55:23.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:22.424350+0000 mgr.y (mgr.24481) 221 : cluster [DBG] pgmap v245: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 703 B/s rd, 0 B/s wr, 367 op/s; 5226/154884 objects misplaced (3.374%); 38 MiB/s, 611 objects/s recovering 2024-02-12T16:55:23.774 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:22.424350+0000 mgr.y (mgr.24481) 221 : cluster [DBG] pgmap v245: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 703 B/s rd, 0 B/s wr, 367 op/s; 5226/154884 objects misplaced (3.374%); 38 MiB/s, 611 objects/s recovering 2024-02-12T16:55:23.775 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:23.794 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:23.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:23.902 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:55:23.961 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:23.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:22.424350+0000 mgr.y (mgr.24481) 221 : cluster [DBG] pgmap v245: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 703 B/s rd, 0 B/s wr, 367 op/s; 5226/154884 objects misplaced (3.374%); 38 MiB/s, 611 objects/s recovering 2024-02-12T16:55:23.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:55:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:55:24.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:23.609174+0000 mgr.y (mgr.24481) 222 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:24.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:23.609174+0000 mgr.y (mgr.24481) 222 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:23.609174+0000 mgr.y (mgr.24481) 222 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:25.518 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:24.425420+0000 mgr.y (mgr.24481) 223 : cluster [DBG] pgmap v246: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 3593/140118 objects misplaced (2.564%); 42 MiB/s, 678 objects/s recovering 2024-02-12T16:55:25.518 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:24.519751+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e152: 8 total, 8 up, 7 in 2024-02-12T16:55:25.622 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:55:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:25.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:55:25.730 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:55:25.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:24.425420+0000 mgr.y (mgr.24481) 223 : cluster [DBG] pgmap v246: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 3593/140118 objects misplaced (2.564%); 42 MiB/s, 678 objects/s recovering 2024-02-12T16:55:25.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:24.519751+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e152: 8 total, 8 up, 7 in 2024-02-12T16:55:25.785 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:25.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:55:25.835 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:55:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:25.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:55:25.940 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:24.425420+0000 mgr.y (mgr.24481) 223 : cluster [DBG] pgmap v246: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 3593/140118 objects misplaced (2.564%); 42 MiB/s, 678 objects/s recovering 2024-02-12T16:55:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:24.519751+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e152: 8 total, 8 up, 7 in 2024-02-12T16:55:25.998 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:26.062 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:26.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:26.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:55:26.201 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:26.235 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:55:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:26.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:55:26.340 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:26.404 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:26.746 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:26.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:25.330407+0000 osd.1 (osd.1) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:26.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:25.330443+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:26.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:25.526166+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e153: 8 total, 8 up, 7 in 2024-02-12T16:55:26.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:25.330407+0000 osd.1 (osd.1) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:26.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:25.330443+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:26.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:25.526166+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e153: 8 total, 8 up, 7 in 2024-02-12T16:55:26.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:25.330407+0000 osd.1 (osd.1) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:26.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:25.330443+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:26.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:25.526166+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e153: 8 total, 8 up, 7 in 2024-02-12T16:55:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:27.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:27.403 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:27.495 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:27.557 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:27.591 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:55:27.620 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:26.083402+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:27.620 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:26.083451+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:27.620 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:26.426258+0000 mgr.y (mgr.24481) 224 : cluster [DBG] pgmap v249: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 3593/140118 objects misplaced (2.564%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:55:27.620 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:26.535206+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e154: 8 total, 8 up, 7 in 2024-02-12T16:55:27.620 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:26.876909+0000 mon.c (mon.2) 96 : audit [INF] from='client.? v1:172.21.15.33:0/1691119751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:55:27.621 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:26.879596+0000 mon.a (mon.0) 892 : audit [INF] from='client.34919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:55:27.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:26.083402+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:27.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:26.083451+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:27.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:26.426258+0000 mgr.y (mgr.24481) 224 : cluster [DBG] pgmap v249: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 3593/140118 objects misplaced (2.564%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:55:27.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:26.535206+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e154: 8 total, 8 up, 7 in 2024-02-12T16:55:27.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:26.876909+0000 mon.c (mon.2) 96 : audit [INF] from='client.? v1:172.21.15.33:0/1691119751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:55:27.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:26.879596+0000 mon.a (mon.0) 892 : audit [INF] from='client.34919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:55:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:26.083402+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:26.083451+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:27.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:26.426258+0000 mgr.y (mgr.24481) 224 : cluster [DBG] pgmap v249: 148 pgs: 145 active+clean, 3 active+remapped+backfilling; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 3593/140118 objects misplaced (2.564%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:55:27.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:26.535206+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e154: 8 total, 8 up, 7 in 2024-02-12T16:55:27.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:26.876909+0000 mon.c (mon.2) 96 : audit [INF] from='client.? v1:172.21.15.33:0/1691119751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:55:27.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:26.879596+0000 mon.a (mon.0) 892 : audit [INF] from='client.34919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:55:28.886 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:27.543494+0000 mon.a (mon.0) 893 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:28.886 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:27.560352+0000 mon.a (mon.0) 894 : audit [INF] from='client.34919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:55:28.886 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:27.560418+0000 mon.a (mon.0) 895 : cluster [DBG] osdmap e155: 8 total, 8 up, 7 in 2024-02-12T16:55:28.887 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:27.543494+0000 mon.a (mon.0) 893 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:28.887 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:27.560352+0000 mon.a (mon.0) 894 : audit [INF] from='client.34919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:55:28.887 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:27.560418+0000 mon.a (mon.0) 895 : cluster [DBG] osdmap e155: 8 total, 8 up, 7 in 2024-02-12T16:55:28.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:27.543494+0000 mon.a (mon.0) 893 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:28.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:27.560352+0000 mon.a (mon.0) 894 : audit [INF] from='client.34919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:55:28.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:27.560418+0000 mon.a (mon.0) 895 : cluster [DBG] osdmap e155: 8 total, 8 up, 7 in 2024-02-12T16:55:29.339 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:55:29.339 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:55:29.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:28.426922+0000 mgr.y (mgr.24481) 225 : cluster [DBG] pgmap v252: 148 pgs: 146 active+clean, 2 active+remapped+backfilling; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 3263/132360 objects misplaced (2.465%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:55:29.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:28.440144+0000 osd.1 (osd.1) 112 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:55:29.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:28.449043+0000 osd.1 (osd.1) 113 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:55:29.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:28.578325+0000 mon.a (mon.0) 896 : cluster [DBG] osdmap e156: 8 total, 8 up, 7 in 2024-02-12T16:55:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:28.426922+0000 mgr.y (mgr.24481) 225 : cluster [DBG] pgmap v252: 148 pgs: 146 active+clean, 2 active+remapped+backfilling; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 3263/132360 objects misplaced (2.465%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:55:30.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:28.440144+0000 osd.1 (osd.1) 112 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:55:30.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:28.449043+0000 osd.1 (osd.1) 113 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:55:30.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:28.578325+0000 mon.a (mon.0) 896 : cluster [DBG] osdmap e156: 8 total, 8 up, 7 in 2024-02-12T16:55:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:28.426922+0000 mgr.y (mgr.24481) 225 : cluster [DBG] pgmap v252: 148 pgs: 146 active+clean, 2 active+remapped+backfilling; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 3263/132360 objects misplaced (2.465%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:55:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:28.440144+0000 osd.1 (osd.1) 112 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:55:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:28.449043+0000 osd.1 (osd.1) 113 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:55:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:28.578325+0000 mon.a (mon.0) 896 : cluster [DBG] osdmap e156: 8 total, 8 up, 7 in 2024-02-12T16:55:30.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:29.587403+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e157: 8 total, 8 up, 7 in 2024-02-12T16:55:30.882 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:29.587403+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e157: 8 total, 8 up, 7 in 2024-02-12T16:55:30.882 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:55:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:30] "GET /metrics HTTP/1.1" 200 175629 "" "Prometheus/2.33.4" 2024-02-12T16:55:30.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:55:30.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:29.587403+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e157: 8 total, 8 up, 7 in 2024-02-12T16:55:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:30.427328+0000 mgr.y (mgr.24481) 226 : cluster [DBG] pgmap v255: 148 pgs: 146 active+clean, 1 active+clean+remapped, 1 active+remapped+backfilling; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 2188/127872 objects misplaced (1.711%); 18 MiB/s, 285 objects/s recovering 2024-02-12T16:55:31.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:30.427328+0000 mgr.y (mgr.24481) 226 : cluster [DBG] pgmap v255: 148 pgs: 146 active+clean, 1 active+clean+remapped, 1 active+remapped+backfilling; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 2188/127872 objects misplaced (1.711%); 18 MiB/s, 285 objects/s recovering 2024-02-12T16:55:31.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:55:31.870 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:31.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:31.976 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:30.427328+0000 mgr.y (mgr.24481) 226 : cluster [DBG] pgmap v255: 148 pgs: 146 active+clean, 1 active+clean+remapped, 1 active+remapped+backfilling; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 2188/127872 objects misplaced (1.711%); 18 MiB/s, 285 objects/s recovering 2024-02-12T16:55:32.041 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:32.217 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:32.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:32.326 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:32.370 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:32.735 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:32.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:32.791 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:32.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:32.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:31.895812+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:32.885 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:31.895876+0000 osd.3 (osd.3) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:32.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:31.895812+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:32.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:31.895876+0000 osd.3 (osd.3) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:32.887 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:32.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:31.895812+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:32.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:31.895876+0000 osd.3 (osd.3) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:32.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:55:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:55:33.616 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:32.429048+0000 mgr.y (mgr.24481) 227 : cluster [DBG] pgmap v256: 148 pgs: 146 active+clean, 1 active+clean+remapped, 1 active+remapped+backfilling; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 868 B/s rd, 0 B/s wr, 693 op/s; 2188/127872 objects misplaced (1.711%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:55:33.616 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:32.614923+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e158: 8 total, 8 up, 7 in 2024-02-12T16:55:33.654 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:33.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:33.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:33.805 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:33.830 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:33.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:33.938 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:33.984 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:32.429048+0000 mgr.y (mgr.24481) 227 : cluster [DBG] pgmap v256: 148 pgs: 146 active+clean, 1 active+clean+remapped, 1 active+remapped+backfilling; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 868 B/s rd, 0 B/s wr, 693 op/s; 2188/127872 objects misplaced (1.711%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:55:33.984 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:32.614923+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e158: 8 total, 8 up, 7 in 2024-02-12T16:55:33.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:55:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:55:33.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:32.429048+0000 mgr.y (mgr.24481) 227 : cluster [DBG] pgmap v256: 148 pgs: 146 active+clean, 1 active+clean+remapped, 1 active+remapped+backfilling; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 868 B/s rd, 0 B/s wr, 693 op/s; 2188/127872 objects misplaced (1.711%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:55:33.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:32.614923+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e158: 8 total, 8 up, 7 in 2024-02-12T16:55:34.009 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:34.371 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:34.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:34.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:55:34.523 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:55:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:34.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:55:34.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:33.604081+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:55:34.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:33.609827+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:55:34.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:33.623794+0000 mgr.y (mgr.24481) 228 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:34.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:34.025335+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.33:0/1077057329' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:55:34.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:34.025735+0000 mon.a (mon.0) 899 : audit [INF] from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:55:34.658 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T16:55:34.663 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:34.721 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:34.749 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:34.761 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:55:34.821 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:55:34.858 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:55:34.890 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:55:34.926 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:55:34.963 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:34.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:34.968 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:55:34.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:33.604081+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:55:34.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:33.609827+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:55:34.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:33.623794+0000 mgr.y (mgr.24481) 228 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:34.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:34.025335+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.33:0/1077057329' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:55:34.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:34.025735+0000 mon.a (mon.0) 899 : audit [INF] from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:55:34.989 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:55:34.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:33.604081+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:55:34.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:33.609827+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:55:34.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:33.623794+0000 mgr.y (mgr.24481) 228 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:34.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:34.025335+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.33:0/1077057329' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:55:34.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:34.025735+0000 mon.a (mon.0) 899 : audit [INF] from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:55:35.017 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:55:35.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:55:35.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:55:35.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:55:35.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:55:35.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:55:35.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:55:35.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:55:35.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:55:35.069 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:35.144 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:35.196 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:35.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:35.304 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:35.365 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:35.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:35.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:35.524 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:35.549 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:35.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:35.641 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:34.429908+0000 mgr.y (mgr.24481) 229 : cluster [DBG] pgmap v258: 148 pgs: 147 active+clean, 1 active+remapped+backfilling; 2.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 999 op/s; 1878/114396 objects misplaced (1.642%); 11 MiB/s, 182 objects/s recovering 2024-02-12T16:55:35.641 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:34.595251+0000 osd.4 (osd.4) 75 : cluster [DBG] 2.f scrub starts 2024-02-12T16:55:35.641 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:34.601445+0000 osd.4 (osd.4) 76 : cluster [DBG] 2.f scrub ok 2024-02-12T16:55:35.641 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:34.640851+0000 mon.a (mon.0) 900 : audit [INF] from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:55:35.641 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:34.640956+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e159: 8 total, 8 up, 7 in 2024-02-12T16:55:35.641 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:34.704472+0000 mon.a (mon.0) 902 : audit [DBG] from='client.? v1:172.21.15.33:0/960218750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:35.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:35.712 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:35.939 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:34.429908+0000 mgr.y (mgr.24481) 229 : cluster [DBG] pgmap v258: 148 pgs: 147 active+clean, 1 active+remapped+backfilling; 2.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 999 op/s; 1878/114396 objects misplaced (1.642%); 11 MiB/s, 182 objects/s recovering 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:34.595251+0000 osd.4 (osd.4) 75 : cluster [DBG] 2.f scrub starts 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:34.601445+0000 osd.4 (osd.4) 76 : cluster [DBG] 2.f scrub ok 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:34.640851+0000 mon.a (mon.0) 900 : audit [INF] from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:34.640956+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e159: 8 total, 8 up, 7 in 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:34.704472+0000 mon.a (mon.0) 902 : audit [DBG] from='client.? v1:172.21.15.33:0/960218750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:34.429908+0000 mgr.y (mgr.24481) 229 : cluster [DBG] pgmap v258: 148 pgs: 147 active+clean, 1 active+remapped+backfilling; 2.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 999 op/s; 1878/114396 objects misplaced (1.642%); 11 MiB/s, 182 objects/s recovering 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:34.595251+0000 osd.4 (osd.4) 75 : cluster [DBG] 2.f scrub starts 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:34.601445+0000 osd.4 (osd.4) 76 : cluster [DBG] 2.f scrub ok 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:34.640851+0000 mon.a (mon.0) 900 : audit [INF] from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:55:35.940 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:34.640956+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e159: 8 total, 8 up, 7 in 2024-02-12T16:55:35.941 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:34.704472+0000 mon.a (mon.0) 902 : audit [DBG] from='client.? v1:172.21.15.33:0/960218750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:35.947 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:35.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:36.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:36.113 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:36.673 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:36.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:36.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T16:55:36.787 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:36.849 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:37.320 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:37.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:37.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:37.488 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:36.114076+0000 osd.6 (osd.6) 86 : cluster [DBG] 7.1 scrub starts 2024-02-12T16:55:37.488 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:36.434765+0000 mgr.y (mgr.24481) 230 : cluster [DBG] pgmap v260: 148 pgs: 147 active+clean, 1 active+remapped+backfilling; 2.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 747 B/s rd, 0 B/s wr, 656 op/s; 1878/114396 objects misplaced (1.642%); 542 KiB/s, 8 objects/s recovering 2024-02-12T16:55:37.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:36.114076+0000 osd.6 (osd.6) 86 : cluster [DBG] 7.1 scrub starts 2024-02-12T16:55:37.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:36.434765+0000 mgr.y (mgr.24481) 230 : cluster [DBG] pgmap v260: 148 pgs: 147 active+clean, 1 active+remapped+backfilling; 2.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 747 B/s rd, 0 B/s wr, 656 op/s; 1878/114396 objects misplaced (1.642%); 542 KiB/s, 8 objects/s recovering 2024-02-12T16:55:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:36.114076+0000 osd.6 (osd.6) 86 : cluster [DBG] 7.1 scrub starts 2024-02-12T16:55:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:36.434765+0000 mgr.y (mgr.24481) 230 : cluster [DBG] pgmap v260: 148 pgs: 147 active+clean, 1 active+remapped+backfilling; 2.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 747 B/s rd, 0 B/s wr, 656 op/s; 1878/114396 objects misplaced (1.642%); 542 KiB/s, 8 objects/s recovering 2024-02-12T16:55:37.490 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:37.966 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:37.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:38.071 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:38.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:38.196 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:38.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:38.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:38.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:36.747144+0000 osd.6 (osd.6) 87 : cluster [DBG] 7.1 scrub ok 2024-02-12T16:55:38.228 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:36.747144+0000 osd.6 (osd.6) 87 : cluster [DBG] 7.1 scrub ok 2024-02-12T16:55:38.300 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:55:38.345 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:38.375 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:55:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:38.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:55:38.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:55:38.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:36.747144+0000 osd.6 (osd.6) 87 : cluster [DBG] 7.1 scrub ok 2024-02-12T16:55:38.535 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:38.568 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:55:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:38.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:55:38.672 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:55:38.738 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:39.172 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T16:55:39.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:38.435482+0000 mgr.y (mgr.24481) 231 : cluster [DBG] pgmap v261: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 971 op/s; 1568/104544 objects misplaced (1.500%); 927 KiB/s, 14 objects/s recovering 2024-02-12T16:55:39.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:39.004880+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.33:0/2316834707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:39.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:39.005278+0000 mon.a (mon.0) 903 : audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:39.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:38.435482+0000 mgr.y (mgr.24481) 231 : cluster [DBG] pgmap v261: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 971 op/s; 1568/104544 objects misplaced (1.500%); 927 KiB/s, 14 objects/s recovering 2024-02-12T16:55:39.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:39.004880+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.33:0/2316834707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:39.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:39.005278+0000 mon.a (mon.0) 903 : audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:39.281 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:55:39.281 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T16:55:39.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T16:55:39.281 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49330 7.f 7:fb50b513:::benchmark_data_smithi033_81060_object49329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.945656+0000", 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 159.32726771899999, 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065405695999999999, 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49330 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945656+0000", 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945656+0000", 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945654+0000", 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2903000000000001e-05 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945677+0000", 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0499000000000001e-05 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945688+0000", 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2190000000000001e-06 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945694+0000", 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0555999999999997e-05 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945725+0000", 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012409199999999999 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945849+0000", 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014497699999999999 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.945994+0000", 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002168957 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948163+0000", 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001750548 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949913+0000", 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.061029917000000003 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.010943+0000", 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8995000000000001e-05 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.010992+0000", 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0012000000000007e-05 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.011062+0000", 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065405695999999999 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49335 7.6 7:6cb921eb:::benchmark_data_smithi033_81060_object49334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.948664+0000", 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 159.32425956200001, 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066631657999999996, 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49335 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948664+0000", 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948664+0000", 2024-02-12T16:55:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948662+0000", 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3328000000000001e-05 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948686+0000", 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0886e-05 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948697+0000", 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5139999999999998e-06 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948703+0000", 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5175e-05 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948748+0000", 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111309 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948860+0000", 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000149159 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949009+0000", 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025638240000000001 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.951573+0000", 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013423020000000001 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.952915+0000", 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062296983 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015212+0000", 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8344000000000003e-05 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015260+0000", 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6001000000000003e-05 2024-02-12T16:55:39.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015296+0000", 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066631657999999996 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49338 7.4 7:21564e79:::benchmark_data_smithi033_81060_object49337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.948823+0000", 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 159.324100953, 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062976378, 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.294 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49338 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948823+0000", 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948823+0000", 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948822+0000", 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1523e-05 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948844+0000", 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0600000000000002e-06 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948851+0000", 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2089999999999996e-06 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948856+0000", 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001957622 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.950814+0000", 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0819e-05 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.950894+0000", 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9448000000000001e-05 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.950994+0000", 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020366469999999999 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953031+0000", 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055526143 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.008557+0000", 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031402140000000001 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.011697+0000", 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4224000000000001e-05 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.011751+0000", 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8350999999999998e-05 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.011799+0000", 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062976378 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49341 7.6 7:69576c26:::benchmark_data_smithi033_81060_object49340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.953198+0000", 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 159.319725764, 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062208072000000003, 2024-02-12T16:55:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49341 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953198+0000", 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953198+0000", 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953197+0000", 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8589999999999999e-05 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953225+0000", 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055228963999999998 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.008454+0000", 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4292999999999998e-05 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.008488+0000", 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105486 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.008594+0000", 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012830799999999999 2024-02-12T16:55:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.008722+0000", 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000284002 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.009006+0000", 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028474009999999998 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.011854+0000", 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018806500000000001 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.012042+0000", 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0032875959999999998 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015329+0000", 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5850000000000003e-05 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015375+0000", 2024-02-12T16:55:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1232000000000002e-05 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015406+0000", 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062208072000000003 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79151 7.6 7:69fe64ad:::benchmark_data_smithi033_81060_object79150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.435509+0000", 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 131.83741489499999, 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.080437704999999998, 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79151 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435509+0000", 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435509+0000", 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435508+0000", 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5213999999999999e-05 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435534+0000", 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0950000000000001e-06 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435540+0000", 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2889999999999999e-06 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435544+0000", 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000888702 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.436433+0000", 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105937 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.436539+0000", 2024-02-12T16:55:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013627600000000001 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.436675+0000", 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048124750000000001 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441487+0000", 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000209242 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441697+0000", 2024-02-12T16:55:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074153001999999996 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515850+0000", 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6749000000000002e-05 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515906+0000", 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0556000000000003e-05 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515947+0000", 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.080437704999999998 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:79148 7.e e120/93 7:7bc62154:::benchmark_data_smithi033_81060_object79147:head v 120'4929, mlcod=120'4927)", 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.437567+0000", 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 131.83535682300001, 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074636276000000001, 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.437567+0000", 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.437567+0000", 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.437567+0000", 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3461000000000001e-05 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.437590+0000", 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9889999999999997e-06 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.437596+0000", 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8419999999999998e-06 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.437600+0000", 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017456380000000001 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.439345+0000", 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5069e-05 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.439371+0000", 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072788098999999995 2024-02-12T16:55:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512159+0000", 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4892999999999999e-05 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512204+0000", 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074636276000000001 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:79156 7.e e120/93 7:7b76d634:::benchmark_data_smithi033_81060_object79155:head v 120'4930, mlcod=120'4927)", 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.440467+0000", 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 131.83245741799999, 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071761282999999995, 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440467+0000", 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440467+0000", 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440466+0000", 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2099999999999998e-05 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440488+0000", 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5579999999999997e-06 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440494+0000", 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4400000000000001e-06 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440498+0000", 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001279395 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441777+0000", 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9480999999999999e-05 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441797+0000", 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070412190999999999 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512209+0000", 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9088000000000001e-05 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512228+0000", 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071761282999999995 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:79158 7.e e120/93 7:76c987fd:::benchmark_data_smithi033_81060_object79157:head v 120'4931, mlcod=120'4927)", 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.442990+0000", 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 131.82993446399999, 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.069258136999999997, 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.442990+0000", 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.442990+0000", 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.442988+0000", 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3274000000000001e-05 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.443011+0000", 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1479999999999995e-06 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.443020+0000", 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9270000000000002e-06 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.443026+0000", 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001339355 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.444366+0000", 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8063999999999998e-05 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.444384+0000", 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.067847168999999999 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512231+0000", 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6725999999999999e-05 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512248+0000", 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.069258136999999997 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79160 7.f 7:f96ee9b4:::benchmark_data_smithi033_81060_object79159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.445660+0000", 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 131.82726431399999, 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077457349999999994, 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.325 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79160 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445660+0000", 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445660+0000", 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445658+0000", 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5193000000000001e-05 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445683+0000", 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9250000000000001e-06 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445692+0000", 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6190000000000002e-06 2024-02-12T16:55:39.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445697+0000", 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4807e-05 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445741+0000", 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012875000000000001 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445870+0000", 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066321670999999999 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512192+0000", 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0033652640000000002 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515557+0000", 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011440129999999999 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.516701+0000", 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0063113459999999998 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.523012+0000", 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4727000000000006e-05 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.523077+0000", 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.998e-05 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.523117+0000", 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077457349999999994 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80687 7.4 7:25452875:::benchmark_data_smithi033_81060_object80686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.034323+0000", 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 130.23860074999999, 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068050093000000006, 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80687 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034323+0000", 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034323+0000", 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034321+0000", 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9020000000000002e-05 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034370+0000", 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2130000000000001e-05 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034382+0000", 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2330000000000003e-06 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034388+0000", 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014932699999999999 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034538+0000", 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117614 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034655+0000", 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015677099999999999 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.034812+0000", 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043466140000000004 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039159+0000", 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012162107 2024-02-12T16:55:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051321+0000", 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024389160000000002 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.053760+0000", 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048554997000000003 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102315+0000", 2024-02-12T16:55:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8618e-05 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102373+0000", 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068050093000000006 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80694 7.4 7:25f53623:::benchmark_data_smithi033_81060_object80693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.039170+0000", 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 130.23375398300001, 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063293487999999995, 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80694 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039170+0000", 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039170+0000", 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:55:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039169+0000", 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1829000000000001e-05 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039191+0000", 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.463e-06 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039197+0000", 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8160000000000004e-06 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039201+0000", 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0015000000000002e-05 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039241+0000", 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014107799999999999 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039382+0000", 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014739899999999999 2024-02-12T16:55:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039529+0000", 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011836342 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051365+0000", 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3636000000000001e-05 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051399+0000", 2024-02-12T16:55:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050988243000000003 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102387+0000", 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2069e-05 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102429+0000", 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4264000000000003e-05 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102464+0000", 2024-02-12T16:55:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063293487999999995 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80696 7.4 7:25e25296:::benchmark_data_smithi033_81060_object80695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.039210+0000", 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 130.233714104, 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063546166000000001, 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80696 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039210+0000", 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039210+0000", 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039209+0000", 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2877000000000001e-05 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039232+0000", 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2099999999999998e-06 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039238+0000", 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2039999999999999e-06 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039243+0000", 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002916559 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.042159+0000", 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012073 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.042280+0000", 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000172273 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.042452+0000", 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089794420000000007 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051432+0000", 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.626e-05 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051448+0000", 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051223839 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102672+0000", 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7210999999999999e-05 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102719+0000", 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7221000000000002e-05 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102756+0000", 2024-02-12T16:55:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063546166000000001 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80697 7.4 7:293c7397:::benchmark_data_smithi033_81060_object80696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.039248+0000", 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 130.23367642299999, 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063590070999999998, 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80697 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039248+0000", 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039248+0000", 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039247+0000", 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2529000000000001e-05 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039269+0000", 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7200000000000003e-06 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039275+0000", 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9820000000000002e-06 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.039279+0000", 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0058180829999999999 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045097+0000", 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000112959 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045210+0000", 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000141991 2024-02-12T16:55:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045352+0000", 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061161870000000004 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051468+0000", 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051131734999999998 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102600+0000", 2024-02-12T16:55:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016393500000000001 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102764+0000", 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9511999999999997e-05 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102803+0000", 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4289000000000003e-05 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102838+0000", 2024-02-12T16:55:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063590070999999998 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80704 7.6 7:653f0e61:::benchmark_data_smithi033_81060_object80703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.045459+0000", 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 130.227465043, 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.061738464999999999, 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80704 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045459+0000", 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045459+0000", 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045457+0000", 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3547999999999999e-05 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045481+0000", 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4350000000000003e-06 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045490+0000", 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7440000000000004e-06 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045496+0000", 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4048999999999998e-05 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045540+0000", 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114169 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045654+0000", 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000142607 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.045797+0000", 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056816590000000004 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051478+0000", 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055611577000000002 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.107090+0000", 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4039999999999998e-06 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.107097+0000", 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4392999999999999e-05 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.107152+0000", 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5781000000000001e-05 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.107197+0000", 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.061738464999999999 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80707 7.4 7:28b32254:::benchmark_data_smithi033_81060_object80706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.047948+0000", 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 130.22497574900001, 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.061916863000000003, 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80707 2024-02-12T16:55:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047948+0000", 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047948+0000", 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047947+0000", 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1271e-05 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047969+0000", 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6529999999999997e-06 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047975+0000", 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7840000000000003e-06 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047980+0000", 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035758600000000001 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051556+0000", 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011100299999999999 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051667+0000", 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010014900000000001 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051767+0000", 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050890412000000003 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102657+0000", 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026453199999999999 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102922+0000", 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068581459999999999 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.109780+0000", 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5154999999999997e-05 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.109825+0000", 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9860000000000001e-05 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.109865+0000", 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.061916863000000003 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80709 7.6 7:6da36266:::benchmark_data_smithi033_81060_object80708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.053559+0000", 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 130.21936488899999, 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066326457000000005, 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80709 2024-02-12T16:55:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.053559+0000", 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.053559+0000", 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.053557+0000", 2024-02-12T16:55:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9181000000000003e-05 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.053596+0000", 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3125000000000001e-05 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.053609+0000", 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4229999999999999e-05 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.053623+0000", 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9437000000000001e-05 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.053673+0000", 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050576709999999997 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.104249+0000", 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000177259 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:55:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.104427+0000", 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004124595 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.108551+0000", 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029275799999999999 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.111479+0000", 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083179570000000008 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.119797+0000", 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1072000000000002e-05 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.119848+0000", 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7846000000000003e-05 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.119886+0000", 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066326457000000005 2024-02-12T16:55:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:155628 7.4 7:28075c1c:::benchmark_data_smithi033_81060_object45176:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:54:58.070391+0000", 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.202533164000002, 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062027256000000003, 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 155628 2024-02-12T16:55:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070391+0000", 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070391+0000", 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070388+0000", 2024-02-12T16:55:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.827e-06 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070395+0000", 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3249999999999997e-06 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070404+0000", 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.767e-06 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070410+0000", 2024-02-12T16:55:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042522300000000001 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.074662+0000", 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7524999999999999e-05 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.074760+0000", 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011731 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.074877+0000", 2024-02-12T16:55:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088606620000000001 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.083738+0000", 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010134494000000001 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.093872+0000", 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038463063999999998 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.132336+0000", 2024-02-12T16:55:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6315e-05 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.132382+0000", 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6294999999999997e-05 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.132418+0000", 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062027256000000003 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:155630 7.4 7:29d04e5e:::benchmark_data_smithi033_81060_object45178:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:54:58.070444+0000", 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.202480010000002, 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062091765, 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 155630 2024-02-12T16:55:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070444+0000", 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070444+0000", 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070443+0000", 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9229999999999998e-06 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070446+0000", 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3709999999999996e-06 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070453+0000", 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8850000000000001e-06 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.070457+0000", 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070019710000000001 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.077459+0000", 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120347 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.077579+0000", 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014167499999999999 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.077721+0000", 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060240060000000002 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.083745+0000", 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010159588000000001 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.093904+0000", 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038556 2024-02-12T16:55:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.132460+0000", 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9817999999999997e-05 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.132500+0000", 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5815000000000002e-05 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.132536+0000", 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062091765 2024-02-12T16:55:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161366 7.6 7:6c8733f1:::benchmark_data_smithi033_81060_object50914:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.116764+0000", 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 33.156160329000002, 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.060861713999999997, 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161366 2024-02-12T16:55:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116764+0000", 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116764+0000", 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116761+0000", 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7610000000000007e-06 2024-02-12T16:55:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116769+0000", 2024-02-12T16:55:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1229e-05 2024-02-12T16:55:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116780+0000", 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6529999999999997e-06 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116787+0000", 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023403349999999998 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.119127+0000", 2024-02-12T16:55:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015106900000000001 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.119278+0000", 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022205 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,7", 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.119500+0000", 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055387140000000001 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.125039+0000", 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6171000000000001e-05 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.125065+0000", 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026157609999999999 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.127681+0000", 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049759752999999997 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.177441+0000", 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4561999999999997e-05 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.177495+0000", 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013027399999999999 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.177625+0000", 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.060861713999999997 2024-02-12T16:55:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161367 7.6 7:6430b1a9:::benchmark_data_smithi033_81060_object50915:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.116793+0000", 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 33.156130697999998, 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.061493691000000003, 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161367 2024-02-12T16:55:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116793+0000", 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116793+0000", 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116793+0000", 2024-02-12T16:55:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8089999999999999e-06 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116796+0000", 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1589999999999998e-06 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116803+0000", 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4859999999999999e-06 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.116807+0000", 2024-02-12T16:55:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053355110000000002 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.122143+0000", 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000109744 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.122252+0000", 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000132805 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,7", 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.122385+0000", 2024-02-12T16:55:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0027277680000000002 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.125113+0000", 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8575999999999999e-05 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.125141+0000", 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025488260000000001 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.127690+0000", 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050406767999999998 2024-02-12T16:55:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.178097+0000", 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2753e-05 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.178160+0000", 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000127272 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.178287+0000", 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.061493691000000003 2024-02-12T16:55:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:39.399 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:55:39.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:38.435482+0000 mgr.y (mgr.24481) 231 : cluster [DBG] pgmap v261: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 971 op/s; 1568/104544 objects misplaced (1.500%); 927 KiB/s, 14 objects/s recovering 2024-02-12T16:55:39.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:39.004880+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.33:0/2316834707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:39.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:39.005278+0000 mon.a (mon.0) 903 : audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:55:40.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:39.128066+0000 mon.a (mon.0) 904 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:55:40.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:39.128095+0000 mon.a (mon.0) 905 : cluster [INF] Cluster is now healthy 2024-02-12T16:55:40.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:39.147443+0000 mon.a (mon.0) 906 : audit [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:55:40.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:39.147507+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e160: 8 total, 8 up, 7 in 2024-02-12T16:55:40.155 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:55:40.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:40.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:55:40.266 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:55:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:40.342 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:55:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:40.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:55:40.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:40.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:39.128066+0000 mon.a (mon.0) 904 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:55:40.486 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:39.128095+0000 mon.a (mon.0) 905 : cluster [INF] Cluster is now healthy 2024-02-12T16:55:40.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:39.147443+0000 mon.a (mon.0) 906 : audit [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:55:40.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:39.147507+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e160: 8 total, 8 up, 7 in 2024-02-12T16:55:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:39.128066+0000 mon.a (mon.0) 904 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:55:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:39.128095+0000 mon.a (mon.0) 905 : cluster [INF] Cluster is now healthy 2024-02-12T16:55:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:39.147443+0000 mon.a (mon.0) 906 : audit [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:55:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:39.147507+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e160: 8 total, 8 up, 7 in 2024-02-12T16:55:40.503 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:40.936 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:40.966 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:40.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:55:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:40.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:41.025 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:55:41.073 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:41.138 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:55:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:40] "GET /metrics HTTP/1.1" 200 175645 "" "Prometheus/2.33.4" 2024-02-12T16:55:41.198 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:41.294 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:41.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:41.397 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:55:41.399 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:40.437939+0000 mgr.y (mgr.24481) 232 : cluster [DBG] pgmap v263: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s; 1568/98658 objects misplaced (1.589%); 949 KiB/s, 14 objects/s recovering 2024-02-12T16:55:41.400 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:40.520347+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.6 deep-scrub starts 2024-02-12T16:55:41.400 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:40.530261+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.6 deep-scrub ok 2024-02-12T16:55:41.400 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:40.437939+0000 mgr.y (mgr.24481) 232 : cluster [DBG] pgmap v263: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s; 1568/98658 objects misplaced (1.589%); 949 KiB/s, 14 objects/s recovering 2024-02-12T16:55:41.400 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:40.520347+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.6 deep-scrub starts 2024-02-12T16:55:41.400 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:40.530261+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.6 deep-scrub ok 2024-02-12T16:55:41.428 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:40.437939+0000 mgr.y (mgr.24481) 232 : cluster [DBG] pgmap v263: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s; 1568/98658 objects misplaced (1.589%); 949 KiB/s, 14 objects/s recovering 2024-02-12T16:55:41.429 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:40.520347+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.6 deep-scrub starts 2024-02-12T16:55:41.429 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:40.530261+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.6 deep-scrub ok 2024-02-12T16:55:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:41.457 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:55:41.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:41.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:55:41.564 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:41.624 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:42.333 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:55:42.599 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:42.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:42.611 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:42.617 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:42.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:55:42.758 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:42.846 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:55:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:42.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:55:42.952 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:42.992 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:55:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:55:42.998 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:43.029 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:43.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:43.136 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:55:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:43.208 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:55:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:43.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:55:43.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:43.395 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:43.549 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:43.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:43.655 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:55:43.694 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:55:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:43.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:55:43.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:42.438351+0000 mgr.y (mgr.24481) 233 : cluster [DBG] pgmap v264: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s; 1568/98658 objects misplaced (1.589%); 463 KiB/s, 7 objects/s recovering 2024-02-12T16:55:43.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:42.438351+0000 mgr.y (mgr.24481) 233 : cluster [DBG] pgmap v264: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s; 1568/98658 objects misplaced (1.589%); 463 KiB/s, 7 objects/s recovering 2024-02-12T16:55:43.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:43.867 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:42.438351+0000 mgr.y (mgr.24481) 233 : cluster [DBG] pgmap v264: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s; 1568/98658 objects misplaced (1.589%); 463 KiB/s, 7 objects/s recovering 2024-02-12T16:55:43.867 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:55:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:55:43.867 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:43.895 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:43.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:44.001 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:44.059 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:44.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:43.609026+0000 mon.a (mon.0) 908 : audit [INF] from='client.? v1:172.21.15.33:0/1900430751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:55:44.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:43.634313+0000 mgr.y (mgr.24481) 234 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:44.541 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T16:55:44.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:43.609026+0000 mon.a (mon.0) 908 : audit [INF] from='client.? v1:172.21.15.33:0/1900430751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:55:44.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:43.634313+0000 mgr.y (mgr.24481) 234 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:44.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:43.609026+0000 mon.a (mon.0) 908 : audit [INF] from='client.? v1:172.21.15.33:0/1900430751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:55:45.002 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:43.634313+0000 mgr.y (mgr.24481) 234 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:45.520 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:45.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:45.556 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-12T16:55:45.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:45.666 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:45.701 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:45.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:45.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:44.439200+0000 mgr.y (mgr.24481) 235 : cluster [DBG] pgmap v265: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1260/82944 objects misplaced (1.519%); 758 KiB/s, 11 objects/s recovering 2024-02-12T16:55:45.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:44.520035+0000 mon.a (mon.0) 909 : audit [INF] from='client.? v1:172.21.15.33:0/1900430751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:55:45.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:44.520087+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e161: 8 total, 8 up, 7 in 2024-02-12T16:55:45.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:44.439200+0000 mgr.y (mgr.24481) 235 : cluster [DBG] pgmap v265: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1260/82944 objects misplaced (1.519%); 758 KiB/s, 11 objects/s recovering 2024-02-12T16:55:45.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:44.520035+0000 mon.a (mon.0) 909 : audit [INF] from='client.? v1:172.21.15.33:0/1900430751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:55:45.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:44.520087+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e161: 8 total, 8 up, 7 in 2024-02-12T16:55:45.807 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:45.888 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:45.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:44.439200+0000 mgr.y (mgr.24481) 235 : cluster [DBG] pgmap v265: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1260/82944 objects misplaced (1.519%); 758 KiB/s, 11 objects/s recovering 2024-02-12T16:55:45.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:44.520035+0000 mon.a (mon.0) 909 : audit [INF] from='client.? v1:172.21.15.33:0/1900430751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:55:45.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:44.520087+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e161: 8 total, 8 up, 7 in 2024-02-12T16:55:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:46.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:46.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:46.387 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:46.418 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:46.435 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:55:46.435 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:55:46.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.435 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:196726 7.2 7:4b4c6e41:::benchmark_data_smithi033_81060_object86274:head [delete] snapc 0=[] ondisk+write+known_if_redirected e161)", 2024-02-12T16:55:46.435 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:46.428784+0000", 2024-02-12T16:55:46.435 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0032469090000000001, 2024-02-12T16:55:46.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0032733530000000001, 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 196726 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.428784+0000", 2024-02-12T16:55:46.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.428784+0000", 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.428782+0000", 2024-02-12T16:55:46.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3459999999999996e-06 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.428789+0000", 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6099999999999998e-05 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.428805+0000", 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.25e-06 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.428812+0000", 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074837199999999997 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.429561+0000", 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000127885 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.429689+0000", 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014360299999999999 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,7", 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:46.429832+0000", 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00104839 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:55:46.440 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 1 2024-02-12T16:55:46.441 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:55:46.458 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:46.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:46.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:46.637 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:46.660 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:46.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:46.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:46.826 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:47.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:46.157167+0000 osd.6 (osd.6) 88 : cluster [DBG] 4.c scrub starts 2024-02-12T16:55:47.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:46.162832+0000 osd.6 (osd.6) 89 : cluster [DBG] 4.c scrub ok 2024-02-12T16:55:47.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:46.439684+0000 mgr.y (mgr.24481) 236 : cluster [DBG] pgmap v267: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 1260/82944 objects misplaced (1.519%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:55:47.134 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:46.157167+0000 osd.6 (osd.6) 88 : cluster [DBG] 4.c scrub starts 2024-02-12T16:55:47.135 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:46.162832+0000 osd.6 (osd.6) 89 : cluster [DBG] 4.c scrub ok 2024-02-12T16:55:47.135 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:46.439684+0000 mgr.y (mgr.24481) 236 : cluster [DBG] pgmap v267: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 1260/82944 objects misplaced (1.519%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:55:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:47.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:47.294 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:47.340 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:47.379 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:47.384 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:46.157167+0000 osd.6 (osd.6) 88 : cluster [DBG] 4.c scrub starts 2024-02-12T16:55:47.384 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:46.162832+0000 osd.6 (osd.6) 89 : cluster [DBG] 4.c scrub ok 2024-02-12T16:55:47.384 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:46.439684+0000 mgr.y (mgr.24481) 236 : cluster [DBG] pgmap v267: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 1260/82944 objects misplaced (1.519%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:55:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:47.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:47.486 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:47.534 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:47.982 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:55:48.617 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:47.470978+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:48.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:47.470978+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:48.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:47.470978+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:48.837 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:48.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:48.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:55:48.958 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:55:48.958 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:55:48.958 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:55:48.958 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:55:48.959 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:55:48.959 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:55:48.959 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:55:48.992 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:55:48.995 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:48.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:55:48.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:55:48.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:55:48.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:55:48.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:55:48.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:55:48.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:55:48.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:55:49.021 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:55:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:49.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:55:49.126 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:49.170 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:49.545 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:49.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:49.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:55:49.698 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:49.739 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:55:49.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:48.443711+0000 mgr.y (mgr.24481) 237 : cluster [DBG] pgmap v268: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 551 B/s rd, 0 B/s wr, 921 op/s; 971/72993 objects misplaced (1.330%); 799 KiB/s, 12 objects/s recovering 2024-02-12T16:55:49.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:48.677647+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.33:0/4111758353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:49.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:55:49.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:48.443711+0000 mgr.y (mgr.24481) 237 : cluster [DBG] pgmap v268: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 551 B/s rd, 0 B/s wr, 921 op/s; 971/72993 objects misplaced (1.330%); 799 KiB/s, 12 objects/s recovering 2024-02-12T16:55:49.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:48.677647+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.33:0/4111758353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:49.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:48.443711+0000 mgr.y (mgr.24481) 237 : cluster [DBG] pgmap v268: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 551 B/s rd, 0 B/s wr, 921 op/s; 971/72993 objects misplaced (1.330%); 799 KiB/s, 12 objects/s recovering 2024-02-12T16:55:49.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:48.677647+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.33:0/4111758353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:55:49.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:49.923 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:49.980 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:49.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:50.085 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:50.152 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:50.191 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:50.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:50.301 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:50.335 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:50.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:50.467 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:55:50.504 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:50.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:50.460313+0000 mon.a (mon.0) 912 : audit [INF] from='client.? v1:172.21.15.33:0/881855501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:55:50.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:50.460313+0000 mon.a (mon.0) 912 : audit [INF] from='client.? v1:172.21.15.33:0/881855501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:55:50.528 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:55:50.531 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:55:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:50.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:55:50.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:50.686 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:50.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:50.460313+0000 mon.a (mon.0) 912 : audit [INF] from='client.? v1:172.21.15.33:0/881855501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:55:50.900 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:50.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:51.007 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:55:51.064 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:51.150 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:55:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:51.154 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:55:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:50] "GET /metrics HTTP/1.1" 200 175646 "" "Prometheus/2.33.4" 2024-02-12T16:55:51.154 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:51.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:55:51.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:51.300 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:51.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:51.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:51.499 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:51.719 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:51.728 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:51.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:50.445771+0000 mgr.y (mgr.24481) 238 : cluster [DBG] pgmap v269: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 971/66798 objects misplaced (1.454%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:55:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:50.485349+0000 mon.a (mon.0) 913 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:50.505820+0000 mon.a (mon.0) 914 : audit [INF] from='client.? v1:172.21.15.33:0/881855501' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:55:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:50.505873+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e162: 8 total, 8 up, 7 in 2024-02-12T16:55:51.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:50.445771+0000 mgr.y (mgr.24481) 238 : cluster [DBG] pgmap v269: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 971/66798 objects misplaced (1.454%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:55:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:50.485349+0000 mon.a (mon.0) 913 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:50.505820+0000 mon.a (mon.0) 914 : audit [INF] from='client.? v1:172.21.15.33:0/881855501' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:55:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:50.505873+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e162: 8 total, 8 up, 7 in 2024-02-12T16:55:51.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:50.445771+0000 mgr.y (mgr.24481) 238 : cluster [DBG] pgmap v269: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 971/66798 objects misplaced (1.454%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:55:51.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:50.485349+0000 mon.a (mon.0) 913 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:55:51.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:50.505820+0000 mon.a (mon.0) 914 : audit [INF] from='client.? v1:172.21.15.33:0/881855501' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:55:51.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:50.505873+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e162: 8 total, 8 up, 7 in 2024-02-12T16:55:51.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:51.899 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:52.007 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:52.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:52.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:55:52.161 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:52.626 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:55:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:52.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:55:52.732 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:52.774 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:55:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:55:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:55:52.775 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:51.989927+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:52.775 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:51.989977+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:52.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:51.989927+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:52.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:51.989977+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:52.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:51.989927+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:52.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:51.989977+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:52.788 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:52.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:52.908 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:55:52.931 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:53.024 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:53.298 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:53.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:53.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:55:53.454 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:53.735 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:55:53.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:52.447340+0000 mgr.y (mgr.24481) 239 : cluster [DBG] pgmap v271: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 672 op/s; 971/66798 objects misplaced (1.454%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:55:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:52.514221+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e163: 8 total, 8 up, 7 in 2024-02-12T16:55:53.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:52.447340+0000 mgr.y (mgr.24481) 239 : cluster [DBG] pgmap v271: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 672 op/s; 971/66798 objects misplaced (1.454%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:55:53.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:52.514221+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e163: 8 total, 8 up, 7 in 2024-02-12T16:55:53.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:52.447340+0000 mgr.y (mgr.24481) 239 : cluster [DBG] pgmap v271: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 672 op/s; 971/66798 objects misplaced (1.454%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:55:53.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:52.514221+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e163: 8 total, 8 up, 7 in 2024-02-12T16:55:53.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:55:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:55:54.708 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:53.645706+0000 mgr.y (mgr.24481) 240 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:54.708 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:54.184954+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:54.709 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:54.184992+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:54.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:53.645706+0000 mgr.y (mgr.24481) 240 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:54.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:54.184954+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:54.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:54.184992+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:54.724 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:55:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:54.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:55:54.831 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:55:54.878 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:53.645706+0000 mgr.y (mgr.24481) 240 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:55:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:54.184954+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:54.184992+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:55:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:55.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:55:55.123 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:55.165 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:55.196 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:55.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:55.301 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:55.375 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:55.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:55.534 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:55.549 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:54.448144+0000 mgr.y (mgr.24481) 241 : cluster [DBG] pgmap v273: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.0 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 695/51087 objects misplaced (1.360%); 927 KiB/s, 14 objects/s recovering 2024-02-12T16:55:55.549 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:54.533798+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e164: 8 total, 8 up, 7 in 2024-02-12T16:55:55.549 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:55.445056+0000 mon.c (mon.2) 100 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:55.585 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:55.612 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:55.615 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:55:55.615 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:55:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:55.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:55.718 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:55:55.793 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:55.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:54.448144+0000 mgr.y (mgr.24481) 241 : cluster [DBG] pgmap v273: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.0 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 695/51087 objects misplaced (1.360%); 927 KiB/s, 14 objects/s recovering 2024-02-12T16:55:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:54.533798+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e164: 8 total, 8 up, 7 in 2024-02-12T16:55:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:55.445056+0000 mon.c (mon.2) 100 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:56.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:54.448144+0000 mgr.y (mgr.24481) 241 : cluster [DBG] pgmap v273: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.0 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 695/51087 objects misplaced (1.360%); 927 KiB/s, 14 objects/s recovering 2024-02-12T16:55:56.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:54.533798+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e164: 8 total, 8 up, 7 in 2024-02-12T16:55:56.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:55.445056+0000 mon.c (mon.2) 100 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:56.641 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:56.658 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:56.665 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:55:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:56.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:55:56.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:55:56.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:55:56.820 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:55:56.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:55:56.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:55:56.951 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:55:56.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:55.791260+0000 mon.a (mon.0) 918 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:56.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:56.128647+0000 mon.a (mon.0) 919 : audit [INF] from='client.? v1:172.21.15.33:0/1695292610' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:55:56.971 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:55.791260+0000 mon.a (mon.0) 918 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:56.972 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:56.128647+0000 mon.a (mon.0) 919 : audit [INF] from='client.? v1:172.21.15.33:0/1695292610' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:55:57.027 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:55:57.120 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.6 to 1 (10000) 2024-02-12T16:55:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:55.791260+0000 mon.a (mon.0) 918 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:56.128647+0000 mon.a (mon.0) 919 : audit [INF] from='client.? v1:172.21.15.33:0/1695292610' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:55:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:56.448546+0000 mgr.y (mgr.24481) 242 : cluster [DBG] pgmap v275: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.0 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 695/51087 objects misplaced (1.360%); 618 KiB/s, 9 objects/s recovering 2024-02-12T16:55:58.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:56.871760+0000 mon.c (mon.2) 101 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:58.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:56.875603+0000 mon.c (mon.2) 102 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:58.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:56.893451+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:58.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:55:57.110667+0000 mon.a (mon.0) 921 : audit [INF] from='client.? v1:172.21.15.33:0/1695292610' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:55:58.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:57.110721+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e165: 8 total, 8 up, 7 in 2024-02-12T16:55:58.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:56.448546+0000 mgr.y (mgr.24481) 242 : cluster [DBG] pgmap v275: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.0 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 695/51087 objects misplaced (1.360%); 618 KiB/s, 9 objects/s recovering 2024-02-12T16:55:58.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:56.871760+0000 mon.c (mon.2) 101 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:58.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:56.875603+0000 mon.c (mon.2) 102 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:58.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:56.893451+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:58.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:55:57.110667+0000 mon.a (mon.0) 921 : audit [INF] from='client.? v1:172.21.15.33:0/1695292610' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:55:58.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:57.110721+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e165: 8 total, 8 up, 7 in 2024-02-12T16:55:58.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:56.448546+0000 mgr.y (mgr.24481) 242 : cluster [DBG] pgmap v275: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.0 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 695/51087 objects misplaced (1.360%); 618 KiB/s, 9 objects/s recovering 2024-02-12T16:55:58.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:56.871760+0000 mon.c (mon.2) 101 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:58.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:56.875603+0000 mon.c (mon.2) 102 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:58.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:56.893451+0000 mon.a (mon.0) 920 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:55:58.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:55:57.110667+0000 mon.a (mon.0) 921 : audit [INF] from='client.? v1:172.21.15.33:0/1695292610' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:55:58.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:57.110721+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e165: 8 total, 8 up, 7 in 2024-02-12T16:56:00.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:55:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:55:58.451721+0000 mgr.y (mgr.24481) 243 : cluster [DBG] pgmap v277: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 861 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.38k op/s; 450/41928 objects misplaced (1.073%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:56:00.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:55:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:55:58.451721+0000 mgr.y (mgr.24481) 243 : cluster [DBG] pgmap v277: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 861 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.38k op/s; 450/41928 objects misplaced (1.073%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:56:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:55:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:55:58.451721+0000 mgr.y (mgr.24481) 243 : cluster [DBG] pgmap v277: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 861 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.38k op/s; 450/41928 objects misplaced (1.073%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:56:01.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:56:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:00] "GET /metrics HTTP/1.1" 200 175646 "" "Prometheus/2.33.4" 2024-02-12T16:56:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:00.452296+0000 mgr.y (mgr.24481) 244 : cluster [DBG] pgmap v278: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 739 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.29k op/s; 450/36033 objects misplaced (1.249%); 943 KiB/s, 14 objects/s recovering 2024-02-12T16:56:02.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:00.452296+0000 mgr.y (mgr.24481) 244 : cluster [DBG] pgmap v278: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 739 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.29k op/s; 450/36033 objects misplaced (1.249%); 943 KiB/s, 14 objects/s recovering 2024-02-12T16:56:02.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:00.452296+0000 mgr.y (mgr.24481) 244 : cluster [DBG] pgmap v278: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 739 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.29k op/s; 450/36033 objects misplaced (1.249%); 943 KiB/s, 14 objects/s recovering 2024-02-12T16:56:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:56:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:56:03.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:02.452616+0000 mgr.y (mgr.24481) 245 : cluster [DBG] pgmap v279: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 739 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 627 op/s; 450/36033 objects misplaced (1.249%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:02.473010+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.122332+0000 mon.c (mon.2) 103 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.122665+0000 mon.c (mon.2) 104 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.122856+0000 mon.c (mon.2) 105 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.123044+0000 mon.c (mon.2) 106 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]: dispatch 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.123098+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.124106+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.124243+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.124372+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]: dispatch 2024-02-12T16:56:03.743 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:56:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:02.452616+0000 mgr.y (mgr.24481) 245 : cluster [DBG] pgmap v279: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 739 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 627 op/s; 450/36033 objects misplaced (1.249%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:02.473010+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.122332+0000 mon.c (mon.2) 103 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.122665+0000 mon.c (mon.2) 104 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.122856+0000 mon.c (mon.2) 105 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.123044+0000 mon.c (mon.2) 106 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]: dispatch 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.123098+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.124106+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.124243+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.124372+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]: dispatch 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:02.452616+0000 mgr.y (mgr.24481) 245 : cluster [DBG] pgmap v279: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 739 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 627 op/s; 450/36033 objects misplaced (1.249%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:56:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:02.473010+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:03.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.122332+0000 mon.c (mon.2) 103 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-12T16:56:03.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.122665+0000 mon.c (mon.2) 104 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.122856+0000 mon.c (mon.2) 105 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.123044+0000 mon.c (mon.2) 106 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]: dispatch 2024-02-12T16:56:03.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.123098+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-12T16:56:03.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.124106+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.124243+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-12T16:56:03.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.124372+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]: dispatch 2024-02-12T16:56:04.328 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:04.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:04.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:04.500 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:04.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:04.603 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:04.628 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:04.654 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:04.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:04.765 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:04.860 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:04.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:04.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:05.032 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:05.063 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:05.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:05.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:05.223 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.655648+0000 mgr.y (mgr.24481) 246 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:05.223 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.834059+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]': finished 2024-02-12T16:56:05.223 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.834124+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-02-12T16:56:05.223 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.834167+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-12T16:56:05.223 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:03.834201+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]': finished 2024-02-12T16:56:05.223 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:03.834233+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e166: 8 total, 8 up, 7 in 2024-02-12T16:56:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:05.264 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:05.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.655648+0000 mgr.y (mgr.24481) 246 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.834059+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]': finished 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.834124+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.834167+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:03.834201+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]': finished 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:03.834233+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e166: 8 total, 8 up, 7 in 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.655648+0000 mgr.y (mgr.24481) 246 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.834059+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]': finished 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.834124+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-02-12T16:56:05.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.834167+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-12T16:56:05.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:03.834201+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 0]}]': finished 2024-02-12T16:56:05.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:03.834233+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e166: 8 total, 8 up, 7 in 2024-02-12T16:56:05.376 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:05.413 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:05.424 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T16:56:05.449 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:05.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:05.557 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:05.595 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:05.616 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:05.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:05.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:05.770 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:05.797 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:05.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:05.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:05.947 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:04.453000+0000 mgr.y (mgr.24481) 247 : cluster [DBG] pgmap v281: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 399 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 210/19728 objects misplaced (1.064%); 935 KiB/s, 14 objects/s recovering 2024-02-12T16:56:05.947 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:04.845398+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-12T16:56:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:05.977 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:05.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:06.081 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:06.112 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:04.453000+0000 mgr.y (mgr.24481) 247 : cluster [DBG] pgmap v281: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 399 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 210/19728 objects misplaced (1.064%); 935 KiB/s, 14 objects/s recovering 2024-02-12T16:56:06.112 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:04.845398+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-12T16:56:06.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:04.453000+0000 mgr.y (mgr.24481) 247 : cluster [DBG] pgmap v281: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 399 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 210/19728 objects misplaced (1.064%); 935 KiB/s, 14 objects/s recovering 2024-02-12T16:56:06.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:04.845398+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-12T16:56:06.121 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:06.684 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:06.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:06.788 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:06.834 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:06.872 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:06.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:06.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:07.040 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:07.204 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:06.455153+0000 mgr.y (mgr.24481) 248 : cluster [DBG] pgmap v283: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 399 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s; 210/19728 objects misplaced (1.064%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:56:07.450 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:06.455153+0000 mgr.y (mgr.24481) 248 : cluster [DBG] pgmap v283: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 399 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s; 210/19728 objects misplaced (1.064%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:56:07.450 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:06.455153+0000 mgr.y (mgr.24481) 248 : cluster [DBG] pgmap v283: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 399 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s; 210/19728 objects misplaced (1.064%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:56:07.809 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:07.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:07.918 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:07.963 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:07.984 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:07.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:08.091 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:08.160 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:08.238 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:08.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:08.345 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:08.392 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:56:08.393 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:08.406 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:08.417 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:08.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:08.443 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:56:08.508 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:56:08.527 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:08.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:08.343299+0000 mon.b (mon.1) 61 : audit [DBG] from='client.? v1:172.21.15.33:0/906101538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:08.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:08.343299+0000 mon.b (mon.1) 61 : audit [DBG] from='client.? v1:172.21.15.33:0/906101538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:08.554 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:56:08.563 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:08.593 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:08.596 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:56:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:08.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:08.645 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:56:08.682 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:56:08.700 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:08.732 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:08.343299+0000 mon.b (mon.1) 61 : audit [DBG] from='client.? v1:172.21.15.33:0/906101538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:08.734 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:56:08.737 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:08.756 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:56:08.756 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:56:08.756 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:56:08.756 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:56:08.757 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:56:08.757 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:56:08.757 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:56:08.757 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:56:08.764 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:08.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:08.876 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:08.944 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:09.035 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:09.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:09.143 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:09.217 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:09.468 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:56:09.669 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:08.455873+0000 mgr.y (mgr.24481) 249 : cluster [DBG] pgmap v284: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 200 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 22/10161 objects misplaced (0.217%); 928 KiB/s, 16 objects/s recovering 2024-02-12T16:56:09.669 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:08.611276+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.33:0/3486571680' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:09.669 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:08.611730+0000 mon.a (mon.0) 934 : audit [INF] from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:09.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:08.455873+0000 mgr.y (mgr.24481) 249 : cluster [DBG] pgmap v284: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 200 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 22/10161 objects misplaced (0.217%); 928 KiB/s, 16 objects/s recovering 2024-02-12T16:56:09.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:08.611276+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.33:0/3486571680' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:09.670 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:08.611730+0000 mon.a (mon.0) 934 : audit [INF] from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:08.455873+0000 mgr.y (mgr.24481) 249 : cluster [DBG] pgmap v284: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 200 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 22/10161 objects misplaced (0.217%); 928 KiB/s, 16 objects/s recovering 2024-02-12T16:56:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:08.611276+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.33:0/3486571680' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:08.611730+0000 mon.a (mon.0) 934 : audit [INF] from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:10.508 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:10.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:10.520 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Removed 110449 objects 2024-02-12T16:56:10.520 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Clean up completed and total clean up time :127.467 2024-02-12T16:56:10.616 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:10.646 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:10.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:09.412261+0000 mon.a (mon.0) 935 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:10.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:09.425739+0000 mon.a (mon.0) 936 : audit [INF] from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:56:10.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:09.425815+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-12T16:56:10.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:09.412261+0000 mon.a (mon.0) 935 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:10.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:09.425739+0000 mon.a (mon.0) 936 : audit [INF] from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:56:10.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:09.425815+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-12T16:56:10.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:09.412261+0000 mon.a (mon.0) 935 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:10.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:09.425739+0000 mon.a (mon.0) 936 : audit [INF] from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:56:10.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:09.425815+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-12T16:56:10.877 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:11.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:56:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:10] "GET /metrics HTTP/1.1" 200 175646 "" "Prometheus/2.33.4" 2024-02-12T16:56:11.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:10.429186+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-12T16:56:11.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:10.456452+0000 mgr.y (mgr.24481) 250 : cluster [DBG] pgmap v287: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 82 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 846 op/s; 22/4503 objects misplaced (0.489%); 618 KiB/s, 12 objects/s recovering 2024-02-12T16:56:11.584 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:10.429186+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-12T16:56:11.584 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:10.456452+0000 mgr.y (mgr.24481) 250 : cluster [DBG] pgmap v287: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 82 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 846 op/s; 22/4503 objects misplaced (0.489%); 618 KiB/s, 12 objects/s recovering 2024-02-12T16:56:11.596 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:11.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:11.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:56:11.702 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:10.429186+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-12T16:56:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:10.456452+0000 mgr.y (mgr.24481) 250 : cluster [DBG] pgmap v287: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 82 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 846 op/s; 22/4503 objects misplaced (0.489%); 618 KiB/s, 12 objects/s recovering 2024-02-12T16:56:11.764 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:11.788 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:11.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:11.892 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:11.945 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:11.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:12.048 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:12.078 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:12.163 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:15788 7.b 7:d9e8c0f0:::benchmark_data_smithi033_81060_object15787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:39.096974+0000", 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 213.06256483999999, 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051344805, 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.164 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 15788 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.096974+0000", 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.096974+0000", 2024-02-12T16:56:12.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.096972+0000", 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4181000000000001e-05 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.096996+0000", 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.132e-05 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097007+0000", 2024-02-12T16:56:12.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5610000000000004e-06 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097014+0000", 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8415999999999999e-05 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097062+0000", 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000121964 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097184+0000", 2024-02-12T16:56:12.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000138486 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.097323+0000", 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00095304999999999999 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098276+0000", 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013995699999999999 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098416+0000", 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049831065000000001 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.148247+0000", 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2997999999999997e-05 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.148290+0000", 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.887e-05 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.148318+0000", 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051344805 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:15796 7.b 7:dfc59b0b:::benchmark_data_smithi033_81060_object15795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:39.098972+0000", 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 213.06056616199999, 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051079531999999997, 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.170 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 15796 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098972+0000", 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098972+0000", 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098970+0000", 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4624999999999999e-05 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.098994+0000", 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1532000000000001e-05 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099006+0000", 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7200000000000006e-06 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099014+0000", 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7688999999999997e-05 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099061+0000", 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00035035300000000003 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099412+0000", 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014778 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099559+0000", 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014316909999999999 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.100991+0000", 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101308 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.101092+0000", 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048877324 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.149970+0000", 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7119000000000001e-05 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150017+0000", 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4996000000000002e-05 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.150052+0000", 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051079531999999997 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:53341 7.4 e111/93 7:29052be9:::benchmark_data_smithi033_81060_object53340:head v 111'3313, mlcod=111'3310)", 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:03.100649+0000", 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 189.05888995699999, 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049599338999999999, 2024-02-12T16:56:12.176 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100649+0000", 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100649+0000", 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100648+0000", 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9190000000000001e-05 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100667+0000", 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3700000000000003e-06 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100672+0000", 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1353e-05 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100684+0000", 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9512000000000001e-05 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100713+0000", 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3009e-05 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.100726+0000", 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0065784140000000003 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.107305+0000", 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042943331000000001 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:03.150248+0000", 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049599338999999999 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.180 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:104505 7.b 7:d5eca461:::benchmark_data_smithi033_81060_object104504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:56.081428+0000", 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 136.078110108, 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050079176000000003, 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 104505 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.081428+0000", 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.081428+0000", 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.081427+0000", 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1869999999999999e-05 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.081449+0000", 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5869999999999998e-06 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.081456+0000", 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.403e-06 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.081460+0000", 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003311184 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.084772+0000", 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000160354 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.084932+0000", 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000200509 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.085132+0000", 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012905811999999999 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.098038+0000", 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000480915 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.098519+0000", 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032889017999999999 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.131408+0000", 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5105000000000002e-05 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.131463+0000", 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4329000000000002e-05 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:56.131508+0000", 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050079176000000003 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:155663 7.2 7:434fb205:::benchmark_data_smithi033_81060_object45211:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:54:58.131766+0000", 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 74.027772761999998, 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050914281999999998, 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.186 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 155663 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.131766+0000", 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.131766+0000", 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.131763+0000", 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8519999999999993e-06 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.131772+0000", 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2588999999999999e-05 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.131784+0000", 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8189999999999992e-06 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.131792+0000", 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015093245999999999 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.146885+0000", 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014098100000000001 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.147026+0000", 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017031300000000001 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.189 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.147197+0000", 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003899118 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.151096+0000", 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019310240999999999 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.170406+0000", 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012072634 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.182479+0000", 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1597999999999998e-05 2024-02-12T16:56:12.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.182510+0000", 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0584000000000002e-05 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.182561+0000", 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000119109 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:58.182680+0000", 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050914281999999998 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.191 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:157193 7.2 7:4ad47d45:::benchmark_data_smithi033_81060_object46741:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:00.180504+0000", 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 71.979034291999994, 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048661805000000002, 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 157193 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.192 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.180504+0000", 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.180504+0000", 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.180501+0000", 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8149999999999995e-06 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.180510+0000", 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2563e-05 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.180523+0000", 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6200000000000001e-06 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.180529+0000", 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003302705 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.194 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.183832+0000", 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013178699999999999 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.183964+0000", 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016440899999999999 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.184128+0000", 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057683530000000004 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.189897+0000", 2024-02-12T16:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000251425 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.190148+0000", 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7110999999999998e-05 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.190185+0000", 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038805478999999997 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.228991+0000", 2024-02-12T16:56:12.196 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0815000000000001e-05 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.229042+0000", 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000124463 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.229166+0000", 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048661805000000002 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:157611 7.b 7:d560648b:::benchmark_data_smithi033_81060_object47159:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T16:56:12.197 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:00.794414+0000", 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 71.365124011999995, 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.06143154, 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 157611 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794414+0000", 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.198 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794414+0000", 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794412+0000", 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5130000000000002e-06 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794419+0000", 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1505000000000001e-05 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794430+0000", 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2909999999999998e-06 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794436+0000", 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7688000000000004e-05 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794533+0000", 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8026000000000001e-05 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794611+0000", 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1385999999999996e-05 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.794693+0000", 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002464122 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.797157+0000", 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050080668000000002 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.847238+0000", 2024-02-12T16:56:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085133480000000004 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.855751+0000", 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1585000000000002e-05 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.855803+0000", 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3514999999999999e-05 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.855846+0000", 2024-02-12T16:56:12.202 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.06143154 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:158325 7.b 7:dfb23831:::benchmark_data_smithi033_81060_object47873:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:01.719529+0000", 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 70.440009575000005, 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051707815999999997, 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.203 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 158325 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719529+0000", 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719529+0000", 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.204 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719527+0000", 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5220000000000004e-06 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719534+0000", 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6500000000000008e-06 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719543+0000", 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7849999999999997e-06 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.205 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719549+0000", 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1891999999999994e-05 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719641+0000", 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104769 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719746+0000", 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013250600000000001 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.206 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.719878+0000", 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021539390000000001 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.722032+0000", 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9500000000000007e-06 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.722041+0000", 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049109432000000001 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.771150+0000", 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7828999999999999e-05 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.771198+0000", 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8423000000000001e-05 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.771237+0000", 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051707815999999997 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.208 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:158327 7.b 7:de392ee9:::benchmark_data_smithi033_81060_object47875:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:01.721738+0000", 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 70.437800945999996, 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049568912999999999, 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 158327 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.209 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.721738+0000", 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.721738+0000", 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.721736+0000", 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.063e-06 2024-02-12T16:56:12.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.721743+0000", 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0659999999999993e-06 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.721752+0000", 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7039999999999999e-06 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.721757+0000", 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000308871 2024-02-12T16:56:12.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.722066+0000", 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000149968 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.722216+0000", 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000152726 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.722369+0000", 2024-02-12T16:56:12.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029557369999999999 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.725325+0000", 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001332529 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.726657+0000", 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044608859000000001 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.771266+0000", 2024-02-12T16:56:12.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1429000000000001e-05 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.771288+0000", 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8899e-05 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.771306+0000", 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049568912999999999 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:159454 7.2 7:4d9dd8fb:::benchmark_data_smithi033_81060_object49002:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:03.385815+0000", 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 68.773723798000006, 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050406205000000003, 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 159454 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.215 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.385815+0000", 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.385815+0000", 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.385813+0000", 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7710000000000001e-06 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.385820+0000", 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.789e-06 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.385828+0000", 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7999999999999998e-06 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.385833+0000", 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037351799999999998 2024-02-12T16:56:12.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.389568+0000", 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000145523 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.389714+0000", 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015552799999999999 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.389869+0000", 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0077231660000000001 2024-02-12T16:56:12.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.397592+0000", 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051158510000000003 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.402708+0000", 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087085440000000004 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.411417+0000", 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024616434999999999 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.436033+0000", 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5040000000000002e-05 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.436088+0000", 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000132533 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.436221+0000", 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050406205000000003 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.220 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:159581 7.2 7:4c668a50:::benchmark_data_smithi033_81060_object49129:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:03.565774+0000", 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 68.593764078000007, 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064772073999999999, 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 159581 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.565774+0000", 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.565774+0000", 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.565772+0000", 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9709999999999998e-06 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.565779+0000", 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2579999999999998e-06 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.565787+0000", 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2830000000000003e-06 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.565791+0000", 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011086554 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.576877+0000", 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013426000000000001 2024-02-12T16:56:12.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.577012+0000", 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016576399999999999 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.577177+0000", 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003905033 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.581083+0000", 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026258969999999999 2024-02-12T16:56:12.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.583708+0000", 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015014718999999999 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.598723+0000", 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031573093000000003 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.630296+0000", 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6166000000000002e-05 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.630372+0000", 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000174086 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.630546+0000", 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064772073999999999 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:159759 7.2 7:42992679:::benchmark_data_smithi033_81060_object49307:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:03.825801+0000", 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 68.333737181000004, 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052902668, 2024-02-12T16:56:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 159759 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.825801+0000", 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.825801+0000", 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:56:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.825798+0000", 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7199999999999995e-06 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.825807+0000", 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3839000000000001e-05 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.825821+0000", 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3280000000000002e-06 2024-02-12T16:56:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.825827+0000", 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028275499999999999 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.828655+0000", 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000134444 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.828789+0000", 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00045219200000000002 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.829241+0000", 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074604160000000001 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.836702+0000", 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006839581 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.843541+0000", 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031165393 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.874707+0000", 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003797239 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.878504+0000", 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7843999999999997e-05 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.878562+0000", 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000142037 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:03.878704+0000", 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052902668 2024-02-12T16:56:12.231 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:160439 7.2 7:4cd7602e:::benchmark_data_smithi033_81060_object49987:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:04.837539+0000", 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 67.321999986999998, 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049747013999999999, 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 160439 2024-02-12T16:56:12.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.837539+0000", 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.837539+0000", 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.837537+0000", 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9740000000000001e-06 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.837542+0000", 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3209999999999999e-06 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.837549+0000", 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1489999999999996e-06 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.837553+0000", 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037220650000000001 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.841275+0000", 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017122400000000001 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.841447+0000", 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017805699999999999 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.841625+0000", 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039389949999999998 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.845564+0000", 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000194816 2024-02-12T16:56:12.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.845758+0000", 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053007619999999997 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.851059+0000", 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036015237999999998 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.887074+0000", 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6456000000000002e-05 2024-02-12T16:56:12.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.887141+0000", 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000144621 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.887286+0000", 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049747013999999999 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161372 7.2 7:4db6b529:::benchmark_data_smithi033_81060_object50920:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.128645+0000", 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 66.030893528999997, 2024-02-12T16:56:12.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050060768999999998, 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161372 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.128645+0000", 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.128645+0000", 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999967 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.128642+0000", 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0695999999999999e-05 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.128672+0000", 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6235e-05 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.128689+0000", 2024-02-12T16:56:12.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2609999999999996e-06 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.128696+0000", 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091098410000000005 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.137806+0000", 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000154993 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.137961+0000", 2024-02-12T16:56:12.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019462900000000001 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.138155+0000", 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054684180000000001 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.143624+0000", 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.033044389 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.176668+0000", 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018444150000000001 2024-02-12T16:56:12.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.178513+0000", 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0421e-05 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.178533+0000", 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3458999999999998e-05 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.178586+0000", 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000119345 2024-02-12T16:56:12.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.178706+0000", 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050060768999999998 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161841 7.2 7:4a5a1512:::benchmark_data_smithi033_81060_object51389:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.763642+0000", 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 65.395896307000001, 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051389602999999999, 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.243 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161841 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763642+0000", 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763642+0000", 2024-02-12T16:56:12.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763642+0000", 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0429999999999999e-06 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763645+0000", 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6330000000000001e-06 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763650+0000", 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3420000000000001e-06 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763655+0000", 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044658600000000003 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.768120+0000", 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015231700000000001 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.768273+0000", 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000195591 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.768468+0000", 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041879500000000002 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.772656+0000", 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084515419999999994 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.781108+0000", 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004914865 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.786023+0000", 2024-02-12T16:56:12.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028806497 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.814829+0000", 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1038999999999996e-05 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.814890+0000", 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014159200000000001 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.815032+0000", 2024-02-12T16:56:12.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051389602999999999 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161843 7.2 7:44935340:::benchmark_data_smithi033_81060_object51391:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.763659+0000", 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 65.395879828000005, 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064849663000000002, 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161843 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.249 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763659+0000", 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763659+0000", 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763658+0000", 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5979999999999999e-06 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763661+0000", 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5230000000000003e-06 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763666+0000", 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.07e-06 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.763670+0000", 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067300820000000001 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.770400+0000", 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5377000000000005e-05 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.770496+0000", 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8064000000000002e-05 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.770594+0000", 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020712790000000001 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.772665+0000", 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086301590000000001 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.781295+0000", 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047604609999999997 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.786056+0000", 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042263712000000002 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.828319+0000", 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.402e-05 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.828383+0000", 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000124864 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.828508+0000", 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064849663000000002 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.254 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161850 7.2 7:48451835:::benchmark_data_smithi033_81060_object51398:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.777606+0000", 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 65.381932183000004, 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051160895999999997, 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161850 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.777606+0000", 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.777606+0000", 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.777605+0000", 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1880000000000001e-06 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.777610+0000", 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7110000000000001e-06 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.777618+0000", 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.921e-06 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.777623+0000", 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00091643400000000004 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.778540+0000", 2024-02-12T16:56:12.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013378199999999999 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.778674+0000", 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000168553 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.778842+0000", 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035548200000000002 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.782397+0000", 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012539699999999999 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.782522+0000", 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082780079999999999 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.790800+0000", 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037764680000000002 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.828565+0000", 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1545999999999997e-05 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.828627+0000", 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000140572 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.828767+0000", 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051160895999999997 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:162741 7.2 7:4d165361:::benchmark_data_smithi033_81060_object52289:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.260 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:07.920578+0000", 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 64.238960863000003, 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052294119999999999, 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 162741 2024-02-12T16:56:12.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920578+0000", 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920578+0000", 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920575+0000", 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8959999999999998e-06 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920584+0000", 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4222e-05 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920599+0000", 2024-02-12T16:56:12.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2120000000000003e-06 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920607+0000", 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042928719999999997 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.924900+0000", 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012006 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.925020+0000", 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014663799999999999 2024-02-12T16:56:12.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.925166+0000", 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051373219999999997 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.930304+0000", 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023815300000000001 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.930542+0000", 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049549429999999998 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.935497+0000", 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037196845999999999 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.972694+0000", 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3749000000000003e-05 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.972747+0000", 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012430199999999999 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.972872+0000", 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052294119999999999 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:162744 7.2 7:4d40efc1:::benchmark_data_smithi033_81060_object52292:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:07.920619+0000", 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 64.238919949999996, 2024-02-12T16:56:12.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052450765000000003, 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 162744 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920619+0000", 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920619+0000", 2024-02-12T16:56:12.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920618+0000", 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.015e-06 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920624+0000", 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7669999999999998e-06 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920630+0000", 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1200000000000001e-06 2024-02-12T16:56:12.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.920636+0000", 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073788669999999999 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.928014+0000", 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3947000000000001e-05 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.928098+0000", 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7320999999999999e-05 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.928196+0000", 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036419180000000001 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.931838+0000", 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3711999999999999e-05 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.931861+0000", 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037293069999999998 2024-02-12T16:56:12.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.935591+0000", 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037317802999999997 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.972908+0000", 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6269e-05 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.972955+0000", 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114559 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:07.973069+0000", 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052450765000000003 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:163233 7.2 7:484685cd:::benchmark_data_smithi033_81060_object52781:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:08.568020+0000", 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 63.591518037999997, 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048105837999999998, 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:12.274 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 163233 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.568020+0000", 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.568020+0000", 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.568018+0000", 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9088000000000001e-05 2024-02-12T16:56:12.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.568037+0000", 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.453e-05 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.568052+0000", 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3260000000000001e-06 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.568058+0000", 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013648914999999999 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.581707+0000", 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000130424 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.581837+0000", 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00039048400000000002 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.582228+0000", 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014285867000000001 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.596514+0000", 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2843000000000001e-05 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.596527+0000", 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00063988700000000001 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.597167+0000", 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018795037000000001 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.615962+0000", 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9295000000000002e-05 2024-02-12T16:56:12.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.616001+0000", 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012545899999999999 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.616126+0000", 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048105837999999998 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:12.279 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:56:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:12.281 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:12.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:12.381 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-02-12T16:56:12.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-02-12T16:56:12.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:12.414 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:12.436 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:12.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:12.451 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:56:12.540 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:12.569 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:12.589 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:12.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:12.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:12.719 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:56:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:56:12.721 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:12.740 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:12.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:12.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:12.872 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:12.971 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:12.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:12.992 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:56:13.075 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:13.103 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:13.156 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:13.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:13.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:13.293 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:13.315 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:13.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:13.419 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:13.448 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:13.468 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:13.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:13.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:12.456793+0000 mgr.y (mgr.24481) 251 : cluster [DBG] pgmap v288: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 82 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 672 B/s rd, 0 B/s wr, 667 op/s; 22/4503 objects misplaced (0.489%); 488 KiB/s, 9 objects/s recovering 2024-02-12T16:56:13.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:12.456793+0000 mgr.y (mgr.24481) 251 : cluster [DBG] pgmap v288: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 82 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 672 B/s rd, 0 B/s wr, 667 op/s; 22/4503 objects misplaced (0.489%); 488 KiB/s, 9 objects/s recovering 2024-02-12T16:56:13.572 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:13.598 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:12.456793+0000 mgr.y (mgr.24481) 251 : cluster [DBG] pgmap v288: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 82 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 672 B/s rd, 0 B/s wr, 667 op/s; 22/4503 objects misplaced (0.489%); 488 KiB/s, 9 objects/s recovering 2024-02-12T16:56:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:13.619 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:13.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:13.723 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:13.751 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:13.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:56:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:56:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:14.046 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:14.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:14.405 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:14.435 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:14.543 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:13.662607+0000 mgr.y (mgr.24481) 252 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:14.544 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:13.662607+0000 mgr.y (mgr.24481) 252 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:14.814 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:13.662607+0000 mgr.y (mgr.24481) 252 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:15.527 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T16:56:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:13.722345+0000 osd.6 (osd.6) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:13.722419+0000 osd.6 (osd.6) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:14.457413+0000 mgr.y (mgr.24481) 253 : cluster [DBG] pgmap v289: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 799 op/s; 552 KiB/s, 10 objects/s recovering 2024-02-12T16:56:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:14.521671+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-12T16:56:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:14.980584+0000 mon.a (mon.0) 940 : audit [INF] from='client.? v1:172.21.15.33:0/3530280552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:56:15.926 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:13.722345+0000 osd.6 (osd.6) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:15.926 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:13.722419+0000 osd.6 (osd.6) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:15.926 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:14.457413+0000 mgr.y (mgr.24481) 253 : cluster [DBG] pgmap v289: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 799 op/s; 552 KiB/s, 10 objects/s recovering 2024-02-12T16:56:15.926 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:14.521671+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-12T16:56:15.926 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:14.980584+0000 mon.a (mon.0) 940 : audit [INF] from='client.? v1:172.21.15.33:0/3530280552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:56:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:13.722345+0000 osd.6 (osd.6) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:13.722419+0000 osd.6 (osd.6) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:14.457413+0000 mgr.y (mgr.24481) 253 : cluster [DBG] pgmap v289: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 799 op/s; 552 KiB/s, 10 objects/s recovering 2024-02-12T16:56:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:14.521671+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-12T16:56:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:14.980584+0000 mon.a (mon.0) 940 : audit [INF] from='client.? v1:172.21.15.33:0/3530280552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:56:16.469 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T16:56:16.469 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:56:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:15.522594+0000 mon.a (mon.0) 941 : audit [INF] from='client.? v1:172.21.15.33:0/3530280552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:56:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:15.522652+0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-12T16:56:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:15.522594+0000 mon.a (mon.0) 941 : audit [INF] from='client.? v1:172.21.15.33:0/3530280552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:56:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:15.522652+0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-12T16:56:16.893 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:16.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:15.522594+0000 mon.a (mon.0) 941 : audit [INF] from='client.? v1:172.21.15.33:0/3530280552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:56:16.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:15.522652+0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-12T16:56:17.538 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:56:17.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:16.457766+0000 mgr.y (mgr.24481) 254 : cluster [DBG] pgmap v292: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 848 B/s rd, 0 B/s wr, 217 op/s; 117 KiB/s, 1 objects/s recovering 2024-02-12T16:56:17.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:17.096736+0000 mon.a (mon.0) 943 : audit [INF] from='client.? v1:172.21.15.33:0/608399390' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:56:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:16.457766+0000 mgr.y (mgr.24481) 254 : cluster [DBG] pgmap v292: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 848 B/s rd, 0 B/s wr, 217 op/s; 117 KiB/s, 1 objects/s recovering 2024-02-12T16:56:17.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:17.096736+0000 mon.a (mon.0) 943 : audit [INF] from='client.? v1:172.21.15.33:0/608399390' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:56:17.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:16.457766+0000 mgr.y (mgr.24481) 254 : cluster [DBG] pgmap v292: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 848 B/s rd, 0 B/s wr, 217 op/s; 117 KiB/s, 1 objects/s recovering 2024-02-12T16:56:17.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:17.096736+0000 mon.a (mon.0) 943 : audit [INF] from='client.? v1:172.21.15.33:0/608399390' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:56:18.288 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:18.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:18.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:18.425 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:18.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:18.542 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:17.527750+0000 mon.a (mon.0) 944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:18.542 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:17.532459+0000 mon.a (mon.0) 945 : audit [INF] from='client.? v1:172.21.15.33:0/608399390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:56:18.542 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:17.532510+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-12T16:56:18.542 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:17.617197+0000 osd.2 (osd.2) 91 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:56:18.542 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:17.626286+0000 osd.2 (osd.2) 92 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:56:18.542 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:18.161218+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:56:18.542 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:18.162184+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:56:18.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:18.580 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:18.600 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:18.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:18.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:18.730 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:17.527750+0000 mon.a (mon.0) 944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:18.730 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:17.532459+0000 mon.a (mon.0) 945 : audit [INF] from='client.? v1:172.21.15.33:0/608399390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:56:18.730 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:17.532510+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-12T16:56:18.730 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:17.617197+0000 osd.2 (osd.2) 91 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:56:18.730 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:17.626286+0000 osd.2 (osd.2) 92 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:56:18.730 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:18.161218+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:56:18.730 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:18.162184+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:56:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:18.751 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:18.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:18.855 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:18.877 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:17.527750+0000 mon.a (mon.0) 944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:18.877 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:17.532459+0000 mon.a (mon.0) 945 : audit [INF] from='client.? v1:172.21.15.33:0/608399390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:56:18.877 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:17.532510+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-12T16:56:18.877 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:17.617197+0000 osd.2 (osd.2) 91 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:56:18.877 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:17.626286+0000 osd.2 (osd.2) 92 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:56:18.877 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:18.161218+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:56:18.877 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:18.162184+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:56:18.886 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:18.458341+0000 mgr.y (mgr.24481) 255 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 218 op/s; 117 KiB/s, 1 objects/s recovering 2024-02-12T16:56:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:18.458993+0000 osd.4 (osd.4) 79 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:56:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:18.464221+0000 osd.4 (osd.4) 80 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:56:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:18.627657+0000 osd.2 (osd.2) 93 : cluster [DBG] 7.a scrub starts 2024-02-12T16:56:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:18.644616+0000 osd.2 (osd.2) 94 : cluster [DBG] 7.a scrub ok 2024-02-12T16:56:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:18.458341+0000 mgr.y (mgr.24481) 255 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 218 op/s; 117 KiB/s, 1 objects/s recovering 2024-02-12T16:56:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:18.458993+0000 osd.4 (osd.4) 79 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:56:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:18.464221+0000 osd.4 (osd.4) 80 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:56:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:18.627657+0000 osd.2 (osd.2) 93 : cluster [DBG] 7.a scrub starts 2024-02-12T16:56:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:18.644616+0000 osd.2 (osd.2) 94 : cluster [DBG] 7.a scrub ok 2024-02-12T16:56:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:18.458341+0000 mgr.y (mgr.24481) 255 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 218 op/s; 117 KiB/s, 1 objects/s recovering 2024-02-12T16:56:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:18.458993+0000 osd.4 (osd.4) 79 : cluster [DBG] 6.1e scrub starts 2024-02-12T16:56:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:18.464221+0000 osd.4 (osd.4) 80 : cluster [DBG] 6.1e scrub ok 2024-02-12T16:56:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:18.627657+0000 osd.2 (osd.2) 93 : cluster [DBG] 7.a scrub starts 2024-02-12T16:56:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:18.644616+0000 osd.2 (osd.2) 94 : cluster [DBG] 7.a scrub ok 2024-02-12T16:56:20.612 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:20.880 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:19.481632+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:56:20.880 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:19.484700+0000 osd.4 (osd.4) 82 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:56:20.880 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:19.481632+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:56:20.881 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:19.484700+0000 osd.4 (osd.4) 82 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:56:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:19.481632+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:56:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:19.484700+0000 osd.4 (osd.4) 82 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:56:21.038 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:21.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:21.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:21.143 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:21.192 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:21.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:21.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:21.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:56:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:20] "GET /metrics HTTP/1.1" 200 175652 "" "Prometheus/2.33.4" 2024-02-12T16:56:21.295 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:21.323 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:21.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:21.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:21.474 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:21.493 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:21.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:21.564 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:20.301609+0000 osd.1 (osd.1) 114 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:56:21.564 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:20.316577+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.2 scrub ok 2024-02-12T16:56:21.564 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:20.459053+0000 mgr.y (mgr.24481) 256 : cluster [DBG] pgmap v295: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:20.675617+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:20.676623+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:20.825980+0000 osd.7 (osd.7) 87 : cluster [DBG] 5.14 scrub starts 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:20.829597+0000 osd.7 (osd.7) 88 : cluster [DBG] 5.14 scrub ok 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:20.301609+0000 osd.1 (osd.1) 114 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:20.316577+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.2 scrub ok 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:20.459053+0000 mgr.y (mgr.24481) 256 : cluster [DBG] pgmap v295: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:20.675617+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:20.676623+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:56:21.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:20.825980+0000 osd.7 (osd.7) 87 : cluster [DBG] 5.14 scrub starts 2024-02-12T16:56:21.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:20.829597+0000 osd.7 (osd.7) 88 : cluster [DBG] 5.14 scrub ok 2024-02-12T16:56:21.598 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:21.629 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:20.301609+0000 osd.1 (osd.1) 114 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:56:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:20.316577+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.2 scrub ok 2024-02-12T16:56:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:20.459053+0000 mgr.y (mgr.24481) 256 : cluster [DBG] pgmap v295: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:20.675617+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-02-12T16:56:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:20.676623+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-02-12T16:56:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:20.825980+0000 osd.7 (osd.7) 87 : cluster [DBG] 5.14 scrub starts 2024-02-12T16:56:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:20.829597+0000 osd.7 (osd.7) 88 : cluster [DBG] 5.14 scrub ok 2024-02-12T16:56:21.793 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:56:21.810 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:56:21.830 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:56:21.848 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:56:21.866 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:56:21.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:56:21.902 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:56:21.920 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:56:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:56:21.921 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:56:21.922 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:56:21.922 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:56:21.922 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:56:21.922 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:56:21.922 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:56:21.922 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:56:22.075 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:56:22.408 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:22.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:22.511 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:22.544 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:22.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:21.697712+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:56:22.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:21.699620+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:56:22.622 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:21.774903+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.33:0/2787245215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:22.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:21.697712+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:56:22.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:21.699620+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:56:22.622 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:21.774903+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.33:0/2787245215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:22.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:56:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:56:22.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:21.697712+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.19 scrub starts 2024-02-12T16:56:22.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:21.699620+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.19 scrub ok 2024-02-12T16:56:22.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:21.774903+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.33:0/2787245215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:23.038 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:23.190 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:23.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:23.294 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:23.323 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:23.739 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_0' removed 2024-02-12T16:56:23.773 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:23.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:23.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:23.885 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:23.914 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:23.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:56:23.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:22.459407+0000 mgr.y (mgr.24481) 257 : cluster [DBG] pgmap v296: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 644 B/s rd, 0 op/s 2024-02-12T16:56:23.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:22.679859+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:23.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:22.698482+0000 osd.2 (osd.2) 99 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:56:23.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:22.699328+0000 osd.2 (osd.2) 100 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:56:23.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:22.716564+0000 mon.c (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.33:0/720325591' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:56:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:22.716912+0000 mon.a (mon.0) 948 : audit [INF] from='client.34979 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:56:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:23.332860+0000 osd.6 (osd.6) 92 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:56:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:23.345712+0000 osd.6 (osd.6) 93 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:56:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:22.459407+0000 mgr.y (mgr.24481) 257 : cluster [DBG] pgmap v296: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 644 B/s rd, 0 op/s 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:22.679859+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:22.698482+0000 osd.2 (osd.2) 99 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:22.699328+0000 osd.2 (osd.2) 100 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:22.716564+0000 mon.c (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.33:0/720325591' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:22.716912+0000 mon.a (mon.0) 948 : audit [INF] from='client.34979 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:23.332860+0000 osd.6 (osd.6) 92 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:23.345712+0000 osd.6 (osd.6) 93 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:22.459407+0000 mgr.y (mgr.24481) 257 : cluster [DBG] pgmap v296: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 644 B/s rd, 0 op/s 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:22.679859+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:22.698482+0000 osd.2 (osd.2) 99 : cluster [DBG] 2.10 scrub starts 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:22.699328+0000 osd.2 (osd.2) 100 : cluster [DBG] 2.10 scrub ok 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:22.716564+0000 mon.c (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.33:0/720325591' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:56:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:22.716912+0000 mon.a (mon.0) 948 : audit [INF] from='client.34979 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:56:24.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:23.332860+0000 osd.6 (osd.6) 92 : cluster [DBG] 6.1f scrub starts 2024-02-12T16:56:24.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:23.345712+0000 osd.6 (osd.6) 93 : cluster [DBG] 6.1f scrub ok 2024-02-12T16:56:24.658 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:56:24.658 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:56:24.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:23.671585+0000 mgr.y (mgr.24481) 258 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:24.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:23.682164+0000 mon.a (mon.0) 949 : audit [INF] from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:56:24.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:23.682226+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-12T16:56:24.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:23.789808+0000 osd.4 (osd.4) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:24.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:23.789887+0000 osd.4 (osd.4) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:24.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:23.896113+0000 osd.0 (osd.0) 83 : cluster [DBG] 5.13 scrub starts 2024-02-12T16:56:24.708 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:23.899131+0000 osd.0 (osd.0) 84 : cluster [DBG] 5.13 scrub ok 2024-02-12T16:56:24.708 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:24.207068+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.1b scrub starts 2024-02-12T16:56:24.708 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:24.210071+0000 osd.5 (osd.5) 87 : cluster [DBG] 5.1b scrub ok 2024-02-12T16:56:24.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:23.671585+0000 mgr.y (mgr.24481) 258 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:24.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:23.682164+0000 mon.a (mon.0) 949 : audit [INF] from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:56:24.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:23.682226+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-12T16:56:24.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:23.789808+0000 osd.4 (osd.4) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:24.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:23.789887+0000 osd.4 (osd.4) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:24.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:23.896113+0000 osd.0 (osd.0) 83 : cluster [DBG] 5.13 scrub starts 2024-02-12T16:56:24.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:23.899131+0000 osd.0 (osd.0) 84 : cluster [DBG] 5.13 scrub ok 2024-02-12T16:56:24.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:24.207068+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.1b scrub starts 2024-02-12T16:56:24.981 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:24.210071+0000 osd.5 (osd.5) 87 : cluster [DBG] 5.1b scrub ok 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:23.671585+0000 mgr.y (mgr.24481) 258 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:23.682164+0000 mon.a (mon.0) 949 : audit [INF] from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:23.682226+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:23.789808+0000 osd.4 (osd.4) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:23.789887+0000 osd.4 (osd.4) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:23.896113+0000 osd.0 (osd.0) 83 : cluster [DBG] 5.13 scrub starts 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:23.899131+0000 osd.0 (osd.0) 84 : cluster [DBG] 5.13 scrub ok 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:24.207068+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.1b scrub starts 2024-02-12T16:56:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:24.210071+0000 osd.5 (osd.5) 87 : cluster [DBG] 5.1b scrub ok 2024-02-12T16:56:25.163 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:25.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:25.268 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:25.297 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:25.547 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:25.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:25.558 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:56:25.651 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:25.678 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:25.802 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:25.803 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:24.459955+0000 mgr.y (mgr.24481) 259 : cluster [DBG] pgmap v298: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:25.803 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:24.707626+0000 mon.a (mon.0) 951 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-12T16:56:25.803 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:24.850540+0000 osd.7 (osd.7) 89 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:56:25.803 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:24.854528+0000 osd.7 (osd.7) 90 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:56:25.803 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:25.224638+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:56:25.803 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:25.227582+0000 osd.5 (osd.5) 89 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:25.339016+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:25.339744+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:24.459955+0000 mgr.y (mgr.24481) 259 : cluster [DBG] pgmap v298: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:24.707626+0000 mon.a (mon.0) 951 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:24.850540+0000 osd.7 (osd.7) 89 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:24.854528+0000 osd.7 (osd.7) 90 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:25.224638+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:25.227582+0000 osd.5 (osd.5) 89 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:56:25.804 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:25.339016+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:56:25.805 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:25.339744+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:56:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:25.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:25.919 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:25.946 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:26.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:24.459955+0000 mgr.y (mgr.24481) 259 : cluster [DBG] pgmap v298: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:26.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:24.707626+0000 mon.a (mon.0) 951 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-12T16:56:26.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:24.850540+0000 osd.7 (osd.7) 89 : cluster [DBG] 4.11 scrub starts 2024-02-12T16:56:26.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:24.854528+0000 osd.7 (osd.7) 90 : cluster [DBG] 4.11 scrub ok 2024-02-12T16:56:26.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:25.224638+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.2 scrub starts 2024-02-12T16:56:26.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:25.227582+0000 osd.5 (osd.5) 89 : cluster [DBG] 2.2 scrub ok 2024-02-12T16:56:26.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:25.339016+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.1a scrub starts 2024-02-12T16:56:26.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:25.339744+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.1a scrub ok 2024-02-12T16:56:26.079 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:56:26.081 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:56:26.272 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:56:26.273 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:56:26.273 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-02-12T16:56:26.273 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-02-12T16:56:26.306 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:56:27.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:26.016896+0000 mgr.y (mgr.24481) 260 : audit [DBG] from='client.34985 v1:172.21.15.33:0/4159692869' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:56:27.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:26.336506+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:56:27.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:26.337450+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:56:27.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:26.460324+0000 mgr.y (mgr.24481) 261 : cluster [DBG] pgmap v300: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:26.016896+0000 mgr.y (mgr.24481) 260 : audit [DBG] from='client.34985 v1:172.21.15.33:0/4159692869' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:56:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:26.336506+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:56:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:26.337450+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:56:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:26.460324+0000 mgr.y (mgr.24481) 261 : cluster [DBG] pgmap v300: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:27.374 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-02-12T16:56:24.705026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":1104,"num_read_kb":856,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":3321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":495,"ondisk_log_size":495,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2118840,"kb_used_data":5504,"kb_used_omap":0,"kb_used_meta":2113280,"kb_avail":654097224,"statfs":{"total":671965249536,"available":669795557376,"internally_reserved":0,"allocated":5636096,"data_stored":2874151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2163998720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-220900,"num_write_kb":-7068737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-31486,"num_bytes_recovered":-2063138816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.174990"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976449+0000","last_change":"2024-02-12T16:54:40.643927+0000","last_active":"2024-02-12T16:56:15.976449+0000","last_peered":"2024-02-12T16:56:15.976449+0000","last_clean":"2024-02-12T16:56:15.976449+0000","last_became_active":"2024-02-12T16:54:40.643598+0000","last_became_peered":"2024-02-12T16:54:40.643598+0000","last_unstale":"2024-02-12T16:56:15.976449+0000","last_undegraded":"2024-02-12T16:56:15.976449+0000","last_fullsized":"2024-02-12T16:56:15.976449+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:45.052938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:51:45.052938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":96,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976544+0000","last_change":"2024-02-12T16:53:49.715871+0000","last_active":"2024-02-12T16:56:15.976544+0000","last_peered":"2024-02-12T16:56:15.976544+0000","last_clean":"2024-02-12T16:56:15.976544+0000","last_became_active":"2024-02-12T16:52:20.563961+0000","last_became_peered":"2024-02-12T16:52:20.563961+0000","last_unstale":"2024-02-12T16:56:15.976544+0000","last_undegraded":"2024-02-12T16:56:15.976544+0000","last_fullsized":"2024-02-12T16:56:15.976544+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:49.715809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.715809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":131,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976264+0000","last_change":"2024-02-12T16:52:41.990018+0000","last_active":"2024-02-12T16:56:15.976264+0000","last_peered":"2024-02-12T16:56:15.976264+0000","last_clean":"2024-02-12T16:56:15.976264+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:56:15.976264+0000","last_undegraded":"2024-02-12T16:56:15.976264+0000","last_fullsized":"2024-02-12T16:56:15.976264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:41.989918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.989918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":104,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:21.699700+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T16:56:21.699700+0000","last_peered":"2024-02-12T16:56:21.699700+0000","last_clean":"2024-02-12T16:56:21.699700+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T16:56:21.699700+0000","last_undegraded":"2024-02-12T16:56:21.699700+0000","last_fullsized":"2024-02-12T16:56:21.699700+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":91,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978623+0000","last_change":"2024-02-12T16:53:44.946159+0000","last_active":"2024-02-12T16:56:15.978623+0000","last_peered":"2024-02-12T16:56:15.978623+0000","last_clean":"2024-02-12T16:56:15.978623+0000","last_became_active":"2024-02-12T16:52:20.561617+0000","last_became_peered":"2024-02-12T16:52:20.561617+0000","last_unstale":"2024-02-12T16:56:15.978623+0000","last_undegraded":"2024-02-12T16:56:15.978623+0000","last_fullsized":"2024-02-12T16:56:15.978623+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":129,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521270+0000","last_change":"2024-02-12T16:52:58.817066+0000","last_active":"2024-02-12T16:56:15.521270+0000","last_peered":"2024-02-12T16:56:15.521270+0000","last_clean":"2024-02-12T16:56:15.521270+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:56:15.521270+0000","last_undegraded":"2024-02-12T16:56:15.521270+0000","last_fullsized":"2024-02-12T16:56:15.521270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:58.816871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.816871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976874+0000","last_change":"2024-02-12T16:54:40.642865+0000","last_active":"2024-02-12T16:56:15.976874+0000","last_peered":"2024-02-12T16:56:15.976874+0000","last_clean":"2024-02-12T16:56:15.976874+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T16:56:15.976874+0000","last_undegraded":"2024-02-12T16:56:15.976874+0000","last_fullsized":"2024-02-12T16:56:15.976874+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:38.073441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:51:38.073441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":154,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521729+0000","last_change":"2024-02-12T16:55:33.609901+0000","last_active":"2024-02-12T16:56:15.521729+0000","last_peered":"2024-02-12T16:56:15.521729+0000","last_clean":"2024-02-12T16:56:15.521729+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:56:15.521729+0000","last_undegraded":"2024-02-12T16:56:15.521729+0000","last_fullsized":"2024-02-12T16:56:15.521729+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:55:33.609834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:33.609834+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":136,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977242+0000","last_change":"2024-02-12T16:54:40.628366+0000","last_active":"2024-02-12T16:56:15.977242+0000","last_peered":"2024-02-12T16:56:15.977242+0000","last_clean":"2024-02-12T16:56:15.977242+0000","last_became_active":"2024-02-12T16:54:40.628037+0000","last_became_peered":"2024-02-12T16:54:40.628037+0000","last_unstale":"2024-02-12T16:56:15.977242+0000","last_undegraded":"2024-02-12T16:56:15.977242+0000","last_fullsized":"2024-02-12T16:56:15.977242+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:43.089004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.089004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977063+0000","last_change":"2024-02-12T16:52:16.185645+0000","last_active":"2024-02-12T16:56:15.977063+0000","last_peered":"2024-02-12T16:56:15.977063+0000","last_clean":"2024-02-12T16:56:15.977063+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:56:15.977063+0000","last_undegraded":"2024-02-12T16:56:15.977063+0000","last_fullsized":"2024-02-12T16:56:15.977063+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:16.185582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:52:16.185582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":144,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977579+0000","last_change":"2024-02-12T16:54:38.784259+0000","last_active":"2024-02-12T16:56:15.977579+0000","last_peered":"2024-02-12T16:56:15.977579+0000","last_clean":"2024-02-12T16:56:15.977579+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:56:15.977579+0000","last_undegraded":"2024-02-12T16:56:15.977579+0000","last_fullsized":"2024-02-12T16:56:15.977579+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":74,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521455+0000","last_change":"2024-02-12T16:54:41.193544+0000","last_active":"2024-02-12T16:56:15.521455+0000","last_peered":"2024-02-12T16:56:15.521455+0000","last_clean":"2024-02-12T16:56:15.521455+0000","last_became_active":"2024-02-12T16:54:40.639106+0000","last_became_peered":"2024-02-12T16:54:40.639106+0000","last_unstale":"2024-02-12T16:56:15.521455+0000","last_undegraded":"2024-02-12T16:56:15.521455+0000","last_fullsized":"2024-02-12T16:56:15.521455+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:53:45.658292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.658292+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976244+0000","last_change":"2024-02-12T16:53:51.204401+0000","last_active":"2024-02-12T16:56:15.976244+0000","last_peered":"2024-02-12T16:56:15.976244+0000","last_clean":"2024-02-12T16:56:15.976244+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:56:15.976244+0000","last_undegraded":"2024-02-12T16:56:15.976244+0000","last_fullsized":"2024-02-12T16:56:15.976244+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":129,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976699+0000","last_change":"2024-02-12T16:52:01.948362+0000","last_active":"2024-02-12T16:56:15.976699+0000","last_peered":"2024-02-12T16:56:15.976699+0000","last_clean":"2024-02-12T16:56:15.976699+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:56:15.976699+0000","last_undegraded":"2024-02-12T16:56:15.976699+0000","last_fullsized":"2024-02-12T16:56:15.976699+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:01.948286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:52:01.948286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":131,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977257+0000","last_change":"2024-02-12T16:52:10.220443+0000","last_active":"2024-02-12T16:56:15.977257+0000","last_peered":"2024-02-12T16:56:15.977257+0000","last_clean":"2024-02-12T16:56:15.977257+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:56:15.977257+0000","last_undegraded":"2024-02-12T16:56:15.977257+0000","last_fullsized":"2024-02-12T16:56:15.977257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:10.220358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:10.220358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":84,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976011+0000","last_change":"2024-02-12T16:54:41.013328+0000","last_active":"2024-02-12T16:56:15.976011+0000","last_peered":"2024-02-12T16:56:15.976011+0000","last_clean":"2024-02-12T16:56:15.976011+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T16:56:15.976011+0000","last_undegraded":"2024-02-12T16:56:15.976011+0000","last_fullsized":"2024-02-12T16:56:15.976011+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:48.029049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.029049+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:23.345846+0000","last_change":"2024-02-12T16:56:23.345846+0000","last_active":"2024-02-12T16:56:23.345846+0000","last_peered":"2024-02-12T16:56:23.345846+0000","last_clean":"2024-02-12T16:56:23.345846+0000","last_became_active":"2024-02-12T16:54:40.628201+0000","last_became_peered":"2024-02-12T16:54:40.628201+0000","last_unstale":"2024-02-12T16:56:23.345846+0000","last_undegraded":"2024-02-12T16:56:23.345846+0000","last_fullsized":"2024-02-12T16:56:23.345846+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":49,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523368+0000","last_change":"2024-02-12T16:54:40.629069+0000","last_active":"2024-02-12T16:56:15.523368+0000","last_peered":"2024-02-12T16:56:15.523368+0000","last_clean":"2024-02-12T16:56:15.523368+0000","last_became_active":"2024-02-12T16:54:40.628744+0000","last_became_peered":"2024-02-12T16:54:40.628744+0000","last_unstale":"2024-02-12T16:56:15.523368+0000","last_undegraded":"2024-02-12T16:56:15.523368+0000","last_fullsized":"2024-02-12T16:56:15.523368+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:40.022594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.022594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976784+0000","last_change":"2024-02-12T16:54:40.633472+0000","last_active":"2024-02-12T16:56:15.976784+0000","last_peered":"2024-02-12T16:56:15.976784+0000","last_clean":"2024-02-12T16:56:15.976784+0000","last_became_active":"2024-02-12T16:54:40.633129+0000","last_became_peered":"2024-02-12T16:54:40.633129+0000","last_unstale":"2024-02-12T16:56:15.976784+0000","last_undegraded":"2024-02-12T16:56:15.976784+0000","last_fullsized":"2024-02-12T16:56:15.976784+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:03.947783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:03.947783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":93,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522399+0000","last_change":"2024-02-12T16:54:41.077705+0000","last_active":"2024-02-12T16:56:15.522399+0000","last_peered":"2024-02-12T16:56:15.522399+0000","last_clean":"2024-02-12T16:56:15.522399+0000","last_became_active":"2024-02-12T16:54:40.626676+0000","last_became_peered":"2024-02-12T16:54:40.626676+0000","last_unstale":"2024-02-12T16:56:15.522399+0000","last_undegraded":"2024-02-12T16:56:15.522399+0000","last_fullsized":"2024-02-12T16:56:15.522399+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:49.624070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.624070+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":138,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:18.464298+0000","last_change":"2024-02-12T16:56:18.464298+0000","last_active":"2024-02-12T16:56:18.464298+0000","last_peered":"2024-02-12T16:56:18.464298+0000","last_clean":"2024-02-12T16:56:18.464298+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:56:18.464298+0000","last_undegraded":"2024-02-12T16:56:18.464298+0000","last_fullsized":"2024-02-12T16:56:18.464298+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":129,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522109+0000","last_change":"2024-02-12T16:53:43.855961+0000","last_active":"2024-02-12T16:56:15.522109+0000","last_peered":"2024-02-12T16:56:15.522109+0000","last_clean":"2024-02-12T16:56:15.522109+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:56:15.522109+0000","last_undegraded":"2024-02-12T16:56:15.522109+0000","last_fullsized":"2024-02-12T16:56:15.522109+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:43.855860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.855860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977357+0000","last_change":"2024-02-12T16:52:58.691294+0000","last_active":"2024-02-12T16:56:15.977357+0000","last_peered":"2024-02-12T16:56:15.977357+0000","last_clean":"2024-02-12T16:56:15.977357+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:56:15.977357+0000","last_undegraded":"2024-02-12T16:56:15.977357+0000","last_fullsized":"2024-02-12T16:56:15.977357+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:58.691233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.691233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":190,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523565+0000","last_change":"2024-02-12T16:54:40.681231+0000","last_active":"2024-02-12T16:56:15.523565+0000","last_peered":"2024-02-12T16:56:15.523565+0000","last_clean":"2024-02-12T16:56:15.523565+0000","last_became_active":"2024-02-12T16:54:40.630167+0000","last_became_peered":"2024-02-12T16:54:40.630167+0000","last_unstale":"2024-02-12T16:56:15.523565+0000","last_undegraded":"2024-02-12T16:56:15.523565+0000","last_fullsized":"2024-02-12T16:56:15.523565+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:53:48.468547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.468547+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521896+0000","last_change":"2024-02-12T16:52:59.833591+0000","last_active":"2024-02-12T16:56:15.521896+0000","last_peered":"2024-02-12T16:56:15.521896+0000","last_clean":"2024-02-12T16:56:15.521896+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:56:15.521896+0000","last_undegraded":"2024-02-12T16:56:15.521896+0000","last_fullsized":"2024-02-12T16:56:15.521896+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:59.833484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.833484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976976+0000","last_change":"2024-02-12T16:53:04.085929+0000","last_active":"2024-02-12T16:56:15.976976+0000","last_peered":"2024-02-12T16:56:15.976976+0000","last_clean":"2024-02-12T16:56:15.976976+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:56:15.976976+0000","last_undegraded":"2024-02-12T16:56:15.976976+0000","last_fullsized":"2024-02-12T16:56:15.976976+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:04.085854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.085854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978387+0000","last_change":"2024-02-12T16:54:40.640797+0000","last_active":"2024-02-12T16:56:15.978387+0000","last_peered":"2024-02-12T16:56:15.978387+0000","last_clean":"2024-02-12T16:56:15.978387+0000","last_became_active":"2024-02-12T16:54:40.640452+0000","last_became_peered":"2024-02-12T16:54:40.640452+0000","last_unstale":"2024-02-12T16:56:15.978387+0000","last_undegraded":"2024-02-12T16:56:15.978387+0000","last_fullsized":"2024-02-12T16:56:15.978387+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.970730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.970730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978676+0000","last_change":"2024-02-12T16:52:52.604196+0000","last_active":"2024-02-12T16:56:15.978676+0000","last_peered":"2024-02-12T16:56:15.978676+0000","last_clean":"2024-02-12T16:56:15.978676+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:56:15.978676+0000","last_undegraded":"2024-02-12T16:56:15.978676+0000","last_fullsized":"2024-02-12T16:56:15.978676+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:52:52.604107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.604107+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977535+0000","last_change":"2024-02-12T16:53:04.169987+0000","last_active":"2024-02-12T16:56:15.977535+0000","last_peered":"2024-02-12T16:56:15.977535+0000","last_clean":"2024-02-12T16:56:15.977535+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:56:15.977535+0000","last_undegraded":"2024-02-12T16:56:15.977535+0000","last_fullsized":"2024-02-12T16:56:15.977535+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:53:04.169892+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.169892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":220,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:17.616899+0000","last_change":"2024-02-12T16:52:44.602751+0000","last_active":"2024-02-12T16:56:17.616899+0000","last_peered":"2024-02-12T16:56:17.616899+0000","last_clean":"2024-02-12T16:56:17.616899+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:56:17.616899+0000","last_undegraded":"2024-02-12T16:56:17.616899+0000","last_fullsized":"2024-02-12T16:56:17.616899+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:44.602656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.602656+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976599+0000","last_change":"2024-02-12T16:52:47.997796+0000","last_active":"2024-02-12T16:56:15.976599+0000","last_peered":"2024-02-12T16:56:15.976599+0000","last_clean":"2024-02-12T16:56:15.976599+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:56:15.976599+0000","last_undegraded":"2024-02-12T16:56:15.976599+0000","last_fullsized":"2024-02-12T16:56:15.976599+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:47.997708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.997708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":165,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976972+0000","last_change":"2024-02-12T16:54:40.691148+0000","last_active":"2024-02-12T16:56:15.976972+0000","last_peered":"2024-02-12T16:56:15.976972+0000","last_clean":"2024-02-12T16:56:15.976972+0000","last_became_active":"2024-02-12T16:54:40.650318+0000","last_became_peered":"2024-02-12T16:54:40.650318+0000","last_unstale":"2024-02-12T16:56:15.976972+0000","last_undegraded":"2024-02-12T16:56:15.976972+0000","last_fullsized":"2024-02-12T16:56:15.976972+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:51:57.693352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:57.693352+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976103+0000","last_change":"2024-02-12T16:54:40.650920+0000","last_active":"2024-02-12T16:56:15.976103+0000","last_peered":"2024-02-12T16:56:15.976103+0000","last_clean":"2024-02-12T16:56:15.976103+0000","last_became_active":"2024-02-12T16:54:40.650168+0000","last_became_peered":"2024-02-12T16:54:40.650168+0000","last_unstale":"2024-02-12T16:56:15.976103+0000","last_undegraded":"2024-02-12T16:56:15.976103+0000","last_fullsized":"2024-02-12T16:56:15.976103+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.665249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.665249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978696+0000","last_change":"2024-02-12T16:53:05.141324+0000","last_active":"2024-02-12T16:56:15.978696+0000","last_peered":"2024-02-12T16:56:15.978696+0000","last_clean":"2024-02-12T16:56:15.978696+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:56:15.978696+0000","last_undegraded":"2024-02-12T16:56:15.978696+0000","last_fullsized":"2024-02-12T16:56:15.978696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978497+0000","last_change":"2024-02-12T16:54:39.665472+0000","last_active":"2024-02-12T16:56:15.978497+0000","last_peered":"2024-02-12T16:56:15.978497+0000","last_clean":"2024-02-12T16:56:15.978497+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:56:15.978497+0000","last_undegraded":"2024-02-12T16:56:15.978497+0000","last_fullsized":"2024-02-12T16:56:15.978497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.665378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.665378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":89,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976431+0000","last_change":"2024-02-12T16:54:41.066818+0000","last_active":"2024-02-12T16:56:15.976431+0000","last_peered":"2024-02-12T16:56:15.976431+0000","last_clean":"2024-02-12T16:56:15.976431+0000","last_became_active":"2024-02-12T16:54:40.632501+0000","last_became_peered":"2024-02-12T16:54:40.632501+0000","last_unstale":"2024-02-12T16:56:15.976431+0000","last_undegraded":"2024-02-12T16:56:15.976431+0000","last_fullsized":"2024-02-12T16:56:15.976431+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:51:57.052428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:57.052428+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":125,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977232+0000","last_change":"2024-02-12T16:52:55.241717+0000","last_active":"2024-02-12T16:56:15.977232+0000","last_peered":"2024-02-12T16:56:15.977232+0000","last_clean":"2024-02-12T16:56:15.977232+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:56:15.977232+0000","last_undegraded":"2024-02-12T16:56:15.977232+0000","last_fullsized":"2024-02-12T16:56:15.977232+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.241622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.241622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978393+0000","last_change":"2024-02-12T16:53:48.719406+0000","last_active":"2024-02-12T16:56:15.978393+0000","last_peered":"2024-02-12T16:56:15.978393+0000","last_clean":"2024-02-12T16:56:15.978393+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:56:15.978393+0000","last_undegraded":"2024-02-12T16:56:15.978393+0000","last_fullsized":"2024-02-12T16:56:15.978393+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.719344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.719344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.979184+0000","last_change":"2024-02-12T16:52:49.986507+0000","last_active":"2024-02-12T16:56:15.979184+0000","last_peered":"2024-02-12T16:56:15.979184+0000","last_clean":"2024-02-12T16:56:15.979184+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:56:15.979184+0000","last_undegraded":"2024-02-12T16:56:15.979184+0000","last_fullsized":"2024-02-12T16:56:15.979184+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.986403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.986403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":70,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521497+0000","last_change":"2024-02-12T16:54:41.081430+0000","last_active":"2024-02-12T16:56:15.521497+0000","last_peered":"2024-02-12T16:56:15.521497+0000","last_clean":"2024-02-12T16:56:15.521497+0000","last_became_active":"2024-02-12T16:54:40.639547+0000","last_became_peered":"2024-02-12T16:54:40.639547+0000","last_unstale":"2024-02-12T16:56:15.521497+0000","last_undegraded":"2024-02-12T16:56:15.521497+0000","last_fullsized":"2024-02-12T16:56:15.521497+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976133+0000","last_change":"2024-02-12T16:54:40.641535+0000","last_active":"2024-02-12T16:56:15.976133+0000","last_peered":"2024-02-12T16:56:15.976133+0000","last_clean":"2024-02-12T16:56:15.976133+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T16:56:15.976133+0000","last_undegraded":"2024-02-12T16:56:15.976133+0000","last_fullsized":"2024-02-12T16:56:15.976133+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:51:42.040828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":143,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521873+0000","last_change":"2024-02-12T16:54:40.629347+0000","last_active":"2024-02-12T16:56:15.521873+0000","last_peered":"2024-02-12T16:56:15.521873+0000","last_clean":"2024-02-12T16:56:15.521873+0000","last_became_active":"2024-02-12T16:54:40.628571+0000","last_became_peered":"2024-02-12T16:54:40.628571+0000","last_unstale":"2024-02-12T16:56:15.521873+0000","last_undegraded":"2024-02-12T16:56:15.521873+0000","last_fullsized":"2024-02-12T16:56:15.521873+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.785477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.785477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522194+0000","last_change":"2024-02-12T16:54:01.773110+0000","last_active":"2024-02-12T16:56:15.522194+0000","last_peered":"2024-02-12T16:56:15.522194+0000","last_clean":"2024-02-12T16:56:15.522194+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:56:15.522194+0000","last_undegraded":"2024-02-12T16:56:15.522194+0000","last_fullsized":"2024-02-12T16:56:15.522194+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":154,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522466+0000","last_change":"2024-02-12T16:54:29.644741+0000","last_active":"2024-02-12T16:56:15.522466+0000","last_peered":"2024-02-12T16:56:15.522466+0000","last_clean":"2024-02-12T16:56:15.522466+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:56:15.522466+0000","last_undegraded":"2024-02-12T16:56:15.522466+0000","last_fullsized":"2024-02-12T16:56:15.522466+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:54:29.644635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:29.644635+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":125,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.975875+0000","last_change":"2024-02-12T16:54:40.194031+0000","last_active":"2024-02-12T16:56:15.975875+0000","last_peered":"2024-02-12T16:56:15.975875+0000","last_clean":"2024-02-12T16:56:15.975875+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:56:15.975875+0000","last_undegraded":"2024-02-12T16:56:15.975875+0000","last_fullsized":"2024-02-12T16:56:15.975875+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":62,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976138+0000","last_change":"2024-02-12T16:55:46.679521+0000","last_active":"2024-02-12T16:56:15.976138+0000","last_peered":"2024-02-12T16:56:15.976138+0000","last_clean":"2024-02-12T16:56:15.976138+0000","last_became_active":"2024-02-12T16:54:40.650612+0000","last_became_peered":"2024-02-12T16:54:40.650612+0000","last_unstale":"2024-02-12T16:56:15.976138+0000","last_undegraded":"2024-02-12T16:56:15.976138+0000","last_fullsized":"2024-02-12T16:56:15.976138+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:51:39.040203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:51:39.040203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":141,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.979002+0000","last_change":"2024-02-12T16:54:40.629935+0000","last_active":"2024-02-12T16:56:15.979002+0000","last_peered":"2024-02-12T16:56:15.979002+0000","last_clean":"2024-02-12T16:56:15.979002+0000","last_became_active":"2024-02-12T16:54:40.629419+0000","last_became_peered":"2024-02-12T16:54:40.629419+0000","last_unstale":"2024-02-12T16:56:15.979002+0000","last_undegraded":"2024-02-12T16:56:15.979002+0000","last_fullsized":"2024-02-12T16:56:15.979002+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.590060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.590060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":156,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521785+0000","last_change":"2024-02-12T16:54:31.954069+0000","last_active":"2024-02-12T16:56:15.521785+0000","last_peered":"2024-02-12T16:56:15.521785+0000","last_clean":"2024-02-12T16:56:15.521785+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:56:15.521785+0000","last_undegraded":"2024-02-12T16:56:15.521785+0000","last_fullsized":"2024-02-12T16:56:15.521785+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":91,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522186+0000","last_change":"2024-02-12T16:52:59.296102+0000","last_active":"2024-02-12T16:56:15.522186+0000","last_peered":"2024-02-12T16:56:15.522186+0000","last_clean":"2024-02-12T16:56:15.522186+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T16:56:15.522186+0000","last_undegraded":"2024-02-12T16:56:15.522186+0000","last_fullsized":"2024-02-12T16:56:15.522186+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:59.296023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.296023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"138'9","reported_seq":116,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:20.829752+0000","last_change":"2024-02-12T16:56:20.829720+0000","last_active":"2024-02-12T16:56:20.829752+0000","last_peered":"2024-02-12T16:56:20.829752+0000","last_clean":"2024-02-12T16:56:20.829752+0000","last_became_active":"2024-02-12T16:54:40.644025+0000","last_became_peered":"2024-02-12T16:54:40.644025+0000","last_unstale":"2024-02-12T16:56:20.829752+0000","last_undegraded":"2024-02-12T16:56:20.829752+0000","last_fullsized":"2024-02-12T16:56:20.829752+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":140,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976258+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T16:56:15.976258+0000","last_peered":"2024-02-12T16:56:15.976258+0000","last_clean":"2024-02-12T16:56:15.976258+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T16:56:15.976258+0000","last_undegraded":"2024-02-12T16:56:15.976258+0000","last_fullsized":"2024-02-12T16:56:15.976258+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":166,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977474+0000","last_change":"2024-02-12T16:54:40.641346+0000","last_active":"2024-02-12T16:56:15.977474+0000","last_peered":"2024-02-12T16:56:15.977474+0000","last_clean":"2024-02-12T16:56:15.977474+0000","last_became_active":"2024-02-12T16:54:40.626031+0000","last_became_peered":"2024-02-12T16:54:40.626031+0000","last_unstale":"2024-02-12T16:56:15.977474+0000","last_undegraded":"2024-02-12T16:56:15.977474+0000","last_fullsized":"2024-02-12T16:56:15.977474+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":134,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976553+0000","last_change":"2024-02-12T16:54:40.633046+0000","last_active":"2024-02-12T16:56:15.976553+0000","last_peered":"2024-02-12T16:56:15.976553+0000","last_clean":"2024-02-12T16:56:15.976553+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T16:56:15.976553+0000","last_undegraded":"2024-02-12T16:56:15.976553+0000","last_fullsized":"2024-02-12T16:56:15.976553+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.105539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.105539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":220,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:17.531577+0000","last_change":"2024-02-12T16:54:00.112924+0000","last_active":"2024-02-12T16:56:17.531577+0000","last_peered":"2024-02-12T16:56:17.531577+0000","last_clean":"2024-02-12T16:56:17.531577+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:56:17.531577+0000","last_undegraded":"2024-02-12T16:56:17.531577+0000","last_fullsized":"2024-02-12T16:56:17.531577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":142,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978789+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T16:56:15.978789+0000","last_peered":"2024-02-12T16:56:15.978789+0000","last_clean":"2024-02-12T16:56:15.978789+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T16:56:15.978789+0000","last_undegraded":"2024-02-12T16:56:15.978789+0000","last_fullsized":"2024-02-12T16:56:15.978789+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":56,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522417+0000","last_change":"2024-02-12T16:54:42.827081+0000","last_active":"2024-02-12T16:56:15.522417+0000","last_peered":"2024-02-12T16:56:15.522417+0000","last_clean":"2024-02-12T16:56:15.522417+0000","last_became_active":"2024-02-12T16:54:40.639018+0000","last_became_peered":"2024-02-12T16:54:40.639018+0000","last_unstale":"2024-02-12T16:56:15.522417+0000","last_undegraded":"2024-02-12T16:56:15.522417+0000","last_fullsized":"2024-02-12T16:56:15.522417+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:03.616717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.616717+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":129,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977541+0000","last_change":"2024-02-12T16:53:51.532654+0000","last_active":"2024-02-12T16:56:15.977541+0000","last_peered":"2024-02-12T16:56:15.977541+0000","last_clean":"2024-02-12T16:56:15.977541+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:56:15.977541+0000","last_undegraded":"2024-02-12T16:56:15.977541+0000","last_fullsized":"2024-02-12T16:56:15.977541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.979030+0000","last_change":"2024-02-12T16:54:40.644416+0000","last_active":"2024-02-12T16:56:15.979030+0000","last_peered":"2024-02-12T16:56:15.979030+0000","last_clean":"2024-02-12T16:56:15.979030+0000","last_became_active":"2024-02-12T16:54:40.644226+0000","last_became_peered":"2024-02-12T16:54:40.644226+0000","last_unstale":"2024-02-12T16:56:15.979030+0000","last_undegraded":"2024-02-12T16:56:15.979030+0000","last_fullsized":"2024-02-12T16:56:15.979030+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.844487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.844487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978473+0000","last_change":"2024-02-12T16:52:43.579914+0000","last_active":"2024-02-12T16:56:15.978473+0000","last_peered":"2024-02-12T16:56:15.978473+0000","last_clean":"2024-02-12T16:56:15.978473+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:56:15.978473+0000","last_undegraded":"2024-02-12T16:56:15.978473+0000","last_fullsized":"2024-02-12T16:56:15.978473+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:43.579811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.579811+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":75,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978008+0000","last_change":"2024-02-12T16:54:40.903364+0000","last_active":"2024-02-12T16:56:15.978008+0000","last_peered":"2024-02-12T16:56:15.978008+0000","last_clean":"2024-02-12T16:56:15.978008+0000","last_became_active":"2024-02-12T16:54:40.640126+0000","last_became_peered":"2024-02-12T16:54:40.640126+0000","last_unstale":"2024-02-12T16:56:15.978008+0000","last_undegraded":"2024-02-12T16:56:15.978008+0000","last_fullsized":"2024-02-12T16:56:15.978008+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523231+0000","last_change":"2024-02-12T16:52:39.278662+0000","last_active":"2024-02-12T16:56:15.523231+0000","last_peered":"2024-02-12T16:56:15.523231+0000","last_clean":"2024-02-12T16:56:15.523231+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:56:15.523231+0000","last_undegraded":"2024-02-12T16:56:15.523231+0000","last_fullsized":"2024-02-12T16:56:15.523231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:39.278592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:39.278592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522250+0000","last_change":"2024-02-12T16:54:40.634603+0000","last_active":"2024-02-12T16:56:15.522250+0000","last_peered":"2024-02-12T16:56:15.522250+0000","last_clean":"2024-02-12T16:56:15.522250+0000","last_became_active":"2024-02-12T16:54:40.634261+0000","last_became_peered":"2024-02-12T16:54:40.634261+0000","last_unstale":"2024-02-12T16:56:15.522250+0000","last_undegraded":"2024-02-12T16:56:15.522250+0000","last_fullsized":"2024-02-12T16:56:15.522250+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:41.740892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:53:41.740892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":138,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:22.699402+0000","last_change":"2024-02-12T16:56:22.699402+0000","last_active":"2024-02-12T16:56:22.699402+0000","last_peered":"2024-02-12T16:56:22.699402+0000","last_clean":"2024-02-12T16:56:22.699402+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:56:22.699402+0000","last_undegraded":"2024-02-12T16:56:22.699402+0000","last_fullsized":"2024-02-12T16:56:22.699402+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":168,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976778+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T16:56:15.976778+0000","last_peered":"2024-02-12T16:56:15.976778+0000","last_clean":"2024-02-12T16:56:15.976778+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T16:56:15.976778+0000","last_undegraded":"2024-02-12T16:56:15.976778+0000","last_fullsized":"2024-02-12T16:56:15.976778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976637+0000","last_change":"2024-02-12T16:54:40.643841+0000","last_active":"2024-02-12T16:56:15.976637+0000","last_peered":"2024-02-12T16:56:15.976637+0000","last_clean":"2024-02-12T16:56:15.976637+0000","last_became_active":"2024-02-12T16:54:40.643529+0000","last_became_peered":"2024-02-12T16:54:40.643529+0000","last_unstale":"2024-02-12T16:56:15.976637+0000","last_undegraded":"2024-02-12T16:56:15.976637+0000","last_fullsized":"2024-02-12T16:56:15.976637+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":159,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522798+0000","last_change":"2024-02-12T16:55:34.601557+0000","last_active":"2024-02-12T16:56:15.522798+0000","last_peered":"2024-02-12T16:56:15.522798+0000","last_clean":"2024-02-12T16:56:15.522798+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:56:15.522798+0000","last_undegraded":"2024-02-12T16:56:15.522798+0000","last_fullsized":"2024-02-12T16:56:15.522798+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:55:34.601453+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:55:34.601453+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":27,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976788+0000","last_change":"2024-02-12T16:56:05.735251+0000","last_active":"2024-02-12T16:56:15.976788+0000","last_peered":"2024-02-12T16:56:15.976788+0000","last_clean":"2024-02-12T16:56:15.976788+0000","last_became_active":"2024-02-12T16:56:04.871405+0000","last_became_peered":"2024-02-12T16:56:04.871405+0000","last_unstale":"2024-02-12T16:56:15.976788+0000","last_undegraded":"2024-02-12T16:56:15.976788+0000","last_fullsized":"2024-02-12T16:56:15.976788+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:52:44.868438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.868438+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":129,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523297+0000","last_change":"2024-02-12T16:52:48.411110+0000","last_active":"2024-02-12T16:56:15.523297+0000","last_peered":"2024-02-12T16:56:15.523297+0000","last_clean":"2024-02-12T16:56:15.523297+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:56:15.523297+0000","last_undegraded":"2024-02-12T16:56:15.523297+0000","last_fullsized":"2024-02-12T16:56:15.523297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.411008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.411008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":134,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.975923+0000","last_change":"2024-02-12T16:54:40.638105+0000","last_active":"2024-02-12T16:56:15.975923+0000","last_peered":"2024-02-12T16:56:15.975923+0000","last_clean":"2024-02-12T16:56:15.975923+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T16:56:15.975923+0000","last_undegraded":"2024-02-12T16:56:15.975923+0000","last_fullsized":"2024-02-12T16:56:15.975923+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.248970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.248970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522499+0000","last_change":"2024-02-12T16:53:40.916900+0000","last_active":"2024-02-12T16:56:15.522499+0000","last_peered":"2024-02-12T16:56:15.522499+0000","last_clean":"2024-02-12T16:56:15.522499+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:56:15.522499+0000","last_undegraded":"2024-02-12T16:56:15.522499+0000","last_fullsized":"2024-02-12T16:56:15.522499+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:40.916831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:40.916831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":122,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523533+0000","last_change":"2024-02-12T16:54:39.421299+0000","last_active":"2024-02-12T16:56:15.523533+0000","last_peered":"2024-02-12T16:56:15.523533+0000","last_clean":"2024-02-12T16:56:15.523533+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:56:15.523533+0000","last_undegraded":"2024-02-12T16:56:15.523533+0000","last_fullsized":"2024-02-12T16:56:15.523533+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:39.421176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.421176+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":216,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:17.948670+0000","last_change":"2024-02-12T16:53:59.173126+0000","last_active":"2024-02-12T16:56:17.948670+0000","last_peered":"2024-02-12T16:56:17.948670+0000","last_clean":"2024-02-12T16:56:17.948670+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:56:17.948670+0000","last_undegraded":"2024-02-12T16:56:17.948670+0000","last_fullsized":"2024-02-12T16:56:17.948670+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522754+0000","last_change":"2024-02-12T16:55:40.530363+0000","last_active":"2024-02-12T16:56:15.522754+0000","last_peered":"2024-02-12T16:56:15.522754+0000","last_clean":"2024-02-12T16:56:15.522754+0000","last_became_active":"2024-02-12T16:54:40.625901+0000","last_became_peered":"2024-02-12T16:54:40.625901+0000","last_unstale":"2024-02-12T16:56:15.522754+0000","last_undegraded":"2024-02-12T16:56:15.522754+0000","last_fullsized":"2024-02-12T16:56:15.522754+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:55:40.530269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978872+0000","last_change":"2024-02-12T16:52:06.907569+0000","last_active":"2024-02-12T16:56:15.978872+0000","last_peered":"2024-02-12T16:56:15.978872+0000","last_clean":"2024-02-12T16:56:15.978872+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:56:15.978872+0000","last_undegraded":"2024-02-12T16:56:15.978872+0000","last_fullsized":"2024-02-12T16:56:15.978872+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:06.907496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:06.907496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"138'29","reported_seq":292,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:17.533013+0000","last_change":"2024-02-12T16:54:42.913130+0000","last_active":"2024-02-12T16:56:17.533013+0000","last_peered":"2024-02-12T16:56:17.533013+0000","last_clean":"2024-02-12T16:56:17.533013+0000","last_became_active":"2024-02-12T16:54:40.639565+0000","last_became_peered":"2024-02-12T16:54:40.639565+0000","last_unstale":"2024-02-12T16:56:17.533013+0000","last_undegraded":"2024-02-12T16:56:17.533013+0000","last_fullsized":"2024-02-12T16:56:17.533013+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-12T16:53:59.751031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.751031+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976774+0000","last_change":"2024-02-12T16:53:01.103805+0000","last_active":"2024-02-12T16:56:15.976774+0000","last_peered":"2024-02-12T16:56:15.976774+0000","last_clean":"2024-02-12T16:56:15.976774+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:56:15.976774+0000","last_undegraded":"2024-02-12T16:56:15.976774+0000","last_fullsized":"2024-02-12T16:56:15.976774+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.103703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.103703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":136,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977679+0000","last_change":"2024-02-12T16:54:40.640502+0000","last_active":"2024-02-12T16:56:15.977679+0000","last_peered":"2024-02-12T16:56:15.977679+0000","last_clean":"2024-02-12T16:56:15.977679+0000","last_became_active":"2024-02-12T16:54:40.640105+0000","last_became_peered":"2024-02-12T16:54:40.640105+0000","last_unstale":"2024-02-12T16:56:15.977679+0000","last_undegraded":"2024-02-12T16:56:15.977679+0000","last_fullsized":"2024-02-12T16:56:15.977679+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":538,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:22.430973+0000","last_change":"2024-02-12T16:54:30.972535+0000","last_active":"2024-02-12T16:56:22.430973+0000","last_peered":"2024-02-12T16:56:22.430973+0000","last_clean":"2024-02-12T16:56:22.430973+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:56:22.430973+0000","last_undegraded":"2024-02-12T16:56:22.430973+0000","last_fullsized":"2024-02-12T16:56:22.430973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":397,"num_read_kb":392,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:20.676702+0000","last_change":"2024-02-12T16:56:20.676702+0000","last_active":"2024-02-12T16:56:20.676702+0000","last_peered":"2024-02-12T16:56:20.676702+0000","last_clean":"2024-02-12T16:56:20.676702+0000","last_became_active":"2024-02-12T16:54:40.632657+0000","last_became_peered":"2024-02-12T16:54:40.632657+0000","last_unstale":"2024-02-12T16:56:20.676702+0000","last_undegraded":"2024-02-12T16:56:20.676702+0000","last_fullsized":"2024-02-12T16:56:20.676702+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":163,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522018+0000","last_change":"2024-02-12T16:53:04.923477+0000","last_active":"2024-02-12T16:56:15.522018+0000","last_peered":"2024-02-12T16:56:15.522018+0000","last_clean":"2024-02-12T16:56:15.522018+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:56:15.522018+0000","last_undegraded":"2024-02-12T16:56:15.522018+0000","last_fullsized":"2024-02-12T16:56:15.522018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.923375+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":129,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523770+0000","last_change":"2024-02-12T16:52:45.314737+0000","last_active":"2024-02-12T16:56:15.523770+0000","last_peered":"2024-02-12T16:56:15.523770+0000","last_clean":"2024-02-12T16:56:15.523770+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:56:15.523770+0000","last_undegraded":"2024-02-12T16:56:15.523770+0000","last_fullsized":"2024-02-12T16:56:15.523770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:45.314637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:45.314637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":136,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978832+0000","last_change":"2024-02-12T16:54:40.628021+0000","last_active":"2024-02-12T16:56:15.978832+0000","last_peered":"2024-02-12T16:56:15.978832+0000","last_clean":"2024-02-12T16:56:15.978832+0000","last_became_active":"2024-02-12T16:54:40.627688+0000","last_became_peered":"2024-02-12T16:54:40.627688+0000","last_unstale":"2024-02-12T16:56:15.978832+0000","last_undegraded":"2024-02-12T16:56:15.978832+0000","last_fullsized":"2024-02-12T16:56:15.978832+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.979059+0000","last_change":"2024-02-12T16:54:40.628668+0000","last_active":"2024-02-12T16:56:15.979059+0000","last_peered":"2024-02-12T16:56:15.979059+0000","last_clean":"2024-02-12T16:56:15.979059+0000","last_became_active":"2024-02-12T16:54:40.628365+0000","last_became_peered":"2024-02-12T16:54:40.628365+0000","last_unstale":"2024-02-12T16:56:15.979059+0000","last_undegraded":"2024-02-12T16:56:15.979059+0000","last_fullsized":"2024-02-12T16:56:15.979059+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.967335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976792+0000","last_change":"2024-02-12T16:52:57.985677+0000","last_active":"2024-02-12T16:56:15.976792+0000","last_peered":"2024-02-12T16:56:15.976792+0000","last_clean":"2024-02-12T16:56:15.976792+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:56:15.976792+0000","last_undegraded":"2024-02-12T16:56:15.976792+0000","last_fullsized":"2024-02-12T16:56:15.976792+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:57.985584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:57.985584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":139,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978491+0000","last_change":"2024-02-12T16:54:30.770032+0000","last_active":"2024-02-12T16:56:15.978491+0000","last_peered":"2024-02-12T16:56:15.978491+0000","last_clean":"2024-02-12T16:56:15.978491+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:56:15.978491+0000","last_undegraded":"2024-02-12T16:56:15.978491+0000","last_fullsized":"2024-02-12T16:56:15.978491+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":157,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978171+0000","last_change":"2024-02-12T16:53:58.740196+0000","last_active":"2024-02-12T16:56:15.978171+0000","last_peered":"2024-02-12T16:56:15.978171+0000","last_clean":"2024-02-12T16:56:15.978171+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:56:15.978171+0000","last_undegraded":"2024-02-12T16:56:15.978171+0000","last_fullsized":"2024-02-12T16:56:15.978171+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:53:58.740122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.740122+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977369+0000","last_change":"2024-02-12T16:52:12.200439+0000","last_active":"2024-02-12T16:56:15.977369+0000","last_peered":"2024-02-12T16:56:15.977369+0000","last_clean":"2024-02-12T16:56:15.977369+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:56:15.977369+0000","last_undegraded":"2024-02-12T16:56:15.977369+0000","last_fullsized":"2024-02-12T16:56:15.977369+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:12.200355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:12.200355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":136,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522928+0000","last_change":"2024-02-12T16:54:40.628087+0000","last_active":"2024-02-12T16:56:15.522928+0000","last_peered":"2024-02-12T16:56:15.522928+0000","last_clean":"2024-02-12T16:56:15.522928+0000","last_became_active":"2024-02-12T16:54:40.626511+0000","last_became_peered":"2024-02-12T16:54:40.626511+0000","last_unstale":"2024-02-12T16:56:15.522928+0000","last_undegraded":"2024-02-12T16:56:15.522928+0000","last_fullsized":"2024-02-12T16:56:15.522928+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.780846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.780846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":140,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976037+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T16:56:15.976037+0000","last_peered":"2024-02-12T16:56:15.976037+0000","last_clean":"2024-02-12T16:56:15.976037+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T16:56:15.976037+0000","last_undegraded":"2024-02-12T16:56:15.976037+0000","last_fullsized":"2024-02-12T16:56:15.976037+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978404+0000","last_change":"2024-02-12T16:52:58.167642+0000","last_active":"2024-02-12T16:56:15.978404+0000","last_peered":"2024-02-12T16:56:15.978404+0000","last_clean":"2024-02-12T16:56:15.978404+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:56:15.978404+0000","last_undegraded":"2024-02-12T16:56:15.978404+0000","last_fullsized":"2024-02-12T16:56:15.978404+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:52:58.167543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.167543+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":147,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976665+0000","last_change":"2024-02-12T16:53:43.229740+0000","last_active":"2024-02-12T16:56:15.976665+0000","last_peered":"2024-02-12T16:56:15.976665+0000","last_clean":"2024-02-12T16:56:15.976665+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:56:15.976665+0000","last_undegraded":"2024-02-12T16:56:15.976665+0000","last_fullsized":"2024-02-12T16:56:15.976665+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:43.229622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.229622+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":129,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:18.162272+0000","last_change":"2024-02-12T16:56:18.162272+0000","last_active":"2024-02-12T16:56:18.162272+0000","last_peered":"2024-02-12T16:56:18.162272+0000","last_clean":"2024-02-12T16:56:18.162272+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:56:18.162272+0000","last_undegraded":"2024-02-12T16:56:18.162272+0000","last_fullsized":"2024-02-12T16:56:18.162272+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521930+0000","last_change":"2024-02-12T16:53:50.836514+0000","last_active":"2024-02-12T16:56:15.521930+0000","last_peered":"2024-02-12T16:56:15.521930+0000","last_clean":"2024-02-12T16:56:15.521930+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:56:15.521930+0000","last_undegraded":"2024-02-12T16:56:15.521930+0000","last_fullsized":"2024-02-12T16:56:15.521930+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.836406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.836406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":136,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977702+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T16:56:15.977702+0000","last_peered":"2024-02-12T16:56:15.977702+0000","last_clean":"2024-02-12T16:56:15.977702+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:56:15.977702+0000","last_undegraded":"2024-02-12T16:56:15.977702+0000","last_fullsized":"2024-02-12T16:56:15.977702+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":165,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976319+0000","last_change":"2024-02-12T16:51:52.745302+0000","last_active":"2024-02-12T16:56:15.976319+0000","last_peered":"2024-02-12T16:56:15.976319+0000","last_clean":"2024-02-12T16:56:15.976319+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:56:15.976319+0000","last_undegraded":"2024-02-12T16:56:15.976319+0000","last_fullsized":"2024-02-12T16:56:15.976319+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:51:52.745190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:52.745190+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977299+0000","last_change":"2024-02-12T16:53:50.681255+0000","last_active":"2024-02-12T16:56:15.977299+0000","last_peered":"2024-02-12T16:56:15.977299+0000","last_clean":"2024-02-12T16:56:15.977299+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:56:15.977299+0000","last_undegraded":"2024-02-12T16:56:15.977299+0000","last_fullsized":"2024-02-12T16:56:15.977299+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.681142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.681142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":49,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523614+0000","last_change":"2024-02-12T16:54:40.632821+0000","last_active":"2024-02-12T16:56:15.523614+0000","last_peered":"2024-02-12T16:56:15.523614+0000","last_clean":"2024-02-12T16:56:15.523614+0000","last_became_active":"2024-02-12T16:54:40.631423+0000","last_became_peered":"2024-02-12T16:54:40.631423+0000","last_unstale":"2024-02-12T16:56:15.523614+0000","last_undegraded":"2024-02-12T16:56:15.523614+0000","last_fullsized":"2024-02-12T16:56:15.523614+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.624865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.624865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522482+0000","last_change":"2024-02-12T16:53:48.828698+0000","last_active":"2024-02-12T16:56:15.522482+0000","last_peered":"2024-02-12T16:56:15.522482+0000","last_clean":"2024-02-12T16:56:15.522482+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:56:15.522482+0000","last_undegraded":"2024-02-12T16:56:15.522482+0000","last_fullsized":"2024-02-12T16:56:15.522482+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":108,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978791+0000","last_change":"2024-02-12T16:54:41.356259+0000","last_active":"2024-02-12T16:56:15.978791+0000","last_peered":"2024-02-12T16:56:15.978791+0000","last_clean":"2024-02-12T16:56:15.978791+0000","last_became_active":"2024-02-12T16:54:40.634781+0000","last_became_peered":"2024-02-12T16:54:40.634781+0000","last_unstale":"2024-02-12T16:56:15.978791+0000","last_undegraded":"2024-02-12T16:56:15.978791+0000","last_fullsized":"2024-02-12T16:56:15.978791+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:51:51.095125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:51:51.095125+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":145,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:19.484845+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T16:56:19.484845+0000","last_peered":"2024-02-12T16:56:19.484845+0000","last_clean":"2024-02-12T16:56:19.484845+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T16:56:19.484845+0000","last_undegraded":"2024-02-12T16:56:19.484845+0000","last_fullsized":"2024-02-12T16:56:19.484845+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":126,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977003+0000","last_change":"2024-02-12T16:52:41.149470+0000","last_active":"2024-02-12T16:56:15.977003+0000","last_peered":"2024-02-12T16:56:15.977003+0000","last_clean":"2024-02-12T16:56:15.977003+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:56:15.977003+0000","last_undegraded":"2024-02-12T16:56:15.977003+0000","last_fullsized":"2024-02-12T16:56:15.977003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:52:41.149382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.149382+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978544+0000","last_change":"2024-02-12T16:52:49.581959+0000","last_active":"2024-02-12T16:56:15.978544+0000","last_peered":"2024-02-12T16:56:15.978544+0000","last_clean":"2024-02-12T16:56:15.978544+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:56:15.978544+0000","last_undegraded":"2024-02-12T16:56:15.978544+0000","last_fullsized":"2024-02-12T16:56:15.978544+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.581862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.581862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":124,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977141+0000","last_change":"2024-02-12T16:52:51.025242+0000","last_active":"2024-02-12T16:56:15.977141+0000","last_peered":"2024-02-12T16:56:15.977141+0000","last_clean":"2024-02-12T16:56:15.977141+0000","last_became_active":"2024-02-12T16:52:20.563744+0000","last_became_peered":"2024-02-12T16:52:20.563744+0000","last_unstale":"2024-02-12T16:56:15.977141+0000","last_undegraded":"2024-02-12T16:56:15.977141+0000","last_fullsized":"2024-02-12T16:56:15.977141+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:52:51.025152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.025152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"138'9","reported_seq":96,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:17.531043+0000","last_change":"2024-02-12T16:55:28.449149+0000","last_active":"2024-02-12T16:56:17.531043+0000","last_peered":"2024-02-12T16:56:17.531043+0000","last_clean":"2024-02-12T16:56:17.531043+0000","last_became_active":"2024-02-12T16:54:40.626646+0000","last_became_peered":"2024-02-12T16:54:40.626646+0000","last_unstale":"2024-02-12T16:56:17.531043+0000","last_undegraded":"2024-02-12T16:56:17.531043+0000","last_fullsized":"2024-02-12T16:56:17.531043+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":46,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978916+0000","last_change":"2024-02-12T16:54:40.629725+0000","last_active":"2024-02-12T16:56:15.978916+0000","last_peered":"2024-02-12T16:56:15.978916+0000","last_clean":"2024-02-12T16:56:15.978916+0000","last_became_active":"2024-02-12T16:54:40.629348+0000","last_became_peered":"2024-02-12T16:54:40.629348+0000","last_unstale":"2024-02-12T16:56:15.978916+0000","last_undegraded":"2024-02-12T16:56:15.978916+0000","last_fullsized":"2024-02-12T16:56:15.978916+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":129,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978577+0000","last_change":"2024-02-12T16:52:52.983089+0000","last_active":"2024-02-12T16:56:15.978577+0000","last_peered":"2024-02-12T16:56:15.978577+0000","last_clean":"2024-02-12T16:56:15.978577+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:56:15.978577+0000","last_undegraded":"2024-02-12T16:56:15.978577+0000","last_fullsized":"2024-02-12T16:56:15.978577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:52.982996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.982996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":110,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523172+0000","last_change":"2024-02-12T16:53:59.566836+0000","last_active":"2024-02-12T16:56:15.523172+0000","last_peered":"2024-02-12T16:56:15.523172+0000","last_clean":"2024-02-12T16:56:15.523172+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:56:15.523172+0000","last_undegraded":"2024-02-12T16:56:15.523172+0000","last_fullsized":"2024-02-12T16:56:15.523172+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.566730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":148,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977783+0000","last_change":"2024-02-12T16:56:04.931453+0000","last_active":"2024-02-12T16:56:15.977783+0000","last_peered":"2024-02-12T16:56:15.977783+0000","last_clean":"2024-02-12T16:56:15.977783+0000","last_became_active":"2024-02-12T16:56:04.863064+0000","last_became_peered":"2024-02-12T16:56:04.863064+0000","last_unstale":"2024-02-12T16:56:15.977783+0000","last_undegraded":"2024-02-12T16:56:15.977783+0000","last_fullsized":"2024-02-12T16:56:15.977783+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:54:36.753321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:36.753321+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977774+0000","last_change":"2024-02-12T16:52:46.979058+0000","last_active":"2024-02-12T16:56:15.977774+0000","last_peered":"2024-02-12T16:56:15.977774+0000","last_clean":"2024-02-12T16:56:15.977774+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:56:15.977774+0000","last_undegraded":"2024-02-12T16:56:15.977774+0000","last_fullsized":"2024-02-12T16:56:15.977774+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:46.978987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:46.978987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"98'10","reported_seq":186,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:17.531689+0000","last_change":"2024-02-12T16:52:20.561940+0000","last_active":"2024-02-12T16:56:17.531689+0000","last_peered":"2024-02-12T16:56:17.531689+0000","last_clean":"2024-02-12T16:56:17.531689+0000","last_became_active":"2024-02-12T16:52:20.561534+0000","last_became_peered":"2024-02-12T16:52:20.561534+0000","last_unstale":"2024-02-12T16:56:17.531689+0000","last_undegraded":"2024-02-12T16:56:17.531689+0000","last_fullsized":"2024-02-12T16:56:17.531689+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-02-12T16:52:12.826762+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:52:12.826762+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":156,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521247+0000","last_change":"2024-02-12T16:56:04.857653+0000","last_active":"2024-02-12T16:56:15.521247+0000","last_peered":"2024-02-12T16:56:15.521247+0000","last_clean":"2024-02-12T16:56:15.521247+0000","last_became_active":"2024-02-12T16:56:04.857412+0000","last_became_peered":"2024-02-12T16:56:04.857412+0000","last_unstale":"2024-02-12T16:56:15.521247+0000","last_undegraded":"2024-02-12T16:56:15.521247+0000","last_fullsized":"2024-02-12T16:56:15.521247+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":193,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522101+0000","last_change":"2024-02-12T16:54:41.243992+0000","last_active":"2024-02-12T16:56:15.522101+0000","last_peered":"2024-02-12T16:56:15.522101+0000","last_clean":"2024-02-12T16:56:15.522101+0000","last_became_active":"2024-02-12T16:54:40.627029+0000","last_became_peered":"2024-02-12T16:54:40.627029+0000","last_unstale":"2024-02-12T16:56:15.522101+0000","last_undegraded":"2024-02-12T16:56:15.522101+0000","last_fullsized":"2024-02-12T16:56:15.522101+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":91,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522031+0000","last_change":"2024-02-12T16:53:57.738352+0000","last_active":"2024-02-12T16:56:15.522031+0000","last_peered":"2024-02-12T16:56:15.522031+0000","last_clean":"2024-02-12T16:56:15.522031+0000","last_became_active":"2024-02-12T16:52:20.560085+0000","last_became_peered":"2024-02-12T16:52:20.560085+0000","last_unstale":"2024-02-12T16:56:15.522031+0000","last_undegraded":"2024-02-12T16:56:15.522031+0000","last_fullsized":"2024-02-12T16:56:15.522031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:57.738249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.738249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":218,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:17.532278+0000","last_change":"2024-02-12T16:52:52.366599+0000","last_active":"2024-02-12T16:56:17.532278+0000","last_peered":"2024-02-12T16:56:17.532278+0000","last_clean":"2024-02-12T16:56:17.532278+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:56:17.532278+0000","last_undegraded":"2024-02-12T16:56:17.532278+0000","last_fullsized":"2024-02-12T16:56:17.532278+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:52.366513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.366513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":147,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521584+0000","last_change":"2024-02-12T16:54:40.629536+0000","last_active":"2024-02-12T16:56:15.521584+0000","last_peered":"2024-02-12T16:56:15.521584+0000","last_clean":"2024-02-12T16:56:15.521584+0000","last_became_active":"2024-02-12T16:54:40.629024+0000","last_became_peered":"2024-02-12T16:54:40.629024+0000","last_unstale":"2024-02-12T16:56:15.521584+0000","last_undegraded":"2024-02-12T16:56:15.521584+0000","last_fullsized":"2024-02-12T16:56:15.521584+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":90,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.979098+0000","last_change":"2024-02-12T16:55:46.162945+0000","last_active":"2024-02-12T16:56:15.979098+0000","last_peered":"2024-02-12T16:56:15.979098+0000","last_clean":"2024-02-12T16:56:15.979098+0000","last_became_active":"2024-02-12T16:54:40.629780+0000","last_became_peered":"2024-02-12T16:54:40.629780+0000","last_unstale":"2024-02-12T16:56:15.979098+0000","last_undegraded":"2024-02-12T16:56:15.979098+0000","last_fullsized":"2024-02-12T16:56:15.979098+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":141,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522271+0000","last_change":"2024-02-12T16:54:40.631210+0000","last_active":"2024-02-12T16:56:15.522271+0000","last_peered":"2024-02-12T16:56:15.522271+0000","last_clean":"2024-02-12T16:56:15.522271+0000","last_became_active":"2024-02-12T16:54:40.630855+0000","last_became_peered":"2024-02-12T16:54:40.630855+0000","last_unstale":"2024-02-12T16:56:15.522271+0000","last_undegraded":"2024-02-12T16:56:15.522271+0000","last_fullsized":"2024-02-12T16:56:15.522271+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.387144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.387144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":129,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522845+0000","last_change":"2024-02-12T16:53:49.832912+0000","last_active":"2024-02-12T16:56:15.522845+0000","last_peered":"2024-02-12T16:56:15.522845+0000","last_clean":"2024-02-12T16:56:15.522845+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:56:15.522845+0000","last_undegraded":"2024-02-12T16:56:15.522845+0000","last_fullsized":"2024-02-12T16:56:15.522845+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:49.832798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.832798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":136,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976537+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T16:56:15.976537+0000","last_peered":"2024-02-12T16:56:15.976537+0000","last_clean":"2024-02-12T16:56:15.976537+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:56:15.976537+0000","last_undegraded":"2024-02-12T16:56:15.976537+0000","last_fullsized":"2024-02-12T16:56:15.976537+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":158,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522008+0000","last_change":"2024-02-12T16:56:04.910150+0000","last_active":"2024-02-12T16:56:15.522008+0000","last_peered":"2024-02-12T16:56:15.522008+0000","last_clean":"2024-02-12T16:56:15.522008+0000","last_became_active":"2024-02-12T16:56:04.869971+0000","last_became_peered":"2024-02-12T16:56:04.869971+0000","last_unstale":"2024-02-12T16:56:15.522008+0000","last_undegraded":"2024-02-12T16:56:15.522008+0000","last_fullsized":"2024-02-12T16:56:15.522008+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:52:44.280844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.280844+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":134,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978310+0000","last_change":"2024-02-12T16:54:40.645951+0000","last_active":"2024-02-12T16:56:15.978310+0000","last_peered":"2024-02-12T16:56:15.978310+0000","last_clean":"2024-02-12T16:56:15.978310+0000","last_became_active":"2024-02-12T16:54:40.645556+0000","last_became_peered":"2024-02-12T16:54:40.645556+0000","last_unstale":"2024-02-12T16:56:15.978310+0000","last_undegraded":"2024-02-12T16:56:15.978310+0000","last_fullsized":"2024-02-12T16:56:15.978310+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.128703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.128703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":134,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.521772+0000","last_change":"2024-02-12T16:53:58.524837+0000","last_active":"2024-02-12T16:56:15.521772+0000","last_peered":"2024-02-12T16:56:15.521772+0000","last_clean":"2024-02-12T16:56:15.521772+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:56:15.521772+0000","last_undegraded":"2024-02-12T16:56:15.521772+0000","last_fullsized":"2024-02-12T16:56:15.521772+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.524736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523470+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T16:56:15.523470+0000","last_peered":"2024-02-12T16:56:15.523470+0000","last_clean":"2024-02-12T16:56:15.523470+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:56:15.523470+0000","last_undegraded":"2024-02-12T16:56:15.523470+0000","last_fullsized":"2024-02-12T16:56:15.523470+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":195,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.977212+0000","last_change":"2024-02-12T16:53:40.847643+0000","last_active":"2024-02-12T16:56:15.977212+0000","last_peered":"2024-02-12T16:56:15.977212+0000","last_clean":"2024-02-12T16:56:15.977212+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:56:15.977212+0000","last_undegraded":"2024-02-12T16:56:15.977212+0000","last_fullsized":"2024-02-12T16:56:15.977212+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:53:40.847540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:40.847540+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":125,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976427+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T16:56:15.976427+0000","last_peered":"2024-02-12T16:56:15.976427+0000","last_clean":"2024-02-12T16:56:15.976427+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:56:15.976427+0000","last_undegraded":"2024-02-12T16:56:15.976427+0000","last_fullsized":"2024-02-12T16:56:15.976427+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523254+0000","last_change":"2024-02-12T16:54:40.629432+0000","last_active":"2024-02-12T16:56:15.523254+0000","last_peered":"2024-02-12T16:56:15.523254+0000","last_clean":"2024-02-12T16:56:15.523254+0000","last_became_active":"2024-02-12T16:54:40.629081+0000","last_became_peered":"2024-02-12T16:54:40.629081+0000","last_unstale":"2024-02-12T16:56:15.523254+0000","last_undegraded":"2024-02-12T16:56:15.523254+0000","last_fullsized":"2024-02-12T16:56:15.523254+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.285961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.285961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":156,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.522295+0000","last_change":"2024-02-12T16:54:41.249353+0000","last_active":"2024-02-12T16:56:15.522295+0000","last_peered":"2024-02-12T16:56:15.522295+0000","last_clean":"2024-02-12T16:56:15.522295+0000","last_became_active":"2024-02-12T16:54:40.639823+0000","last_became_peered":"2024-02-12T16:54:40.639823+0000","last_unstale":"2024-02-12T16:56:15.522295+0000","last_undegraded":"2024-02-12T16:56:15.522295+0000","last_fullsized":"2024-02-12T16:56:15.522295+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":145,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976375+0000","last_change":"2024-02-12T16:52:08.569339+0000","last_active":"2024-02-12T16:56:15.976375+0000","last_peered":"2024-02-12T16:56:15.976375+0000","last_clean":"2024-02-12T16:56:15.976375+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:56:15.976375+0000","last_undegraded":"2024-02-12T16:56:15.976375+0000","last_fullsized":"2024-02-12T16:56:15.976375+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:08.569246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:08.569246+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":127,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.978708+0000","last_change":"2024-02-12T16:54:03.996224+0000","last_active":"2024-02-12T16:56:15.978708+0000","last_peered":"2024-02-12T16:56:15.978708+0000","last_clean":"2024-02-12T16:56:15.978708+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:56:15.978708+0000","last_undegraded":"2024-02-12T16:56:15.978708+0000","last_fullsized":"2024-02-12T16:56:15.978708+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":147,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.523715+0000","last_change":"2024-02-12T16:54:40.630996+0000","last_active":"2024-02-12T16:56:15.523715+0000","last_peered":"2024-02-12T16:56:15.523715+0000","last_clean":"2024-02-12T16:56:15.523715+0000","last_became_active":"2024-02-12T16:54:40.630768+0000","last_became_peered":"2024-02-12T16:54:40.630768+0000","last_unstale":"2024-02-12T16:56:15.523715+0000","last_undegraded":"2024-02-12T16:56:15.523715+0000","last_fullsized":"2024-02-12T16:56:15.523715+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:00.294633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:00.294633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":156,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-02-12T16:56:15.976698+0000","last_change":"2024-02-12T16:52:40.091071+0000","last_active":"2024-02-12T16:56:15.976698+0000","last_peered":"2024-02-12T16:56:15.976698+0000","last_clean":"2024-02-12T16:56:15.976698+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:56:15.976698+0000","last_undegraded":"2024-02-12T16:56:15.976698+0000","last_fullsized":"2024-02-12T16:56:15.976698+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:52:40.090931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.090931+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":216,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-02-12T16:56:17.951485+0000","last_change":"2024-02-12T16:52:59.185115+0000","last_active":"2024-02-12T16:56:17.951485+0000","last_peered":"2024-02-12T16:56:17.951485+0000","last_clean":"2024-02-12T16:56:17.951485+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:56:17.951485+0000","last_undegraded":"2024-02-12T16:56:17.951485+0000","last_fullsized":"2024-02-12T16:56:17.951485+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":397,"num_read_kb":392,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234068,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":802816,"data_stored":410305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":2.0129999999999999,"5min":1.7310000000000001,"15min":0.89200000000000002},"min":{"1min":0.22600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":7.3239999999999998,"5min":36.439,"15min":36.439},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.5620000000000001,"15min":0.86699999999999999},"min":{"1min":0.24099999999999999,"5min":0.218,"15min":0.218},"max":{"1min":3.5609999999999999,"5min":36.490000000000002,"15min":36.490000000000002},"last":0.68700000000000006}]},{"osd":1,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":1.0369999999999999,"15min":0.70599999999999996},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":12.032,"5min":12.032,"15min":12.032},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.1480000000000001,"5min":1.022,"15min":0.70499999999999996},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":10.054,"5min":10.054,"15min":10.054},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.0860000000000001,"15min":0.68300000000000005},"min":{"1min":0.30199999999999999,"5min":0.245,"15min":0.245},"max":{"1min":7.1230000000000002,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.806,"5min":1.2350000000000001,"15min":0.75800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.9989999999999997,"5min":20.338000000000001,"15min":20.338000000000001},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":2.2429999999999999,"5min":1.6599999999999999,"15min":0.94399999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.1069999999999993,"5min":20.433,"15min":20.433},"last":0.63100000000000001},{"interface":"front","average":{"1min":2.5049999999999999,"5min":1.7270000000000001,"15min":0.95299999999999996},"min":{"1min":0.33400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":9.9659999999999993,"5min":20.393000000000001,"15min":20.393000000000001},"last":0.622}]},{"osd":4,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.59099999999999997,"15min":0.52600000000000002},"min":{"1min":0.26800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.997,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.972,"5min":2.972,"15min":2.972},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.58799999999999997,"15min":0.56100000000000005},"min":{"1min":0.27400000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.65800000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.60899999999999999,"15min":0.54100000000000004},"min":{"1min":0.317,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.042,"5min":0.69399999999999995,"15min":0.60899999999999999},"min":{"1min":0.309,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.5920000000000001,"5min":3.5920000000000001,"15min":3.5920000000000001},"last":0.70399999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528409,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310308,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434844,"statfs":{"total":95995035648,"available":95677280256,"internally_reserved":0,"allocated":815104,"data_stored":410453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3279999999999998,"5min":1.522,"15min":0.82999999999999996},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":10.949,"5min":10.949,"15min":10.949},"last":0.63300000000000001},{"interface":"front","average":{"1min":2.794,"5min":1.5680000000000001,"15min":0.84399999999999997},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.6050000000000004,"5min":10.56,"15min":10.56},"last":0.55100000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.4550000000000001,"15min":0.83199999999999996},"min":{"1min":0.252,"5min":0.251,"15min":0.251},"max":{"1min":3.7000000000000002,"5min":15.987,"15min":15.987},"last":0.54600000000000004},{"interface":"front","average":{"1min":2.3319999999999999,"5min":1.5740000000000001,"15min":0.88800000000000001},"min":{"1min":0.34200000000000003,"5min":0.25,"15min":0.25},"max":{"1min":5.9770000000000003,"5min":15.997,"15min":15.997},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":1.357,"15min":0.78900000000000003},"min":{"1min":0.22900000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":2.3090000000000002,"5min":1.333,"15min":0.78400000000000003},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.319,"5min":6.3680000000000003,"15min":6.3680000000000003},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.3280000000000001,"15min":0.77500000000000002},"min":{"1min":0.29199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.6900000000000004,"5min":11.212,"15min":11.212},"last":0.57199999999999995},{"interface":"front","average":{"1min":2.1059999999999999,"5min":1.6719999999999999,"15min":0.90500000000000003},"min":{"1min":0.32900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":6.0380000000000003,"5min":10.875999999999999,"15min":10.875999999999999},"last":0.57799999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.61399999999999999,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.217},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.353,"5min":0.68300000000000005,"15min":0.495},"min":{"1min":0.26600000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.624,"15min":0.48799999999999999},"min":{"1min":0.309,"5min":0.245,"15min":0.245},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.755,"15min":0.54100000000000004},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":3.5859999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.45400000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:55:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.68799999999999994,"15min":0.53200000000000003},"min":{"1min":0.31900000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.452,"5min":5.452,"15min":5.452},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.67300000000000004,"15min":0.53900000000000003},"min":{"1min":0.35799999999999998,"5min":0.29299999999999998,"15min":0.27900000000000003},"max":{"1min":2.407,"5min":3.9049999999999998,"15min":3.9049999999999998},"last":0.55900000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298028,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447124,"statfs":{"total":95995035648,"available":95689854976,"internally_reserved":0,"allocated":823296,"data_stored":411737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":5.2329999999999997,"5min":2.262,"15min":0.97799999999999998},"min":{"1min":0.20599999999999999,"5min":0.182,"15min":0.182},"max":{"1min":25.585000000000001,"5min":25.585000000000001,"15min":25.585000000000001},"last":0.371},{"interface":"front","average":{"1min":3.528,"5min":2.423,"15min":1.0409999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":14.92,"5min":26.798999999999999,"15min":26.798999999999999},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:55:49 2024","interfaces":[{"interface":"back","average":{"1min":4.1269999999999998,"5min":2.1520000000000001,"15min":0.96799999999999997},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.859,"5min":11.581,"15min":11.581},"last":0.44500000000000001},{"interface":"front","average":{"1min":2.403,"5min":2.4620000000000002,"15min":1.0820000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":8.6519999999999992,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.495}]},{"osd":3,"last update":"Mon Feb 12 16:56:14 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.552,"15min":1.155},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.5220000000000002,"5min":26.649999999999999,"15min":26.649999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":3.21,"5min":2.2999999999999998,"15min":1.0409999999999999},"min":{"1min":0.314,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":17.821000000000002,"5min":17.821000000000002,"15min":17.821000000000002},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":3.4540000000000002,"5min":1.5209999999999999,"15min":0.88100000000000001},"min":{"1min":0.39100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":8.8249999999999993,"5min":8.8249999999999993,"15min":8.8249999999999993},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.8439999999999999,"5min":2.0649999999999999,"15min":1.0880000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":9.6739999999999995,"5min":26.748999999999999,"15min":26.748999999999999},"last":0.54900000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":3.2000000000000002,"5min":1.7110000000000001,"15min":0.93999999999999995},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":8.7959999999999994,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":0.53400000000000003},{"interface":"front","average":{"1min":2.4009999999999998,"5min":2.2519999999999998,"15min":1.1499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.5960000000000001,"5min":26.722999999999999,"15min":26.722999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.9490000000000001,"15min":1.042},"min":{"1min":0.38600000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6509999999999998,"5min":13.834,"15min":13.834},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.2749999999999999,"5min":2.1200000000000001,"15min":1.1299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.5550000000000002,"5min":26.774999999999999,"15min":26.774999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:55:43 2024","interfaces":[{"interface":"back","average":{"1min":2.6429999999999998,"5min":1.613,"15min":0.97899999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":9.1940000000000008,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.55900000000000005},{"interface":"front","average":{"1min":4.0369999999999999,"5min":1.966,"15min":1.0880000000000001},"min":{"1min":0.42199999999999999,"5min":0.31,"15min":0.31},"max":{"1min":8.8829999999999991,"5min":8.8829999999999991,"15min":8.8829999999999991},"last":0.59499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302104,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93443048,"statfs":{"total":95995035648,"available":95685681152,"internally_reserved":0,"allocated":802816,"data_stored":410099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:55:23 2024","interfaces":[{"interface":"back","average":{"1min":3.919,"5min":1.708,"15min":0.78900000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":23.510999999999999,"5min":23.510999999999999,"15min":23.510999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":2.431,"5min":1.5089999999999999,"15min":0.71499999999999997},"min":{"1min":0.26000000000000001,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":10.186999999999999,"5min":10.186999999999999,"15min":10.186999999999999},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:55:53 2024","interfaces":[{"interface":"back","average":{"1min":3.714,"5min":2.3900000000000001,"15min":1.052},"min":{"1min":0.26000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":16.521000000000001,"5min":16.521000000000001,"15min":16.521000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":2.8290000000000002,"5min":1.9059999999999999,"15min":0.90300000000000002},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":9.9429999999999996,"5min":9.9429999999999996,"15min":9.9429999999999996},"last":0.64700000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:56:17 2024","interfaces":[{"interface":"back","average":{"1min":2.5329999999999999,"5min":2.5430000000000001,"15min":1.121},"min":{"1min":0.28799999999999998,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.3029999999999999,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.879,"5min":2.52,"15min":1.145},"min":{"1min":0.29799999999999999,"5min":0.28100000000000003,"15min":0.27600000000000002},"max":{"1min":7.2030000000000003,"5min":16.446000000000002,"15min":16.446000000000002},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:55:28 2024","interfaces":[{"interface":"back","average":{"1min":2.8879999999999999,"5min":1.5549999999999999,"15min":0.89800000000000002},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":16.027000000000001,"5min":16.027000000000001,"15min":16.027000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.927,"5min":1.2250000000000001,"15min":0.79400000000000004},"min":{"1min":0.39500000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.141,"5min":10.141,"15min":10.141},"last":0.83999999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":2.46,"5min":1.6739999999999999,"15min":0.93700000000000006},"min":{"1min":0.32100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.9930000000000003,"5min":7.4340000000000002,"15min":7.4340000000000002},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.6479999999999999,"15min":0.94299999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.8799999999999999,"5min":16.138999999999999,"15min":16.138999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:55:23 2024","interfaces":[{"interface":"back","average":{"1min":2.242,"5min":1.329,"15min":0.82499999999999996},"min":{"1min":0.38500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":9.9849999999999994,"5min":9.9849999999999994,"15min":9.9849999999999994},"last":0.871},{"interface":"front","average":{"1min":2.6579999999999999,"5min":1.5569999999999999,"15min":0.90500000000000003},"min":{"1min":0.36899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":21.062000000000001,"5min":21.062000000000001,"15min":21.062000000000001},"last":0.88900000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4180000000000001,"5min":1.6419999999999999,"15min":0.93799999999999994},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":16.175000000000001,"5min":16.175000000000001,"15min":16.175000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.714,"5min":1.2909999999999999,"15min":0.82699999999999996},"min":{"1min":0.35499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":10.025,"5min":10.025,"15min":10.025},"last":0.91200000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306196,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438956,"statfs":{"total":95995035648,"available":95681490944,"internally_reserved":0,"allocated":798720,"data_stored":410991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:49 2024","interfaces":[{"interface":"back","average":{"1min":4.1509999999999998,"5min":2.4340000000000002,"15min":1.069},"min":{"1min":0.27600000000000002,"5min":0.19,"15min":0.19},"max":{"1min":15.808,"5min":15.808,"15min":15.808},"last":0.26000000000000001},{"interface":"front","average":{"1min":2.9729999999999999,"5min":1.9019999999999999,"15min":0.90700000000000003},"min":{"1min":0.20499999999999999,"5min":0.158,"15min":0.158},"max":{"1min":8.6620000000000008,"5min":19.893000000000001,"15min":19.893000000000001},"last":0.28999999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:55:49 2024","interfaces":[{"interface":"back","average":{"1min":3.9220000000000002,"5min":2.46,"15min":1.109},"min":{"1min":0.22800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":17.739000000000001,"5min":17.739000000000001,"15min":17.739000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":4.2530000000000001,"5min":2.6080000000000001,"15min":1.1499999999999999},"min":{"1min":0.373,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":15.766999999999999,"5min":15.766999999999999,"15min":15.766999999999999},"last":0.34399999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":3.2909999999999999,"15min":1.4079999999999999},"min":{"1min":0.35999999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":13.596,"5min":51.579999999999998,"15min":51.579999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":2.0649999999999999,"5min":2.4569999999999999,"15min":1.143},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":13.521000000000001,"5min":48.988999999999997,"15min":48.988999999999997},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:55:39 2024","interfaces":[{"interface":"back","average":{"1min":3.4940000000000002,"5min":1.7370000000000001,"15min":0.999},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":15.737,"5min":15.737,"15min":15.737},"last":0.51800000000000002},{"interface":"front","average":{"1min":2.5529999999999999,"5min":1.3120000000000001,"15min":0.84699999999999998},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7390000000000008,"5min":9.2119999999999997,"15min":9.2119999999999997},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:55:56 2024","interfaces":[{"interface":"back","average":{"1min":3.3500000000000001,"5min":2.0369999999999999,"15min":1.169},"min":{"1min":0.32400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":15.699,"5min":15.699,"15min":15.699},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.988,"15min":1.0620000000000001},"min":{"1min":0.34200000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":8.4610000000000003,"5min":11.319000000000001,"15min":11.319000000000001},"last":0.53900000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.0449999999999999,"5min":2.3599999999999999,"15min":1.1870000000000001},"min":{"1min":0.40600000000000003,"5min":0.31,"15min":0.31},"max":{"1min":12.215,"5min":17.782,"15min":17.782},"last":0.56100000000000005},{"interface":"front","average":{"1min":2.226,"5min":1.736,"15min":1.056},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":13.446999999999999,"5min":13.446999999999999,"15min":13.446999999999999},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:55:49 2024","interfaces":[{"interface":"back","average":{"1min":2.4449999999999998,"5min":1.6279999999999999,"15min":1.1970000000000001},"min":{"1min":0.36299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":8.7880000000000003,"5min":9.2550000000000008,"15min":9.2550000000000008},"last":0.57799999999999996},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.9770000000000001,"15min":1.079},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":10.691000000000001,"5min":11.356,"15min":11.356},"last":0.58999999999999997}]}]},{"osd":3,"up_from":0,"seq":111669149799,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953571,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306196,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438956,"statfs":{"total":95995035648,"available":95681490944,"internally_reserved":0,"allocated":798720,"data_stored":410548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:32 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.4630000000000001,"15min":0.85799999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":7.9950000000000001,"5min":38.179000000000002,"15min":38.179000000000002},"last":0.503},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.51,"15min":0.84999999999999998},"min":{"1min":0.22700000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.202,"5min":38.151000000000003,"15min":38.151000000000003},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:55:32 2024","interfaces":[{"interface":"back","average":{"1min":1.907,"5min":1.05,"15min":0.70199999999999996},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.1879999999999997,"5min":7.673,"15min":7.673},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.913,"5min":1.0700000000000001,"15min":0.72099999999999997},"min":{"1min":0.254,"5min":0.249,"15min":0.249},"max":{"1min":6.367,"5min":6.367,"15min":6.367},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:55:32 2024","interfaces":[{"interface":"back","average":{"1min":2.1280000000000001,"5min":1.087,"15min":0.745},"min":{"1min":0.28799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.6349999999999998,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.3250000000000002,"5min":1.2050000000000001,"15min":0.78300000000000003},"min":{"1min":0.307,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:55:32 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":2.0449999999999999,"15min":1.0549999999999999},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.4539999999999997,"5min":38.170999999999999,"15min":38.170999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.609,"5min":1.744,"15min":0.95099999999999996},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.4859999999999998,"5min":38.311999999999998,"15min":38.311999999999998},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:55:54 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":0.76600000000000001,"15min":0.48299999999999998},"min":{"1min":0.27000000000000002,"5min":0.247,"15min":0.24299999999999999},"max":{"1min":5.4109999999999996,"5min":5.4109999999999996,"15min":5.4109999999999996},"last":0.433},{"interface":"front","average":{"1min":1.3,"5min":0.61799999999999999,"15min":0.42799999999999999},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:55:28 2024","interfaces":[{"interface":"back","average":{"1min":6.5350000000000001,"5min":1.6919999999999999,"15min":0.81000000000000005},"min":{"1min":0.27000000000000002,"5min":0.255,"15min":0.255},"max":{"1min":104.907,"5min":104.907,"15min":104.907},"last":0.52100000000000002},{"interface":"front","average":{"1min":6.5510000000000002,"5min":1.722,"15min":0.82499999999999996},"min":{"1min":0.28799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":105.395,"5min":105.395,"15min":105.395},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.57399999999999995,"15min":0.46800000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.2829999999999999,"5min":0.69199999999999995,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.7329999999999997,"5min":5.7329999999999997,"15min":5.7329999999999997},"last":0.58399999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757343,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298000,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447152,"statfs":{"total":95995035648,"available":95689883648,"internally_reserved":0,"allocated":794624,"data_stored":410018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.536,"15min":0.85999999999999999},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.1349999999999998,"5min":36.880000000000003,"15min":36.880000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":2.4769999999999999,"5min":1.7549999999999999,"15min":0.90900000000000003},"min":{"1min":0.32700000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":6.7919999999999998,"5min":36.954000000000001,"15min":36.954000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":1.427,"15min":0.81799999999999995},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":15.920999999999999,"5min":15.920999999999999,"15min":15.920999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":2.7200000000000002,"5min":1.571,"15min":0.85999999999999999},"min":{"1min":0.32400000000000001,"5min":0.29099999999999998,"15min":0.28100000000000003},"max":{"1min":11.48,"5min":12.221,"15min":12.221},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":1.5700000000000001,"15min":0.877},"min":{"1min":0.247,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":11.542999999999999,"5min":18.218,"15min":18.218},"last":0.59799999999999998},{"interface":"front","average":{"1min":2.1259999999999999,"5min":1.595,"15min":0.86599999999999999},"min":{"1min":0.33100000000000002,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":7.0869999999999997,"5min":18.327000000000002,"15min":18.327000000000002},"last":0.64800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.758,"15min":0.93799999999999994},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.3179999999999996,"5min":18.251000000000001,"15min":18.251000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.601,"5min":1.647,"15min":0.88900000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":6.024,"5min":20.263000000000002,"15min":20.263000000000002},"last":0.68799999999999994}]},{"osd":4,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.64600000000000002,"15min":0.46600000000000003},"min":{"1min":0.26500000000000001,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":4.3710000000000004,"5min":4.3710000000000004,"15min":4.3710000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.65100000000000002,"15min":0.47199999999999998},"min":{"1min":0.29999999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.1600000000000001,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":0.33100000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.55700000000000005},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.65100000000000002,"15min":0.51100000000000001},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.8159999999999998,"5min":2.827,"15min":2.827},"last":0.371}]},{"osd":7,"last update":"Mon Feb 12 16:55:45 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":0.72799999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.312},{"interface":"front","average":{"1min":1.202,"5min":0.752,"15min":0.57399999999999995},"min":{"1min":0.40999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.38900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:56:27.380 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:56:27.399 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:27.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:26.016896+0000 mgr.y (mgr.24481) 260 : audit [DBG] from='client.34985 v1:172.21.15.33:0/4159692869' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:56:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:26.336506+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:56:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:26.337450+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.7 scrub ok 2024-02-12T16:56:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:26.460324+0000 mgr.y (mgr.24481) 261 : cluster [DBG] pgmap v300: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:27.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:27.556 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:27.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:27.573 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:27.588 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:27.593 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:27.598 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:27.660 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:27.688 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:27.837 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:27.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:27.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:27.974 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:27.996 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:27.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:28.099 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:28.130 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:28.151 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:28.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:26.671346+0000 osd.2 (osd.2) 101 : cluster [DBG] 5.d scrub starts 2024-02-12T16:56:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:26.673478+0000 osd.2 (osd.2) 102 : cluster [DBG] 5.d scrub ok 2024-02-12T16:56:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:26.885764+0000 osd.7 (osd.7) 91 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:56:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:26.889435+0000 osd.7 (osd.7) 92 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:56:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:27.291689+0000 osd.6 (osd.6) 98 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:56:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:27.293677+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:56:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:27.838585+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.1e scrub starts 2024-02-12T16:56:28.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:27.839759+0000 osd.0 (osd.0) 86 : cluster [DBG] 2.1e scrub ok 2024-02-12T16:56:28.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:28.283 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:28.302 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:28.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:28.406 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:28.435 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:28.455 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:28.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:28.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:28.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:26.671346+0000 osd.2 (osd.2) 101 : cluster [DBG] 5.d scrub starts 2024-02-12T16:56:28.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:26.673478+0000 osd.2 (osd.2) 102 : cluster [DBG] 5.d scrub ok 2024-02-12T16:56:28.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:26.885764+0000 osd.7 (osd.7) 91 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:56:28.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:26.889435+0000 osd.7 (osd.7) 92 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:27.291689+0000 osd.6 (osd.6) 98 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:27.293677+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:27.838585+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.1e scrub starts 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:27.839759+0000 osd.0 (osd.0) 86 : cluster [DBG] 2.1e scrub ok 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:26.671346+0000 osd.2 (osd.2) 101 : cluster [DBG] 5.d scrub starts 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:26.673478+0000 osd.2 (osd.2) 102 : cluster [DBG] 5.d scrub ok 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:26.885764+0000 osd.7 (osd.7) 91 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:26.889435+0000 osd.7 (osd.7) 92 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:27.291689+0000 osd.6 (osd.6) 98 : cluster [DBG] 4.1f scrub starts 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:27.293677+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.1f scrub ok 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:27.838585+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.1e scrub starts 2024-02-12T16:56:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:27.839759+0000 osd.0 (osd.0) 86 : cluster [DBG] 2.1e scrub ok 2024-02-12T16:56:28.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:28.586 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:28.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:28.710 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:28.740 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:28.989 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:28.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:29.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:29.126 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:29.146 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:29.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:29.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:29.250 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:27.668174+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:56:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:27.696167+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:56:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:28.192515+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:56:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:28.195985+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:56:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:28.460877+0000 mgr.y (mgr.24481) 262 : cluster [DBG] pgmap v301: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T16:56:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:27.668174+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:56:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:27.696167+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:56:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:28.192515+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:56:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:28.195985+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:56:29.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:28.460877+0000 mgr.y (mgr.24481) 262 : cluster [DBG] pgmap v301: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T16:56:29.279 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:29.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:29.403 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:27.668174+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.17 scrub starts 2024-02-12T16:56:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:27.696167+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.17 scrub ok 2024-02-12T16:56:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:28.192515+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:56:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:28.195985+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:56:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:28.460877+0000 mgr.y (mgr.24481) 262 : cluster [DBG] pgmap v301: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T16:56:30.184 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:30.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:29.857384+0000 osd.0 (osd.0) 87 : cluster [DBG] 6.11 scrub starts 2024-02-12T16:56:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:29.870071+0000 osd.0 (osd.0) 88 : cluster [DBG] 6.11 scrub ok 2024-02-12T16:56:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:29.857384+0000 osd.0 (osd.0) 87 : cluster [DBG] 6.11 scrub starts 2024-02-12T16:56:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:29.870071+0000 osd.0 (osd.0) 88 : cluster [DBG] 6.11 scrub ok 2024-02-12T16:56:30.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:30.322 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:30.343 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:30.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:30.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:30.475 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:29.857384+0000 osd.0 (osd.0) 87 : cluster [DBG] 6.11 scrub starts 2024-02-12T16:56:30.475 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:29.870071+0000 osd.0 (osd.0) 88 : cluster [DBG] 6.11 scrub ok 2024-02-12T16:56:30.477 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:30.497 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:30.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:30.601 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:30.628 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:30.647 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:30.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:30.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:30.778 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:30.797 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:30.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:30.901 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:30.931 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:30.984 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:30.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:31.099 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:31.125 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:31.137 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:56:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:30] "GET /metrics HTTP/1.1" 200 175652 "" "Prometheus/2.33.4" 2024-02-12T16:56:31.144 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:31.146 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T16:56:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:31.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:31.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:31.293 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:31.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:31.396 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:31.418 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:30.145410+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:56:31.418 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:30.146639+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:30.461298+0000 mgr.y (mgr.24481) 263 : cluster [DBG] pgmap v302: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:30.627565+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.33:0/3166878222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:30.627873+0000 mon.a (mon.0) 952 : audit [INF] from='client.34991 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:30.809177+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:30.812676+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:30.145410+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:30.146639+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:30.461298+0000 mgr.y (mgr.24481) 263 : cluster [DBG] pgmap v302: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:30.627565+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.33:0/3166878222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:30.627873+0000 mon.a (mon.0) 952 : audit [INF] from='client.34991 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:30.809177+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:56:31.419 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:30.812676+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:56:31.425 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:30.145410+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.19 scrub starts 2024-02-12T16:56:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:30.146639+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.19 scrub ok 2024-02-12T16:56:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:30.461298+0000 mgr.y (mgr.24481) 263 : cluster [DBG] pgmap v302: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T16:56:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:30.627565+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.33:0/3166878222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:30.627873+0000 mon.a (mon.0) 952 : audit [INF] from='client.34991 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:56:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:30.809177+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:56:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:30.812676+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:56:31.676 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:31.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:31.780 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:31.808 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:31.828 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:31.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:31.931 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:31.959 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:31.977 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:31.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:32.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:32.108 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:32.126 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:32.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:32.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:32.256 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:32.274 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:32.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:32.378 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:32.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:30.712518+0000 osd.2 (osd.2) 105 : cluster [DBG] 4.d scrub starts 2024-02-12T16:56:32.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:30.714424+0000 osd.2 (osd.2) 106 : cluster [DBG] 4.d scrub ok 2024-02-12T16:56:32.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.112568+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:56:32.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.122553+0000 osd.1 (osd.1) 121 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:56:32.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.129734+0000 mon.a (mon.0) 953 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:56:32.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.129760+0000 mon.a (mon.0) 954 : cluster [INF] Cluster is now healthy 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:31.138602+0000 mon.a (mon.0) 955 : audit [INF] from='client.34991 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.138666+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.833309+0000 osd.0 (osd.0) 91 : cluster [DBG] 4.b scrub starts 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.836769+0000 osd.0 (osd.0) 92 : cluster [DBG] 4.b scrub ok 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:30.712518+0000 osd.2 (osd.2) 105 : cluster [DBG] 4.d scrub starts 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:30.714424+0000 osd.2 (osd.2) 106 : cluster [DBG] 4.d scrub ok 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.112568+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.122553+0000 osd.1 (osd.1) 121 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.129734+0000 mon.a (mon.0) 953 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:56:32.407 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.129760+0000 mon.a (mon.0) 954 : cluster [INF] Cluster is now healthy 2024-02-12T16:56:32.408 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:31.138602+0000 mon.a (mon.0) 955 : audit [INF] from='client.34991 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:56:32.408 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.138666+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-12T16:56:32.408 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.833309+0000 osd.0 (osd.0) 91 : cluster [DBG] 4.b scrub starts 2024-02-12T16:56:32.408 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.836769+0000 osd.0 (osd.0) 92 : cluster [DBG] 4.b scrub ok 2024-02-12T16:56:32.408 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:30.712518+0000 osd.2 (osd.2) 105 : cluster [DBG] 4.d scrub starts 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:30.714424+0000 osd.2 (osd.2) 106 : cluster [DBG] 4.d scrub ok 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.112568+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1d scrub starts 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.122553+0000 osd.1 (osd.1) 121 : cluster [DBG] 6.1d scrub ok 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.129734+0000 mon.a (mon.0) 953 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.129760+0000 mon.a (mon.0) 954 : cluster [INF] Cluster is now healthy 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:31.138602+0000 mon.a (mon.0) 955 : audit [INF] from='client.34991 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.138666+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.833309+0000 osd.0 (osd.0) 91 : cluster [DBG] 4.b scrub starts 2024-02-12T16:56:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.836769+0000 osd.0 (osd.0) 92 : cluster [DBG] 4.b scrub ok 2024-02-12T16:56:32.501 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:32.507 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:32.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:32.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:32.611 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:32.640 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:32.716 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:32.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:32.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:32.820 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:32.852 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:32.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:56:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:56:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.940406+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.1b scrub starts 2024-02-12T16:56:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:31.965913+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.1b scrub ok 2024-02-12T16:56:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.099025+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:56:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.102018+0000 osd.1 (osd.1) 123 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:56:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.291320+0000 osd.5 (osd.5) 90 : cluster [DBG] 5.e scrub starts 2024-02-12T16:56:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.292721+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.e scrub ok 2024-02-12T16:56:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.461580+0000 mgr.y (mgr.24481) 264 : cluster [DBG] pgmap v304: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:32.465577+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.823286+0000 osd.0 (osd.0) 93 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.825178+0000 osd.0 (osd.0) 94 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.940406+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.1b scrub starts 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:31.965913+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.1b scrub ok 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.099025+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.102018+0000 osd.1 (osd.1) 123 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.291320+0000 osd.5 (osd.5) 90 : cluster [DBG] 5.e scrub starts 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.292721+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.e scrub ok 2024-02-12T16:56:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.461580+0000 mgr.y (mgr.24481) 264 : cluster [DBG] pgmap v304: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:56:33.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:32.465577+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:33.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.823286+0000 osd.0 (osd.0) 93 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:56:33.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.825178+0000 osd.0 (osd.0) 94 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:56:33.404 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:33.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.940406+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.1b scrub starts 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:31.965913+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.1b scrub ok 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.099025+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.1d scrub starts 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.102018+0000 osd.1 (osd.1) 123 : cluster [DBG] 5.1d scrub ok 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.291320+0000 osd.5 (osd.5) 90 : cluster [DBG] 5.e scrub starts 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.292721+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.e scrub ok 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.461580+0000 mgr.y (mgr.24481) 264 : cluster [DBG] pgmap v304: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:32.465577+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.823286+0000 osd.0 (osd.0) 93 : cluster [DBG] 4.1e scrub starts 2024-02-12T16:56:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.825178+0000 osd.0 (osd.0) 94 : cluster [DBG] 4.1e scrub ok 2024-02-12T16:56:33.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:33.542 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:33.562 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:33.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:33.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:33.697 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:33.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:56:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:56:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.716197+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-02-12T16:56:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:32.717098+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-02-12T16:56:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:33.049506+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.c deep-scrub starts 2024-02-12T16:56:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:33.052439+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.c deep-scrub ok 2024-02-12T16:56:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:33.290986+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:56:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:33.291985+0000 osd.6 (osd.6) 101 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:56:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:33.681426+0000 mgr.y (mgr.24481) 265 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.716197+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-02-12T16:56:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:32.717098+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-02-12T16:56:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:33.049506+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.c deep-scrub starts 2024-02-12T16:56:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:33.052439+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.c deep-scrub ok 2024-02-12T16:56:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:33.290986+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:56:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:33.291985+0000 osd.6 (osd.6) 101 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:56:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:33.681426+0000 mgr.y (mgr.24481) 265 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.716197+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-02-12T16:56:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:32.717098+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-02-12T16:56:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:33.049506+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.c deep-scrub starts 2024-02-12T16:56:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:33.052439+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.c deep-scrub ok 2024-02-12T16:56:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:33.290986+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.19 scrub starts 2024-02-12T16:56:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:33.291985+0000 osd.6 (osd.6) 101 : cluster [DBG] 2.19 scrub ok 2024-02-12T16:56:34.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:33.681426+0000 mgr.y (mgr.24481) 265 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:34.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:35.081 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:35.110 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:35.212 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:35.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:35.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:35.343 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:35.446 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_1' created 2024-02-12T16:56:35.456 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:35.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:35.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:34.423103+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:34.426795+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:34.462500+0000 mgr.y (mgr.24481) 266 : cluster [DBG] pgmap v305: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:34.942351+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.33:0/2287230686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:34.942676+0000 mon.a (mon.0) 958 : audit [INF] from='client.34997 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:34.423103+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:34.426795+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:34.462500+0000 mgr.y (mgr.24481) 266 : cluster [DBG] pgmap v305: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:34.942351+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.33:0/2287230686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:56:35.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:34.942676+0000 mon.a (mon.0) 958 : audit [INF] from='client.34997 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:56:35.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:35.592 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:35.613 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:35.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:56:35.671 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:56:35.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:34.423103+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:56:35.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:34.426795+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:56:35.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:34.462500+0000 mgr.y (mgr.24481) 266 : cluster [DBG] pgmap v305: 132 pgs: 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T16:56:35.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:34.942351+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.33:0/2287230686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:56:35.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:34.942676+0000 mon.a (mon.0) 958 : audit [INF] from='client.34997 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:56:35.746 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:35.766 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:35.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:35.870 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:35.899 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:35.919 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:35.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:36.014 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:56:36.014 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:56:36.023 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:36.025 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:56:36.050 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:36.069 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:36.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:36.173 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:36.206 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:36.259 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:36.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:36.364 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:36.392 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:36.411 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:36.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:36.523 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.096162+0000 osd.1 (osd.1) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.096217+0000 osd.1 (osd.1) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:35.438716+0000 mon.a (mon.0) 959 : audit [INF] from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.438793+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:35.650772+0000 mgr.y (mgr.24481) 267 : audit [DBG] from='client.35003 v1:172.21.15.33:0/3998345494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.852376+0000 osd.0 (osd.0) 95 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.853326+0000 osd.0 (osd.0) 96 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.894159+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.897081+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.979254+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:35.979340+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:36.440361+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.096162+0000 osd.1 (osd.1) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.096217+0000 osd.1 (osd.1) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:35.438716+0000 mon.a (mon.0) 959 : audit [INF] from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.438793+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:35.650772+0000 mgr.y (mgr.24481) 267 : audit [DBG] from='client.35003 v1:172.21.15.33:0/3998345494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.852376+0000 osd.0 (osd.0) 95 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.853326+0000 osd.0 (osd.0) 96 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.894159+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.897081+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.979254+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:35.979340+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:36.440361+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-12T16:56:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:36.568 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:36.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:36.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:36.697 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:36.715 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:36.717 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.096162+0000 osd.1 (osd.1) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:36.717 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.096217+0000 osd.1 (osd.1) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:35.438716+0000 mon.a (mon.0) 959 : audit [INF] from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.438793+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:35.650772+0000 mgr.y (mgr.24481) 267 : audit [DBG] from='client.35003 v1:172.21.15.33:0/3998345494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.852376+0000 osd.0 (osd.0) 95 : cluster [DBG] 2.1f scrub starts 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.853326+0000 osd.0 (osd.0) 96 : cluster [DBG] 2.1f scrub ok 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.894159+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.897081+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.979254+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:35.979340+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:36.718 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:36.440361+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-12T16:56:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:36.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:36.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:36.845 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:36.865 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:36.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:36.969 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:36.995 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:37.013 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:37.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:37.117 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:37.146 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:37.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:36.209622+0000 osd.6 (osd.6) 102 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:56:37.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:36.211095+0000 osd.6 (osd.6) 103 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:56:37.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:36.462829+0000 mgr.y (mgr.24481) 268 : cluster [DBG] pgmap v308: 148 pgs: 16 unknown, 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:37.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:37.154522+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:56:37.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:37.155307+0000 osd.5 (osd.5) 93 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:56:37.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:37.441474+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-12T16:56:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:36.209622+0000 osd.6 (osd.6) 102 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:56:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:36.211095+0000 osd.6 (osd.6) 103 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:56:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:36.462829+0000 mgr.y (mgr.24481) 268 : cluster [DBG] pgmap v308: 148 pgs: 16 unknown, 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:37.154522+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:56:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:37.155307+0000 osd.5 (osd.5) 93 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:56:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:37.441474+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-12T16:56:37.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:36.209622+0000 osd.6 (osd.6) 102 : cluster [DBG] 5.1f scrub starts 2024-02-12T16:56:37.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:36.211095+0000 osd.6 (osd.6) 103 : cluster [DBG] 5.1f scrub ok 2024-02-12T16:56:37.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:36.462829+0000 mgr.y (mgr.24481) 268 : cluster [DBG] pgmap v308: 148 pgs: 16 unknown, 132 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:37.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:37.154522+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.16 scrub starts 2024-02-12T16:56:37.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:37.155307+0000 osd.5 (osd.5) 93 : cluster [DBG] 5.16 scrub ok 2024-02-12T16:56:37.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:37.441474+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-12T16:56:38.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:37.006225+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:56:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:37.009091+0000 osd.1 (osd.1) 129 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:56:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:37.006225+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:56:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:37.009091+0000 osd.1 (osd.1) 129 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:56:38.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:37.006225+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.4 scrub starts 2024-02-12T16:56:38.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:37.009091+0000 osd.1 (osd.1) 129 : cluster [DBG] 2.4 scrub ok 2024-02-12T16:56:39.083 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-02-12T16:56:35.436393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":1114,"num_read_kb":866,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":3321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":495,"ondisk_log_size":495,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2119120,"kb_used_data":5784,"kb_used_omap":0,"kb_used_meta":2113280,"kb_avail":654096944,"statfs":{"total":671965249536,"available":669795270656,"internally_reserved":0,"allocated":5922816,"data_stored":3013864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2163998720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.976233"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":54,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.966054+0000","last_change":"2024-02-12T16:56:31.966054+0000","last_active":"2024-02-12T16:56:31.966054+0000","last_peered":"2024-02-12T16:56:31.966054+0000","last_clean":"2024-02-12T16:56:31.966054+0000","last_became_active":"2024-02-12T16:54:40.643598+0000","last_became_peered":"2024-02-12T16:54:40.643598+0000","last_unstale":"2024-02-12T16:56:31.966054+0000","last_undegraded":"2024-02-12T16:56:31.966054+0000","last_fullsized":"2024-02-12T16:56:31.966054+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":99,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853064+0000","last_change":"2024-02-12T16:53:49.715871+0000","last_active":"2024-02-12T16:56:24.853064+0000","last_peered":"2024-02-12T16:56:24.853064+0000","last_clean":"2024-02-12T16:56:24.853064+0000","last_became_active":"2024-02-12T16:52:20.563961+0000","last_became_peered":"2024-02-12T16:52:20.563961+0000","last_unstale":"2024-02-12T16:56:24.853064+0000","last_undegraded":"2024-02-12T16:56:24.853064+0000","last_fullsized":"2024-02-12T16:56:24.853064+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:49.715809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.715809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":134,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.817060+0000","last_change":"2024-02-12T16:52:41.990018+0000","last_active":"2024-02-12T16:56:24.817060+0000","last_peered":"2024-02-12T16:56:24.817060+0000","last_clean":"2024-02-12T16:56:24.817060+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:56:24.817060+0000","last_undegraded":"2024-02-12T16:56:24.817060+0000","last_fullsized":"2024-02-12T16:56:24.817060+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:41.989918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.989918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":107,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706957+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T16:56:24.706957+0000","last_peered":"2024-02-12T16:56:24.706957+0000","last_clean":"2024-02-12T16:56:24.706957+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T16:56:24.706957+0000","last_undegraded":"2024-02-12T16:56:24.706957+0000","last_fullsized":"2024-02-12T16:56:24.706957+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":94,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.854299+0000","last_change":"2024-02-12T16:53:44.946159+0000","last_active":"2024-02-12T16:56:24.854299+0000","last_peered":"2024-02-12T16:56:24.854299+0000","last_clean":"2024-02-12T16:56:24.854299+0000","last_became_active":"2024-02-12T16:52:20.561617+0000","last_became_peered":"2024-02-12T16:52:20.561617+0000","last_unstale":"2024-02-12T16:56:24.854299+0000","last_undegraded":"2024-02-12T16:56:24.854299+0000","last_fullsized":"2024-02-12T16:56:24.854299+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:30.146733+0000","last_change":"2024-02-12T16:56:30.146707+0000","last_active":"2024-02-12T16:56:30.146733+0000","last_peered":"2024-02-12T16:56:30.146733+0000","last_clean":"2024-02-12T16:56:30.146733+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:56:30.146733+0000","last_undegraded":"2024-02-12T16:56:30.146733+0000","last_fullsized":"2024-02-12T16:56:30.146733+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:30.146643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.146643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":54,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:27.839878+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T16:56:27.839878+0000","last_peered":"2024-02-12T16:56:27.839878+0000","last_clean":"2024-02-12T16:56:27.839878+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T16:56:27.839878+0000","last_undegraded":"2024-02-12T16:56:27.839878+0000","last_fullsized":"2024-02-12T16:56:27.839878+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":156,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.703995+0000","last_change":"2024-02-12T16:55:33.609901+0000","last_active":"2024-02-12T16:56:24.703995+0000","last_peered":"2024-02-12T16:56:24.703995+0000","last_clean":"2024-02-12T16:56:24.703995+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:56:24.703995+0000","last_undegraded":"2024-02-12T16:56:24.703995+0000","last_fullsized":"2024-02-12T16:56:24.703995+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:55:33.609834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:33.609834+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":139,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853596+0000","last_change":"2024-02-12T16:54:40.628366+0000","last_active":"2024-02-12T16:56:24.853596+0000","last_peered":"2024-02-12T16:56:24.853596+0000","last_clean":"2024-02-12T16:56:24.853596+0000","last_became_active":"2024-02-12T16:54:40.628037+0000","last_became_peered":"2024-02-12T16:54:40.628037+0000","last_unstale":"2024-02-12T16:56:24.853596+0000","last_undegraded":"2024-02-12T16:56:24.853596+0000","last_fullsized":"2024-02-12T16:56:24.853596+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:43.089004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.089004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851050+0000","last_change":"2024-02-12T16:52:16.185645+0000","last_active":"2024-02-12T16:56:24.851050+0000","last_peered":"2024-02-12T16:56:24.851050+0000","last_clean":"2024-02-12T16:56:24.851050+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:56:24.851050+0000","last_undegraded":"2024-02-12T16:56:24.851050+0000","last_fullsized":"2024-02-12T16:56:24.851050+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:16.185582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:52:16.185582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":147,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851378+0000","last_change":"2024-02-12T16:54:38.784259+0000","last_active":"2024-02-12T16:56:24.851378+0000","last_peered":"2024-02-12T16:56:24.851378+0000","last_clean":"2024-02-12T16:56:24.851378+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:56:24.851378+0000","last_undegraded":"2024-02-12T16:56:24.851378+0000","last_fullsized":"2024-02-12T16:56:24.851378+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":77,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.705299+0000","last_change":"2024-02-12T16:54:41.193544+0000","last_active":"2024-02-12T16:56:24.705299+0000","last_peered":"2024-02-12T16:56:24.705299+0000","last_clean":"2024-02-12T16:56:24.705299+0000","last_became_active":"2024-02-12T16:54:40.639106+0000","last_became_peered":"2024-02-12T16:54:40.639106+0000","last_unstale":"2024-02-12T16:56:24.705299+0000","last_undegraded":"2024-02-12T16:56:24.705299+0000","last_fullsized":"2024-02-12T16:56:24.705299+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:53:45.658292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.658292+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.815979+0000","last_change":"2024-02-12T16:53:51.204401+0000","last_active":"2024-02-12T16:56:24.815979+0000","last_peered":"2024-02-12T16:56:24.815979+0000","last_clean":"2024-02-12T16:56:24.815979+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:56:24.815979+0000","last_undegraded":"2024-02-12T16:56:24.815979+0000","last_fullsized":"2024-02-12T16:56:24.815979+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.854451+0000","last_change":"2024-02-12T16:56:24.210155+0000","last_active":"2024-02-12T16:56:24.854451+0000","last_peered":"2024-02-12T16:56:24.854451+0000","last_clean":"2024-02-12T16:56:24.854451+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:56:24.854451+0000","last_undegraded":"2024-02-12T16:56:24.854451+0000","last_fullsized":"2024-02-12T16:56:24.854451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:24.210075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.210075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":134,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851363+0000","last_change":"2024-02-12T16:52:10.220443+0000","last_active":"2024-02-12T16:56:24.851363+0000","last_peered":"2024-02-12T16:56:24.851363+0000","last_clean":"2024-02-12T16:56:24.851363+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:56:24.851363+0000","last_undegraded":"2024-02-12T16:56:24.851363+0000","last_fullsized":"2024-02-12T16:56:24.851363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:10.220358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:10.220358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":87,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.815598+0000","last_change":"2024-02-12T16:54:41.013328+0000","last_active":"2024-02-12T16:56:24.815598+0000","last_peered":"2024-02-12T16:56:24.815598+0000","last_clean":"2024-02-12T16:56:24.815598+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T16:56:24.815598+0000","last_undegraded":"2024-02-12T16:56:24.815598+0000","last_fullsized":"2024-02-12T16:56:24.815598+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:48.029049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.029049+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":56,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853543+0000","last_change":"2024-02-12T16:56:23.345846+0000","last_active":"2024-02-12T16:56:24.853543+0000","last_peered":"2024-02-12T16:56:24.853543+0000","last_clean":"2024-02-12T16:56:24.853543+0000","last_became_active":"2024-02-12T16:54:40.628201+0000","last_became_peered":"2024-02-12T16:54:40.628201+0000","last_unstale":"2024-02-12T16:56:24.853543+0000","last_undegraded":"2024-02-12T16:56:24.853543+0000","last_fullsized":"2024-02-12T16:56:24.853543+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":52,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708450+0000","last_change":"2024-02-12T16:54:40.629069+0000","last_active":"2024-02-12T16:56:24.708450+0000","last_peered":"2024-02-12T16:56:24.708450+0000","last_clean":"2024-02-12T16:56:24.708450+0000","last_became_active":"2024-02-12T16:54:40.628744+0000","last_became_peered":"2024-02-12T16:54:40.628744+0000","last_unstale":"2024-02-12T16:56:24.708450+0000","last_undegraded":"2024-02-12T16:56:24.708450+0000","last_fullsized":"2024-02-12T16:56:24.708450+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:40.022594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.022594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":49,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.850737+0000","last_change":"2024-02-12T16:54:40.633472+0000","last_active":"2024-02-12T16:56:24.850737+0000","last_peered":"2024-02-12T16:56:24.850737+0000","last_clean":"2024-02-12T16:56:24.850737+0000","last_became_active":"2024-02-12T16:54:40.633129+0000","last_became_peered":"2024-02-12T16:54:40.633129+0000","last_unstale":"2024-02-12T16:56:24.850737+0000","last_undegraded":"2024-02-12T16:56:24.850737+0000","last_fullsized":"2024-02-12T16:56:24.850737+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:03.947783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:03.947783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":96,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.705505+0000","last_change":"2024-02-12T16:54:41.077705+0000","last_active":"2024-02-12T16:56:24.705505+0000","last_peered":"2024-02-12T16:56:24.705505+0000","last_clean":"2024-02-12T16:56:24.705505+0000","last_became_active":"2024-02-12T16:54:40.626676+0000","last_became_peered":"2024-02-12T16:54:40.626676+0000","last_unstale":"2024-02-12T16:56:24.705505+0000","last_undegraded":"2024-02-12T16:56:24.705505+0000","last_fullsized":"2024-02-12T16:56:24.705505+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:49.624070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:49.624070+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":140,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.705590+0000","last_change":"2024-02-12T16:56:18.464298+0000","last_active":"2024-02-12T16:56:24.705590+0000","last_peered":"2024-02-12T16:56:24.705590+0000","last_clean":"2024-02-12T16:56:24.705590+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:56:24.705590+0000","last_undegraded":"2024-02-12T16:56:24.705590+0000","last_fullsized":"2024-02-12T16:56:24.705590+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:32.102101+0000","last_change":"2024-02-12T16:56:32.102101+0000","last_active":"2024-02-12T16:56:32.102101+0000","last_peered":"2024-02-12T16:56:32.102101+0000","last_clean":"2024-02-12T16:56:32.102101+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:56:32.102101+0000","last_undegraded":"2024-02-12T16:56:32.102101+0000","last_fullsized":"2024-02-12T16:56:32.102101+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.102022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.102022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":145,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:25.339851+0000","last_change":"2024-02-12T16:56:25.339812+0000","last_active":"2024-02-12T16:56:25.339851+0000","last_peered":"2024-02-12T16:56:25.339851+0000","last_clean":"2024-02-12T16:56:25.339851+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:56:25.339851+0000","last_undegraded":"2024-02-12T16:56:25.339851+0000","last_fullsized":"2024-02-12T16:56:25.339851+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":193,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708993+0000","last_change":"2024-02-12T16:54:40.681231+0000","last_active":"2024-02-12T16:56:24.708993+0000","last_peered":"2024-02-12T16:56:24.708993+0000","last_clean":"2024-02-12T16:56:24.708993+0000","last_became_active":"2024-02-12T16:54:40.630167+0000","last_became_peered":"2024-02-12T16:54:40.630167+0000","last_unstale":"2024-02-12T16:56:24.708993+0000","last_undegraded":"2024-02-12T16:56:24.708993+0000","last_fullsized":"2024-02-12T16:56:24.708993+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:53:48.468547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.468547+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":135,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.122670+0000","last_change":"2024-02-12T16:56:31.122644+0000","last_active":"2024-02-12T16:56:31.122670+0000","last_peered":"2024-02-12T16:56:31.122670+0000","last_clean":"2024-02-12T16:56:31.122670+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:56:31.122670+0000","last_undegraded":"2024-02-12T16:56:31.122670+0000","last_fullsized":"2024-02-12T16:56:31.122670+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.817381+0000","last_change":"2024-02-12T16:53:04.085929+0000","last_active":"2024-02-12T16:56:24.817381+0000","last_peered":"2024-02-12T16:56:24.817381+0000","last_clean":"2024-02-12T16:56:24.817381+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:56:24.817381+0000","last_undegraded":"2024-02-12T16:56:24.817381+0000","last_fullsized":"2024-02-12T16:56:24.817381+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:04.085854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.085854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":54,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:33.292072+0000","last_change":"2024-02-12T16:56:33.292072+0000","last_active":"2024-02-12T16:56:33.292072+0000","last_peered":"2024-02-12T16:56:33.292072+0000","last_clean":"2024-02-12T16:56:33.292072+0000","last_became_active":"2024-02-12T16:54:40.640452+0000","last_became_peered":"2024-02-12T16:54:40.640452+0000","last_unstale":"2024-02-12T16:56:33.292072+0000","last_undegraded":"2024-02-12T16:56:33.292072+0000","last_fullsized":"2024-02-12T16:56:33.292072+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":157,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:27.293799+0000","last_change":"2024-02-12T16:56:27.293762+0000","last_active":"2024-02-12T16:56:27.293799+0000","last_peered":"2024-02-12T16:56:27.293799+0000","last_clean":"2024-02-12T16:56:27.293799+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:56:27.293799+0000","last_undegraded":"2024-02-12T16:56:27.293799+0000","last_fullsized":"2024-02-12T16:56:27.293799+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851655+0000","last_change":"2024-02-12T16:53:04.169987+0000","last_active":"2024-02-12T16:56:24.851655+0000","last_peered":"2024-02-12T16:56:24.851655+0000","last_clean":"2024-02-12T16:56:24.851655+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:56:24.851655+0000","last_undegraded":"2024-02-12T16:56:24.851655+0000","last_fullsized":"2024-02-12T16:56:24.851655+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:53:04.169892+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.169892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":225,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.136427+0000","last_change":"2024-02-12T16:52:44.602751+0000","last_active":"2024-02-12T16:56:31.136427+0000","last_peered":"2024-02-12T16:56:31.136427+0000","last_clean":"2024-02-12T16:56:31.136427+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:56:31.136427+0000","last_undegraded":"2024-02-12T16:56:31.136427+0000","last_fullsized":"2024-02-12T16:56:31.136427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:44.602656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.602656+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.852902+0000","last_change":"2024-02-12T16:52:47.997796+0000","last_active":"2024-02-12T16:56:24.852902+0000","last_peered":"2024-02-12T16:56:24.852902+0000","last_clean":"2024-02-12T16:56:24.852902+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:56:24.852902+0000","last_undegraded":"2024-02-12T16:56:24.852902+0000","last_fullsized":"2024-02-12T16:56:24.852902+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:47.997708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.997708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":173,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:32.825256+0000","last_change":"2024-02-12T16:56:32.825256+0000","last_active":"2024-02-12T16:56:32.825256+0000","last_peered":"2024-02-12T16:56:32.825256+0000","last_clean":"2024-02-12T16:56:32.825256+0000","last_became_active":"2024-02-12T16:54:40.650318+0000","last_became_peered":"2024-02-12T16:54:40.650318+0000","last_unstale":"2024-02-12T16:56:32.825256+0000","last_undegraded":"2024-02-12T16:56:32.825256+0000","last_fullsized":"2024-02-12T16:56:32.825256+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":49,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.816503+0000","last_change":"2024-02-12T16:54:40.650920+0000","last_active":"2024-02-12T16:56:24.816503+0000","last_peered":"2024-02-12T16:56:24.816503+0000","last_clean":"2024-02-12T16:56:24.816503+0000","last_became_active":"2024-02-12T16:54:40.650168+0000","last_became_peered":"2024-02-12T16:54:40.650168+0000","last_unstale":"2024-02-12T16:56:24.816503+0000","last_undegraded":"2024-02-12T16:56:24.816503+0000","last_fullsized":"2024-02-12T16:56:24.816503+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.665249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.665249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851047+0000","last_change":"2024-02-12T16:53:05.141324+0000","last_active":"2024-02-12T16:56:24.851047+0000","last_peered":"2024-02-12T16:56:24.851047+0000","last_clean":"2024-02-12T16:56:24.851047+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:56:24.851047+0000","last_undegraded":"2024-02-12T16:56:24.851047+0000","last_fullsized":"2024-02-12T16:56:24.851047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":140,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853163+0000","last_change":"2024-02-12T16:54:39.665472+0000","last_active":"2024-02-12T16:56:24.853163+0000","last_peered":"2024-02-12T16:56:24.853163+0000","last_clean":"2024-02-12T16:56:24.853163+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:56:24.853163+0000","last_undegraded":"2024-02-12T16:56:24.853163+0000","last_fullsized":"2024-02-12T16:56:24.853163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.665378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.665378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":97,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.854646+0000","last_change":"2024-02-12T16:56:24.854608+0000","last_active":"2024-02-12T16:56:24.854646+0000","last_peered":"2024-02-12T16:56:24.854646+0000","last_clean":"2024-02-12T16:56:24.854646+0000","last_became_active":"2024-02-12T16:54:40.632501+0000","last_became_peered":"2024-02-12T16:54:40.632501+0000","last_unstale":"2024-02-12T16:56:24.854646+0000","last_undegraded":"2024-02-12T16:56:24.854646+0000","last_fullsized":"2024-02-12T16:56:24.854646+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":128,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851125+0000","last_change":"2024-02-12T16:52:55.241717+0000","last_active":"2024-02-12T16:56:24.851125+0000","last_peered":"2024-02-12T16:56:24.851125+0000","last_clean":"2024-02-12T16:56:24.851125+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:56:24.851125+0000","last_undegraded":"2024-02-12T16:56:24.851125+0000","last_fullsized":"2024-02-12T16:56:24.851125+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.241622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.241622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853004+0000","last_change":"2024-02-12T16:53:48.719406+0000","last_active":"2024-02-12T16:56:24.853004+0000","last_peered":"2024-02-12T16:56:24.853004+0000","last_clean":"2024-02-12T16:56:24.853004+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:56:24.853004+0000","last_undegraded":"2024-02-12T16:56:24.853004+0000","last_fullsized":"2024-02-12T16:56:24.853004+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.719344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.719344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.855367+0000","last_change":"2024-02-12T16:52:49.986507+0000","last_active":"2024-02-12T16:56:24.855367+0000","last_peered":"2024-02-12T16:56:24.855367+0000","last_clean":"2024-02-12T16:56:24.855367+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:56:24.855367+0000","last_undegraded":"2024-02-12T16:56:24.855367+0000","last_fullsized":"2024-02-12T16:56:24.855367+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.986403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.986403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":73,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.705269+0000","last_change":"2024-02-12T16:54:41.081430+0000","last_active":"2024-02-12T16:56:24.705269+0000","last_peered":"2024-02-12T16:56:24.705269+0000","last_clean":"2024-02-12T16:56:24.705269+0000","last_became_active":"2024-02-12T16:54:40.639547+0000","last_became_peered":"2024-02-12T16:54:40.639547+0000","last_unstale":"2024-02-12T16:56:24.705269+0000","last_undegraded":"2024-02-12T16:56:24.705269+0000","last_fullsized":"2024-02-12T16:56:24.705269+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":54,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:29.870200+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T16:56:29.870200+0000","last_peered":"2024-02-12T16:56:29.870200+0000","last_clean":"2024-02-12T16:56:29.870200+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T16:56:29.870200+0000","last_undegraded":"2024-02-12T16:56:29.870200+0000","last_fullsized":"2024-02-12T16:56:29.870200+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":146,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706110+0000","last_change":"2024-02-12T16:54:40.629347+0000","last_active":"2024-02-12T16:56:24.706110+0000","last_peered":"2024-02-12T16:56:24.706110+0000","last_clean":"2024-02-12T16:56:24.706110+0000","last_became_active":"2024-02-12T16:54:40.628571+0000","last_became_peered":"2024-02-12T16:54:40.628571+0000","last_unstale":"2024-02-12T16:56:24.706110+0000","last_undegraded":"2024-02-12T16:56:24.706110+0000","last_fullsized":"2024-02-12T16:56:24.706110+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.785477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.785477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708115+0000","last_change":"2024-02-12T16:54:01.773110+0000","last_active":"2024-02-12T16:56:24.708115+0000","last_peered":"2024-02-12T16:56:24.708115+0000","last_clean":"2024-02-12T16:56:24.708115+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:56:24.708115+0000","last_undegraded":"2024-02-12T16:56:24.708115+0000","last_fullsized":"2024-02-12T16:56:24.708115+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":156,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.705123+0000","last_change":"2024-02-12T16:54:29.644741+0000","last_active":"2024-02-12T16:56:24.705123+0000","last_peered":"2024-02-12T16:56:24.705123+0000","last_clean":"2024-02-12T16:56:24.705123+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:56:24.705123+0000","last_undegraded":"2024-02-12T16:56:24.705123+0000","last_fullsized":"2024-02-12T16:56:24.705123+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:54:29.644635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:29.644635+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":128,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.815730+0000","last_change":"2024-02-12T16:54:40.194031+0000","last_active":"2024-02-12T16:56:24.815730+0000","last_peered":"2024-02-12T16:56:24.815730+0000","last_clean":"2024-02-12T16:56:24.815730+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:56:24.815730+0000","last_undegraded":"2024-02-12T16:56:24.815730+0000","last_fullsized":"2024-02-12T16:56:24.815730+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":70,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.816581+0000","last_change":"2024-02-12T16:56:23.899212+0000","last_active":"2024-02-12T16:56:24.816581+0000","last_peered":"2024-02-12T16:56:24.816581+0000","last_clean":"2024-02-12T16:56:24.816581+0000","last_became_active":"2024-02-12T16:54:40.650612+0000","last_became_peered":"2024-02-12T16:54:40.650612+0000","last_unstale":"2024-02-12T16:56:24.816581+0000","last_undegraded":"2024-02-12T16:56:24.816581+0000","last_fullsized":"2024-02-12T16:56:24.816581+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":144,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853455+0000","last_change":"2024-02-12T16:54:40.629935+0000","last_active":"2024-02-12T16:56:24.853455+0000","last_peered":"2024-02-12T16:56:24.853455+0000","last_clean":"2024-02-12T16:56:24.853455+0000","last_became_active":"2024-02-12T16:54:40.629419+0000","last_became_peered":"2024-02-12T16:54:40.629419+0000","last_unstale":"2024-02-12T16:56:24.853455+0000","last_undegraded":"2024-02-12T16:56:24.853455+0000","last_fullsized":"2024-02-12T16:56:24.853455+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.590060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.590060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":159,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706023+0000","last_change":"2024-02-12T16:54:31.954069+0000","last_active":"2024-02-12T16:56:24.706023+0000","last_peered":"2024-02-12T16:56:24.706023+0000","last_clean":"2024-02-12T16:56:24.706023+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:56:24.706023+0000","last_undegraded":"2024-02-12T16:56:24.706023+0000","last_fullsized":"2024-02-12T16:56:24.706023+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":99,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:27.696384+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T16:56:27.696384+0000","last_peered":"2024-02-12T16:56:27.696384+0000","last_clean":"2024-02-12T16:56:27.696384+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T16:56:27.696384+0000","last_undegraded":"2024-02-12T16:56:27.696384+0000","last_fullsized":"2024-02-12T16:56:27.696384+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"138'9","reported_seq":121,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.326601+0000","last_change":"2024-02-12T16:56:20.829720+0000","last_active":"2024-02-12T16:56:31.326601+0000","last_peered":"2024-02-12T16:56:31.326601+0000","last_clean":"2024-02-12T16:56:31.326601+0000","last_became_active":"2024-02-12T16:54:40.644025+0000","last_became_peered":"2024-02-12T16:54:40.644025+0000","last_unstale":"2024-02-12T16:56:31.326601+0000","last_undegraded":"2024-02-12T16:56:31.326601+0000","last_fullsized":"2024-02-12T16:56:31.326601+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":143,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.815737+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T16:56:24.815737+0000","last_peered":"2024-02-12T16:56:24.815737+0000","last_clean":"2024-02-12T16:56:24.815737+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T16:56:24.815737+0000","last_undegraded":"2024-02-12T16:56:24.815737+0000","last_fullsized":"2024-02-12T16:56:24.815737+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":169,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.855014+0000","last_change":"2024-02-12T16:54:40.641346+0000","last_active":"2024-02-12T16:56:24.855014+0000","last_peered":"2024-02-12T16:56:24.855014+0000","last_clean":"2024-02-12T16:56:24.855014+0000","last_became_active":"2024-02-12T16:54:40.626031+0000","last_became_peered":"2024-02-12T16:54:40.626031+0000","last_unstale":"2024-02-12T16:56:24.855014+0000","last_undegraded":"2024-02-12T16:56:24.855014+0000","last_fullsized":"2024-02-12T16:56:24.855014+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.815667+0000","last_change":"2024-02-12T16:54:40.633046+0000","last_active":"2024-02-12T16:56:24.815667+0000","last_peered":"2024-02-12T16:56:24.815667+0000","last_clean":"2024-02-12T16:56:24.815667+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T16:56:24.815667+0000","last_undegraded":"2024-02-12T16:56:24.815667+0000","last_fullsized":"2024-02-12T16:56:24.815667+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.105539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.105539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":225,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.298350+0000","last_change":"2024-02-12T16:54:00.112924+0000","last_active":"2024-02-12T16:56:31.298350+0000","last_peered":"2024-02-12T16:56:31.298350+0000","last_clean":"2024-02-12T16:56:31.298350+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:56:31.298350+0000","last_undegraded":"2024-02-12T16:56:31.298350+0000","last_fullsized":"2024-02-12T16:56:31.298350+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":145,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.854382+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T16:56:24.854382+0000","last_peered":"2024-02-12T16:56:24.854382+0000","last_clean":"2024-02-12T16:56:24.854382+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T16:56:24.854382+0000","last_undegraded":"2024-02-12T16:56:24.854382+0000","last_fullsized":"2024-02-12T16:56:24.854382+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":59,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708232+0000","last_change":"2024-02-12T16:54:42.827081+0000","last_active":"2024-02-12T16:56:24.708232+0000","last_peered":"2024-02-12T16:56:24.708232+0000","last_clean":"2024-02-12T16:56:24.708232+0000","last_became_active":"2024-02-12T16:54:40.639018+0000","last_became_peered":"2024-02-12T16:54:40.639018+0000","last_unstale":"2024-02-12T16:56:24.708232+0000","last_undegraded":"2024-02-12T16:56:24.708232+0000","last_fullsized":"2024-02-12T16:56:24.708232+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:03.616717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.616717+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":132,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851359+0000","last_change":"2024-02-12T16:53:51.532654+0000","last_active":"2024-02-12T16:56:24.851359+0000","last_peered":"2024-02-12T16:56:24.851359+0000","last_clean":"2024-02-12T16:56:24.851359+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:56:24.851359+0000","last_undegraded":"2024-02-12T16:56:24.851359+0000","last_fullsized":"2024-02-12T16:56:24.851359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.855292+0000","last_change":"2024-02-12T16:54:40.644416+0000","last_active":"2024-02-12T16:56:24.855292+0000","last_peered":"2024-02-12T16:56:24.855292+0000","last_clean":"2024-02-12T16:56:24.855292+0000","last_became_active":"2024-02-12T16:54:40.644226+0000","last_became_peered":"2024-02-12T16:54:40.644226+0000","last_unstale":"2024-02-12T16:56:24.855292+0000","last_undegraded":"2024-02-12T16:56:24.855292+0000","last_fullsized":"2024-02-12T16:56:24.855292+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:15.844487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:15.844487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":135,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853064+0000","last_change":"2024-02-12T16:52:43.579914+0000","last_active":"2024-02-12T16:56:24.853064+0000","last_peered":"2024-02-12T16:56:24.853064+0000","last_clean":"2024-02-12T16:56:24.853064+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:56:24.853064+0000","last_undegraded":"2024-02-12T16:56:24.853064+0000","last_fullsized":"2024-02-12T16:56:24.853064+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:43.579811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:43.579811+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":78,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851713+0000","last_change":"2024-02-12T16:54:40.903364+0000","last_active":"2024-02-12T16:56:24.851713+0000","last_peered":"2024-02-12T16:56:24.851713+0000","last_clean":"2024-02-12T16:56:24.851713+0000","last_became_active":"2024-02-12T16:54:40.640126+0000","last_became_peered":"2024-02-12T16:54:40.640126+0000","last_unstale":"2024-02-12T16:56:24.851713+0000","last_undegraded":"2024-02-12T16:56:24.851713+0000","last_fullsized":"2024-02-12T16:56:24.851713+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708660+0000","last_change":"2024-02-12T16:52:39.278662+0000","last_active":"2024-02-12T16:56:24.708660+0000","last_peered":"2024-02-12T16:56:24.708660+0000","last_clean":"2024-02-12T16:56:24.708660+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:56:24.708660+0000","last_undegraded":"2024-02-12T16:56:24.708660+0000","last_fullsized":"2024-02-12T16:56:24.708660+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:39.278592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:39.278592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708489+0000","last_change":"2024-02-12T16:54:40.634603+0000","last_active":"2024-02-12T16:56:24.708489+0000","last_peered":"2024-02-12T16:56:24.708489+0000","last_clean":"2024-02-12T16:56:24.708489+0000","last_became_active":"2024-02-12T16:54:40.634261+0000","last_became_peered":"2024-02-12T16:54:40.634261+0000","last_unstale":"2024-02-12T16:56:24.708489+0000","last_undegraded":"2024-02-12T16:56:24.708489+0000","last_fullsized":"2024-02-12T16:56:24.708489+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:41.740892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:53:41.740892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706740+0000","last_change":"2024-02-12T16:56:22.699402+0000","last_active":"2024-02-12T16:56:24.706740+0000","last_peered":"2024-02-12T16:56:24.706740+0000","last_clean":"2024-02-12T16:56:24.706740+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:56:24.706740+0000","last_undegraded":"2024-02-12T16:56:24.706740+0000","last_fullsized":"2024-02-12T16:56:24.706740+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":171,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.817060+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T16:56:24.817060+0000","last_peered":"2024-02-12T16:56:24.817060+0000","last_clean":"2024-02-12T16:56:24.817060+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T16:56:24.817060+0000","last_undegraded":"2024-02-12T16:56:24.817060+0000","last_fullsized":"2024-02-12T16:56:24.817060+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":49,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851215+0000","last_change":"2024-02-12T16:54:40.643841+0000","last_active":"2024-02-12T16:56:24.851215+0000","last_peered":"2024-02-12T16:56:24.851215+0000","last_clean":"2024-02-12T16:56:24.851215+0000","last_became_active":"2024-02-12T16:54:40.643529+0000","last_became_peered":"2024-02-12T16:54:40.643529+0000","last_unstale":"2024-02-12T16:56:24.851215+0000","last_undegraded":"2024-02-12T16:56:24.851215+0000","last_fullsized":"2024-02-12T16:56:24.851215+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":161,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706182+0000","last_change":"2024-02-12T16:55:34.601557+0000","last_active":"2024-02-12T16:56:24.706182+0000","last_peered":"2024-02-12T16:56:24.706182+0000","last_clean":"2024-02-12T16:56:24.706182+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:56:24.706182+0000","last_undegraded":"2024-02-12T16:56:24.706182+0000","last_fullsized":"2024-02-12T16:56:24.706182+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:55:34.601453+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:55:34.601453+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":30,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.852797+0000","last_change":"2024-02-12T16:56:05.735251+0000","last_active":"2024-02-12T16:56:24.852797+0000","last_peered":"2024-02-12T16:56:24.852797+0000","last_clean":"2024-02-12T16:56:24.852797+0000","last_became_active":"2024-02-12T16:56:04.871405+0000","last_became_peered":"2024-02-12T16:56:04.871405+0000","last_unstale":"2024-02-12T16:56:24.852797+0000","last_undegraded":"2024-02-12T16:56:24.852797+0000","last_fullsized":"2024-02-12T16:56:24.852797+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:52:44.868438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.868438+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":132,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706743+0000","last_change":"2024-02-12T16:52:48.411110+0000","last_active":"2024-02-12T16:56:24.706743+0000","last_peered":"2024-02-12T16:56:24.706743+0000","last_clean":"2024-02-12T16:56:24.706743+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:56:24.706743+0000","last_undegraded":"2024-02-12T16:56:24.706743+0000","last_fullsized":"2024-02-12T16:56:24.706743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.411008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.411008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.815787+0000","last_change":"2024-02-12T16:54:40.638105+0000","last_active":"2024-02-12T16:56:24.815787+0000","last_peered":"2024-02-12T16:56:24.815787+0000","last_clean":"2024-02-12T16:56:24.815787+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T16:56:24.815787+0000","last_undegraded":"2024-02-12T16:56:24.815787+0000","last_fullsized":"2024-02-12T16:56:24.815787+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.248970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.248970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708274+0000","last_change":"2024-02-12T16:53:40.916900+0000","last_active":"2024-02-12T16:56:24.708274+0000","last_peered":"2024-02-12T16:56:24.708274+0000","last_clean":"2024-02-12T16:56:24.708274+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:56:24.708274+0000","last_undegraded":"2024-02-12T16:56:24.708274+0000","last_fullsized":"2024-02-12T16:56:24.708274+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:40.916831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:40.916831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":125,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.707230+0000","last_change":"2024-02-12T16:54:39.421299+0000","last_active":"2024-02-12T16:56:24.707230+0000","last_peered":"2024-02-12T16:56:24.707230+0000","last_clean":"2024-02-12T16:56:24.707230+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:56:24.707230+0000","last_undegraded":"2024-02-12T16:56:24.707230+0000","last_fullsized":"2024-02-12T16:56:24.707230+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:39.421176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.421176+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":221,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.141598+0000","last_change":"2024-02-12T16:53:59.173126+0000","last_active":"2024-02-12T16:56:31.141598+0000","last_peered":"2024-02-12T16:56:31.141598+0000","last_clean":"2024-02-12T16:56:31.141598+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:56:31.141598+0000","last_undegraded":"2024-02-12T16:56:31.141598+0000","last_fullsized":"2024-02-12T16:56:31.141598+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.704014+0000","last_change":"2024-02-12T16:55:40.530363+0000","last_active":"2024-02-12T16:56:24.704014+0000","last_peered":"2024-02-12T16:56:24.704014+0000","last_clean":"2024-02-12T16:56:24.704014+0000","last_became_active":"2024-02-12T16:54:40.625901+0000","last_became_peered":"2024-02-12T16:54:40.625901+0000","last_unstale":"2024-02-12T16:56:24.704014+0000","last_undegraded":"2024-02-12T16:56:24.704014+0000","last_fullsized":"2024-02-12T16:56:24.704014+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:55:40.530269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:25.227698+0000","last_change":"2024-02-12T16:56:25.227663+0000","last_active":"2024-02-12T16:56:25.227698+0000","last_peered":"2024-02-12T16:56:25.227698+0000","last_clean":"2024-02-12T16:56:25.227698+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:56:25.227698+0000","last_undegraded":"2024-02-12T16:56:25.227698+0000","last_fullsized":"2024-02-12T16:56:25.227698+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"138'29","reported_seq":297,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.140467+0000","last_change":"2024-02-12T16:54:42.913130+0000","last_active":"2024-02-12T16:56:31.140467+0000","last_peered":"2024-02-12T16:56:31.140467+0000","last_clean":"2024-02-12T16:56:31.140467+0000","last_became_active":"2024-02-12T16:54:40.639565+0000","last_became_peered":"2024-02-12T16:54:40.639565+0000","last_unstale":"2024-02-12T16:56:31.140467+0000","last_undegraded":"2024-02-12T16:56:31.140467+0000","last_fullsized":"2024-02-12T16:56:31.140467+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-12T16:53:59.751031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.751031+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.817688+0000","last_change":"2024-02-12T16:53:01.103805+0000","last_active":"2024-02-12T16:56:24.817688+0000","last_peered":"2024-02-12T16:56:24.817688+0000","last_clean":"2024-02-12T16:56:24.817688+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:56:24.817688+0000","last_undegraded":"2024-02-12T16:56:24.817688+0000","last_fullsized":"2024-02-12T16:56:24.817688+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.103703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.103703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":139,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851476+0000","last_change":"2024-02-12T16:54:40.640502+0000","last_active":"2024-02-12T16:56:24.851476+0000","last_peered":"2024-02-12T16:56:24.851476+0000","last_clean":"2024-02-12T16:56:24.851476+0000","last_became_active":"2024-02-12T16:54:40.640105+0000","last_became_peered":"2024-02-12T16:54:40.640105+0000","last_unstale":"2024-02-12T16:56:24.851476+0000","last_undegraded":"2024-02-12T16:56:24.851476+0000","last_fullsized":"2024-02-12T16:56:24.851476+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":551,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:32.446676+0000","last_change":"2024-02-12T16:54:30.972535+0000","last_active":"2024-02-12T16:56:32.446676+0000","last_peered":"2024-02-12T16:56:32.446676+0000","last_clean":"2024-02-12T16:56:32.446676+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:56:32.446676+0000","last_undegraded":"2024-02-12T16:56:32.446676+0000","last_fullsized":"2024-02-12T16:56:32.446676+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":407,"num_read_kb":402,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":150,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.707030+0000","last_change":"2024-02-12T16:56:20.676702+0000","last_active":"2024-02-12T16:56:24.707030+0000","last_peered":"2024-02-12T16:56:24.707030+0000","last_clean":"2024-02-12T16:56:24.707030+0000","last_became_active":"2024-02-12T16:54:40.632657+0000","last_became_peered":"2024-02-12T16:54:40.632657+0000","last_unstale":"2024-02-12T16:56:24.707030+0000","last_undegraded":"2024-02-12T16:56:24.707030+0000","last_fullsized":"2024-02-12T16:56:24.707030+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":171,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:28.196087+0000","last_change":"2024-02-12T16:56:28.196061+0000","last_active":"2024-02-12T16:56:28.196087+0000","last_peered":"2024-02-12T16:56:28.196087+0000","last_clean":"2024-02-12T16:56:28.196087+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:56:28.196087+0000","last_undegraded":"2024-02-12T16:56:28.196087+0000","last_fullsized":"2024-02-12T16:56:28.196087+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":132,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.707616+0000","last_change":"2024-02-12T16:52:45.314737+0000","last_active":"2024-02-12T16:56:24.707616+0000","last_peered":"2024-02-12T16:56:24.707616+0000","last_clean":"2024-02-12T16:56:24.707616+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:56:24.707616+0000","last_undegraded":"2024-02-12T16:56:24.707616+0000","last_fullsized":"2024-02-12T16:56:24.707616+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:45.314637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:45.314637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":139,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.852764+0000","last_change":"2024-02-12T16:54:40.628021+0000","last_active":"2024-02-12T16:56:24.852764+0000","last_peered":"2024-02-12T16:56:24.852764+0000","last_clean":"2024-02-12T16:56:24.852764+0000","last_became_active":"2024-02-12T16:54:40.627688+0000","last_became_peered":"2024-02-12T16:54:40.627688+0000","last_unstale":"2024-02-12T16:56:24.852764+0000","last_undegraded":"2024-02-12T16:56:24.852764+0000","last_fullsized":"2024-02-12T16:56:24.852764+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":49,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853069+0000","last_change":"2024-02-12T16:54:40.628668+0000","last_active":"2024-02-12T16:56:24.853069+0000","last_peered":"2024-02-12T16:56:24.853069+0000","last_clean":"2024-02-12T16:56:24.853069+0000","last_became_active":"2024-02-12T16:54:40.628365+0000","last_became_peered":"2024-02-12T16:54:40.628365+0000","last_unstale":"2024-02-12T16:56:24.853069+0000","last_undegraded":"2024-02-12T16:56:24.853069+0000","last_fullsized":"2024-02-12T16:56:24.853069+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.967335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.854459+0000","last_change":"2024-02-12T16:52:57.985677+0000","last_active":"2024-02-12T16:56:24.854459+0000","last_peered":"2024-02-12T16:56:24.854459+0000","last_clean":"2024-02-12T16:56:24.854459+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:56:24.854459+0000","last_undegraded":"2024-02-12T16:56:24.854459+0000","last_fullsized":"2024-02-12T16:56:24.854459+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:57.985584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:57.985584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":142,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.850907+0000","last_change":"2024-02-12T16:54:30.770032+0000","last_active":"2024-02-12T16:56:24.850907+0000","last_peered":"2024-02-12T16:56:24.850907+0000","last_clean":"2024-02-12T16:56:24.850907+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:56:24.850907+0000","last_undegraded":"2024-02-12T16:56:24.850907+0000","last_fullsized":"2024-02-12T16:56:24.850907+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":160,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853367+0000","last_change":"2024-02-12T16:53:58.740196+0000","last_active":"2024-02-12T16:56:24.853367+0000","last_peered":"2024-02-12T16:56:24.853367+0000","last_clean":"2024-02-12T16:56:24.853367+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:56:24.853367+0000","last_undegraded":"2024-02-12T16:56:24.853367+0000","last_fullsized":"2024-02-12T16:56:24.853367+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:53:58.740122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.740122+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851545+0000","last_change":"2024-02-12T16:52:12.200439+0000","last_active":"2024-02-12T16:56:24.851545+0000","last_peered":"2024-02-12T16:56:24.851545+0000","last_clean":"2024-02-12T16:56:24.851545+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:56:24.851545+0000","last_undegraded":"2024-02-12T16:56:24.851545+0000","last_fullsized":"2024-02-12T16:56:24.851545+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:12.200355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:12.200355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":139,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708588+0000","last_change":"2024-02-12T16:54:40.628087+0000","last_active":"2024-02-12T16:56:24.708588+0000","last_peered":"2024-02-12T16:56:24.708588+0000","last_clean":"2024-02-12T16:56:24.708588+0000","last_became_active":"2024-02-12T16:54:40.626511+0000","last_became_peered":"2024-02-12T16:54:40.626511+0000","last_unstale":"2024-02-12T16:56:24.708588+0000","last_undegraded":"2024-02-12T16:56:24.708588+0000","last_fullsized":"2024-02-12T16:56:24.708588+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.780846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.780846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":143,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.815598+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T16:56:24.815598+0000","last_peered":"2024-02-12T16:56:24.815598+0000","last_clean":"2024-02-12T16:56:24.815598+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T16:56:24.815598+0000","last_undegraded":"2024-02-12T16:56:24.815598+0000","last_fullsized":"2024-02-12T16:56:24.815598+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":140,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.850859+0000","last_change":"2024-02-12T16:52:58.167642+0000","last_active":"2024-02-12T16:56:24.850859+0000","last_peered":"2024-02-12T16:56:24.850859+0000","last_clean":"2024-02-12T16:56:24.850859+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:56:24.850859+0000","last_undegraded":"2024-02-12T16:56:24.850859+0000","last_fullsized":"2024-02-12T16:56:24.850859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:52:58.167543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.167543+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":150,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.816959+0000","last_change":"2024-02-12T16:53:43.229740+0000","last_active":"2024-02-12T16:56:24.816959+0000","last_peered":"2024-02-12T16:56:24.816959+0000","last_clean":"2024-02-12T16:56:24.816959+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:56:24.816959+0000","last_undegraded":"2024-02-12T16:56:24.816959+0000","last_fullsized":"2024-02-12T16:56:24.816959+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:43.229622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.229622+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":132,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.855456+0000","last_change":"2024-02-12T16:56:18.162272+0000","last_active":"2024-02-12T16:56:24.855456+0000","last_peered":"2024-02-12T16:56:24.855456+0000","last_clean":"2024-02-12T16:56:24.855456+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:56:24.855456+0000","last_undegraded":"2024-02-12T16:56:24.855456+0000","last_fullsized":"2024-02-12T16:56:24.855456+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706304+0000","last_change":"2024-02-12T16:53:50.836514+0000","last_active":"2024-02-12T16:56:24.706304+0000","last_peered":"2024-02-12T16:56:24.706304+0000","last_clean":"2024-02-12T16:56:24.706304+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:56:24.706304+0000","last_undegraded":"2024-02-12T16:56:24.706304+0000","last_fullsized":"2024-02-12T16:56:24.706304+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.836406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.836406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851523+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T16:56:24.851523+0000","last_peered":"2024-02-12T16:56:24.851523+0000","last_clean":"2024-02-12T16:56:24.851523+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:56:24.851523+0000","last_undegraded":"2024-02-12T16:56:24.851523+0000","last_fullsized":"2024-02-12T16:56:24.851523+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":173,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:30.812766+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T16:56:30.812766+0000","last_peered":"2024-02-12T16:56:30.812766+0000","last_clean":"2024-02-12T16:56:30.812766+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:56:30.812766+0000","last_undegraded":"2024-02-12T16:56:30.812766+0000","last_fullsized":"2024-02-12T16:56:30.812766+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853091+0000","last_change":"2024-02-12T16:53:50.681255+0000","last_active":"2024-02-12T16:56:24.853091+0000","last_peered":"2024-02-12T16:56:24.853091+0000","last_clean":"2024-02-12T16:56:24.853091+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:56:24.853091+0000","last_undegraded":"2024-02-12T16:56:24.853091+0000","last_fullsized":"2024-02-12T16:56:24.853091+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.681142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.681142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":52,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706528+0000","last_change":"2024-02-12T16:54:40.632821+0000","last_active":"2024-02-12T16:56:24.706528+0000","last_peered":"2024-02-12T16:56:24.706528+0000","last_clean":"2024-02-12T16:56:24.706528+0000","last_became_active":"2024-02-12T16:54:40.631423+0000","last_became_peered":"2024-02-12T16:54:40.631423+0000","last_unstale":"2024-02-12T16:56:24.706528+0000","last_undegraded":"2024-02-12T16:56:24.706528+0000","last_fullsized":"2024-02-12T16:56:24.706528+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.624865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.624865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.707991+0000","last_change":"2024-02-12T16:53:48.828698+0000","last_active":"2024-02-12T16:56:24.707991+0000","last_peered":"2024-02-12T16:56:24.707991+0000","last_clean":"2024-02-12T16:56:24.707991+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:56:24.707991+0000","last_undegraded":"2024-02-12T16:56:24.707991+0000","last_fullsized":"2024-02-12T16:56:24.707991+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":116,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:26.889589+0000","last_change":"2024-02-12T16:56:26.889557+0000","last_active":"2024-02-12T16:56:26.889589+0000","last_peered":"2024-02-12T16:56:26.889589+0000","last_clean":"2024-02-12T16:56:26.889589+0000","last_became_active":"2024-02-12T16:54:40.634781+0000","last_became_peered":"2024-02-12T16:54:40.634781+0000","last_unstale":"2024-02-12T16:56:26.889589+0000","last_undegraded":"2024-02-12T16:56:26.889589+0000","last_fullsized":"2024-02-12T16:56:26.889589+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":147,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.703604+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T16:56:24.703604+0000","last_peered":"2024-02-12T16:56:24.703604+0000","last_clean":"2024-02-12T16:56:24.703604+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T16:56:24.703604+0000","last_undegraded":"2024-02-12T16:56:24.703604+0000","last_fullsized":"2024-02-12T16:56:24.703604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":129,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851366+0000","last_change":"2024-02-12T16:52:41.149470+0000","last_active":"2024-02-12T16:56:24.851366+0000","last_peered":"2024-02-12T16:56:24.851366+0000","last_clean":"2024-02-12T16:56:24.851366+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:56:24.851366+0000","last_undegraded":"2024-02-12T16:56:24.851366+0000","last_fullsized":"2024-02-12T16:56:24.851366+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:52:41.149382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.149382+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":140,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:26.337578+0000","last_change":"2024-02-12T16:56:26.337548+0000","last_active":"2024-02-12T16:56:26.337578+0000","last_peered":"2024-02-12T16:56:26.337578+0000","last_clean":"2024-02-12T16:56:26.337578+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:56:26.337578+0000","last_undegraded":"2024-02-12T16:56:26.337578+0000","last_fullsized":"2024-02-12T16:56:26.337578+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":127,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853551+0000","last_change":"2024-02-12T16:52:51.025242+0000","last_active":"2024-02-12T16:56:24.853551+0000","last_peered":"2024-02-12T16:56:24.853551+0000","last_clean":"2024-02-12T16:56:24.853551+0000","last_became_active":"2024-02-12T16:52:20.563744+0000","last_became_peered":"2024-02-12T16:52:20.563744+0000","last_unstale":"2024-02-12T16:56:24.853551+0000","last_undegraded":"2024-02-12T16:56:24.853551+0000","last_fullsized":"2024-02-12T16:56:24.853551+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:52:51.025152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.025152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"138'9","reported_seq":101,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.139615+0000","last_change":"2024-02-12T16:55:28.449149+0000","last_active":"2024-02-12T16:56:31.139615+0000","last_peered":"2024-02-12T16:56:31.139615+0000","last_clean":"2024-02-12T16:56:31.139615+0000","last_became_active":"2024-02-12T16:54:40.626646+0000","last_became_peered":"2024-02-12T16:54:40.626646+0000","last_unstale":"2024-02-12T16:56:31.139615+0000","last_undegraded":"2024-02-12T16:56:31.139615+0000","last_fullsized":"2024-02-12T16:56:31.139615+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":49,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853363+0000","last_change":"2024-02-12T16:54:40.629725+0000","last_active":"2024-02-12T16:56:24.853363+0000","last_peered":"2024-02-12T16:56:24.853363+0000","last_clean":"2024-02-12T16:56:24.853363+0000","last_became_active":"2024-02-12T16:54:40.629348+0000","last_became_peered":"2024-02-12T16:54:40.629348+0000","last_unstale":"2024-02-12T16:56:24.853363+0000","last_undegraded":"2024-02-12T16:56:24.853363+0000","last_fullsized":"2024-02-12T16:56:24.853363+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":132,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.855410+0000","last_change":"2024-02-12T16:52:52.983089+0000","last_active":"2024-02-12T16:56:24.855410+0000","last_peered":"2024-02-12T16:56:24.855410+0000","last_clean":"2024-02-12T16:56:24.855410+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:56:24.855410+0000","last_undegraded":"2024-02-12T16:56:24.855410+0000","last_fullsized":"2024-02-12T16:56:24.855410+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:52.982996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.982996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":113,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706661+0000","last_change":"2024-02-12T16:53:59.566836+0000","last_active":"2024-02-12T16:56:24.706661+0000","last_peered":"2024-02-12T16:56:24.706661+0000","last_clean":"2024-02-12T16:56:24.706661+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:56:24.706661+0000","last_undegraded":"2024-02-12T16:56:24.706661+0000","last_fullsized":"2024-02-12T16:56:24.706661+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.566730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853399+0000","last_change":"2024-02-12T16:56:04.931453+0000","last_active":"2024-02-12T16:56:24.853399+0000","last_peered":"2024-02-12T16:56:24.853399+0000","last_clean":"2024-02-12T16:56:24.853399+0000","last_became_active":"2024-02-12T16:56:04.863064+0000","last_became_peered":"2024-02-12T16:56:04.863064+0000","last_unstale":"2024-02-12T16:56:24.853399+0000","last_undegraded":"2024-02-12T16:56:24.853399+0000","last_fullsized":"2024-02-12T16:56:24.853399+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:54:36.753321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:36.753321+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.855212+0000","last_change":"2024-02-12T16:52:46.979058+0000","last_active":"2024-02-12T16:56:24.855212+0000","last_peered":"2024-02-12T16:56:24.855212+0000","last_clean":"2024-02-12T16:56:24.855212+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:56:24.855212+0000","last_undegraded":"2024-02-12T16:56:24.855212+0000","last_fullsized":"2024-02-12T16:56:24.855212+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:46.978987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:46.978987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"98'10","reported_seq":196,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:32.292817+0000","last_change":"2024-02-12T16:56:32.292792+0000","last_active":"2024-02-12T16:56:32.292817+0000","last_peered":"2024-02-12T16:56:32.292817+0000","last_clean":"2024-02-12T16:56:32.292817+0000","last_became_active":"2024-02-12T16:52:20.561534+0000","last_became_peered":"2024-02-12T16:52:20.561534+0000","last_unstale":"2024-02-12T16:56:32.292817+0000","last_undegraded":"2024-02-12T16:56:32.292817+0000","last_fullsized":"2024-02-12T16:56:32.292817+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":159,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706218+0000","last_change":"2024-02-12T16:56:04.857653+0000","last_active":"2024-02-12T16:56:24.706218+0000","last_peered":"2024-02-12T16:56:24.706218+0000","last_clean":"2024-02-12T16:56:24.706218+0000","last_became_active":"2024-02-12T16:56:04.857412+0000","last_became_peered":"2024-02-12T16:56:04.857412+0000","last_unstale":"2024-02-12T16:56:24.706218+0000","last_undegraded":"2024-02-12T16:56:24.706218+0000","last_fullsized":"2024-02-12T16:56:24.706218+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":196,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708211+0000","last_change":"2024-02-12T16:54:41.243992+0000","last_active":"2024-02-12T16:56:24.708211+0000","last_peered":"2024-02-12T16:56:24.708211+0000","last_clean":"2024-02-12T16:56:24.708211+0000","last_became_active":"2024-02-12T16:54:40.627029+0000","last_became_peered":"2024-02-12T16:54:40.627029+0000","last_unstale":"2024-02-12T16:56:24.708211+0000","last_undegraded":"2024-02-12T16:56:24.708211+0000","last_fullsized":"2024-02-12T16:56:24.708211+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":94,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.707889+0000","last_change":"2024-02-12T16:53:57.738352+0000","last_active":"2024-02-12T16:56:24.707889+0000","last_peered":"2024-02-12T16:56:24.707889+0000","last_clean":"2024-02-12T16:56:24.707889+0000","last_became_active":"2024-02-12T16:52:20.560085+0000","last_became_peered":"2024-02-12T16:52:20.560085+0000","last_unstale":"2024-02-12T16:56:24.707889+0000","last_undegraded":"2024-02-12T16:56:24.707889+0000","last_fullsized":"2024-02-12T16:56:24.707889+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:57.738249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.738249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":228,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.326830+0000","last_change":"2024-02-12T16:56:26.673555+0000","last_active":"2024-02-12T16:56:31.326830+0000","last_peered":"2024-02-12T16:56:31.326830+0000","last_clean":"2024-02-12T16:56:31.326830+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:56:31.326830+0000","last_undegraded":"2024-02-12T16:56:31.326830+0000","last_fullsized":"2024-02-12T16:56:31.326830+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":150,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.705934+0000","last_change":"2024-02-12T16:54:40.629536+0000","last_active":"2024-02-12T16:56:24.705934+0000","last_peered":"2024-02-12T16:56:24.705934+0000","last_clean":"2024-02-12T16:56:24.705934+0000","last_became_active":"2024-02-12T16:54:40.629024+0000","last_became_peered":"2024-02-12T16:54:40.629024+0000","last_unstale":"2024-02-12T16:56:24.705934+0000","last_undegraded":"2024-02-12T16:56:24.705934+0000","last_fullsized":"2024-02-12T16:56:24.705934+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":93,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.852618+0000","last_change":"2024-02-12T16:55:46.162945+0000","last_active":"2024-02-12T16:56:24.852618+0000","last_peered":"2024-02-12T16:56:24.852618+0000","last_clean":"2024-02-12T16:56:24.852618+0000","last_became_active":"2024-02-12T16:54:40.629780+0000","last_became_peered":"2024-02-12T16:54:40.629780+0000","last_unstale":"2024-02-12T16:56:24.852618+0000","last_undegraded":"2024-02-12T16:56:24.852618+0000","last_fullsized":"2024-02-12T16:56:24.852618+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":144,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706527+0000","last_change":"2024-02-12T16:54:40.631210+0000","last_active":"2024-02-12T16:56:24.706527+0000","last_peered":"2024-02-12T16:56:24.706527+0000","last_clean":"2024-02-12T16:56:24.706527+0000","last_became_active":"2024-02-12T16:54:40.630855+0000","last_became_peered":"2024-02-12T16:54:40.630855+0000","last_unstale":"2024-02-12T16:56:24.706527+0000","last_undegraded":"2024-02-12T16:56:24.706527+0000","last_fullsized":"2024-02-12T16:56:24.706527+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.387144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.387144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:33.053023+0000","last_change":"2024-02-12T16:56:33.053023+0000","last_active":"2024-02-12T16:56:33.053023+0000","last_peered":"2024-02-12T16:56:33.053023+0000","last_clean":"2024-02-12T16:56:33.053023+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:56:33.053023+0000","last_undegraded":"2024-02-12T16:56:33.053023+0000","last_fullsized":"2024-02-12T16:56:33.053023+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":139,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.851159+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T16:56:24.851159+0000","last_peered":"2024-02-12T16:56:24.851159+0000","last_clean":"2024-02-12T16:56:24.851159+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:56:24.851159+0000","last_undegraded":"2024-02-12T16:56:24.851159+0000","last_fullsized":"2024-02-12T16:56:24.851159+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":166,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:30.714593+0000","last_change":"2024-02-12T16:56:30.714551+0000","last_active":"2024-02-12T16:56:30.714593+0000","last_peered":"2024-02-12T16:56:30.714593+0000","last_clean":"2024-02-12T16:56:30.714593+0000","last_became_active":"2024-02-12T16:56:04.869971+0000","last_became_peered":"2024-02-12T16:56:04.869971+0000","last_unstale":"2024-02-12T16:56:30.714593+0000","last_undegraded":"2024-02-12T16:56:30.714593+0000","last_fullsized":"2024-02-12T16:56:30.714593+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.850811+0000","last_change":"2024-02-12T16:54:40.645951+0000","last_active":"2024-02-12T16:56:24.850811+0000","last_peered":"2024-02-12T16:56:24.850811+0000","last_clean":"2024-02-12T16:56:24.850811+0000","last_became_active":"2024-02-12T16:54:40.645556+0000","last_became_peered":"2024-02-12T16:54:40.645556+0000","last_unstale":"2024-02-12T16:56:24.850811+0000","last_undegraded":"2024-02-12T16:56:24.850811+0000","last_fullsized":"2024-02-12T16:56:24.850811+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.128703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.128703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.707394+0000","last_change":"2024-02-12T16:53:58.524837+0000","last_active":"2024-02-12T16:56:24.707394+0000","last_peered":"2024-02-12T16:56:24.707394+0000","last_clean":"2024-02-12T16:56:24.707394+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:56:24.707394+0000","last_undegraded":"2024-02-12T16:56:24.707394+0000","last_fullsized":"2024-02-12T16:56:24.707394+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.524736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.708934+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T16:56:24.708934+0000","last_peered":"2024-02-12T16:56:24.708934+0000","last_clean":"2024-02-12T16:56:24.708934+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:56:24.708934+0000","last_undegraded":"2024-02-12T16:56:24.708934+0000","last_fullsized":"2024-02-12T16:56:24.708934+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":198,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.853012+0000","last_change":"2024-02-12T16:53:40.847643+0000","last_active":"2024-02-12T16:56:24.853012+0000","last_peered":"2024-02-12T16:56:24.853012+0000","last_clean":"2024-02-12T16:56:24.853012+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:56:24.853012+0000","last_undegraded":"2024-02-12T16:56:24.853012+0000","last_fullsized":"2024-02-12T16:56:24.853012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:53:40.847540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:40.847540+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":128,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.816382+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T16:56:24.816382+0000","last_peered":"2024-02-12T16:56:24.816382+0000","last_clean":"2024-02-12T16:56:24.816382+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:56:24.816382+0000","last_undegraded":"2024-02-12T16:56:24.816382+0000","last_fullsized":"2024-02-12T16:56:24.816382+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.706644+0000","last_change":"2024-02-12T16:54:40.629432+0000","last_active":"2024-02-12T16:56:24.706644+0000","last_peered":"2024-02-12T16:56:24.706644+0000","last_clean":"2024-02-12T16:56:24.706644+0000","last_became_active":"2024-02-12T16:54:40.629081+0000","last_became_peered":"2024-02-12T16:54:40.629081+0000","last_unstale":"2024-02-12T16:56:24.706644+0000","last_undegraded":"2024-02-12T16:56:24.706644+0000","last_fullsized":"2024-02-12T16:56:24.706644+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.285961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.285961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":159,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.705629+0000","last_change":"2024-02-12T16:54:41.249353+0000","last_active":"2024-02-12T16:56:24.705629+0000","last_peered":"2024-02-12T16:56:24.705629+0000","last_clean":"2024-02-12T16:56:24.705629+0000","last_became_active":"2024-02-12T16:54:40.639823+0000","last_became_peered":"2024-02-12T16:54:40.639823+0000","last_unstale":"2024-02-12T16:56:24.705629+0000","last_undegraded":"2024-02-12T16:56:24.705629+0000","last_fullsized":"2024-02-12T16:56:24.705629+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":153,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.836846+0000","last_change":"2024-02-12T16:56:31.836846+0000","last_active":"2024-02-12T16:56:31.836846+0000","last_peered":"2024-02-12T16:56:31.836846+0000","last_clean":"2024-02-12T16:56:31.836846+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:56:31.836846+0000","last_undegraded":"2024-02-12T16:56:31.836846+0000","last_fullsized":"2024-02-12T16:56:31.836846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:56:35.434299+0000","last_change":"2024-02-12T16:56:35.434299+0000","last_active":"2024-02-12T16:56:35.434299+0000","last_peered":"2024-02-12T16:56:35.434299+0000","last_clean":"2024-02-12T16:56:35.434299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:56:35.434299+0000","last_undegraded":"2024-02-12T16:56:35.434299+0000","last_fullsized":"2024-02-12T16:56:35.434299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":130,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.854331+0000","last_change":"2024-02-12T16:54:03.996224+0000","last_active":"2024-02-12T16:56:24.854331+0000","last_peered":"2024-02-12T16:56:24.854331+0000","last_clean":"2024-02-12T16:56:24.854331+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:56:24.854331+0000","last_undegraded":"2024-02-12T16:56:24.854331+0000","last_fullsized":"2024-02-12T16:56:24.854331+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":155,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:32.717219+0000","last_change":"2024-02-12T16:56:32.717219+0000","last_active":"2024-02-12T16:56:32.717219+0000","last_peered":"2024-02-12T16:56:32.717219+0000","last_clean":"2024-02-12T16:56:32.717219+0000","last_became_active":"2024-02-12T16:54:40.630768+0000","last_became_peered":"2024-02-12T16:54:40.630768+0000","last_unstale":"2024-02-12T16:56:32.717219+0000","last_undegraded":"2024-02-12T16:56:32.717219+0000","last_fullsized":"2024-02-12T16:56:32.717219+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":159,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-12T16:56:24.852980+0000","last_change":"2024-02-12T16:52:40.091071+0000","last_active":"2024-02-12T16:56:24.852980+0000","last_peered":"2024-02-12T16:56:24.852980+0000","last_clean":"2024-02-12T16:56:24.852980+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:56:24.852980+0000","last_undegraded":"2024-02-12T16:56:24.852980+0000","last_fullsized":"2024-02-12T16:56:24.852980+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:52:40.090931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:40.090931+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":221,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-02-12T16:56:31.325836+0000","last_change":"2024-02-12T16:52:59.185115+0000","last_active":"2024-02-12T16:56:31.325836+0000","last_peered":"2024-02-12T16:56:31.325836+0000","last_clean":"2024-02-12T16:56:31.325836+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:56:31.325836+0000","last_undegraded":"2024-02-12T16:56:31.325836+0000","last_fullsized":"2024-02-12T16:56:31.325836+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":407,"num_read_kb":402,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234070,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298048,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447104,"statfs":{"total":95995035648,"available":95689834496,"internally_reserved":0,"allocated":843776,"data_stored":430264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":2.0129999999999999,"5min":1.7310000000000001,"15min":0.89200000000000002},"min":{"1min":0.22600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":7.3239999999999998,"5min":36.439,"15min":36.439},"last":0.495},{"interface":"front","average":{"1min":1.389,"5min":1.5620000000000001,"15min":0.86699999999999999},"min":{"1min":0.24099999999999999,"5min":0.218,"15min":0.218},"max":{"1min":3.5609999999999999,"5min":36.490000000000002,"15min":36.490000000000002},"last":0.374}]},{"osd":1,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":1.0369999999999999,"15min":0.70599999999999996},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":12.032,"5min":12.032,"15min":12.032},"last":0.47399999999999998},{"interface":"front","average":{"1min":2.1480000000000001,"5min":1.022,"15min":0.70499999999999996},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":10.054,"5min":10.054,"15min":10.054},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.0860000000000001,"15min":0.68300000000000005},"min":{"1min":0.30199999999999999,"5min":0.245,"15min":0.245},"max":{"1min":7.1230000000000002,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.806,"5min":1.2350000000000001,"15min":0.75800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.9989999999999997,"5min":20.338000000000001,"15min":20.338000000000001},"last":0.54400000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":2.2429999999999999,"5min":1.6599999999999999,"15min":0.94399999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.1069999999999993,"5min":20.433,"15min":20.433},"last":0.34599999999999997},{"interface":"front","average":{"1min":2.5049999999999999,"5min":1.7270000000000001,"15min":0.95299999999999996},"min":{"1min":0.33400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":9.9659999999999993,"5min":20.393000000000001,"15min":20.393000000000001},"last":0.56299999999999994}]},{"osd":4,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.59099999999999997,"15min":0.52600000000000002},"min":{"1min":0.26800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.997,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":0.51900000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.972,"5min":2.972,"15min":2.972},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.58799999999999997,"15min":0.56100000000000005},"min":{"1min":0.27400000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.60899999999999999,"15min":0.54100000000000004},"min":{"1min":0.317,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.042,"5min":0.69399999999999995,"15min":0.60899999999999999},"min":{"1min":0.309,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.5920000000000001,"5min":3.5920000000000001,"15min":3.5920000000000001},"last":0.53200000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528411,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310348,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434804,"statfs":{"total":95995035648,"available":95677239296,"internally_reserved":0,"allocated":856064,"data_stored":430412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3279999999999998,"5min":1.522,"15min":0.82999999999999996},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":10.949,"5min":10.949,"15min":10.949},"last":0.60199999999999998},{"interface":"front","average":{"1min":2.794,"5min":1.5680000000000001,"15min":0.84399999999999997},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.6050000000000004,"5min":10.56,"15min":10.56},"last":0.375}]},{"osd":1,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.4550000000000001,"15min":0.83199999999999996},"min":{"1min":0.252,"5min":0.251,"15min":0.251},"max":{"1min":3.7000000000000002,"5min":15.987,"15min":15.987},"last":0.55700000000000005},{"interface":"front","average":{"1min":2.3319999999999999,"5min":1.5740000000000001,"15min":0.88800000000000001},"min":{"1min":0.34200000000000003,"5min":0.25,"15min":0.25},"max":{"1min":5.9770000000000003,"5min":15.997,"15min":15.997},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":1.357,"15min":0.78900000000000003},"min":{"1min":0.22900000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.435},{"interface":"front","average":{"1min":2.3090000000000002,"5min":1.333,"15min":0.78400000000000003},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.319,"5min":6.3680000000000003,"15min":6.3680000000000003},"last":0.56999999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.3280000000000001,"15min":0.77500000000000002},"min":{"1min":0.29199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.6900000000000004,"5min":11.212,"15min":11.212},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.1059999999999999,"5min":1.6719999999999999,"15min":0.90500000000000003},"min":{"1min":0.32900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":6.0380000000000003,"5min":10.875999999999999,"15min":10.875999999999999},"last":0.47499999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.61399999999999999,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.217},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.353,"5min":0.68300000000000005,"15min":0.495},"min":{"1min":0.26600000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.624,"15min":0.48799999999999999},"min":{"1min":0.309,"5min":0.245,"15min":0.245},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.497},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.755,"15min":0.54100000000000004},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":3.5859999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.65600000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:55:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.68799999999999994,"15min":0.53200000000000003},"min":{"1min":0.31900000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.452,"5min":5.452,"15min":5.452},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.67300000000000004,"15min":0.53900000000000003},"min":{"1min":0.35799999999999998,"5min":0.29299999999999998,"15min":0.27900000000000003},"max":{"1min":2.407,"5min":3.9049999999999998,"15min":3.9049999999999998},"last":0.622}]}]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298068,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447084,"statfs":{"total":95995035648,"available":95689814016,"internally_reserved":0,"allocated":864256,"data_stored":431696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4660000000000002,"5min":2.6760000000000002,"15min":1.121},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":12.592000000000001,"5min":25.585000000000001,"15min":25.585000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.0029999999999999,"5min":2.5299999999999998,"15min":1.085},"min":{"1min":0.219,"5min":0.215,"15min":0.215},"max":{"1min":3.9390000000000001,"5min":26.798999999999999,"15min":26.798999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:55:49 2024","interfaces":[{"interface":"back","average":{"1min":4.1269999999999998,"5min":2.1520000000000001,"15min":0.96799999999999997},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.859,"5min":11.581,"15min":11.581},"last":0.311},{"interface":"front","average":{"1min":2.403,"5min":2.4620000000000002,"15min":1.0820000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":8.6519999999999992,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.40500000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:56:14 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.552,"15min":1.155},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.5220000000000002,"5min":26.649999999999999,"15min":26.649999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":3.21,"5min":2.2999999999999998,"15min":1.0409999999999999},"min":{"1min":0.314,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":17.821000000000002,"5min":17.821000000000002,"15min":17.821000000000002},"last":0.376}]},{"osd":4,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.6970000000000001,"15min":0.93999999999999995},"min":{"1min":0.34399999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.7030000000000003,"5min":8.8249999999999993,"15min":8.8249999999999993},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.1659999999999999,"5min":2.1819999999999999,"15min":1.1259999999999999},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.8879999999999999,"5min":26.748999999999999,"15min":26.748999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":3.2000000000000002,"5min":1.7110000000000001,"15min":0.93999999999999995},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":8.7959999999999994,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":0.434},{"interface":"front","average":{"1min":2.4009999999999998,"5min":2.2519999999999998,"15min":1.1499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.5960000000000001,"5min":26.722999999999999,"15min":26.722999999999999},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.9490000000000001,"15min":1.042},"min":{"1min":0.38600000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6509999999999998,"5min":13.834,"15min":13.834},"last":0.497},{"interface":"front","average":{"1min":1.2749999999999999,"5min":2.1200000000000001,"15min":1.1299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.5550000000000002,"5min":26.774999999999999,"15min":26.774999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:55:43 2024","interfaces":[{"interface":"back","average":{"1min":2.6429999999999998,"5min":1.613,"15min":0.97899999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":9.1940000000000008,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.505},{"interface":"front","average":{"1min":4.0369999999999999,"5min":1.966,"15min":1.0880000000000001},"min":{"1min":0.42199999999999999,"5min":0.31,"15min":0.31},"max":{"1min":8.8829999999999991,"5min":8.8829999999999991,"15min":8.8829999999999991},"last":0.51700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302144,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93443008,"statfs":{"total":95995035648,"available":95685640192,"internally_reserved":0,"allocated":843776,"data_stored":430058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:56:24 2024","interfaces":[{"interface":"back","average":{"1min":1.895,"5min":2.0169999999999999,"15min":0.89500000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.1859999999999999,"5min":23.510999999999999,"15min":23.510999999999999},"last":0.221},{"interface":"front","average":{"1min":2.8679999999999999,"5min":2.008,"15min":0.88700000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":8.6500000000000004,"5min":10.186999999999999,"15min":10.186999999999999},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:55:53 2024","interfaces":[{"interface":"back","average":{"1min":3.714,"5min":2.3900000000000001,"15min":1.052},"min":{"1min":0.26000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":16.521000000000001,"5min":16.521000000000001,"15min":16.521000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.8290000000000002,"5min":1.9059999999999999,"15min":0.90300000000000002},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":9.9429999999999996,"5min":9.9429999999999996,"15min":9.9429999999999996},"last":0.23599999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:56:17 2024","interfaces":[{"interface":"back","average":{"1min":2.5329999999999999,"5min":2.5430000000000001,"15min":1.121},"min":{"1min":0.28799999999999998,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.3029999999999999,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.311},{"interface":"front","average":{"1min":1.879,"5min":2.52,"15min":1.145},"min":{"1min":0.29799999999999999,"5min":0.28100000000000003,"15min":0.27600000000000002},"max":{"1min":7.2030000000000003,"5min":16.446000000000002,"15min":16.446000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.6919999999999999,"15min":0.94299999999999995},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.8330000000000002,"5min":16.027000000000001,"15min":16.027000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.4399999999999999,"15min":0.86299999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.7619999999999996,"5min":10.141,"15min":10.141},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":2.46,"5min":1.6739999999999999,"15min":0.93700000000000006},"min":{"1min":0.32100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.9930000000000003,"5min":7.4340000000000002,"15min":7.4340000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.6479999999999999,"15min":0.94299999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.8799999999999999,"5min":16.138999999999999,"15min":16.138999999999999},"last":0.52500000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:56:24 2024","interfaces":[{"interface":"back","average":{"1min":2.4319999999999999,"5min":1.704,"15min":0.94899999999999995},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":12.135,"5min":12.135,"15min":12.135},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.6829999999999998,"5min":1.982,"15min":1.0449999999999999},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.618,"5min":21.062000000000001,"15min":21.062000000000001},"last":0.53800000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4180000000000001,"5min":1.6419999999999999,"15min":0.93799999999999994},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":16.175000000000001,"5min":16.175000000000001,"15min":16.175000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.714,"5min":1.2909999999999999,"15min":0.82699999999999996},"min":{"1min":0.35499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":10.025,"5min":10.025,"15min":10.025},"last":0.63}]}]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306236,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438916,"statfs":{"total":95995035648,"available":95681449984,"internally_reserved":0,"allocated":839680,"data_stored":430950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:49 2024","interfaces":[{"interface":"back","average":{"1min":4.1509999999999998,"5min":2.4340000000000002,"15min":1.069},"min":{"1min":0.27600000000000002,"5min":0.19,"15min":0.19},"max":{"1min":15.808,"5min":15.808,"15min":15.808},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.9729999999999999,"5min":1.9019999999999999,"15min":0.90700000000000003},"min":{"1min":0.20499999999999999,"5min":0.158,"15min":0.158},"max":{"1min":8.6620000000000008,"5min":19.893000000000001,"15min":19.893000000000001},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:55:49 2024","interfaces":[{"interface":"back","average":{"1min":3.9220000000000002,"5min":2.46,"15min":1.109},"min":{"1min":0.22800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":17.739000000000001,"5min":17.739000000000001,"15min":17.739000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":4.2530000000000001,"5min":2.6080000000000001,"15min":1.1499999999999999},"min":{"1min":0.373,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":15.766999999999999,"5min":15.766999999999999,"15min":15.766999999999999},"last":0.40699999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":3.2909999999999999,"15min":1.4079999999999999},"min":{"1min":0.35999999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":13.596,"5min":51.579999999999998,"15min":51.579999999999998},"last":0.499},{"interface":"front","average":{"1min":2.0649999999999999,"5min":2.4569999999999999,"15min":1.143},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":13.521000000000001,"5min":48.988999999999997,"15min":48.988999999999997},"last":0.47799999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:55:39 2024","interfaces":[{"interface":"back","average":{"1min":3.4940000000000002,"5min":1.7370000000000001,"15min":0.999},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":15.737,"5min":15.737,"15min":15.737},"last":0.56599999999999995},{"interface":"front","average":{"1min":2.5529999999999999,"5min":1.3120000000000001,"15min":0.84699999999999998},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7390000000000008,"5min":9.2119999999999997,"15min":9.2119999999999997},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:55:56 2024","interfaces":[{"interface":"back","average":{"1min":3.3500000000000001,"5min":2.0369999999999999,"15min":1.169},"min":{"1min":0.32400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":15.699,"5min":15.699,"15min":15.699},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.988,"15min":1.0620000000000001},"min":{"1min":0.34200000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":8.4610000000000003,"5min":11.319000000000001,"15min":11.319000000000001},"last":0.55400000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.0449999999999999,"5min":2.3599999999999999,"15min":1.1870000000000001},"min":{"1min":0.40600000000000003,"5min":0.31,"15min":0.31},"max":{"1min":12.215,"5min":17.782,"15min":17.782},"last":0.61799999999999999},{"interface":"front","average":{"1min":2.226,"5min":1.736,"15min":1.056},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":13.446999999999999,"5min":13.446999999999999,"15min":13.446999999999999},"last":0.68700000000000006}]},{"osd":7,"last update":"Mon Feb 12 16:55:49 2024","interfaces":[{"interface":"back","average":{"1min":2.4449999999999998,"5min":1.6279999999999999,"15min":1.1970000000000001},"min":{"1min":0.36299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":8.7880000000000003,"5min":9.2550000000000008,"15min":9.2550000000000008},"last":0.66200000000000003},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.9770000000000001,"15min":1.079},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":10.691000000000001,"5min":11.356,"15min":11.356},"last":0.59499999999999997}]}]},{"osd":3,"up_from":0,"seq":111669149801,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953573,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306236,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438916,"statfs":{"total":95995035648,"available":95681449984,"internally_reserved":0,"allocated":839680,"data_stored":430507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:32 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.4630000000000001,"15min":0.85799999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":7.9950000000000001,"5min":38.179000000000002,"15min":38.179000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.51,"15min":0.84999999999999998},"min":{"1min":0.22700000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.202,"5min":38.151000000000003,"15min":38.151000000000003},"last":0.35099999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:55:32 2024","interfaces":[{"interface":"back","average":{"1min":1.907,"5min":1.05,"15min":0.70199999999999996},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.1879999999999997,"5min":7.673,"15min":7.673},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.913,"5min":1.0700000000000001,"15min":0.72099999999999997},"min":{"1min":0.254,"5min":0.249,"15min":0.249},"max":{"1min":6.367,"5min":6.367,"15min":6.367},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:55:32 2024","interfaces":[{"interface":"back","average":{"1min":2.1280000000000001,"5min":1.087,"15min":0.745},"min":{"1min":0.28799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.6349999999999998,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":2.3250000000000002,"5min":1.2050000000000001,"15min":0.78300000000000003},"min":{"1min":0.307,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":9.6460000000000008,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.374}]},{"osd":3,"last update":"Mon Feb 12 16:55:32 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":2.0449999999999999,"15min":1.0549999999999999},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.4539999999999997,"5min":38.170999999999999,"15min":38.170999999999999},"last":0.315},{"interface":"front","average":{"1min":1.609,"5min":1.744,"15min":0.95099999999999996},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.4859999999999998,"5min":38.311999999999998,"15min":38.311999999999998},"last":0.36599999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:55:54 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":0.76600000000000001,"15min":0.48299999999999998},"min":{"1min":0.27000000000000002,"5min":0.247,"15min":0.24299999999999999},"max":{"1min":5.4109999999999996,"5min":5.4109999999999996,"15min":5.4109999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.3,"5min":0.61799999999999999,"15min":0.42799999999999999},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"last":0.26000000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.8009999999999999,"15min":0.84899999999999998},"min":{"1min":0.30399999999999999,"5min":0.255,"15min":0.255},"max":{"1min":2.4430000000000001,"5min":104.907,"15min":104.907},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.9650000000000001,"15min":0.90900000000000003},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.4560000000000004,"5min":105.395,"15min":105.395},"last":0.28699999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.57399999999999995,"15min":0.46800000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":0.69199999999999995,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.7329999999999997,"5min":5.7329999999999997,"15min":5.7329999999999997},"last":0.40100000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757345,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298040,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447112,"statfs":{"total":95995035648,"available":95689842688,"internally_reserved":0,"allocated":835584,"data_stored":429977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.536,"15min":0.85999999999999999},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.1349999999999998,"5min":36.880000000000003,"15min":36.880000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":2.4769999999999999,"5min":1.7549999999999999,"15min":0.90900000000000003},"min":{"1min":0.32700000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":6.7919999999999998,"5min":36.954000000000001,"15min":36.954000000000001},"last":0.56799999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":1.427,"15min":0.81799999999999995},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":15.920999999999999,"5min":15.920999999999999,"15min":15.920999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":2.7200000000000002,"5min":1.571,"15min":0.85999999999999999},"min":{"1min":0.32400000000000001,"5min":0.29099999999999998,"15min":0.28100000000000003},"max":{"1min":11.48,"5min":12.221,"15min":12.221},"last":0.51000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":1.5700000000000001,"15min":0.877},"min":{"1min":0.247,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":11.542999999999999,"5min":18.218,"15min":18.218},"last":0.52500000000000002},{"interface":"front","average":{"1min":2.1259999999999999,"5min":1.595,"15min":0.86599999999999999},"min":{"1min":0.33100000000000002,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":7.0869999999999997,"5min":18.327000000000002,"15min":18.327000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.758,"15min":0.93799999999999994},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.3179999999999996,"5min":18.251000000000001,"15min":18.251000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.647,"15min":0.88900000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":6.024,"5min":20.263000000000002,"15min":20.263000000000002},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.64600000000000002,"15min":0.46600000000000003},"min":{"1min":0.26500000000000001,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":4.3710000000000004,"5min":4.3710000000000004,"15min":4.3710000000000004},"last":0.432},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.65100000000000002,"15min":0.47199999999999998},"min":{"1min":0.29999999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.1600000000000001,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":0.30199999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.55700000000000005},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.65100000000000002,"15min":0.51100000000000001},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.8159999999999998,"5min":2.827,"15min":2.827},"last":0.39100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:55:45 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":0.72799999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.202,"5min":0.752,"15min":0.57399999999999995},"min":{"1min":0.40999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:56:39.089 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T16:56:39.438 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:56:39.438 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:56:39.438 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:56:39.438 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:56:39.439 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:56:39.445 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:56:39.464 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:56:39.483 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:56:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:56:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:56:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:56:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:56:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:56:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:56:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:56:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:56:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:38.216914+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:56:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:38.218022+0000 osd.6 (osd.6) 105 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:56:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:38.463476+0000 mgr.y (mgr.24481) 269 : cluster [DBG] pgmap v310: 148 pgs: 7 unknown, 141 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:38.698471+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 deep-scrub starts 2024-02-12T16:56:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:38.699596+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 deep-scrub ok 2024-02-12T16:56:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:39.132151+0000 osd.5 (osd.5) 94 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T16:56:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:39.138751+0000 osd.5 (osd.5) 95 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T16:56:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:39.332532+0000 mon.a (mon.0) 963 : audit [DBG] from='client.? v1:172.21.15.33:0/548594332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:39.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:38.216914+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:56:39.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:38.218022+0000 osd.6 (osd.6) 105 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:56:39.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:38.463476+0000 mgr.y (mgr.24481) 269 : cluster [DBG] pgmap v310: 148 pgs: 7 unknown, 141 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:38.698471+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 deep-scrub starts 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:38.699596+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 deep-scrub ok 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:39.132151+0000 osd.5 (osd.5) 94 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:39.138751+0000 osd.5 (osd.5) 95 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:39.332532+0000 mon.a (mon.0) 963 : audit [DBG] from='client.? v1:172.21.15.33:0/548594332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:38.216914+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.11 scrub starts 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:38.218022+0000 osd.6 (osd.6) 105 : cluster [DBG] 2.11 scrub ok 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:38.463476+0000 mgr.y (mgr.24481) 269 : cluster [DBG] pgmap v310: 148 pgs: 7 unknown, 141 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:38.698471+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 deep-scrub starts 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:38.699596+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 deep-scrub ok 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:39.132151+0000 osd.5 (osd.5) 94 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:39.138751+0000 osd.5 (osd.5) 95 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T16:56:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:39.332532+0000 mon.a (mon.0) 963 : audit [DBG] from='client.? v1:172.21.15.33:0/548594332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:39.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:56:40.167 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-02-12T16:56:40.179 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:40.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:39.247643+0000 osd.6 (osd.6) 106 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:56:40.654 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:39.247643+0000 osd.6 (osd.6) 106 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:56:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:39.248557+0000 osd.6 (osd.6) 107 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:56:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:40.129058+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub starts 2024-02-12T16:56:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:40.152260+0000 osd.5 (osd.5) 97 : cluster [DBG] 6.d scrub ok 2024-02-12T16:56:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:39.248557+0000 osd.6 (osd.6) 107 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:56:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:40.129058+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub starts 2024-02-12T16:56:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:40.152260+0000 osd.5 (osd.5) 97 : cluster [DBG] 6.d scrub ok 2024-02-12T16:56:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:39.247643+0000 osd.6 (osd.6) 106 : cluster [DBG] 2.14 scrub starts 2024-02-12T16:56:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:39.248557+0000 osd.6 (osd.6) 107 : cluster [DBG] 2.14 scrub ok 2024-02-12T16:56:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:40.129058+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub starts 2024-02-12T16:56:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:40.152260+0000 osd.5 (osd.5) 97 : cluster [DBG] 6.d scrub ok 2024-02-12T16:56:40.927 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:40.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:41.032 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:41.065 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:41.163 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:41.165 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:56:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:40] "GET /metrics HTTP/1.1" 200 173279 "" "Prometheus/2.33.4" 2024-02-12T16:56:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:41.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:41.268 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:41.302 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:41.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:41.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:41.457 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:41.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:40.463944+0000 mgr.y (mgr.24481) 270 : cluster [DBG] pgmap v311: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:41.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:40.671549+0000 osd.2 (osd.2) 111 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:56:41.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:40.672723+0000 osd.2 (osd.2) 112 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:56:41.479 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:41.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:41.583 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:41.585 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:41.614 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:41.714 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:41.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:40.463944+0000 mgr.y (mgr.24481) 270 : cluster [DBG] pgmap v311: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:40.671549+0000 osd.2 (osd.2) 111 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:56:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:40.672723+0000 osd.2 (osd.2) 112 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:56:41.748 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:40.463944+0000 mgr.y (mgr.24481) 270 : cluster [DBG] pgmap v311: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:41.748 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:40.671549+0000 osd.2 (osd.2) 111 : cluster [DBG] 5.1c scrub starts 2024-02-12T16:56:41.748 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:40.672723+0000 osd.2 (osd.2) 112 : cluster [DBG] 5.1c scrub ok 2024-02-12T16:56:41.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:41.851 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:41.872 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:41.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:41.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:42.026 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:42.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:42.130 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:42.158 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:42.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:40.893252+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:42.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:40.893326+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:42.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:41.048678+0000 osd.1 (osd.1) 130 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:56:42.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:41.053583+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:56:42.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:41.287883+0000 osd.6 (osd.6) 108 : cluster [DBG] 4.a scrub starts 2024-02-12T16:56:42.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:41.290232+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub ok 2024-02-12T16:56:42.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:41.474223+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-12T16:56:42.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:41.968713+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.1a scrub starts 2024-02-12T16:56:42.481 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:41.972197+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub ok 2024-02-12T16:56:42.482 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:42.114057+0000 osd.2 (osd.2) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:42.482 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:42.114136+0000 osd.2 (osd.2) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:42.482 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:42.161554+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:56:42.482 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:42.174986+0000 osd.5 (osd.5) 99 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:56:42.671 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:56:42.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:56:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:40.893252+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:40.893326+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:41.048678+0000 osd.1 (osd.1) 130 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:41.053583+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:41.287883+0000 osd.6 (osd.6) 108 : cluster [DBG] 4.a scrub starts 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:41.290232+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub ok 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:41.474223+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:41.968713+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.1a scrub starts 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:41.972197+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub ok 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:42.114057+0000 osd.2 (osd.2) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:42.114136+0000 osd.2 (osd.2) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:42.161554+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:56:42.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:42.174986+0000 osd.5 (osd.5) 99 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:56:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:40.893252+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:40.893326+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:41.048678+0000 osd.1 (osd.1) 130 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:41.053583+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:41.287883+0000 osd.6 (osd.6) 108 : cluster [DBG] 4.a scrub starts 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:41.290232+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub ok 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:41.474223+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:41.968713+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.1a scrub starts 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:41.972197+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub ok 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:42.114057+0000 osd.2 (osd.2) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:42.114136+0000 osd.2 (osd.2) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:42.161554+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.19 scrub starts 2024-02-12T16:56:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:42.174986+0000 osd.5 (osd.5) 99 : cluster [DBG] 6.19 scrub ok 2024-02-12T16:56:43.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:56:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:56:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:42.464250+0000 mgr.y (mgr.24481) 271 : cluster [DBG] pgmap v313: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 849 B/s rd, 0 op/s 2024-02-12T16:56:43.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:42.480455+0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-12T16:56:43.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:42.464250+0000 mgr.y (mgr.24481) 271 : cluster [DBG] pgmap v313: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 849 B/s rd, 0 op/s 2024-02-12T16:56:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:42.480455+0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-12T16:56:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:42.464250+0000 mgr.y (mgr.24481) 271 : cluster [DBG] pgmap v313: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 849 B/s rd, 0 op/s 2024-02-12T16:56:43.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:42.480455+0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-12T16:56:44.045 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:56:44.045 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T16:56:44.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T16:56:44.045 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:56:44.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.045 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:15800 7.e 7:7bb04606:::benchmark_data_smithi033_81060_object15799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:39.099070+0000", 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 244.94222576499999, 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052536100000000002, 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 15800 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099070+0000", 2024-02-12T16:56:44.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099070+0000", 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099069+0000", 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3039000000000001e-05 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099092+0000", 2024-02-12T16:56:44.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6980000000000001e-06 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099099+0000", 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.831e-06 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.099103+0000", 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001796593 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.100900+0000", 2024-02-12T16:56:44.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011629999999999999 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.101016+0000", 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9702000000000002e-05 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.101116+0000", 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069400099999999997 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.101810+0000", 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001458198 2024-02-12T16:56:44.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.103268+0000", 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048241877000000002 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.151510+0000", 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1771000000000003e-05 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.151562+0000", 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3912000000000002e-05 2024-02-12T16:56:44.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:39.151606+0000", 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052536100000000002 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49339 7.e 7:784bec61:::benchmark_data_smithi033_81060_object49338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.953170+0000", 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 224.08812572400001, 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.059088168000000003, 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.051 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49339 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953170+0000", 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953170+0000", 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953167+0000", 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2877000000000001e-05 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953190+0000", 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2561000000000001e-05 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953203+0000", 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1310000000000003e-06 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953209+0000", 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6390999999999998e-05 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953245+0000", 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6520000000000005e-05 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953332+0000", 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011073600000000001 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.953442+0000", 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014593099999999999 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.954902+0000", 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013875000000000001 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.955041+0000", 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.057118052000000002 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.012159+0000", 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7213000000000003e-05 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.012216+0000", 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2135000000000002e-05 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.012258+0000", 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.059088168000000003 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:50739 7.e 7:74d27a3b:::benchmark_data_smithi033_81060_object50738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:01.048922+0000", 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 222.99237308900001, 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053326244000000002, 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 50739 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.057 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.048922+0000", 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.048922+0000", 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.048920+0000", 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3774e-05 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.048944+0000", 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1959e-05 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.048956+0000", 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1160000000000001e-06 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.048963+0000", 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6789000000000003e-05 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.048999+0000", 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101803 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.049101+0000", 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5203000000000003e-05 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.060 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.049196+0000", 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001603628 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.050800+0000", 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023887299999999999 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.051039+0000", 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051110396000000002 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.102149+0000", 2024-02-12T16:56:44.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0424000000000003e-05 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.102210+0000", 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8893999999999997e-05 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:01.102249+0000", 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053326244000000002 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.062 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:52932 7.e 7:7c1de004:::benchmark_data_smithi033_81060_object52931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:02.776302+0000", 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 221.26499329200001, 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054558453999999999, 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 52932 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776302+0000", 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776302+0000", 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776300+0000", 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9297000000000001e-05 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776330+0000", 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1063999999999999e-05 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776341+0000", 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6019999999999996e-06 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776347+0000", 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0858000000000005e-05 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776418+0000", 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014633499999999999 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776565+0000", 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017340199999999999 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.776738+0000", 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022935 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.779031+0000", 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001678946 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.780710+0000", 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050045649999999997 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.830756+0000", 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9567000000000001e-05 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.830816+0000", 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5043999999999998e-05 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:02.830861+0000", 2024-02-12T16:56:44.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054558453999999999 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79148 7.e 7:7bc62154:::benchmark_data_smithi033_81060_object79147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.432695+0000", 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 196.60860052199999, 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.081656918999999994, 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.068 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79148 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.432695+0000", 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.432695+0000", 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999962 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.432691+0000", 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1615999999999997e-05 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.432723+0000", 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4508000000000001e-05 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.432737+0000", 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8839999999999994e-06 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.432745+0000", 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8204999999999997e-05 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.432794+0000", 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015226700000000001 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.432946+0000", 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000159624 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.433105+0000", 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043472629999999996 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.437453+0000", 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061528600000000004 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.438068+0000", 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.076192857000000003 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514261+0000", 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2546000000000001e-05 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514313+0000", 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8507999999999998e-05 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514352+0000", 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.081656918999999994 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79156 7.e 7:7b76d634:::benchmark_data_smithi033_81060_object79155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.439993+0000", 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 196.60130234600001, 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074474914000000003, 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79156 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.439993+0000", 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.439993+0000", 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.439991+0000", 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2014999999999998e-05 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440013+0000", 2024-02-12T16:56:44.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0197000000000001e-05 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440023+0000", 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7160000000000002e-06 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440028+0000", 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1242999999999998e-05 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440069+0000", 2024-02-12T16:56:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013491100000000001 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440204+0000", 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010147199999999999 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440306+0000", 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004542206 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.444848+0000", 2024-02-12T16:56:44.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1022000000000001e-05 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.444859+0000", 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.069539567999999996 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514398+0000", 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7320999999999997e-05 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514436+0000", 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2330999999999999e-05 2024-02-12T16:56:44.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514468+0000", 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074474914000000003 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79158 7.e 7:76c987fd:::benchmark_data_smithi033_81060_object79157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.441538+0000", 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 196.59975761600001, 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.073023429000000001, 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.079 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79158 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441538+0000", 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441538+0000", 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441536+0000", 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3479e-05 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441560+0000", 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0379e-05 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441570+0000", 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0939999999999996e-06 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441576+0000", 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010536370000000001 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.442630+0000", 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011433 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.442744+0000", 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000145647 2024-02-12T16:56:44.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.442890+0000", 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020536080000000002 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.444943+0000", 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025985800000000002 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445203+0000", 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.069294128999999996 2024-02-12T16:56:44.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514497+0000", 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4310999999999997e-05 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514532+0000", 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9694999999999999e-05 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514561+0000", 2024-02-12T16:56:44.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.073023429000000001 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80705 7.e 7:7a0a69d6:::benchmark_data_smithi033_81060_object80704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.047835+0000", 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 194.99346052000001, 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.060484980000000001, 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.085 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80705 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047835+0000", 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047835+0000", 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047833+0000", 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1815999999999999e-05 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047855+0000", 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6239999999999997e-06 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047864+0000", 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2480000000000004e-06 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047870+0000", 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4368e-05 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047904+0000", 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5879999999999998e-05 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.047990+0000", 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9106999999999998e-05 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.048089+0000", 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031142919999999998 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.051203+0000", 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011950629999999999 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.052398+0000", 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.05582757 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.108226+0000", 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3736000000000002e-05 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.108280+0000", 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0355999999999998e-05 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.108320+0000", 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.060484980000000001 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:140327 7.e 7:7e3e6219:::benchmark_data_smithi033_81060_object29875:head [delete] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:54:35.677667+0000", 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 128.363628928, 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056878164000000002, 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 140327 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.677667+0000", 2024-02-12T16:56:44.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.677667+0000", 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.677664+0000", 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8789999999999992e-06 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.677673+0000", 2024-02-12T16:56:44.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1698999999999999e-05 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.677685+0000", 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6080000000000001e-06 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.677691+0000", 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00098931299999999991 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.678681+0000", 2024-02-12T16:56:44.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012108399999999999 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.678802+0000", 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014557900000000001 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.678947+0000", 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097954819999999995 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.688743+0000", 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047071739999999997 2024-02-12T16:56:44.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.693450+0000", 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041004578999999999 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.734455+0000", 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6894000000000002e-05 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.734501+0000", 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3306999999999997e-05 2024-02-12T16:56:44.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.734545+0000", 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056878164000000002 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:140364 7.e 7:79eec378:::benchmark_data_smithi033_81060_object29912:head [delete] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:54:35.730787+0000", 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 128.31050888199999, 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.057883268000000002, 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.096 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 140364 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.730787+0000", 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.730787+0000", 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.730785+0000", 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0500000000000003e-06 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.730792+0000", 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4570000000000005e-06 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.730800+0000", 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6210000000000003e-06 2024-02-12T16:56:44.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.730805+0000", 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.693e-05 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.730842+0000", 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000136744 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.730978+0000", 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016422100000000001 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.731143+0000", 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095426120000000007 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.740685+0000", 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011518940000000001 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.741837+0000", 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046741058000000002 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.788578+0000", 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8860000000000003e-05 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.788627+0000", 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2806000000000003e-05 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:35.788670+0000", 2024-02-12T16:56:44.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.057883268000000002 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:140642 7.e 7:74e8715a:::benchmark_data_smithi033_81060_object30190:head [delete] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:54:36.100985+0000", 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 127.940310228, 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056383419999999997, 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 140642 2024-02-12T16:56:44.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.100985+0000", 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.100985+0000", 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.100983+0000", 2024-02-12T16:56:44.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0999999999999993e-06 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.100992+0000", 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3087e-05 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.101005+0000", 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1442e-05 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.101016+0000", 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010629151999999999 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.111645+0000", 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000141283 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.111787+0000", 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000185615 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.111972+0000", 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044621579999999999 2024-02-12T16:56:44.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.116434+0000", 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019772056 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.136206+0000", 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021066281999999999 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.157273+0000", 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0182999999999997e-05 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.157323+0000", 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5809999999999997e-05 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:54:36.157369+0000", 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056383419999999997 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:157571 7.a 7:5a786ba6:::benchmark_data_smithi033_81060_object47119:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T16:56:44.107 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:00.732329+0000", 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 103.308966591, 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053872528000000003, 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 157571 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.732329+0000", 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.732329+0000", 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.732327+0000", 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3499999999999999e-06 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.732334+0000", 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0380999999999999e-05 2024-02-12T16:56:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.732345+0000", 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.703e-06 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.732351+0000", 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014740542000000001 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.747092+0000", 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114183 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.747206+0000", 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00048185300000000002 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5", 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.747688+0000", 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026698137 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.774386+0000", 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3522000000000002e-05 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.774420+0000", 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2959000000000001e-05 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.774452+0000", 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011575907999999999 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.786028+0000", 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7382999999999997e-05 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.786076+0000", 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125659 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:00.786201+0000", 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053872528000000003 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:158316 7.3 7:cec75787:::benchmark_data_smithi033_81060_object47864:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:01.706260+0000", 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 102.335035772, 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.060578052, 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 158316 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706260+0000", 2024-02-12T16:56:44.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706260+0000", 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706257+0000", 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0999999999999993e-06 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706266+0000", 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4501000000000001e-05 2024-02-12T16:56:44.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706280+0000", 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3280000000000002e-06 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706287+0000", 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012421599999999999 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706411+0000", 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000140154 2024-02-12T16:56:44.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706551+0000", 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017701300000000001 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,6", 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.706728+0000", 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025177649999999999 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.709246+0000", 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047020639999999997 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.713948+0000", 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021505949999999999 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.716098+0000", 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050537055999999997 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.766636+0000", 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3387999999999997e-05 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.766699+0000", 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013883799999999999 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:01.766838+0000", 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.060578052 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:158862 7.a 7:53ca8660:::benchmark_data_smithi033_81060_object48410:head [delete] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-02-12T16:56:44.119 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:02.475068+0000", 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 101.566227908, 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056476563, 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 158862 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475068+0000", 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475068+0000", 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475065+0000", 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5490000000000003e-06 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475072+0000", 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9370000000000004e-06 2024-02-12T16:56:44.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475080+0000", 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7999999999999998e-06 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475085+0000", 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021057700000000001 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475295+0000", 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015245800000000001 2024-02-12T16:56:44.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475448+0000", 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018007799999999999 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5", 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.475628+0000", 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031658459999999999 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.478794+0000", 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002069324 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.480863+0000", 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028851328999999998 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.509714+0000", 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021715359 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.531430+0000", 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9847e-05 2024-02-12T16:56:44.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.531470+0000", 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4603000000000004e-05 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:02.531544+0000", 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056476563 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:159956 7.a 7:50584551:::benchmark_data_smithi033_81060_object49504:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:44.125 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:04.161484+0000", 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 99.879811501999995, 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062230184000000001, 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 159956 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.161484+0000", 2024-02-12T16:56:44.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.161484+0000", 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.161484+0000", 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6510000000000001e-06 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.161486+0000", 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3309999999999997e-06 2024-02-12T16:56:44.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.161491+0000", 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4570000000000001e-06 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.161494+0000", 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019763395999999999 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.181257+0000", 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000141884 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.181399+0000", 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000100889 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5", 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.181500+0000", 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039182849999999997 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.185418+0000", 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069643400000000003 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.192383+0000", 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012678824 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.205062+0000", 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018460462 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.223522+0000", 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7630000000000002e-05 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.223580+0000", 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000134504 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.223714+0000", 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062230184000000001 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:159972 7.a 7:592d0aa2:::benchmark_data_smithi033_81060_object49520:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:04.177777+0000", 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 99.863518045999996, 2024-02-12T16:56:44.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.059262376999999998, 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 159972 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.177777+0000", 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.132 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.177777+0000", 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.177775+0000", 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2470000000000008e-06 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.177783+0000", 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1533e-05 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.177795+0000", 2024-02-12T16:56:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4180000000000002e-06 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.177801+0000", 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0076407829999999999 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.185442+0000", 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000132638 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.185575+0000", 2024-02-12T16:56:44.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016184399999999999 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5", 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.185736+0000", 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028611000000000001 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.188598+0000", 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038160529999999998 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.192414+0000", 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019034582000000001 2024-02-12T16:56:44.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.211448+0000", 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.02542049 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.236869+0000", 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9693e-05 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.236918+0000", 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012141700000000001 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:04.237040+0000", 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.059262376999999998 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161134 7.a 7:516aebe6:::benchmark_data_smithi033_81060_object50682:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:05.810397+0000", 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 98.230898248000003, 2024-02-12T16:56:44.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053382242000000003, 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161134 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.810397+0000", 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.138 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.810397+0000", 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.810395+0000", 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3280000000000002e-06 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.810402+0000", 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8199999999999992e-06 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.810411+0000", 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8930000000000002e-06 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.810417+0000", 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0066401359999999996 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.817057+0000", 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000131464 2024-02-12T16:56:44.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.817189+0000", 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013174899999999999 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5", 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.817321+0000", 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053877179999999997 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.822708+0000", 2024-02-12T16:56:44.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022626500000000002 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.824971+0000", 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083167870000000008 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.833288+0000", 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030305963000000002 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.863594+0000", 2024-02-12T16:56:44.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9549000000000002e-05 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.863653+0000", 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000126134 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:05.863779+0000", 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053382242000000003 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:161364 7.3 7:c97272a8:::benchmark_data_smithi033_81060_object50912:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:06.114642+0000", 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 97.926653583000004, 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064490308999999996, 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 161364 2024-02-12T16:56:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.114642+0000", 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.114642+0000", 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.114639+0000", 2024-02-12T16:56:44.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.772e-05 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.114657+0000", 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1277999999999999e-05 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.114668+0000", 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0010000000000004e-06 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.114675+0000", 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018795796 2024-02-12T16:56:44.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.133471+0000", 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015483000000000001 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.133626+0000", 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000128515 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,6", 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.133755+0000", 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0065556920000000001 2024-02-12T16:56:44.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.140310+0000", 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010334971 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.150645+0000", 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.259e-06 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.150654+0000", 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028305525000000002 2024-02-12T16:56:44.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.178959+0000", 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2471e-05 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.179012+0000", 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120667 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:06.179132+0000", 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064490308999999996 2024-02-12T16:56:44.149 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:163161 7.a 7:57a274ca:::benchmark_data_smithi033_81060_object52709:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:08.463419+0000", 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 95.577876759999995, 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053468500000000002, 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.150 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 163161 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.463419+0000", 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.463419+0000", 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.463416+0000", 2024-02-12T16:56:44.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9850000000000006e-06 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.463424+0000", 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7363999999999998e-05 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.463441+0000", 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5859999999999999e-06 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.463449+0000", 2024-02-12T16:56:44.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060883500000000002 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.469537+0000", 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047596299999999998 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.470013+0000", 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015236000000000001 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5", 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.470166+0000", 2024-02-12T16:56:44.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054095580000000001 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.475575+0000", 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0667999999999998e-05 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.475616+0000", 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017473787000000001 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.493090+0000", 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023691791 2024-02-12T16:56:44.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.516782+0000", 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0423e-05 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.516812+0000", 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5278000000000007e-05 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.516887+0000", 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053468500000000002 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:163165 7.a 7:55ef0c72:::benchmark_data_smithi033_81060_object52713:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:55:08.464159+0000", 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 95.577136772000003, 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056732507000000001, 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:56:44.156 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 163165 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.464159+0000", 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.464159+0000", 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.464157+0000", 2024-02-12T16:56:44.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4670000000000001e-06 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.464163+0000", 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1959999999999997e-06 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.464173+0000", 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6899999999999997e-06 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.464178+0000", 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086730890000000001 2024-02-12T16:56:44.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.472851+0000", 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000150397 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.473002+0000", 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016468599999999999 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5", 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.473167+0000", 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025811919999999999 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.475748+0000", 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059887630000000002 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.481737+0000", 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011387456000000001 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.493124+0000", 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027587202000000002 2024-02-12T16:56:44.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.520711+0000", 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1887e-05 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.520763+0000", 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012818000000000001 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:55:08.520891+0000", 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056732507000000001 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.161 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.162 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.162 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:56:44.162 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:56:44.162 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:56:44.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:43.075961+0000 osd.1 (osd.1) 132 : cluster [DBG] 4.4 deep-scrub starts 2024-02-12T16:56:44.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:43.083646+0000 osd.1 (osd.1) 133 : cluster [DBG] 4.4 deep-scrub ok 2024-02-12T16:56:44.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:43.269832+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:56:44.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:43.270751+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:56:44.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:43.690636+0000 mgr.y (mgr.24481) 272 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:43.075961+0000 osd.1 (osd.1) 132 : cluster [DBG] 4.4 deep-scrub starts 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:43.083646+0000 osd.1 (osd.1) 133 : cluster [DBG] 4.4 deep-scrub ok 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:43.269832+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:43.270751+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:43.690636+0000 mgr.y (mgr.24481) 272 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:43.075961+0000 osd.1 (osd.1) 132 : cluster [DBG] 4.4 deep-scrub starts 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:43.083646+0000 osd.1 (osd.1) 133 : cluster [DBG] 4.4 deep-scrub ok 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:43.269832+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.18 scrub starts 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:43.270751+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.18 scrub ok 2024-02-12T16:56:44.755 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:43.690636+0000 mgr.y (mgr.24481) 272 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:44.782 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:56:45.135 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:45.138 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:45.140 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:45.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:45.245 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:45.275 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:45.372 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:45.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:45.477 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:45.507 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:44.465017+0000 mgr.y (mgr.24481) 273 : cluster [DBG] pgmap v315: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-12T16:56:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:45.215026+0000 osd.5 (osd.5) 100 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:56:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:45.218022+0000 osd.5 (osd.5) 101 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:56:45.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:44.465017+0000 mgr.y (mgr.24481) 273 : cluster [DBG] pgmap v315: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-12T16:56:45.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:45.215026+0000 osd.5 (osd.5) 100 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:56:45.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:45.218022+0000 osd.5 (osd.5) 101 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:56:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:44.465017+0000 mgr.y (mgr.24481) 273 : cluster [DBG] pgmap v315: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-12T16:56:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:45.215026+0000 osd.5 (osd.5) 100 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:56:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:45.218022+0000 osd.5 (osd.5) 101 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:56:45.956 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:45.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:46.025 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:56:46.025 INFO:teuthology.orchestra.run.smithi033.stdout:{"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":638,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":180,"num_osds":8,"num_up_osds":8,"osd_up_since":1707756570,"num_in_osds":7,"osd_in_since":1707756879,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":148}],"num_pgs":148,"num_pools":7,"num_objects":192,"data_bytes":5796,"bytes_used":2170359808,"bytes_avail":669794889728,"bytes_total":671965249536,"read_bytes_sec":1457,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.33:8443/","prometheus":"http://172.21.15.33:9283/"}},"servicemap":{"epoch":13,"modified":"2024-02-12T16:51:04.286395+0000","services":{"rgw":{"daemons":{"summary":"","24397":{"start_epoch":12,"start_stamp":"2024-02-12T16:49:46.308902+0000","gid":24397,"addr":"172.21.15.33:0/1589842856","metadata":{"arch":"x86_64","ceph_release":"pacific","ceph_version":"ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)","ceph_version_short":"16.2.14-521-g0e714d9a","container_hostname":"smithi033","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi033","id":"foo.a","kernel_description":"#1 SMP Thu Jan 18 15:15:29 UTC 2024","kernel_version":"4.18.0-536.el8.x86_64","mem_swap_kb":"0","mem_total_kb":"32543856","num_handles":"1","os":"Linux","pid":"8","realm_id":"","realm_name":"","zone_id":"4799771a-d28e-4dc8-a5a0-179e1408b6e6","zone_name":"default","zonegroup_id":"0384a922-9dfb-46b6-aed7-e430c5071cb8","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-02-12T16:56:46.061 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:46.093 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:46.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:46.218 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:46.249 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:46.372 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:46.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:46.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:46.477 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:46.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:45.029873+0000 osd.1 (osd.1) 134 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:56:46.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:45.033434+0000 osd.1 (osd.1) 135 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:56:46.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:45.225077+0000 osd.6 (osd.6) 112 : cluster [DBG] 5.11 deep-scrub starts 2024-02-12T16:56:46.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:45.225988+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 deep-scrub ok 2024-02-12T16:56:46.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:45.646806+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:56:46.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:45.647624+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:56:46.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:46.024405+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.33:0/1002100946' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:56:46.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:46.116795+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:46.252478+0000 mon.a (mon.0) 967 : audit [INF] from='client.? v1:172.21.15.33:0/679900966' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:45.029873+0000 osd.1 (osd.1) 134 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:45.033434+0000 osd.1 (osd.1) 135 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:45.225077+0000 osd.6 (osd.6) 112 : cluster [DBG] 5.11 deep-scrub starts 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:45.225988+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 deep-scrub ok 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:45.646806+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:45.647624+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:46.024405+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.33:0/1002100946' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:46.116795+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-12T16:56:46.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:46.252478+0000 mon.a (mon.0) 967 : audit [INF] from='client.? v1:172.21.15.33:0/679900966' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:56:46.507 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:46.606 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:46.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:46.710 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:45.029873+0000 osd.1 (osd.1) 134 : cluster [DBG] 4.1d scrub starts 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:45.033434+0000 osd.1 (osd.1) 135 : cluster [DBG] 4.1d scrub ok 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:45.225077+0000 osd.6 (osd.6) 112 : cluster [DBG] 5.11 deep-scrub starts 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:45.225988+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 deep-scrub ok 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:45.646806+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:45.647624+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:46.024405+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.33:0/1002100946' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:46.116795+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-12T16:56:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:46.252478+0000 mon.a (mon.0) 967 : audit [INF] from='client.? v1:172.21.15.33:0/679900966' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:56:46.742 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:46.763 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:46.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:46.867 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:46.896 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:47.130 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:56:47.345 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:47.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:47.449 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:47.478 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:46.465442+0000 mgr.y (mgr.24481) 274 : cluster [DBG] pgmap v317: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:46.718458+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:56:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:46.719698+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:56:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:47.118461+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:47.125161+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.33:0/679900966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:56:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:47.125218+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-12T16:56:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:47.465399+0000 mon.a (mon.0) 971 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:46.465442+0000 mgr.y (mgr.24481) 274 : cluster [DBG] pgmap v317: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:46.718458+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:46.719698+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:47.118461+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:47.125161+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.33:0/679900966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:47.125218+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:47.465399+0000 mon.a (mon.0) 971 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:46.465442+0000 mgr.y (mgr.24481) 274 : cluster [DBG] pgmap v317: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:47.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:46.718458+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:56:47.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:46.719698+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:56:47.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:47.118461+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:56:47.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:47.125161+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.33:0/679900966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:56:47.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:47.125218+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-12T16:56:47.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:47.465399+0000 mon.a (mon.0) 971 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:47.003570+0000 osd.1 (osd.1) 136 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:56:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:47.004464+0000 osd.1 (osd.1) 137 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:56:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:47.003570+0000 osd.1 (osd.1) 136 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:56:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:47.004464+0000 osd.1 (osd.1) 137 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:56:48.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:47.003570+0000 osd.1 (osd.1) 136 : cluster [DBG] 5.12 scrub starts 2024-02-12T16:56:48.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:47.004464+0000 osd.1 (osd.1) 137 : cluster [DBG] 5.12 scrub ok 2024-02-12T16:56:49.031 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:49.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:49.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:49.167 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:56:49.617 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:48.466133+0000 mgr.y (mgr.24481) 275 : cluster [DBG] pgmap v319: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:49.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:48.466133+0000 mgr.y (mgr.24481) 275 : cluster [DBG] pgmap v319: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:49.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:56:49.719 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:49.749 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:48.466133+0000 mgr.y (mgr.24481) 275 : cluster [DBG] pgmap v319: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:49.751 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:49.772 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:49.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:49.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:49.854 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:56:49.869 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:49.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:49.893 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-02-12T16:56:49.893 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:56:49.906 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:49.927 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:49.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:50.031 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:50.060 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:50.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:50.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:56:50.209 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:50.228 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:56:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:50.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:56:50.331 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:50.360 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:50.434 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:50.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:50.508 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-02-12T16:56:50.538 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:50.567 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:50.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:49.786919+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.33:0/4098335438' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:56:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:49.787257+0000 mon.a (mon.0) 972 : audit [INF] from='client.35027 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:56:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:49.786919+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.33:0/4098335438' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:56:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:49.787257+0000 mon.a (mon.0) 972 : audit [INF] from='client.35027 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:56:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:49.786919+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.33:0/4098335438' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:56:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:49.787257+0000 mon.a (mon.0) 972 : audit [INF] from='client.35027 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:56:51.118 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:51.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:51.195 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:56:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:50] "GET /metrics HTTP/1.1" 200 175689 "" "Prometheus/2.33.4" 2024-02-12T16:56:51.225 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:51.256 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:51.277 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:51.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:51.380 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:51.412 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:51.470 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:56:51.470 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:56:51.470 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:56:51.477 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:56:51.484 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:51.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:51.496 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:56:51.504 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:50.466591+0000 mgr.y (mgr.24481) 276 : cluster [DBG] pgmap v320: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-12T16:56:51.515 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:56:51.533 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:56:51.552 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:56:51.553 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:56:51.553 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:56:51.553 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:56:51.553 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:56:51.553 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:56:51.554 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:56:51.554 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:56:51.554 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:56:51.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:50.466591+0000 mgr.y (mgr.24481) 276 : cluster [DBG] pgmap v320: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:50.502580+0000 mon.a (mon.0) 973 : audit [INF] from='client.35027 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:50.502653+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:51.118061+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:51.405198+0000 mon.a (mon.0) 976 : audit [DBG] from='client.? v1:172.21.15.33:0/4115665683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:50.466591+0000 mgr.y (mgr.24481) 276 : cluster [DBG] pgmap v320: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:50.502580+0000 mon.a (mon.0) 973 : audit [INF] from='client.35027 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:50.502653+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:51.118061+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-12T16:56:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:51.405198+0000 mon.a (mon.0) 976 : audit [DBG] from='client.? v1:172.21.15.33:0/4115665683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:51.965 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:51.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:50.502580+0000 mon.a (mon.0) 973 : audit [INF] from='client.35027 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:56:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:50.502653+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-12T16:56:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:51.118061+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-12T16:56:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:51.405198+0000 mon.a (mon.0) 976 : audit [DBG] from='client.? v1:172.21.15.33:0/4115665683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:56:52.070 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:52.097 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:52.669 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:52.669 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:52.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:56:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:56:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:56:53.410 DEBUG:teuthology.orchestra.run.smithi033:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-02-12T16:56:53.413 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:56:53.413 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:56:53.461 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:hints = 1 2024-02-12T16:56:53.461 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:56:53.461 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object prefix: benchmark_data_smithi033_105085 2024-02-12T16:56:53.462 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:56:53.462 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:56:53.636 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:56:53.636 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:56:53.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:52.467084+0000 mgr.y (mgr.24481) 277 : cluster [DBG] pgmap v323: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:52.467084+0000 mgr.y (mgr.24481) 277 : cluster [DBG] pgmap v323: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:53.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:56:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:56:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:52.467084+0000 mgr.y (mgr.24481) 277 : cluster [DBG] pgmap v323: 148 pgs: 148 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T16:56:54.279 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:54.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:54.300 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:56:54.388 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:54.426 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:54.455 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:54.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:54.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 1 16 3757 3741 233.781 233.812 0.00219479 0.0042611 2024-02-12T16:56:54.521 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:53.700844+0000 mgr.y (mgr.24481) 278 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:54.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:54.601 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:54.625 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:54.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:54.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:54.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:53.700844+0000 mgr.y (mgr.24481) 278 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:54.772 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:54.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:53.700844+0000 mgr.y (mgr.24481) 278 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:56:54.800 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:56:54.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:54.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:56:54.905 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:54.958 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:55.462 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 2 16 7017 7001 218.748 203.75 0.00457092 0.00456188 2024-02-12T16:56:55.504 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:56:55.519 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:54.467993+0000 mgr.y (mgr.24481) 279 : cluster [DBG] pgmap v324: 148 pgs: 148 active+clean; 3.3 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 453 KiB/s wr, 8 op/s 2024-02-12T16:56:55.716 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:56:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:55.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:56:55.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:54.467993+0000 mgr.y (mgr.24481) 279 : cluster [DBG] pgmap v324: 148 pgs: 148 active+clean; 3.3 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 453 KiB/s wr, 8 op/s 2024-02-12T16:56:55.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:55.869 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:55.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:54.467993+0000 mgr.y (mgr.24481) 279 : cluster [DBG] pgmap v324: 148 pgs: 148 active+clean; 3.3 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 453 KiB/s wr, 8 op/s 2024-02-12T16:56:56.339 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:56:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:56.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:56:56.444 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:56.462 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 3 16 9768 9752 203.133 171.938 0.00458117 0.00490932 2024-02-12T16:56:56.490 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:56.950 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:56.950 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:56:56.959 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:56.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:56:57.060 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:57.113 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:56:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:57.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:56:57.213 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:56.125694+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-12T16:56:57.213 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:56.468921+0000 mgr.y (mgr.24481) 280 : cluster [DBG] pgmap v326: 148 pgs: 148 active+clean; 3.3 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 554 KiB/s wr, 9 op/s 2024-02-12T16:56:57.213 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:56.903917+0000 mon.c (mon.2) 114 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:56:57.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:56.125694+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-12T16:56:57.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:56.468921+0000 mgr.y (mgr.24481) 280 : cluster [DBG] pgmap v326: 148 pgs: 148 active+clean; 3.3 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 554 KiB/s wr, 9 op/s 2024-02-12T16:56:57.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:56.903917+0000 mon.c (mon.2) 114 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:56:57.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:56:57.288 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:56:57.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:56:57.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:56:57.418 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:57.462 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 4 16 12457 12441 194.359 168.062 0.00494369 0.00513906 2024-02-12T16:56:57.471 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:56:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:56.125694+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-12T16:56:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:56.468921+0000 mgr.y (mgr.24481) 280 : cluster [DBG] pgmap v326: 148 pgs: 148 active+clean; 3.3 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 554 KiB/s wr, 9 op/s 2024-02-12T16:56:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:56.903917+0000 mon.c (mon.2) 114 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:56:58.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:57.384594+0000 mon.c (mon.2) 115 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:56:58.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:57.386777+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:56:58.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:57.394836+0000 mon.a (mon.0) 978 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:58.405 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:57.384594+0000 mon.c (mon.2) 115 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:56:58.405 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:57.386777+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:56:58.405 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:57.394836+0000 mon.a (mon.0) 978 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:58.462 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 5 16 15223 15207 190.059 172.875 0.011118 0.00525498 2024-02-12T16:56:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:57.384594+0000 mon.c (mon.2) 115 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:56:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:57.386777+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:56:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:57.394836+0000 mon.a (mon.0) 978 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:56:59.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 6 16 17912 17896 186.391 168.062 0.00700328 0.00535829 2024-02-12T16:56:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:56:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:56:58.469694+0000 mgr.y (mgr.24481) 281 : cluster [DBG] pgmap v327: 148 pgs: 148 active+clean; 523 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 642 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-12T16:56:59.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:56:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:56:58.469694+0000 mgr.y (mgr.24481) 281 : cluster [DBG] pgmap v327: 148 pgs: 148 active+clean; 523 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 642 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-12T16:56:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:56:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:56:58.469694+0000 mgr.y (mgr.24481) 281 : cluster [DBG] pgmap v327: 148 pgs: 148 active+clean; 523 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 642 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-12T16:56:59.746 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:56:59.755 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:57:00.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:56:59.720131+0000 mgr.y (mgr.24481) 282 : audit [DBG] from='client.15150 v1:172.21.15.33:0/2182503109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:00.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:00.059311+0000 mon.a (mon.0) 979 : audit [INF] from='client.? v1:172.21.15.33:0/3501986223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:00.165 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:56:59.720131+0000 mgr.y (mgr.24481) 282 : audit [DBG] from='client.15150 v1:172.21.15.33:0/2182503109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:00.165 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:00.059311+0000 mon.a (mon.0) 979 : audit [INF] from='client.? v1:172.21.15.33:0/3501986223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:00.183 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:57:00.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 7 15 20260 20245 180.734 146.812 0.00755481 0.00552771 2024-02-12T16:57:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:56:59.720131+0000 mgr.y (mgr.24481) 282 : audit [DBG] from='client.15150 v1:172.21.15.33:0/2182503109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:00.059311+0000 mon.a (mon.0) 979 : audit [INF] from='client.? v1:172.21.15.33:0/3501986223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:01.165 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:57:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:00] "GET /metrics HTTP/1.1" 200 175689 "" "Prometheus/2.33.4" 2024-02-12T16:57:01.216 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:01.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:01.322 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:01.383 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:01.441 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:00.154713+0000 mon.a (mon.0) 980 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:00.168777+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.33:0/3501986223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:00.168819+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:00.470025+0000 mgr.y (mgr.24481) 283 : cluster [DBG] pgmap v329: 148 pgs: 148 active+clean; 942 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.88k op/s 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:01.128191+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:00.154713+0000 mon.a (mon.0) 980 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:00.168777+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.33:0/3501986223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:00.168819+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:00.470025+0000 mgr.y (mgr.24481) 283 : cluster [DBG] pgmap v329: 148 pgs: 148 active+clean; 942 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.88k op/s 2024-02-12T16:57:01.442 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:01.128191+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-12T16:57:01.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 8 16 22724 22708 177.379 153.938 0.00564574 0.00563273 2024-02-12T16:57:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:00.154713+0000 mon.a (mon.0) 980 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:00.168777+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.33:0/3501986223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:00.168819+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-12T16:57:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:00.470025+0000 mgr.y (mgr.24481) 283 : cluster [DBG] pgmap v329: 148 pgs: 148 active+clean; 942 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.88k op/s 2024-02-12T16:57:01.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:01.128191+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-12T16:57:01.927 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-02-12T16:56:58.469286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":548804260,"num_objects":8566,"num_object_clones":0,"num_object_copies":25698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8566,"num_whiteouts":0,"num_read":1134,"num_read_kb":886,"num_write":8802,"num_write_kb":535977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":3321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8886,"ondisk_log_size":8886,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4004800,"kb_used_data":1879112,"kb_used_omap":0,"kb_used_meta":2125568,"kb_avail":652211264,"statfs":{"total":671965249536,"available":667864334336,"internally_reserved":0,"allocated":1924210688,"data_stored":1920942160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2176581632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":548798464,"num_objects":8374,"num_object_clones":0,"num_object_copies":25122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8374,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8374,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.968983"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":64,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230781+0000","last_change":"2024-02-12T16:56:31.966054+0000","last_active":"2024-02-12T16:56:51.230781+0000","last_peered":"2024-02-12T16:56:51.230781+0000","last_clean":"2024-02-12T16:56:51.230781+0000","last_became_active":"2024-02-12T16:54:40.643598+0000","last_became_peered":"2024-02-12T16:54:40.643598+0000","last_unstale":"2024-02-12T16:56:51.230781+0000","last_undegraded":"2024-02-12T16:56:51.230781+0000","last_fullsized":"2024-02-12T16:56:51.230781+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":114,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.191627+0000","last_change":"2024-02-12T16:56:43.270830+0000","last_active":"2024-02-12T16:56:51.191627+0000","last_peered":"2024-02-12T16:56:51.191627+0000","last_clean":"2024-02-12T16:56:51.191627+0000","last_became_active":"2024-02-12T16:52:20.563961+0000","last_became_peered":"2024-02-12T16:52:20.563961+0000","last_unstale":"2024-02-12T16:56:51.191627+0000","last_undegraded":"2024-02-12T16:56:51.191627+0000","last_fullsized":"2024-02-12T16:56:51.191627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:43.270756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.270756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":148,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721370+0000","last_change":"2024-02-12T16:56:35.853405+0000","last_active":"2024-02-12T16:56:50.721370+0000","last_peered":"2024-02-12T16:56:50.721370+0000","last_clean":"2024-02-12T16:56:50.721370+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:56:50.721370+0000","last_undegraded":"2024-02-12T16:56:50.721370+0000","last_fullsized":"2024-02-12T16:56:50.721370+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.853330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.853330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":117,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229649+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T16:56:51.229649+0000","last_peered":"2024-02-12T16:56:51.229649+0000","last_clean":"2024-02-12T16:56:51.229649+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T16:56:51.229649+0000","last_undegraded":"2024-02-12T16:56:51.229649+0000","last_fullsized":"2024-02-12T16:56:51.229649+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":104,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117558+0000","last_change":"2024-02-12T16:53:44.946159+0000","last_active":"2024-02-12T16:56:51.117558+0000","last_peered":"2024-02-12T16:56:51.117558+0000","last_clean":"2024-02-12T16:56:51.117558+0000","last_became_active":"2024-02-12T16:52:20.561617+0000","last_became_peered":"2024-02-12T16:52:20.561617+0000","last_unstale":"2024-02-12T16:56:51.117558+0000","last_undegraded":"2024-02-12T16:56:51.117558+0000","last_fullsized":"2024-02-12T16:56:51.117558+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":146,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503814+0000","last_change":"2024-02-12T16:56:30.146707+0000","last_active":"2024-02-12T16:56:50.503814+0000","last_peered":"2024-02-12T16:56:50.503814+0000","last_clean":"2024-02-12T16:56:50.503814+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:56:50.503814+0000","last_undegraded":"2024-02-12T16:56:50.503814+0000","last_fullsized":"2024-02-12T16:56:50.503814+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:30.146643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.146643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":63,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721754+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T16:56:50.721754+0000","last_peered":"2024-02-12T16:56:50.721754+0000","last_clean":"2024-02-12T16:56:50.721754+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T16:56:50.721754+0000","last_undegraded":"2024-02-12T16:56:50.721754+0000","last_fullsized":"2024-02-12T16:56:50.721754+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":166,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229359+0000","last_change":"2024-02-12T16:55:33.609901+0000","last_active":"2024-02-12T16:56:51.229359+0000","last_peered":"2024-02-12T16:56:51.229359+0000","last_clean":"2024-02-12T16:56:51.229359+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:56:51.229359+0000","last_undegraded":"2024-02-12T16:56:51.229359+0000","last_fullsized":"2024-02-12T16:56:51.229359+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:55:33.609834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:33.609834+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":154,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117644+0000","last_change":"2024-02-12T16:56:42.175069+0000","last_active":"2024-02-12T16:56:51.117644+0000","last_peered":"2024-02-12T16:56:51.117644+0000","last_clean":"2024-02-12T16:56:51.117644+0000","last_became_active":"2024-02-12T16:54:40.628037+0000","last_became_peered":"2024-02-12T16:54:40.628037+0000","last_unstale":"2024-02-12T16:56:51.117644+0000","last_undegraded":"2024-02-12T16:56:51.117644+0000","last_fullsized":"2024-02-12T16:56:51.117644+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:42.174992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:42.174992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":145,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230715+0000","last_change":"2024-02-12T16:56:41.972274+0000","last_active":"2024-02-12T16:56:51.230715+0000","last_peered":"2024-02-12T16:56:51.230715+0000","last_clean":"2024-02-12T16:56:51.230715+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:56:51.230715+0000","last_undegraded":"2024-02-12T16:56:51.230715+0000","last_fullsized":"2024-02-12T16:56:51.230715+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.972202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.972202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":157,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231357+0000","last_change":"2024-02-12T16:54:38.784259+0000","last_active":"2024-02-12T16:56:51.231357+0000","last_peered":"2024-02-12T16:56:51.231357+0000","last_clean":"2024-02-12T16:56:51.231357+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:56:51.231357+0000","last_undegraded":"2024-02-12T16:56:51.231357+0000","last_fullsized":"2024-02-12T16:56:51.231357+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":86,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.502554+0000","last_change":"2024-02-12T16:54:41.193544+0000","last_active":"2024-02-12T16:56:50.502554+0000","last_peered":"2024-02-12T16:56:50.502554+0000","last_clean":"2024-02-12T16:56:50.502554+0000","last_became_active":"2024-02-12T16:54:40.639106+0000","last_became_peered":"2024-02-12T16:54:40.639106+0000","last_unstale":"2024-02-12T16:56:50.502554+0000","last_undegraded":"2024-02-12T16:56:50.502554+0000","last_fullsized":"2024-02-12T16:56:50.502554+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:53:45.658292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.658292+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":139,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.720998+0000","last_change":"2024-02-12T16:53:51.204401+0000","last_active":"2024-02-12T16:56:50.720998+0000","last_peered":"2024-02-12T16:56:50.720998+0000","last_clean":"2024-02-12T16:56:50.720998+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:56:50.720998+0000","last_undegraded":"2024-02-12T16:56:50.720998+0000","last_fullsized":"2024-02-12T16:56:50.720998+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":147,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117886+0000","last_change":"2024-02-12T16:56:24.210155+0000","last_active":"2024-02-12T16:56:51.117886+0000","last_peered":"2024-02-12T16:56:51.117886+0000","last_clean":"2024-02-12T16:56:51.117886+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:56:51.117886+0000","last_undegraded":"2024-02-12T16:56:51.117886+0000","last_fullsized":"2024-02-12T16:56:51.117886+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:24.210075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.210075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":149,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231211+0000","last_change":"2024-02-12T16:56:35.897211+0000","last_active":"2024-02-12T16:56:51.231211+0000","last_peered":"2024-02-12T16:56:51.231211+0000","last_clean":"2024-02-12T16:56:51.231211+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:56:51.231211+0000","last_undegraded":"2024-02-12T16:56:51.231211+0000","last_fullsized":"2024-02-12T16:56:51.231211+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.897088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.897088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":96,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721164+0000","last_change":"2024-02-12T16:54:41.013328+0000","last_active":"2024-02-12T16:56:50.721164+0000","last_peered":"2024-02-12T16:56:50.721164+0000","last_clean":"2024-02-12T16:56:50.721164+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T16:56:50.721164+0000","last_undegraded":"2024-02-12T16:56:50.721164+0000","last_fullsized":"2024-02-12T16:56:50.721164+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:48.029049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.029049+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":66,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192564+0000","last_change":"2024-02-12T16:56:23.345846+0000","last_active":"2024-02-12T16:56:51.192564+0000","last_peered":"2024-02-12T16:56:51.192564+0000","last_clean":"2024-02-12T16:56:51.192564+0000","last_became_active":"2024-02-12T16:54:40.628201+0000","last_became_peered":"2024-02-12T16:54:40.628201+0000","last_unstale":"2024-02-12T16:56:51.192564+0000","last_undegraded":"2024-02-12T16:56:51.192564+0000","last_fullsized":"2024-02-12T16:56:51.192564+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":67,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229548+0000","last_change":"2024-02-12T16:56:40.672807+0000","last_active":"2024-02-12T16:56:51.229548+0000","last_peered":"2024-02-12T16:56:51.229548+0000","last_clean":"2024-02-12T16:56:51.229548+0000","last_became_active":"2024-02-12T16:54:40.628744+0000","last_became_peered":"2024-02-12T16:54:40.628744+0000","last_unstale":"2024-02-12T16:56:51.229548+0000","last_undegraded":"2024-02-12T16:56:51.229548+0000","last_fullsized":"2024-02-12T16:56:51.229548+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.672727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.672727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":59,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231089+0000","last_change":"2024-02-12T16:54:40.633472+0000","last_active":"2024-02-12T16:56:51.231089+0000","last_peered":"2024-02-12T16:56:51.231089+0000","last_clean":"2024-02-12T16:56:51.231089+0000","last_became_active":"2024-02-12T16:54:40.633129+0000","last_became_peered":"2024-02-12T16:54:40.633129+0000","last_unstale":"2024-02-12T16:56:51.231089+0000","last_undegraded":"2024-02-12T16:56:51.231089+0000","last_fullsized":"2024-02-12T16:56:51.231089+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:03.947783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:03.947783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":110,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503605+0000","last_change":"2024-02-12T16:56:45.033557+0000","last_active":"2024-02-12T16:56:50.503605+0000","last_peered":"2024-02-12T16:56:50.503605+0000","last_clean":"2024-02-12T16:56:50.503605+0000","last_became_active":"2024-02-12T16:54:40.626676+0000","last_became_peered":"2024-02-12T16:54:40.626676+0000","last_unstale":"2024-02-12T16:56:50.503605+0000","last_undegraded":"2024-02-12T16:56:50.503605+0000","last_fullsized":"2024-02-12T16:56:50.503605+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:56:45.033441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.033441+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":150,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229619+0000","last_change":"2024-02-12T16:56:18.464298+0000","last_active":"2024-02-12T16:56:51.229619+0000","last_peered":"2024-02-12T16:56:51.229619+0000","last_clean":"2024-02-12T16:56:51.229619+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:56:51.229619+0000","last_undegraded":"2024-02-12T16:56:51.229619+0000","last_fullsized":"2024-02-12T16:56:51.229619+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":146,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503516+0000","last_change":"2024-02-12T16:56:32.102101+0000","last_active":"2024-02-12T16:56:50.503516+0000","last_peered":"2024-02-12T16:56:50.503516+0000","last_clean":"2024-02-12T16:56:50.503516+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:56:50.503516+0000","last_undegraded":"2024-02-12T16:56:50.503516+0000","last_fullsized":"2024-02-12T16:56:50.503516+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.102022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.102022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":155,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192107+0000","last_change":"2024-02-12T16:56:25.339812+0000","last_active":"2024-02-12T16:56:51.192107+0000","last_peered":"2024-02-12T16:56:51.192107+0000","last_clean":"2024-02-12T16:56:51.192107+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:56:51.192107+0000","last_undegraded":"2024-02-12T16:56:51.192107+0000","last_fullsized":"2024-02-12T16:56:51.192107+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":203,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229705+0000","last_change":"2024-02-12T16:54:40.681231+0000","last_active":"2024-02-12T16:56:51.229705+0000","last_peered":"2024-02-12T16:56:51.229705+0000","last_clean":"2024-02-12T16:56:51.229705+0000","last_became_active":"2024-02-12T16:54:40.630167+0000","last_became_peered":"2024-02-12T16:54:40.630167+0000","last_unstale":"2024-02-12T16:56:51.229705+0000","last_undegraded":"2024-02-12T16:56:51.229705+0000","last_fullsized":"2024-02-12T16:56:51.229705+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:53:48.468547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.468547+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":144,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503322+0000","last_change":"2024-02-12T16:56:31.122644+0000","last_active":"2024-02-12T16:56:50.503322+0000","last_peered":"2024-02-12T16:56:50.503322+0000","last_clean":"2024-02-12T16:56:50.503322+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:56:50.503322+0000","last_undegraded":"2024-02-12T16:56:50.503322+0000","last_fullsized":"2024-02-12T16:56:50.503322+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":139,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721849+0000","last_change":"2024-02-12T16:53:04.085929+0000","last_active":"2024-02-12T16:56:50.721849+0000","last_peered":"2024-02-12T16:56:50.721849+0000","last_clean":"2024-02-12T16:56:50.721849+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:56:50.721849+0000","last_undegraded":"2024-02-12T16:56:50.721849+0000","last_fullsized":"2024-02-12T16:56:50.721849+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:04.085854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.085854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":64,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.191826+0000","last_change":"2024-02-12T16:56:33.292072+0000","last_active":"2024-02-12T16:56:51.191826+0000","last_peered":"2024-02-12T16:56:51.191826+0000","last_clean":"2024-02-12T16:56:51.191826+0000","last_became_active":"2024-02-12T16:54:40.640452+0000","last_became_peered":"2024-02-12T16:54:40.640452+0000","last_unstale":"2024-02-12T16:56:51.191826+0000","last_undegraded":"2024-02-12T16:56:51.191826+0000","last_fullsized":"2024-02-12T16:56:51.191826+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":167,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192519+0000","last_change":"2024-02-12T16:56:27.293762+0000","last_active":"2024-02-12T16:56:51.192519+0000","last_peered":"2024-02-12T16:56:51.192519+0000","last_clean":"2024-02-12T16:56:51.192519+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:56:51.192519+0000","last_undegraded":"2024-02-12T16:56:51.192519+0000","last_fullsized":"2024-02-12T16:56:51.192519+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":140,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231455+0000","last_change":"2024-02-12T16:53:04.169987+0000","last_active":"2024-02-12T16:56:51.231455+0000","last_peered":"2024-02-12T16:56:51.231455+0000","last_clean":"2024-02-12T16:56:51.231455+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:56:51.231455+0000","last_undegraded":"2024-02-12T16:56:51.231455+0000","last_fullsized":"2024-02-12T16:56:51.231455+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:53:04.169892+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.169892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":245,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:56.126573+0000","last_change":"2024-02-12T16:56:36.211166+0000","last_active":"2024-02-12T16:56:56.126573+0000","last_peered":"2024-02-12T16:56:56.126573+0000","last_clean":"2024-02-12T16:56:56.126573+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:56:56.126573+0000","last_undegraded":"2024-02-12T16:56:56.126573+0000","last_fullsized":"2024-02-12T16:56:56.126573+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:36.211099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:36.211099+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":146,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117440+0000","last_change":"2024-02-12T16:52:47.997796+0000","last_active":"2024-02-12T16:56:51.117440+0000","last_peered":"2024-02-12T16:56:51.117440+0000","last_clean":"2024-02-12T16:56:51.117440+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:56:51.117440+0000","last_undegraded":"2024-02-12T16:56:51.117440+0000","last_fullsized":"2024-02-12T16:56:51.117440+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:47.997708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.997708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":182,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721801+0000","last_change":"2024-02-12T16:56:32.825256+0000","last_active":"2024-02-12T16:56:50.721801+0000","last_peered":"2024-02-12T16:56:50.721801+0000","last_clean":"2024-02-12T16:56:50.721801+0000","last_became_active":"2024-02-12T16:54:40.650318+0000","last_became_peered":"2024-02-12T16:54:40.650318+0000","last_unstale":"2024-02-12T16:56:50.721801+0000","last_undegraded":"2024-02-12T16:56:50.721801+0000","last_fullsized":"2024-02-12T16:56:50.721801+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":58,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721173+0000","last_change":"2024-02-12T16:54:40.650920+0000","last_active":"2024-02-12T16:56:50.721173+0000","last_peered":"2024-02-12T16:56:50.721173+0000","last_clean":"2024-02-12T16:56:50.721173+0000","last_became_active":"2024-02-12T16:54:40.650168+0000","last_became_peered":"2024-02-12T16:54:40.650168+0000","last_unstale":"2024-02-12T16:56:50.721173+0000","last_undegraded":"2024-02-12T16:56:50.721173+0000","last_fullsized":"2024-02-12T16:56:50.721173+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.665249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.665249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":140,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230943+0000","last_change":"2024-02-12T16:53:05.141324+0000","last_active":"2024-02-12T16:56:51.230943+0000","last_peered":"2024-02-12T16:56:51.230943+0000","last_clean":"2024-02-12T16:56:51.230943+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:56:51.230943+0000","last_undegraded":"2024-02-12T16:56:51.230943+0000","last_fullsized":"2024-02-12T16:56:51.230943+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":150,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.191365+0000","last_change":"2024-02-12T16:54:39.665472+0000","last_active":"2024-02-12T16:56:51.191365+0000","last_peered":"2024-02-12T16:56:51.191365+0000","last_clean":"2024-02-12T16:56:51.191365+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:56:51.191365+0000","last_undegraded":"2024-02-12T16:56:51.191365+0000","last_fullsized":"2024-02-12T16:56:51.191365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.665378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.665378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":107,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230304+0000","last_change":"2024-02-12T16:56:24.854608+0000","last_active":"2024-02-12T16:56:51.230304+0000","last_peered":"2024-02-12T16:56:51.230304+0000","last_clean":"2024-02-12T16:56:51.230304+0000","last_became_active":"2024-02-12T16:54:40.632501+0000","last_became_peered":"2024-02-12T16:54:40.632501+0000","last_unstale":"2024-02-12T16:56:51.230304+0000","last_undegraded":"2024-02-12T16:56:51.230304+0000","last_fullsized":"2024-02-12T16:56:51.230304+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":138,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231376+0000","last_change":"2024-02-12T16:52:55.241717+0000","last_active":"2024-02-12T16:56:51.231376+0000","last_peered":"2024-02-12T16:56:51.231376+0000","last_clean":"2024-02-12T16:56:51.231376+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:56:51.231376+0000","last_undegraded":"2024-02-12T16:56:51.231376+0000","last_fullsized":"2024-02-12T16:56:51.231376+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.241622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.241622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":151,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.191894+0000","last_change":"2024-02-12T16:56:45.226096+0000","last_active":"2024-02-12T16:56:51.191894+0000","last_peered":"2024-02-12T16:56:51.191894+0000","last_clean":"2024-02-12T16:56:51.191894+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:56:51.191894+0000","last_undegraded":"2024-02-12T16:56:51.191894+0000","last_fullsized":"2024-02-12T16:56:51.191894+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.225996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":146,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.118631+0000","last_change":"2024-02-12T16:52:49.986507+0000","last_active":"2024-02-12T16:56:51.118631+0000","last_peered":"2024-02-12T16:56:51.118631+0000","last_clean":"2024-02-12T16:56:51.118631+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:56:51.118631+0000","last_undegraded":"2024-02-12T16:56:51.118631+0000","last_fullsized":"2024-02-12T16:56:51.118631+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.986403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.986403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":82,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.501929+0000","last_change":"2024-02-12T16:54:41.081430+0000","last_active":"2024-02-12T16:56:50.501929+0000","last_peered":"2024-02-12T16:56:50.501929+0000","last_clean":"2024-02-12T16:56:50.501929+0000","last_became_active":"2024-02-12T16:54:40.639547+0000","last_became_peered":"2024-02-12T16:54:40.639547+0000","last_unstale":"2024-02-12T16:56:50.501929+0000","last_undegraded":"2024-02-12T16:56:50.501929+0000","last_fullsized":"2024-02-12T16:56:50.501929+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":63,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721027+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T16:56:50.721027+0000","last_peered":"2024-02-12T16:56:50.721027+0000","last_clean":"2024-02-12T16:56:50.721027+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T16:56:50.721027+0000","last_undegraded":"2024-02-12T16:56:50.721027+0000","last_fullsized":"2024-02-12T16:56:50.721027+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":160,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503055+0000","last_change":"2024-02-12T16:56:47.004534+0000","last_active":"2024-02-12T16:56:50.503055+0000","last_peered":"2024-02-12T16:56:50.503055+0000","last_clean":"2024-02-12T16:56:50.503055+0000","last_became_active":"2024-02-12T16:54:40.628571+0000","last_became_peered":"2024-02-12T16:54:40.628571+0000","last_unstale":"2024-02-12T16:56:50.503055+0000","last_undegraded":"2024-02-12T16:56:50.503055+0000","last_fullsized":"2024-02-12T16:56:50.503055+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":150,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503522+0000","last_change":"2024-02-12T16:54:01.773110+0000","last_active":"2024-02-12T16:56:50.503522+0000","last_peered":"2024-02-12T16:56:50.503522+0000","last_clean":"2024-02-12T16:56:50.503522+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:56:50.503522+0000","last_undegraded":"2024-02-12T16:56:50.503522+0000","last_fullsized":"2024-02-12T16:56:50.503522+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":171,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229854+0000","last_change":"2024-02-12T16:56:34.426891+0000","last_active":"2024-02-12T16:56:51.229854+0000","last_peered":"2024-02-12T16:56:51.229854+0000","last_clean":"2024-02-12T16:56:51.229854+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:56:51.229854+0000","last_undegraded":"2024-02-12T16:56:51.229854+0000","last_fullsized":"2024-02-12T16:56:51.229854+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:34.426800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:34.426800+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":137,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.720741+0000","last_change":"2024-02-12T16:54:40.194031+0000","last_active":"2024-02-12T16:56:50.720741+0000","last_peered":"2024-02-12T16:56:50.720741+0000","last_clean":"2024-02-12T16:56:50.720741+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:56:50.720741+0000","last_undegraded":"2024-02-12T16:56:50.720741+0000","last_fullsized":"2024-02-12T16:56:50.720741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":79,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721089+0000","last_change":"2024-02-12T16:56:23.899212+0000","last_active":"2024-02-12T16:56:50.721089+0000","last_peered":"2024-02-12T16:56:50.721089+0000","last_clean":"2024-02-12T16:56:50.721089+0000","last_became_active":"2024-02-12T16:54:40.650612+0000","last_became_peered":"2024-02-12T16:54:40.650612+0000","last_unstale":"2024-02-12T16:56:50.721089+0000","last_undegraded":"2024-02-12T16:56:50.721089+0000","last_fullsized":"2024-02-12T16:56:50.721089+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":159,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192189+0000","last_change":"2024-02-12T16:56:39.248638+0000","last_active":"2024-02-12T16:56:51.192189+0000","last_peered":"2024-02-12T16:56:51.192189+0000","last_clean":"2024-02-12T16:56:51.192189+0000","last_became_active":"2024-02-12T16:54:40.629419+0000","last_became_peered":"2024-02-12T16:54:40.629419+0000","last_unstale":"2024-02-12T16:56:51.192189+0000","last_undegraded":"2024-02-12T16:56:51.192189+0000","last_fullsized":"2024-02-12T16:56:51.192189+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:39.248562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.248562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":168,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.502980+0000","last_change":"2024-02-12T16:54:31.954069+0000","last_active":"2024-02-12T16:56:50.502980+0000","last_peered":"2024-02-12T16:56:50.502980+0000","last_clean":"2024-02-12T16:56:50.502980+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:56:50.502980+0000","last_undegraded":"2024-02-12T16:56:50.502980+0000","last_fullsized":"2024-02-12T16:56:50.502980+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":109,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230352+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T16:56:51.230352+0000","last_peered":"2024-02-12T16:56:51.230352+0000","last_clean":"2024-02-12T16:56:51.230352+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T16:56:51.230352+0000","last_undegraded":"2024-02-12T16:56:51.230352+0000","last_fullsized":"2024-02-12T16:56:51.230352+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"138'9","reported_seq":136,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:56.134169+0000","last_change":"2024-02-12T16:56:20.829720+0000","last_active":"2024-02-12T16:56:56.134169+0000","last_peered":"2024-02-12T16:56:56.134169+0000","last_clean":"2024-02-12T16:56:56.134169+0000","last_became_active":"2024-02-12T16:54:40.644025+0000","last_became_peered":"2024-02-12T16:54:40.644025+0000","last_unstale":"2024-02-12T16:56:56.134169+0000","last_undegraded":"2024-02-12T16:56:56.134169+0000","last_fullsized":"2024-02-12T16:56:56.134169+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":152,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721288+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T16:56:50.721288+0000","last_peered":"2024-02-12T16:56:50.721288+0000","last_clean":"2024-02-12T16:56:50.721288+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T16:56:50.721288+0000","last_undegraded":"2024-02-12T16:56:50.721288+0000","last_fullsized":"2024-02-12T16:56:50.721288+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":179,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.118259+0000","last_change":"2024-02-12T16:54:40.641346+0000","last_active":"2024-02-12T16:56:51.118259+0000","last_peered":"2024-02-12T16:56:51.118259+0000","last_clean":"2024-02-12T16:56:51.118259+0000","last_became_active":"2024-02-12T16:54:40.626031+0000","last_became_peered":"2024-02-12T16:54:40.626031+0000","last_unstale":"2024-02-12T16:56:51.118259+0000","last_undegraded":"2024-02-12T16:56:51.118259+0000","last_fullsized":"2024-02-12T16:56:51.118259+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":146,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721655+0000","last_change":"2024-02-12T16:54:40.633046+0000","last_active":"2024-02-12T16:56:50.721655+0000","last_peered":"2024-02-12T16:56:50.721655+0000","last_clean":"2024-02-12T16:56:50.721655+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T16:56:50.721655+0000","last_undegraded":"2024-02-12T16:56:50.721655+0000","last_fullsized":"2024-02-12T16:56:50.721655+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.105539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.105539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":240,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:56.132863+0000","last_change":"2024-02-12T16:54:00.112924+0000","last_active":"2024-02-12T16:56:56.132863+0000","last_peered":"2024-02-12T16:56:56.132863+0000","last_clean":"2024-02-12T16:56:56.132863+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:56:56.132863+0000","last_undegraded":"2024-02-12T16:56:56.132863+0000","last_fullsized":"2024-02-12T16:56:56.132863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":155,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117659+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T16:56:51.117659+0000","last_peered":"2024-02-12T16:56:51.117659+0000","last_clean":"2024-02-12T16:56:51.117659+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T16:56:51.117659+0000","last_undegraded":"2024-02-12T16:56:51.117659+0000","last_fullsized":"2024-02-12T16:56:51.117659+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":68,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.502947+0000","last_change":"2024-02-12T16:54:42.827081+0000","last_active":"2024-02-12T16:56:50.502947+0000","last_peered":"2024-02-12T16:56:50.502947+0000","last_clean":"2024-02-12T16:56:50.502947+0000","last_became_active":"2024-02-12T16:54:40.639018+0000","last_became_peered":"2024-02-12T16:54:40.639018+0000","last_unstale":"2024-02-12T16:56:50.502947+0000","last_undegraded":"2024-02-12T16:56:50.502947+0000","last_fullsized":"2024-02-12T16:56:50.502947+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:03.616717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.616717+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":142,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231256+0000","last_change":"2024-02-12T16:53:51.532654+0000","last_active":"2024-02-12T16:56:51.231256+0000","last_peered":"2024-02-12T16:56:51.231256+0000","last_clean":"2024-02-12T16:56:51.231256+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:56:51.231256+0000","last_undegraded":"2024-02-12T16:56:51.231256+0000","last_fullsized":"2024-02-12T16:56:51.231256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":156,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.118461+0000","last_change":"2024-02-12T16:56:37.155376+0000","last_active":"2024-02-12T16:56:51.118461+0000","last_peered":"2024-02-12T16:56:51.118461+0000","last_clean":"2024-02-12T16:56:51.118461+0000","last_became_active":"2024-02-12T16:54:40.644226+0000","last_became_peered":"2024-02-12T16:54:40.644226+0000","last_unstale":"2024-02-12T16:56:51.118461+0000","last_undegraded":"2024-02-12T16:56:51.118461+0000","last_fullsized":"2024-02-12T16:56:51.118461+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.155311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.155311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":150,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.191326+0000","last_change":"2024-02-12T16:56:38.218147+0000","last_active":"2024-02-12T16:56:51.191326+0000","last_peered":"2024-02-12T16:56:51.191326+0000","last_clean":"2024-02-12T16:56:51.191326+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:56:51.191326+0000","last_undegraded":"2024-02-12T16:56:51.191326+0000","last_fullsized":"2024-02-12T16:56:51.191326+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.218031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.218031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":88,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230599+0000","last_change":"2024-02-12T16:54:40.903364+0000","last_active":"2024-02-12T16:56:51.230599+0000","last_peered":"2024-02-12T16:56:51.230599+0000","last_clean":"2024-02-12T16:56:51.230599+0000","last_became_active":"2024-02-12T16:54:40.640126+0000","last_became_peered":"2024-02-12T16:54:40.640126+0000","last_unstale":"2024-02-12T16:56:51.230599+0000","last_undegraded":"2024-02-12T16:56:51.230599+0000","last_fullsized":"2024-02-12T16:56:51.230599+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":145,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229416+0000","last_change":"2024-02-12T16:56:38.699688+0000","last_active":"2024-02-12T16:56:51.229416+0000","last_peered":"2024-02-12T16:56:51.229416+0000","last_clean":"2024-02-12T16:56:51.229416+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:56:51.229416+0000","last_undegraded":"2024-02-12T16:56:51.229416+0000","last_fullsized":"2024-02-12T16:56:51.229416+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.699602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":58,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503778+0000","last_change":"2024-02-12T16:54:40.634603+0000","last_active":"2024-02-12T16:56:50.503778+0000","last_peered":"2024-02-12T16:56:50.503778+0000","last_clean":"2024-02-12T16:56:50.503778+0000","last_became_active":"2024-02-12T16:54:40.634261+0000","last_became_peered":"2024-02-12T16:54:40.634261+0000","last_unstale":"2024-02-12T16:56:50.503778+0000","last_undegraded":"2024-02-12T16:56:50.503778+0000","last_fullsized":"2024-02-12T16:56:50.503778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:41.740892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:53:41.740892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":151,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.228765+0000","last_change":"2024-02-12T16:56:22.699402+0000","last_active":"2024-02-12T16:56:51.228765+0000","last_peered":"2024-02-12T16:56:51.228765+0000","last_clean":"2024-02-12T16:56:51.228765+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:56:51.228765+0000","last_undegraded":"2024-02-12T16:56:51.228765+0000","last_fullsized":"2024-02-12T16:56:51.228765+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":180,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721707+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T16:56:50.721707+0000","last_peered":"2024-02-12T16:56:50.721707+0000","last_clean":"2024-02-12T16:56:50.721707+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T16:56:50.721707+0000","last_undegraded":"2024-02-12T16:56:50.721707+0000","last_fullsized":"2024-02-12T16:56:50.721707+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"185'947","reported_seq":964,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.347074+0000","last_change":"2024-02-12T16:56:36.789085+0000","last_active":"2024-02-12T16:56:58.347074+0000","last_peered":"2024-02-12T16:56:58.347074+0000","last_clean":"2024-02-12T16:56:58.347074+0000","last_became_active":"2024-02-12T16:56:36.788797+0000","last_became_peered":"2024-02-12T16:56:36.788797+0000","last_unstale":"2024-02-12T16:56:58.347074+0000","last_undegraded":"2024-02-12T16:56:58.347074+0000","last_fullsized":"2024-02-12T16:56:58.347074+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":947,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62062592,"num_objects":947,"num_object_clones":0,"num_object_copies":2841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":60608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":59,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230909+0000","last_change":"2024-02-12T16:54:40.643841+0000","last_active":"2024-02-12T16:56:51.230909+0000","last_peered":"2024-02-12T16:56:51.230909+0000","last_clean":"2024-02-12T16:56:51.230909+0000","last_became_active":"2024-02-12T16:54:40.643529+0000","last_became_peered":"2024-02-12T16:54:40.643529+0000","last_unstale":"2024-02-12T16:56:51.230909+0000","last_undegraded":"2024-02-12T16:56:51.230909+0000","last_fullsized":"2024-02-12T16:56:51.230909+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":171,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230338+0000","last_change":"2024-02-12T16:55:34.601557+0000","last_active":"2024-02-12T16:56:51.230338+0000","last_peered":"2024-02-12T16:56:51.230338+0000","last_clean":"2024-02-12T16:56:51.230338+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:56:51.230338+0000","last_undegraded":"2024-02-12T16:56:51.230338+0000","last_fullsized":"2024-02-12T16:56:51.230338+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:55:34.601453+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:55:34.601453+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":40,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117354+0000","last_change":"2024-02-12T16:56:05.735251+0000","last_active":"2024-02-12T16:56:51.117354+0000","last_peered":"2024-02-12T16:56:51.117354+0000","last_clean":"2024-02-12T16:56:51.117354+0000","last_became_active":"2024-02-12T16:56:04.871405+0000","last_became_peered":"2024-02-12T16:56:04.871405+0000","last_unstale":"2024-02-12T16:56:51.117354+0000","last_undegraded":"2024-02-12T16:56:51.117354+0000","last_fullsized":"2024-02-12T16:56:51.117354+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:52:44.868438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.868438+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":147,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.228950+0000","last_change":"2024-02-12T16:56:45.647692+0000","last_active":"2024-02-12T16:56:51.228950+0000","last_peered":"2024-02-12T16:56:51.228950+0000","last_clean":"2024-02-12T16:56:51.228950+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:56:51.228950+0000","last_undegraded":"2024-02-12T16:56:51.228950+0000","last_fullsized":"2024-02-12T16:56:51.228950+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.647627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.647627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"184'5","reported_seq":21,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:53.489384+0000","last_change":"2024-02-12T16:56:36.446431+0000","last_active":"2024-02-12T16:56:53.489384+0000","last_peered":"2024-02-12T16:56:53.489384+0000","last_clean":"2024-02-12T16:56:53.489384+0000","last_became_active":"2024-02-12T16:56:36.446107+0000","last_became_peered":"2024-02-12T16:56:36.446107+0000","last_unstale":"2024-02-12T16:56:53.489384+0000","last_undegraded":"2024-02-12T16:56:53.489384+0000","last_fullsized":"2024-02-12T16:56:53.489384+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":146,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.720748+0000","last_change":"2024-02-12T16:54:40.638105+0000","last_active":"2024-02-12T16:56:50.720748+0000","last_peered":"2024-02-12T16:56:50.720748+0000","last_clean":"2024-02-12T16:56:50.720748+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T16:56:50.720748+0000","last_undegraded":"2024-02-12T16:56:50.720748+0000","last_fullsized":"2024-02-12T16:56:50.720748+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.248970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.248970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":150,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503031+0000","last_change":"2024-02-12T16:56:37.009164+0000","last_active":"2024-02-12T16:56:50.503031+0000","last_peered":"2024-02-12T16:56:50.503031+0000","last_clean":"2024-02-12T16:56:50.503031+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:56:50.503031+0000","last_undegraded":"2024-02-12T16:56:50.503031+0000","last_fullsized":"2024-02-12T16:56:50.503031+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.009094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.009094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":135,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230231+0000","last_change":"2024-02-12T16:54:39.421299+0000","last_active":"2024-02-12T16:56:51.230231+0000","last_peered":"2024-02-12T16:56:51.230231+0000","last_clean":"2024-02-12T16:56:51.230231+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:56:51.230231+0000","last_undegraded":"2024-02-12T16:56:51.230231+0000","last_fullsized":"2024-02-12T16:56:51.230231+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:39.421176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.421176+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":234,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229510+0000","last_change":"2024-02-12T16:53:59.173126+0000","last_active":"2024-02-12T16:56:51.229510+0000","last_peered":"2024-02-12T16:56:51.229510+0000","last_clean":"2024-02-12T16:56:51.229510+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:56:51.229510+0000","last_undegraded":"2024-02-12T16:56:51.229510+0000","last_fullsized":"2024-02-12T16:56:51.229510+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"185'917","reported_seq":934,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.321859+0000","last_change":"2024-02-12T16:56:36.788519+0000","last_active":"2024-02-12T16:56:58.321859+0000","last_peered":"2024-02-12T16:56:58.321859+0000","last_clean":"2024-02-12T16:56:58.321859+0000","last_became_active":"2024-02-12T16:56:36.788101+0000","last_became_peered":"2024-02-12T16:56:36.788101+0000","last_unstale":"2024-02-12T16:56:58.321859+0000","last_undegraded":"2024-02-12T16:56:58.321859+0000","last_fullsized":"2024-02-12T16:56:58.321859+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":917,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60096512,"num_objects":917,"num_object_clones":0,"num_object_copies":2751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":917,"num_write_kb":58688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":63,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229836+0000","last_change":"2024-02-12T16:55:40.530363+0000","last_active":"2024-02-12T16:56:51.229836+0000","last_peered":"2024-02-12T16:56:51.229836+0000","last_clean":"2024-02-12T16:56:51.229836+0000","last_became_active":"2024-02-12T16:54:40.625901+0000","last_became_peered":"2024-02-12T16:54:40.625901+0000","last_unstale":"2024-02-12T16:56:51.229836+0000","last_undegraded":"2024-02-12T16:56:51.229836+0000","last_fullsized":"2024-02-12T16:56:51.229836+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:55:40.530269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":151,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117228+0000","last_change":"2024-02-12T16:56:25.227663+0000","last_active":"2024-02-12T16:56:51.117228+0000","last_peered":"2024-02-12T16:56:51.117228+0000","last_clean":"2024-02-12T16:56:51.117228+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:56:51.117228+0000","last_undegraded":"2024-02-12T16:56:51.117228+0000","last_fullsized":"2024-02-12T16:56:51.117228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"180'39","reported_seq":325,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.118506+0000","last_change":"2024-02-12T16:56:43.083756+0000","last_active":"2024-02-12T16:56:51.118506+0000","last_peered":"2024-02-12T16:56:51.118506+0000","last_clean":"2024-02-12T16:56:51.118506+0000","last_became_active":"2024-02-12T16:54:40.639565+0000","last_became_peered":"2024-02-12T16:54:40.639565+0000","last_unstale":"2024-02-12T16:56:51.118506+0000","last_undegraded":"2024-02-12T16:56:51.118506+0000","last_fullsized":"2024-02-12T16:56:51.118506+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'39","last_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.083656+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":144,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721459+0000","last_change":"2024-02-12T16:56:46.719822+0000","last_active":"2024-02-12T16:56:50.721459+0000","last_peered":"2024-02-12T16:56:50.721459+0000","last_clean":"2024-02-12T16:56:50.721459+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:56:50.721459+0000","last_undegraded":"2024-02-12T16:56:50.721459+0000","last_fullsized":"2024-02-12T16:56:50.721459+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:46.719706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:46.719706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"185'868","reported_seq":884,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.330457+0000","last_change":"2024-02-12T16:56:36.443593+0000","last_active":"2024-02-12T16:56:58.330457+0000","last_peered":"2024-02-12T16:56:58.330457+0000","last_clean":"2024-02-12T16:56:58.330457+0000","last_became_active":"2024-02-12T16:56:36.443309+0000","last_became_peered":"2024-02-12T16:56:36.443309+0000","last_unstale":"2024-02-12T16:56:58.330457+0000","last_undegraded":"2024-02-12T16:56:58.330457+0000","last_fullsized":"2024-02-12T16:56:58.330457+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":868,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56885248,"num_objects":868,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":868,"num_write_kb":55552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":149,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231540+0000","last_change":"2024-02-12T16:54:40.640502+0000","last_active":"2024-02-12T16:56:51.231540+0000","last_peered":"2024-02-12T16:56:51.231540+0000","last_clean":"2024-02-12T16:56:51.231540+0000","last_became_active":"2024-02-12T16:54:40.640105+0000","last_became_peered":"2024-02-12T16:54:40.640105+0000","last_unstale":"2024-02-12T16:56:51.231540+0000","last_undegraded":"2024-02-12T16:56:51.231540+0000","last_fullsized":"2024-02-12T16:56:51.231540+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":580,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:52.476606+0000","last_change":"2024-02-12T16:54:30.972535+0000","last_active":"2024-02-12T16:56:52.476606+0000","last_peered":"2024-02-12T16:56:52.476606+0000","last_clean":"2024-02-12T16:56:52.476606+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:56:52.476606+0000","last_undegraded":"2024-02-12T16:56:52.476606+0000","last_fullsized":"2024-02-12T16:56:52.476606+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":427,"num_read_kb":422,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":160,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229771+0000","last_change":"2024-02-12T16:56:20.676702+0000","last_active":"2024-02-12T16:56:51.229771+0000","last_peered":"2024-02-12T16:56:51.229771+0000","last_clean":"2024-02-12T16:56:51.229771+0000","last_became_active":"2024-02-12T16:54:40.632657+0000","last_became_peered":"2024-02-12T16:54:40.632657+0000","last_unstale":"2024-02-12T16:56:51.229771+0000","last_undegraded":"2024-02-12T16:56:51.229771+0000","last_fullsized":"2024-02-12T16:56:51.229771+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":180,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503592+0000","last_change":"2024-02-12T16:56:28.196061+0000","last_active":"2024-02-12T16:56:50.503592+0000","last_peered":"2024-02-12T16:56:50.503592+0000","last_clean":"2024-02-12T16:56:50.503592+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:56:50.503592+0000","last_undegraded":"2024-02-12T16:56:50.503592+0000","last_fullsized":"2024-02-12T16:56:50.503592+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":142,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230793+0000","last_change":"2024-02-12T16:52:45.314737+0000","last_active":"2024-02-12T16:56:51.230793+0000","last_peered":"2024-02-12T16:56:51.230793+0000","last_clean":"2024-02-12T16:56:51.230793+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:56:51.230793+0000","last_undegraded":"2024-02-12T16:56:51.230793+0000","last_fullsized":"2024-02-12T16:56:51.230793+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:45.314637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:45.314637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"185'883","reported_seq":900,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.357504+0000","last_change":"2024-02-12T16:56:36.447717+0000","last_active":"2024-02-12T16:56:58.357504+0000","last_peered":"2024-02-12T16:56:58.357504+0000","last_clean":"2024-02-12T16:56:58.357504+0000","last_became_active":"2024-02-12T16:56:36.447417+0000","last_became_peered":"2024-02-12T16:56:36.447417+0000","last_unstale":"2024-02-12T16:56:58.357504+0000","last_undegraded":"2024-02-12T16:56:58.357504+0000","last_fullsized":"2024-02-12T16:56:58.357504+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":883,"ondisk_log_size":883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57868288,"num_objects":883,"num_object_clones":0,"num_object_copies":2649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":883,"num_write_kb":56512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":149,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.118649+0000","last_change":"2024-02-12T16:54:40.628021+0000","last_active":"2024-02-12T16:56:51.118649+0000","last_peered":"2024-02-12T16:56:51.118649+0000","last_clean":"2024-02-12T16:56:51.118649+0000","last_became_active":"2024-02-12T16:54:40.627688+0000","last_became_peered":"2024-02-12T16:54:40.627688+0000","last_unstale":"2024-02-12T16:56:51.118649+0000","last_undegraded":"2024-02-12T16:56:51.118649+0000","last_fullsized":"2024-02-12T16:56:51.118649+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":64,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117799+0000","last_change":"2024-02-12T16:56:45.218102+0000","last_active":"2024-02-12T16:56:51.117799+0000","last_peered":"2024-02-12T16:56:51.117799+0000","last_clean":"2024-02-12T16:56:51.117799+0000","last_became_active":"2024-02-12T16:54:40.628365+0000","last_became_peered":"2024-02-12T16:54:40.628365+0000","last_unstale":"2024-02-12T16:56:51.117799+0000","last_undegraded":"2024-02-12T16:56:51.117799+0000","last_fullsized":"2024-02-12T16:56:51.117799+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":146,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117909+0000","last_change":"2024-02-12T16:52:57.985677+0000","last_active":"2024-02-12T16:56:51.117909+0000","last_peered":"2024-02-12T16:56:51.117909+0000","last_clean":"2024-02-12T16:56:51.117909+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:56:51.117909+0000","last_undegraded":"2024-02-12T16:56:51.117909+0000","last_fullsized":"2024-02-12T16:56:51.117909+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:57.985584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:57.985584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":152,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230740+0000","last_change":"2024-02-12T16:54:30.770032+0000","last_active":"2024-02-12T16:56:51.230740+0000","last_peered":"2024-02-12T16:56:51.230740+0000","last_clean":"2024-02-12T16:56:51.230740+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:56:51.230740+0000","last_undegraded":"2024-02-12T16:56:51.230740+0000","last_fullsized":"2024-02-12T16:56:51.230740+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":170,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192250+0000","last_change":"2024-02-12T16:53:58.740196+0000","last_active":"2024-02-12T16:56:51.192250+0000","last_peered":"2024-02-12T16:56:51.192250+0000","last_clean":"2024-02-12T16:56:51.192250+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:56:51.192250+0000","last_undegraded":"2024-02-12T16:56:51.192250+0000","last_fullsized":"2024-02-12T16:56:51.192250+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:53:58.740122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.740122+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":140,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231329+0000","last_change":"2024-02-12T16:52:12.200439+0000","last_active":"2024-02-12T16:56:51.231329+0000","last_peered":"2024-02-12T16:56:51.231329+0000","last_clean":"2024-02-12T16:56:51.231329+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:56:51.231329+0000","last_undegraded":"2024-02-12T16:56:51.231329+0000","last_fullsized":"2024-02-12T16:56:51.231329+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:12.200355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:12.200355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"184'9","reported_seq":25,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:53.500167+0000","last_change":"2024-02-12T16:56:36.447068+0000","last_active":"2024-02-12T16:56:53.500167+0000","last_peered":"2024-02-12T16:56:53.500167+0000","last_clean":"2024-02-12T16:56:53.500167+0000","last_became_active":"2024-02-12T16:56:36.446744+0000","last_became_peered":"2024-02-12T16:56:36.446744+0000","last_unstale":"2024-02-12T16:56:53.500167+0000","last_undegraded":"2024-02-12T16:56:53.500167+0000","last_fullsized":"2024-02-12T16:56:53.500167+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":148,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503429+0000","last_change":"2024-02-12T16:54:40.628087+0000","last_active":"2024-02-12T16:56:50.503429+0000","last_peered":"2024-02-12T16:56:50.503429+0000","last_clean":"2024-02-12T16:56:50.503429+0000","last_became_active":"2024-02-12T16:54:40.626511+0000","last_became_peered":"2024-02-12T16:54:40.626511+0000","last_unstale":"2024-02-12T16:56:50.503429+0000","last_undegraded":"2024-02-12T16:56:50.503429+0000","last_fullsized":"2024-02-12T16:56:50.503429+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.780846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.780846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":152,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.720958+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T16:56:50.720958+0000","last_peered":"2024-02-12T16:56:50.720958+0000","last_clean":"2024-02-12T16:56:50.720958+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T16:56:50.720958+0000","last_undegraded":"2024-02-12T16:56:50.720958+0000","last_fullsized":"2024-02-12T16:56:50.720958+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":150,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229478+0000","last_change":"2024-02-12T16:52:58.167642+0000","last_active":"2024-02-12T16:56:51.229478+0000","last_peered":"2024-02-12T16:56:51.229478+0000","last_clean":"2024-02-12T16:56:51.229478+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:56:51.229478+0000","last_undegraded":"2024-02-12T16:56:51.229478+0000","last_fullsized":"2024-02-12T16:56:51.229478+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:52:58.167543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.167543+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":159,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721711+0000","last_change":"2024-02-12T16:53:43.229740+0000","last_active":"2024-02-12T16:56:50.721711+0000","last_peered":"2024-02-12T16:56:50.721711+0000","last_clean":"2024-02-12T16:56:50.721711+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:56:50.721711+0000","last_undegraded":"2024-02-12T16:56:50.721711+0000","last_fullsized":"2024-02-12T16:56:50.721711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:43.229622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.229622+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":142,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.118743+0000","last_change":"2024-02-12T16:56:18.162272+0000","last_active":"2024-02-12T16:56:51.118743+0000","last_peered":"2024-02-12T16:56:51.118743+0000","last_clean":"2024-02-12T16:56:51.118743+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:56:51.118743+0000","last_undegraded":"2024-02-12T16:56:51.118743+0000","last_fullsized":"2024-02-12T16:56:51.118743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"185'913","reported_seq":930,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.424440+0000","last_change":"2024-02-12T16:56:36.443457+0000","last_active":"2024-02-12T16:56:58.424440+0000","last_peered":"2024-02-12T16:56:58.424440+0000","last_clean":"2024-02-12T16:56:58.424440+0000","last_became_active":"2024-02-12T16:56:36.443153+0000","last_became_peered":"2024-02-12T16:56:36.443153+0000","last_unstale":"2024-02-12T16:56:58.424440+0000","last_undegraded":"2024-02-12T16:56:58.424440+0000","last_fullsized":"2024-02-12T16:56:58.424440+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":913,"ondisk_log_size":913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59834368,"num_objects":913,"num_object_clones":0,"num_object_copies":2739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":913,"num_write_kb":58432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":144,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503857+0000","last_change":"2024-02-12T16:56:41.053659+0000","last_active":"2024-02-12T16:56:50.503857+0000","last_peered":"2024-02-12T16:56:50.503857+0000","last_clean":"2024-02-12T16:56:50.503857+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:56:50.503857+0000","last_undegraded":"2024-02-12T16:56:50.503857+0000","last_fullsized":"2024-02-12T16:56:50.503857+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.053587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.053587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":149,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231578+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T16:56:51.231578+0000","last_peered":"2024-02-12T16:56:51.231578+0000","last_clean":"2024-02-12T16:56:51.231578+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:56:51.231578+0000","last_undegraded":"2024-02-12T16:56:51.231578+0000","last_fullsized":"2024-02-12T16:56:51.231578+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":182,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721254+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T16:56:50.721254+0000","last_peered":"2024-02-12T16:56:50.721254+0000","last_clean":"2024-02-12T16:56:50.721254+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:56:50.721254+0000","last_undegraded":"2024-02-12T16:56:50.721254+0000","last_fullsized":"2024-02-12T16:56:50.721254+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":146,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.191964+0000","last_change":"2024-02-12T16:53:50.681255+0000","last_active":"2024-02-12T16:56:51.191964+0000","last_peered":"2024-02-12T16:56:51.191964+0000","last_clean":"2024-02-12T16:56:51.191964+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:56:51.191964+0000","last_undegraded":"2024-02-12T16:56:51.191964+0000","last_fullsized":"2024-02-12T16:56:51.191964+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:50.681142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:50.681142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"184'7","reported_seq":23,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:53.497342+0000","last_change":"2024-02-12T16:56:36.789146+0000","last_active":"2024-02-12T16:56:53.497342+0000","last_peered":"2024-02-12T16:56:53.497342+0000","last_clean":"2024-02-12T16:56:53.497342+0000","last_became_active":"2024-02-12T16:56:36.788871+0000","last_became_peered":"2024-02-12T16:56:36.788871+0000","last_unstale":"2024-02-12T16:56:53.497342+0000","last_undegraded":"2024-02-12T16:56:53.497342+0000","last_fullsized":"2024-02-12T16:56:53.497342+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":62,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230318+0000","last_change":"2024-02-12T16:54:40.632821+0000","last_active":"2024-02-12T16:56:51.230318+0000","last_peered":"2024-02-12T16:56:51.230318+0000","last_clean":"2024-02-12T16:56:51.230318+0000","last_became_active":"2024-02-12T16:54:40.631423+0000","last_became_peered":"2024-02-12T16:54:40.631423+0000","last_unstale":"2024-02-12T16:56:51.230318+0000","last_undegraded":"2024-02-12T16:56:51.230318+0000","last_fullsized":"2024-02-12T16:56:51.230318+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.624865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.624865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.502613+0000","last_change":"2024-02-12T16:53:48.828698+0000","last_active":"2024-02-12T16:56:50.502613+0000","last_peered":"2024-02-12T16:56:50.502613+0000","last_clean":"2024-02-12T16:56:50.502613+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:56:50.502613+0000","last_undegraded":"2024-02-12T16:56:50.502613+0000","last_fullsized":"2024-02-12T16:56:50.502613+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":126,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231041+0000","last_change":"2024-02-12T16:56:26.889557+0000","last_active":"2024-02-12T16:56:51.231041+0000","last_peered":"2024-02-12T16:56:51.231041+0000","last_clean":"2024-02-12T16:56:51.231041+0000","last_became_active":"2024-02-12T16:54:40.634781+0000","last_became_peered":"2024-02-12T16:54:40.634781+0000","last_unstale":"2024-02-12T16:56:51.231041+0000","last_undegraded":"2024-02-12T16:56:51.231041+0000","last_fullsized":"2024-02-12T16:56:51.231041+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":157,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230348+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T16:56:51.230348+0000","last_peered":"2024-02-12T16:56:51.230348+0000","last_clean":"2024-02-12T16:56:51.230348+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T16:56:51.230348+0000","last_undegraded":"2024-02-12T16:56:51.230348+0000","last_fullsized":"2024-02-12T16:56:51.230348+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"185'931","reported_seq":948,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.309603+0000","last_change":"2024-02-12T16:56:36.788810+0000","last_active":"2024-02-12T16:56:58.309603+0000","last_peered":"2024-02-12T16:56:58.309603+0000","last_clean":"2024-02-12T16:56:58.309603+0000","last_became_active":"2024-02-12T16:56:36.788518+0000","last_became_peered":"2024-02-12T16:56:36.788518+0000","last_unstale":"2024-02-12T16:56:58.309603+0000","last_undegraded":"2024-02-12T16:56:58.309603+0000","last_fullsized":"2024-02-12T16:56:58.309603+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":931,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61014016,"num_objects":931,"num_object_clones":0,"num_object_copies":2793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":931,"num_write_kb":59584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":139,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.231452+0000","last_change":"2024-02-12T16:52:41.149470+0000","last_active":"2024-02-12T16:56:51.231452+0000","last_peered":"2024-02-12T16:56:51.231452+0000","last_clean":"2024-02-12T16:56:51.231452+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:56:51.231452+0000","last_undegraded":"2024-02-12T16:56:51.231452+0000","last_fullsized":"2024-02-12T16:56:51.231452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:52:41.149382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.149382+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":150,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192429+0000","last_change":"2024-02-12T16:56:26.337548+0000","last_active":"2024-02-12T16:56:51.192429+0000","last_peered":"2024-02-12T16:56:51.192429+0000","last_clean":"2024-02-12T16:56:51.192429+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:56:51.192429+0000","last_undegraded":"2024-02-12T16:56:51.192429+0000","last_fullsized":"2024-02-12T16:56:51.192429+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":137,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117583+0000","last_change":"2024-02-12T16:52:51.025242+0000","last_active":"2024-02-12T16:56:51.117583+0000","last_peered":"2024-02-12T16:56:51.117583+0000","last_clean":"2024-02-12T16:56:51.117583+0000","last_became_active":"2024-02-12T16:52:20.563744+0000","last_became_peered":"2024-02-12T16:52:20.563744+0000","last_unstale":"2024-02-12T16:56:51.117583+0000","last_undegraded":"2024-02-12T16:56:51.117583+0000","last_fullsized":"2024-02-12T16:56:51.117583+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:52:51.025152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.025152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"138'9","reported_seq":114,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117265+0000","last_change":"2024-02-12T16:55:28.449149+0000","last_active":"2024-02-12T16:56:51.117265+0000","last_peered":"2024-02-12T16:56:51.117265+0000","last_clean":"2024-02-12T16:56:51.117265+0000","last_became_active":"2024-02-12T16:54:40.626646+0000","last_became_peered":"2024-02-12T16:54:40.626646+0000","last_unstale":"2024-02-12T16:56:51.117265+0000","last_undegraded":"2024-02-12T16:56:51.117265+0000","last_fullsized":"2024-02-12T16:56:51.117265+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"185'926","reported_seq":943,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.320392+0000","last_change":"2024-02-12T16:56:36.445962+0000","last_active":"2024-02-12T16:56:58.320392+0000","last_peered":"2024-02-12T16:56:58.320392+0000","last_clean":"2024-02-12T16:56:58.320392+0000","last_became_active":"2024-02-12T16:56:36.445638+0000","last_became_peered":"2024-02-12T16:56:36.445638+0000","last_unstale":"2024-02-12T16:56:58.320392+0000","last_undegraded":"2024-02-12T16:56:58.320392+0000","last_fullsized":"2024-02-12T16:56:58.320392+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":926,"ondisk_log_size":926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60686336,"num_objects":926,"num_object_clones":0,"num_object_copies":2778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":926,"num_write_kb":59264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":59,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192086+0000","last_change":"2024-02-12T16:54:40.629725+0000","last_active":"2024-02-12T16:56:51.192086+0000","last_peered":"2024-02-12T16:56:51.192086+0000","last_clean":"2024-02-12T16:56:51.192086+0000","last_became_active":"2024-02-12T16:54:40.629348+0000","last_became_peered":"2024-02-12T16:54:40.629348+0000","last_unstale":"2024-02-12T16:56:51.192086+0000","last_undegraded":"2024-02-12T16:56:51.192086+0000","last_fullsized":"2024-02-12T16:56:51.192086+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":142,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.118519+0000","last_change":"2024-02-12T16:52:52.983089+0000","last_active":"2024-02-12T16:56:51.118519+0000","last_peered":"2024-02-12T16:56:51.118519+0000","last_clean":"2024-02-12T16:56:51.118519+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:56:51.118519+0000","last_undegraded":"2024-02-12T16:56:51.118519+0000","last_fullsized":"2024-02-12T16:56:51.118519+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:52.982996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.982996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":123,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.228912+0000","last_change":"2024-02-12T16:53:59.566836+0000","last_active":"2024-02-12T16:56:51.228912+0000","last_peered":"2024-02-12T16:56:51.228912+0000","last_clean":"2024-02-12T16:56:51.228912+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:56:51.228912+0000","last_undegraded":"2024-02-12T16:56:51.228912+0000","last_fullsized":"2024-02-12T16:56:51.228912+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.566730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":161,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192453+0000","last_change":"2024-02-12T16:56:04.931453+0000","last_active":"2024-02-12T16:56:51.192453+0000","last_peered":"2024-02-12T16:56:51.192453+0000","last_clean":"2024-02-12T16:56:51.192453+0000","last_became_active":"2024-02-12T16:56:04.863064+0000","last_became_peered":"2024-02-12T16:56:04.863064+0000","last_unstale":"2024-02-12T16:56:51.192453+0000","last_undegraded":"2024-02-12T16:56:51.192453+0000","last_fullsized":"2024-02-12T16:56:51.192453+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:54:36.753321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:36.753321+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"184'7","reported_seq":23,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:53.503517+0000","last_change":"2024-02-12T16:56:36.447192+0000","last_active":"2024-02-12T16:56:53.503517+0000","last_peered":"2024-02-12T16:56:53.503517+0000","last_clean":"2024-02-12T16:56:53.503517+0000","last_became_active":"2024-02-12T16:56:36.446926+0000","last_became_peered":"2024-02-12T16:56:36.446926+0000","last_unstale":"2024-02-12T16:56:53.503517+0000","last_undegraded":"2024-02-12T16:56:53.503517+0000","last_fullsized":"2024-02-12T16:56:53.503517+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":145,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.118560+0000","last_change":"2024-02-12T16:56:40.152368+0000","last_active":"2024-02-12T16:56:51.118560+0000","last_peered":"2024-02-12T16:56:51.118560+0000","last_clean":"2024-02-12T16:56:51.118560+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:56:51.118560+0000","last_undegraded":"2024-02-12T16:56:51.118560+0000","last_fullsized":"2024-02-12T16:56:51.118560+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.152265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"98'10","reported_seq":211,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:56.132778+0000","last_change":"2024-02-12T16:56:32.292792+0000","last_active":"2024-02-12T16:56:56.132778+0000","last_peered":"2024-02-12T16:56:56.132778+0000","last_clean":"2024-02-12T16:56:56.132778+0000","last_became_active":"2024-02-12T16:52:20.561534+0000","last_became_peered":"2024-02-12T16:52:20.561534+0000","last_unstale":"2024-02-12T16:56:56.132778+0000","last_undegraded":"2024-02-12T16:56:56.132778+0000","last_fullsized":"2024-02-12T16:56:56.132778+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":168,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503759+0000","last_change":"2024-02-12T16:56:04.857653+0000","last_active":"2024-02-12T16:56:50.503759+0000","last_peered":"2024-02-12T16:56:50.503759+0000","last_clean":"2024-02-12T16:56:50.503759+0000","last_became_active":"2024-02-12T16:56:04.857412+0000","last_became_peered":"2024-02-12T16:56:04.857412+0000","last_unstale":"2024-02-12T16:56:50.503759+0000","last_undegraded":"2024-02-12T16:56:50.503759+0000","last_fullsized":"2024-02-12T16:56:50.503759+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":205,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503996+0000","last_change":"2024-02-12T16:54:41.243992+0000","last_active":"2024-02-12T16:56:50.503996+0000","last_peered":"2024-02-12T16:56:50.503996+0000","last_clean":"2024-02-12T16:56:50.503996+0000","last_became_active":"2024-02-12T16:54:40.627029+0000","last_became_peered":"2024-02-12T16:54:40.627029+0000","last_unstale":"2024-02-12T16:56:50.503996+0000","last_undegraded":"2024-02-12T16:56:50.503996+0000","last_fullsized":"2024-02-12T16:56:50.503996+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"185'965","reported_seq":982,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.327443+0000","last_change":"2024-02-12T16:56:36.449103+0000","last_active":"2024-02-12T16:56:58.327443+0000","last_peered":"2024-02-12T16:56:58.327443+0000","last_clean":"2024-02-12T16:56:58.327443+0000","last_became_active":"2024-02-12T16:56:36.448827+0000","last_became_peered":"2024-02-12T16:56:36.448827+0000","last_unstale":"2024-02-12T16:56:58.327443+0000","last_undegraded":"2024-02-12T16:56:58.327443+0000","last_fullsized":"2024-02-12T16:56:58.327443+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":965,"ondisk_log_size":965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63242240,"num_objects":965,"num_object_clones":0,"num_object_copies":2895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":965,"num_write_kb":61760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":103,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.502347+0000","last_change":"2024-02-12T16:53:57.738352+0000","last_active":"2024-02-12T16:56:50.502347+0000","last_peered":"2024-02-12T16:56:50.502347+0000","last_clean":"2024-02-12T16:56:50.502347+0000","last_became_active":"2024-02-12T16:52:20.560085+0000","last_became_peered":"2024-02-12T16:52:20.560085+0000","last_unstale":"2024-02-12T16:56:50.502347+0000","last_undegraded":"2024-02-12T16:56:50.502347+0000","last_fullsized":"2024-02-12T16:56:50.502347+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:57.738249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.738249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":243,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:56.135381+0000","last_change":"2024-02-12T16:56:26.673555+0000","last_active":"2024-02-12T16:56:56.135381+0000","last_peered":"2024-02-12T16:56:56.135381+0000","last_clean":"2024-02-12T16:56:56.135381+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:56:56.135381+0000","last_undegraded":"2024-02-12T16:56:56.135381+0000","last_fullsized":"2024-02-12T16:56:56.135381+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":159,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.502904+0000","last_change":"2024-02-12T16:54:40.629536+0000","last_active":"2024-02-12T16:56:50.502904+0000","last_peered":"2024-02-12T16:56:50.502904+0000","last_clean":"2024-02-12T16:56:50.502904+0000","last_became_active":"2024-02-12T16:54:40.629024+0000","last_became_peered":"2024-02-12T16:54:40.629024+0000","last_unstale":"2024-02-12T16:56:50.502904+0000","last_undegraded":"2024-02-12T16:56:50.502904+0000","last_fullsized":"2024-02-12T16:56:50.502904+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":103,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.192449+0000","last_change":"2024-02-12T16:55:46.162945+0000","last_active":"2024-02-12T16:56:51.192449+0000","last_peered":"2024-02-12T16:56:51.192449+0000","last_clean":"2024-02-12T16:56:51.192449+0000","last_became_active":"2024-02-12T16:54:40.629780+0000","last_became_peered":"2024-02-12T16:54:40.629780+0000","last_unstale":"2024-02-12T16:56:51.192449+0000","last_undegraded":"2024-02-12T16:56:51.192449+0000","last_fullsized":"2024-02-12T16:56:51.192449+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"184'3","reported_seq":19,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:53.479083+0000","last_change":"2024-02-12T16:56:36.447081+0000","last_active":"2024-02-12T16:56:53.479083+0000","last_peered":"2024-02-12T16:56:53.479083+0000","last_clean":"2024-02-12T16:56:53.479083+0000","last_became_active":"2024-02-12T16:56:36.446786+0000","last_became_peered":"2024-02-12T16:56:36.446786+0000","last_unstale":"2024-02-12T16:56:53.479083+0000","last_undegraded":"2024-02-12T16:56:53.479083+0000","last_fullsized":"2024-02-12T16:56:53.479083+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":154,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230268+0000","last_change":"2024-02-12T16:54:40.631210+0000","last_active":"2024-02-12T16:56:51.230268+0000","last_peered":"2024-02-12T16:56:51.230268+0000","last_clean":"2024-02-12T16:56:51.230268+0000","last_became_active":"2024-02-12T16:54:40.630855+0000","last_became_peered":"2024-02-12T16:54:40.630855+0000","last_unstale":"2024-02-12T16:56:51.230268+0000","last_undegraded":"2024-02-12T16:56:51.230268+0000","last_fullsized":"2024-02-12T16:56:51.230268+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.387144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.387144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":146,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503325+0000","last_change":"2024-02-12T16:56:33.053023+0000","last_active":"2024-02-12T16:56:50.503325+0000","last_peered":"2024-02-12T16:56:50.503325+0000","last_clean":"2024-02-12T16:56:50.503325+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:56:50.503325+0000","last_undegraded":"2024-02-12T16:56:50.503325+0000","last_fullsized":"2024-02-12T16:56:50.503325+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":149,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229209+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T16:56:51.229209+0000","last_peered":"2024-02-12T16:56:51.229209+0000","last_clean":"2024-02-12T16:56:51.229209+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:56:51.229209+0000","last_undegraded":"2024-02-12T16:56:51.229209+0000","last_fullsized":"2024-02-12T16:56:51.229209+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":176,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229246+0000","last_change":"2024-02-12T16:56:30.714551+0000","last_active":"2024-02-12T16:56:51.229246+0000","last_peered":"2024-02-12T16:56:51.229246+0000","last_clean":"2024-02-12T16:56:51.229246+0000","last_became_active":"2024-02-12T16:56:04.869971+0000","last_became_peered":"2024-02-12T16:56:04.869971+0000","last_unstale":"2024-02-12T16:56:51.229246+0000","last_undegraded":"2024-02-12T16:56:51.229246+0000","last_fullsized":"2024-02-12T16:56:51.229246+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"184'7","reported_seq":23,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:53.501178+0000","last_change":"2024-02-12T16:56:36.788521+0000","last_active":"2024-02-12T16:56:53.501178+0000","last_peered":"2024-02-12T16:56:53.501178+0000","last_clean":"2024-02-12T16:56:53.501178+0000","last_became_active":"2024-02-12T16:56:36.788231+0000","last_became_peered":"2024-02-12T16:56:36.788231+0000","last_unstale":"2024-02-12T16:56:53.501178+0000","last_undegraded":"2024-02-12T16:56:53.501178+0000","last_fullsized":"2024-02-12T16:56:53.501178+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":147,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230549+0000","last_change":"2024-02-12T16:54:40.645951+0000","last_active":"2024-02-12T16:56:51.230549+0000","last_peered":"2024-02-12T16:56:51.230549+0000","last_clean":"2024-02-12T16:56:51.230549+0000","last_became_active":"2024-02-12T16:54:40.645556+0000","last_became_peered":"2024-02-12T16:54:40.645556+0000","last_unstale":"2024-02-12T16:56:51.230549+0000","last_undegraded":"2024-02-12T16:56:51.230549+0000","last_fullsized":"2024-02-12T16:56:51.230549+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.128703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.128703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":147,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229629+0000","last_change":"2024-02-12T16:53:58.524837+0000","last_active":"2024-02-12T16:56:51.229629+0000","last_peered":"2024-02-12T16:56:51.229629+0000","last_clean":"2024-02-12T16:56:51.229629+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:56:51.229629+0000","last_undegraded":"2024-02-12T16:56:51.229629+0000","last_fullsized":"2024-02-12T16:56:51.229629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.524736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":151,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.229096+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T16:56:51.229096+0000","last_peered":"2024-02-12T16:56:51.229096+0000","last_clean":"2024-02-12T16:56:51.229096+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:56:51.229096+0000","last_undegraded":"2024-02-12T16:56:51.229096+0000","last_fullsized":"2024-02-12T16:56:51.229096+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":213,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.191952+0000","last_change":"2024-02-12T16:56:41.290320+0000","last_active":"2024-02-12T16:56:51.191952+0000","last_peered":"2024-02-12T16:56:51.191952+0000","last_clean":"2024-02-12T16:56:51.191952+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:56:51.191952+0000","last_undegraded":"2024-02-12T16:56:51.191952+0000","last_fullsized":"2024-02-12T16:56:51.191952+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:41.290236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.290236+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"184'14","reported_seq":30,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:53.504099+0000","last_change":"2024-02-12T16:56:36.446663+0000","last_active":"2024-02-12T16:56:53.504099+0000","last_peered":"2024-02-12T16:56:53.504099+0000","last_clean":"2024-02-12T16:56:53.504099+0000","last_became_active":"2024-02-12T16:56:36.446299+0000","last_became_peered":"2024-02-12T16:56:36.446299+0000","last_unstale":"2024-02-12T16:56:53.504099+0000","last_undegraded":"2024-02-12T16:56:53.504099+0000","last_fullsized":"2024-02-12T16:56:53.504099+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":137,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721277+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T16:56:50.721277+0000","last_peered":"2024-02-12T16:56:50.721277+0000","last_clean":"2024-02-12T16:56:50.721277+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:56:50.721277+0000","last_undegraded":"2024-02-12T16:56:50.721277+0000","last_fullsized":"2024-02-12T16:56:50.721277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":151,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230023+0000","last_change":"2024-02-12T16:54:40.629432+0000","last_active":"2024-02-12T16:56:51.230023+0000","last_peered":"2024-02-12T16:56:51.230023+0000","last_clean":"2024-02-12T16:56:51.230023+0000","last_became_active":"2024-02-12T16:54:40.629081+0000","last_became_peered":"2024-02-12T16:54:40.629081+0000","last_unstale":"2024-02-12T16:56:51.230023+0000","last_undegraded":"2024-02-12T16:56:51.230023+0000","last_fullsized":"2024-02-12T16:56:51.230023+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.285961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.285961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":168,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.503575+0000","last_change":"2024-02-12T16:54:41.249353+0000","last_active":"2024-02-12T16:56:50.503575+0000","last_peered":"2024-02-12T16:56:50.503575+0000","last_clean":"2024-02-12T16:56:50.503575+0000","last_became_active":"2024-02-12T16:54:40.639823+0000","last_became_peered":"2024-02-12T16:54:40.639823+0000","last_unstale":"2024-02-12T16:56:50.503575+0000","last_undegraded":"2024-02-12T16:56:50.503575+0000","last_fullsized":"2024-02-12T16:56:50.503575+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":162,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-02-12T16:56:50.721408+0000","last_change":"2024-02-12T16:56:31.836846+0000","last_active":"2024-02-12T16:56:50.721408+0000","last_peered":"2024-02-12T16:56:50.721408+0000","last_clean":"2024-02-12T16:56:50.721408+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:56:50.721408+0000","last_undegraded":"2024-02-12T16:56:50.721408+0000","last_fullsized":"2024-02-12T16:56:50.721408+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"185'972","reported_seq":989,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:58.323393+0000","last_change":"2024-02-12T16:56:36.445585+0000","last_active":"2024-02-12T16:56:58.323393+0000","last_peered":"2024-02-12T16:56:58.323393+0000","last_clean":"2024-02-12T16:56:58.323393+0000","last_became_active":"2024-02-12T16:56:36.445291+0000","last_became_peered":"2024-02-12T16:56:36.445291+0000","last_unstale":"2024-02-12T16:56:58.323393+0000","last_undegraded":"2024-02-12T16:56:58.323393+0000","last_fullsized":"2024-02-12T16:56:58.323393+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":972,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63700992,"num_objects":972,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":972,"num_write_kb":62208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":140,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117697+0000","last_change":"2024-02-12T16:54:03.996224+0000","last_active":"2024-02-12T16:56:51.117697+0000","last_peered":"2024-02-12T16:56:51.117697+0000","last_clean":"2024-02-12T16:56:51.117697+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:56:51.117697+0000","last_undegraded":"2024-02-12T16:56:51.117697+0000","last_fullsized":"2024-02-12T16:56:51.117697+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":165,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.230652+0000","last_change":"2024-02-12T16:56:32.717219+0000","last_active":"2024-02-12T16:56:51.230652+0000","last_peered":"2024-02-12T16:56:51.230652+0000","last_clean":"2024-02-12T16:56:51.230652+0000","last_became_active":"2024-02-12T16:54:40.630768+0000","last_became_peered":"2024-02-12T16:54:40.630768+0000","last_unstale":"2024-02-12T16:56:51.230652+0000","last_undegraded":"2024-02-12T16:56:51.230652+0000","last_fullsized":"2024-02-12T16:56:51.230652+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"178'22","reported_seq":181,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-02-12T16:56:51.117439+0000","last_change":"2024-02-12T16:56:39.138852+0000","last_active":"2024-02-12T16:56:51.117439+0000","last_peered":"2024-02-12T16:56:51.117439+0000","last_clean":"2024-02-12T16:56:51.117439+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:56:51.117439+0000","last_undegraded":"2024-02-12T16:56:51.117439+0000","last_fullsized":"2024-02-12T16:56:51.117439+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'22","last_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.138758+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":236,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-02-12T16:56:56.133656+0000","last_change":"2024-02-12T16:52:59.185115+0000","last_active":"2024-02-12T16:56:56.133656+0000","last_peered":"2024-02-12T16:56:56.133656+0000","last_clean":"2024-02-12T16:56:56.133656+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:56:56.133656+0000","last_undegraded":"2024-02-12T16:56:56.133656+0000","last_fullsized":"2024-02-12T16:56:56.133656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":548798464,"num_objects":8374,"num_object_clones":0,"num_object_copies":25122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8374,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2092302336,"data_stored":2092302336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8374,"ondisk_log_size":8374,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":427,"num_read_kb":422,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":429,"ondisk_log_size":429,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234075,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":575800,"kb_used_data":278576,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93169352,"statfs":{"total":95995035648,"available":95405416448,"internally_reserved":0,"allocated":285261824,"data_stored":284795398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.788,"15min":0.91200000000000003},"min":{"1min":0.25600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":3.2469999999999999,"5min":36.439,"15min":36.439},"last":2.6549999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.6200000000000001,"15min":0.88300000000000001},"min":{"1min":0.28599999999999998,"5min":0.218,"15min":0.218},"max":{"1min":4.0910000000000002,"5min":36.490000000000002,"15min":36.490000000000002},"last":0.26600000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":1.069,"15min":0.71499999999999997},"min":{"1min":0.29799999999999999,"5min":0.221,"15min":0.221},"max":{"1min":2.2839999999999998,"5min":12.032,"15min":12.032},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.089,"15min":0.72399999999999998},"min":{"1min":0.38700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":4.1130000000000004,"5min":10.054,"15min":10.054},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.1599999999999999,"15min":0.70999999999999996},"min":{"1min":0.25,"5min":0.245,"15min":0.245},"max":{"1min":4.0759999999999996,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.292,"15min":0.77600000000000002},"min":{"1min":0.33600000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.6589999999999998,"5min":20.338000000000001,"15min":20.338000000000001},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.77,"15min":0.97499999999999998},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.8150000000000004,"5min":20.433,"15min":20.433},"last":0.443},{"interface":"front","average":{"1min":0.73299999999999998,"5min":1.77,"15min":0.96399999999999997},"min":{"1min":0.42699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.5979999999999999,"5min":20.393000000000001,"15min":20.393000000000001},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.65200000000000002,"15min":0.54300000000000004},"min":{"1min":0.29199999999999998,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":3.3479999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.54700000000000004},"min":{"1min":0.34799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.6869999999999998,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.41099999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.33100000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.3390000000000004,"5min":5.3390000000000004,"15min":5.3390000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.58999999999999997,"15min":0.55700000000000005},"min":{"1min":0.33700000000000002,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":0.65800000000000003,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.34999999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.66600000000000004,"15min":0.55900000000000005},"min":{"1min":0.36899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.581,"5min":2.6970000000000001,"15min":2.6970000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.746,"15min":0.625},"min":{"1min":0.34699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.1720000000000002,"5min":3.5920000000000001,"15min":3.5920000000000001},"last":0.46600000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528416,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":602820,"kb_used_data":293308,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93142332,"statfs":{"total":95995035648,"available":95377747968,"internally_reserved":0,"allocated":300347392,"data_stored":299868826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3279999999999998,"5min":1.522,"15min":0.82999999999999996},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":10.949,"5min":10.949,"15min":10.949},"last":0.80600000000000005},{"interface":"front","average":{"1min":2.794,"5min":1.5680000000000001,"15min":0.84399999999999997},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.6050000000000004,"5min":10.56,"15min":10.56},"last":0.24199999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.4550000000000001,"15min":0.83199999999999996},"min":{"1min":0.252,"5min":0.251,"15min":0.251},"max":{"1min":3.7000000000000002,"5min":15.987,"15min":15.987},"last":0.52900000000000003},{"interface":"front","average":{"1min":2.3319999999999999,"5min":1.5740000000000001,"15min":0.88800000000000001},"min":{"1min":0.34200000000000003,"5min":0.25,"15min":0.25},"max":{"1min":5.9770000000000003,"5min":15.997,"15min":15.997},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":1.357,"15min":0.78900000000000003},"min":{"1min":0.22900000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.63},{"interface":"front","average":{"1min":2.3090000000000002,"5min":1.333,"15min":0.78400000000000003},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.319,"5min":6.3680000000000003,"15min":6.3680000000000003},"last":0.86399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.3280000000000001,"15min":0.77500000000000002},"min":{"1min":0.29199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.6900000000000004,"5min":11.212,"15min":11.212},"last":2.6179999999999999},{"interface":"front","average":{"1min":2.1059999999999999,"5min":1.6719999999999999,"15min":0.90500000000000003},"min":{"1min":0.32900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":6.0380000000000003,"5min":10.875999999999999,"15min":10.875999999999999},"last":0.55000000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.61399999999999999,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.217},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.353,"5min":0.68300000000000005,"15min":0.495},"min":{"1min":0.26600000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"last":2.6619999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.624,"15min":0.48799999999999999},"min":{"1min":0.309,"5min":0.245,"15min":0.245},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.755,"15min":0.54100000000000004},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":3.5859999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.89000000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:56:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":0.83999999999999997,"15min":0.58499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.73799999999999999,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.29299999999999998,"15min":0.27900000000000003},"max":{"1min":2.7050000000000001,"5min":3.9049999999999998,"15min":3.9049999999999998},"last":0.83899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298176,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446976,"statfs":{"total":95995035648,"available":95689703424,"internally_reserved":0,"allocated":974848,"data_stored":494785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4660000000000002,"5min":2.6760000000000002,"15min":1.121},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":12.592000000000001,"5min":25.585000000000001,"15min":25.585000000000001},"last":0.313},{"interface":"front","average":{"1min":1.0029999999999999,"5min":2.5299999999999998,"15min":1.085},"min":{"1min":0.219,"5min":0.215,"15min":0.215},"max":{"1min":3.9390000000000001,"5min":26.798999999999999,"15min":26.798999999999999},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":2.234,"15min":0.996},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.22500000000000001},"max":{"1min":3.274,"5min":11.581,"15min":11.581},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":2.5499999999999998,"15min":1.113},"min":{"1min":0.29799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":3.1280000000000001,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:56:14 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.552,"15min":1.155},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.5220000000000002,"5min":26.649999999999999,"15min":26.649999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":3.21,"5min":2.2999999999999998,"15min":1.0409999999999999},"min":{"1min":0.314,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":17.821000000000002,"5min":17.821000000000002,"15min":17.821000000000002},"last":0.48999999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.6970000000000001,"15min":0.93999999999999995},"min":{"1min":0.34399999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.7030000000000003,"5min":8.8249999999999993,"15min":8.8249999999999993},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":2.1819999999999999,"15min":1.1259999999999999},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.8879999999999999,"5min":26.748999999999999,"15min":26.748999999999999},"last":0.69999999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":3.2000000000000002,"5min":1.7110000000000001,"15min":0.93999999999999995},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":8.7959999999999994,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":0.70799999999999996},{"interface":"front","average":{"1min":2.4009999999999998,"5min":2.2519999999999998,"15min":1.1499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.5960000000000001,"5min":26.722999999999999,"15min":26.722999999999999},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.9490000000000001,"15min":1.042},"min":{"1min":0.38600000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6509999999999998,"5min":13.834,"15min":13.834},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":2.1200000000000001,"15min":1.1299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.5550000000000002,"5min":26.774999999999999,"15min":26.774999999999999},"last":0.72199999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.6879999999999999,"15min":0.999},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":3.3079999999999998,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":2,"15min":1.0980000000000001},"min":{"1min":0.371,"5min":0.31,"15min":0.31},"max":{"1min":2.9710000000000001,"5min":8.8829999999999991,"15min":8.8829999999999991},"last":0.626}]}]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302252,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442900,"statfs":{"total":95995035648,"available":95685529600,"internally_reserved":0,"allocated":954368,"data_stored":493147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:56:24 2024","interfaces":[{"interface":"back","average":{"1min":1.895,"5min":2.0169999999999999,"15min":0.89500000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.1859999999999999,"5min":23.510999999999999,"15min":23.510999999999999},"last":0.191},{"interface":"front","average":{"1min":2.8679999999999999,"5min":2.008,"15min":0.88700000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":8.6500000000000004,"5min":10.186999999999999,"15min":10.186999999999999},"last":0.21099999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:55:53 2024","interfaces":[{"interface":"back","average":{"1min":3.714,"5min":2.3900000000000001,"15min":1.052},"min":{"1min":0.26000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":16.521000000000001,"5min":16.521000000000001,"15min":16.521000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":2.8290000000000002,"5min":1.9059999999999999,"15min":0.90300000000000002},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":9.9429999999999996,"5min":9.9429999999999996,"15min":9.9429999999999996},"last":0.28299999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:56:17 2024","interfaces":[{"interface":"back","average":{"1min":2.5329999999999999,"5min":2.5430000000000001,"15min":1.121},"min":{"1min":0.28799999999999998,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.3029999999999999,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.879,"5min":2.52,"15min":1.145},"min":{"1min":0.29799999999999999,"5min":0.28100000000000003,"15min":0.27600000000000002},"max":{"1min":7.2030000000000003,"5min":16.446000000000002,"15min":16.446000000000002},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.6919999999999999,"15min":0.94299999999999995},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.8330000000000002,"5min":16.027000000000001,"15min":16.027000000000001},"last":0.622},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.4399999999999999,"15min":0.86299999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.7619999999999996,"5min":10.141,"15min":10.141},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":2.46,"5min":1.6739999999999999,"15min":0.93700000000000006},"min":{"1min":0.32100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.9930000000000003,"5min":7.4340000000000002,"15min":7.4340000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.6479999999999999,"15min":0.94299999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.8799999999999999,"5min":16.138999999999999,"15min":16.138999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:56:24 2024","interfaces":[{"interface":"back","average":{"1min":2.4319999999999999,"5min":1.704,"15min":0.94899999999999995},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":12.135,"5min":12.135,"15min":12.135},"last":0.57199999999999995},{"interface":"front","average":{"1min":2.6829999999999998,"5min":1.982,"15min":1.0449999999999999},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.618,"5min":21.062000000000001,"15min":21.062000000000001},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4180000000000001,"5min":1.6419999999999999,"15min":0.93799999999999994},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":16.175000000000001,"5min":16.175000000000001,"15min":16.175000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.714,"5min":1.2909999999999999,"15min":0.82699999999999996},"min":{"1min":0.35499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":10.025,"5min":10.025,"15min":10.025},"last":0.54200000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":721204,"kb_used_data":415788,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93023948,"statfs":{"total":95995035648,"available":95256522752,"internally_reserved":0,"allocated":425766912,"data_stored":425305268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":2.6150000000000002,"15min":1.1259999999999999},"min":{"1min":0.26000000000000001,"5min":0.19,"15min":0.19},"max":{"1min":11.906000000000001,"5min":15.808,"15min":15.808},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.331,"5min":2.1000000000000001,"15min":0.96999999999999997},"min":{"1min":0.26400000000000001,"5min":0.158,"15min":0.158},"max":{"1min":13.497,"5min":19.893000000000001,"15min":19.893000000000001},"last":1.8029999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":2.6000000000000001,"15min":1.1519999999999999},"min":{"1min":0.27400000000000002,"5min":0.191,"15min":0.191},"max":{"1min":12.173,"5min":17.739000000000001,"15min":17.739000000000001},"last":3.085},{"interface":"front","average":{"1min":1.0800000000000001,"5min":2.75,"15min":1.1950000000000001},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":6.7130000000000001,"5min":15.766999999999999,"15min":15.766999999999999},"last":0.94199999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":3.2909999999999999,"15min":1.4079999999999999},"min":{"1min":0.35999999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":13.596,"5min":51.579999999999998,"15min":51.579999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":2.0649999999999999,"5min":2.4569999999999999,"15min":1.143},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":13.521000000000001,"5min":48.988999999999997,"15min":48.988999999999997},"last":0.37}]},{"osd":4,"last update":"Mon Feb 12 16:56:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.9039999999999999,"15min":1.048},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.3339999999999996,"5min":15.737,"15min":15.737},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.712,"5min":1.544,"15min":0.92100000000000004},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:56:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":2.214,"15min":1.2150000000000001},"min":{"1min":0.34799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.0410000000000004,"5min":15.699,"15min":15.699},"last":2.77},{"interface":"front","average":{"1min":1.3959999999999999,"5min":2.1579999999999999,"15min":1.115},"min":{"1min":0.46999999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.239000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":3.1179999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.0449999999999999,"5min":2.3599999999999999,"15min":1.1870000000000001},"min":{"1min":0.40600000000000003,"5min":0.31,"15min":0.31},"max":{"1min":12.215,"5min":17.782,"15min":17.782},"last":2.9710000000000001},{"interface":"front","average":{"1min":2.226,"5min":1.736,"15min":1.056},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":13.446999999999999,"5min":13.446999999999999,"15min":13.446999999999999},"last":1.837}]},{"osd":7,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.7889999999999999,"15min":1.222},"min":{"1min":0.42299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":1.9410000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":2.1429999999999998,"15min":1.1299999999999999},"min":{"1min":0.49299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":12.294,"5min":12.294,"15min":12.294},"last":3.149}]}]},{"osd":3,"up_from":0,"seq":111669149806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953578,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":669172,"kb_used_data":359596,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93075980,"statfs":{"total":95995035648,"available":95309803520,"internally_reserved":0,"allocated":368226304,"data_stored":367764217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.5760000000000001,"15min":0.88900000000000001},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.192},"max":{"1min":3.681,"5min":38.179000000000002,"15min":38.179000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.6399999999999999,"15min":0.88900000000000001},"min":{"1min":0.25800000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.9649999999999999,"5min":38.151000000000003,"15min":38.151000000000003},"last":0.38900000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.266,"15min":0.77100000000000002},"min":{"1min":0.28199999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.4829999999999997,"5min":7.673,"15min":7.673},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.272,"15min":0.78300000000000003},"min":{"1min":0.34100000000000003,"5min":0.249,"15min":0.249},"max":{"1min":5.6340000000000003,"5min":6.367,"15min":6.367},"last":0.85699999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.3160000000000001,"15min":0.81499999999999995},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.8769999999999998,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3540000000000001,"15min":0.82899999999999996},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.3330000000000002,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":2.2360000000000002,"15min":1.113},"min":{"1min":0.28999999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.7699999999999996,"5min":38.170999999999999,"15min":38.170999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.885,"15min":0.995},"min":{"1min":0.34300000000000003,"5min":0.32400000000000001,"15min":0.32000000000000001},"max":{"1min":4.5469999999999997,"5min":38.311999999999998,"15min":38.311999999999998},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.78700000000000003,"15min":0.497},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5009999999999999,"5min":5.4109999999999996,"15min":5.4109999999999996},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.68400000000000005,"15min":0.45800000000000002},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.0070000000000001,"5min":6.7960000000000003,"15min":6.7960000000000003},"last":1.1240000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.8009999999999999,"15min":0.84899999999999998},"min":{"1min":0.30399999999999999,"5min":0.255,"15min":0.255},"max":{"1min":2.4430000000000001,"5min":104.907,"15min":104.907},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.9650000000000001,"15min":0.90900000000000003},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.4560000000000004,"5min":105.395,"15min":105.395},"last":0.40500000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:56:42 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.67400000000000004,"15min":0.50700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":3.7559999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.77800000000000002,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.931,"5min":5.7329999999999997,"15min":5.7329999999999997},"last":0.55400000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757350,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":835376,"kb_used_data":529960,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92909776,"statfs":{"total":95995035648,"available":95139610624,"internally_reserved":0,"allocated":542679040,"data_stored":542220519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":1.516,"15min":0.85199999999999998},"min":{"1min":0.22500000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.64900000000000002,"5min":36.880000000000003,"15min":36.880000000000003},"last":1.119},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.782,"15min":0.92300000000000004},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2,"5min":36.954000000000001,"15min":36.954000000000001},"last":0.28899999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.6699999999999999,"15min":0.90000000000000002},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.962,"5min":1.849,"15min":0.95699999999999996},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":16.712,"5min":16.712,"15min":16.712},"last":0.318}]},{"osd":2,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2559999999999998,"5min":1.909,"15min":0.99099999999999999},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":16.940999999999999,"5min":18.218,"15min":18.218},"last":1.04},{"interface":"front","average":{"1min":2.2109999999999999,"5min":1.9319999999999999,"15min":0.97999999999999998},"min":{"1min":0.35399999999999998,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":17.103999999999999,"5min":18.327000000000002,"15min":18.327000000000002},"last":0.67500000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.825,"15min":0.96099999999999997},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.7189999999999999,"5min":18.251000000000001,"15min":18.251000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.714,"15min":0.91700000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.75,"5min":20.263000000000002,"15min":20.263000000000002},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.623,"15min":0.46800000000000003},"min":{"1min":0.23999999999999999,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":0.63500000000000001,"5min":4.3710000000000004,"15min":4.3710000000000004},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.64700000000000002,"15min":0.48199999999999998},"min":{"1min":0.30099999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.55,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":0.433}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.55700000000000005},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.65100000000000002,"15min":0.51100000000000001},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.8159999999999998,"5min":2.827,"15min":2.827},"last":0.41499999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.76100000000000001,"15min":0.57299999999999995},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.8079999999999998,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.73899999999999999,"15min":0.57499999999999996},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.98299999999999998,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3473408,"data_stored":3473408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3735552,"data_stored":3735552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":493027328,"data_stored":493027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":427819008,"data_stored":427819008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":551092224,"data_stored":551092224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":305004544,"data_stored":305004544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":308150272,"data_stored":308150272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:57:02.146 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:02.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:02.253 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:57:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:02.321 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:57:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:02.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:57:02.428 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:02.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 9 15 25037 25022 173.738 144.625 0.00886118 0.00574752 2024-02-12T16:57:02.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:01.215524+0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:02.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:01.215575+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:02.484 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:01.215524+0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:02.484 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:01.215575+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:02.486 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:02.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:01.215524+0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:02.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:01.215575+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:02.648 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:57:02.648 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:57:02.648 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:57:02.649 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:57:02.649 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:57:02.654 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:57:02.689 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:57:02.718 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:57:02.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:57:02.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:57:02.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:57:02.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:57:02.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:57:02.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:57:02.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:57:02.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:57:02.941 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:57:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T16:57:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T16:57:02.941 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T16:57:02.941 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:57:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:57:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:57:03.464 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 10 16 27153 27137 169.582 132.188 0.00812295 0.0058924 2024-02-12T16:57:03.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:02.192873+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-12T16:57:03.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:02.446281+0000 mon.b (mon.1) 62 : audit [DBG] from='client.? v1:172.21.15.33:0/1696243499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:03.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:02.473676+0000 mgr.y (mgr.24481) 284 : cluster [DBG] pgmap v332: 148 pgs: 148 active+clean; 942 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 156 MiB/s wr, 2.50k op/s 2024-02-12T16:57:03.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:02.192873+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-12T16:57:03.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:02.446281+0000 mon.b (mon.1) 62 : audit [DBG] from='client.? v1:172.21.15.33:0/1696243499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:03.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:02.473676+0000 mgr.y (mgr.24481) 284 : cluster [DBG] pgmap v332: 148 pgs: 148 active+clean; 942 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 156 MiB/s wr, 2.50k op/s 2024-02-12T16:57:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:02.192873+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-12T16:57:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:02.446281+0000 mon.b (mon.1) 62 : audit [DBG] from='client.? v1:172.21.15.33:0/1696243499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:02.473676+0000 mgr.y (mgr.24481) 284 : cluster [DBG] pgmap v332: 148 pgs: 148 active+clean; 942 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 156 MiB/s wr, 2.50k op/s 2024-02-12T16:57:03.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:57:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:57:04.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:03.712954+0000 mgr.y (mgr.24481) 285 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:04.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:03.712954+0000 mgr.y (mgr.24481) 285 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:04.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:04.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 11 16 29360 29344 166.705 137.938 0.0299669 0.00598402 2024-02-12T16:57:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:03.712954+0000 mgr.y (mgr.24481) 285 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:04.505 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:04.562 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:04.934 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:57:05.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:04.474440+0000 mgr.y (mgr.24481) 286 : cluster [DBG] pgmap v333: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 194 MiB/s wr, 3.11k op/s 2024-02-12T16:57:05.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:04.474440+0000 mgr.y (mgr.24481) 286 : cluster [DBG] pgmap v333: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 194 MiB/s wr, 3.11k op/s 2024-02-12T16:57:05.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:57:05.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 12 16 31292 31276 162.874 120.75 0.00269975 0.00613277 2024-02-12T16:57:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:04.474440+0000 mgr.y (mgr.24481) 286 : cluster [DBG] pgmap v333: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 194 MiB/s wr, 3.11k op/s 2024-02-12T16:57:06.341 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:57:06.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 13 16 33075 33059 158.917 111.438 0.0119658 0.00628759 2024-02-12T16:57:07.463 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 14 16 34914 34898 155.775 114.938 0.00632732 0.00641517 2024-02-12T16:57:07.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:06.126756+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-12T16:57:07.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:06.481742+0000 mgr.y (mgr.24481) 287 : cluster [DBG] pgmap v335: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 851 B/s rd, 125 MiB/s wr, 1.99k op/s 2024-02-12T16:57:07.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:06.126756+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-12T16:57:07.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:06.481742+0000 mgr.y (mgr.24481) 287 : cluster [DBG] pgmap v335: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 851 B/s rd, 125 MiB/s wr, 1.99k op/s 2024-02-12T16:57:07.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:06.126756+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-12T16:57:07.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:06.481742+0000 mgr.y (mgr.24481) 287 : cluster [DBG] pgmap v335: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 851 B/s rd, 125 MiB/s wr, 1.99k op/s 2024-02-12T16:57:08.464 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 15 16 36711 36695 152.875 112.312 0.0081842 0.00653656 2024-02-12T16:57:09.464 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 16 16 38467 38451 150.18 109.75 0.00406898 0.00665333 2024-02-12T16:57:09.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:08.482442+0000 mgr.y (mgr.24481) 288 : cluster [DBG] pgmap v336: 148 pgs: 148 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 696 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-12T16:57:10.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:08.482442+0000 mgr.y (mgr.24481) 288 : cluster [DBG] pgmap v336: 148 pgs: 148 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 696 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-12T16:57:10.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:08.482442+0000 mgr.y (mgr.24481) 288 : cluster [DBG] pgmap v336: 148 pgs: 148 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 696 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-12T16:57:10.464 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 17 16 40183 40167 147.654 107.25 0.00434479 0.00676803 2024-02-12T16:57:10.506 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:10.512 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:57:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:10.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:10.603 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:10.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:57:10.618 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:10.635 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:10.651 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:10.677 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:57:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:10.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:57:10.781 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:10.825 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:10.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:10.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:11.014 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:11.043 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:11.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:11.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:11.197 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:11.225 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:11.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:11.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:57:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:10] "GET /metrics HTTP/1.1" 200 175783 "" "Prometheus/2.33.4" 2024-02-12T16:57:11.331 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:11.380 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:11.464 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 18 16 41710 41694 144.753 95.4375 0.0120542 0.00690286 2024-02-12T16:57:11.811 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:10.482997+0000 mgr.y (mgr.24481) 289 : cluster [DBG] pgmap v337: 148 pgs: 148 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 162 MiB/s wr, 2.60k op/s 2024-02-12T16:57:11.811 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:10.482997+0000 mgr.y (mgr.24481) 289 : cluster [DBG] pgmap v337: 148 pgs: 148 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 162 MiB/s wr, 2.60k op/s 2024-02-12T16:57:11.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:10.482997+0000 mgr.y (mgr.24481) 289 : cluster [DBG] pgmap v337: 148 pgs: 148 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 162 MiB/s wr, 2.60k op/s 2024-02-12T16:57:12.236 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:12.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:12.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:57:12.397 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:12.464 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 19 16 43413 43397 142.736 106.438 0.0159972 0.00700045 2024-02-12T16:57:12.992 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:57:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:57:13.467 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T16:57:13.465834+0000 min lat: 0.00131853 max lat: 0.0747942 avg lat: 0.007084 2024-02-12T16:57:13.467 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:57:13.467 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 20 16 45169 45153 141.07 109.75 0.00696617 0.007084 2024-02-12T16:57:13.669 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:13.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:13.776 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:13.822 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:12.487117+0000 mgr.y (mgr.24481) 290 : cluster [DBG] pgmap v338: 148 pgs: 148 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 920 B/s rd, 134 MiB/s wr, 2.15k op/s 2024-02-12T16:57:13.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:12.487117+0000 mgr.y (mgr.24481) 290 : cluster [DBG] pgmap v338: 148 pgs: 148 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 920 B/s rd, 134 MiB/s wr, 2.15k op/s 2024-02-12T16:57:13.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:57:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:57:13.824 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:12.487117+0000 mgr.y (mgr.24481) 290 : cluster [DBG] pgmap v338: 148 pgs: 148 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 920 B/s rd, 134 MiB/s wr, 2.15k op/s 2024-02-12T16:57:13.831 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:13.868 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:13.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:13.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:14.025 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:14.397 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:14.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:14.466 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 21 16 46783 46767 139.156 100.875 0.007102 0.00718126 2024-02-12T16:57:14.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:14.552 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:14.587 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:14.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:14.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:14.754 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:13.726263+0000 mgr.y (mgr.24481) 291 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:14.754 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:13.726263+0000 mgr.y (mgr.24481) 291 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:14.763 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:14.852 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:14.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:14.959 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:13.726263+0000 mgr.y (mgr.24481) 291 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:15.026 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:57:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:57:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:57:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:57:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:57:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:57:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:57:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:57:15.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:57:15.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:57:15.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:57:15.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:57:15.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:57:15.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:57:15.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:57:15.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:57:15.466 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 22 16 48297 48281 137.132 94.625 0.00581877 0.00728745 2024-02-12T16:57:15.632 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:15.988 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:15.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:14.489135+0000 mgr.y (mgr.24481) 292 : cluster [DBG] pgmap v339: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 920 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-02-12T16:57:15.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:14.789816+0000 mon.b (mon.1) 63 : audit [DBG] from='client.? v1:172.21.15.33:0/3734465138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:15.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:15.997 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:14.489135+0000 mgr.y (mgr.24481) 292 : cluster [DBG] pgmap v339: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 920 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-02-12T16:57:15.997 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:14.789816+0000 mon.b (mon.1) 63 : audit [DBG] from='client.? v1:172.21.15.33:0/3734465138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:15.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:14.489135+0000 mgr.y (mgr.24481) 292 : cluster [DBG] pgmap v339: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 920 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-02-12T16:57:15.998 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:14.789816+0000 mon.b (mon.1) 63 : audit [DBG] from='client.? v1:172.21.15.33:0/3734465138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:16.098 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:57:16.151 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:16.467 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 23 16 49654 49638 134.857 84.8125 0.00952327 0.00740994 2024-02-12T16:57:17.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:16.238090+0000 mon.a (mon.0) 986 : audit [INF] from='client.? v1:172.21.15.33:0/4054667840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:57:17.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:16.489502+0000 mgr.y (mgr.24481) 293 : cluster [DBG] pgmap v340: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 889 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-12T16:57:17.175 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:57:17.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 24 15 51255 51240 133.398 100.125 0.0115441 0.00749161 2024-02-12T16:57:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:16.238090+0000 mon.a (mon.0) 986 : audit [INF] from='client.? v1:172.21.15.33:0/4054667840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:57:17.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:16.489502+0000 mgr.y (mgr.24481) 293 : cluster [DBG] pgmap v340: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 889 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-12T16:57:17.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:16.238090+0000 mon.a (mon.0) 986 : audit [INF] from='client.? v1:172.21.15.33:0/4054667840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:57:17.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:16.489502+0000 mgr.y (mgr.24481) 293 : cluster [DBG] pgmap v340: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 889 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-12T16:57:18.061 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:18.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:18.166 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:18.214 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:17.148724+0000 mon.a (mon.0) 987 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:17.163080+0000 mon.a (mon.0) 988 : audit [INF] from='client.? v1:172.21.15.33:0/4054667840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:17.163128+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:17.279940+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:17.283570+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:17.148724+0000 mon.a (mon.0) 987 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:17.163080+0000 mon.a (mon.0) 988 : audit [INF] from='client.? v1:172.21.15.33:0/4054667840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:17.163128+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-12T16:57:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:17.279940+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:57:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:17.283570+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:57:18.298 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:18.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:18.404 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:18.448 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:17.148724+0000 mon.a (mon.0) 987 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:18.448 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:17.163080+0000 mon.a (mon.0) 988 : audit [INF] from='client.? v1:172.21.15.33:0/4054667840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:57:18.448 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:17.163128+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-12T16:57:18.448 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:17.279940+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:57:18.448 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:17.283570+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:57:18.451 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:18.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 25 16 52769 52753 131.843 94.5625 0.0140337 0.0075803 2024-02-12T16:57:18.477 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:18.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:18.583 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:18.630 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:18.659 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:18.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:18.765 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:18.817 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:18.874 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:57:19.169 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:57:19.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:17.688268+0000 osd.2 (osd.2) 117 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:57:19.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:17.691000+0000 osd.2 (osd.2) 118 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:57:19.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:17.939496+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:57:19.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:17.944939+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:57:19.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:18.049605+0000 osd.5 (osd.5) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:19.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:18.049650+0000 osd.5 (osd.5) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:19.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:18.171583+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-12T16:57:19.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:18.490251+0000 mgr.y (mgr.24481) 294 : cluster [DBG] pgmap v343: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-12T16:57:19.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:18.860845+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:57:19.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:18.865799+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:57:19.200 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:57:19.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 26 16 54124 54108 130.03 84.6875 0.0112508 0.00768539 2024-02-12T16:57:19.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:17.688268+0000 osd.2 (osd.2) 117 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:17.691000+0000 osd.2 (osd.2) 118 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:17.939496+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:17.944939+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:18.049605+0000 osd.5 (osd.5) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:18.049650+0000 osd.5 (osd.5) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:18.171583+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:18.490251+0000 mgr.y (mgr.24481) 294 : cluster [DBG] pgmap v343: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:18.860845+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:57:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:18.865799+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:57:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:17.688268+0000 osd.2 (osd.2) 117 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:57:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:17.691000+0000 osd.2 (osd.2) 118 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:57:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:17.939496+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:57:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:17.944939+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:57:19.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:18.049605+0000 osd.5 (osd.5) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:19.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:18.049650+0000 osd.5 (osd.5) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:19.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:18.171583+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-12T16:57:19.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:18.490251+0000 mgr.y (mgr.24481) 294 : cluster [DBG] pgmap v343: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-12T16:57:19.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:18.860845+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.1a scrub starts 2024-02-12T16:57:19.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:18.865799+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.1a scrub ok 2024-02-12T16:57:19.822 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:19.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:19.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:19.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:19.988 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:20.022 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:20.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:20.130 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:20.180 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:20.187 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:18.672157+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:57:20.187 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:18.681480+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:57:20.187 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:19.140743+0000 mgr.y (mgr.24481) 295 : audit [DBG] from='client.35072 v1:172.21.15.33:0/4019371871' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:20.187 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:19.278308+0000 osd.6 (osd.6) 116 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:57:20.187 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:19.280888+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:57:20.213 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:20.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:20.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:18.672157+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:57:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:18.681480+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:57:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:19.140743+0000 mgr.y (mgr.24481) 295 : audit [DBG] from='client.35072 v1:172.21.15.33:0/4019371871' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:19.278308+0000 osd.6 (osd.6) 116 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:57:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:19.280888+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:57:20.320 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:20.376 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:20.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 27 16 55570 55554 128.561 90.375 0.013274 0.00777352 2024-02-12T16:57:20.500 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:57:20.500 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:57:20.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:18.672157+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:57:20.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:18.681480+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:57:20.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:19.140743+0000 mgr.y (mgr.24481) 295 : audit [DBG] from='client.35072 v1:172.21.15.33:0/4019371871' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:20.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:19.278308+0000 osd.6 (osd.6) 116 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:57:20.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:19.280888+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:57:20.745 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:20.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:20.790 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-02-12T16:57:18.489724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3235190436,"num_objects":49557,"num_object_clones":0,"num_object_copies":148671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49557,"num_whiteouts":0,"num_read":1153,"num_read_kb":905,"num_write":49793,"num_write_kb":3159401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":3321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49877,"ondisk_log_size":49877,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11654256,"kb_used_data":9536824,"kb_used_omap":0,"kb_used_meta":2117376,"kb_avail":644561808,"statfs":{"total":671965249536,"available":660031291392,"internally_reserved":0,"allocated":9765707776,"data_stored":9762244852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2168193024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,5,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":835911680,"num_objects":12755,"num_object_clones":0,"num_object_copies":38265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12755,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12755,"num_write_kb":816320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007091"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":70,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.177464+0000","last_change":"2024-02-12T16:56:31.966054+0000","last_active":"2024-02-12T16:57:17.177464+0000","last_peered":"2024-02-12T16:57:17.177464+0000","last_clean":"2024-02-12T16:57:17.177464+0000","last_became_active":"2024-02-12T16:54:40.643598+0000","last_became_peered":"2024-02-12T16:54:40.643598+0000","last_unstale":"2024-02-12T16:57:17.177464+0000","last_undegraded":"2024-02-12T16:57:17.177464+0000","last_fullsized":"2024-02-12T16:57:17.177464+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":120,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.170157+0000","last_change":"2024-02-12T16:56:43.270830+0000","last_active":"2024-02-12T16:57:17.170157+0000","last_peered":"2024-02-12T16:57:17.170157+0000","last_clean":"2024-02-12T16:57:17.170157+0000","last_became_active":"2024-02-12T16:52:20.563961+0000","last_became_peered":"2024-02-12T16:52:20.563961+0000","last_unstale":"2024-02-12T16:57:17.170157+0000","last_undegraded":"2024-02-12T16:57:17.170157+0000","last_fullsized":"2024-02-12T16:57:17.170157+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:43.270756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.270756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":153,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.199910+0000","last_change":"2024-02-12T16:56:35.853405+0000","last_active":"2024-02-12T16:57:02.199910+0000","last_peered":"2024-02-12T16:57:02.199910+0000","last_clean":"2024-02-12T16:57:02.199910+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:57:02.199910+0000","last_undegraded":"2024-02-12T16:57:02.199910+0000","last_fullsized":"2024-02-12T16:57:02.199910+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.853330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.853330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":123,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169165+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T16:57:17.169165+0000","last_peered":"2024-02-12T16:57:17.169165+0000","last_clean":"2024-02-12T16:57:17.169165+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T16:57:17.169165+0000","last_undegraded":"2024-02-12T16:57:17.169165+0000","last_fullsized":"2024-02-12T16:57:17.169165+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":110,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.171959+0000","last_change":"2024-02-12T16:53:44.946159+0000","last_active":"2024-02-12T16:57:17.171959+0000","last_peered":"2024-02-12T16:57:17.171959+0000","last_clean":"2024-02-12T16:57:17.171959+0000","last_became_active":"2024-02-12T16:52:20.561617+0000","last_became_peered":"2024-02-12T16:52:20.561617+0000","last_unstale":"2024-02-12T16:57:17.171959+0000","last_undegraded":"2024-02-12T16:57:17.171959+0000","last_fullsized":"2024-02-12T16:57:17.171959+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":151,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.197437+0000","last_change":"2024-02-12T16:56:30.146707+0000","last_active":"2024-02-12T16:57:02.197437+0000","last_peered":"2024-02-12T16:57:02.197437+0000","last_clean":"2024-02-12T16:57:02.197437+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:57:02.197437+0000","last_undegraded":"2024-02-12T16:57:02.197437+0000","last_fullsized":"2024-02-12T16:57:02.197437+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:30.146643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.146643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":68,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.200191+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T16:57:02.200191+0000","last_peered":"2024-02-12T16:57:02.200191+0000","last_clean":"2024-02-12T16:57:02.200191+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T16:57:02.200191+0000","last_undegraded":"2024-02-12T16:57:02.200191+0000","last_fullsized":"2024-02-12T16:57:02.200191+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":172,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.161907+0000","last_change":"2024-02-12T16:55:33.609901+0000","last_active":"2024-02-12T16:57:17.161907+0000","last_peered":"2024-02-12T16:57:17.161907+0000","last_clean":"2024-02-12T16:57:17.161907+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:57:17.161907+0000","last_undegraded":"2024-02-12T16:57:17.161907+0000","last_fullsized":"2024-02-12T16:57:17.161907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:55:33.609834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:33.609834+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":160,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172830+0000","last_change":"2024-02-12T16:56:42.175069+0000","last_active":"2024-02-12T16:57:17.172830+0000","last_peered":"2024-02-12T16:57:17.172830+0000","last_clean":"2024-02-12T16:57:17.172830+0000","last_became_active":"2024-02-12T16:54:40.628037+0000","last_became_peered":"2024-02-12T16:54:40.628037+0000","last_unstale":"2024-02-12T16:57:17.172830+0000","last_undegraded":"2024-02-12T16:57:17.172830+0000","last_fullsized":"2024-02-12T16:57:17.172830+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:42.174992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:42.174992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":151,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.177222+0000","last_change":"2024-02-12T16:56:41.972274+0000","last_active":"2024-02-12T16:57:17.177222+0000","last_peered":"2024-02-12T16:57:17.177222+0000","last_clean":"2024-02-12T16:57:17.177222+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:57:17.177222+0000","last_undegraded":"2024-02-12T16:57:17.177222+0000","last_fullsized":"2024-02-12T16:57:17.177222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.972202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.972202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":163,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178646+0000","last_change":"2024-02-12T16:54:38.784259+0000","last_active":"2024-02-12T16:57:17.178646+0000","last_peered":"2024-02-12T16:57:17.178646+0000","last_clean":"2024-02-12T16:57:17.178646+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:57:17.178646+0000","last_undegraded":"2024-02-12T16:57:17.178646+0000","last_fullsized":"2024-02-12T16:57:17.178646+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":91,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194503+0000","last_change":"2024-02-12T16:54:41.193544+0000","last_active":"2024-02-12T16:57:02.194503+0000","last_peered":"2024-02-12T16:57:02.194503+0000","last_clean":"2024-02-12T16:57:02.194503+0000","last_became_active":"2024-02-12T16:54:40.639106+0000","last_became_peered":"2024-02-12T16:54:40.639106+0000","last_unstale":"2024-02-12T16:57:02.194503+0000","last_undegraded":"2024-02-12T16:57:02.194503+0000","last_fullsized":"2024-02-12T16:57:02.194503+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:53:45.658292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.658292+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":144,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.200801+0000","last_change":"2024-02-12T16:53:51.204401+0000","last_active":"2024-02-12T16:57:02.200801+0000","last_peered":"2024-02-12T16:57:02.200801+0000","last_clean":"2024-02-12T16:57:02.200801+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:57:02.200801+0000","last_undegraded":"2024-02-12T16:57:02.200801+0000","last_fullsized":"2024-02-12T16:57:02.200801+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":153,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172695+0000","last_change":"2024-02-12T16:56:24.210155+0000","last_active":"2024-02-12T16:57:17.172695+0000","last_peered":"2024-02-12T16:57:17.172695+0000","last_clean":"2024-02-12T16:57:17.172695+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:57:17.172695+0000","last_undegraded":"2024-02-12T16:57:17.172695+0000","last_fullsized":"2024-02-12T16:57:17.172695+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:24.210075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.210075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":155,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178706+0000","last_change":"2024-02-12T16:56:35.897211+0000","last_active":"2024-02-12T16:57:17.178706+0000","last_peered":"2024-02-12T16:57:17.178706+0000","last_clean":"2024-02-12T16:57:17.178706+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:57:17.178706+0000","last_undegraded":"2024-02-12T16:57:17.178706+0000","last_fullsized":"2024-02-12T16:57:17.178706+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.897088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.897088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":101,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.197600+0000","last_change":"2024-02-12T16:54:41.013328+0000","last_active":"2024-02-12T16:57:02.197600+0000","last_peered":"2024-02-12T16:57:02.197600+0000","last_clean":"2024-02-12T16:57:02.197600+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T16:57:02.197600+0000","last_undegraded":"2024-02-12T16:57:02.197600+0000","last_fullsized":"2024-02-12T16:57:02.197600+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:52:48.029049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.029049+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":72,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.174736+0000","last_change":"2024-02-12T16:56:23.345846+0000","last_active":"2024-02-12T16:57:17.174736+0000","last_peered":"2024-02-12T16:57:17.174736+0000","last_clean":"2024-02-12T16:57:17.174736+0000","last_became_active":"2024-02-12T16:54:40.628201+0000","last_became_peered":"2024-02-12T16:54:40.628201+0000","last_unstale":"2024-02-12T16:57:17.174736+0000","last_undegraded":"2024-02-12T16:57:17.174736+0000","last_fullsized":"2024-02-12T16:57:17.174736+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":73,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169509+0000","last_change":"2024-02-12T16:56:40.672807+0000","last_active":"2024-02-12T16:57:17.169509+0000","last_peered":"2024-02-12T16:57:17.169509+0000","last_clean":"2024-02-12T16:57:17.169509+0000","last_became_active":"2024-02-12T16:54:40.628744+0000","last_became_peered":"2024-02-12T16:54:40.628744+0000","last_unstale":"2024-02-12T16:57:17.169509+0000","last_undegraded":"2024-02-12T16:57:17.169509+0000","last_fullsized":"2024-02-12T16:57:17.169509+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.672727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.672727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":65,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.177704+0000","last_change":"2024-02-12T16:54:40.633472+0000","last_active":"2024-02-12T16:57:17.177704+0000","last_peered":"2024-02-12T16:57:17.177704+0000","last_clean":"2024-02-12T16:57:17.177704+0000","last_became_active":"2024-02-12T16:54:40.633129+0000","last_became_peered":"2024-02-12T16:54:40.633129+0000","last_unstale":"2024-02-12T16:57:17.177704+0000","last_undegraded":"2024-02-12T16:57:17.177704+0000","last_fullsized":"2024-02-12T16:57:17.177704+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:03.947783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:03.947783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":115,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.195392+0000","last_change":"2024-02-12T16:56:45.033557+0000","last_active":"2024-02-12T16:57:02.195392+0000","last_peered":"2024-02-12T16:57:02.195392+0000","last_clean":"2024-02-12T16:57:02.195392+0000","last_became_active":"2024-02-12T16:54:40.626676+0000","last_became_peered":"2024-02-12T16:54:40.626676+0000","last_unstale":"2024-02-12T16:57:02.195392+0000","last_undegraded":"2024-02-12T16:57:02.195392+0000","last_fullsized":"2024-02-12T16:57:02.195392+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:56:45.033441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.033441+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":156,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.163100+0000","last_change":"2024-02-12T16:56:18.464298+0000","last_active":"2024-02-12T16:57:17.163100+0000","last_peered":"2024-02-12T16:57:17.163100+0000","last_clean":"2024-02-12T16:57:17.163100+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:57:17.163100+0000","last_undegraded":"2024-02-12T16:57:17.163100+0000","last_fullsized":"2024-02-12T16:57:17.163100+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":151,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.195144+0000","last_change":"2024-02-12T16:56:32.102101+0000","last_active":"2024-02-12T16:57:02.195144+0000","last_peered":"2024-02-12T16:57:02.195144+0000","last_clean":"2024-02-12T16:57:02.195144+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:57:02.195144+0000","last_undegraded":"2024-02-12T16:57:02.195144+0000","last_fullsized":"2024-02-12T16:57:02.195144+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.102022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.102022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":161,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.173161+0000","last_change":"2024-02-12T16:56:25.339812+0000","last_active":"2024-02-12T16:57:17.173161+0000","last_peered":"2024-02-12T16:57:17.173161+0000","last_clean":"2024-02-12T16:57:17.173161+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:57:17.173161+0000","last_undegraded":"2024-02-12T16:57:17.173161+0000","last_fullsized":"2024-02-12T16:57:17.173161+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":209,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169697+0000","last_change":"2024-02-12T16:54:40.681231+0000","last_active":"2024-02-12T16:57:17.169697+0000","last_peered":"2024-02-12T16:57:17.169697+0000","last_clean":"2024-02-12T16:57:17.169697+0000","last_became_active":"2024-02-12T16:54:40.630167+0000","last_became_peered":"2024-02-12T16:54:40.630167+0000","last_unstale":"2024-02-12T16:57:17.169697+0000","last_undegraded":"2024-02-12T16:57:17.169697+0000","last_fullsized":"2024-02-12T16:57:17.169697+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:53:48.468547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.468547+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":149,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194957+0000","last_change":"2024-02-12T16:56:31.122644+0000","last_active":"2024-02-12T16:57:02.194957+0000","last_peered":"2024-02-12T16:57:02.194957+0000","last_clean":"2024-02-12T16:57:02.194957+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:57:02.194957+0000","last_undegraded":"2024-02-12T16:57:02.194957+0000","last_fullsized":"2024-02-12T16:57:02.194957+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":144,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.201883+0000","last_change":"2024-02-12T16:53:04.085929+0000","last_active":"2024-02-12T16:57:02.201883+0000","last_peered":"2024-02-12T16:57:02.201883+0000","last_clean":"2024-02-12T16:57:02.201883+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:57:02.201883+0000","last_undegraded":"2024-02-12T16:57:02.201883+0000","last_fullsized":"2024-02-12T16:57:02.201883+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:04.085854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.085854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":70,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.173393+0000","last_change":"2024-02-12T16:56:33.292072+0000","last_active":"2024-02-12T16:57:17.173393+0000","last_peered":"2024-02-12T16:57:17.173393+0000","last_clean":"2024-02-12T16:57:17.173393+0000","last_became_active":"2024-02-12T16:54:40.640452+0000","last_became_peered":"2024-02-12T16:54:40.640452+0000","last_unstale":"2024-02-12T16:57:17.173393+0000","last_undegraded":"2024-02-12T16:57:17.173393+0000","last_fullsized":"2024-02-12T16:57:17.173393+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":173,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.174618+0000","last_change":"2024-02-12T16:56:27.293762+0000","last_active":"2024-02-12T16:57:17.174618+0000","last_peered":"2024-02-12T16:57:17.174618+0000","last_clean":"2024-02-12T16:57:17.174618+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:57:17.174618+0000","last_undegraded":"2024-02-12T16:57:17.174618+0000","last_fullsized":"2024-02-12T16:57:17.174618+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":146,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178877+0000","last_change":"2024-02-12T16:53:04.169987+0000","last_active":"2024-02-12T16:57:17.178877+0000","last_peered":"2024-02-12T16:57:17.178877+0000","last_clean":"2024-02-12T16:57:17.178877+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:57:17.178877+0000","last_undegraded":"2024-02-12T16:57:17.178877+0000","last_fullsized":"2024-02-12T16:57:17.178877+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:53:04.169892+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.169892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":255,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.332890+0000","last_change":"2024-02-12T16:56:36.211166+0000","last_active":"2024-02-12T16:57:17.332890+0000","last_peered":"2024-02-12T16:57:17.332890+0000","last_clean":"2024-02-12T16:57:17.332890+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:57:17.332890+0000","last_undegraded":"2024-02-12T16:57:17.332890+0000","last_fullsized":"2024-02-12T16:57:17.332890+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:36.211099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:36.211099+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":152,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172515+0000","last_change":"2024-02-12T16:52:47.997796+0000","last_active":"2024-02-12T16:57:17.172515+0000","last_peered":"2024-02-12T16:57:17.172515+0000","last_clean":"2024-02-12T16:57:17.172515+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:57:17.172515+0000","last_undegraded":"2024-02-12T16:57:17.172515+0000","last_fullsized":"2024-02-12T16:57:17.172515+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:47.997708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.997708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":187,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.200282+0000","last_change":"2024-02-12T16:56:32.825256+0000","last_active":"2024-02-12T16:57:02.200282+0000","last_peered":"2024-02-12T16:57:02.200282+0000","last_clean":"2024-02-12T16:57:02.200282+0000","last_became_active":"2024-02-12T16:54:40.650318+0000","last_became_peered":"2024-02-12T16:54:40.650318+0000","last_unstale":"2024-02-12T16:57:02.200282+0000","last_undegraded":"2024-02-12T16:57:02.200282+0000","last_fullsized":"2024-02-12T16:57:02.200282+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":63,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.197559+0000","last_change":"2024-02-12T16:54:40.650920+0000","last_active":"2024-02-12T16:57:02.197559+0000","last_peered":"2024-02-12T16:57:02.197559+0000","last_clean":"2024-02-12T16:57:02.197559+0000","last_became_active":"2024-02-12T16:54:40.650168+0000","last_became_peered":"2024-02-12T16:54:40.650168+0000","last_unstale":"2024-02-12T16:57:02.197559+0000","last_undegraded":"2024-02-12T16:57:02.197559+0000","last_fullsized":"2024-02-12T16:57:02.197559+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.665249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.665249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":146,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178880+0000","last_change":"2024-02-12T16:53:05.141324+0000","last_active":"2024-02-12T16:57:17.178880+0000","last_peered":"2024-02-12T16:57:17.178880+0000","last_clean":"2024-02-12T16:57:17.178880+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:57:17.178880+0000","last_undegraded":"2024-02-12T16:57:17.178880+0000","last_fullsized":"2024-02-12T16:57:17.178880+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":156,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.174352+0000","last_change":"2024-02-12T16:54:39.665472+0000","last_active":"2024-02-12T16:57:17.174352+0000","last_peered":"2024-02-12T16:57:17.174352+0000","last_clean":"2024-02-12T16:57:17.174352+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:57:17.174352+0000","last_undegraded":"2024-02-12T16:57:17.174352+0000","last_fullsized":"2024-02-12T16:57:17.174352+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.665378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.665378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":113,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.176814+0000","last_change":"2024-02-12T16:56:24.854608+0000","last_active":"2024-02-12T16:57:17.176814+0000","last_peered":"2024-02-12T16:57:17.176814+0000","last_clean":"2024-02-12T16:57:17.176814+0000","last_became_active":"2024-02-12T16:54:40.632501+0000","last_became_peered":"2024-02-12T16:54:40.632501+0000","last_unstale":"2024-02-12T16:57:17.176814+0000","last_undegraded":"2024-02-12T16:57:17.176814+0000","last_fullsized":"2024-02-12T16:57:17.176814+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":144,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.177337+0000","last_change":"2024-02-12T16:52:55.241717+0000","last_active":"2024-02-12T16:57:17.177337+0000","last_peered":"2024-02-12T16:57:17.177337+0000","last_clean":"2024-02-12T16:57:17.177337+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:57:17.177337+0000","last_undegraded":"2024-02-12T16:57:17.177337+0000","last_fullsized":"2024-02-12T16:57:17.177337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.241622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.241622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":157,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.173454+0000","last_change":"2024-02-12T16:56:45.226096+0000","last_active":"2024-02-12T16:57:17.173454+0000","last_peered":"2024-02-12T16:57:17.173454+0000","last_clean":"2024-02-12T16:57:17.173454+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:57:17.173454+0000","last_undegraded":"2024-02-12T16:57:17.173454+0000","last_fullsized":"2024-02-12T16:57:17.173454+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.225996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":152,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172627+0000","last_change":"2024-02-12T16:52:49.986507+0000","last_active":"2024-02-12T16:57:17.172627+0000","last_peered":"2024-02-12T16:57:17.172627+0000","last_clean":"2024-02-12T16:57:17.172627+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:57:17.172627+0000","last_undegraded":"2024-02-12T16:57:17.172627+0000","last_fullsized":"2024-02-12T16:57:17.172627+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.986403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.986403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":87,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194403+0000","last_change":"2024-02-12T16:54:41.081430+0000","last_active":"2024-02-12T16:57:02.194403+0000","last_peered":"2024-02-12T16:57:02.194403+0000","last_clean":"2024-02-12T16:57:02.194403+0000","last_became_active":"2024-02-12T16:54:40.639547+0000","last_became_peered":"2024-02-12T16:54:40.639547+0000","last_unstale":"2024-02-12T16:57:02.194403+0000","last_undegraded":"2024-02-12T16:57:02.194403+0000","last_fullsized":"2024-02-12T16:57:02.194403+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":68,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.196879+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T16:57:02.196879+0000","last_peered":"2024-02-12T16:57:02.196879+0000","last_clean":"2024-02-12T16:57:02.196879+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T16:57:02.196879+0000","last_undegraded":"2024-02-12T16:57:02.196879+0000","last_fullsized":"2024-02-12T16:57:02.196879+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":165,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.195909+0000","last_change":"2024-02-12T16:56:47.004534+0000","last_active":"2024-02-12T16:57:02.195909+0000","last_peered":"2024-02-12T16:57:02.195909+0000","last_clean":"2024-02-12T16:57:02.195909+0000","last_became_active":"2024-02-12T16:54:40.628571+0000","last_became_peered":"2024-02-12T16:54:40.628571+0000","last_unstale":"2024-02-12T16:57:02.195909+0000","last_undegraded":"2024-02-12T16:57:02.195909+0000","last_fullsized":"2024-02-12T16:57:02.195909+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":155,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.195216+0000","last_change":"2024-02-12T16:54:01.773110+0000","last_active":"2024-02-12T16:57:02.195216+0000","last_peered":"2024-02-12T16:57:02.195216+0000","last_clean":"2024-02-12T16:57:02.195216+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:57:02.195216+0000","last_undegraded":"2024-02-12T16:57:02.195216+0000","last_fullsized":"2024-02-12T16:57:02.195216+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":177,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.162562+0000","last_change":"2024-02-12T16:56:34.426891+0000","last_active":"2024-02-12T16:57:17.162562+0000","last_peered":"2024-02-12T16:57:17.162562+0000","last_clean":"2024-02-12T16:57:17.162562+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:57:17.162562+0000","last_undegraded":"2024-02-12T16:57:17.162562+0000","last_fullsized":"2024-02-12T16:57:17.162562+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:34.426800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:34.426800+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":142,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.198475+0000","last_change":"2024-02-12T16:54:40.194031+0000","last_active":"2024-02-12T16:57:02.198475+0000","last_peered":"2024-02-12T16:57:02.198475+0000","last_clean":"2024-02-12T16:57:02.198475+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:57:02.198475+0000","last_undegraded":"2024-02-12T16:57:02.198475+0000","last_fullsized":"2024-02-12T16:57:02.198475+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":84,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.197639+0000","last_change":"2024-02-12T16:56:23.899212+0000","last_active":"2024-02-12T16:57:02.197639+0000","last_peered":"2024-02-12T16:57:02.197639+0000","last_clean":"2024-02-12T16:57:02.197639+0000","last_became_active":"2024-02-12T16:54:40.650612+0000","last_became_peered":"2024-02-12T16:54:40.650612+0000","last_unstale":"2024-02-12T16:57:02.197639+0000","last_undegraded":"2024-02-12T16:57:02.197639+0000","last_fullsized":"2024-02-12T16:57:02.197639+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":165,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.175630+0000","last_change":"2024-02-12T16:56:39.248638+0000","last_active":"2024-02-12T16:57:17.175630+0000","last_peered":"2024-02-12T16:57:17.175630+0000","last_clean":"2024-02-12T16:57:17.175630+0000","last_became_active":"2024-02-12T16:54:40.629419+0000","last_became_peered":"2024-02-12T16:54:40.629419+0000","last_unstale":"2024-02-12T16:57:17.175630+0000","last_undegraded":"2024-02-12T16:57:17.175630+0000","last_fullsized":"2024-02-12T16:57:17.175630+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:39.248562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.248562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":173,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194313+0000","last_change":"2024-02-12T16:54:31.954069+0000","last_active":"2024-02-12T16:57:02.194313+0000","last_peered":"2024-02-12T16:57:02.194313+0000","last_clean":"2024-02-12T16:57:02.194313+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:57:02.194313+0000","last_undegraded":"2024-02-12T16:57:02.194313+0000","last_fullsized":"2024-02-12T16:57:02.194313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":115,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.170235+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T16:57:17.170235+0000","last_peered":"2024-02-12T16:57:17.170235+0000","last_clean":"2024-02-12T16:57:17.170235+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T16:57:17.170235+0000","last_undegraded":"2024-02-12T16:57:17.170235+0000","last_fullsized":"2024-02-12T16:57:17.170235+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"138'9","reported_seq":146,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.337642+0000","last_change":"2024-02-12T16:56:20.829720+0000","last_active":"2024-02-12T16:57:17.337642+0000","last_peered":"2024-02-12T16:57:17.337642+0000","last_clean":"2024-02-12T16:57:17.337642+0000","last_became_active":"2024-02-12T16:54:40.644025+0000","last_became_peered":"2024-02-12T16:54:40.644025+0000","last_unstale":"2024-02-12T16:57:17.337642+0000","last_undegraded":"2024-02-12T16:57:17.337642+0000","last_fullsized":"2024-02-12T16:57:17.337642+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":157,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.197822+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T16:57:02.197822+0000","last_peered":"2024-02-12T16:57:02.197822+0000","last_clean":"2024-02-12T16:57:02.197822+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T16:57:02.197822+0000","last_undegraded":"2024-02-12T16:57:02.197822+0000","last_fullsized":"2024-02-12T16:57:02.197822+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":185,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172692+0000","last_change":"2024-02-12T16:54:40.641346+0000","last_active":"2024-02-12T16:57:17.172692+0000","last_peered":"2024-02-12T16:57:17.172692+0000","last_clean":"2024-02-12T16:57:17.172692+0000","last_became_active":"2024-02-12T16:54:40.626031+0000","last_became_peered":"2024-02-12T16:54:40.626031+0000","last_unstale":"2024-02-12T16:57:17.172692+0000","last_undegraded":"2024-02-12T16:57:17.172692+0000","last_fullsized":"2024-02-12T16:57:17.172692+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":151,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.200006+0000","last_change":"2024-02-12T16:54:40.633046+0000","last_active":"2024-02-12T16:57:02.200006+0000","last_peered":"2024-02-12T16:57:02.200006+0000","last_clean":"2024-02-12T16:57:02.200006+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T16:57:02.200006+0000","last_undegraded":"2024-02-12T16:57:02.200006+0000","last_fullsized":"2024-02-12T16:57:02.200006+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.105539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.105539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":250,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.332913+0000","last_change":"2024-02-12T16:54:00.112924+0000","last_active":"2024-02-12T16:57:17.332913+0000","last_peered":"2024-02-12T16:57:17.332913+0000","last_clean":"2024-02-12T16:57:17.332913+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:57:17.332913+0000","last_undegraded":"2024-02-12T16:57:17.332913+0000","last_fullsized":"2024-02-12T16:57:17.332913+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":161,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172101+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T16:57:17.172101+0000","last_peered":"2024-02-12T16:57:17.172101+0000","last_clean":"2024-02-12T16:57:17.172101+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T16:57:17.172101+0000","last_undegraded":"2024-02-12T16:57:17.172101+0000","last_fullsized":"2024-02-12T16:57:17.172101+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":73,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194451+0000","last_change":"2024-02-12T16:54:42.827081+0000","last_active":"2024-02-12T16:57:02.194451+0000","last_peered":"2024-02-12T16:57:02.194451+0000","last_clean":"2024-02-12T16:57:02.194451+0000","last_became_active":"2024-02-12T16:54:40.639018+0000","last_became_peered":"2024-02-12T16:54:40.639018+0000","last_unstale":"2024-02-12T16:57:02.194451+0000","last_undegraded":"2024-02-12T16:57:02.194451+0000","last_fullsized":"2024-02-12T16:57:02.194451+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:03.616717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.616717+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":148,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178572+0000","last_change":"2024-02-12T16:53:51.532654+0000","last_active":"2024-02-12T16:57:17.178572+0000","last_peered":"2024-02-12T16:57:17.178572+0000","last_clean":"2024-02-12T16:57:17.178572+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:57:17.178572+0000","last_undegraded":"2024-02-12T16:57:17.178572+0000","last_fullsized":"2024-02-12T16:57:17.178572+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":162,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172460+0000","last_change":"2024-02-12T16:56:37.155376+0000","last_active":"2024-02-12T16:57:17.172460+0000","last_peered":"2024-02-12T16:57:17.172460+0000","last_clean":"2024-02-12T16:57:17.172460+0000","last_became_active":"2024-02-12T16:54:40.644226+0000","last_became_peered":"2024-02-12T16:54:40.644226+0000","last_unstale":"2024-02-12T16:57:17.172460+0000","last_undegraded":"2024-02-12T16:57:17.172460+0000","last_fullsized":"2024-02-12T16:57:17.172460+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.155311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.155311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":156,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.173496+0000","last_change":"2024-02-12T16:56:38.218147+0000","last_active":"2024-02-12T16:57:17.173496+0000","last_peered":"2024-02-12T16:57:17.173496+0000","last_clean":"2024-02-12T16:57:17.173496+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:57:17.173496+0000","last_undegraded":"2024-02-12T16:57:17.173496+0000","last_fullsized":"2024-02-12T16:57:17.173496+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.218031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.218031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":94,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.179166+0000","last_change":"2024-02-12T16:54:40.903364+0000","last_active":"2024-02-12T16:57:17.179166+0000","last_peered":"2024-02-12T16:57:17.179166+0000","last_clean":"2024-02-12T16:57:17.179166+0000","last_became_active":"2024-02-12T16:54:40.640126+0000","last_became_peered":"2024-02-12T16:54:40.640126+0000","last_unstale":"2024-02-12T16:57:17.179166+0000","last_undegraded":"2024-02-12T16:57:17.179166+0000","last_fullsized":"2024-02-12T16:57:17.179166+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":151,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169391+0000","last_change":"2024-02-12T16:56:38.699688+0000","last_active":"2024-02-12T16:57:17.169391+0000","last_peered":"2024-02-12T16:57:17.169391+0000","last_clean":"2024-02-12T16:57:17.169391+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:57:17.169391+0000","last_undegraded":"2024-02-12T16:57:17.169391+0000","last_fullsized":"2024-02-12T16:57:17.169391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.699602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":63,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194866+0000","last_change":"2024-02-12T16:54:40.634603+0000","last_active":"2024-02-12T16:57:02.194866+0000","last_peered":"2024-02-12T16:57:02.194866+0000","last_clean":"2024-02-12T16:57:02.194866+0000","last_became_active":"2024-02-12T16:54:40.634261+0000","last_became_peered":"2024-02-12T16:54:40.634261+0000","last_unstale":"2024-02-12T16:57:02.194866+0000","last_undegraded":"2024-02-12T16:57:02.194866+0000","last_fullsized":"2024-02-12T16:57:02.194866+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:41.740892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:53:41.740892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":157,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.168325+0000","last_change":"2024-02-12T16:56:22.699402+0000","last_active":"2024-02-12T16:57:17.168325+0000","last_peered":"2024-02-12T16:57:17.168325+0000","last_clean":"2024-02-12T16:57:17.168325+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:57:17.168325+0000","last_undegraded":"2024-02-12T16:57:17.168325+0000","last_fullsized":"2024-02-12T16:57:17.168325+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":185,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.200154+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T16:57:02.200154+0000","last_peered":"2024-02-12T16:57:02.200154+0000","last_clean":"2024-02-12T16:57:02.200154+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T16:57:02.200154+0000","last_undegraded":"2024-02-12T16:57:02.200154+0000","last_fullsized":"2024-02-12T16:57:02.200154+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"191'3271","reported_seq":3294,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.330164+0000","last_change":"2024-02-12T16:56:36.789085+0000","last_active":"2024-02-12T16:57:18.330164+0000","last_peered":"2024-02-12T16:57:18.330164+0000","last_clean":"2024-02-12T16:57:18.330164+0000","last_became_active":"2024-02-12T16:56:36.788797+0000","last_became_peered":"2024-02-12T16:56:36.788797+0000","last_unstale":"2024-02-12T16:57:18.330164+0000","last_undegraded":"2024-02-12T16:57:18.330164+0000","last_fullsized":"2024-02-12T16:57:18.330164+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3271,"ondisk_log_size":3271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":214368256,"num_objects":3271,"num_object_clones":0,"num_object_copies":9813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3271,"num_write_kb":209344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":65,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.177598+0000","last_change":"2024-02-12T16:54:40.643841+0000","last_active":"2024-02-12T16:57:17.177598+0000","last_peered":"2024-02-12T16:57:17.177598+0000","last_clean":"2024-02-12T16:57:17.177598+0000","last_became_active":"2024-02-12T16:54:40.643529+0000","last_became_peered":"2024-02-12T16:54:40.643529+0000","last_unstale":"2024-02-12T16:57:17.177598+0000","last_undegraded":"2024-02-12T16:57:17.177598+0000","last_fullsized":"2024-02-12T16:57:17.177598+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":177,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.165313+0000","last_change":"2024-02-12T16:55:34.601557+0000","last_active":"2024-02-12T16:57:17.165313+0000","last_peered":"2024-02-12T16:57:17.165313+0000","last_clean":"2024-02-12T16:57:17.165313+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:57:17.165313+0000","last_undegraded":"2024-02-12T16:57:17.165313+0000","last_fullsized":"2024-02-12T16:57:17.165313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:55:34.601453+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:55:34.601453+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":46,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.171852+0000","last_change":"2024-02-12T16:56:05.735251+0000","last_active":"2024-02-12T16:57:17.171852+0000","last_peered":"2024-02-12T16:57:17.171852+0000","last_clean":"2024-02-12T16:57:17.171852+0000","last_became_active":"2024-02-12T16:56:04.871405+0000","last_became_peered":"2024-02-12T16:56:04.871405+0000","last_unstale":"2024-02-12T16:57:17.171852+0000","last_undegraded":"2024-02-12T16:57:17.171852+0000","last_fullsized":"2024-02-12T16:57:17.171852+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:52:44.868438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:44.868438+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":153,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.168403+0000","last_change":"2024-02-12T16:56:45.647692+0000","last_active":"2024-02-12T16:57:17.168403+0000","last_peered":"2024-02-12T16:57:17.168403+0000","last_clean":"2024-02-12T16:57:17.168403+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:57:17.168403+0000","last_undegraded":"2024-02-12T16:57:17.168403+0000","last_fullsized":"2024-02-12T16:57:17.168403+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.647627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.647627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"189'2855","reported_seq":2876,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:13.501798+0000","last_change":"2024-02-12T16:56:36.446431+0000","last_active":"2024-02-12T16:57:13.501798+0000","last_peered":"2024-02-12T16:57:13.501798+0000","last_clean":"2024-02-12T16:57:13.501798+0000","last_became_active":"2024-02-12T16:56:36.446107+0000","last_became_peered":"2024-02-12T16:56:36.446107+0000","last_unstale":"2024-02-12T16:57:13.501798+0000","last_undegraded":"2024-02-12T16:57:13.501798+0000","last_fullsized":"2024-02-12T16:57:13.501798+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":2855,"ondisk_log_size":2855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":187105280,"num_objects":2855,"num_object_clones":0,"num_object_copies":8565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2855,"num_write_kb":182720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":151,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.198560+0000","last_change":"2024-02-12T16:54:40.638105+0000","last_active":"2024-02-12T16:57:02.198560+0000","last_peered":"2024-02-12T16:57:02.198560+0000","last_clean":"2024-02-12T16:57:02.198560+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T16:57:02.198560+0000","last_undegraded":"2024-02-12T16:57:02.198560+0000","last_fullsized":"2024-02-12T16:57:02.198560+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.248970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.248970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.195841+0000","last_change":"2024-02-12T16:56:37.009164+0000","last_active":"2024-02-12T16:57:02.195841+0000","last_peered":"2024-02-12T16:57:02.195841+0000","last_clean":"2024-02-12T16:57:02.195841+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:57:02.195841+0000","last_undegraded":"2024-02-12T16:57:02.195841+0000","last_fullsized":"2024-02-12T16:57:02.195841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.009094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.009094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":141,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169641+0000","last_change":"2024-02-12T16:54:39.421299+0000","last_active":"2024-02-12T16:57:17.169641+0000","last_peered":"2024-02-12T16:57:17.169641+0000","last_clean":"2024-02-12T16:57:17.169641+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:57:17.169641+0000","last_undegraded":"2024-02-12T16:57:17.169641+0000","last_fullsized":"2024-02-12T16:57:17.169641+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:39.421176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.421176+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":243,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:07.332521+0000","last_change":"2024-02-12T16:53:59.173126+0000","last_active":"2024-02-12T16:57:07.332521+0000","last_peered":"2024-02-12T16:57:07.332521+0000","last_clean":"2024-02-12T16:57:07.332521+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:57:07.332521+0000","last_undegraded":"2024-02-12T16:57:07.332521+0000","last_fullsized":"2024-02-12T16:57:07.332521+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"191'3347","reported_seq":3370,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.329797+0000","last_change":"2024-02-12T16:56:36.788519+0000","last_active":"2024-02-12T16:57:18.329797+0000","last_peered":"2024-02-12T16:57:18.329797+0000","last_clean":"2024-02-12T16:57:18.329797+0000","last_became_active":"2024-02-12T16:56:36.788101+0000","last_became_peered":"2024-02-12T16:56:36.788101+0000","last_unstale":"2024-02-12T16:57:18.329797+0000","last_undegraded":"2024-02-12T16:57:18.329797+0000","last_fullsized":"2024-02-12T16:57:18.329797+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3347,"ondisk_log_size":3347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":219348992,"num_objects":3347,"num_object_clones":0,"num_object_copies":10041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3347,"num_write_kb":214208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":69,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.163166+0000","last_change":"2024-02-12T16:55:40.530363+0000","last_active":"2024-02-12T16:57:17.163166+0000","last_peered":"2024-02-12T16:57:17.163166+0000","last_clean":"2024-02-12T16:57:17.163166+0000","last_became_active":"2024-02-12T16:54:40.625901+0000","last_became_peered":"2024-02-12T16:54:40.625901+0000","last_unstale":"2024-02-12T16:57:17.163166+0000","last_undegraded":"2024-02-12T16:57:17.163166+0000","last_fullsized":"2024-02-12T16:57:17.163166+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:55:40.530269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":157,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172188+0000","last_change":"2024-02-12T16:56:25.227663+0000","last_active":"2024-02-12T16:57:17.172188+0000","last_peered":"2024-02-12T16:57:17.172188+0000","last_clean":"2024-02-12T16:57:17.172188+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:57:17.172188+0000","last_undegraded":"2024-02-12T16:57:17.172188+0000","last_fullsized":"2024-02-12T16:57:17.172188+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"180'39","reported_seq":334,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:07.333004+0000","last_change":"2024-02-12T16:56:43.083756+0000","last_active":"2024-02-12T16:57:07.333004+0000","last_peered":"2024-02-12T16:57:07.333004+0000","last_clean":"2024-02-12T16:57:07.333004+0000","last_became_active":"2024-02-12T16:54:40.639565+0000","last_became_peered":"2024-02-12T16:54:40.639565+0000","last_unstale":"2024-02-12T16:57:07.333004+0000","last_undegraded":"2024-02-12T16:57:07.333004+0000","last_fullsized":"2024-02-12T16:57:07.333004+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'39","last_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.083656+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":149,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.199975+0000","last_change":"2024-02-12T16:56:46.719822+0000","last_active":"2024-02-12T16:57:02.199975+0000","last_peered":"2024-02-12T16:57:02.199975+0000","last_clean":"2024-02-12T16:57:02.199975+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:57:02.199975+0000","last_undegraded":"2024-02-12T16:57:02.199975+0000","last_fullsized":"2024-02-12T16:57:02.199975+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:46.719706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:46.719706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"191'3202","reported_seq":3225,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.313283+0000","last_change":"2024-02-12T16:56:36.443593+0000","last_active":"2024-02-12T16:57:18.313283+0000","last_peered":"2024-02-12T16:57:18.313283+0000","last_clean":"2024-02-12T16:57:18.313283+0000","last_became_active":"2024-02-12T16:56:36.443309+0000","last_became_peered":"2024-02-12T16:56:36.443309+0000","last_unstale":"2024-02-12T16:57:18.313283+0000","last_undegraded":"2024-02-12T16:57:18.313283+0000","last_fullsized":"2024-02-12T16:57:18.313283+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3202,"ondisk_log_size":3202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":209846272,"num_objects":3202,"num_object_clones":0,"num_object_copies":9606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3202,"num_write_kb":204928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":155,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178773+0000","last_change":"2024-02-12T16:54:40.640502+0000","last_active":"2024-02-12T16:57:17.178773+0000","last_peered":"2024-02-12T16:57:17.178773+0000","last_clean":"2024-02-12T16:57:17.178773+0000","last_became_active":"2024-02-12T16:54:40.640105+0000","last_became_peered":"2024-02-12T16:54:40.640105+0000","last_unstale":"2024-02-12T16:57:17.178773+0000","last_undegraded":"2024-02-12T16:57:17.178773+0000","last_fullsized":"2024-02-12T16:57:17.178773+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":604,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:11.546719+0000","last_change":"2024-02-12T16:54:30.972535+0000","last_active":"2024-02-12T16:57:11.546719+0000","last_peered":"2024-02-12T16:57:11.546719+0000","last_clean":"2024-02-12T16:57:11.546719+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:57:11.546719+0000","last_undegraded":"2024-02-12T16:57:11.546719+0000","last_fullsized":"2024-02-12T16:57:11.546719+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":446,"num_read_kb":441,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":166,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169240+0000","last_change":"2024-02-12T16:56:20.676702+0000","last_active":"2024-02-12T16:57:17.169240+0000","last_peered":"2024-02-12T16:57:17.169240+0000","last_clean":"2024-02-12T16:57:17.169240+0000","last_became_active":"2024-02-12T16:54:40.632657+0000","last_became_peered":"2024-02-12T16:54:40.632657+0000","last_unstale":"2024-02-12T16:57:17.169240+0000","last_undegraded":"2024-02-12T16:57:17.169240+0000","last_fullsized":"2024-02-12T16:57:17.169240+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":185,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194662+0000","last_change":"2024-02-12T16:56:28.196061+0000","last_active":"2024-02-12T16:57:02.194662+0000","last_peered":"2024-02-12T16:57:02.194662+0000","last_clean":"2024-02-12T16:57:02.194662+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:57:02.194662+0000","last_undegraded":"2024-02-12T16:57:02.194662+0000","last_fullsized":"2024-02-12T16:57:02.194662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":153,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.691136+0000","last_change":"2024-02-12T16:57:17.691096+0000","last_active":"2024-02-12T16:57:17.691136+0000","last_peered":"2024-02-12T16:57:17.691136+0000","last_clean":"2024-02-12T16:57:17.691136+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:57:17.691136+0000","last_undegraded":"2024-02-12T16:57:17.691136+0000","last_fullsized":"2024-02-12T16:57:17.691136+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"191'3222","reported_seq":3245,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.333232+0000","last_change":"2024-02-12T16:56:36.447717+0000","last_active":"2024-02-12T16:57:18.333232+0000","last_peered":"2024-02-12T16:57:18.333232+0000","last_clean":"2024-02-12T16:57:18.333232+0000","last_became_active":"2024-02-12T16:56:36.447417+0000","last_became_peered":"2024-02-12T16:56:36.447417+0000","last_unstale":"2024-02-12T16:57:18.333232+0000","last_undegraded":"2024-02-12T16:57:18.333232+0000","last_fullsized":"2024-02-12T16:57:18.333232+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3222,"ondisk_log_size":3222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":211156992,"num_objects":3222,"num_object_clones":0,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3222,"num_write_kb":206208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":155,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172964+0000","last_change":"2024-02-12T16:54:40.628021+0000","last_active":"2024-02-12T16:57:17.172964+0000","last_peered":"2024-02-12T16:57:17.172964+0000","last_clean":"2024-02-12T16:57:17.172964+0000","last_became_active":"2024-02-12T16:54:40.627688+0000","last_became_peered":"2024-02-12T16:54:40.627688+0000","last_unstale":"2024-02-12T16:57:17.172964+0000","last_undegraded":"2024-02-12T16:57:17.172964+0000","last_fullsized":"2024-02-12T16:57:17.172964+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":70,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172315+0000","last_change":"2024-02-12T16:56:45.218102+0000","last_active":"2024-02-12T16:57:17.172315+0000","last_peered":"2024-02-12T16:57:17.172315+0000","last_clean":"2024-02-12T16:57:17.172315+0000","last_became_active":"2024-02-12T16:54:40.628365+0000","last_became_peered":"2024-02-12T16:54:40.628365+0000","last_unstale":"2024-02-12T16:57:17.172315+0000","last_undegraded":"2024-02-12T16:57:17.172315+0000","last_fullsized":"2024-02-12T16:57:17.172315+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":152,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.173015+0000","last_change":"2024-02-12T16:52:57.985677+0000","last_active":"2024-02-12T16:57:17.173015+0000","last_peered":"2024-02-12T16:57:17.173015+0000","last_clean":"2024-02-12T16:57:17.173015+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:57:17.173015+0000","last_undegraded":"2024-02-12T16:57:17.173015+0000","last_fullsized":"2024-02-12T16:57:17.173015+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:57.985584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:57.985584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":158,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178703+0000","last_change":"2024-02-12T16:54:30.770032+0000","last_active":"2024-02-12T16:57:17.178703+0000","last_peered":"2024-02-12T16:57:17.178703+0000","last_clean":"2024-02-12T16:57:17.178703+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:57:17.178703+0000","last_undegraded":"2024-02-12T16:57:17.178703+0000","last_fullsized":"2024-02-12T16:57:17.178703+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":176,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.174876+0000","last_change":"2024-02-12T16:53:58.740196+0000","last_active":"2024-02-12T16:57:17.174876+0000","last_peered":"2024-02-12T16:57:17.174876+0000","last_clean":"2024-02-12T16:57:17.174876+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:57:17.174876+0000","last_undegraded":"2024-02-12T16:57:17.174876+0000","last_fullsized":"2024-02-12T16:57:17.174876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:53:58.740122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.740122+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":151,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.945080+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T16:57:17.945080+0000","last_peered":"2024-02-12T16:57:17.945080+0000","last_clean":"2024-02-12T16:57:17.945080+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:57:17.945080+0000","last_undegraded":"2024-02-12T16:57:17.945080+0000","last_fullsized":"2024-02-12T16:57:17.945080+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"189'2806","reported_seq":2827,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:13.491716+0000","last_change":"2024-02-12T16:56:36.447068+0000","last_active":"2024-02-12T16:57:13.491716+0000","last_peered":"2024-02-12T16:57:13.491716+0000","last_clean":"2024-02-12T16:57:13.491716+0000","last_became_active":"2024-02-12T16:56:36.446744+0000","last_became_peered":"2024-02-12T16:56:36.446744+0000","last_unstale":"2024-02-12T16:57:13.491716+0000","last_undegraded":"2024-02-12T16:57:13.491716+0000","last_fullsized":"2024-02-12T16:57:13.491716+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":183894016,"num_objects":2806,"num_object_clones":0,"num_object_copies":8418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":179584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":153,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.195792+0000","last_change":"2024-02-12T16:54:40.628087+0000","last_active":"2024-02-12T16:57:02.195792+0000","last_peered":"2024-02-12T16:57:02.195792+0000","last_clean":"2024-02-12T16:57:02.195792+0000","last_became_active":"2024-02-12T16:54:40.626511+0000","last_became_peered":"2024-02-12T16:54:40.626511+0000","last_unstale":"2024-02-12T16:57:02.195792+0000","last_undegraded":"2024-02-12T16:57:02.195792+0000","last_fullsized":"2024-02-12T16:57:02.195792+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.780846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.780846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":157,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.196798+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T16:57:02.196798+0000","last_peered":"2024-02-12T16:57:02.196798+0000","last_clean":"2024-02-12T16:57:02.196798+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T16:57:02.196798+0000","last_undegraded":"2024-02-12T16:57:02.196798+0000","last_fullsized":"2024-02-12T16:57:02.196798+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":156,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178278+0000","last_change":"2024-02-12T16:52:58.167642+0000","last_active":"2024-02-12T16:57:17.178278+0000","last_peered":"2024-02-12T16:57:17.178278+0000","last_clean":"2024-02-12T16:57:17.178278+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:57:17.178278+0000","last_undegraded":"2024-02-12T16:57:17.178278+0000","last_fullsized":"2024-02-12T16:57:17.178278+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:52:58.167543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:58.167543+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":164,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.200059+0000","last_change":"2024-02-12T16:53:43.229740+0000","last_active":"2024-02-12T16:57:02.200059+0000","last_peered":"2024-02-12T16:57:02.200059+0000","last_clean":"2024-02-12T16:57:02.200059+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:57:02.200059+0000","last_undegraded":"2024-02-12T16:57:02.200059+0000","last_fullsized":"2024-02-12T16:57:02.200059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:43.229622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.229622+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":148,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.173072+0000","last_change":"2024-02-12T16:56:18.162272+0000","last_active":"2024-02-12T16:57:17.173072+0000","last_peered":"2024-02-12T16:57:17.173072+0000","last_clean":"2024-02-12T16:57:17.173072+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:57:17.173072+0000","last_undegraded":"2024-02-12T16:57:17.173072+0000","last_fullsized":"2024-02-12T16:57:17.173072+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"191'3258","reported_seq":3281,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.419774+0000","last_change":"2024-02-12T16:56:36.443457+0000","last_active":"2024-02-12T16:57:18.419774+0000","last_peered":"2024-02-12T16:57:18.419774+0000","last_clean":"2024-02-12T16:57:18.419774+0000","last_became_active":"2024-02-12T16:56:36.443153+0000","last_became_peered":"2024-02-12T16:56:36.443153+0000","last_unstale":"2024-02-12T16:57:18.419774+0000","last_undegraded":"2024-02-12T16:57:18.419774+0000","last_fullsized":"2024-02-12T16:57:18.419774+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3258,"ondisk_log_size":3258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":213516288,"num_objects":3258,"num_object_clones":0,"num_object_copies":9774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3258,"num_write_kb":208512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":149,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194476+0000","last_change":"2024-02-12T16:56:41.053659+0000","last_active":"2024-02-12T16:57:02.194476+0000","last_peered":"2024-02-12T16:57:02.194476+0000","last_clean":"2024-02-12T16:57:02.194476+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:57:02.194476+0000","last_undegraded":"2024-02-12T16:57:02.194476+0000","last_fullsized":"2024-02-12T16:57:02.194476+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.053587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.053587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":155,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178847+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T16:57:17.178847+0000","last_peered":"2024-02-12T16:57:17.178847+0000","last_clean":"2024-02-12T16:57:17.178847+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:57:17.178847+0000","last_undegraded":"2024-02-12T16:57:17.178847+0000","last_fullsized":"2024-02-12T16:57:17.178847+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":187,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.197757+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T16:57:02.197757+0000","last_peered":"2024-02-12T16:57:02.197757+0000","last_clean":"2024-02-12T16:57:02.197757+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:57:02.197757+0000","last_undegraded":"2024-02-12T16:57:02.197757+0000","last_fullsized":"2024-02-12T16:57:02.197757+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":157,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.283726+0000","last_change":"2024-02-12T16:57:17.283687+0000","last_active":"2024-02-12T16:57:17.283726+0000","last_peered":"2024-02-12T16:57:17.283726+0000","last_clean":"2024-02-12T16:57:17.283726+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:57:17.283726+0000","last_undegraded":"2024-02-12T16:57:17.283726+0000","last_fullsized":"2024-02-12T16:57:17.283726+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"189'2927","reported_seq":2948,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:13.483257+0000","last_change":"2024-02-12T16:56:36.789146+0000","last_active":"2024-02-12T16:57:13.483257+0000","last_peered":"2024-02-12T16:57:13.483257+0000","last_clean":"2024-02-12T16:57:13.483257+0000","last_became_active":"2024-02-12T16:56:36.788871+0000","last_became_peered":"2024-02-12T16:56:36.788871+0000","last_unstale":"2024-02-12T16:57:13.483257+0000","last_undegraded":"2024-02-12T16:57:13.483257+0000","last_fullsized":"2024-02-12T16:57:13.483257+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":2927,"ondisk_log_size":2927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":191823872,"num_objects":2927,"num_object_clones":0,"num_object_copies":8781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":187328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":68,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169712+0000","last_change":"2024-02-12T16:54:40.632821+0000","last_active":"2024-02-12T16:57:17.169712+0000","last_peered":"2024-02-12T16:57:17.169712+0000","last_clean":"2024-02-12T16:57:17.169712+0000","last_became_active":"2024-02-12T16:54:40.631423+0000","last_became_peered":"2024-02-12T16:54:40.631423+0000","last_unstale":"2024-02-12T16:57:17.169712+0000","last_undegraded":"2024-02-12T16:57:17.169712+0000","last_fullsized":"2024-02-12T16:57:17.169712+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.624865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.624865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194850+0000","last_change":"2024-02-12T16:53:48.828698+0000","last_active":"2024-02-12T16:57:02.194850+0000","last_peered":"2024-02-12T16:57:02.194850+0000","last_clean":"2024-02-12T16:57:02.194850+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:57:02.194850+0000","last_undegraded":"2024-02-12T16:57:02.194850+0000","last_fullsized":"2024-02-12T16:57:02.194850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":132,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178955+0000","last_change":"2024-02-12T16:56:26.889557+0000","last_active":"2024-02-12T16:57:17.178955+0000","last_peered":"2024-02-12T16:57:17.178955+0000","last_clean":"2024-02-12T16:57:17.178955+0000","last_became_active":"2024-02-12T16:54:40.634781+0000","last_became_peered":"2024-02-12T16:54:40.634781+0000","last_unstale":"2024-02-12T16:57:17.178955+0000","last_undegraded":"2024-02-12T16:57:17.178955+0000","last_fullsized":"2024-02-12T16:57:17.178955+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":163,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.162283+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T16:57:17.162283+0000","last_peered":"2024-02-12T16:57:17.162283+0000","last_clean":"2024-02-12T16:57:17.162283+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T16:57:17.162283+0000","last_undegraded":"2024-02-12T16:57:17.162283+0000","last_fullsized":"2024-02-12T16:57:17.162283+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"191'3315","reported_seq":3338,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.301112+0000","last_change":"2024-02-12T16:56:36.788810+0000","last_active":"2024-02-12T16:57:18.301112+0000","last_peered":"2024-02-12T16:57:18.301112+0000","last_clean":"2024-02-12T16:57:18.301112+0000","last_became_active":"2024-02-12T16:56:36.788518+0000","last_became_peered":"2024-02-12T16:56:36.788518+0000","last_unstale":"2024-02-12T16:57:18.301112+0000","last_undegraded":"2024-02-12T16:57:18.301112+0000","last_fullsized":"2024-02-12T16:57:18.301112+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3315,"ondisk_log_size":3315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":217251840,"num_objects":3315,"num_object_clones":0,"num_object_copies":9945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":212160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":145,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.177865+0000","last_change":"2024-02-12T16:52:41.149470+0000","last_active":"2024-02-12T16:57:17.177865+0000","last_peered":"2024-02-12T16:57:17.177865+0000","last_clean":"2024-02-12T16:57:17.177865+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:57:17.177865+0000","last_undegraded":"2024-02-12T16:57:17.177865+0000","last_fullsized":"2024-02-12T16:57:17.177865+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:52:41.149382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:52:41.149382+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":156,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.174419+0000","last_change":"2024-02-12T16:56:26.337548+0000","last_active":"2024-02-12T16:57:17.174419+0000","last_peered":"2024-02-12T16:57:17.174419+0000","last_clean":"2024-02-12T16:57:17.174419+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:57:17.174419+0000","last_undegraded":"2024-02-12T16:57:17.174419+0000","last_fullsized":"2024-02-12T16:57:17.174419+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":143,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172141+0000","last_change":"2024-02-12T16:52:51.025242+0000","last_active":"2024-02-12T16:57:17.172141+0000","last_peered":"2024-02-12T16:57:17.172141+0000","last_clean":"2024-02-12T16:57:17.172141+0000","last_became_active":"2024-02-12T16:52:20.563744+0000","last_became_peered":"2024-02-12T16:52:20.563744+0000","last_unstale":"2024-02-12T16:57:17.172141+0000","last_undegraded":"2024-02-12T16:57:17.172141+0000","last_fullsized":"2024-02-12T16:57:17.172141+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:52:51.025152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:52:51.025152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"138'9","reported_seq":123,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:07.332650+0000","last_change":"2024-02-12T16:55:28.449149+0000","last_active":"2024-02-12T16:57:07.332650+0000","last_peered":"2024-02-12T16:57:07.332650+0000","last_clean":"2024-02-12T16:57:07.332650+0000","last_became_active":"2024-02-12T16:54:40.626646+0000","last_became_peered":"2024-02-12T16:54:40.626646+0000","last_unstale":"2024-02-12T16:57:07.332650+0000","last_undegraded":"2024-02-12T16:57:07.332650+0000","last_fullsized":"2024-02-12T16:57:07.332650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"191'3257","reported_seq":3280,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.299981+0000","last_change":"2024-02-12T16:56:36.445962+0000","last_active":"2024-02-12T16:57:18.299981+0000","last_peered":"2024-02-12T16:57:18.299981+0000","last_clean":"2024-02-12T16:57:18.299981+0000","last_became_active":"2024-02-12T16:56:36.445638+0000","last_became_peered":"2024-02-12T16:56:36.445638+0000","last_unstale":"2024-02-12T16:57:18.299981+0000","last_undegraded":"2024-02-12T16:57:18.299981+0000","last_fullsized":"2024-02-12T16:57:18.299981+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3257,"ondisk_log_size":3257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":213450752,"num_objects":3257,"num_object_clones":0,"num_object_copies":9771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":208448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":65,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.175554+0000","last_change":"2024-02-12T16:54:40.629725+0000","last_active":"2024-02-12T16:57:17.175554+0000","last_peered":"2024-02-12T16:57:17.175554+0000","last_clean":"2024-02-12T16:57:17.175554+0000","last_became_active":"2024-02-12T16:54:40.629348+0000","last_became_peered":"2024-02-12T16:54:40.629348+0000","last_unstale":"2024-02-12T16:57:17.175554+0000","last_undegraded":"2024-02-12T16:57:17.175554+0000","last_fullsized":"2024-02-12T16:57:17.175554+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":148,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172810+0000","last_change":"2024-02-12T16:52:52.983089+0000","last_active":"2024-02-12T16:57:17.172810+0000","last_peered":"2024-02-12T16:57:17.172810+0000","last_clean":"2024-02-12T16:57:17.172810+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:57:17.172810+0000","last_undegraded":"2024-02-12T16:57:17.172810+0000","last_fullsized":"2024-02-12T16:57:17.172810+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:52.982996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.982996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":129,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.168253+0000","last_change":"2024-02-12T16:53:59.566836+0000","last_active":"2024-02-12T16:57:17.168253+0000","last_peered":"2024-02-12T16:57:17.168253+0000","last_clean":"2024-02-12T16:57:17.168253+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:57:17.168253+0000","last_undegraded":"2024-02-12T16:57:17.168253+0000","last_fullsized":"2024-02-12T16:57:17.168253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.566730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":167,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.174513+0000","last_change":"2024-02-12T16:56:04.931453+0000","last_active":"2024-02-12T16:57:17.174513+0000","last_peered":"2024-02-12T16:57:17.174513+0000","last_clean":"2024-02-12T16:57:17.174513+0000","last_became_active":"2024-02-12T16:56:04.863064+0000","last_became_peered":"2024-02-12T16:56:04.863064+0000","last_unstale":"2024-02-12T16:57:17.174513+0000","last_undegraded":"2024-02-12T16:57:17.174513+0000","last_fullsized":"2024-02-12T16:57:17.174513+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:54:36.753321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:36.753321+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"189'2761","reported_seq":2782,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:13.471522+0000","last_change":"2024-02-12T16:56:36.447192+0000","last_active":"2024-02-12T16:57:13.471522+0000","last_peered":"2024-02-12T16:57:13.471522+0000","last_clean":"2024-02-12T16:57:13.471522+0000","last_became_active":"2024-02-12T16:56:36.446926+0000","last_became_peered":"2024-02-12T16:56:36.446926+0000","last_unstale":"2024-02-12T16:57:13.471522+0000","last_undegraded":"2024-02-12T16:57:13.471522+0000","last_fullsized":"2024-02-12T16:57:13.471522+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":2761,"ondisk_log_size":2761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2761,"num_write_kb":176704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":151,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172947+0000","last_change":"2024-02-12T16:56:40.152368+0000","last_active":"2024-02-12T16:57:17.172947+0000","last_peered":"2024-02-12T16:57:17.172947+0000","last_clean":"2024-02-12T16:57:17.172947+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:57:17.172947+0000","last_undegraded":"2024-02-12T16:57:17.172947+0000","last_fullsized":"2024-02-12T16:57:17.172947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.152265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"98'10","reported_seq":221,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.332926+0000","last_change":"2024-02-12T16:56:32.292792+0000","last_active":"2024-02-12T16:57:17.332926+0000","last_peered":"2024-02-12T16:57:17.332926+0000","last_clean":"2024-02-12T16:57:17.332926+0000","last_became_active":"2024-02-12T16:52:20.561534+0000","last_became_peered":"2024-02-12T16:52:20.561534+0000","last_unstale":"2024-02-12T16:57:17.332926+0000","last_undegraded":"2024-02-12T16:57:17.332926+0000","last_fullsized":"2024-02-12T16:57:17.332926+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":173,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.193945+0000","last_change":"2024-02-12T16:56:04.857653+0000","last_active":"2024-02-12T16:57:02.193945+0000","last_peered":"2024-02-12T16:57:02.193945+0000","last_clean":"2024-02-12T16:57:02.193945+0000","last_became_active":"2024-02-12T16:56:04.857412+0000","last_became_peered":"2024-02-12T16:56:04.857412+0000","last_unstale":"2024-02-12T16:57:02.193945+0000","last_undegraded":"2024-02-12T16:57:02.193945+0000","last_fullsized":"2024-02-12T16:57:02.193945+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":210,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194720+0000","last_change":"2024-02-12T16:54:41.243992+0000","last_active":"2024-02-12T16:57:02.194720+0000","last_peered":"2024-02-12T16:57:02.194720+0000","last_clean":"2024-02-12T16:57:02.194720+0000","last_became_active":"2024-02-12T16:54:40.627029+0000","last_became_peered":"2024-02-12T16:54:40.627029+0000","last_unstale":"2024-02-12T16:57:02.194720+0000","last_undegraded":"2024-02-12T16:57:02.194720+0000","last_fullsized":"2024-02-12T16:57:02.194720+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"191'3286","reported_seq":3309,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.314615+0000","last_change":"2024-02-12T16:56:36.449103+0000","last_active":"2024-02-12T16:57:18.314615+0000","last_peered":"2024-02-12T16:57:18.314615+0000","last_clean":"2024-02-12T16:57:18.314615+0000","last_became_active":"2024-02-12T16:56:36.448827+0000","last_became_peered":"2024-02-12T16:56:36.448827+0000","last_unstale":"2024-02-12T16:57:18.314615+0000","last_undegraded":"2024-02-12T16:57:18.314615+0000","last_fullsized":"2024-02-12T16:57:18.314615+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3286,"ondisk_log_size":3286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":215351296,"num_objects":3286,"num_object_clones":0,"num_object_copies":9858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3286,"num_write_kb":210304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194639+0000","last_change":"2024-02-12T16:53:57.738352+0000","last_active":"2024-02-12T16:57:02.194639+0000","last_peered":"2024-02-12T16:57:02.194639+0000","last_clean":"2024-02-12T16:57:02.194639+0000","last_became_active":"2024-02-12T16:52:20.560085+0000","last_became_peered":"2024-02-12T16:52:20.560085+0000","last_unstale":"2024-02-12T16:57:02.194639+0000","last_undegraded":"2024-02-12T16:57:02.194639+0000","last_fullsized":"2024-02-12T16:57:02.194639+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:57.738249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.738249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":253,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.336504+0000","last_change":"2024-02-12T16:56:26.673555+0000","last_active":"2024-02-12T16:57:17.336504+0000","last_peered":"2024-02-12T16:57:17.336504+0000","last_clean":"2024-02-12T16:57:17.336504+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:57:17.336504+0000","last_undegraded":"2024-02-12T16:57:17.336504+0000","last_fullsized":"2024-02-12T16:57:17.336504+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":164,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.194192+0000","last_change":"2024-02-12T16:54:40.629536+0000","last_active":"2024-02-12T16:57:02.194192+0000","last_peered":"2024-02-12T16:57:02.194192+0000","last_clean":"2024-02-12T16:57:02.194192+0000","last_became_active":"2024-02-12T16:54:40.629024+0000","last_became_peered":"2024-02-12T16:54:40.629024+0000","last_unstale":"2024-02-12T16:57:02.194192+0000","last_undegraded":"2024-02-12T16:57:02.194192+0000","last_fullsized":"2024-02-12T16:57:02.194192+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":109,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.175709+0000","last_change":"2024-02-12T16:55:46.162945+0000","last_active":"2024-02-12T16:57:17.175709+0000","last_peered":"2024-02-12T16:57:17.175709+0000","last_clean":"2024-02-12T16:57:17.175709+0000","last_became_active":"2024-02-12T16:54:40.629780+0000","last_became_peered":"2024-02-12T16:54:40.629780+0000","last_unstale":"2024-02-12T16:57:17.175709+0000","last_undegraded":"2024-02-12T16:57:17.175709+0000","last_fullsized":"2024-02-12T16:57:17.175709+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"189'2760","reported_seq":2781,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:13.476402+0000","last_change":"2024-02-12T16:56:36.447081+0000","last_active":"2024-02-12T16:57:13.476402+0000","last_peered":"2024-02-12T16:57:13.476402+0000","last_clean":"2024-02-12T16:57:13.476402+0000","last_became_active":"2024-02-12T16:56:36.446786+0000","last_became_peered":"2024-02-12T16:56:36.446786+0000","last_unstale":"2024-02-12T16:57:13.476402+0000","last_undegraded":"2024-02-12T16:57:13.476402+0000","last_fullsized":"2024-02-12T16:57:13.476402+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":2760,"ondisk_log_size":2760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":180879360,"num_objects":2760,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2760,"num_write_kb":176640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":160,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.170157+0000","last_change":"2024-02-12T16:54:40.631210+0000","last_active":"2024-02-12T16:57:17.170157+0000","last_peered":"2024-02-12T16:57:17.170157+0000","last_clean":"2024-02-12T16:57:17.170157+0000","last_became_active":"2024-02-12T16:54:40.630855+0000","last_became_peered":"2024-02-12T16:54:40.630855+0000","last_unstale":"2024-02-12T16:57:17.170157+0000","last_undegraded":"2024-02-12T16:57:17.170157+0000","last_fullsized":"2024-02-12T16:57:17.170157+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.387144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.387144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":151,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.195708+0000","last_change":"2024-02-12T16:56:33.053023+0000","last_active":"2024-02-12T16:57:02.195708+0000","last_peered":"2024-02-12T16:57:02.195708+0000","last_clean":"2024-02-12T16:57:02.195708+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:57:02.195708+0000","last_undegraded":"2024-02-12T16:57:02.195708+0000","last_fullsized":"2024-02-12T16:57:02.195708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":155,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.177526+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T16:57:17.177526+0000","last_peered":"2024-02-12T16:57:17.177526+0000","last_clean":"2024-02-12T16:57:17.177526+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:57:17.177526+0000","last_undegraded":"2024-02-12T16:57:17.177526+0000","last_fullsized":"2024-02-12T16:57:17.177526+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":182,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.170046+0000","last_change":"2024-02-12T16:56:30.714551+0000","last_active":"2024-02-12T16:57:17.170046+0000","last_peered":"2024-02-12T16:57:17.170046+0000","last_clean":"2024-02-12T16:57:17.170046+0000","last_became_active":"2024-02-12T16:56:04.869971+0000","last_became_peered":"2024-02-12T16:56:04.869971+0000","last_unstale":"2024-02-12T16:57:17.170046+0000","last_undegraded":"2024-02-12T16:57:17.170046+0000","last_fullsized":"2024-02-12T16:57:17.170046+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"189'2877","reported_seq":2898,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:13.501398+0000","last_change":"2024-02-12T16:56:36.788521+0000","last_active":"2024-02-12T16:57:13.501398+0000","last_peered":"2024-02-12T16:57:13.501398+0000","last_clean":"2024-02-12T16:57:13.501398+0000","last_became_active":"2024-02-12T16:56:36.788231+0000","last_became_peered":"2024-02-12T16:56:36.788231+0000","last_unstale":"2024-02-12T16:57:13.501398+0000","last_undegraded":"2024-02-12T16:57:13.501398+0000","last_fullsized":"2024-02-12T16:57:13.501398+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":2877,"ondisk_log_size":2877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":188547072,"num_objects":2877,"num_object_clones":0,"num_object_copies":8631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2877,"num_write_kb":184128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":153,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.178196+0000","last_change":"2024-02-12T16:54:40.645951+0000","last_active":"2024-02-12T16:57:17.178196+0000","last_peered":"2024-02-12T16:57:17.178196+0000","last_clean":"2024-02-12T16:57:17.178196+0000","last_became_active":"2024-02-12T16:54:40.645556+0000","last_became_peered":"2024-02-12T16:54:40.645556+0000","last_unstale":"2024-02-12T16:57:17.178196+0000","last_undegraded":"2024-02-12T16:57:17.178196+0000","last_fullsized":"2024-02-12T16:57:17.178196+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.128703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.128703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":153,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.168794+0000","last_change":"2024-02-12T16:53:58.524837+0000","last_active":"2024-02-12T16:57:17.168794+0000","last_peered":"2024-02-12T16:57:17.168794+0000","last_clean":"2024-02-12T16:57:17.168794+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:57:17.168794+0000","last_undegraded":"2024-02-12T16:57:17.168794+0000","last_fullsized":"2024-02-12T16:57:17.168794+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.524736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":157,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169622+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T16:57:17.169622+0000","last_peered":"2024-02-12T16:57:17.169622+0000","last_clean":"2024-02-12T16:57:17.169622+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:57:17.169622+0000","last_undegraded":"2024-02-12T16:57:17.169622+0000","last_fullsized":"2024-02-12T16:57:17.169622+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":219,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.173112+0000","last_change":"2024-02-12T16:56:41.290320+0000","last_active":"2024-02-12T16:57:17.173112+0000","last_peered":"2024-02-12T16:57:17.173112+0000","last_clean":"2024-02-12T16:57:17.173112+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:57:17.173112+0000","last_undegraded":"2024-02-12T16:57:17.173112+0000","last_fullsized":"2024-02-12T16:57:17.173112+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:41.290236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.290236+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"189'2848","reported_seq":2869,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-02-12T16:57:13.482985+0000","last_change":"2024-02-12T16:56:36.446663+0000","last_active":"2024-02-12T16:57:13.482985+0000","last_peered":"2024-02-12T16:57:13.482985+0000","last_clean":"2024-02-12T16:57:13.482985+0000","last_became_active":"2024-02-12T16:56:36.446299+0000","last_became_peered":"2024-02-12T16:56:36.446299+0000","last_unstale":"2024-02-12T16:57:13.482985+0000","last_undegraded":"2024-02-12T16:57:13.482985+0000","last_fullsized":"2024-02-12T16:57:13.482985+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":2848,"ondisk_log_size":2848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":186646528,"num_objects":2848,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":182272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":142,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.197148+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T16:57:02.197148+0000","last_peered":"2024-02-12T16:57:02.197148+0000","last_clean":"2024-02-12T16:57:02.197148+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:57:02.197148+0000","last_undegraded":"2024-02-12T16:57:02.197148+0000","last_fullsized":"2024-02-12T16:57:02.197148+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":157,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.169461+0000","last_change":"2024-02-12T16:54:40.629432+0000","last_active":"2024-02-12T16:57:17.169461+0000","last_peered":"2024-02-12T16:57:17.169461+0000","last_clean":"2024-02-12T16:57:17.169461+0000","last_became_active":"2024-02-12T16:54:40.629081+0000","last_became_peered":"2024-02-12T16:54:40.629081+0000","last_unstale":"2024-02-12T16:57:17.169461+0000","last_undegraded":"2024-02-12T16:57:17.169461+0000","last_fullsized":"2024-02-12T16:57:17.169461+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:01.285961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:01.285961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":173,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.195299+0000","last_change":"2024-02-12T16:54:41.249353+0000","last_active":"2024-02-12T16:57:02.195299+0000","last_peered":"2024-02-12T16:57:02.195299+0000","last_clean":"2024-02-12T16:57:02.195299+0000","last_became_active":"2024-02-12T16:54:40.639823+0000","last_became_peered":"2024-02-12T16:54:40.639823+0000","last_unstale":"2024-02-12T16:57:02.195299+0000","last_undegraded":"2024-02-12T16:57:02.195299+0000","last_fullsized":"2024-02-12T16:57:02.195299+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":167,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-12T16:57:02.197867+0000","last_change":"2024-02-12T16:56:31.836846+0000","last_active":"2024-02-12T16:57:02.197867+0000","last_peered":"2024-02-12T16:57:02.197867+0000","last_clean":"2024-02-12T16:57:02.197867+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:57:02.197867+0000","last_undegraded":"2024-02-12T16:57:02.197867+0000","last_fullsized":"2024-02-12T16:57:02.197867+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"191'3373","reported_seq":3396,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.322821+0000","last_change":"2024-02-12T16:56:36.445585+0000","last_active":"2024-02-12T16:57:18.322821+0000","last_peered":"2024-02-12T16:57:18.322821+0000","last_clean":"2024-02-12T16:57:18.322821+0000","last_became_active":"2024-02-12T16:56:36.445291+0000","last_became_peered":"2024-02-12T16:56:36.445291+0000","last_unstale":"2024-02-12T16:57:18.322821+0000","last_undegraded":"2024-02-12T16:57:18.322821+0000","last_fullsized":"2024-02-12T16:57:18.322821+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3373,"ondisk_log_size":3373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":221052928,"num_objects":3373,"num_object_clones":0,"num_object_copies":10119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3373,"num_write_kb":215872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":146,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172030+0000","last_change":"2024-02-12T16:54:03.996224+0000","last_active":"2024-02-12T16:57:17.172030+0000","last_peered":"2024-02-12T16:57:17.172030+0000","last_clean":"2024-02-12T16:57:17.172030+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:57:17.172030+0000","last_undegraded":"2024-02-12T16:57:17.172030+0000","last_fullsized":"2024-02-12T16:57:17.172030+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":171,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.168801+0000","last_change":"2024-02-12T16:56:32.717219+0000","last_active":"2024-02-12T16:57:17.168801+0000","last_peered":"2024-02-12T16:57:17.168801+0000","last_clean":"2024-02-12T16:57:17.168801+0000","last_became_active":"2024-02-12T16:54:40.630768+0000","last_became_peered":"2024-02-12T16:54:40.630768+0000","last_unstale":"2024-02-12T16:57:17.168801+0000","last_undegraded":"2024-02-12T16:57:17.168801+0000","last_fullsized":"2024-02-12T16:57:17.168801+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"178'22","reported_seq":187,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.172591+0000","last_change":"2024-02-12T16:56:39.138852+0000","last_active":"2024-02-12T16:57:17.172591+0000","last_peered":"2024-02-12T16:57:17.172591+0000","last_clean":"2024-02-12T16:57:17.172591+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:57:17.172591+0000","last_undegraded":"2024-02-12T16:57:17.172591+0000","last_fullsized":"2024-02-12T16:57:17.172591+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'22","last_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.138758+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":246,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-12T16:57:17.333164+0000","last_change":"2024-02-12T16:52:59.185115+0000","last_active":"2024-02-12T16:57:17.333164+0000","last_peered":"2024-02-12T16:57:17.333164+0000","last_clean":"2024-02-12T16:57:17.333164+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:57:17.333164+0000","last_undegraded":"2024-02-12T16:57:17.333164+0000","last_fullsized":"2024-02-12T16:57:17.333164+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3235184640,"num_objects":49365,"num_object_clones":0,"num_object_copies":148095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49365,"num_write_kb":3159360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9921953792,"data_stored":9921953792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49365,"ondisk_log_size":49365,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":446,"num_read_kb":441,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":429,"ondisk_log_size":429,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234079,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1359284,"kb_used_data":1053868,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92385868,"statfs":{"total":95995035648,"available":94603128832,"internally_reserved":0,"allocated":1079160832,"data_stored":1078667401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.788,"15min":0.91200000000000003},"min":{"1min":0.25600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":3.2469999999999999,"5min":36.439,"15min":36.439},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.6200000000000001,"15min":0.88300000000000001},"min":{"1min":0.28599999999999998,"5min":0.218,"15min":0.218},"max":{"1min":4.0910000000000002,"5min":36.490000000000002,"15min":36.490000000000002},"last":0.45100000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":1.069,"15min":0.71499999999999997},"min":{"1min":0.29799999999999999,"5min":0.221,"15min":0.221},"max":{"1min":2.2839999999999998,"5min":12.032,"15min":12.032},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.089,"15min":0.72399999999999998},"min":{"1min":0.38700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":4.1130000000000004,"5min":10.054,"15min":10.054},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.1599999999999999,"15min":0.70999999999999996},"min":{"1min":0.25,"5min":0.245,"15min":0.245},"max":{"1min":4.0759999999999996,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.292,"15min":0.77600000000000002},"min":{"1min":0.33600000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.6589999999999998,"5min":20.338000000000001,"15min":20.338000000000001},"last":0.39800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.77,"15min":0.97499999999999998},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.8150000000000004,"5min":20.433,"15min":20.433},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":1.77,"15min":0.96399999999999997},"min":{"1min":0.42699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.5979999999999999,"5min":20.393000000000001,"15min":20.393000000000001},"last":0.439}]},{"osd":4,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.65200000000000002,"15min":0.54300000000000004},"min":{"1min":0.29199999999999998,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":3.3479999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.54700000000000004},"min":{"1min":0.34799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.6869999999999998,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.33100000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.3390000000000004,"5min":5.3390000000000004,"15min":5.3390000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.58999999999999997,"15min":0.55700000000000005},"min":{"1min":0.33700000000000002,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":0.65800000000000003,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.64000000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.66600000000000004,"15min":0.55900000000000005},"min":{"1min":0.36899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.581,"5min":2.6970000000000001,"15min":2.6970000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.746,"15min":0.625},"min":{"1min":0.34699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.1720000000000002,"5min":3.5920000000000001,"15min":3.5920000000000001},"last":0.52600000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528420,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1347596,"kb_used_data":1046276,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92397556,"statfs":{"total":95995035648,"available":94615097344,"internally_reserved":0,"allocated":1071386624,"data_stored":1070875692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3279999999999998,"5min":1.522,"15min":0.82999999999999996},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":10.949,"5min":10.949,"15min":10.949},"last":4.5650000000000004},{"interface":"front","average":{"1min":2.794,"5min":1.5680000000000001,"15min":0.84399999999999997},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.6050000000000004,"5min":10.56,"15min":10.56},"last":0.317}]},{"osd":1,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.4550000000000001,"15min":0.83199999999999996},"min":{"1min":0.252,"5min":0.251,"15min":0.251},"max":{"1min":3.7000000000000002,"5min":15.987,"15min":15.987},"last":0.80100000000000005},{"interface":"front","average":{"1min":2.3319999999999999,"5min":1.5740000000000001,"15min":0.88800000000000001},"min":{"1min":0.34200000000000003,"5min":0.25,"15min":0.25},"max":{"1min":5.9770000000000003,"5min":15.997,"15min":15.997},"last":0.78900000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":1.357,"15min":0.78900000000000003},"min":{"1min":0.22900000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":2.3090000000000002,"5min":1.333,"15min":0.78400000000000003},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.319,"5min":6.3680000000000003,"15min":6.3680000000000003},"last":0.28999999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.3280000000000001,"15min":0.77500000000000002},"min":{"1min":0.29199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.6900000000000004,"5min":11.212,"15min":11.212},"last":0.40400000000000003},{"interface":"front","average":{"1min":2.1059999999999999,"5min":1.6719999999999999,"15min":0.90500000000000003},"min":{"1min":0.32900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":6.0380000000000003,"5min":10.875999999999999,"15min":10.875999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.61399999999999999,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.217},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.353,"5min":0.68300000000000005,"15min":0.495},"min":{"1min":0.26600000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"last":0.32600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.624,"15min":0.48799999999999999},"min":{"1min":0.309,"5min":0.245,"15min":0.245},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.371},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.755,"15min":0.54100000000000004},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":3.5859999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.36299999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:56:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":0.83999999999999997,"15min":0.58499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.73799999999999999,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.29299999999999998,"15min":0.27900000000000003},"max":{"1min":2.7050000000000001,"5min":3.9049999999999998,"15min":3.9049999999999998},"last":0.39300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550140,"kb_used_data":1240628,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92195012,"statfs":{"total":95995035648,"available":94407692288,"internally_reserved":0,"allocated":1270403072,"data_stored":1269896004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4660000000000002,"5min":2.6760000000000002,"15min":1.121},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":12.592000000000001,"5min":25.585000000000001,"15min":25.585000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.0029999999999999,"5min":2.5299999999999998,"15min":1.085},"min":{"1min":0.219,"5min":0.215,"15min":0.215},"max":{"1min":3.9390000000000001,"5min":26.798999999999999,"15min":26.798999999999999},"last":1.395}]},{"osd":2,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":2.234,"15min":0.996},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.22500000000000001},"max":{"1min":3.274,"5min":11.581,"15min":11.581},"last":1.8819999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":2.5499999999999998,"15min":1.113},"min":{"1min":0.29799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":3.1280000000000001,"5min":26.693999999999999,"15min":26.693999999999999},"last":1.492}]},{"osd":3,"last update":"Mon Feb 12 16:56:14 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.552,"15min":1.155},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.5220000000000002,"5min":26.649999999999999,"15min":26.649999999999999},"last":1.5269999999999999},{"interface":"front","average":{"1min":3.21,"5min":2.2999999999999998,"15min":1.0409999999999999},"min":{"1min":0.314,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":17.821000000000002,"5min":17.821000000000002,"15min":17.821000000000002},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.6970000000000001,"15min":0.93999999999999995},"min":{"1min":0.34399999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.7030000000000003,"5min":8.8249999999999993,"15min":8.8249999999999993},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.1659999999999999,"5min":2.1819999999999999,"15min":1.1259999999999999},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.8879999999999999,"5min":26.748999999999999,"15min":26.748999999999999},"last":1.637}]},{"osd":5,"last update":"Mon Feb 12 16:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.7629999999999999,"15min":0.95899999999999996},"min":{"1min":0.29499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.2469999999999999,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":0.505},{"interface":"front","average":{"1min":0.73799999999999999,"5min":2.2759999999999998,"15min":1.159},"min":{"1min":0.32000000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0299999999999998,"5min":26.722999999999999,"15min":26.722999999999999},"last":1.988}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.9490000000000001,"15min":1.042},"min":{"1min":0.38600000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6509999999999998,"5min":13.834,"15min":13.834},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.2749999999999999,"5min":2.1200000000000001,"15min":1.1299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.5550000000000002,"5min":26.774999999999999,"15min":26.774999999999999},"last":1.613}]},{"osd":7,"last update":"Mon Feb 12 16:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.6879999999999999,"15min":0.999},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":3.3079999999999998,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":2.077},{"interface":"front","average":{"1min":0.79300000000000004,"5min":2,"15min":1.0980000000000001},"min":{"1min":0.371,"5min":0.31,"15min":0.31},"max":{"1min":2.9710000000000001,"5min":8.8829999999999991,"15min":8.8829999999999991},"last":0.65200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1535912,"kb_used_data":1238688,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92209240,"statfs":{"total":95995035648,"available":94422261760,"internally_reserved":0,"allocated":1268416512,"data_stored":1267928286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:56:24 2024","interfaces":[{"interface":"back","average":{"1min":1.895,"5min":2.0169999999999999,"15min":0.89500000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.1859999999999999,"5min":23.510999999999999,"15min":23.510999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":2.8679999999999999,"5min":2.008,"15min":0.88700000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":8.6500000000000004,"5min":10.186999999999999,"15min":10.186999999999999},"last":0.501}]},{"osd":2,"last update":"Mon Feb 12 16:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":2.4700000000000002,"15min":1.089},"min":{"1min":0.23799999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.5730000000000004,"5min":16.521000000000001,"15min":16.521000000000001},"last":3.1269999999999998},{"interface":"front","average":{"1min":1.1910000000000001,"5min":2.0750000000000002,"15min":0.95599999999999996},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":10.340999999999999,"5min":10.340999999999999,"15min":10.340999999999999},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:56:17 2024","interfaces":[{"interface":"back","average":{"1min":2.5329999999999999,"5min":2.5430000000000001,"15min":1.121},"min":{"1min":0.28799999999999998,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.3029999999999999,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.879,"5min":2.52,"15min":1.145},"min":{"1min":0.29799999999999999,"5min":0.28100000000000003,"15min":0.27600000000000002},"max":{"1min":7.2030000000000003,"5min":16.446000000000002,"15min":16.446000000000002},"last":0.94999999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.6919999999999999,"15min":0.94299999999999995},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.8330000000000002,"5min":16.027000000000001,"15min":16.027000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.4399999999999999,"15min":0.86299999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.7619999999999996,"5min":10.141,"15min":10.141},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:57:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.8899999999999999,"15min":1.008},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":10.851000000000001,"5min":10.851000000000001,"15min":10.851000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.806,"15min":0.997},"min":{"1min":0.40600000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":12.364000000000001,"5min":16.138999999999999,"15min":16.138999999999999},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:56:24 2024","interfaces":[{"interface":"back","average":{"1min":2.4319999999999999,"5min":1.704,"15min":0.94899999999999995},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":12.135,"5min":12.135,"15min":12.135},"last":0.41599999999999998},{"interface":"front","average":{"1min":2.6829999999999998,"5min":1.982,"15min":1.0449999999999999},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.618,"5min":21.062000000000001,"15min":21.062000000000001},"last":0.75600000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.6990000000000001,"15min":0.95799999999999996},"min":{"1min":0.432,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.0960000000000001,"5min":16.175000000000001,"15min":16.175000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.286,"5min":1.4259999999999999,"15min":0.873},"min":{"1min":0.35599999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":6.7350000000000003,"5min":10.025,"15min":10.025},"last":0.47299999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1951152,"kb_used_data":1649832,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91794000,"statfs":{"total":95995035648,"available":93997056000,"internally_reserved":0,"allocated":1689427968,"data_stored":1688939319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":2.6150000000000002,"15min":1.1259999999999999},"min":{"1min":0.26000000000000001,"5min":0.19,"15min":0.19},"max":{"1min":11.906000000000001,"5min":15.808,"15min":15.808},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.331,"5min":2.1000000000000001,"15min":0.96999999999999997},"min":{"1min":0.26400000000000001,"5min":0.158,"15min":0.158},"max":{"1min":13.497,"5min":19.893000000000001,"15min":19.893000000000001},"last":4.3680000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":2.6000000000000001,"15min":1.1519999999999999},"min":{"1min":0.27400000000000002,"5min":0.191,"15min":0.191},"max":{"1min":12.173,"5min":17.739000000000001,"15min":17.739000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.0800000000000001,"5min":2.75,"15min":1.1950000000000001},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":6.7130000000000001,"5min":15.766999999999999,"15min":15.766999999999999},"last":4.282}]},{"osd":3,"last update":"Mon Feb 12 16:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":3.351,"15min":1.423},"min":{"1min":0.32400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.6699999999999999,"5min":51.579999999999998,"15min":51.579999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.86799999999999999,"5min":2.5529999999999999,"15min":1.1699999999999999},"min":{"1min":0.30499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.3380000000000001,"5min":48.988999999999997,"15min":48.988999999999997},"last":4.3380000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:56:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.9039999999999999,"15min":1.048},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.3339999999999996,"5min":15.737,"15min":15.737},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.712,"5min":1.544,"15min":0.92100000000000004},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":4.3090000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:56:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":2.214,"15min":1.2150000000000001},"min":{"1min":0.34799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.0410000000000004,"5min":15.699,"15min":15.699},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":2.1579999999999999,"15min":1.115},"min":{"1min":0.46999999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.239000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.39500000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":2.403,"15min":1.2010000000000001},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.31},"max":{"1min":2.9710000000000001,"5min":17.782,"15min":17.782},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.847,"15min":1.083},"min":{"1min":0.501,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.2590000000000003,"5min":13.446999999999999,"15min":13.446999999999999},"last":4.2590000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.7889999999999999,"15min":1.222},"min":{"1min":0.42299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":6.0469999999999997},{"interface":"front","average":{"1min":1.3879999999999999,"5min":2.1429999999999998,"15min":1.1299999999999999},"min":{"1min":0.49299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":12.294,"5min":12.294,"15min":12.294},"last":0.42599999999999999}]}]},{"osd":3,"up_from":0,"seq":111669149810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953582,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1740016,"kb_used_data":1442792,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92005136,"statfs":{"total":95995035648,"available":94213259264,"internally_reserved":0,"allocated":1477419008,"data_stored":1476929916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.5760000000000001,"15min":0.88900000000000001},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.192},"max":{"1min":3.681,"5min":38.179000000000002,"15min":38.179000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.6399999999999999,"15min":0.88900000000000001},"min":{"1min":0.25800000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.9649999999999999,"5min":38.151000000000003,"15min":38.151000000000003},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.266,"15min":0.77100000000000002},"min":{"1min":0.28199999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.4829999999999997,"5min":7.673,"15min":7.673},"last":1.829},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.272,"15min":0.78300000000000003},"min":{"1min":0.34100000000000003,"5min":0.249,"15min":0.249},"max":{"1min":5.6340000000000003,"5min":6.367,"15min":6.367},"last":2.0499999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.3160000000000001,"15min":0.81499999999999995},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.8769999999999998,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3540000000000001,"15min":0.82899999999999996},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.3330000000000002,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":1.25}]},{"osd":3,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":2.2360000000000002,"15min":1.113},"min":{"1min":0.28999999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.7699999999999996,"5min":38.170999999999999,"15min":38.170999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.885,"15min":0.995},"min":{"1min":0.34300000000000003,"5min":0.32400000000000001,"15min":0.32000000000000001},"max":{"1min":4.5469999999999997,"5min":38.311999999999998,"15min":38.311999999999998},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.78700000000000003,"15min":0.497},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5009999999999999,"5min":5.4109999999999996,"15min":5.4109999999999996},"last":2.7010000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.68400000000000005,"15min":0.45800000000000002},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.0070000000000001,"5min":6.7960000000000003,"15min":6.7960000000000003},"last":2.7189999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.8009999999999999,"15min":0.84899999999999998},"min":{"1min":0.30399999999999999,"5min":0.255,"15min":0.255},"max":{"1min":2.4430000000000001,"5min":104.907,"15min":104.907},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.9650000000000001,"15min":0.90900000000000003},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.4560000000000004,"5min":105.395,"15min":105.395},"last":0.436}]},{"osd":7,"last update":"Mon Feb 12 16:56:42 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.67400000000000004,"15min":0.50700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":3.7559999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.77800000000000002,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.931,"5min":5.7329999999999997,"15min":5.7329999999999997},"last":0.44500000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757354,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2170156,"kb_used_data":1864740,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91574996,"statfs":{"total":95995035648,"available":93772795904,"internally_reserved":0,"allocated":1909493760,"data_stored":1909008234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":1.516,"15min":0.85199999999999998},"min":{"1min":0.22500000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.64900000000000002,"5min":36.880000000000003,"15min":36.880000000000003},"last":1.7809999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.782,"15min":0.92300000000000004},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2,"5min":36.954000000000001,"15min":36.954000000000001},"last":1.675}]},{"osd":1,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.6699999999999999,"15min":0.90000000000000002},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":1.756},{"interface":"front","average":{"1min":1.962,"5min":1.849,"15min":0.95699999999999996},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":16.712,"5min":16.712,"15min":16.712},"last":0.92700000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2559999999999998,"5min":1.909,"15min":0.99099999999999999},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":16.940999999999999,"5min":18.218,"15min":18.218},"last":2.1739999999999999},{"interface":"front","average":{"1min":2.2109999999999999,"5min":1.9319999999999999,"15min":0.97999999999999998},"min":{"1min":0.35399999999999998,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":17.103999999999999,"5min":18.327000000000002,"15min":18.327000000000002},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.825,"15min":0.96099999999999997},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.7189999999999999,"5min":18.251000000000001,"15min":18.251000000000001},"last":3.0009999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.714,"15min":0.91700000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.75,"5min":20.263000000000002,"15min":20.263000000000002},"last":1.0069999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.623,"15min":0.46800000000000003},"min":{"1min":0.23999999999999999,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":0.63500000000000001,"5min":4.3710000000000004,"15min":4.3710000000000004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.64700000000000002,"15min":0.48199999999999998},"min":{"1min":0.30099999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.55,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":2.1150000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.55700000000000005},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.65100000000000002,"15min":0.51100000000000001},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.8159999999999998,"5min":2.827,"15min":2.827},"last":2.1459999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.76100000000000001,"15min":0.57299999999999995},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.8079999999999998,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.73899999999999999,"15min":0.57499999999999996},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.98299999999999998,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.90100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1302790144,"data_stored":1302790144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1281818624,"data_stored":1281818624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1724514304,"data_stored":1724514304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1505361920,"data_stored":1505361920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1945370624,"data_stored":1945370624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1071906816,"data_stored":1071906816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1090191360,"data_stored":1090191360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:57:20.796 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T16:57:20.796 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:57:20.797 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.control 2024-02-12T16:57:20.797 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.control by 10 min 8 2024-02-12T16:57:20.797 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:57:20.797 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:20.850 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:20.892 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:20.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:21.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:57:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:20] "GET /metrics HTTP/1.1" 200 175831 "" "Prometheus/2.33.4" 2024-02-12T16:57:21.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:21.084 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:21.210 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:19.701924+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.a scrub starts 2024-02-12T16:57:21.210 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:19.703736+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.a scrub ok 2024-02-12T16:57:21.210 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:19.930004+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:57:21.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:19.955304+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:57:21.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:20.492165+0000 mgr.y (mgr.24481) 296 : cluster [DBG] pgmap v344: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-02-12T16:57:21.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:57:21.400 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:19.701924+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.a scrub starts 2024-02-12T16:57:21.400 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:19.703736+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.a scrub ok 2024-02-12T16:57:21.400 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:19.930004+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:57:21.400 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:19.955304+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:57:21.400 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:20.492165+0000 mgr.y (mgr.24481) 296 : cluster [DBG] pgmap v344: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-02-12T16:57:21.400 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:19.701924+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.a scrub starts 2024-02-12T16:57:21.401 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:19.703736+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.a scrub ok 2024-02-12T16:57:21.401 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:19.930004+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.3 scrub starts 2024-02-12T16:57:21.401 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:19.955304+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.3 scrub ok 2024-02-12T16:57:21.401 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:20.492165+0000 mgr.y (mgr.24481) 296 : cluster [DBG] pgmap v344: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-02-12T16:57:21.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:57:21.449 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:21.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:21.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:21.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 28 15 56935 56920 127.019 85.375 0.0183872 0.00786711 2024-02-12T16:57:21.558 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:21.624 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:21.709 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:21.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:21.818 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:21.841 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:21.862 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:22.017 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:22.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:22.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:22.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:22.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:21.636973+0000 osd.4 (osd.4) 87 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:57:22.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:21.653295+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:57:22.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:21.636973+0000 osd.4 (osd.4) 87 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:57:22.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:21.653295+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:57:22.302 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:22.353 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:22.376 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:21.636973+0000 osd.4 (osd.4) 87 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:57:22.376 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:21.653295+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:57:22.383 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:22.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:22.440 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:22.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 29 16 58192 58176 125.346 78.5 0.0104783 0.00797365 2024-02-12T16:57:22.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:22.544 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:22.681 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:22.685 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:57:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:57:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:22.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:22.787 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:22.837 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:22.947 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:23.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:21.750442+0000 osd.5 (osd.5) 106 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:57:23.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:21.756195+0000 osd.5 (osd.5) 107 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:57:23.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:21.904254+0000 osd.7 (osd.7) 105 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:57:23.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:21.909630+0000 osd.7 (osd.7) 106 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:57:23.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:21.933638+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:23.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:21.933691+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:23.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:22.217934+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:22.319719+0000 osd.1 (osd.1) 138 : cluster [DBG] 6.e scrub starts 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:21.750442+0000 osd.5 (osd.5) 106 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:21.756195+0000 osd.5 (osd.5) 107 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:21.904254+0000 osd.7 (osd.7) 105 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:21.909630+0000 osd.7 (osd.7) 106 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:21.933638+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:21.933691+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:22.217934+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:22.319719+0000 osd.1 (osd.1) 138 : cluster [DBG] 6.e scrub starts 2024-02-12T16:57:23.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:22.329759+0000 osd.1 (osd.1) 139 : cluster [DBG] 6.e scrub ok 2024-02-12T16:57:23.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:22.492561+0000 mgr.y (mgr.24481) 297 : cluster [DBG] pgmap v346: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-02-12T16:57:23.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:22.673633+0000 osd.4 (osd.4) 89 : cluster [DBG] 2.f scrub starts 2024-02-12T16:57:23.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:22.679698+0000 osd.4 (osd.4) 90 : cluster [DBG] 2.f scrub ok 2024-02-12T16:57:23.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 30 16 59557 59541 124.011 85.3125 0.00940596 0.00805785 2024-02-12T16:57:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:21.750442+0000 osd.5 (osd.5) 106 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:57:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:21.756195+0000 osd.5 (osd.5) 107 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:57:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:21.904254+0000 osd.7 (osd.7) 105 : cluster [DBG] 2.0 scrub starts 2024-02-12T16:57:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:21.909630+0000 osd.7 (osd.7) 106 : cluster [DBG] 2.0 scrub ok 2024-02-12T16:57:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:21.933638+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:21.933691+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:22.217934+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-12T16:57:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:22.319719+0000 osd.1 (osd.1) 138 : cluster [DBG] 6.e scrub starts 2024-02-12T16:57:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:22.329759+0000 osd.1 (osd.1) 139 : cluster [DBG] 6.e scrub ok 2024-02-12T16:57:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:22.492561+0000 mgr.y (mgr.24481) 297 : cluster [DBG] pgmap v346: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-02-12T16:57:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:22.673633+0000 osd.4 (osd.4) 89 : cluster [DBG] 2.f scrub starts 2024-02-12T16:57:23.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:22.679698+0000 osd.4 (osd.4) 90 : cluster [DBG] 2.f scrub ok 2024-02-12T16:57:23.508 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:23.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:22.329759+0000 osd.1 (osd.1) 139 : cluster [DBG] 6.e scrub ok 2024-02-12T16:57:23.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:22.492561+0000 mgr.y (mgr.24481) 297 : cluster [DBG] pgmap v346: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-02-12T16:57:23.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:22.673633+0000 osd.4 (osd.4) 89 : cluster [DBG] 2.f scrub starts 2024-02-12T16:57:23.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:22.679698+0000 osd.4 (osd.4) 90 : cluster [DBG] 2.f scrub ok 2024-02-12T16:57:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:23.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:23.616 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:23.667 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:23.829 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:57:23.829 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:57:23.853 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:57:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:57:24.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:22.760299+0000 osd.5 (osd.5) 108 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:57:24.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:22.764978+0000 osd.5 (osd.5) 109 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:57:24.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:23.740345+0000 mgr.y (mgr.24481) 298 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:24.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:23.807437+0000 osd.0 (osd.0) 103 : cluster [DBG] 5.1e scrub starts 2024-02-12T16:57:24.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:23.810649+0000 osd.0 (osd.0) 104 : cluster [DBG] 5.1e scrub ok 2024-02-12T16:57:24.459 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:23.890866+0000 mon.a (mon.0) 992 : audit [DBG] from='client.? v1:172.21.15.33:0/461753632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:24.459 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:22.760299+0000 osd.5 (osd.5) 108 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:57:24.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:22.764978+0000 osd.5 (osd.5) 109 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:57:24.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:23.740345+0000 mgr.y (mgr.24481) 298 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:24.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:23.807437+0000 osd.0 (osd.0) 103 : cluster [DBG] 5.1e scrub starts 2024-02-12T16:57:24.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:23.810649+0000 osd.0 (osd.0) 104 : cluster [DBG] 5.1e scrub ok 2024-02-12T16:57:24.460 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:23.890866+0000 mon.a (mon.0) 992 : audit [DBG] from='client.? v1:172.21.15.33:0/461753632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:24.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 31 16 60830 60814 122.578 79.5625 0.0127517 0.00815338 2024-02-12T16:57:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:22.760299+0000 osd.5 (osd.5) 108 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:57:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:22.764978+0000 osd.5 (osd.5) 109 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:57:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:23.740345+0000 mgr.y (mgr.24481) 298 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:24.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:23.807437+0000 osd.0 (osd.0) 103 : cluster [DBG] 5.1e scrub starts 2024-02-12T16:57:24.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:23.810649+0000 osd.0 (osd.0) 104 : cluster [DBG] 5.1e scrub ok 2024-02-12T16:57:24.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:23.890866+0000 mon.a (mon.0) 992 : audit [DBG] from='client.? v1:172.21.15.33:0/461753632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:24.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:57:24.973 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:57:24.991 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:57:24.992 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:57:24.992 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:57:24.992 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:57:24.992 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:57:24.992 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:57:25.276 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:24.406505+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:57:25.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:24.413368+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:57:25.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:24.495093+0000 mgr.y (mgr.24481) 299 : cluster [DBG] pgmap v347: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-02-12T16:57:25.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 32 16 62141 62125 121.308 81.9375 0.00948184 0.00823881 2024-02-12T16:57:25.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:24.406505+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:57:25.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:24.413368+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:57:25.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:24.495093+0000 mgr.y (mgr.24481) 299 : cluster [DBG] pgmap v347: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-02-12T16:57:25.628 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:25.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:25.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:25.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:25.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:24.406505+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.1b scrub starts 2024-02-12T16:57:25.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:24.413368+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.1b scrub ok 2024-02-12T16:57:25.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:24.495093+0000 mgr.y (mgr.24481) 299 : cluster [DBG] pgmap v347: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-02-12T16:57:25.777 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:25.817 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:25.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:25.924 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:57:25.982 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:26.070 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:26.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:26.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:26.235 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:26.264 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:26.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:26.370 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:26.418 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:26.469 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 33 16 63434 63418 120.081 80.8125 0.00740055 0.00832356 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:24.747926+0000 osd.2 (osd.2) 121 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:24.750240+0000 osd.2 (osd.2) 122 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:24.800250+0000 osd.5 (osd.5) 110 : cluster [DBG] 2.16 scrub starts 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:24.805269+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub ok 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:25.275140+0000 osd.6 (osd.6) 120 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:25.278403+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:25.821252+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:25.835877+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:57:26.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:26.139611+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-12T16:57:26.609 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:24.747926+0000 osd.2 (osd.2) 121 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:24.750240+0000 osd.2 (osd.2) 122 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:24.800250+0000 osd.5 (osd.5) 110 : cluster [DBG] 2.16 scrub starts 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:24.805269+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub ok 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:25.275140+0000 osd.6 (osd.6) 120 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:25.278403+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:25.821252+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:25.835877+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:57:26.613 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:26.139611+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:24.747926+0000 osd.2 (osd.2) 121 : cluster [DBG] 4.1c scrub starts 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:24.750240+0000 osd.2 (osd.2) 122 : cluster [DBG] 4.1c scrub ok 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:24.800250+0000 osd.5 (osd.5) 110 : cluster [DBG] 2.16 scrub starts 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:24.805269+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub ok 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:25.275140+0000 osd.6 (osd.6) 120 : cluster [DBG] 2.17 scrub starts 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:25.278403+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.17 scrub ok 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:25.821252+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:25.835877+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:57:26.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:26.139611+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-12T16:57:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:26.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:26.715 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:26.782 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:26.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:26.932 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:57:26.932 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T16:57:26.932 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T16:57:26.932 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T16:57:26.932 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:57:26.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:27.064 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:27.329 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:27.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:27.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:27.470 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 34 16 64610 64594 118.711 73.5 0.0328473 0.0084184 2024-02-12T16:57:27.494 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:27.575 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:26.262992+0000 osd.6 (osd.6) 122 : cluster [DBG] 4.e scrub starts 2024-02-12T16:57:27.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:26.266518+0000 osd.6 (osd.6) 123 : cluster [DBG] 4.e scrub ok 2024-02-12T16:57:27.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:26.495468+0000 mgr.y (mgr.24481) 300 : cluster [DBG] pgmap v349: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-12T16:57:27.600 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:57:27.614 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:57:27.617 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:27.624 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:26.262992+0000 osd.6 (osd.6) 122 : cluster [DBG] 4.e scrub starts 2024-02-12T16:57:27.624 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:26.266518+0000 osd.6 (osd.6) 123 : cluster [DBG] 4.e scrub ok 2024-02-12T16:57:27.624 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:26.495468+0000 mgr.y (mgr.24481) 300 : cluster [DBG] pgmap v349: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-12T16:57:27.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:27.627 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:26.262992+0000 osd.6 (osd.6) 122 : cluster [DBG] 4.e scrub starts 2024-02-12T16:57:27.627 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:26.266518+0000 osd.6 (osd.6) 123 : cluster [DBG] 4.e scrub ok 2024-02-12T16:57:27.627 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:26.495468+0000 mgr.y (mgr.24481) 300 : cluster [DBG] pgmap v349: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-12T16:57:27.635 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:57:27.650 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:57:27.726 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:27.773 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:27.800 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:27.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:27.907 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:27.966 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:28.035 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:28.049 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:28.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:28.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:28.222 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:28.316 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T16:57:28.470 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 35 16 65720 65704 117.301 69.375 0.0095074 0.0085203 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:26.785962+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:26.790295+0000 osd.5 (osd.5) 113 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:26.855023+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.8 scrub starts 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:26.862071+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.8 scrub ok 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:27.369525+0000 osd.1 (osd.1) 142 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:27.383012+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:27.589871+0000 mgr.y (mgr.24481) 301 : audit [DBG] from='client.25114 v1:172.21.15.33:0/2006330360' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:27.826029+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.33:0/3166047981' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:28.502 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:27.827445+0000 mon.a (mon.0) 994 : audit [INF] from='client.25117 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:26.785962+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:26.790295+0000 osd.5 (osd.5) 113 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:26.855023+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.8 scrub starts 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:26.862071+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.8 scrub ok 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:27.369525+0000 osd.1 (osd.1) 142 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:27.383012+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:27.589871+0000 mgr.y (mgr.24481) 301 : audit [DBG] from='client.25114 v1:172.21.15.33:0/2006330360' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:27.826029+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.33:0/3166047981' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:28.503 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:27.827445+0000 mon.a (mon.0) 994 : audit [INF] from='client.25117 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:28.529 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-02-12T16:57:26.495256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3899397796,"num_objects":59692,"num_object_clones":0,"num_object_copies":179076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59692,"num_whiteouts":0,"num_read":1163,"num_read_kb":915,"num_write":59928,"num_write_kb":3808041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":3321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60012,"ondisk_log_size":60012,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13432868,"kb_used_data":11294956,"kb_used_omap":0,"kb_used_meta":2137856,"kb_avail":642783196,"statfs":{"total":671965249536,"available":658209992704,"internally_reserved":0,"allocated":11566034944,"data_stored":11562491505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2189164544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":664207360,"num_objects":10135,"num_object_clones":0,"num_object_copies":30405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10135,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10135,"num_write_kb":648640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005531"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":71,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188329+0000","last_change":"2024-02-12T16:56:31.966054+0000","last_active":"2024-02-12T16:57:18.188329+0000","last_peered":"2024-02-12T16:57:18.188329+0000","last_clean":"2024-02-12T16:57:18.188329+0000","last_became_active":"2024-02-12T16:54:40.643598+0000","last_became_peered":"2024-02-12T16:54:40.643598+0000","last_unstale":"2024-02-12T16:57:18.188329+0000","last_undegraded":"2024-02-12T16:57:18.188329+0000","last_fullsized":"2024-02-12T16:57:18.188329+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":121,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182354+0000","last_change":"2024-02-12T16:56:43.270830+0000","last_active":"2024-02-12T16:57:18.182354+0000","last_peered":"2024-02-12T16:57:18.182354+0000","last_clean":"2024-02-12T16:57:18.182354+0000","last_became_active":"2024-02-12T16:52:20.563961+0000","last_became_peered":"2024-02-12T16:52:20.563961+0000","last_unstale":"2024-02-12T16:57:18.182354+0000","last_undegraded":"2024-02-12T16:57:18.182354+0000","last_fullsized":"2024-02-12T16:57:18.182354+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:43.270756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:51.823967+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.270756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":156,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180177+0000","last_change":"2024-02-12T16:56:35.853405+0000","last_active":"2024-02-12T16:57:18.180177+0000","last_peered":"2024-02-12T16:57:18.180177+0000","last_clean":"2024-02-12T16:57:18.180177+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:57:18.180177+0000","last_undegraded":"2024-02-12T16:57:18.180177+0000","last_fullsized":"2024-02-12T16:57:18.180177+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.853330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.853330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":124,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178848+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T16:57:18.178848+0000","last_peered":"2024-02-12T16:57:18.178848+0000","last_clean":"2024-02-12T16:57:18.178848+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T16:57:18.178848+0000","last_undegraded":"2024-02-12T16:57:18.178848+0000","last_fullsized":"2024-02-12T16:57:18.178848+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":111,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182705+0000","last_change":"2024-02-12T16:53:44.946159+0000","last_active":"2024-02-12T16:57:18.182705+0000","last_peered":"2024-02-12T16:57:18.182705+0000","last_clean":"2024-02-12T16:57:18.182705+0000","last_became_active":"2024-02-12T16:52:20.561617+0000","last_became_peered":"2024-02-12T16:52:20.561617+0000","last_unstale":"2024-02-12T16:57:18.182705+0000","last_undegraded":"2024-02-12T16:57:18.182705+0000","last_fullsized":"2024-02-12T16:57:18.182705+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170259+0000","last_change":"2024-02-12T16:56:30.146707+0000","last_active":"2024-02-12T16:57:18.170259+0000","last_peered":"2024-02-12T16:57:18.170259+0000","last_clean":"2024-02-12T16:57:18.170259+0000","last_became_active":"2024-02-12T16:49:43.202837+0000","last_became_peered":"2024-02-12T16:49:43.202837+0000","last_unstale":"2024-02-12T16:57:18.170259+0000","last_undegraded":"2024-02-12T16:57:18.170259+0000","last_fullsized":"2024-02-12T16:57:18.170259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:30.146643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.146643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":71,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180249+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T16:57:18.180249+0000","last_peered":"2024-02-12T16:57:18.180249+0000","last_clean":"2024-02-12T16:57:18.180249+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T16:57:18.180249+0000","last_undegraded":"2024-02-12T16:57:18.180249+0000","last_fullsized":"2024-02-12T16:57:18.180249+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":173,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.171974+0000","last_change":"2024-02-12T16:55:33.609901+0000","last_active":"2024-02-12T16:57:18.171974+0000","last_peered":"2024-02-12T16:57:18.171974+0000","last_clean":"2024-02-12T16:57:18.171974+0000","last_became_active":"2024-02-12T16:49:41.178538+0000","last_became_peered":"2024-02-12T16:49:41.178538+0000","last_unstale":"2024-02-12T16:57:18.171974+0000","last_undegraded":"2024-02-12T16:57:18.171974+0000","last_fullsized":"2024-02-12T16:57:18.171974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:55:33.609834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:33.609834+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":161,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182900+0000","last_change":"2024-02-12T16:56:42.175069+0000","last_active":"2024-02-12T16:57:18.182900+0000","last_peered":"2024-02-12T16:57:18.182900+0000","last_clean":"2024-02-12T16:57:18.182900+0000","last_became_active":"2024-02-12T16:54:40.628037+0000","last_became_peered":"2024-02-12T16:54:40.628037+0000","last_unstale":"2024-02-12T16:57:18.182900+0000","last_undegraded":"2024-02-12T16:57:18.182900+0000","last_fullsized":"2024-02-12T16:57:18.182900+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:42.174992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:42.174992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":152,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188769+0000","last_change":"2024-02-12T16:56:41.972274+0000","last_active":"2024-02-12T16:57:18.188769+0000","last_peered":"2024-02-12T16:57:18.188769+0000","last_clean":"2024-02-12T16:57:18.188769+0000","last_became_active":"2024-02-12T16:49:43.205967+0000","last_became_peered":"2024-02-12T16:49:43.205967+0000","last_unstale":"2024-02-12T16:57:18.188769+0000","last_undegraded":"2024-02-12T16:57:18.188769+0000","last_fullsized":"2024-02-12T16:57:18.188769+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.972202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.095924+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.972202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":164,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.191994+0000","last_change":"2024-02-12T16:54:38.784259+0000","last_active":"2024-02-12T16:57:18.191994+0000","last_peered":"2024-02-12T16:57:18.191994+0000","last_clean":"2024-02-12T16:57:18.191994+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:57:18.191994+0000","last_undegraded":"2024-02-12T16:57:18.191994+0000","last_fullsized":"2024-02-12T16:57:18.191994+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":94,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170223+0000","last_change":"2024-02-12T16:54:41.193544+0000","last_active":"2024-02-12T16:57:18.170223+0000","last_peered":"2024-02-12T16:57:18.170223+0000","last_clean":"2024-02-12T16:57:18.170223+0000","last_became_active":"2024-02-12T16:54:40.639106+0000","last_became_peered":"2024-02-12T16:54:40.639106+0000","last_unstale":"2024-02-12T16:57:18.170223+0000","last_undegraded":"2024-02-12T16:57:18.170223+0000","last_fullsized":"2024-02-12T16:57:18.170223+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:53:45.658292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:45.658292+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":147,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178673+0000","last_change":"2024-02-12T16:53:51.204401+0000","last_active":"2024-02-12T16:57:18.178673+0000","last_peered":"2024-02-12T16:57:18.178673+0000","last_clean":"2024-02-12T16:57:18.178673+0000","last_became_active":"2024-02-12T16:49:45.209951+0000","last_became_peered":"2024-02-12T16:49:45.209951+0000","last_unstale":"2024-02-12T16:57:18.178673+0000","last_undegraded":"2024-02-12T16:57:18.178673+0000","last_fullsized":"2024-02-12T16:57:18.178673+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.184303+0000","last_change":"2024-02-12T16:56:24.210155+0000","last_active":"2024-02-12T16:57:18.184303+0000","last_peered":"2024-02-12T16:57:18.184303+0000","last_clean":"2024-02-12T16:57:18.184303+0000","last_became_active":"2024-02-12T16:49:43.198993+0000","last_became_peered":"2024-02-12T16:49:43.198993+0000","last_unstale":"2024-02-12T16:57:18.184303+0000","last_undegraded":"2024-02-12T16:57:18.184303+0000","last_fullsized":"2024-02-12T16:57:18.184303+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:24.210075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:55.322446+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.210075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":156,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.189459+0000","last_change":"2024-02-12T16:56:35.897211+0000","last_active":"2024-02-12T16:57:18.189459+0000","last_peered":"2024-02-12T16:57:18.189459+0000","last_clean":"2024-02-12T16:57:18.189459+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:57:18.189459+0000","last_undegraded":"2024-02-12T16:57:18.189459+0000","last_fullsized":"2024-02-12T16:57:18.189459+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.897088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.897088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":109,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.865955+0000","last_change":"2024-02-12T16:57:18.865914+0000","last_active":"2024-02-12T16:57:18.865955+0000","last_peered":"2024-02-12T16:57:18.865955+0000","last_clean":"2024-02-12T16:57:18.865955+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T16:57:18.865955+0000","last_undegraded":"2024-02-12T16:57:18.865955+0000","last_fullsized":"2024-02-12T16:57:18.865955+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:18.865807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.865807+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":73,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.184492+0000","last_change":"2024-02-12T16:56:23.345846+0000","last_active":"2024-02-12T16:57:18.184492+0000","last_peered":"2024-02-12T16:57:18.184492+0000","last_clean":"2024-02-12T16:57:18.184492+0000","last_became_active":"2024-02-12T16:54:40.628201+0000","last_became_peered":"2024-02-12T16:54:40.628201+0000","last_unstale":"2024-02-12T16:57:18.184492+0000","last_undegraded":"2024-02-12T16:57:18.184492+0000","last_fullsized":"2024-02-12T16:57:18.184492+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":74,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179142+0000","last_change":"2024-02-12T16:56:40.672807+0000","last_active":"2024-02-12T16:57:18.179142+0000","last_peered":"2024-02-12T16:57:18.179142+0000","last_clean":"2024-02-12T16:57:18.179142+0000","last_became_active":"2024-02-12T16:54:40.628744+0000","last_became_peered":"2024-02-12T16:54:40.628744+0000","last_unstale":"2024-02-12T16:57:18.179142+0000","last_undegraded":"2024-02-12T16:57:18.179142+0000","last_fullsized":"2024-02-12T16:57:18.179142+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.672727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.672727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":66,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188628+0000","last_change":"2024-02-12T16:54:40.633472+0000","last_active":"2024-02-12T16:57:18.188628+0000","last_peered":"2024-02-12T16:57:18.188628+0000","last_clean":"2024-02-12T16:57:18.188628+0000","last_became_active":"2024-02-12T16:54:40.633129+0000","last_became_peered":"2024-02-12T16:54:40.633129+0000","last_unstale":"2024-02-12T16:57:18.188628+0000","last_undegraded":"2024-02-12T16:57:18.188628+0000","last_fullsized":"2024-02-12T16:57:18.188628+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:03.947783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:03.947783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":118,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170495+0000","last_change":"2024-02-12T16:56:45.033557+0000","last_active":"2024-02-12T16:57:18.170495+0000","last_peered":"2024-02-12T16:57:18.170495+0000","last_clean":"2024-02-12T16:57:18.170495+0000","last_became_active":"2024-02-12T16:54:40.626676+0000","last_became_peered":"2024-02-12T16:54:40.626676+0000","last_unstale":"2024-02-12T16:57:18.170495+0000","last_undegraded":"2024-02-12T16:57:18.170495+0000","last_fullsized":"2024-02-12T16:57:18.170495+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:56:45.033441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.033441+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":157,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.172800+0000","last_change":"2024-02-12T16:56:18.464298+0000","last_active":"2024-02-12T16:57:18.172800+0000","last_peered":"2024-02-12T16:57:18.172800+0000","last_clean":"2024-02-12T16:57:18.172800+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:57:18.172800+0000","last_undegraded":"2024-02-12T16:57:18.172800+0000","last_fullsized":"2024-02-12T16:57:18.172800+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170327+0000","last_change":"2024-02-12T16:56:32.102101+0000","last_active":"2024-02-12T16:57:18.170327+0000","last_peered":"2024-02-12T16:57:18.170327+0000","last_clean":"2024-02-12T16:57:18.170327+0000","last_became_active":"2024-02-12T16:49:43.198806+0000","last_became_peered":"2024-02-12T16:49:43.198806+0000","last_unstale":"2024-02-12T16:57:18.170327+0000","last_undegraded":"2024-02-12T16:57:18.170327+0000","last_fullsized":"2024-02-12T16:57:18.170327+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.102022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.102022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":162,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.183468+0000","last_change":"2024-02-12T16:56:25.339812+0000","last_active":"2024-02-12T16:57:18.183468+0000","last_peered":"2024-02-12T16:57:18.183468+0000","last_clean":"2024-02-12T16:57:18.183468+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:57:18.183468+0000","last_undegraded":"2024-02-12T16:57:18.183468+0000","last_fullsized":"2024-02-12T16:57:18.183468+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":210,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179293+0000","last_change":"2024-02-12T16:54:40.681231+0000","last_active":"2024-02-12T16:57:18.179293+0000","last_peered":"2024-02-12T16:57:18.179293+0000","last_clean":"2024-02-12T16:57:18.179293+0000","last_became_active":"2024-02-12T16:54:40.630167+0000","last_became_peered":"2024-02-12T16:54:40.630167+0000","last_unstale":"2024-02-12T16:57:18.179293+0000","last_undegraded":"2024-02-12T16:57:18.179293+0000","last_fullsized":"2024-02-12T16:57:18.179293+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:53:48.468547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.468547+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":152,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170197+0000","last_change":"2024-02-12T16:56:31.122644+0000","last_active":"2024-02-12T16:57:18.170197+0000","last_peered":"2024-02-12T16:57:18.170197+0000","last_clean":"2024-02-12T16:57:18.170197+0000","last_became_active":"2024-02-12T16:49:45.218104+0000","last_became_peered":"2024-02-12T16:49:45.218104+0000","last_unstale":"2024-02-12T16:57:18.170197+0000","last_undegraded":"2024-02-12T16:57:18.170197+0000","last_fullsized":"2024-02-12T16:57:18.170197+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":147,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180438+0000","last_change":"2024-02-12T16:53:04.085929+0000","last_active":"2024-02-12T16:57:18.180438+0000","last_peered":"2024-02-12T16:57:18.180438+0000","last_clean":"2024-02-12T16:57:18.180438+0000","last_became_active":"2024-02-12T16:49:43.201933+0000","last_became_peered":"2024-02-12T16:49:43.201933+0000","last_unstale":"2024-02-12T16:57:18.180438+0000","last_undegraded":"2024-02-12T16:57:18.180438+0000","last_fullsized":"2024-02-12T16:57:18.180438+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:04.085854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:46.144010+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.085854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":71,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182359+0000","last_change":"2024-02-12T16:56:33.292072+0000","last_active":"2024-02-12T16:57:18.182359+0000","last_peered":"2024-02-12T16:57:18.182359+0000","last_clean":"2024-02-12T16:57:18.182359+0000","last_became_active":"2024-02-12T16:54:40.640452+0000","last_became_peered":"2024-02-12T16:54:40.640452+0000","last_unstale":"2024-02-12T16:57:18.182359+0000","last_undegraded":"2024-02-12T16:57:18.182359+0000","last_fullsized":"2024-02-12T16:57:18.182359+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":174,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.184321+0000","last_change":"2024-02-12T16:56:27.293762+0000","last_active":"2024-02-12T16:57:18.184321+0000","last_peered":"2024-02-12T16:57:18.184321+0000","last_clean":"2024-02-12T16:57:18.184321+0000","last_became_active":"2024-02-12T16:49:41.183212+0000","last_became_peered":"2024-02-12T16:49:41.183212+0000","last_unstale":"2024-02-12T16:57:18.184321+0000","last_undegraded":"2024-02-12T16:57:18.184321+0000","last_fullsized":"2024-02-12T16:57:18.184321+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":147,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.189760+0000","last_change":"2024-02-12T16:53:04.169987+0000","last_active":"2024-02-12T16:57:18.189760+0000","last_peered":"2024-02-12T16:57:18.189760+0000","last_clean":"2024-02-12T16:57:18.189760+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:57:18.189760+0000","last_undegraded":"2024-02-12T16:57:18.189760+0000","last_fullsized":"2024-02-12T16:57:18.189760+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:53:04.169892+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:53:04.169892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":257,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.234173+0000","last_change":"2024-02-12T16:56:36.211166+0000","last_active":"2024-02-12T16:57:22.234173+0000","last_peered":"2024-02-12T16:57:22.234173+0000","last_clean":"2024-02-12T16:57:22.234173+0000","last_became_active":"2024-02-12T16:49:43.197745+0000","last_became_peered":"2024-02-12T16:49:43.197745+0000","last_unstale":"2024-02-12T16:57:22.234173+0000","last_undegraded":"2024-02-12T16:57:22.234173+0000","last_fullsized":"2024-02-12T16:57:22.234173+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:36.211099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:36.211099+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":153,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.181365+0000","last_change":"2024-02-12T16:52:47.997796+0000","last_active":"2024-02-12T16:57:18.181365+0000","last_peered":"2024-02-12T16:57:18.181365+0000","last_clean":"2024-02-12T16:57:18.181365+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:57:18.181365+0000","last_undegraded":"2024-02-12T16:57:18.181365+0000","last_fullsized":"2024-02-12T16:57:18.181365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:47.997708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:52:47.997708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":190,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180340+0000","last_change":"2024-02-12T16:56:32.825256+0000","last_active":"2024-02-12T16:57:18.180340+0000","last_peered":"2024-02-12T16:57:18.180340+0000","last_clean":"2024-02-12T16:57:18.180340+0000","last_became_active":"2024-02-12T16:54:40.650318+0000","last_became_peered":"2024-02-12T16:54:40.650318+0000","last_unstale":"2024-02-12T16:57:18.180340+0000","last_undegraded":"2024-02-12T16:57:18.180340+0000","last_fullsized":"2024-02-12T16:57:18.180340+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":66,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179735+0000","last_change":"2024-02-12T16:54:40.650920+0000","last_active":"2024-02-12T16:57:18.179735+0000","last_peered":"2024-02-12T16:57:18.179735+0000","last_clean":"2024-02-12T16:57:18.179735+0000","last_became_active":"2024-02-12T16:54:40.650168+0000","last_became_peered":"2024-02-12T16:54:40.650168+0000","last_unstale":"2024-02-12T16:57:18.179735+0000","last_undegraded":"2024-02-12T16:57:18.179735+0000","last_fullsized":"2024-02-12T16:57:18.179735+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.665249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.665249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":147,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188571+0000","last_change":"2024-02-12T16:53:05.141324+0000","last_active":"2024-02-12T16:57:18.188571+0000","last_peered":"2024-02-12T16:57:18.188571+0000","last_clean":"2024-02-12T16:57:18.188571+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:57:18.188571+0000","last_undegraded":"2024-02-12T16:57:18.188571+0000","last_fullsized":"2024-02-12T16:57:18.188571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":159,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.316762+0000","last_change":"2024-02-12T16:57:22.321765+0000","last_active":"2024-02-12T16:57:22.316762+0000","last_peered":"2024-02-12T16:57:22.316762+0000","last_clean":"2024-02-12T16:57:22.316762+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:57:22.316762+0000","last_undegraded":"2024-02-12T16:57:22.316762+0000","last_fullsized":"2024-02-12T16:57:22.316762+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.665378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.665378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":114,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188361+0000","last_change":"2024-02-12T16:56:24.854608+0000","last_active":"2024-02-12T16:57:18.188361+0000","last_peered":"2024-02-12T16:57:18.188361+0000","last_clean":"2024-02-12T16:57:18.188361+0000","last_became_active":"2024-02-12T16:54:40.632501+0000","last_became_peered":"2024-02-12T16:54:40.632501+0000","last_unstale":"2024-02-12T16:57:18.188361+0000","last_undegraded":"2024-02-12T16:57:18.188361+0000","last_fullsized":"2024-02-12T16:57:18.188361+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":145,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188839+0000","last_change":"2024-02-12T16:52:55.241717+0000","last_active":"2024-02-12T16:57:18.188839+0000","last_peered":"2024-02-12T16:57:18.188839+0000","last_clean":"2024-02-12T16:57:18.188839+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:57:18.188839+0000","last_undegraded":"2024-02-12T16:57:18.188839+0000","last_fullsized":"2024-02-12T16:57:18.188839+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.241622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:15.965186+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.241622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":158,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182384+0000","last_change":"2024-02-12T16:56:45.226096+0000","last_active":"2024-02-12T16:57:18.182384+0000","last_peered":"2024-02-12T16:57:18.182384+0000","last_clean":"2024-02-12T16:57:18.182384+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:57:18.182384+0000","last_undegraded":"2024-02-12T16:57:18.182384+0000","last_fullsized":"2024-02-12T16:57:18.182384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.225996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":153,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.186763+0000","last_change":"2024-02-12T16:52:49.986507+0000","last_active":"2024-02-12T16:57:18.186763+0000","last_peered":"2024-02-12T16:57:18.186763+0000","last_clean":"2024-02-12T16:57:18.186763+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:57:18.186763+0000","last_undegraded":"2024-02-12T16:57:18.186763+0000","last_fullsized":"2024-02-12T16:57:18.186763+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:49.986403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:52:49.986403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":90,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.169857+0000","last_change":"2024-02-12T16:54:41.081430+0000","last_active":"2024-02-12T16:57:18.169857+0000","last_peered":"2024-02-12T16:57:18.169857+0000","last_clean":"2024-02-12T16:57:18.169857+0000","last_became_active":"2024-02-12T16:54:40.639547+0000","last_became_peered":"2024-02-12T16:54:40.639547+0000","last_unstale":"2024-02-12T16:57:18.169857+0000","last_undegraded":"2024-02-12T16:57:18.169857+0000","last_fullsized":"2024-02-12T16:57:18.169857+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":71,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179434+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T16:57:18.179434+0000","last_peered":"2024-02-12T16:57:18.179434+0000","last_clean":"2024-02-12T16:57:18.179434+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T16:57:18.179434+0000","last_undegraded":"2024-02-12T16:57:18.179434+0000","last_fullsized":"2024-02-12T16:57:18.179434+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":168,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.171681+0000","last_change":"2024-02-12T16:56:47.004534+0000","last_active":"2024-02-12T16:57:18.171681+0000","last_peered":"2024-02-12T16:57:18.171681+0000","last_clean":"2024-02-12T16:57:18.171681+0000","last_became_active":"2024-02-12T16:54:40.628571+0000","last_became_peered":"2024-02-12T16:54:40.628571+0000","last_unstale":"2024-02-12T16:57:18.171681+0000","last_undegraded":"2024-02-12T16:57:18.171681+0000","last_fullsized":"2024-02-12T16:57:18.171681+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":158,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170404+0000","last_change":"2024-02-12T16:54:01.773110+0000","last_active":"2024-02-12T16:57:18.170404+0000","last_peered":"2024-02-12T16:57:18.170404+0000","last_clean":"2024-02-12T16:57:18.170404+0000","last_became_active":"2024-02-12T16:49:39.101381+0000","last_became_peered":"2024-02-12T16:49:39.101381+0000","last_unstale":"2024-02-12T16:57:18.170404+0000","last_undegraded":"2024-02-12T16:57:18.170404+0000","last_fullsized":"2024-02-12T16:57:18.170404+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":178,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.172466+0000","last_change":"2024-02-12T16:56:34.426891+0000","last_active":"2024-02-12T16:57:18.172466+0000","last_peered":"2024-02-12T16:57:18.172466+0000","last_clean":"2024-02-12T16:57:18.172466+0000","last_became_active":"2024-02-12T16:49:41.179226+0000","last_became_peered":"2024-02-12T16:49:41.179226+0000","last_unstale":"2024-02-12T16:57:18.172466+0000","last_undegraded":"2024-02-12T16:57:18.172466+0000","last_fullsized":"2024-02-12T16:57:18.172466+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:34.426800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:34.426800+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":145,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178200+0000","last_change":"2024-02-12T16:54:40.194031+0000","last_active":"2024-02-12T16:57:18.178200+0000","last_peered":"2024-02-12T16:57:18.178200+0000","last_clean":"2024-02-12T16:57:18.178200+0000","last_became_active":"2024-02-12T16:49:45.210620+0000","last_became_peered":"2024-02-12T16:49:45.210620+0000","last_unstale":"2024-02-12T16:57:18.178200+0000","last_undegraded":"2024-02-12T16:57:18.178200+0000","last_fullsized":"2024-02-12T16:57:18.178200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":87,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179841+0000","last_change":"2024-02-12T16:56:23.899212+0000","last_active":"2024-02-12T16:57:18.179841+0000","last_peered":"2024-02-12T16:57:18.179841+0000","last_clean":"2024-02-12T16:57:18.179841+0000","last_became_active":"2024-02-12T16:54:40.650612+0000","last_became_peered":"2024-02-12T16:54:40.650612+0000","last_unstale":"2024-02-12T16:57:18.179841+0000","last_undegraded":"2024-02-12T16:57:18.179841+0000","last_fullsized":"2024-02-12T16:57:18.179841+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":166,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.183678+0000","last_change":"2024-02-12T16:56:39.248638+0000","last_active":"2024-02-12T16:57:18.183678+0000","last_peered":"2024-02-12T16:57:18.183678+0000","last_clean":"2024-02-12T16:57:18.183678+0000","last_became_active":"2024-02-12T16:54:40.629419+0000","last_became_peered":"2024-02-12T16:54:40.629419+0000","last_unstale":"2024-02-12T16:57:18.183678+0000","last_undegraded":"2024-02-12T16:57:18.183678+0000","last_fullsized":"2024-02-12T16:57:18.183678+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:39.248562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.248562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":176,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.171606+0000","last_change":"2024-02-12T16:54:31.954069+0000","last_active":"2024-02-12T16:57:18.171606+0000","last_peered":"2024-02-12T16:57:18.171606+0000","last_clean":"2024-02-12T16:57:18.171606+0000","last_became_active":"2024-02-12T16:49:41.182330+0000","last_became_peered":"2024-02-12T16:49:41.182330+0000","last_unstale":"2024-02-12T16:57:18.171606+0000","last_undegraded":"2024-02-12T16:57:18.171606+0000","last_fullsized":"2024-02-12T16:57:18.171606+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":116,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.184899+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T16:57:18.184899+0000","last_peered":"2024-02-12T16:57:18.184899+0000","last_clean":"2024-02-12T16:57:18.184899+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T16:57:18.184899+0000","last_undegraded":"2024-02-12T16:57:18.184899+0000","last_fullsized":"2024-02-12T16:57:18.184899+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"138'9","reported_seq":148,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.229743+0000","last_change":"2024-02-12T16:56:20.829720+0000","last_active":"2024-02-12T16:57:22.229743+0000","last_peered":"2024-02-12T16:57:22.229743+0000","last_clean":"2024-02-12T16:57:22.229743+0000","last_became_active":"2024-02-12T16:54:40.644025+0000","last_became_peered":"2024-02-12T16:54:40.644025+0000","last_unstale":"2024-02-12T16:57:22.229743+0000","last_undegraded":"2024-02-12T16:57:22.229743+0000","last_fullsized":"2024-02-12T16:57:22.229743+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":160,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179994+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T16:57:18.179994+0000","last_peered":"2024-02-12T16:57:18.179994+0000","last_clean":"2024-02-12T16:57:18.179994+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T16:57:18.179994+0000","last_undegraded":"2024-02-12T16:57:18.179994+0000","last_fullsized":"2024-02-12T16:57:18.179994+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":186,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.185942+0000","last_change":"2024-02-12T16:54:40.641346+0000","last_active":"2024-02-12T16:57:18.185942+0000","last_peered":"2024-02-12T16:57:18.185942+0000","last_clean":"2024-02-12T16:57:18.185942+0000","last_became_active":"2024-02-12T16:54:40.626031+0000","last_became_peered":"2024-02-12T16:54:40.626031+0000","last_unstale":"2024-02-12T16:57:18.185942+0000","last_undegraded":"2024-02-12T16:57:18.185942+0000","last_fullsized":"2024-02-12T16:57:18.185942+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180056+0000","last_change":"2024-02-12T16:54:40.633046+0000","last_active":"2024-02-12T16:57:18.180056+0000","last_peered":"2024-02-12T16:57:18.180056+0000","last_clean":"2024-02-12T16:57:18.180056+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T16:57:18.180056+0000","last_undegraded":"2024-02-12T16:57:18.180056+0000","last_fullsized":"2024-02-12T16:57:18.180056+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:55.105539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:52:55.105539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":252,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.221810+0000","last_change":"2024-02-12T16:54:00.112924+0000","last_active":"2024-02-12T16:57:22.221810+0000","last_peered":"2024-02-12T16:57:22.221810+0000","last_clean":"2024-02-12T16:57:22.221810+0000","last_became_active":"2024-02-12T16:49:43.197998+0000","last_became_peered":"2024-02-12T16:49:43.197998+0000","last_unstale":"2024-02-12T16:57:22.221810+0000","last_undegraded":"2024-02-12T16:57:22.221810+0000","last_fullsized":"2024-02-12T16:57:22.221810+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":162,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182795+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T16:57:18.182795+0000","last_peered":"2024-02-12T16:57:18.182795+0000","last_clean":"2024-02-12T16:57:18.182795+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T16:57:18.182795+0000","last_undegraded":"2024-02-12T16:57:18.182795+0000","last_fullsized":"2024-02-12T16:57:18.182795+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":76,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.169963+0000","last_change":"2024-02-12T16:54:42.827081+0000","last_active":"2024-02-12T16:57:18.169963+0000","last_peered":"2024-02-12T16:57:18.169963+0000","last_clean":"2024-02-12T16:57:18.169963+0000","last_became_active":"2024-02-12T16:54:40.639018+0000","last_became_peered":"2024-02-12T16:54:40.639018+0000","last_unstale":"2024-02-12T16:57:18.169963+0000","last_undegraded":"2024-02-12T16:57:18.169963+0000","last_fullsized":"2024-02-12T16:57:18.169963+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:03.616717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.616717+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":149,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.191927+0000","last_change":"2024-02-12T16:53:51.532654+0000","last_active":"2024-02-12T16:57:18.191927+0000","last_peered":"2024-02-12T16:57:18.191927+0000","last_clean":"2024-02-12T16:57:18.191927+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:57:18.191927+0000","last_undegraded":"2024-02-12T16:57:18.191927+0000","last_fullsized":"2024-02-12T16:57:18.191927+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":163,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.185860+0000","last_change":"2024-02-12T16:56:37.155376+0000","last_active":"2024-02-12T16:57:18.185860+0000","last_peered":"2024-02-12T16:57:18.185860+0000","last_clean":"2024-02-12T16:57:18.185860+0000","last_became_active":"2024-02-12T16:54:40.644226+0000","last_became_peered":"2024-02-12T16:54:40.644226+0000","last_unstale":"2024-02-12T16:57:18.185860+0000","last_undegraded":"2024-02-12T16:57:18.185860+0000","last_fullsized":"2024-02-12T16:57:18.185860+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.155311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.155311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":157,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182430+0000","last_change":"2024-02-12T16:56:38.218147+0000","last_active":"2024-02-12T16:57:18.182430+0000","last_peered":"2024-02-12T16:57:18.182430+0000","last_clean":"2024-02-12T16:57:18.182430+0000","last_became_active":"2024-02-12T16:49:39.107168+0000","last_became_peered":"2024-02-12T16:49:39.107168+0000","last_unstale":"2024-02-12T16:57:18.182430+0000","last_undegraded":"2024-02-12T16:57:18.182430+0000","last_fullsized":"2024-02-12T16:57:18.182430+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.218031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.218031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":95,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.191052+0000","last_change":"2024-02-12T16:54:40.903364+0000","last_active":"2024-02-12T16:57:18.191052+0000","last_peered":"2024-02-12T16:57:18.191052+0000","last_clean":"2024-02-12T16:57:18.191052+0000","last_became_active":"2024-02-12T16:54:40.640126+0000","last_became_peered":"2024-02-12T16:54:40.640126+0000","last_unstale":"2024-02-12T16:57:18.191052+0000","last_undegraded":"2024-02-12T16:57:18.191052+0000","last_fullsized":"2024-02-12T16:57:18.191052+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":152,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178984+0000","last_change":"2024-02-12T16:56:38.699688+0000","last_active":"2024-02-12T16:57:18.178984+0000","last_peered":"2024-02-12T16:57:18.178984+0000","last_clean":"2024-02-12T16:57:18.178984+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:57:18.178984+0000","last_undegraded":"2024-02-12T16:57:18.178984+0000","last_fullsized":"2024-02-12T16:57:18.178984+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.699602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":66,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.171256+0000","last_change":"2024-02-12T16:54:40.634603+0000","last_active":"2024-02-12T16:57:18.171256+0000","last_peered":"2024-02-12T16:57:18.171256+0000","last_clean":"2024-02-12T16:57:18.171256+0000","last_became_active":"2024-02-12T16:54:40.634261+0000","last_became_peered":"2024-02-12T16:54:40.634261+0000","last_unstale":"2024-02-12T16:57:18.171256+0000","last_undegraded":"2024-02-12T16:57:18.171256+0000","last_fullsized":"2024-02-12T16:57:18.171256+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:41.740892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.014772+0000","last_clean_scrub_stamp":"2024-02-12T16:53:41.740892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":158,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178761+0000","last_change":"2024-02-12T16:56:22.699402+0000","last_active":"2024-02-12T16:57:18.178761+0000","last_peered":"2024-02-12T16:57:18.178761+0000","last_clean":"2024-02-12T16:57:18.178761+0000","last_became_active":"2024-02-12T16:49:39.105630+0000","last_became_peered":"2024-02-12T16:49:39.105630+0000","last_unstale":"2024-02-12T16:57:18.178761+0000","last_undegraded":"2024-02-12T16:57:18.178761+0000","last_fullsized":"2024-02-12T16:57:18.178761+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":188,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180185+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T16:57:18.180185+0000","last_peered":"2024-02-12T16:57:18.180185+0000","last_clean":"2024-02-12T16:57:18.180185+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T16:57:18.180185+0000","last_undegraded":"2024-02-12T16:57:18.180185+0000","last_fullsized":"2024-02-12T16:57:18.180185+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"192'3673","reported_seq":3697,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.347281+0000","last_change":"2024-02-12T16:56:36.789085+0000","last_active":"2024-02-12T16:57:23.347281+0000","last_peered":"2024-02-12T16:57:23.347281+0000","last_clean":"2024-02-12T16:57:23.347281+0000","last_became_active":"2024-02-12T16:56:36.788797+0000","last_became_peered":"2024-02-12T16:56:36.788797+0000","last_unstale":"2024-02-12T16:57:23.347281+0000","last_undegraded":"2024-02-12T16:57:23.347281+0000","last_fullsized":"2024-02-12T16:57:23.347281+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3673,"ondisk_log_size":3673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":240713728,"num_objects":3673,"num_object_clones":0,"num_object_copies":11019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3673,"num_write_kb":235072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":66,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188492+0000","last_change":"2024-02-12T16:54:40.643841+0000","last_active":"2024-02-12T16:57:18.188492+0000","last_peered":"2024-02-12T16:57:18.188492+0000","last_clean":"2024-02-12T16:57:18.188492+0000","last_became_active":"2024-02-12T16:54:40.643529+0000","last_became_peered":"2024-02-12T16:54:40.643529+0000","last_unstale":"2024-02-12T16:57:18.188492+0000","last_undegraded":"2024-02-12T16:57:18.188492+0000","last_fullsized":"2024-02-12T16:57:18.188492+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":183,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.679795+0000","last_change":"2024-02-12T16:57:22.679795+0000","last_active":"2024-02-12T16:57:22.679795+0000","last_peered":"2024-02-12T16:57:22.679795+0000","last_clean":"2024-02-12T16:57:22.679795+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:57:22.679795+0000","last_undegraded":"2024-02-12T16:57:22.679795+0000","last_fullsized":"2024-02-12T16:57:22.679795+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:57:22.679705+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.679705+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":52,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.765081+0000","last_change":"2024-02-12T16:57:22.765081+0000","last_active":"2024-02-12T16:57:22.765081+0000","last_peered":"2024-02-12T16:57:22.765081+0000","last_clean":"2024-02-12T16:57:22.765081+0000","last_became_active":"2024-02-12T16:56:04.871405+0000","last_became_peered":"2024-02-12T16:56:04.871405+0000","last_unstale":"2024-02-12T16:57:22.765081+0000","last_undegraded":"2024-02-12T16:57:22.765081+0000","last_fullsized":"2024-02-12T16:57:22.765081+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:22.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.764984+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178834+0000","last_change":"2024-02-12T16:56:45.647692+0000","last_active":"2024-02-12T16:57:18.178834+0000","last_peered":"2024-02-12T16:57:18.178834+0000","last_clean":"2024-02-12T16:57:18.178834+0000","last_became_active":"2024-02-12T16:49:43.203943+0000","last_became_peered":"2024-02-12T16:49:43.203943+0000","last_unstale":"2024-02-12T16:57:18.178834+0000","last_undegraded":"2024-02-12T16:57:18.178834+0000","last_fullsized":"2024-02-12T16:57:18.178834+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.647627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.647627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"192'3753","reported_seq":3777,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.498010+0000","last_change":"2024-02-12T16:56:36.446431+0000","last_active":"2024-02-12T16:57:23.498010+0000","last_peered":"2024-02-12T16:57:23.498010+0000","last_clean":"2024-02-12T16:57:23.498010+0000","last_became_active":"2024-02-12T16:56:36.446107+0000","last_became_peered":"2024-02-12T16:56:36.446107+0000","last_unstale":"2024-02-12T16:57:23.498010+0000","last_undegraded":"2024-02-12T16:57:23.498010+0000","last_fullsized":"2024-02-12T16:57:23.498010+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3753,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":245956608,"num_objects":3753,"num_object_clones":0,"num_object_copies":11259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3753,"num_write_kb":240192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178285+0000","last_change":"2024-02-12T16:54:40.638105+0000","last_active":"2024-02-12T16:57:18.178285+0000","last_peered":"2024-02-12T16:57:18.178285+0000","last_clean":"2024-02-12T16:57:18.178285+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T16:57:18.178285+0000","last_undegraded":"2024-02-12T16:57:18.178285+0000","last_fullsized":"2024-02-12T16:57:18.178285+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.248970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.248970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":158,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170609+0000","last_change":"2024-02-12T16:56:37.009164+0000","last_active":"2024-02-12T16:57:18.170609+0000","last_peered":"2024-02-12T16:57:18.170609+0000","last_clean":"2024-02-12T16:57:18.170609+0000","last_became_active":"2024-02-12T16:49:39.101273+0000","last_became_peered":"2024-02-12T16:49:39.101273+0000","last_unstale":"2024-02-12T16:57:18.170609+0000","last_undegraded":"2024-02-12T16:57:18.170609+0000","last_fullsized":"2024-02-12T16:57:18.170609+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.009094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.009094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":142,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179742+0000","last_change":"2024-02-12T16:54:39.421299+0000","last_active":"2024-02-12T16:57:18.179742+0000","last_peered":"2024-02-12T16:57:18.179742+0000","last_clean":"2024-02-12T16:57:18.179742+0000","last_became_active":"2024-02-12T16:51:04.110893+0000","last_became_peered":"2024-02-12T16:51:04.110893+0000","last_unstale":"2024-02-12T16:57:18.179742+0000","last_undegraded":"2024-02-12T16:57:18.179742+0000","last_fullsized":"2024-02-12T16:57:18.179742+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:54:39.421176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.421176+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":248,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.220710+0000","last_change":"2024-02-12T16:53:59.173126+0000","last_active":"2024-02-12T16:57:22.220710+0000","last_peered":"2024-02-12T16:57:22.220710+0000","last_clean":"2024-02-12T16:57:22.220710+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:57:22.220710+0000","last_undegraded":"2024-02-12T16:57:22.220710+0000","last_fullsized":"2024-02-12T16:57:22.220710+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"192'3796","reported_seq":3820,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.340468+0000","last_change":"2024-02-12T16:56:36.788519+0000","last_active":"2024-02-12T16:57:23.340468+0000","last_peered":"2024-02-12T16:57:23.340468+0000","last_clean":"2024-02-12T16:57:23.340468+0000","last_became_active":"2024-02-12T16:56:36.788101+0000","last_became_peered":"2024-02-12T16:56:36.788101+0000","last_unstale":"2024-02-12T16:57:23.340468+0000","last_undegraded":"2024-02-12T16:57:23.340468+0000","last_fullsized":"2024-02-12T16:57:23.340468+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3796,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":248774656,"num_objects":3796,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3796,"num_write_kb":242944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":75,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:21.653429+0000","last_change":"2024-02-12T16:57:21.653397+0000","last_active":"2024-02-12T16:57:21.653429+0000","last_peered":"2024-02-12T16:57:21.653429+0000","last_clean":"2024-02-12T16:57:21.653429+0000","last_became_active":"2024-02-12T16:54:40.625901+0000","last_became_peered":"2024-02-12T16:54:40.625901+0000","last_unstale":"2024-02-12T16:57:21.653429+0000","last_undegraded":"2024-02-12T16:57:21.653429+0000","last_fullsized":"2024-02-12T16:57:21.653429+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.653301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.653301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":158,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182859+0000","last_change":"2024-02-12T16:56:25.227663+0000","last_active":"2024-02-12T16:57:18.182859+0000","last_peered":"2024-02-12T16:57:18.182859+0000","last_clean":"2024-02-12T16:57:18.182859+0000","last_became_active":"2024-02-12T16:49:39.103443+0000","last_became_peered":"2024-02-12T16:49:39.103443+0000","last_unstale":"2024-02-12T16:57:18.182859+0000","last_undegraded":"2024-02-12T16:57:18.182859+0000","last_fullsized":"2024-02-12T16:57:18.182859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"180'39","reported_seq":339,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.217704+0000","last_change":"2024-02-12T16:56:43.083756+0000","last_active":"2024-02-12T16:57:22.217704+0000","last_peered":"2024-02-12T16:57:22.217704+0000","last_clean":"2024-02-12T16:57:22.217704+0000","last_became_active":"2024-02-12T16:54:40.639565+0000","last_became_peered":"2024-02-12T16:54:40.639565+0000","last_unstale":"2024-02-12T16:57:22.217704+0000","last_undegraded":"2024-02-12T16:57:22.217704+0000","last_fullsized":"2024-02-12T16:57:22.217704+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'39","last_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.083656+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":152,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180104+0000","last_change":"2024-02-12T16:56:46.719822+0000","last_active":"2024-02-12T16:57:18.180104+0000","last_peered":"2024-02-12T16:57:18.180104+0000","last_clean":"2024-02-12T16:57:18.180104+0000","last_became_active":"2024-02-12T16:49:43.202058+0000","last_became_peered":"2024-02-12T16:49:43.202058+0000","last_unstale":"2024-02-12T16:57:18.180104+0000","last_undegraded":"2024-02-12T16:57:18.180104+0000","last_fullsized":"2024-02-12T16:57:18.180104+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:46.719706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:46.719706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"192'3632","reported_seq":3656,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.315993+0000","last_change":"2024-02-12T16:56:36.443593+0000","last_active":"2024-02-12T16:57:23.315993+0000","last_peered":"2024-02-12T16:57:23.315993+0000","last_clean":"2024-02-12T16:57:23.315993+0000","last_became_active":"2024-02-12T16:56:36.443309+0000","last_became_peered":"2024-02-12T16:56:36.443309+0000","last_unstale":"2024-02-12T16:57:23.315993+0000","last_undegraded":"2024-02-12T16:57:23.315993+0000","last_fullsized":"2024-02-12T16:57:23.315993+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3632,"ondisk_log_size":3632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":238026752,"num_objects":3632,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3632,"num_write_kb":232448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":156,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.192115+0000","last_change":"2024-02-12T16:54:40.640502+0000","last_active":"2024-02-12T16:57:18.192115+0000","last_peered":"2024-02-12T16:57:18.192115+0000","last_clean":"2024-02-12T16:57:18.192115+0000","last_became_active":"2024-02-12T16:54:40.640105+0000","last_became_peered":"2024-02-12T16:54:40.640105+0000","last_unstale":"2024-02-12T16:57:18.192115+0000","last_undegraded":"2024-02-12T16:57:18.192115+0000","last_fullsized":"2024-02-12T16:57:18.192115+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":617,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.217464+0000","last_change":"2024-02-12T16:54:30.972535+0000","last_active":"2024-02-12T16:57:22.217464+0000","last_peered":"2024-02-12T16:57:22.217464+0000","last_clean":"2024-02-12T16:57:22.217464+0000","last_became_active":"2024-02-12T16:49:39.101048+0000","last_became_peered":"2024-02-12T16:49:39.101048+0000","last_unstale":"2024-02-12T16:57:22.217464+0000","last_undegraded":"2024-02-12T16:57:22.217464+0000","last_fullsized":"2024-02-12T16:57:22.217464+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":456,"num_read_kb":451,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":167,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178977+0000","last_change":"2024-02-12T16:56:20.676702+0000","last_active":"2024-02-12T16:57:18.178977+0000","last_peered":"2024-02-12T16:57:18.178977+0000","last_clean":"2024-02-12T16:57:18.178977+0000","last_became_active":"2024-02-12T16:54:40.632657+0000","last_became_peered":"2024-02-12T16:54:40.632657+0000","last_unstale":"2024-02-12T16:57:18.178977+0000","last_undegraded":"2024-02-12T16:57:18.178977+0000","last_fullsized":"2024-02-12T16:57:18.178977+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":188,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.171040+0000","last_change":"2024-02-12T16:56:28.196061+0000","last_active":"2024-02-12T16:57:18.171040+0000","last_peered":"2024-02-12T16:57:18.171040+0000","last_clean":"2024-02-12T16:57:18.171040+0000","last_became_active":"2024-02-12T16:49:41.183615+0000","last_became_peered":"2024-02-12T16:49:41.183615+0000","last_unstale":"2024-02-12T16:57:18.171040+0000","last_undegraded":"2024-02-12T16:57:18.171040+0000","last_fullsized":"2024-02-12T16:57:18.171040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179574+0000","last_change":"2024-02-12T16:57:17.691096+0000","last_active":"2024-02-12T16:57:18.179574+0000","last_peered":"2024-02-12T16:57:18.179574+0000","last_clean":"2024-02-12T16:57:18.179574+0000","last_became_active":"2024-02-12T16:49:43.198897+0000","last_became_peered":"2024-02-12T16:49:43.198897+0000","last_unstale":"2024-02-12T16:57:18.179574+0000","last_undegraded":"2024-02-12T16:57:18.179574+0000","last_fullsized":"2024-02-12T16:57:18.179574+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"192'3651","reported_seq":3675,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.341151+0000","last_change":"2024-02-12T16:56:36.447717+0000","last_active":"2024-02-12T16:57:23.341151+0000","last_peered":"2024-02-12T16:57:23.341151+0000","last_clean":"2024-02-12T16:57:23.341151+0000","last_became_active":"2024-02-12T16:56:36.447417+0000","last_became_peered":"2024-02-12T16:56:36.447417+0000","last_unstale":"2024-02-12T16:57:23.341151+0000","last_undegraded":"2024-02-12T16:57:23.341151+0000","last_fullsized":"2024-02-12T16:57:23.341151+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3651,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":239271936,"num_objects":3651,"num_object_clones":0,"num_object_copies":10953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":233664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":156,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.186637+0000","last_change":"2024-02-12T16:54:40.628021+0000","last_active":"2024-02-12T16:57:18.186637+0000","last_peered":"2024-02-12T16:57:18.186637+0000","last_clean":"2024-02-12T16:57:18.186637+0000","last_became_active":"2024-02-12T16:54:40.627688+0000","last_became_peered":"2024-02-12T16:54:40.627688+0000","last_unstale":"2024-02-12T16:57:18.186637+0000","last_undegraded":"2024-02-12T16:57:18.186637+0000","last_fullsized":"2024-02-12T16:57:18.186637+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":71,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182949+0000","last_change":"2024-02-12T16:56:45.218102+0000","last_active":"2024-02-12T16:57:18.182949+0000","last_peered":"2024-02-12T16:57:18.182949+0000","last_clean":"2024-02-12T16:57:18.182949+0000","last_became_active":"2024-02-12T16:54:40.628365+0000","last_became_peered":"2024-02-12T16:54:40.628365+0000","last_unstale":"2024-02-12T16:57:18.182949+0000","last_undegraded":"2024-02-12T16:57:18.182949+0000","last_fullsized":"2024-02-12T16:57:18.182949+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":158,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:21.756318+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T16:57:21.756318+0000","last_peered":"2024-02-12T16:57:21.756318+0000","last_clean":"2024-02-12T16:57:21.756318+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:57:21.756318+0000","last_undegraded":"2024-02-12T16:57:21.756318+0000","last_fullsized":"2024-02-12T16:57:21.756318+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":159,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188395+0000","last_change":"2024-02-12T16:54:30.770032+0000","last_active":"2024-02-12T16:57:18.188395+0000","last_peered":"2024-02-12T16:57:18.188395+0000","last_clean":"2024-02-12T16:57:18.188395+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:57:18.188395+0000","last_undegraded":"2024-02-12T16:57:18.188395+0000","last_fullsized":"2024-02-12T16:57:18.188395+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":182,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:19.281030+0000","last_change":"2024-02-12T16:57:19.280994+0000","last_active":"2024-02-12T16:57:19.281030+0000","last_peered":"2024-02-12T16:57:19.281030+0000","last_clean":"2024-02-12T16:57:19.281030+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:57:19.281030+0000","last_undegraded":"2024-02-12T16:57:19.281030+0000","last_fullsized":"2024-02-12T16:57:19.281030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:19.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.280894+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":152,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.189616+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T16:57:18.189616+0000","last_peered":"2024-02-12T16:57:18.189616+0000","last_clean":"2024-02-12T16:57:18.189616+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:57:18.189616+0000","last_undegraded":"2024-02-12T16:57:18.189616+0000","last_fullsized":"2024-02-12T16:57:18.189616+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"192'3698","reported_seq":3722,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.487666+0000","last_change":"2024-02-12T16:56:36.447068+0000","last_active":"2024-02-12T16:57:23.487666+0000","last_peered":"2024-02-12T16:57:23.487666+0000","last_clean":"2024-02-12T16:57:23.487666+0000","last_became_active":"2024-02-12T16:56:36.446744+0000","last_became_peered":"2024-02-12T16:56:36.446744+0000","last_unstale":"2024-02-12T16:57:23.487666+0000","last_undegraded":"2024-02-12T16:57:23.487666+0000","last_fullsized":"2024-02-12T16:57:23.487666+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3698,"ondisk_log_size":3698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":242352128,"num_objects":3698,"num_object_clones":0,"num_object_copies":11094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":236672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":156,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170787+0000","last_change":"2024-02-12T16:54:40.628087+0000","last_active":"2024-02-12T16:57:18.170787+0000","last_peered":"2024-02-12T16:57:18.170787+0000","last_clean":"2024-02-12T16:57:18.170787+0000","last_became_active":"2024-02-12T16:54:40.626511+0000","last_became_peered":"2024-02-12T16:54:40.626511+0000","last_unstale":"2024-02-12T16:57:18.170787+0000","last_undegraded":"2024-02-12T16:57:18.170787+0000","last_fullsized":"2024-02-12T16:57:18.170787+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.780846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.780846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":160,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179407+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T16:57:18.179407+0000","last_peered":"2024-02-12T16:57:18.179407+0000","last_clean":"2024-02-12T16:57:18.179407+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T16:57:18.179407+0000","last_undegraded":"2024-02-12T16:57:18.179407+0000","last_fullsized":"2024-02-12T16:57:18.179407+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":162,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:21.909771+0000","last_change":"2024-02-12T16:57:21.909729+0000","last_active":"2024-02-12T16:57:21.909771+0000","last_peered":"2024-02-12T16:57:21.909771+0000","last_clean":"2024-02-12T16:57:21.909771+0000","last_became_active":"2024-02-12T16:49:39.100132+0000","last_became_peered":"2024-02-12T16:49:39.100132+0000","last_unstale":"2024-02-12T16:57:21.909771+0000","last_undegraded":"2024-02-12T16:57:21.909771+0000","last_fullsized":"2024-02-12T16:57:21.909771+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":167,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180162+0000","last_change":"2024-02-12T16:53:43.229740+0000","last_active":"2024-02-12T16:57:18.180162+0000","last_peered":"2024-02-12T16:57:18.180162+0000","last_clean":"2024-02-12T16:57:18.180162+0000","last_became_active":"2024-02-12T16:49:41.184003+0000","last_became_peered":"2024-02-12T16:49:41.184003+0000","last_unstale":"2024-02-12T16:57:18.180162+0000","last_undegraded":"2024-02-12T16:57:18.180162+0000","last_fullsized":"2024-02-12T16:57:18.180162+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:53:43.229622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:43.229622+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":149,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.186566+0000","last_change":"2024-02-12T16:56:18.162272+0000","last_active":"2024-02-12T16:57:18.186566+0000","last_peered":"2024-02-12T16:57:18.186566+0000","last_clean":"2024-02-12T16:57:18.186566+0000","last_became_active":"2024-02-12T16:49:43.197975+0000","last_became_peered":"2024-02-12T16:49:43.197975+0000","last_unstale":"2024-02-12T16:57:18.186566+0000","last_undegraded":"2024-02-12T16:57:18.186566+0000","last_fullsized":"2024-02-12T16:57:18.186566+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"192'3706","reported_seq":3730,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.437265+0000","last_change":"2024-02-12T16:56:36.443457+0000","last_active":"2024-02-12T16:57:23.437265+0000","last_peered":"2024-02-12T16:57:23.437265+0000","last_clean":"2024-02-12T16:57:23.437265+0000","last_became_active":"2024-02-12T16:56:36.443153+0000","last_became_peered":"2024-02-12T16:56:36.443153+0000","last_unstale":"2024-02-12T16:57:23.437265+0000","last_undegraded":"2024-02-12T16:57:23.437265+0000","last_fullsized":"2024-02-12T16:57:23.437265+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3706,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3706,"num_write_kb":237184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":152,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170334+0000","last_change":"2024-02-12T16:56:41.053659+0000","last_active":"2024-02-12T16:57:18.170334+0000","last_peered":"2024-02-12T16:57:18.170334+0000","last_clean":"2024-02-12T16:57:18.170334+0000","last_became_active":"2024-02-12T16:49:45.222514+0000","last_became_peered":"2024-02-12T16:49:45.222514+0000","last_unstale":"2024-02-12T16:57:18.170334+0000","last_undegraded":"2024-02-12T16:57:18.170334+0000","last_fullsized":"2024-02-12T16:57:18.170334+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.053587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.053587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":156,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.192189+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T16:57:18.192189+0000","last_peered":"2024-02-12T16:57:18.192189+0000","last_clean":"2024-02-12T16:57:18.192189+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:57:18.192189+0000","last_undegraded":"2024-02-12T16:57:18.192189+0000","last_fullsized":"2024-02-12T16:57:18.192189+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":190,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179961+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T16:57:18.179961+0000","last_peered":"2024-02-12T16:57:18.179961+0000","last_clean":"2024-02-12T16:57:18.179961+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:57:18.179961+0000","last_undegraded":"2024-02-12T16:57:18.179961+0000","last_fullsized":"2024-02-12T16:57:18.179961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":158,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.183389+0000","last_change":"2024-02-12T16:57:17.283687+0000","last_active":"2024-02-12T16:57:18.183389+0000","last_peered":"2024-02-12T16:57:18.183389+0000","last_clean":"2024-02-12T16:57:18.183389+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:57:18.183389+0000","last_undegraded":"2024-02-12T16:57:18.183389+0000","last_fullsized":"2024-02-12T16:57:18.183389+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"192'3820","reported_seq":3844,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.499129+0000","last_change":"2024-02-12T16:56:36.789146+0000","last_active":"2024-02-12T16:57:23.499129+0000","last_peered":"2024-02-12T16:57:23.499129+0000","last_clean":"2024-02-12T16:57:23.499129+0000","last_became_active":"2024-02-12T16:56:36.788871+0000","last_became_peered":"2024-02-12T16:56:36.788871+0000","last_unstale":"2024-02-12T16:57:23.499129+0000","last_undegraded":"2024-02-12T16:57:23.499129+0000","last_fullsized":"2024-02-12T16:57:23.499129+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3820,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":250347520,"num_objects":3820,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":244480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":69,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179814+0000","last_change":"2024-02-12T16:54:40.632821+0000","last_active":"2024-02-12T16:57:18.179814+0000","last_peered":"2024-02-12T16:57:18.179814+0000","last_clean":"2024-02-12T16:57:18.179814+0000","last_became_active":"2024-02-12T16:54:40.631423+0000","last_became_peered":"2024-02-12T16:54:40.631423+0000","last_unstale":"2024-02-12T16:57:18.179814+0000","last_undegraded":"2024-02-12T16:57:18.179814+0000","last_fullsized":"2024-02-12T16:57:18.179814+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:02.624865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.624865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":155,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170837+0000","last_change":"2024-02-12T16:53:48.828698+0000","last_active":"2024-02-12T16:57:18.170837+0000","last_peered":"2024-02-12T16:57:18.170837+0000","last_clean":"2024-02-12T16:57:18.170837+0000","last_became_active":"2024-02-12T16:49:39.099753+0000","last_became_peered":"2024-02-12T16:49:39.099753+0000","last_unstale":"2024-02-12T16:57:18.170837+0000","last_undegraded":"2024-02-12T16:57:18.170837+0000","last_fullsized":"2024-02-12T16:57:18.170837+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":133,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188651+0000","last_change":"2024-02-12T16:56:26.889557+0000","last_active":"2024-02-12T16:57:18.188651+0000","last_peered":"2024-02-12T16:57:18.188651+0000","last_clean":"2024-02-12T16:57:18.188651+0000","last_became_active":"2024-02-12T16:54:40.634781+0000","last_became_peered":"2024-02-12T16:54:40.634781+0000","last_unstale":"2024-02-12T16:57:18.188651+0000","last_undegraded":"2024-02-12T16:57:18.188651+0000","last_fullsized":"2024-02-12T16:57:18.188651+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":164,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.172213+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T16:57:18.172213+0000","last_peered":"2024-02-12T16:57:18.172213+0000","last_clean":"2024-02-12T16:57:18.172213+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T16:57:18.172213+0000","last_undegraded":"2024-02-12T16:57:18.172213+0000","last_fullsized":"2024-02-12T16:57:18.172213+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"192'3753","reported_seq":3777,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.327557+0000","last_change":"2024-02-12T16:56:36.788810+0000","last_active":"2024-02-12T16:57:23.327557+0000","last_peered":"2024-02-12T16:57:23.327557+0000","last_clean":"2024-02-12T16:57:23.327557+0000","last_became_active":"2024-02-12T16:56:36.788518+0000","last_became_peered":"2024-02-12T16:56:36.788518+0000","last_unstale":"2024-02-12T16:57:23.327557+0000","last_undegraded":"2024-02-12T16:57:23.327557+0000","last_fullsized":"2024-02-12T16:57:23.327557+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3753,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":245956608,"num_objects":3753,"num_object_clones":0,"num_object_copies":11259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3753,"num_write_kb":240192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":151,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:19.955453+0000","last_change":"2024-02-12T16:57:19.955408+0000","last_active":"2024-02-12T16:57:19.955453+0000","last_peered":"2024-02-12T16:57:19.955453+0000","last_clean":"2024-02-12T16:57:19.955453+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:57:19.955453+0000","last_undegraded":"2024-02-12T16:57:19.955453+0000","last_fullsized":"2024-02-12T16:57:19.955453+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:57:19.955311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.955311+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.184114+0000","last_change":"2024-02-12T16:56:26.337548+0000","last_active":"2024-02-12T16:57:18.184114+0000","last_peered":"2024-02-12T16:57:18.184114+0000","last_clean":"2024-02-12T16:57:18.184114+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:57:18.184114+0000","last_undegraded":"2024-02-12T16:57:18.184114+0000","last_fullsized":"2024-02-12T16:57:18.184114+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":149,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.681627+0000","last_change":"2024-02-12T16:57:18.681588+0000","last_active":"2024-02-12T16:57:18.681627+0000","last_peered":"2024-02-12T16:57:18.681627+0000","last_clean":"2024-02-12T16:57:18.681627+0000","last_became_active":"2024-02-12T16:52:20.563744+0000","last_became_peered":"2024-02-12T16:52:20.563744+0000","last_unstale":"2024-02-12T16:57:18.681627+0000","last_undegraded":"2024-02-12T16:57:18.681627+0000","last_fullsized":"2024-02-12T16:57:18.681627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"138'9","reported_seq":128,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.217023+0000","last_change":"2024-02-12T16:55:28.449149+0000","last_active":"2024-02-12T16:57:22.217023+0000","last_peered":"2024-02-12T16:57:22.217023+0000","last_clean":"2024-02-12T16:57:22.217023+0000","last_became_active":"2024-02-12T16:54:40.626646+0000","last_became_peered":"2024-02-12T16:54:40.626646+0000","last_unstale":"2024-02-12T16:57:22.217023+0000","last_undegraded":"2024-02-12T16:57:22.217023+0000","last_fullsized":"2024-02-12T16:57:22.217023+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"192'3670","reported_seq":3694,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.324398+0000","last_change":"2024-02-12T16:56:36.445962+0000","last_active":"2024-02-12T16:57:23.324398+0000","last_peered":"2024-02-12T16:57:23.324398+0000","last_clean":"2024-02-12T16:57:23.324398+0000","last_became_active":"2024-02-12T16:56:36.445638+0000","last_became_peered":"2024-02-12T16:56:36.445638+0000","last_unstale":"2024-02-12T16:57:23.324398+0000","last_undegraded":"2024-02-12T16:57:23.324398+0000","last_fullsized":"2024-02-12T16:57:23.324398+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3670,"ondisk_log_size":3670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":240517120,"num_objects":3670,"num_object_clones":0,"num_object_copies":11010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3670,"num_write_kb":234880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":66,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.183605+0000","last_change":"2024-02-12T16:54:40.629725+0000","last_active":"2024-02-12T16:57:18.183605+0000","last_peered":"2024-02-12T16:57:18.183605+0000","last_clean":"2024-02-12T16:57:18.183605+0000","last_became_active":"2024-02-12T16:54:40.629348+0000","last_became_peered":"2024-02-12T16:54:40.629348+0000","last_unstale":"2024-02-12T16:57:18.183605+0000","last_undegraded":"2024-02-12T16:57:18.183605+0000","last_fullsized":"2024-02-12T16:57:18.183605+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":149,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.186379+0000","last_change":"2024-02-12T16:52:52.983089+0000","last_active":"2024-02-12T16:57:18.186379+0000","last_peered":"2024-02-12T16:57:18.186379+0000","last_clean":"2024-02-12T16:57:18.186379+0000","last_became_active":"2024-02-12T16:49:43.196420+0000","last_became_peered":"2024-02-12T16:49:43.196420+0000","last_unstale":"2024-02-12T16:57:18.186379+0000","last_undegraded":"2024-02-12T16:57:18.186379+0000","last_fullsized":"2024-02-12T16:57:18.186379+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:52.982996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:52:52.982996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":130,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.178679+0000","last_change":"2024-02-12T16:53:59.566836+0000","last_active":"2024-02-12T16:57:18.178679+0000","last_peered":"2024-02-12T16:57:18.178679+0000","last_clean":"2024-02-12T16:57:18.178679+0000","last_became_active":"2024-02-12T16:51:04.205996+0000","last_became_peered":"2024-02-12T16:51:04.205996+0000","last_unstale":"2024-02-12T16:57:18.178679+0000","last_undegraded":"2024-02-12T16:57:18.178679+0000","last_fullsized":"2024-02-12T16:57:18.178679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:59.566730+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.566730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":168,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.184936+0000","last_change":"2024-02-12T16:56:04.931453+0000","last_active":"2024-02-12T16:57:18.184936+0000","last_peered":"2024-02-12T16:57:18.184936+0000","last_clean":"2024-02-12T16:57:18.184936+0000","last_became_active":"2024-02-12T16:56:04.863064+0000","last_became_peered":"2024-02-12T16:56:04.863064+0000","last_unstale":"2024-02-12T16:57:18.184936+0000","last_undegraded":"2024-02-12T16:57:18.184936+0000","last_fullsized":"2024-02-12T16:57:18.184936+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:54:36.753321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:36.753321+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"192'3713","reported_seq":3737,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.479032+0000","last_change":"2024-02-12T16:56:36.447192+0000","last_active":"2024-02-12T16:57:23.479032+0000","last_peered":"2024-02-12T16:57:23.479032+0000","last_clean":"2024-02-12T16:57:23.479032+0000","last_became_active":"2024-02-12T16:56:36.446926+0000","last_became_peered":"2024-02-12T16:56:36.446926+0000","last_unstale":"2024-02-12T16:57:23.479032+0000","last_undegraded":"2024-02-12T16:57:23.479032+0000","last_fullsized":"2024-02-12T16:57:23.479032+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3713,"ondisk_log_size":3713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":243335168,"num_objects":3713,"num_object_clones":0,"num_object_copies":11139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3713,"num_write_kb":237632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":152,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.186172+0000","last_change":"2024-02-12T16:56:40.152368+0000","last_active":"2024-02-12T16:57:18.186172+0000","last_peered":"2024-02-12T16:57:18.186172+0000","last_clean":"2024-02-12T16:57:18.186172+0000","last_became_active":"2024-02-12T16:49:45.218135+0000","last_became_peered":"2024-02-12T16:49:45.218135+0000","last_unstale":"2024-02-12T16:57:18.186172+0000","last_undegraded":"2024-02-12T16:57:18.186172+0000","last_fullsized":"2024-02-12T16:57:18.186172+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.152265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"98'10","reported_seq":223,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.220918+0000","last_change":"2024-02-12T16:56:32.292792+0000","last_active":"2024-02-12T16:57:22.220918+0000","last_peered":"2024-02-12T16:57:22.220918+0000","last_clean":"2024-02-12T16:57:22.220918+0000","last_became_active":"2024-02-12T16:52:20.561534+0000","last_became_peered":"2024-02-12T16:52:20.561534+0000","last_unstale":"2024-02-12T16:57:22.220918+0000","last_undegraded":"2024-02-12T16:57:22.220918+0000","last_fullsized":"2024-02-12T16:57:22.220918+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":176,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170181+0000","last_change":"2024-02-12T16:56:04.857653+0000","last_active":"2024-02-12T16:57:18.170181+0000","last_peered":"2024-02-12T16:57:18.170181+0000","last_clean":"2024-02-12T16:57:18.170181+0000","last_became_active":"2024-02-12T16:56:04.857412+0000","last_became_peered":"2024-02-12T16:56:04.857412+0000","last_unstale":"2024-02-12T16:57:18.170181+0000","last_undegraded":"2024-02-12T16:57:18.170181+0000","last_fullsized":"2024-02-12T16:57:18.170181+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":213,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.171124+0000","last_change":"2024-02-12T16:54:41.243992+0000","last_active":"2024-02-12T16:57:18.171124+0000","last_peered":"2024-02-12T16:57:18.171124+0000","last_clean":"2024-02-12T16:57:18.171124+0000","last_became_active":"2024-02-12T16:54:40.627029+0000","last_became_peered":"2024-02-12T16:54:40.627029+0000","last_unstale":"2024-02-12T16:57:18.171124+0000","last_undegraded":"2024-02-12T16:57:18.171124+0000","last_fullsized":"2024-02-12T16:57:18.171124+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"192'3725","reported_seq":3749,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.320510+0000","last_change":"2024-02-12T16:56:36.449103+0000","last_active":"2024-02-12T16:57:23.320510+0000","last_peered":"2024-02-12T16:57:23.320510+0000","last_clean":"2024-02-12T16:57:23.320510+0000","last_became_active":"2024-02-12T16:56:36.448827+0000","last_became_peered":"2024-02-12T16:56:36.448827+0000","last_unstale":"2024-02-12T16:57:23.320510+0000","last_undegraded":"2024-02-12T16:57:23.320510+0000","last_fullsized":"2024-02-12T16:57:23.320510+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3725,"ondisk_log_size":3725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":244121600,"num_objects":3725,"num_object_clones":0,"num_object_copies":11175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":238400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":116,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.329875+0000","last_change":"2024-02-12T16:57:22.329875+0000","last_active":"2024-02-12T16:57:22.329875+0000","last_peered":"2024-02-12T16:57:22.329875+0000","last_clean":"2024-02-12T16:57:22.329875+0000","last_became_active":"2024-02-12T16:52:20.560085+0000","last_became_peered":"2024-02-12T16:52:20.560085+0000","last_unstale":"2024-02-12T16:57:22.329875+0000","last_undegraded":"2024-02-12T16:57:22.329875+0000","last_fullsized":"2024-02-12T16:57:22.329875+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":255,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.226203+0000","last_change":"2024-02-12T16:56:26.673555+0000","last_active":"2024-02-12T16:57:22.226203+0000","last_peered":"2024-02-12T16:57:22.226203+0000","last_clean":"2024-02-12T16:57:22.226203+0000","last_became_active":"2024-02-12T16:49:43.198141+0000","last_became_peered":"2024-02-12T16:49:43.198141+0000","last_unstale":"2024-02-12T16:57:22.226203+0000","last_undegraded":"2024-02-12T16:57:22.226203+0000","last_fullsized":"2024-02-12T16:57:22.226203+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":167,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.171525+0000","last_change":"2024-02-12T16:54:40.629536+0000","last_active":"2024-02-12T16:57:18.171525+0000","last_peered":"2024-02-12T16:57:18.171525+0000","last_clean":"2024-02-12T16:57:18.171525+0000","last_became_active":"2024-02-12T16:54:40.629024+0000","last_became_peered":"2024-02-12T16:54:40.629024+0000","last_unstale":"2024-02-12T16:57:18.171525+0000","last_undegraded":"2024-02-12T16:57:18.171525+0000","last_fullsized":"2024-02-12T16:57:18.171525+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":110,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.183752+0000","last_change":"2024-02-12T16:55:46.162945+0000","last_active":"2024-02-12T16:57:18.183752+0000","last_peered":"2024-02-12T16:57:18.183752+0000","last_clean":"2024-02-12T16:57:18.183752+0000","last_became_active":"2024-02-12T16:54:40.629780+0000","last_became_peered":"2024-02-12T16:54:40.629780+0000","last_unstale":"2024-02-12T16:57:18.183752+0000","last_undegraded":"2024-02-12T16:57:18.183752+0000","last_fullsized":"2024-02-12T16:57:18.183752+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"192'3632","reported_seq":3656,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.459117+0000","last_change":"2024-02-12T16:56:36.447081+0000","last_active":"2024-02-12T16:57:23.459117+0000","last_peered":"2024-02-12T16:57:23.459117+0000","last_clean":"2024-02-12T16:57:23.459117+0000","last_became_active":"2024-02-12T16:56:36.446786+0000","last_became_peered":"2024-02-12T16:56:36.446786+0000","last_unstale":"2024-02-12T16:57:23.459117+0000","last_undegraded":"2024-02-12T16:57:23.459117+0000","last_fullsized":"2024-02-12T16:57:23.459117+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3632,"ondisk_log_size":3632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":238026752,"num_objects":3632,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3632,"num_write_kb":232448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":161,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.184825+0000","last_change":"2024-02-12T16:54:40.631210+0000","last_active":"2024-02-12T16:57:18.184825+0000","last_peered":"2024-02-12T16:57:18.184825+0000","last_clean":"2024-02-12T16:57:18.184825+0000","last_became_active":"2024-02-12T16:54:40.630855+0000","last_became_peered":"2024-02-12T16:54:40.630855+0000","last_unstale":"2024-02-12T16:57:18.184825+0000","last_undegraded":"2024-02-12T16:57:18.184825+0000","last_fullsized":"2024-02-12T16:57:18.184825+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.387144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.387144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.172875+0000","last_change":"2024-02-12T16:56:33.053023+0000","last_active":"2024-02-12T16:57:18.172875+0000","last_peered":"2024-02-12T16:57:18.172875+0000","last_clean":"2024-02-12T16:57:18.172875+0000","last_became_active":"2024-02-12T16:49:43.198103+0000","last_became_peered":"2024-02-12T16:49:43.198103+0000","last_unstale":"2024-02-12T16:57:18.172875+0000","last_undegraded":"2024-02-12T16:57:18.172875+0000","last_fullsized":"2024-02-12T16:57:18.172875+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":156,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188412+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T16:57:18.188412+0000","last_peered":"2024-02-12T16:57:18.188412+0000","last_clean":"2024-02-12T16:57:18.188412+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:57:18.188412+0000","last_undegraded":"2024-02-12T16:57:18.188412+0000","last_fullsized":"2024-02-12T16:57:18.188412+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":183,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182511+0000","last_change":"2024-02-12T16:56:30.714551+0000","last_active":"2024-02-12T16:57:18.182511+0000","last_peered":"2024-02-12T16:57:18.182511+0000","last_clean":"2024-02-12T16:57:18.182511+0000","last_became_active":"2024-02-12T16:56:04.869971+0000","last_became_peered":"2024-02-12T16:56:04.869971+0000","last_unstale":"2024-02-12T16:57:18.182511+0000","last_undegraded":"2024-02-12T16:57:18.182511+0000","last_fullsized":"2024-02-12T16:57:18.182511+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"192'3765","reported_seq":3789,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.482479+0000","last_change":"2024-02-12T16:56:36.788521+0000","last_active":"2024-02-12T16:57:23.482479+0000","last_peered":"2024-02-12T16:57:23.482479+0000","last_clean":"2024-02-12T16:57:23.482479+0000","last_became_active":"2024-02-12T16:56:36.788231+0000","last_became_peered":"2024-02-12T16:56:36.788231+0000","last_unstale":"2024-02-12T16:57:23.482479+0000","last_undegraded":"2024-02-12T16:57:23.482479+0000","last_fullsized":"2024-02-12T16:57:23.482479+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3765,"ondisk_log_size":3765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":246743040,"num_objects":3765,"num_object_clones":0,"num_object_copies":11295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":240960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.188241+0000","last_change":"2024-02-12T16:54:40.645951+0000","last_active":"2024-02-12T16:57:18.188241+0000","last_peered":"2024-02-12T16:57:18.188241+0000","last_clean":"2024-02-12T16:57:18.188241+0000","last_became_active":"2024-02-12T16:54:40.645556+0000","last_became_peered":"2024-02-12T16:54:40.645556+0000","last_unstale":"2024-02-12T16:57:18.188241+0000","last_undegraded":"2024-02-12T16:57:18.188241+0000","last_fullsized":"2024-02-12T16:57:18.188241+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:52:48.128703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:52:48.128703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":154,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179488+0000","last_change":"2024-02-12T16:53:58.524837+0000","last_active":"2024-02-12T16:57:18.179488+0000","last_peered":"2024-02-12T16:57:18.179488+0000","last_clean":"2024-02-12T16:57:18.179488+0000","last_became_active":"2024-02-12T16:49:43.207396+0000","last_became_peered":"2024-02-12T16:49:43.207396+0000","last_unstale":"2024-02-12T16:57:18.179488+0000","last_undegraded":"2024-02-12T16:57:18.179488+0000","last_fullsized":"2024-02-12T16:57:18.179488+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:53:58.524736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":158,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179257+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T16:57:18.179257+0000","last_peered":"2024-02-12T16:57:18.179257+0000","last_clean":"2024-02-12T16:57:18.179257+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:57:18.179257+0000","last_undegraded":"2024-02-12T16:57:18.179257+0000","last_fullsized":"2024-02-12T16:57:18.179257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":220,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.183308+0000","last_change":"2024-02-12T16:56:41.290320+0000","last_active":"2024-02-12T16:57:18.183308+0000","last_peered":"2024-02-12T16:57:18.183308+0000","last_clean":"2024-02-12T16:57:18.183308+0000","last_became_active":"2024-02-12T16:51:04.116400+0000","last_became_peered":"2024-02-12T16:51:04.116400+0000","last_unstale":"2024-02-12T16:57:18.183308+0000","last_undegraded":"2024-02-12T16:57:18.183308+0000","last_fullsized":"2024-02-12T16:57:18.183308+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:41.290236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.290236+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"192'3718","reported_seq":3742,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.467616+0000","last_change":"2024-02-12T16:56:36.446663+0000","last_active":"2024-02-12T16:57:23.467616+0000","last_peered":"2024-02-12T16:57:23.467616+0000","last_clean":"2024-02-12T16:57:23.467616+0000","last_became_active":"2024-02-12T16:56:36.446299+0000","last_became_peered":"2024-02-12T16:56:36.446299+0000","last_unstale":"2024-02-12T16:57:23.467616+0000","last_undegraded":"2024-02-12T16:57:23.467616+0000","last_fullsized":"2024-02-12T16:57:23.467616+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3718,"ondisk_log_size":3718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":243662848,"num_objects":3718,"num_object_clones":0,"num_object_copies":11154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":237952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":145,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179573+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T16:57:18.179573+0000","last_peered":"2024-02-12T16:57:18.179573+0000","last_clean":"2024-02-12T16:57:18.179573+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:57:18.179573+0000","last_undegraded":"2024-02-12T16:57:18.179573+0000","last_fullsized":"2024-02-12T16:57:18.179573+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":163,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:19.703913+0000","last_change":"2024-02-12T16:57:19.703875+0000","last_active":"2024-02-12T16:57:19.703913+0000","last_peered":"2024-02-12T16:57:19.703913+0000","last_clean":"2024-02-12T16:57:19.703913+0000","last_became_active":"2024-02-12T16:54:40.629081+0000","last_became_peered":"2024-02-12T16:54:40.629081+0000","last_unstale":"2024-02-12T16:57:19.703913+0000","last_undegraded":"2024-02-12T16:57:19.703913+0000","last_fullsized":"2024-02-12T16:57:19.703913+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:19.703739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.703739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":176,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.170463+0000","last_change":"2024-02-12T16:54:41.249353+0000","last_active":"2024-02-12T16:57:18.170463+0000","last_peered":"2024-02-12T16:57:18.170463+0000","last_clean":"2024-02-12T16:57:18.170463+0000","last_became_active":"2024-02-12T16:54:40.639823+0000","last_became_peered":"2024-02-12T16:54:40.639823+0000","last_unstale":"2024-02-12T16:57:18.170463+0000","last_undegraded":"2024-02-12T16:57:18.170463+0000","last_fullsized":"2024-02-12T16:57:18.170463+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":170,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.180056+0000","last_change":"2024-02-12T16:56:31.836846+0000","last_active":"2024-02-12T16:57:18.180056+0000","last_peered":"2024-02-12T16:57:18.180056+0000","last_clean":"2024-02-12T16:57:18.180056+0000","last_became_active":"2024-02-12T16:49:41.184975+0000","last_became_peered":"2024-02-12T16:49:41.184975+0000","last_unstale":"2024-02-12T16:57:18.180056+0000","last_undegraded":"2024-02-12T16:57:18.180056+0000","last_fullsized":"2024-02-12T16:57:18.180056+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"192'3795","reported_seq":3818,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:23.327663+0000","last_change":"2024-02-12T16:56:36.445585+0000","last_active":"2024-02-12T16:57:23.327663+0000","last_peered":"2024-02-12T16:57:23.327663+0000","last_clean":"2024-02-12T16:57:23.327663+0000","last_became_active":"2024-02-12T16:56:36.445291+0000","last_became_peered":"2024-02-12T16:56:36.445291+0000","last_unstale":"2024-02-12T16:57:23.327663+0000","last_undegraded":"2024-02-12T16:57:23.327663+0000","last_fullsized":"2024-02-12T16:57:23.327663+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3795,"ondisk_log_size":3795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":248709120,"num_objects":3795,"num_object_clones":0,"num_object_copies":11385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":242880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":147,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.182780+0000","last_change":"2024-02-12T16:54:03.996224+0000","last_active":"2024-02-12T16:57:18.182780+0000","last_peered":"2024-02-12T16:57:18.182780+0000","last_clean":"2024-02-12T16:57:18.182780+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:57:18.182780+0000","last_undegraded":"2024-02-12T16:57:18.182780+0000","last_fullsized":"2024-02-12T16:57:18.182780+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":172,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.179475+0000","last_change":"2024-02-12T16:56:32.717219+0000","last_active":"2024-02-12T16:57:18.179475+0000","last_peered":"2024-02-12T16:57:18.179475+0000","last_clean":"2024-02-12T16:57:18.179475+0000","last_became_active":"2024-02-12T16:54:40.630768+0000","last_became_peered":"2024-02-12T16:54:40.630768+0000","last_unstale":"2024-02-12T16:57:18.179475+0000","last_undegraded":"2024-02-12T16:57:18.179475+0000","last_fullsized":"2024-02-12T16:57:18.179475+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"178'22","reported_seq":188,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-02-12T16:57:18.181396+0000","last_change":"2024-02-12T16:56:39.138852+0000","last_active":"2024-02-12T16:57:18.181396+0000","last_peered":"2024-02-12T16:57:18.181396+0000","last_clean":"2024-02-12T16:57:18.181396+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:57:18.181396+0000","last_undegraded":"2024-02-12T16:57:18.181396+0000","last_fullsized":"2024-02-12T16:57:18.181396+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'22","last_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.138758+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":248,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-02-12T16:57:22.228246+0000","last_change":"2024-02-12T16:52:59.185115+0000","last_active":"2024-02-12T16:57:22.228246+0000","last_peered":"2024-02-12T16:57:22.228246+0000","last_clean":"2024-02-12T16:57:22.228246+0000","last_became_active":"2024-02-12T16:49:43.197957+0000","last_became_peered":"2024-02-12T16:49:43.197957+0000","last_unstale":"2024-02-12T16:57:22.228246+0000","last_undegraded":"2024-02-12T16:57:22.228246+0000","last_fullsized":"2024-02-12T16:57:22.228246+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3899392000,"num_objects":59500,"num_object_clones":0,"num_object_copies":178500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59500,"num_write_kb":3808000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11693850624,"data_stored":11693850624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59500,"ondisk_log_size":59500,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":456,"num_read_kb":451,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":429,"ondisk_log_size":429,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234080,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1489804,"kb_used_data":1192580,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92255348,"statfs":{"total":95995035648,"available":94469476352,"internally_reserved":0,"allocated":1221201920,"data_stored":1220697767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.788,"15min":0.91200000000000003},"min":{"1min":0.25600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":3.2469999999999999,"5min":36.439,"15min":36.439},"last":1.4359999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.6200000000000001,"15min":0.88300000000000001},"min":{"1min":0.28599999999999998,"5min":0.218,"15min":0.218},"max":{"1min":4.0910000000000002,"5min":36.490000000000002,"15min":36.490000000000002},"last":0.29499999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":1.069,"15min":0.71499999999999997},"min":{"1min":0.29799999999999999,"5min":0.221,"15min":0.221},"max":{"1min":2.2839999999999998,"5min":12.032,"15min":12.032},"last":2.468},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.089,"15min":0.72399999999999998},"min":{"1min":0.38700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":4.1130000000000004,"5min":10.054,"15min":10.054},"last":0.249}]},{"osd":2,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.1599999999999999,"15min":0.70999999999999996},"min":{"1min":0.25,"5min":0.245,"15min":0.245},"max":{"1min":4.0759999999999996,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":1.4870000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.292,"15min":0.77600000000000002},"min":{"1min":0.33600000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.6589999999999998,"5min":20.338000000000001,"15min":20.338000000000001},"last":2.5259999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.77,"15min":0.97499999999999998},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.8150000000000004,"5min":20.433,"15min":20.433},"last":1.508},{"interface":"front","average":{"1min":0.73299999999999998,"5min":1.77,"15min":0.96399999999999997},"min":{"1min":0.42699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.5979999999999999,"5min":20.393000000000001,"15min":20.393000000000001},"last":2.548}]},{"osd":4,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.65200000000000002,"15min":0.54300000000000004},"min":{"1min":0.29199999999999998,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":3.3479999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":1.546},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.54700000000000004},"min":{"1min":0.34799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.6869999999999998,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":2.5009999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.33100000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.3390000000000004,"5min":5.3390000000000004,"15min":5.3390000000000004},"last":1.468},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.58999999999999997,"15min":0.55700000000000005},"min":{"1min":0.33700000000000002,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":0.65800000000000003,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":2.5670000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.66600000000000004,"15min":0.55900000000000005},"min":{"1min":0.36899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.581,"5min":2.6970000000000001,"15min":2.6970000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.746,"15min":0.625},"min":{"1min":0.34699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.1720000000000002,"5min":3.5920000000000001,"15min":3.5920000000000001},"last":1.5249999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528421,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491224,"kb_used_data":1181712,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92253928,"statfs":{"total":95995035648,"available":94468022272,"internally_reserved":0,"allocated":1210073088,"data_stored":1209556795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:57:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.635,"15min":0.86799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.5650000000000004,"5min":10.949,"15min":10.949},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.6259999999999999,"15min":0.86799999999999999},"min":{"1min":0.24199999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.883,"5min":10.56,"15min":10.56},"last":0.29999999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:57:18 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.556,"15min":0.86699999999999999},"min":{"1min":0.253,"5min":0.251,"15min":0.251},"max":{"1min":6.3860000000000001,"5min":15.987,"15min":15.987},"last":2.3210000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.661,"15min":0.91900000000000004},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.3929999999999998,"5min":15.997,"15min":15.997},"last":1.3100000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.4339999999999999,"15min":0.81699999999999995},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":8.2759999999999998,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.3899999999999999,"15min":0.80400000000000005},"min":{"1min":0.28899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.1269999999999998,"5min":6.3680000000000003,"15min":6.3680000000000003},"last":1.1479999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.407,"15min":0.80200000000000005},"min":{"1min":0.34499999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.4590000000000001,"5min":11.212,"15min":11.212},"last":3.629},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.6950000000000001,"15min":0.91600000000000004},"min":{"1min":0.35599999999999998,"5min":0.249,"15min":0.249},"max":{"1min":1.552,"5min":10.875999999999999,"15min":10.875999999999999},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65800000000000003,"15min":0.48099999999999998},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.217},"max":{"1min":3.226,"5min":3.226,"15min":3.226},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.747,"15min":0.52200000000000002},"min":{"1min":0.30499999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.6619999999999999,"5min":6.0220000000000002,"15min":6.0220000000000002},"last":1.427}]},{"osd":5,"last update":"Mon Feb 12 16:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.66800000000000004,"15min":0.50800000000000001},"min":{"1min":0.34000000000000002,"5min":0.245,"15min":0.245},"max":{"1min":1.726,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":1.4930000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.78500000000000003,"15min":0.55600000000000005},"min":{"1min":0.26700000000000002,"5min":0.247,"15min":0.247},"max":{"1min":1.542,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":2.3980000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:56:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":0.83999999999999997,"15min":0.58499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.73799999999999999,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.29299999999999998,"15min":0.27900000000000003},"max":{"1min":2.7050000000000001,"5min":3.9049999999999998,"15min":3.9049999999999998},"last":0.52200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574974,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1959840,"kb_used_data":1650328,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91785312,"statfs":{"total":95995035648,"available":93988159488,"internally_reserved":0,"allocated":1689935872,"data_stored":1689412671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4660000000000002,"5min":2.6760000000000002,"15min":1.121},"min":{"1min":0.26800000000000002,"5min":0.182,"15min":0.182},"max":{"1min":12.592000000000001,"5min":25.585000000000001,"15min":25.585000000000001},"last":1.24},{"interface":"front","average":{"1min":1.0029999999999999,"5min":2.5299999999999998,"15min":1.085},"min":{"1min":0.219,"5min":0.215,"15min":0.215},"max":{"1min":3.9390000000000001,"5min":26.798999999999999,"15min":26.798999999999999},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":2.234,"15min":0.996},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.22500000000000001},"max":{"1min":3.274,"5min":11.581,"15min":11.581},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":2.5499999999999998,"15min":1.113},"min":{"1min":0.29799999999999999,"5min":0.19,"15min":0.19},"max":{"1min":3.1280000000000001,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.29899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":2.6499999999999999,"15min":1.1839999999999999},"min":{"1min":0.26900000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4689999999999994,"5min":26.649999999999999,"15min":26.649999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.879,"5min":2.391,"15min":1.073},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":8.8499999999999996,"5min":17.821000000000002,"15min":17.821000000000002},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.6970000000000001,"15min":0.93999999999999995},"min":{"1min":0.34399999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.7030000000000003,"5min":8.8249999999999993,"15min":8.8249999999999993},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":2.1819999999999999,"15min":1.1259999999999999},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.8879999999999999,"5min":26.748999999999999,"15min":26.748999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.7629999999999999,"15min":0.95899999999999996},"min":{"1min":0.29499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.2469999999999999,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":2.2759999999999998,"15min":1.159},"min":{"1min":0.32000000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0299999999999998,"5min":26.722999999999999,"15min":26.722999999999999},"last":0.48299999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:56:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.9490000000000001,"15min":1.042},"min":{"1min":0.38600000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6509999999999998,"5min":13.834,"15min":13.834},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.2749999999999999,"5min":2.1200000000000001,"15min":1.1299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.5550000000000002,"5min":26.774999999999999,"15min":26.774999999999999},"last":0.498}]},{"osd":7,"last update":"Mon Feb 12 16:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.6879999999999999,"15min":0.999},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":3.3079999999999998,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":2,"15min":1.0980000000000001},"min":{"1min":0.371,"5min":0.31,"15min":0.31},"max":{"1min":2.9710000000000001,"5min":8.8829999999999991,"15min":8.8829999999999991},"last":0.88300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1949580,"kb_used_data":1648260,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91795572,"statfs":{"total":95995035648,"available":93998665728,"internally_reserved":0,"allocated":1687818240,"data_stored":1687313881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:56:24 2024","interfaces":[{"interface":"back","average":{"1min":1.895,"5min":2.0169999999999999,"15min":0.89500000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":9.1859999999999999,"5min":23.510999999999999,"15min":23.510999999999999},"last":0.622},{"interface":"front","average":{"1min":2.8679999999999999,"5min":2.008,"15min":0.88700000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":8.6500000000000004,"5min":10.186999999999999,"15min":10.186999999999999},"last":0.373}]},{"osd":2,"last update":"Mon Feb 12 16:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":2.4700000000000002,"15min":1.089},"min":{"1min":0.23799999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.5730000000000004,"5min":16.521000000000001,"15min":16.521000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.1910000000000001,"5min":2.0750000000000002,"15min":0.95599999999999996},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":10.340999999999999,"5min":10.340999999999999,"15min":10.340999999999999},"last":5.149}]},{"osd":3,"last update":"Mon Feb 12 16:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":2.6440000000000001,"15min":1.1479999999999999},"min":{"1min":0.28100000000000003,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.823,"5min":21.100000000000001,"15min":21.100000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.03,"5min":2.6509999999999998,"15min":1.1830000000000001},"min":{"1min":0.29299999999999998,"5min":0.28100000000000003,"15min":0.27600000000000002},"max":{"1min":8.7400000000000002,"5min":16.446000000000002,"15min":16.446000000000002},"last":0.68899999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.6919999999999999,"15min":0.94299999999999995},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.8330000000000002,"5min":16.027000000000001,"15min":16.027000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.4399999999999999,"15min":0.86299999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.7619999999999996,"5min":10.141,"15min":10.141},"last":0.437}]},{"osd":5,"last update":"Mon Feb 12 16:57:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.8899999999999999,"15min":1.008},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":10.851000000000001,"5min":10.851000000000001,"15min":10.851000000000001},"last":3.9860000000000002},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.806,"15min":0.997},"min":{"1min":0.40600000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":12.364000000000001,"5min":16.138999999999999,"15min":16.138999999999999},"last":1.8280000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:56:24 2024","interfaces":[{"interface":"back","average":{"1min":2.4319999999999999,"5min":1.704,"15min":0.94899999999999995},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":12.135,"5min":12.135,"15min":12.135},"last":0.40899999999999997},{"interface":"front","average":{"1min":2.6829999999999998,"5min":1.982,"15min":1.0449999999999999},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.618,"5min":21.062000000000001,"15min":21.062000000000001},"last":3.9500000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.6990000000000001,"15min":0.95799999999999996},"min":{"1min":0.432,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.0960000000000001,"5min":16.175000000000001,"15min":16.175000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.4259999999999999,"15min":0.873},"min":{"1min":0.35599999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":6.7350000000000003,"5min":10.025,"15min":10.025},"last":1.204}]}]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2177544,"kb_used_data":1868032,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91567608,"statfs":{"total":95995035648,"available":93765230592,"internally_reserved":0,"allocated":1912864768,"data_stored":1912365397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":2.6150000000000002,"15min":1.1259999999999999},"min":{"1min":0.26000000000000001,"5min":0.19,"15min":0.19},"max":{"1min":11.906000000000001,"5min":15.808,"15min":15.808},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.331,"5min":2.1000000000000001,"15min":0.96999999999999997},"min":{"1min":0.26400000000000001,"5min":0.158,"15min":0.158},"max":{"1min":13.497,"5min":19.893000000000001,"15min":19.893000000000001},"last":4.5910000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":2.6000000000000001,"15min":1.1519999999999999},"min":{"1min":0.27400000000000002,"5min":0.191,"15min":0.191},"max":{"1min":12.173,"5min":17.739000000000001,"15min":17.739000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":2.75,"15min":1.1950000000000001},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":6.7130000000000001,"5min":15.766999999999999,"15min":15.766999999999999},"last":0.29299999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":3.351,"15min":1.423},"min":{"1min":0.32400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.6699999999999999,"5min":51.579999999999998,"15min":51.579999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":2.5529999999999999,"15min":1.1699999999999999},"min":{"1min":0.30499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.3380000000000001,"5min":48.988999999999997,"15min":48.988999999999997},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:56:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.9039999999999999,"15min":1.048},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.3339999999999996,"5min":15.737,"15min":15.737},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.712,"5min":1.544,"15min":0.92100000000000004},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.433}]},{"osd":5,"last update":"Mon Feb 12 16:56:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":2.214,"15min":1.2150000000000001},"min":{"1min":0.34799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.0410000000000004,"5min":15.699,"15min":15.699},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":2.1579999999999999,"15min":1.115},"min":{"1min":0.46999999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.239000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.79200000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":2.403,"15min":1.2010000000000001},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.31},"max":{"1min":2.9710000000000001,"5min":17.782,"15min":17.782},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.135,"5min":1.847,"15min":1.083},"min":{"1min":0.501,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.2590000000000003,"5min":13.446999999999999,"15min":13.446999999999999},"last":4.5590000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:56:51 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.7889999999999999,"15min":1.222},"min":{"1min":0.42299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":0.629},{"interface":"front","average":{"1min":1.3879999999999999,"5min":2.1429999999999998,"15min":1.1299999999999999},"min":{"1min":0.49299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":12.294,"5min":12.294,"15min":12.294},"last":2.375}]}]},{"osd":3,"up_from":0,"seq":111669149811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953583,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1943368,"kb_used_data":1633856,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91801784,"statfs":{"total":95995035648,"available":94005026816,"internally_reserved":0,"allocated":1673068544,"data_stored":1672568730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.5760000000000001,"15min":0.88900000000000001},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.192},"max":{"1min":3.681,"5min":38.179000000000002,"15min":38.179000000000002},"last":0.20499999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.6399999999999999,"15min":0.88900000000000001},"min":{"1min":0.25800000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.9649999999999999,"5min":38.151000000000003,"15min":38.151000000000003},"last":0.48099999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.266,"15min":0.77100000000000002},"min":{"1min":0.28199999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.4829999999999997,"5min":7.673,"15min":7.673},"last":0.218},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.272,"15min":0.78300000000000003},"min":{"1min":0.34100000000000003,"5min":0.249,"15min":0.249},"max":{"1min":5.6340000000000003,"5min":6.367,"15min":6.367},"last":1.726}]},{"osd":2,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.3160000000000001,"15min":0.81499999999999995},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.8769999999999998,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3540000000000001,"15min":0.82899999999999996},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.3330000000000002,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":2.9260000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":2.2360000000000002,"15min":1.113},"min":{"1min":0.28999999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.7699999999999996,"5min":38.170999999999999,"15min":38.170999999999999},"last":2.891},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.885,"15min":0.995},"min":{"1min":0.34300000000000003,"5min":0.32400000000000001,"15min":0.32000000000000001},"max":{"1min":4.5469999999999997,"5min":38.311999999999998,"15min":38.311999999999998},"last":0.69699999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.78700000000000003,"15min":0.497},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5009999999999999,"5min":5.4109999999999996,"15min":5.4109999999999996},"last":0.253},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.68400000000000005,"15min":0.45800000000000002},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.0070000000000001,"5min":6.7960000000000003,"15min":6.7960000000000003},"last":0.40000000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.8009999999999999,"15min":0.84899999999999998},"min":{"1min":0.30399999999999999,"5min":0.255,"15min":0.255},"max":{"1min":2.4430000000000001,"5min":104.907,"15min":104.907},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.9650000000000001,"15min":0.90900000000000003},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.4560000000000004,"5min":105.395,"15min":105.395},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:56:42 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.67400000000000004,"15min":0.50700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":3.7559999999999998},"last":0.373},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.77800000000000002,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.931,"5min":5.7329999999999997,"15min":5.7329999999999997},"last":0.45400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757355,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2421508,"kb_used_data":2120188,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91323644,"statfs":{"total":95995035648,"available":93515411456,"internally_reserved":0,"allocated":2171072512,"data_stored":2170576264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":1.516,"15min":0.85199999999999998},"min":{"1min":0.22500000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.64900000000000002,"5min":36.880000000000003,"15min":36.880000000000003},"last":2.6179999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.782,"15min":0.92300000000000004},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2,"5min":36.954000000000001,"15min":36.954000000000001},"last":4.1639999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.6699999999999999,"15min":0.90000000000000002},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":5.6319999999999997},{"interface":"front","average":{"1min":1.962,"5min":1.849,"15min":0.95699999999999996},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":16.712,"5min":16.712,"15min":16.712},"last":4.1959999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2559999999999998,"5min":1.909,"15min":0.99099999999999999},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":16.940999999999999,"5min":18.218,"15min":18.218},"last":2.6909999999999998},{"interface":"front","average":{"1min":2.2109999999999999,"5min":1.9319999999999999,"15min":0.97999999999999998},"min":{"1min":0.35399999999999998,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":17.103999999999999,"5min":18.327000000000002,"15min":18.327000000000002},"last":2.3839999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.825,"15min":0.96099999999999997},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.7189999999999999,"5min":18.251000000000001,"15min":18.251000000000001},"last":2.141},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.714,"15min":0.91700000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.75,"5min":20.263000000000002,"15min":20.263000000000002},"last":2.0790000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.623,"15min":0.46800000000000003},"min":{"1min":0.23999999999999999,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":0.63500000000000001,"5min":4.3710000000000004,"15min":4.3710000000000004},"last":4.101},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.64700000000000002,"15min":0.48199999999999998},"min":{"1min":0.30099999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.55,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":2.2010000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.91700000000000004,"15min":0.60099999999999998},"min":{"1min":0.36799999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.1319999999999997,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":4.1319999999999997},{"interface":"front","average":{"1min":1.036,"5min":0.73999999999999999,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.512,"5min":3.512,"15min":3.512},"last":2.798}]},{"osd":7,"last update":"Mon Feb 12 16:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.76100000000000001,"15min":0.57299999999999995},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.8079999999999998,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":2.3460000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.73899999999999999,"15min":0.57499999999999996},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.98299999999999998,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":4.2140000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1712324608,"data_stored":1712324608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1695088640,"data_stored":1695088640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1945567232,"data_stored":1945567232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1698889728,"data_stored":1698889728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2200305664,"data_stored":2200305664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1211564032,"data_stored":1211564032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1230110720,"data_stored":1230110720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:57:28.534 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-12T16:57:28.534 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.control pg_num 22 2024-02-12T16:57:28.575 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:26.785962+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.18 scrub starts 2024-02-12T16:57:28.575 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:26.790295+0000 osd.5 (osd.5) 113 : cluster [DBG] 2.18 scrub ok 2024-02-12T16:57:28.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:26.855023+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.8 scrub starts 2024-02-12T16:57:28.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:26.862071+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.8 scrub ok 2024-02-12T16:57:28.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:27.369525+0000 osd.1 (osd.1) 142 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:57:28.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:27.383012+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:57:28.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:27.589871+0000 mgr.y (mgr.24481) 301 : audit [DBG] from='client.25114 v1:172.21.15.33:0/2006330360' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:57:28.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:27.826029+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.33:0/3166047981' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:28.576 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:27.827445+0000 mon.a (mon.0) 994 : audit [INF] from='client.25117 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:29.288 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:29.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:29.334 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:29.343 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:29.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:29.470 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 36 16 66926 66910 116.136 75.375 0.0203346 0.00860543 2024-02-12T16:57:29.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:28.293061+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:57:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:28.293086+0000 mon.a (mon.0) 996 : cluster [INF] Cluster is now healthy 2024-02-12T16:57:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:28.299018+0000 mon.a (mon.0) 997 : audit [INF] from='client.25117 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:28.299092+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-12T16:57:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:28.496207+0000 mgr.y (mgr.24481) 302 : cluster [DBG] pgmap v351: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 815 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-12T16:57:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:28.293061+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:57:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:28.293086+0000 mon.a (mon.0) 996 : cluster [INF] Cluster is now healthy 2024-02-12T16:57:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:28.299018+0000 mon.a (mon.0) 997 : audit [INF] from='client.25117 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:29.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:28.299092+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-12T16:57:29.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:28.496207+0000 mgr.y (mgr.24481) 302 : cluster [DBG] pgmap v351: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 815 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-12T16:57:29.581 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:29.583 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:28.293061+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:57:29.583 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:28.293086+0000 mon.a (mon.0) 996 : cluster [INF] Cluster is now healthy 2024-02-12T16:57:29.583 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:28.299018+0000 mon.a (mon.0) 997 : audit [INF] from='client.25117 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:29.583 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:28.299092+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-12T16:57:29.583 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:28.496207+0000 mgr.y (mgr.24481) 302 : cluster [DBG] pgmap v351: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 815 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-12T16:57:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:29.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:29.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:29.730 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:29.871 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:29.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:29.919 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:57:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T16:57:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T16:57:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T16:57:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:32161 8.b 8:dd6ba873:::benchmark_data_smithi033_105085_object32160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e188)", 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:05.927688+0000", 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.985679357999999, 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0548376, 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32161 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.927688+0000", 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.927688+0000", 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.927685+0000", 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7321000000000002e-05 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.927742+0000", 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7212000000000001e-05 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.927760+0000", 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9470000000000003e-06 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.927768+0000", 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2012999999999998e-05 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.927820+0000", 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014048199999999999 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.927960+0000", 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015721100000000001 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.928117+0000", 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001693307 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.929811+0000", 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036929089999999999 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.933503+0000", 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048943352000000002 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.982447+0000", 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1522e-05 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.982488+0000", 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7035e-05 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.982525+0000", 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0548376 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:32177 8.b 8:d29477c0:::benchmark_data_smithi033_105085_object32176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e188)", 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:05.936938+0000", 2024-02-12T16:57:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.976428696999999, 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052158090999999997, 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32177 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.936938+0000", 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.936938+0000", 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.936936+0000", 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3285000000000002e-05 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.936990+0000", 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3543000000000001e-05 2024-02-12T16:57:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.937003+0000", 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5880000000000004e-06 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.937013+0000", 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9066000000000001e-05 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.937062+0000", 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018047699999999999 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.937242+0000", 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017940400000000001 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.937422+0000", 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00108073 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.938502+0000", 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00036447699999999999 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.938867+0000", 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050164474000000001 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.989031+0000", 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4879000000000003e-05 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.989066+0000", 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0284999999999999e-05 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:05.989097+0000", 2024-02-12T16:57:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052158090999999997 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:32864 8.b 8:d3c88a03:::benchmark_data_smithi033_105085_object32863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:06.307731+0000", 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.605635667000001, 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048562594000000001, 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32864 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.307731+0000", 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.307731+0000", 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.307730+0000", 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0217000000000003e-05 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.307790+0000", 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0180000000000001e-05 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.307800+0000", 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6910000000000002e-06 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.307806+0000", 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0009000000000002e-05 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.307856+0000", 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011590399999999999 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.307972+0000", 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014130300000000001 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.308113+0000", 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014360969999999999 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.309549+0000", 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00034536300000000001 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.309894+0000", 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046314121999999999 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.356209+0000", 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8538000000000002e-05 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.356257+0000", 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6959999999999998e-05 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:06.356294+0000", 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048562594000000001 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:34812 8.2 8:4db00c8d:::benchmark_data_smithi033_105085_object34811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:07.381338+0000", 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.532029520000002, 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048895702999999999, 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34812 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381338+0000", 2024-02-12T16:57:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381338+0000", 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381336+0000", 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365999999999998e-05 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381397+0000", 2024-02-12T16:57:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6159999999999996e-06 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381404+0000", 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6189999999999999e-06 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381410+0000", 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0917999999999998e-05 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381441+0000", 2024-02-12T16:57:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0080999999999995e-05 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381511+0000", 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3512999999999997e-05 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.381604+0000", 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012988699999999999 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.382903+0000", 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014404000000000001 2024-02-12T16:57:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.383047+0000", 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047079650000000001 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.430127+0000", 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2021999999999997e-05 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.430189+0000", 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4419000000000003e-05 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.430233+0000", 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048895702999999999 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:34819 8.2 8:4b295911:::benchmark_data_smithi033_105085_object34818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:07.385546+0000", 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.527820664, 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048702124999999999, 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34819 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385546+0000", 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385546+0000", 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385544+0000", 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8532000000000001e-05 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385603+0000", 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2699e-05 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385615+0000", 2024-02-12T16:57:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.348e-06 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385622+0000", 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1297000000000006e-05 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385683+0000", 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120963 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385804+0000", 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000149343 2024-02-12T16:57:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385953+0000", 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022446179999999999 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.388198+0000", 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000189359 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.388387+0000", 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045777963999999997 2024-02-12T16:57:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.434165+0000", 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4469999999999999e-05 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.434210+0000", 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8843000000000001e-05 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.434249+0000", 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048702124999999999 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:34820 8.2 8:4a530265:::benchmark_data_smithi033_105085_object34819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:07.385639+0000", 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.52772839, 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048750366000000003, 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34820 2024-02-12T16:57:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385639+0000", 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385639+0000", 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385638+0000", 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0898000000000001e-05 2024-02-12T16:57:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385669+0000", 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5540000000000003e-06 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385675+0000", 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7579999999999992e-06 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.385683+0000", 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001517729 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.387201+0000", 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011789 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.387319+0000", 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000105665 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.387424+0000", 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096911499999999999 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.388393+0000", 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000107252 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.388501+0000", 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045802112999999998 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.434303+0000", 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1589999999999999e-05 2024-02-12T16:57:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.434354+0000", 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4721000000000002e-05 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.434389+0000", 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048750366000000003 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:35234 8.b 8:d2c033bf:::benchmark_data_smithi033_105085_object35233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:07.621289+0000", 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.292078283999999, 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.061378212000000001, 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35234 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621289+0000", 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621289+0000", 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999847 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621274+0000", 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1094999999999999e-05 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621315+0000", 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2129999999999997e-06 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621324+0000", 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1009999999999997e-06 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621330+0000", 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6668000000000001e-05 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621377+0000", 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010995800000000001 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621487+0000", 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012074900000000001 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.621608+0000", 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016427119999999999 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.623250+0000", 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011284159999999999 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.624379+0000", 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058202012999999997 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.682581+0000", 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8656999999999998e-05 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.682629+0000", 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7721e-05 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.682667+0000", 2024-02-12T16:57:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.061378212000000001 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:35618 8.b 8:d6949987:::benchmark_data_smithi033_105085_object35617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:07.860356+0000", 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.053011293000001, 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.065072397000000004, 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35618 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860356+0000", 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860356+0000", 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860354+0000", 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6739999999999997e-05 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860391+0000", 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0113e-05 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860401+0000", 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1075e-05 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860412+0000", 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6091999999999998e-05 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860458+0000", 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111282 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860569+0000", 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000145606 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.860715+0000", 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027962070000000002 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.863511+0000", 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038990729999999999 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.867410+0000", 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057944013000000003 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925354+0000", 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0198000000000001e-05 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925384+0000", 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3967000000000001e-05 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925428+0000", 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.065072397000000004 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:35622 8.b 8:d7bb87c7:::benchmark_data_smithi033_105085_object35621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:07.861982+0000", 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.051385334999999, 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063592522999999998, 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35622 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.861982+0000", 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.861982+0000", 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.861980+0000", 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1769e-05 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.862012+0000", 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0860000000000008e-06 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.862021+0000", 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7089999999999999e-06 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.862027+0000", 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2308000000000001e-05 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.862079+0000", 2024-02-12T16:57:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012209100000000001 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.862201+0000", 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000101086 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.862302+0000", 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029893950000000002 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.865292+0000", 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021441160000000002 2024-02-12T16:57:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.867436+0000", 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058047762000000003 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925483+0000", 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1727000000000001e-05 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925535+0000", 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9125999999999998e-05 2024-02-12T16:57:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925574+0000", 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063592522999999998 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:35630 8.b 8:d5ea0eda:::benchmark_data_smithi033_105085_object35629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:07.866167+0000", 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.047200176, 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059543486, 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35630 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.866167+0000", 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.866167+0000", 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.866166+0000", 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2821999999999998e-05 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.866199+0000", 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0569999999999998e-06 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.866205+0000", 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6419999999999998e-06 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.866210+0000", 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012444870000000001 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.867454+0000", 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000101858 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.867556+0000", 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000135539 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.867692+0000", 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025130339999999999 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870205+0000", 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5296000000000001e-05 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870220+0000", 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.055408453000000003 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925628+0000", 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4710999999999999e-05 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925673+0000", 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7367999999999999e-05 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925710+0000", 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059543486 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:35635 8.b 8:d2824ec8:::benchmark_data_smithi033_105085_object35634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:07.870342+0000", 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.043024978999998, 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.055482570000000002, 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35635 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870342+0000", 2024-02-12T16:57:29.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870342+0000", 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870341+0000", 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2980999999999999e-05 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870404+0000", 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1179999999999992e-06 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870412+0000", 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0799999999999999e-06 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870416+0000", 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0300000000000001e-05 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870446+0000", 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2582000000000004e-05 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870519+0000", 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8590000000000004e-05 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.870607+0000", 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000958201 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.871566+0000", 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1319e-05 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.871577+0000", 2024-02-12T16:57:29.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054177615999999998 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925754+0000", 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9725999999999999e-05 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925794+0000", 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0540000000000002e-05 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:07.925825+0000", 2024-02-12T16:57:29.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.055482570000000002 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:37080 8.b 8:d2498496:::benchmark_data_smithi033_105085_object37079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:08.652514+0000", 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.260853320999999, 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068990145000000003, 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37080 2024-02-12T16:57:29.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652514+0000", 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652514+0000", 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652512+0000", 2024-02-12T16:57:29.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1099000000000005e-05 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652583+0000", 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7881999999999999e-05 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652601+0000", 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.037e-06 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652607+0000", 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8402000000000003e-05 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652645+0000", 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8405000000000006e-05 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652723+0000", 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9599000000000006e-05 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.652823+0000", 2024-02-12T16:57:29.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019614400000000001 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.654784+0000", 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001136368 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.655921+0000", 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.065504769000000004 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.721426+0000", 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8319999999999999e-05 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.721464+0000", 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0022999999999999e-05 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.721504+0000", 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068990145000000003 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:37094 8.b 8:db300118:::benchmark_data_smithi033_105085_object37093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:08.659569+0000", 2024-02-12T16:57:29.986 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.253798094, 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067271469, 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37094 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659569+0000", 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659569+0000", 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999905 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659559+0000", 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0856999999999998e-05 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659610+0000", 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3954000000000001e-05 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659624+0000", 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2670000000000003e-06 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659630+0000", 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8179999999999997e-05 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659669+0000", 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013047800000000001 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659799+0000", 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012118800000000001 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.659920+0000", 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014576979999999999 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.661378+0000", 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025976599999999998 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.661638+0000", 2024-02-12T16:57:29.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.065104466 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.726742+0000", 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0222000000000001e-05 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.726792+0000", 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8044000000000003e-05 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.726840+0000", 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067271469 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.991 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:37098 8.b 8:d01419c4:::benchmark_data_smithi033_105085_object37097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:08.663455+0000", 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.249911694000001, 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063532316000000005, 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37098 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.992 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663455+0000", 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663455+0000", 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663454+0000", 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7669999999999994e-05 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663522+0000", 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5349999999999993e-06 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663531+0000", 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2609999999999993e-06 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663540+0000", 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9353000000000001e-05 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:29.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663579+0000", 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116243 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663695+0000", 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015082800000000001 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.663846+0000", 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188873 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.665035+0000", 2024-02-12T16:57:29.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.596e-05 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.665051+0000", 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.061843944999999997 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.726895+0000", 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5520999999999999e-05 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.726950+0000", 2024-02-12T16:57:29.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7255999999999997e-05 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.726988+0000", 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063532316000000005 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:37471 8.b 8:df6b1cfa:::benchmark_data_smithi033_105085_object37470:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:08.907604+0000", 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.005763416000001, 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051426238999999999, 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:29.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37471 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907604+0000", 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907604+0000", 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907602+0000", 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6589999999999999e-05 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907658+0000", 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0631999999999999e-05 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907669+0000", 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8259999999999998e-06 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:29.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907675+0000", 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4511000000000001e-05 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907719+0000", 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8729999999999998e-05 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907818+0000", 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114589 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.907933+0000", 2024-02-12T16:57:30.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001203696 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.909136+0000", 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00261575 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.911752+0000", 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047208802000000001 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.958961+0000", 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0873e-05 2024-02-12T16:57:30.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.959002+0000", 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8136999999999998e-05 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.959030+0000", 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051426238999999999 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:37478 8.b 8:d35c30e1:::benchmark_data_smithi033_105085_object37477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:30.002 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:08.909739+0000", 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.003628587000001, 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050407797999999997, 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37478 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.909739+0000", 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.909739+0000", 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.909737+0000", 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019184200000000001 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.911655+0000", 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6379999999999993e-06 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.911664+0000", 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2070000000000003e-06 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.911669+0000", 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106319 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.911775+0000", 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018588900000000001 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.911961+0000", 2024-02-12T16:57:30.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018050999999999999 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.912142+0000", 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012699289999999999 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.913412+0000", 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000641732 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.914054+0000", 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046041179000000002 2024-02-12T16:57:30.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.960095+0000", 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8963e-05 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.960124+0000", 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2628000000000001e-05 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:08.960146+0000", 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050407797999999997 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.007 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:38519 8.b 8:d67311e1:::benchmark_data_smithi033_105085_object38518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:09.492429+0000", 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.420938360000001, 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046955152, 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 38519 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492429+0000", 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492429+0000", 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492428+0000", 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4736e-05 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492473+0000", 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5140000000000001e-06 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492481+0000", 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7169999999999999e-06 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492486+0000", 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6313999999999998e-05 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492522+0000", 2024-02-12T16:57:30.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000107562 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492630+0000", 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012253200000000001 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.492752+0000", 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011851920000000001 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.493937+0000", 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078491699999999999 2024-02-12T16:57:30.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.494722+0000", 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.044576287999999999 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.539299+0000", 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9237999999999999e-05 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.539348+0000", 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6072000000000003e-05 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.539384+0000", 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046955152 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:38936 8.2 8:44ea4f4e:::benchmark_data_smithi033_105085_object38935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:09.763719+0000", 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.149647712, 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051289790000000002, 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:30.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 38936 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.763719+0000", 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.763719+0000", 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999948 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:30.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.763714+0000", 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020156699999999999 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.763916+0000", 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1179999999999992e-06 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.763924+0000", 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3190000000000002e-06 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.763929+0000", 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1663999999999998e-05 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.763991+0000", 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112453 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.764103+0000", 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014252899999999999 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:57:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.764246+0000", 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059743840000000001 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.770220+0000", 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040993760000000001 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.774319+0000", 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040592215000000001 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.814912+0000", 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3902e-05 2024-02-12T16:57:30.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.814965+0000", 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3689999999999997e-05 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.815009+0000", 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051289790000000002 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.018 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:38946 8.2 8:4c168aca:::benchmark_data_smithi033_105085_object38945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:09.767107+0000", 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.146259651000001, 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048071095000000001, 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 38946 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.767107+0000", 2024-02-12T16:57:30.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.767107+0000", 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.767105+0000", 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9915e-05 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:30.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.767145+0000", 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4630000000000001e-05 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.767160+0000", 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5649999999999998e-06 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.767166+0000", 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001628067 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.768794+0000", 2024-02-12T16:57:30.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012544300000000001 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.768920+0000", 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106692 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.769026+0000", 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002569277 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.771596+0000", 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002741884 2024-02-12T16:57:30.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.774338+0000", 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040742383 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.815080+0000", 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8894000000000003e-05 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.815129+0000", 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9753999999999998e-05 2024-02-12T16:57:30.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.815179+0000", 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048071095000000001 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:38950 8.2 8:427f31c9:::benchmark_data_smithi033_105085_object38949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:09.769751+0000", 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.143616588, 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047950661999999998, 2024-02-12T16:57:30.024 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 38950 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.769751+0000", 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.769751+0000", 2024-02-12T16:57:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.769749+0000", 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6005999999999999e-05 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.769805+0000", 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0438999999999999e-05 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.769815+0000", 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2029999999999997e-06 2024-02-12T16:57:30.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.769821+0000", 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052069200000000001 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.770342+0000", 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8160999999999997e-05 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.770440+0000", 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100258 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,5", 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.770541+0000", 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012875810000000001 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.771828+0000", 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025235909999999999 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.774352+0000", 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043270156999999997 2024-02-12T16:57:30.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.817622+0000", 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2447999999999997e-05 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.817664+0000", 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6857999999999998e-05 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:09.817701+0000", 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047950661999999998 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.029 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:30.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:30.030 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:30.030 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:57:30.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:30.083 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:30.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:30.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:57:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:30.470 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 37 16 68114 68098 115.004 74.25 0.0124997 0.00869024 2024-02-12T16:57:30.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:28.843710+0000 osd.7 (osd.7) 109 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T16:57:30.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:28.846752+0000 osd.7 (osd.7) 110 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T16:57:30.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:29.916708+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:57:30.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:29.933945+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:57:30.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:57:30.667 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:28.843710+0000 osd.7 (osd.7) 109 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T16:57:30.667 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:28.846752+0000 osd.7 (osd.7) 110 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T16:57:30.667 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:29.916708+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:57:30.667 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:29.933945+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:57:30.668 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:28.843710+0000 osd.7 (osd.7) 109 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T16:57:30.668 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:28.846752+0000 osd.7 (osd.7) 110 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T16:57:30.668 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:29.916708+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:57:30.668 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:29.933945+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:57:30.680 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:57:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:30.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:57:30.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:57:30.858 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:30.891 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:57:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:30.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:57:30.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:31.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:57:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:30] "GET /metrics HTTP/1.1" 200 175831 "" "Prometheus/2.33.4" 2024-02-12T16:57:31.048 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:31.077 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:31.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:31.184 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:31.213 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:31.274 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:57:31.470 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 38 16 69235 69219 113.822 70.0625 0.0103579 0.00878214 2024-02-12T16:57:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:29.783312+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.f scrub starts 2024-02-12T16:57:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:29.805320+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.f scrub ok 2024-02-12T16:57:31.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:30.496791+0000 mgr.y (mgr.24481) 303 : cluster [DBG] pgmap v352: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-02-12T16:57:31.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:31.137125+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-12T16:57:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:29.783312+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.f scrub starts 2024-02-12T16:57:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:29.805320+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.f scrub ok 2024-02-12T16:57:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:30.496791+0000 mgr.y (mgr.24481) 303 : cluster [DBG] pgmap v352: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-02-12T16:57:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:31.137125+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-12T16:57:31.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:29.783312+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.f scrub starts 2024-02-12T16:57:31.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:29.805320+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.f scrub ok 2024-02-12T16:57:31.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:30.496791+0000 mgr.y (mgr.24481) 303 : cluster [DBG] pgmap v352: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-02-12T16:57:31.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:31.137125+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-12T16:57:31.778 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:31.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:31.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:31.933 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:32.039 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:32.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:32.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:32.206 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:32.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:32.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:32.470 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 39 16 70444 70428 112.84 75.5625 0.0125195 0.00885749 2024-02-12T16:57:32.501 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:32.728 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:32.734 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:30.750526+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-02-12T16:57:32.734 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:30.754646+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-02-12T16:57:32.734 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:31.804129+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:57:32.734 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:31.812636+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:57:32.734 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:31.927819+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:57:32.734 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:31.978832+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:57:32.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:30.750526+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-02-12T16:57:32.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:30.754646+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-02-12T16:57:32.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:31.804129+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:57:32.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:31.812636+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:57:32.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:31.927819+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:57:32.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:31.978832+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:57:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:32.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:30.750526+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-02-12T16:57:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:30.754646+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-02-12T16:57:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:31.804129+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.18 scrub starts 2024-02-12T16:57:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:31.812636+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.18 scrub ok 2024-02-12T16:57:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:31.927819+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.13 scrub starts 2024-02-12T16:57:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:31.978832+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.13 scrub ok 2024-02-12T16:57:32.741 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:57:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:57:32.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:33.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:71448 8.0 8:016ccd65:::benchmark_data_smithi033_105085_object71447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.389286+0000", 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.013873171, 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013923207999999999, 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:33.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71448 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.389286+0000", 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.389286+0000", 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.389286+0000", 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1183000000000003e-05 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.389317+0000", 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.277e-06 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.389324+0000", 2024-02-12T16:57:33.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0509999999999998e-06 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.389328+0000", 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025386200000000001 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.391867+0000", 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117281 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.391984+0000", 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000141998 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.392126+0000", 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072872179999999998 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399413+0000", 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030283900000000001 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399716+0000", 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034732040000000001 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403189+0000", 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013902868000000001 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.411 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:71455 8.0 8:0d873149:::benchmark_data_smithi033_105085_object71454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.393741+0000", 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0094186579999999999, 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095591340000000004, 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71455 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393741+0000", 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393741+0000", 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393740+0000", 2024-02-12T16:57:33.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1625999999999998e-05 2024-02-12T16:57:33.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393772+0000", 2024-02-12T16:57:33.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3119999999999996e-06 2024-02-12T16:57:33.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393779+0000", 2024-02-12T16:57:33.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393784+0000", 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030021280000000002 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396786+0000", 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121607 2024-02-12T16:57:33.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396907+0000", 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017847800000000001 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.397086+0000", 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036365189999999999 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.400722+0000", 2024-02-12T16:57:33.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069813999999999996 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:71456 8.8 8:1dbb1913:::benchmark_data_smithi033_105085_object71455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.393788+0000", 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0093713700000000004, 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095862069999999994, 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:33.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71456 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393788+0000", 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393788+0000", 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393788+0000", 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8806000000000002e-05 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393816+0000", 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5829999999999997e-06 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393822+0000", 2024-02-12T16:57:33.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8959999999999996e-06 2024-02-12T16:57:33.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.393826+0000", 2024-02-12T16:57:33.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040726110000000003 2024-02-12T16:57:33.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.397898+0000", 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109551 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.398008+0000", 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013359600000000001 2024-02-12T16:57:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,7", 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.398142+0000", 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0046340009999999996 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402776+0000", 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0089874259999999997 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:71459 8.4 8:208da4e0:::benchmark_data_smithi033_105085_object71458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.396357+0000", 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0068026880000000003, 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.007076939, 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71459 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396357+0000", 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396357+0000", 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396355+0000", 2024-02-12T16:57:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0707000000000003e-05 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396386+0000", 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6610000000000006e-06 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396395+0000", 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3870000000000003e-06 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396400+0000", 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8893000000000003e-05 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396499+0000", 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014272300000000001 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396642+0000", 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000371529 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,4", 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.397013+0000", 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037617739999999998 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.400775+0000", 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001955072 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402730+0000", 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063730560000000002 2024-02-12T16:57:33.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:71460 8.d 8:be87784e:::benchmark_data_smithi033_105085_object71459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.396939+0000", 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0062209850000000001, 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006593214, 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:33.430 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71460 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396939+0000", 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396939+0000", 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396937+0000", 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4425000000000002e-05 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396981+0000", 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8063e-05 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.396999+0000", 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9670000000000001e-06 2024-02-12T16:57:33.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.397007+0000", 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6715000000000003e-05 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.397054+0000", 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000110931 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:33.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.397165+0000", 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014220199999999999 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.397307+0000", 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0032824019999999998 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.400590+0000", 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036509899999999998 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.434 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:71458 8.2 e195/176 8:4652a709:::benchmark_data_smithi033_105085_object71457:head v 195'4464, mlcod=195'4460)", 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.399816+0000", 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0033433579999999998, 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003768343, 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399816+0000", 2024-02-12T16:57:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399816+0000", 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399815+0000", 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2121e-05 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399847+0000", 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3700000000000001e-06 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399857+0000", 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4389999999999999e-06 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399862+0000", 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000126809 2024-02-12T16:57:33.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.399989+0000", 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.933e-05 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.400008+0000", 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019200700000000001 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.438 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:71457 8.9 e195/176 8:9c78bccd:::benchmark_data_smithi033_105085_object71456:head v 195'4388, mlcod=195'4386)", 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.402266+0000", 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.000894045, 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013668549999999999, 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402266+0000", 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402266+0000", 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402264+0000", 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4275999999999999e-05 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402288+0000", 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8699999999999998e-06 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402297+0000", 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3170000000000001e-06 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402303+0000", 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6928000000000003e-05 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402339+0000", 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0673999999999998e-05 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402370+0000", 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010464200000000001 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:71461 8.9 e195/176)", 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.402995+0000", 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.00016485100000000001, 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00067134400000000004, 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "queued for pg", 2024-02-12T16:57:33.442 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402995+0000", 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402995+0000", 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.402993+0000", 2024-02-12T16:57:33.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3153e-05 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403017+0000", 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1348e-05 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403028+0000", 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.972e-06 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403033+0000", 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8263000000000003e-05 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.35042.0:71460 8.d e195/176)", 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:33.403678+0000", 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4294967295.9994822, 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6536000000000001e-05, 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "queued for pg", 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403678+0000", 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403678+0000", 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403676+0000", 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5670000000000002e-06 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403682+0000", 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6309999999999992e-06 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403691+0000", 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1220000000000003e-06 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:33.403696+0000", 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8199e-05 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:33.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:33.448 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T16:57:33.448 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 9 2024-02-12T16:57:33.448 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:57:33.448 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:57:33.448 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:57:33.448 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:57:33.454 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:57:33.470 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T16:57:33.470005+0000 min lat: 0.00131853 max lat: 0.0769216 avg lat: 0.00894629 2024-02-12T16:57:33.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:57:33.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 40 16 71533 71517 111.721 68.0625 0.00986888 0.00894629 2024-02-12T16:57:33.471 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:33.505 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:57:33.509 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:33.521 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:33.542 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:57:33.583 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:57:33.618 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:57:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:57:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:57:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:57:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:57:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:57:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:57:33.631 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:57:33.632 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:57:33.649 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:32.476432+0000 mon.a (mon.0) 1000 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:33.649 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:32.497452+0000 mgr.y (mgr.24481) 304 : cluster [DBG] pgmap v354: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 805 B/s rd, 61 MiB/s wr, 969 op/s 2024-02-12T16:57:33.649 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:32.773840+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.2 deep-scrub starts 2024-02-12T16:57:33.649 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:32.780108+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.2 deep-scrub ok 2024-02-12T16:57:33.650 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:33.291017+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.33:0/2502572084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:33.650 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:32.476432+0000 mon.a (mon.0) 1000 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:33.650 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:32.497452+0000 mgr.y (mgr.24481) 304 : cluster [DBG] pgmap v354: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 805 B/s rd, 61 MiB/s wr, 969 op/s 2024-02-12T16:57:33.650 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:32.773840+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.2 deep-scrub starts 2024-02-12T16:57:33.650 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:32.780108+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.2 deep-scrub ok 2024-02-12T16:57:33.650 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:33.291017+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.33:0/2502572084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:33.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:33.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:32.476432+0000 mon.a (mon.0) 1000 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:33.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:32.497452+0000 mgr.y (mgr.24481) 304 : cluster [DBG] pgmap v354: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 805 B/s rd, 61 MiB/s wr, 969 op/s 2024-02-12T16:57:33.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:32.773840+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.2 deep-scrub starts 2024-02-12T16:57:33.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:32.780108+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.2 deep-scrub ok 2024-02-12T16:57:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:33.291017+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.33:0/2502572084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:33.772 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:57:33.825 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:33.916 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:33.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:34.020 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:34.079 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:34.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:57:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:57:34.127 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:57:34.191 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:34.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:34.301 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:34.349 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:34.470 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 41 16 72629 72613 110.666 68.5 0.0162663 0.00903135 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:33.661511+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.33:0/3214336727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:33.661852+0000 mon.a (mon.0) 1001 : audit [INF] from='client.35099 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:33.717339+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.1b deep-scrub starts 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:33.724883+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.1b deep-scrub ok 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:33.758488+0000 mgr.y (mgr.24481) 305 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:33.800686+0000 osd.2 (osd.2) 127 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:33.806959+0000 osd.2 (osd.2) 128 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:33.889273+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:57:34.497 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:33.892173+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:57:34.521 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 5 pg_num to 22 2024-02-12T16:57:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:34.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:33.661511+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.33:0/3214336727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T16:57:34.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:33.661852+0000 mon.a (mon.0) 1001 : audit [INF] from='client.35099 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T16:57:34.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:33.717339+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.1b deep-scrub starts 2024-02-12T16:57:34.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:33.724883+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.1b deep-scrub ok 2024-02-12T16:57:34.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:33.758488+0000 mgr.y (mgr.24481) 305 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:34.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:33.800686+0000 osd.2 (osd.2) 127 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:57:34.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:33.806959+0000 osd.2 (osd.2) 128 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:57:34.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:33.889273+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:57:34.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:33.892173+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:57:34.851 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:33.661511+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.33:0/3214336727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T16:57:34.851 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:33.661852+0000 mon.a (mon.0) 1001 : audit [INF] from='client.35099 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T16:57:34.852 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:33.717339+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.1b deep-scrub starts 2024-02-12T16:57:34.852 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:33.724883+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.1b deep-scrub ok 2024-02-12T16:57:34.852 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:33.758488+0000 mgr.y (mgr.24481) 305 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:34.852 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:33.800686+0000 osd.2 (osd.2) 127 : cluster [DBG] 2.8 deep-scrub starts 2024-02-12T16:57:34.852 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:33.806959+0000 osd.2 (osd.2) 128 : cluster [DBG] 2.8 deep-scrub ok 2024-02-12T16:57:34.852 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:33.889273+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:57:34.852 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:33.892173+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:57:35.315 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:35.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:35.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:35.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 42 16 73819 73803 109.802 74.375 0.0186084 0.0091031 2024-02-12T16:57:35.474 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:35.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:34.498908+0000 mgr.y (mgr.24481) 306 : cluster [DBG] pgmap v355: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-02-12T16:57:35.735 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:34.502478+0000 mon.a (mon.0) 1002 : audit [INF] from='client.35099 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-12T16:57:35.736 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:34.502542+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-12T16:57:35.736 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:34.498908+0000 mgr.y (mgr.24481) 306 : cluster [DBG] pgmap v355: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-02-12T16:57:35.736 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:34.502478+0000 mon.a (mon.0) 1002 : audit [INF] from='client.35099 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-12T16:57:35.736 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:34.502542+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-12T16:57:35.737 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:35.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:35.848 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:35.849 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:34.498908+0000 mgr.y (mgr.24481) 306 : cluster [DBG] pgmap v355: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-02-12T16:57:35.849 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:34.502478+0000 mon.a (mon.0) 1002 : audit [INF] from='client.35099 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-12T16:57:35.849 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:34.502542+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-12T16:57:35.916 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:36.172 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:36.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:36.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:57:36.324 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:36.382 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:36.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:36.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 43 16 74901 74885 108.821 67.625 0.0281903 0.00918445 2024-02-12T16:57:36.488 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:36.500 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:57:36.500 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:57:36.544 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:36.575 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:36.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:36.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:57:36.681 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:36.738 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:36.774 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:36.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:36.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:37.098 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:37.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 44 16 75856 75840 107.705 59.6875 0.0146797 0.00928017 2024-02-12T16:57:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:35.775549+0000 osd.2 (osd.2) 129 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:57:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:35.783897+0000 osd.2 (osd.2) 130 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:57:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:36.135450+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-12T16:57:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:36.500393+0000 mgr.y (mgr.24481) 307 : cluster [DBG] pgmap v358: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 60 MiB/s wr, 968 op/s 2024-02-12T16:57:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:36.500954+0000 mon.c (mon.2) 119 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:57:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:36.502140+0000 mon.a (mon.0) 1005 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:35.775549+0000 osd.2 (osd.2) 129 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:35.783897+0000 osd.2 (osd.2) 130 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:36.135450+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:36.500393+0000 mgr.y (mgr.24481) 307 : cluster [DBG] pgmap v358: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 60 MiB/s wr, 968 op/s 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:36.500954+0000 mon.c (mon.2) 119 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:36.502140+0000 mon.a (mon.0) 1005 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:35.775549+0000 osd.2 (osd.2) 129 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:35.783897+0000 osd.2 (osd.2) 130 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:36.135450+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:36.500393+0000 mgr.y (mgr.24481) 307 : cluster [DBG] pgmap v358: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 60 MiB/s wr, 968 op/s 2024-02-12T16:57:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:36.500954+0000 mon.c (mon.2) 119 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:57:37.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:36.502140+0000 mon.a (mon.0) 1005 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:57:38.203 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:57:38.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 45 15 76943 76928 106.822 68 0.00897533 0.0093574 2024-02-12T16:57:38.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:36.751914+0000 osd.2 (osd.2) 131 : cluster [DBG] 6.f scrub starts 2024-02-12T16:57:38.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:36.798230+0000 osd.2 (osd.2) 132 : cluster [DBG] 6.f scrub ok 2024-02-12T16:57:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:37.154167+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:57:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.154225+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-12T16:57:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:37.212807+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.33:0/3739796084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:57:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.215797+0000 osd.1 (osd.1) 144 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:57:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.219872+0000 osd.1 (osd.1) 145 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:57:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.863638+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:57:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.895279+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:57:38.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:36.751914+0000 osd.2 (osd.2) 131 : cluster [DBG] 6.f scrub starts 2024-02-12T16:57:38.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:36.798230+0000 osd.2 (osd.2) 132 : cluster [DBG] 6.f scrub ok 2024-02-12T16:57:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:37.154167+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:57:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.154225+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-12T16:57:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:37.212807+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.33:0/3739796084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:57:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.215797+0000 osd.1 (osd.1) 144 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:57:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.219872+0000 osd.1 (osd.1) 145 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:57:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.863638+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:57:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.895279+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:57:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:36.751914+0000 osd.2 (osd.2) 131 : cluster [DBG] 6.f scrub starts 2024-02-12T16:57:38.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:36.798230+0000 osd.2 (osd.2) 132 : cluster [DBG] 6.f scrub ok 2024-02-12T16:57:38.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:37.154167+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:57:38.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.154225+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-12T16:57:38.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:37.212807+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.33:0/3739796084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:57:38.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.215797+0000 osd.1 (osd.1) 144 : cluster [DBG] 4.14 scrub starts 2024-02-12T16:57:38.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.219872+0000 osd.1 (osd.1) 145 : cluster [DBG] 4.14 scrub ok 2024-02-12T16:57:38.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.863638+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.13 scrub starts 2024-02-12T16:57:38.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.895279+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.13 scrub ok 2024-02-12T16:57:39.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 46 16 77998 77982 105.932 65.875 0.0110115 0.00943542 2024-02-12T16:57:39.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.199433+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:39.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.199489+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:39.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.657181+0000 osd.7 (osd.7) 113 : cluster [DBG] 6.1c scrub starts 2024-02-12T16:57:39.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:37.670697+0000 osd.7 (osd.7) 114 : cluster [DBG] 6.1c scrub ok 2024-02-12T16:57:39.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:38.167601+0000 mon.a (mon.0) 1009 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:39.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:38.173796+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? v1:172.21.15.33:0/3739796084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:57:39.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:38.173857+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-12T16:57:39.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:38.501024+0000 mgr.y (mgr.24481) 308 : cluster [DBG] pgmap v361: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 48 MiB/s wr, 760 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:39.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:38.501446+0000 mon.c (mon.2) 120 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:57:39.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:38.502590+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:57:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.199433+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.199489+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.657181+0000 osd.7 (osd.7) 113 : cluster [DBG] 6.1c scrub starts 2024-02-12T16:57:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:37.670697+0000 osd.7 (osd.7) 114 : cluster [DBG] 6.1c scrub ok 2024-02-12T16:57:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:38.167601+0000 mon.a (mon.0) 1009 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:38.173796+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? v1:172.21.15.33:0/3739796084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:57:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:38.173857+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-12T16:57:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:38.501024+0000 mgr.y (mgr.24481) 308 : cluster [DBG] pgmap v361: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 48 MiB/s wr, 760 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:38.501446+0000 mon.c (mon.2) 120 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:38.502590+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.199433+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.199489+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.657181+0000 osd.7 (osd.7) 113 : cluster [DBG] 6.1c scrub starts 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:37.670697+0000 osd.7 (osd.7) 114 : cluster [DBG] 6.1c scrub ok 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:38.167601+0000 mon.a (mon.0) 1009 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:38.173796+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? v1:172.21.15.33:0/3739796084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:38.173857+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-12T16:57:39.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:38.501024+0000 mgr.y (mgr.24481) 308 : cluster [DBG] pgmap v361: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 48 MiB/s wr, 760 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:39.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:38.501446+0000 mon.c (mon.2) 120 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:57:39.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:38.502590+0000 mon.a (mon.0) 1012 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:57:40.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 47 16 79116 79100 105.165 69.875 0.00899419 0.00950422 2024-02-12T16:57:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:39.183002+0000 mon.a (mon.0) 1013 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T16:57:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:39.183064+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-12T16:57:40.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:39.183002+0000 mon.a (mon.0) 1013 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T16:57:40.506 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:39.183064+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-12T16:57:40.506 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:39.183002+0000 mon.a (mon.0) 1013 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T16:57:40.506 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:39.183064+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-12T16:57:41.193 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:57:41.193 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:57:41.194 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 4, 5, 6, 7] 2024-02-12T16:57:41.194 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd out 1 2024-02-12T16:57:41.212 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:57:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:40] "GET /metrics HTTP/1.1" 200 175836 "" "Prometheus/2.33.4" 2024-02-12T16:57:41.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 48 15 80257 80242 104.461 71.375 0.012141 0.00956845 2024-02-12T16:57:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:40.206929+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-12T16:57:41.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:40.501501+0000 mgr.y (mgr.24481) 309 : cluster [DBG] pgmap v364: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:41.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:40.206929+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-12T16:57:41.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:40.501501+0000 mgr.y (mgr.24481) 309 : cluster [DBG] pgmap v364: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:41.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:40.206929+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-12T16:57:41.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:40.501501+0000 mgr.y (mgr.24481) 309 : cluster [DBG] pgmap v364: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:41.802 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:41.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:41.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:57:41.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:41.958 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:42.121 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:42.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:42.185 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:57:42.188 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:42.221 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:41.210216+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-12T16:57:42.222 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:42.150731+0000 mon.b (mon.1) 65 : audit [DBG] from='client.? v1:172.21.15.33:0/796262016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:42.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:42.235 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:57:42.275 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:57:42.277 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:42.289 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:42.303 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:57:42.305 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:42.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:42.356 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:57:42.393 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:57:42.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:42.423 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:57:42.458 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:57:42.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:41.210216+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-12T16:57:42.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:42.150731+0000 mon.b (mon.1) 65 : audit [DBG] from='client.? v1:172.21.15.33:0/796262016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:42.461 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:57:42.461 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:57:42.461 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:57:42.462 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:57:42.462 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:57:42.462 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:57:42.462 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:57:42.462 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:57:42.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:41.210216+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-12T16:57:42.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:42.150731+0000 mon.b (mon.1) 65 : audit [DBG] from='client.? v1:172.21.15.33:0/796262016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:42.467 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:42.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 49 16 81284 81268 103.638 64.125 0.0181885 0.0096445 2024-02-12T16:57:42.499 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:42.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:42.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:42.679 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:42.739 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:42.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:42.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:57:42.897 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:42.991 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:57:42.993 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:57:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:57:43.261 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:43.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:43.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:57:43.402 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:42.213242+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-12T16:57:43.402 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:42.502867+0000 mgr.y (mgr.24481) 310 : cluster [DBG] pgmap v367: 147 pgs: 147 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 599 op/s 2024-02-12T16:57:43.402 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:42.503472+0000 mon.c (mon.2) 121 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T16:57:43.402 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:42.504610+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T16:57:43.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:42.213242+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-12T16:57:43.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:42.502867+0000 mgr.y (mgr.24481) 310 : cluster [DBG] pgmap v367: 147 pgs: 147 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 599 op/s 2024-02-12T16:57:43.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:42.503472+0000 mon.c (mon.2) 121 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T16:57:43.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:42.504610+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T16:57:43.434 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:43.471 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 50 16 82315 82299 102.854 64.4375 0.0109251 0.0097184 2024-02-12T16:57:43.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:42.213242+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-12T16:57:43.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:42.502867+0000 mgr.y (mgr.24481) 310 : cluster [DBG] pgmap v367: 147 pgs: 147 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 599 op/s 2024-02-12T16:57:43.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:42.503472+0000 mon.c (mon.2) 121 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T16:57:43.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:42.504610+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T16:57:44.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:57:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:57:44.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 51 16 83349 83333 102.1 64.625 0.0159886 0.0097898 2024-02-12T16:57:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:43.234716+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T16:57:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:43.234776+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-12T16:57:44.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:43.773698+0000 mgr.y (mgr.24481) 311 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:44.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:43.234716+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T16:57:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:43.234776+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-12T16:57:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:43.773698+0000 mgr.y (mgr.24481) 311 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:43.234716+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T16:57:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:43.234776+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-12T16:57:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:43.773698+0000 mgr.y (mgr.24481) 311 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:44.806 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:57:44.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:44.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:57:44.911 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:44.985 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:44.996 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:45.033 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:45.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:45.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:45.193 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:45.279 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:44.248280+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-12T16:57:45.279 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:44.503254+0000 mgr.y (mgr.24481) 312 : cluster [DBG] pgmap v370: 147 pgs: 2 peering, 145 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-12T16:57:45.360 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:45.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:45.466 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:45.473 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 52 16 84315 84299 101.298 60.375 0.00670595 0.00986794 2024-02-12T16:57:45.524 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:45.610 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:45.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:44.248280+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-12T16:57:45.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:44.503254+0000 mgr.y (mgr.24481) 312 : cluster [DBG] pgmap v370: 147 pgs: 2 peering, 145 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-12T16:57:45.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:44.248280+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-12T16:57:45.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:44.503254+0000 mgr.y (mgr.24481) 312 : cluster [DBG] pgmap v370: 147 pgs: 2 peering, 145 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-12T16:57:45.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:57:45.339+0000 7fd96a9de700 -1 mon.a@0(leader).osd e206 definitely_dead 0 2024-02-12T16:57:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:45.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:45.716 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:45.772 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:45.914 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:45.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:46.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:46.093 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:46.139 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:46.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:46.180 INFO:teuthology.orchestra.run.smithi033.stderr:marked out osd.1. 2024-02-12T16:57:46.255 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:46.284 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220600 7.0 7:02c49166:::benchmark_data_smithi033_81060_object110148:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.423888+0000", 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.967960356999995, 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0364286869999999, 2024-02-12T16:57:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220600 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423888+0000", 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423888+0000", 2024-02-12T16:57:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423885+0000", 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0101e-05 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423895+0000", 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7076999999999999e-05 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423912+0000", 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8669999999999994e-06 2024-02-12T16:57:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423920+0000", 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2386999999999998e-05 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423972+0000", 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013168699999999999 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.424104+0000", 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000158741 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,3", 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.424263+0000", 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028780569999999998 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.427141+0000", 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0050549430000000001 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.432196+0000", 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1647999999999999e-05 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.432207+0000", 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028020624 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460228+0000", 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8412999999999995e-05 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460316+0000", 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0364286869999999 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220610 7.0 7:099dd978:::benchmark_data_smithi033_81060_object110158:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.440693+0000", 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.951154606000003, 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0445725020000001, 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220610 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440693+0000", 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440693+0000", 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440691+0000", 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2689999999999999e-06 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440699+0000", 2024-02-12T16:57:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6155e-05 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440716+0000", 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2880000000000004e-06 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440723+0000", 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5838999999999999e-05 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440769+0000", 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8959999999999992e-06 2024-02-12T16:57:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440778+0000", 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0195559999999999 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460334+0000", 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4116000000000006e-05 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460418+0000", 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9007999999999998e-05 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460517+0000", 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010719841000000001 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471237+0000", 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013909055 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485146+0000", 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3446000000000003e-05 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485199+0000", 2024-02-12T16:57:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9681000000000002e-05 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485239+0000", 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7188000000000001e-05 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485266+0000", 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0445725020000001 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220613 7.0 7:04831311:::benchmark_data_smithi033_81060_object110161:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.444906+0000", 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.946942407999998, 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.040521985, 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220613 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444906+0000", 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444906+0000", 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444904+0000", 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5519999999999999e-06 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444911+0000", 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2004000000000001e-05 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444923+0000", 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.815e-06 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444930+0000", 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1479e-05 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444972+0000", 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6400000000000002e-06 2024-02-12T16:57:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444977+0000", 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0176013660000001 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.462579+0000", 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000110575 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.462689+0000", 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015559300000000001 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.462845+0000", 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0084006129999999995 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471245+0000", 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014058726000000001 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485304+0000", 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4366e-05 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485368+0000", 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4146999999999997e-05 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485403+0000", 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5015999999999999e-05 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485428+0000", 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.040521985 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220649 7.0 7:0e13de3b:::benchmark_data_smithi033_81060_object110197:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.474384+0000", 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.917463706999996, 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0112301079999999, 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220649 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474384+0000", 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474384+0000", 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474382+0000", 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8539999999999997e-06 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474390+0000", 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3152999999999999e-05 2024-02-12T16:57:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474403+0000", 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3549999999999999e-06 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474410+0000", 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9561000000000002e-05 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474460+0000", 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0509999999999999e-05 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474470+0000", 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99010147599999998 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.464572+0000", 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010268900000000001 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.464674+0000", 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011038 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.464785+0000", 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064674779999999996 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471252+0000", 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014089404 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485342+0000", 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018344800000000001 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485525+0000", 2024-02-12T16:57:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6623999999999999e-05 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485572+0000", 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765999999999997e-05 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485614+0000", 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0112301079999999 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220655 7.0 7:094c2cc9:::benchmark_data_smithi033_81060_object110203:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.480491+0000", 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.911357086999999, 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.005281632, 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220655 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480491+0000", 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480491+0000", 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480489+0000", 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5780000000000001e-06 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480496+0000", 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0825000000000001e-05 2024-02-12T16:57:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480507+0000", 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5570000000000001e-06 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480514+0000", 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2985000000000002e-05 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480567+0000", 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.003e-06 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480576+0000", 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98622844899999995 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.466804+0000", 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012368399999999999 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.466928+0000", 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000128603 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.467056+0000", 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042049790000000002 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471261+0000", 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014414544 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485676+0000", 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7676999999999999e-05 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485713+0000", 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5408000000000001e-05 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485739+0000", 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3747000000000001e-05 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485773+0000", 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.005281632 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220656 7.0 7:07768b72:::benchmark_data_smithi033_81060_object110204:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.481291+0000", 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.910557448000006, 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.004739396, 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220656 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481291+0000", 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481291+0000", 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481289+0000", 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6919999999999999e-06 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481295+0000", 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6899999999999998e-06 2024-02-12T16:57:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481304+0000", 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4210000000000003e-06 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481309+0000", 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6100000000000002e-05 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481355+0000", 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6790000000000005e-06 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481363+0000", 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98715179399999997 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.468515+0000", 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6761000000000005e-05 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.468602+0000", 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4424000000000003e-05 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.468686+0000", 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062958279999999998 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.474982+0000", 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010832078 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485814+0000", 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012336199999999999 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485937+0000", 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7234999999999997e-05 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485995+0000", 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5515000000000001e-05 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486030+0000", 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.004739396 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220706 7.0 7:06895523:::benchmark_data_smithi033_81060_object110254:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.524105+0000", 2024-02-12T16:57:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.867743150999999, 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96202424399999997, 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220706 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524105+0000", 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524105+0000", 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524103+0000", 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3710000000000002e-06 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524110+0000", 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1291000000000001e-05 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524122+0000", 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0849999999999996e-06 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524127+0000", 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9991999999999998e-05 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524187+0000", 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8079999999999999e-06 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524196+0000", 2024-02-12T16:57:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94564106599999997 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.469837+0000", 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113767 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.469951+0000", 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6594000000000005e-05 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.470038+0000", 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092142419999999992 2024-02-12T16:57:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.479252+0000", 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065906769999999996 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485843+0000", 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021390100000000001 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486057+0000", 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8380000000000002e-05 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486095+0000", 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4137999999999997e-05 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486129+0000", 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96202424399999997 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220711 7.0 7:05b344d8:::benchmark_data_smithi033_81060_object110259:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.528517+0000", 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.863330963999999, 2024-02-12T16:57:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95770867299999995, 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220711 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.528517+0000", 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.528517+0000", 2024-02-12T16:57:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.528515+0000", 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9070000000000007e-06 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.528523+0000", 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0478e-05 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.528533+0000", 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.973e-06 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.528540+0000", 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7231000000000002e-05 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.528587+0000", 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2559999999999999e-06 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.528594+0000", 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94271025900000005 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471304+0000", 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2148e-05 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471386+0000", 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1038999999999994e-05 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471477+0000", 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833759999999998 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.479260+0000", 2024-02-12T16:57:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066188139999999998 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485879+0000", 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00029010899999999998 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486169+0000", 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5250999999999998e-05 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486205+0000", 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1148000000000001e-05 2024-02-12T16:57:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486226+0000", 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95770867299999995 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220723 7.0 7:070ee907:::benchmark_data_smithi033_81060_object110271:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.543251+0000", 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.848597573000006, 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94312711199999999, 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220723 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.543251+0000", 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.543251+0000", 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.543248+0000", 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2910000000000007e-06 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.543257+0000", 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1083999999999999e-05 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.543268+0000", 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4221e-05 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.543282+0000", 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8066000000000003e-05 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.543340+0000", 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9569999999999999e-06 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.543346+0000", 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93030939400000001 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.473655+0000", 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9984999999999999e-05 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.473755+0000", 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2998999999999996e-05 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.473838+0000", 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0083241319999999997 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.482162+0000", 2024-02-12T16:57:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0041208829999999997 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486283+0000", 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1662000000000002e-05 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486325+0000", 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3592e-05 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486349+0000", 2024-02-12T16:57:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9099999999999999e-05 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486378+0000", 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94312711199999999 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220730 7.0 7:036d925e:::benchmark_data_smithi033_81060_object110278:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.548301+0000", 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.843546911999994, 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93821209699999997, 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220730 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.548301+0000", 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.548301+0000", 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.548299+0000", 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0819999999999999e-06 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.548307+0000", 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3132000000000001e-05 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.548320+0000", 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4969999999999998e-06 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.548327+0000", 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7265999999999998e-05 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.548374+0000", 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7400000000000001e-06 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.548380+0000", 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.92665195899999997 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.475031+0000", 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1383000000000003e-05 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.475113+0000", 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5175999999999998e-05 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.475208+0000", 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069612750000000003 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.482169+0000", 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042457900000000002 2024-02-12T16:57:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486415+0000", 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3881999999999998e-05 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486459+0000", 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2586999999999998e-05 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486492+0000", 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1696e-05 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486513+0000", 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93821209699999997 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220769 7.0 7:0535c892:::benchmark_data_smithi033_81060_object110317:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.595004+0000", 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.796844359000005, 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.89164605100000005, 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220769 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.595004+0000", 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.595004+0000", 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.595001+0000", 2024-02-12T16:57:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2620000000000004e-06 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.595010+0000", 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3963e-05 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.595024+0000", 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4949999999999996e-06 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.595031+0000", 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1740999999999998e-05 2024-02-12T16:57:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.595083+0000", 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2669999999999995e-06 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.595092+0000", 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.88132147500000002 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.476413+0000", 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6402000000000006e-05 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.476480+0000", 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7859000000000001e-05 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.476558+0000", 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056243669999999999 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.482182+0000", 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043661070000000001 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486548+0000", 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2386000000000003e-05 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486600+0000", 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5216999999999999e-05 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486626+0000", 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4108e-05 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486650+0000", 2024-02-12T16:57:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.89164605100000005 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220788 7.0 7:047ce615:::benchmark_data_smithi033_81060_object110336:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.612497+0000", 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.779351258000005, 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87423543199999998, 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220788 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.612497+0000", 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.612497+0000", 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.612494+0000", 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7880000000000007e-06 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.612502+0000", 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1533e-05 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.612514+0000", 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.322e-06 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.612519+0000", 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0760000000000002e-05 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.612570+0000", 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5789999999999993e-06 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.612579+0000", 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.865131753 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.477711+0000", 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011106 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.477822+0000", 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2560999999999998e-05 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.477905+0000", 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070041840000000001 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.484909+0000", 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001667221 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486576+0000", 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114002 2024-02-12T16:57:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486690+0000", 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3172000000000001e-05 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486713+0000", 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9017000000000001e-05 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486732+0000", 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87423543199999998 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220792 7.0 7:03f97ede:::benchmark_data_smithi033_81060_object110340:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.613551+0000", 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.778296845, 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87327720600000003, 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220792 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.613551+0000", 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.613551+0000", 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.613550+0000", 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-06 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.613555+0000", 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0725e-05 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.613566+0000", 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5990000000000004e-06 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.613570+0000", 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4530000000000003e-05 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.613605+0000", 2024-02-12T16:57:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8559999999999998e-06 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.613610+0000", 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.86568617999999997 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.479296+0000", 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3059999999999995e-05 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.479369+0000", 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3669000000000001e-05 2024-02-12T16:57:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.479453+0000", 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054645090000000002 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.484917+0000", 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001845429 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486763+0000", 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6248e-05 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486789+0000", 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1126000000000001e-05 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486810+0000", 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8405000000000001e-05 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486828+0000", 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87327720600000003 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220823 7.0 7:0d1c6ada:::benchmark_data_smithi033_81060_object110371:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.654790+0000", 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.737057840000006, 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.832132765, 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220823 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.654790+0000", 2024-02-12T16:57:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.654790+0000", 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.654788+0000", 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0839999999999992e-06 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.654796+0000", 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2896e-05 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.654809+0000", 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4049999999999997e-06 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.654815+0000", 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0713000000000001e-05 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.654856+0000", 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5379999999999999e-06 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.654862+0000", 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82577639300000005 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.480639+0000", 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010552600000000001 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.480744+0000", 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4588999999999999e-05 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.480829+0000", 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043270799999999996 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485156+0000", 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00170235 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486858+0000", 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3073000000000001e-05 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486881+0000", 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2030000000000001e-05 2024-02-12T16:57:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486903+0000", 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9542e-05 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486923+0000", 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.832132765 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220833 7.0 7:0ff00b82:::benchmark_data_smithi033_81060_object110381:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.672350+0000", 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.719498478999995, 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.81989158500000003, 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220833 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.672350+0000", 2024-02-12T16:57:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.672350+0000", 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.672348+0000", 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6059999999999997e-06 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.672355+0000", 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1430000000000001e-05 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.672367+0000", 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1170000000000003e-06 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.672372+0000", 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3993999999999999e-05 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.672416+0000", 2024-02-12T16:57:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4590000000000004e-06 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.672425+0000", 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.80979774100000002 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.482223+0000", 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6097e-05 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.482299+0000", 2024-02-12T16:57:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7841000000000001e-05 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.482397+0000", 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027661809999999999 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485163+0000", 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001790963 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486954+0000", 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051808000000000002 2024-02-12T16:57:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.492135+0000", 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4040000000000003e-05 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.492199+0000", 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2333000000000003e-05 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.492241+0000", 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.81989158500000003 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220856 7.0 7:009b566b:::benchmark_data_smithi033_81060_object110404:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.737192+0000", 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.654656270000004, 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.75515619, 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220856 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.737192+0000", 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.737192+0000", 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.737190+0000", 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1969999999999999e-06 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.737197+0000", 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.737208+0000", 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7659999999999998e-06 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.737213+0000", 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7540000000000003e-05 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.737250+0000", 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2219999999999999e-06 2024-02-12T16:57:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.737256+0000", 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.74633482799999995 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.483591+0000", 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6836999999999997e-05 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.483658+0000", 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3484000000000002e-05 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.483741+0000", 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015681359999999999 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485310+0000", 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001673349 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486983+0000", 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052937510000000002 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.492277+0000", 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9304000000000002e-05 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.492316+0000", 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1998000000000001e-05 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.492348+0000", 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.75515619 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:83368 8.0 8:0e1ea8c4:::benchmark_data_smithi033_105085_object83367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:44.487681+0000", 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.904166883, 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.111381665, 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83368 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487681+0000", 2024-02-12T16:57:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487681+0000", 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487680+0000", 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2999e-05 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487713+0000", 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6019999999999996e-06 2024-02-12T16:57:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487719+0000", 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.164e-06 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487725+0000", 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.097986824 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.585711+0000", 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000127823 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.585839+0000", 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000192974 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.586032+0000", 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088064219999999995 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.594839+0000", 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004100139 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.598939+0000", 2024-02-12T16:57:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1393000000000003e-05 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.598970+0000", 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0393999999999999e-05 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599021+0000", 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2308999999999998e-05 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599063+0000", 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.111381665 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:83369 8.0 8:0a30d280:::benchmark_data_smithi033_105085_object83368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:44.487730+0000", 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.9041184449999999, 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.112591222, 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83369 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487730+0000", 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487730+0000", 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487729+0000", 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4533000000000003e-05 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487763+0000", 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7819999999999996e-06 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487770+0000", 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1370000000000004e-06 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487774+0000", 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10060920299999999 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.588383+0000", 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120072 2024-02-12T16:57:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.588503+0000", 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014852999999999999 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4", 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.588652+0000", 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088827179999999995 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.597535+0000", 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001563023 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599098+0000", 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001133083 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.600231+0000", 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1232e-05 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.600282+0000", 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8819999999999998e-05 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.600321+0000", 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.112591222 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:83370 8.8 8:11de6d2c:::benchmark_data_smithi033_105085_object83369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:44.487779+0000", 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.904069316, 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.11149394999999999, 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83370 2024-02-12T16:57:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487779+0000", 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487779+0000", 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487778+0000", 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4598000000000002e-05 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487823+0000", 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2999999999999998e-06 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487829+0000", 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4220000000000002e-06 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.487833+0000", 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103320364 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.591154+0000", 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014848199999999999 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.591302+0000", 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017118499999999999 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,7", 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.591473+0000", 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076587729999999998 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599132+0000", 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9508e-05 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599192+0000", 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2539999999999992e-06 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599200+0000", 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7224999999999997e-05 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599237+0000", 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5565000000000003e-05 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599273+0000", 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.11149394999999999 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:83374 8.8 8:1a35a800:::benchmark_data_smithi033_105085_object83373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:44.492362+0000", 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.8994857509999998, 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.114030125, 2024-02-12T16:57:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83374 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.492362+0000", 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.492362+0000", 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:57:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.492361+0000", 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5529999999999999e-05 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.492416+0000", 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.090807185999999998 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.583224+0000", 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8619999999999992e-06 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.583233+0000", 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013115378 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.596348+0000", 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114338 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.596462+0000", 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014904100000000001 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,7", 2024-02-12T16:57:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.596611+0000", 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030650909999999998 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599676+0000", 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00031564599999999999 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.599992+0000", 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063201689999999996 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.606312+0000", 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4217000000000001e-05 2024-02-12T16:57:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.606346+0000", 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6038999999999997e-05 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:44.606392+0000", 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.114030125 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:46.511 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:57:46.512 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 53 16 85096 85080 100.308 48.8125 0.00295693 0.00990994 2024-02-12T16:57:46.512 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:46.512 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:45.278397+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-12T16:57:46.512 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:45.339409+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.33:0/1932155325' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-12T16:57:46.512 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:45.340150+0000 mon.a (mon.0) 1023 : audit [INF] from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-12T16:57:46.513 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:45.340252+0000 mon.a (mon.0) 1024 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2024-02-12T16:57:46.513 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:46.142276+0000 mon.a (mon.0) 1025 : audit [INF] from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-02-12T16:57:46.513 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:46.142345+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e207: 8 total, 8 up, 6 in 2024-02-12T16:57:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:46.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:45.278397+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-12T16:57:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:45.339409+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.33:0/1932155325' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-12T16:57:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:45.340150+0000 mon.a (mon.0) 1023 : audit [INF] from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:45.340252+0000 mon.a (mon.0) 1024 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:46.142276+0000 mon.a (mon.0) 1025 : audit [INF] from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:46.142345+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e207: 8 total, 8 up, 6 in 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:45.278397+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:45.339409+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.33:0/1932155325' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:45.340150+0000 mon.a (mon.0) 1023 : audit [INF] from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:45.340252+0000 mon.a (mon.0) 1024 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:46.142276+0000 mon.a (mon.0) 1025 : audit [INF] from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-02-12T16:57:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:46.142345+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e207: 8 total, 8 up, 6 in 2024-02-12T16:57:46.614 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:46.620 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:57:46.642 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:46.991 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:46.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:57:46.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:47.094 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:57:47.124 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:47.278 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:46.503639+0000 mgr.y (mgr.24481) 313 : cluster [DBG] pgmap v373: 146 pgs: 1 peering, 145 active+clean; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:47.278 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:46.503907+0000 mon.c (mon.2) 123 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T16:57:47.278 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:46.504581+0000 mon.a (mon.0) 1027 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T16:57:47.279 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:47.158874+0000 mon.a (mon.0) 1028 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T16:57:47.279 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:47.158958+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e208: 8 total, 8 up, 6 in 2024-02-12T16:57:47.323 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:47.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:47.359 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:47.428 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:47.454 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:47.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 54 16 85096 85080 98.4503 0 - 0.00990994 2024-02-12T16:57:47.555 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:47.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:46.503639+0000 mgr.y (mgr.24481) 313 : cluster [DBG] pgmap v373: 146 pgs: 1 peering, 145 active+clean; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:46.503907+0000 mon.c (mon.2) 123 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:46.504581+0000 mon.a (mon.0) 1027 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:47.158874+0000 mon.a (mon.0) 1028 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:47.158958+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e208: 8 total, 8 up, 6 in 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:46.503639+0000 mgr.y (mgr.24481) 313 : cluster [DBG] pgmap v373: 146 pgs: 1 peering, 145 active+clean; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 0 B/s, 0 objects/s recovering 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:46.503907+0000 mon.c (mon.2) 123 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:46.504581+0000 mon.a (mon.0) 1027 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T16:57:47.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:47.158874+0000 mon.a (mon.0) 1028 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T16:57:47.560 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:47.158958+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e208: 8 total, 8 up, 6 in 2024-02-12T16:57:47.659 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:47.697 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:47.726 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:47.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:47.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:47.862 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:47.868 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:48.174 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:57:48.215 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:48.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:48.325 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:48.372 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:48.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 55 16 85226 85210 96.8082 4.0625 0.0138339 0.0103255 2024-02-12T16:57:48.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:47.475695+0000 mon.a (mon.0) 1030 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:48.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:48.038275+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? v1:172.21.15.33:0/2028001882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:48.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.156058+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:48.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.160543+0000 osd.5 (osd.5) 116 : cluster [DBG] 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 206'5337 2024-02-12T16:57:48.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:48.160558+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? v1:172.21.15.33:0/2028001882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:48.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.160604+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e209: 8 total, 8 up, 6 in 2024-02-12T16:57:48.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.161058+0000 osd.5 (osd.5) 117 : cluster [DBG] 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 206'5369 2024-02-12T16:57:48.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.162861+0000 osd.0 (osd.0) 115 : cluster [DBG] 8.1 starting backfill to osd.5 from (0'0,0'0] MAX to 206'5217 2024-02-12T16:57:48.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.163308+0000 osd.2 (osd.2) 133 : cluster [DBG] 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 206'5315 2024-02-12T16:57:48.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.164314+0000 osd.2 (osd.2) 134 : cluster [DBG] 8.a starting backfill to osd.4 from (0'0,0'0] MAX to 206'5252 2024-02-12T16:57:48.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:47.475695+0000 mon.a (mon.0) 1030 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:48.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:48.038275+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? v1:172.21.15.33:0/2028001882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:48.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.156058+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:48.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.160543+0000 osd.5 (osd.5) 116 : cluster [DBG] 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 206'5337 2024-02-12T16:57:48.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:48.160558+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? v1:172.21.15.33:0/2028001882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:48.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.160604+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e209: 8 total, 8 up, 6 in 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.161058+0000 osd.5 (osd.5) 117 : cluster [DBG] 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 206'5369 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.162861+0000 osd.0 (osd.0) 115 : cluster [DBG] 8.1 starting backfill to osd.5 from (0'0,0'0] MAX to 206'5217 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.163308+0000 osd.2 (osd.2) 133 : cluster [DBG] 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 206'5315 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.164314+0000 osd.2 (osd.2) 134 : cluster [DBG] 8.a starting backfill to osd.4 from (0'0,0'0] MAX to 206'5252 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:47.475695+0000 mon.a (mon.0) 1030 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:48.038275+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? v1:172.21.15.33:0/2028001882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.156058+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.160543+0000 osd.5 (osd.5) 116 : cluster [DBG] 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 206'5337 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:48.160558+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? v1:172.21.15.33:0/2028001882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.160604+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e209: 8 total, 8 up, 6 in 2024-02-12T16:57:48.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.161058+0000 osd.5 (osd.5) 117 : cluster [DBG] 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 206'5369 2024-02-12T16:57:48.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.162861+0000 osd.0 (osd.0) 115 : cluster [DBG] 8.1 starting backfill to osd.5 from (0'0,0'0] MAX to 206'5217 2024-02-12T16:57:48.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.163308+0000 osd.2 (osd.2) 133 : cluster [DBG] 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 206'5315 2024-02-12T16:57:48.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.164314+0000 osd.2 (osd.2) 134 : cluster [DBG] 8.a starting backfill to osd.4 from (0'0,0'0] MAX to 206'5252 2024-02-12T16:57:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:57:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T16:57:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:85982 8.6 e210/176 8:65a5d65f:::benchmark_data_smithi033_105085_object85981:head v 210'5356, mlcod=210'5354)", 2024-02-12T16:57:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:49.454956+0000", 2024-02-12T16:57:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.00092770500000000002, 2024-02-12T16:57:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000983741, 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.454956+0000", 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.454956+0000", 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.454955+0000", 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8217999999999999e-05 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.454993+0000", 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1503e-05 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455004+0000", 2024-02-12T16:57:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0269999999999998e-06 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455011+0000", 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2057000000000001e-05 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455053+0000", 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8797999999999999e-05 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455072+0000", 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011574300000000001 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:85971 8.3 e210/208 8:ce9c71f2:::benchmark_data_smithi033_105085_object85970:head v 210'5395, mlcod=210'5392)", 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:49.455117+0000", 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.00076718000000000001, 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089299900000000001, 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T16:57:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455117+0000", 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455117+0000", 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455116+0000", 2024-02-12T16:57:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0094e-05 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455146+0000", 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6429999999999998e-06 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455153+0000", 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8729999999999998e-06 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455158+0000", 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113006 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455271+0000", 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3059000000000001e-05 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455294+0000", 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017703900000000001 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(8.0 210/208 [PushOp(8:0006d9b1:::benchmark_data_smithi033_105085_object2020:head, version: 184'132, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:0006d9b1:::benchmark_data_smithi033_105085_object2020:head@184'132, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:49.455170+0000", 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.000714557, 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093033000000000005, 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "queued for pg", 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455170+0000", 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455170+0000", 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455169+0000", 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.629e-05 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455205+0000", 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5143000000000001e-05 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455220+0000", 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0066e-05 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:49.455230+0000", 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0853000000000002e-05 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 3 2024-02-12T16:57:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:57:49.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 56 16 85999 85983 95.9423 48.3125 0.0298141 0.0104191 2024-02-12T16:57:49.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.163066+0000 osd.4 (osd.4) 95 : cluster [DBG] 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 206'5253 2024-02-12T16:57:49.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.167929+0000 osd.6 (osd.6) 124 : cluster [DBG] 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 206'5331 2024-02-12T16:57:49.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.349723+0000 osd.4 (osd.4) 96 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:49.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.349773+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:49.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:48.504472+0000 mgr.y (mgr.24481) 314 : cluster [DBG] pgmap v376: 146 pgs: 1 active+recovering+remapped, 5 active+remapped+backfilling, 2 clean+premerge+peered, 1 active+recovering, 137 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 34 MiB/s wr, 546 op/s; 26604/253959 objects misplaced (10.476%); 496 KiB/s, 26 objects/s recovering 2024-02-12T16:57:49.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:49.162034+0000 mon.a (mon.0) 1035 : cluster [DBG] osdmap e210: 8 total, 8 up, 6 in 2024-02-12T16:57:49.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.163066+0000 osd.4 (osd.4) 95 : cluster [DBG] 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 206'5253 2024-02-12T16:57:49.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.167929+0000 osd.6 (osd.6) 124 : cluster [DBG] 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 206'5331 2024-02-12T16:57:49.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.349723+0000 osd.4 (osd.4) 96 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:49.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.349773+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:49.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:48.504472+0000 mgr.y (mgr.24481) 314 : cluster [DBG] pgmap v376: 146 pgs: 1 active+recovering+remapped, 5 active+remapped+backfilling, 2 clean+premerge+peered, 1 active+recovering, 137 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 34 MiB/s wr, 546 op/s; 26604/253959 objects misplaced (10.476%); 496 KiB/s, 26 objects/s recovering 2024-02-12T16:57:49.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:49.162034+0000 mon.a (mon.0) 1035 : cluster [DBG] osdmap e210: 8 total, 8 up, 6 in 2024-02-12T16:57:49.880 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.163066+0000 osd.4 (osd.4) 95 : cluster [DBG] 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 206'5253 2024-02-12T16:57:49.880 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.167929+0000 osd.6 (osd.6) 124 : cluster [DBG] 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 206'5331 2024-02-12T16:57:49.880 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.349723+0000 osd.4 (osd.4) 96 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:49.880 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.349773+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:49.880 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:48.504472+0000 mgr.y (mgr.24481) 314 : cluster [DBG] pgmap v376: 146 pgs: 1 active+recovering+remapped, 5 active+remapped+backfilling, 2 clean+premerge+peered, 1 active+recovering, 137 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 34 MiB/s wr, 546 op/s; 26604/253959 objects misplaced (10.476%); 496 KiB/s, 26 objects/s recovering 2024-02-12T16:57:49.880 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:49.162034+0000 mon.a (mon.0) 1035 : cluster [DBG] osdmap e210: 8 total, 8 up, 6 in 2024-02-12T16:57:50.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:57:50.236 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:50.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:50.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:50.388 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:50.411 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:57:50.418 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:50.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:50.438 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:57:50.462 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:57:50.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 57 16 86772 86756 95.1067 48.3125 0.0155372 0.0105088 2024-02-12T16:57:50.492 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:57:50.523 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:50.528 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:57:50.559 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:50.566 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:57:50.585 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:50.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:50.611 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:57:50.632 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:57:50.633 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:57:50.633 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:57:50.634 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:57:50.634 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:57:50.634 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:57:50.634 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:57:50.634 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:57:50.634 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:57:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:50.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:50.734 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:50.935 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:50.937 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:57:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:50] "GET /metrics HTTP/1.1" 200 175857 "" "Prometheus/2.33.4" 2024-02-12T16:57:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:50.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:51.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:57:51.136 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:51.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:50.178804+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e211: 8 total, 8 up, 6 in 2024-02-12T16:57:51.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:50.376179+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.33:0/3147334697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:51.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:50.504922+0000 mgr.y (mgr.24481) 315 : cluster [DBG] pgmap v379: 145 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 5 active+remapped+backfilling, 1 clean+premerge+peered, 1 active+recovering, 136 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 45 MiB/s wr, 726 op/s; 31843/256116 objects misplaced (12.433%); 496 KiB/s, 30 objects/s recovering 2024-02-12T16:57:51.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:50.505487+0000 mon.c (mon.2) 125 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T16:57:51.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:50.507122+0000 mon.a (mon.0) 1037 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T16:57:51.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:50.178804+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e211: 8 total, 8 up, 6 in 2024-02-12T16:57:51.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:50.376179+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.33:0/3147334697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:51.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:50.504922+0000 mgr.y (mgr.24481) 315 : cluster [DBG] pgmap v379: 145 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 5 active+remapped+backfilling, 1 clean+premerge+peered, 1 active+recovering, 136 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 45 MiB/s wr, 726 op/s; 31843/256116 objects misplaced (12.433%); 496 KiB/s, 30 objects/s recovering 2024-02-12T16:57:51.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:50.505487+0000 mon.c (mon.2) 125 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T16:57:51.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:50.507122+0000 mon.a (mon.0) 1037 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T16:57:51.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 58 16 87537 87521 94.2913 47.8125 0.0445345 0.0106 2024-02-12T16:57:51.491 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:50.178804+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e211: 8 total, 8 up, 6 in 2024-02-12T16:57:51.491 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:50.376179+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.33:0/3147334697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:57:51.491 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:50.504922+0000 mgr.y (mgr.24481) 315 : cluster [DBG] pgmap v379: 145 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 5 active+remapped+backfilling, 1 clean+premerge+peered, 1 active+recovering, 136 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 45 MiB/s wr, 726 op/s; 31843/256116 objects misplaced (12.433%); 496 KiB/s, 30 objects/s recovering 2024-02-12T16:57:51.491 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:50.505487+0000 mon.c (mon.2) 125 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T16:57:51.491 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:50.507122+0000 mon.a (mon.0) 1037 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T16:57:51.724 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:57:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:51.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:57:51.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:57:51.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:51.876 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:51.913 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:51.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:52.017 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:52.058 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:52.086 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:52.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:52.171 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:52.190 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:57:52.276 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:52.323 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:57:52.328 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 6, 7] out_osds: [3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:57:52.328 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:57:52.328 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-12T16:57:52.328 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.540846 2024-02-12T16:57:52.328 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 4 0.5408463175554836 2024-02-12T16:57:52.329 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:50.686331+0000 osd.3 (osd.3) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:52.330 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:50.686387+0000 osd.3 (osd.3) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:52.330 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:51.176211+0000 mon.a (mon.0) 1038 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:57:52.330 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:51.192728+0000 mon.a (mon.0) 1039 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T16:57:52.330 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:51.192862+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e212: 8 total, 8 up, 6 in 2024-02-12T16:57:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:52.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:57:52.359 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:50.686331+0000 osd.3 (osd.3) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:52.359 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:50.686387+0000 osd.3 (osd.3) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:52.359 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:51.176211+0000 mon.a (mon.0) 1038 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:57:52.359 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:51.192728+0000 mon.a (mon.0) 1039 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T16:57:52.359 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:51.192862+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e212: 8 total, 8 up, 6 in 2024-02-12T16:57:52.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:50.686331+0000 osd.3 (osd.3) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:52.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:50.686387+0000 osd.3 (osd.3) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:52.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:51.176211+0000 mon.a (mon.0) 1038 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:57:52.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:51.192728+0000 mon.a (mon.0) 1039 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T16:57:52.360 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:51.192862+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e212: 8 total, 8 up, 6 in 2024-02-12T16:57:52.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:52.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 59 16 88315 88299 93.5173 48.625 0.0174628 0.0106876 2024-02-12T16:57:52.525 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:52.637 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:52.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:52.642 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:57:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:57:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:57:52.686 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:57:52.686 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:57:52.742 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:57:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:53.015 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:57:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:53.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:57:53.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:53.165 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:53.451 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:52.228956+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e213: 8 total, 8 up, 6 in 2024-02-12T16:57:53.451 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:52.505442+0000 mgr.y (mgr.24481) 316 : cluster [DBG] pgmap v382: 145 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 5 active+remapped+backfilling, 1 clean+premerge+peered, 1 active+recovering, 136 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 11 MiB/s wr, 179 op/s; 31843/256116 objects misplaced (12.433%); 27 B/s, 3 objects/s recovering 2024-02-12T16:57:53.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T16:57:53.473689+0000 min lat: 0.00131853 max lat: 2.15627 avg lat: 0.0107797 2024-02-12T16:57:53.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:57:53.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 60 16 89044 89028 92.7181 45.5625 0.0231579 0.0107797 2024-02-12T16:57:53.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:57:53.514 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:57:53.522 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:53.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:52.228956+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e213: 8 total, 8 up, 6 in 2024-02-12T16:57:53.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:52.505442+0000 mgr.y (mgr.24481) 316 : cluster [DBG] pgmap v382: 145 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 5 active+remapped+backfilling, 1 clean+premerge+peered, 1 active+recovering, 136 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 11 MiB/s wr, 179 op/s; 31843/256116 objects misplaced (12.433%); 27 B/s, 3 objects/s recovering 2024-02-12T16:57:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:52.228956+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e213: 8 total, 8 up, 6 in 2024-02-12T16:57:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:52.505442+0000 mgr.y (mgr.24481) 316 : cluster [DBG] pgmap v382: 145 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 5 active+remapped+backfilling, 1 clean+premerge+peered, 1 active+recovering, 136 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 11 MiB/s wr, 179 op/s; 31843/256116 objects misplaced (12.433%); 27 B/s, 3 objects/s recovering 2024-02-12T16:57:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:53.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:53.536 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:53.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:53.695 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:53.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:53.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:53.913 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:53.970 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:57:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:53.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:57:53.983 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:54.077 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:54.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:54.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:57:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:57:54.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 61 16 89768 89752 91.94 45.25 0.0156878 0.0108723 2024-02-12T16:57:54.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:53.242278+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e214: 8 total, 8 up, 6 in 2024-02-12T16:57:54.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:53.785504+0000 mgr.y (mgr.24481) 317 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:54.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:53.242278+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e214: 8 total, 8 up, 6 in 2024-02-12T16:57:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:53.785504+0000 mgr.y (mgr.24481) 317 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:54.579 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:53.242278+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e214: 8 total, 8 up, 6 in 2024-02-12T16:57:54.579 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:53.785504+0000 mgr.y (mgr.24481) 317 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:57:55.389 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:54.243388+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e215: 8 total, 8 up, 6 in 2024-02-12T16:57:55.389 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:54.505857+0000 mgr.y (mgr.24481) 318 : cluster [DBG] pgmap v385: 144 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 136 active+clean; 5.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 37140/267537 objects misplaced (13.882%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:57:55.389 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:54.506195+0000 mon.c (mon.2) 126 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T16:57:55.389 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:54.507037+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T16:57:55.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 62 16 90623 90607 91.319 53.4375 0.0214575 0.0109456 2024-02-12T16:57:55.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:54.243388+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e215: 8 total, 8 up, 6 in 2024-02-12T16:57:55.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:54.505857+0000 mgr.y (mgr.24481) 318 : cluster [DBG] pgmap v385: 144 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 136 active+clean; 5.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 37140/267537 objects misplaced (13.882%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:57:55.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:54.506195+0000 mon.c (mon.2) 126 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T16:57:55.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:54.507037+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T16:57:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:54.243388+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e215: 8 total, 8 up, 6 in 2024-02-12T16:57:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:54.505857+0000 mgr.y (mgr.24481) 318 : cluster [DBG] pgmap v385: 144 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 136 active+clean; 5.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 37140/267537 objects misplaced (13.882%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:57:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:54.506195+0000 mon.c (mon.2) 126 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T16:57:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:54.507037+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T16:57:56.132 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:57:56.132 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:57:56.182 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:57:56.197 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:56.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:56.302 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:56.363 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:56.476 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 63 16 91360 91344 90.6006 46.0625 0.0124694 0.0110322 2024-02-12T16:57:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:55.272152+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T16:57:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:55.272219+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e216: 8 total, 8 up, 6 in 2024-02-12T16:57:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:56.004932+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? v1:172.21.15.33:0/3433835981' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:57:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:56.141207+0000 mon.a (mon.0) 1048 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:56.151044+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? v1:172.21.15.33:0/3433835981' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:57:56.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:56.151096+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e217: 8 total, 8 up, 6 in 2024-02-12T16:57:56.500 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:56.504 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:55.272152+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T16:57:56.504 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:55.272219+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e216: 8 total, 8 up, 6 in 2024-02-12T16:57:56.504 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:56.004932+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? v1:172.21.15.33:0/3433835981' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:57:56.504 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:56.141207+0000 mon.a (mon.0) 1048 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:56.504 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:56.151044+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? v1:172.21.15.33:0/3433835981' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:57:56.504 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:56.151096+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e217: 8 total, 8 up, 6 in 2024-02-12T16:57:56.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:55.272152+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T16:57:56.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:55.272219+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e216: 8 total, 8 up, 6 in 2024-02-12T16:57:56.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:56.004932+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? v1:172.21.15.33:0/3433835981' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:57:56.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:56.141207+0000 mon.a (mon.0) 1048 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:57:56.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:56.151044+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? v1:172.21.15.33:0/3433835981' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:57:56.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:56.151096+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e217: 8 total, 8 up, 6 in 2024-02-12T16:57:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:56.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:56.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:57:56.669 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:56.903 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:57:56.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:57:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:56.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:57:57.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:57:57.049 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:57.084 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:57:57.096 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:57:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:57.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:57:57.182 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.4 primary-affinity to 0.540846 (8354442) 2024-02-12T16:57:57.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:57.233 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:57.302 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:56.461501+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? v1:172.21.15.33:0/640299707' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5408463175554836}]: dispatch 2024-02-12T16:57:57.302 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:56.506299+0000 mgr.y (mgr.24481) 319 : cluster [DBG] pgmap v388: 144 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 136 active+clean; 5.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 37140/267537 objects misplaced (13.882%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:57:57.302 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:57.174425+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? v1:172.21.15.33:0/640299707' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5408463175554836}]': finished 2024-02-12T16:57:57.302 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:57.174496+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e218: 8 total, 8 up, 6 in 2024-02-12T16:57:57.440 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:57:57.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 64 16 92075 92059 89.8833 44.6875 0.0125317 0.0111188 2024-02-12T16:57:57.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:56.461501+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? v1:172.21.15.33:0/640299707' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5408463175554836}]: dispatch 2024-02-12T16:57:57.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:56.506299+0000 mgr.y (mgr.24481) 319 : cluster [DBG] pgmap v388: 144 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 136 active+clean; 5.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 37140/267537 objects misplaced (13.882%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:57:57.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:57.174425+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? v1:172.21.15.33:0/640299707' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5408463175554836}]': finished 2024-02-12T16:57:57.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:57.174496+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e218: 8 total, 8 up, 6 in 2024-02-12T16:57:57.593 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:57.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:57.697 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:57:57.710 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:56.461501+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? v1:172.21.15.33:0/640299707' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5408463175554836}]: dispatch 2024-02-12T16:57:57.710 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:56.506299+0000 mgr.y (mgr.24481) 319 : cluster [DBG] pgmap v388: 144 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 136 active+clean; 5.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 37140/267537 objects misplaced (13.882%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:57:57.710 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:57.174425+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? v1:172.21.15.33:0/640299707' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5408463175554836}]': finished 2024-02-12T16:57:57.710 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:57.174496+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e218: 8 total, 8 up, 6 in 2024-02-12T16:57:57.734 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:58.310 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:57.302412+0000 osd.1 (osd.1) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:58.310 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:57.302468+0000 osd.1 (osd.1) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:58.310 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:57.405496+0000 mon.c (mon.2) 127 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:57:58.310 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:57.734898+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:58.310 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:57.918392+0000 mon.c (mon.2) 128 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:57:58.311 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:57.919271+0000 mon.c (mon.2) 129 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:57:58.311 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:57.925583+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:58.311 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:58.176847+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e219: 8 total, 8 up, 6 in 2024-02-12T16:57:58.475 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 65 16 92813 92797 89.2101 46.125 0.0281992 0.0112047 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:57.302412+0000 osd.1 (osd.1) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:57.302468+0000 osd.1 (osd.1) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:57.405496+0000 mon.c (mon.2) 127 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:57.734898+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:57.918392+0000 mon.c (mon.2) 128 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:57.919271+0000 mon.c (mon.2) 129 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:57.925583+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:58.176847+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e219: 8 total, 8 up, 6 in 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:57.302412+0000 osd.1 (osd.1) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:57.302468+0000 osd.1 (osd.1) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:57:58.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:57.405496+0000 mon.c (mon.2) 127 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:57:58.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:57.734898+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:58.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:57.918392+0000 mon.c (mon.2) 128 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:57:58.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:57.919271+0000 mon.c (mon.2) 129 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:57:58.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:57.925583+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:57:58.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:58.176847+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e219: 8 total, 8 up, 6 in 2024-02-12T16:57:58.874 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:57:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:58.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:57:58.981 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:57:59.024 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:59.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:57:59.385 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:57:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:59.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:57:59.474 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 66 16 93609 93593 88.6122 49.75 0.0216576 0.01128 2024-02-12T16:57:59.490 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:57:59.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:58.507111+0000 mgr.y (mgr.24481) 320 : cluster [DBG] pgmap v391: 143 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 136 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 715 op/s; 36939/276105 objects misplaced (13.379%); 24 MiB/s, 377 objects/s recovering 2024-02-12T16:57:59.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:58.508615+0000 mon.c (mon.2) 130 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:57:59.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:57:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:58.509577+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:57:59.500 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T16:57:59.500 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:57:59.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:58.507111+0000 mgr.y (mgr.24481) 320 : cluster [DBG] pgmap v391: 143 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 136 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 715 op/s; 36939/276105 objects misplaced (13.379%); 24 MiB/s, 377 objects/s recovering 2024-02-12T16:57:59.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:58.508615+0000 mon.c (mon.2) 130 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:57:59.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:57:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:58.509577+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:57:59.525 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:58.507111+0000 mgr.y (mgr.24481) 320 : cluster [DBG] pgmap v391: 143 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 136 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 715 op/s; 36939/276105 objects misplaced (13.379%); 24 MiB/s, 377 objects/s recovering 2024-02-12T16:57:59.525 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:58.508615+0000 mon.c (mon.2) 130 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:57:59.525 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:57:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:58.509577+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:57:59.531 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:57:59.636 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:57:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:57:59.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:57:59.743 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:59.819 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:00.017 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:00.030 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:58:00.061 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:58:00.090 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:58:00.117 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:58:00.150 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:58:00.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:58:00.221 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:58:00.251 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:58:00.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:58:00.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:58:00.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:58:00.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:58:00.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:58:00.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:58:00.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:58:00.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:58:00.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:58:00.290 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:00.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:00.397 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:00.441 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:00.476 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 67 16 94369 94353 87.9961 47.5 0.027968 0.0113578 2024-02-12T16:58:00.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:59.321740+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:58:00.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:57:59.321859+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e220: 8 total, 8 up, 6 in 2024-02-12T16:58:00.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:57:59.984904+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.33:0/2992563689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:00.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:59.321740+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:58:00.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:57:59.321859+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e220: 8 total, 8 up, 6 in 2024-02-12T16:58:00.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:57:59.984904+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.33:0/2992563689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:00.677 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:00.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:59.321740+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:58:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:57:59.321859+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e220: 8 total, 8 up, 6 in 2024-02-12T16:58:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:57:59.984904+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.33:0/2992563689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:00.783 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:00.816 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:00.819 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:58:01.031 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:58:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:00] "GET /metrics HTTP/1.1" 200 175857 "" "Prometheus/2.33.4" 2024-02-12T16:58:01.433 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:00.325433+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e221: 8 total, 8 up, 6 in 2024-02-12T16:58:01.433 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:00.507548+0000 mgr.y (mgr.24481) 321 : cluster [DBG] pgmap v394: 143 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 136 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 941 op/s; 37161/278823 objects misplaced (13.328%); 24 MiB/s, 377 objects/s recovering 2024-02-12T16:58:01.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 68 16 95065 95049 87.3417 43.5 0.0231627 0.0114428 2024-02-12T16:58:01.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:00.325433+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e221: 8 total, 8 up, 6 in 2024-02-12T16:58:01.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:00.507548+0000 mgr.y (mgr.24481) 321 : cluster [DBG] pgmap v394: 143 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 136 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 941 op/s; 37161/278823 objects misplaced (13.328%); 24 MiB/s, 377 objects/s recovering 2024-02-12T16:58:01.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:00.325433+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e221: 8 total, 8 up, 6 in 2024-02-12T16:58:01.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:00.507548+0000 mgr.y (mgr.24481) 321 : cluster [DBG] pgmap v394: 143 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 136 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 941 op/s; 37161/278823 objects misplaced (13.328%); 24 MiB/s, 377 objects/s recovering 2024-02-12T16:58:02.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 69 16 95857 95841 86.7933 49.5 0.0124922 0.0115144 2024-02-12T16:58:02.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:01.378146+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e222: 8 total, 8 up, 6 in 2024-02-12T16:58:02.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:01.378146+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e222: 8 total, 8 up, 6 in 2024-02-12T16:58:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:01.378146+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e222: 8 total, 8 up, 6 in 2024-02-12T16:58:02.918 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:58:02.918 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T16:58:02.918 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T16:58:02.918 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T16:58:02.918 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:58:02.941 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:58:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:58:03.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 70 16 96584 96568 86.2026 45.4375 0.0258238 0.0115959 2024-02-12T16:58:03.629 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:02.355331+0000 mon.a (mon.0) 1062 : cluster [DBG] osdmap e223: 8 total, 8 up, 6 in 2024-02-12T16:58:03.629 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:02.476023+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:03.629 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:02.508199+0000 mgr.y (mgr.24481) 322 : cluster [DBG] pgmap v397: 142 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 135 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 14 MiB/s wr, 226 op/s; 37161/278823 objects misplaced (13.328%) 2024-02-12T16:58:03.629 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:02.508855+0000 mon.c (mon.2) 132 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T16:58:03.629 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:02.510706+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T16:58:03.658 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:58:03.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:02.355331+0000 mon.a (mon.0) 1062 : cluster [DBG] osdmap e223: 8 total, 8 up, 6 in 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:02.476023+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:02.508199+0000 mgr.y (mgr.24481) 322 : cluster [DBG] pgmap v397: 142 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 135 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 14 MiB/s wr, 226 op/s; 37161/278823 objects misplaced (13.328%) 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:02.508855+0000 mon.c (mon.2) 132 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:02.510706+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:02.355331+0000 mon.a (mon.0) 1062 : cluster [DBG] osdmap e223: 8 total, 8 up, 6 in 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:02.476023+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:02.508199+0000 mgr.y (mgr.24481) 322 : cluster [DBG] pgmap v397: 142 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 135 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 14 MiB/s wr, 226 op/s; 37161/278823 objects misplaced (13.328%) 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:02.508855+0000 mon.c (mon.2) 132 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T16:58:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:02.510706+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T16:58:03.993 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:58:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:58:04.154 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:58:04.385 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 6, 7] out_osds: [3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:58:04.385 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:58:04.386 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-02-12T16:58:04.386 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:58:04.386 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:58:04.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-02-12T16:58:04.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 71 16 97302 97286 85.6205 44.875 0.0260448 0.0116741 2024-02-12T16:58:04.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:03.487600+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T16:58:04.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:03.487727+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e224: 8 total, 8 up, 6 in 2024-02-12T16:58:04.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:03.805654+0000 mgr.y (mgr.24481) 323 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:03.487600+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T16:58:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:03.487727+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e224: 8 total, 8 up, 6 in 2024-02-12T16:58:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:03.805654+0000 mgr.y (mgr.24481) 323 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:03.487600+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T16:58:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:03.487727+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e224: 8 total, 8 up, 6 in 2024-02-12T16:58:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:03.805654+0000 mgr.y (mgr.24481) 323 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:05.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 72 16 97981 97965 85.0208 42.4375 0.0142253 0.011757 2024-02-12T16:58:05.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:04.503470+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e225: 8 total, 8 up, 6 in 2024-02-12T16:58:05.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:04.508719+0000 mgr.y (mgr.24481) 324 : cluster [DBG] pgmap v400: 142 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 135 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 37029/290160 objects misplaced (12.762%); 25 MiB/s, 404 objects/s recovering 2024-02-12T16:58:05.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:05.347864+0000 osd.2 (osd.2) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:05.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:05.347923+0000 osd.2 (osd.2) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:06.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:04.503470+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e225: 8 total, 8 up, 6 in 2024-02-12T16:58:06.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:04.508719+0000 mgr.y (mgr.24481) 324 : cluster [DBG] pgmap v400: 142 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 135 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 37029/290160 objects misplaced (12.762%); 25 MiB/s, 404 objects/s recovering 2024-02-12T16:58:06.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:05.347864+0000 osd.2 (osd.2) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:06.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:05.347923+0000 osd.2 (osd.2) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:06.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:04.503470+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e225: 8 total, 8 up, 6 in 2024-02-12T16:58:06.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:04.508719+0000 mgr.y (mgr.24481) 324 : cluster [DBG] pgmap v400: 142 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 135 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 37029/290160 objects misplaced (12.762%); 25 MiB/s, 404 objects/s recovering 2024-02-12T16:58:06.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:05.347864+0000 osd.2 (osd.2) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:06.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:05.347923+0000 osd.2 (osd.2) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:58:06.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 73 16 98739 98723 84.5052 47.375 0.0212114 0.0118284 2024-02-12T16:58:06.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:05.540058+0000 mon.a (mon.0) 1068 : cluster [DBG] osdmap e226: 8 total, 8 up, 6 in 2024-02-12T16:58:06.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:06.173690+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e227: 8 total, 8 up, 6 in 2024-02-12T16:58:06.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:06.509348+0000 mon.c (mon.2) 133 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T16:58:06.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:06.510293+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T16:58:06.926 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:58:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T16:58:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:52165 7.5 7:a87ced8b:::benchmark_data_smithi033_81060_object52164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:02.180637+0000", 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 304.657350769, 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040851483000000001, 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52165 2024-02-12T16:58:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180637+0000", 2024-02-12T16:58:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180637+0000", 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180635+0000", 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1456000000000002e-05 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180696+0000", 2024-02-12T16:58:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4055e-05 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180710+0000", 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3990000000000001e-06 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180716+0000", 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3244999999999997e-05 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180760+0000", 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011835700000000001 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.180878+0000", 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014653499999999999 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.181025+0000", 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001759942 2024-02-12T16:58:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.182785+0000", 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024822687 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.207607+0000", 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013803553 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.221411+0000", 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0213999999999998e-05 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.221451+0000", 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7537000000000003e-05 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.221489+0000", 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040851483000000001 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:52933 7.5 7:a3163bed:::benchmark_data_smithi033_81060_object52932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:02.776319+0000", 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 304.06166890499998, 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045632915000000003, 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52933 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776319+0000", 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776319+0000", 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776317+0000", 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2389000000000003e-05 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776369+0000", 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2075e-05 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776381+0000", 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9429999999999999e-06 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776386+0000", 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3195999999999999e-05 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776419+0000", 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9982999999999995e-05 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776509+0000", 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000101128 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.776611+0000", 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015528060000000001 2024-02-12T16:58:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.778163+0000", 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012289829999999999 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779392+0000", 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042494756000000002 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.821887+0000", 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5698999999999998e-05 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.821923+0000", 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9062000000000001e-05 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.821952+0000", 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045632915000000003 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:52947 7.5 7:aefd2641:::benchmark_data_smithi033_81060_object52946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:02.779519+0000", 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 304.05846901899997, 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043052356, 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52947 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779519+0000", 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779519+0000", 2024-02-12T16:58:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779517+0000", 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0080000000000001e-05 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779577+0000", 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1793999999999999e-05 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779589+0000", 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5640000000000002e-06 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779595+0000", 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5726000000000003e-05 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779631+0000", 2024-02-12T16:58:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2948000000000006e-05 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779714+0000", 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9182e-05 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.779813+0000", 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013416700000000001 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.781155+0000", 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022305290000000002 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.783386+0000", 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.039099086999999998 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.822485+0000", 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8486000000000003e-05 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.822533+0000", 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8021000000000001e-05 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:02.822571+0000", 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043052356 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:80708 7.5 7:a793f08c:::benchmark_data_smithi033_81060_object80707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:29.050600+0000", 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 277.78738774700003, 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057351045000000003, 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 80708 2024-02-12T16:58:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050600+0000", 2024-02-12T16:58:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050600+0000", 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050598+0000", 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3399000000000002e-05 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050651+0000", 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2693e-05 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050664+0000", 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8950000000000001e-06 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050671+0000", 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6180999999999997e-05 2024-02-12T16:58:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050717+0000", 2024-02-12T16:58:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0001003 2024-02-12T16:58:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050817+0000", 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112668 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050930+0000", 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021345280000000001 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053065+0000", 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024767410000000002 2024-02-12T16:58:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.055541+0000", 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052323124999999998 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107864+0000", 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3448000000000001e-05 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107918+0000", 2024-02-12T16:58:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3254999999999999e-05 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107951+0000", 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057351045000000003 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:80710 7.5 7:a5b76bb0:::benchmark_data_smithi033_81060_object80709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:29.053581+0000", 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 277.78440652199998, 2024-02-12T16:58:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062171000999999997, 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 80710 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053581+0000", 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053581+0000", 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053580+0000", 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9789000000000001e-05 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053629+0000", 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6770000000000001e-06 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053638+0000", 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5739999999999997e-06 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053643+0000", 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3254999999999998e-05 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053686+0000", 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013029299999999999 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053816+0000", 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018151500000000001 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053998+0000", 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016970679999999999 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.055695+0000", 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053556299999999997 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.056230+0000", 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059463378999999997 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115694+0000", 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5593999999999998e-05 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115729+0000", 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2915999999999999e-05 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115752+0000", 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062171000999999997 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:99072 7.5 7:a97cf8b5:::benchmark_data_smithi033_81060_object99071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:49.556027+0000", 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 257.28196101700001, 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038903828000000001, 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 99072 2024-02-12T16:58:06.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556027+0000", 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556027+0000", 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556024+0000", 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5201999999999999e-05 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556069+0000", 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9437999999999999e-05 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556088+0000", 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1380000000000007e-06 2024-02-12T16:58:06.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556096+0000", 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6246999999999999e-05 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556153+0000", 2024-02-12T16:58:06.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138104 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556291+0000", 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000154594 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.556445+0000", 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029249580000000001 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.559370+0000", 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01541178 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.574782+0000", 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020090665000000001 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.594873+0000", 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4709000000000003e-05 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.594907+0000", 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3195999999999999e-05 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:49.594931+0000", 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038903828000000001 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:104495 7.5 7:a1d02bf9:::benchmark_data_smithi033_81060_object104494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-12T16:58:06.968 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:56.075999+0000", 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.76198911099999, 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042203978000000003, 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 104495 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.075999+0000", 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.075999+0000", 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999919 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.075991+0000", 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3384000000000003e-05 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076044+0000", 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2148e-05 2024-02-12T16:58:06.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076056+0000", 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8499999999999999e-06 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076062+0000", 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1137999999999997e-05 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076113+0000", 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014489 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076258+0000", 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017179599999999999 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.076430+0000", 2024-02-12T16:58:06.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003172008 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.079602+0000", 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067976299999999998 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.086399+0000", 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031726231000000001 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.118126+0000", 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7032000000000001e-05 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.118163+0000", 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0058999999999997e-05 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:56.118203+0000", 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042203978000000003 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:107212 7.5 7:ae1c802b:::benchmark_data_smithi033_81060_object107211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:59.278400+0000", 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 247.55958761799999, 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043589808000000001, 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 107212 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278400+0000", 2024-02-12T16:58:06.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278400+0000", 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278398+0000", 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1880999999999998e-05 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278480+0000", 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0026000000000001e-05 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278500+0000", 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6369999999999999e-06 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278508+0000", 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4691999999999998e-05 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278552+0000", 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013292800000000001 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278685+0000", 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019186600000000001 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278877+0000", 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0046846479999999996 2024-02-12T16:58:06.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.283562+0000", 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012374098 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.295936+0000", 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025991435 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.321927+0000", 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6848999999999999e-05 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.321964+0000", 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5956e-05 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.321990+0000", 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043589808000000001 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.983 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:107213 7.5 7:ab05a132:::benchmark_data_smithi033_81060_object107212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:59.278791+0000", 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 247.559197052, 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043341330999999997, 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 107213 2024-02-12T16:58:06.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278791+0000", 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278791+0000", 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278789+0000", 2024-02-12T16:58:06.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2889000000000002e-05 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278832+0000", 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278843+0000", 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6170000000000003e-06 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.278850+0000", 2024-02-12T16:58:06.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025485339999999999 2024-02-12T16:58:06.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.281399+0000", 2024-02-12T16:58:06.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000127766 2024-02-12T16:58:06.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.281526+0000", 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016311200000000001 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.281689+0000", 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020537590000000001 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.283743+0000", 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012221242 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.295964+0000", 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.026105811999999999 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.322070+0000", 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7021999999999999e-05 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.322107+0000", 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4808999999999999e-05 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:59.322132+0000", 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043341330999999997 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.990 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:143742 7.5 7:afb72632:::benchmark_data_smithi033_81060_object33290:head [delete] snapc 0=[] ondisk+write+known_if_redirected e136)", 2024-02-12T16:58:06.991 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:54:39.605709+0000", 2024-02-12T16:58:06.991 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 207.23227927100001, 2024-02-12T16:58:06.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037265803, 2024-02-12T16:58:06.991 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.991 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 143742 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605709+0000", 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605709+0000", 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:06.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605707+0000", 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7029999999999995e-06 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605716+0000", 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5427999999999999e-05 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605731+0000", 2024-02-12T16:58:06.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1840000000000004e-06 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.605737+0000", 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0085558779999999994 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.614293+0000", 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000139746 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:06.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.614433+0000", 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000147718 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.614581+0000", 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0032810280000000001 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.617862+0000", 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022027228999999999 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.639889+0000", 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030113649999999998 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.642900+0000", 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5810999999999999e-05 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.642946+0000", 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8206000000000001e-05 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:06.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:54:39.642974+0000", 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037265803 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:157562 7.5 7:abd29c58:::benchmark_data_smithi033_81060_object47110:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:55:00.723417+0000", 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 186.11457040100001, 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048460371000000002, 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:06.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 157562 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723417+0000", 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723417+0000", 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723415+0000", 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8360000000000004e-06 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723425+0000", 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8109999999999997e-06 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723435+0000", 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4220000000000002e-06 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:06.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723439+0000", 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5435000000000001e-05 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723485+0000", 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013159099999999999 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723616+0000", 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012698599999999999 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723743+0000", 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017941299999999999 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.725537+0000", 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0089994559999999994 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.734537+0000", 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.03725999 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771797+0000", 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7404000000000002e-05 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771844+0000", 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3670000000000001e-05 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771878+0000", 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048460371000000002 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:157570 7.5 7:a07ab10b:::benchmark_data_smithi033_81060_object47118:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:55:00.731444+0000", 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 186.10654423599999, 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040539275, 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 157570 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731444+0000", 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731444+0000", 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731442+0000", 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2160000000000006e-06 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731450+0000", 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3397999999999999e-05 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731463+0000", 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1909999999999997e-06 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731469+0000", 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7744999999999999e-05 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731507+0000", 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8629999999999998e-05 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731586+0000", 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1322000000000004e-05 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.731677+0000", 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003628643 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.735306+0000", 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000358597 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.735664+0000", 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.036254190999999998 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771918+0000", 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9085000000000003e-05 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771958+0000", 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5320000000000002e-05 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771983+0000", 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040539275 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:159169 7.5 7:a0bc834f:::benchmark_data_smithi033_81060_object48717:head [delete] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-02-12T16:58:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:55:02.971073+0000", 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.866914853, 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038733267000000002, 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 159169 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.971073+0000", 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.971073+0000", 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.971072+0000", 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0249999999999998e-06 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.971075+0000", 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9580000000000007e-06 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.971083+0000", 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2189999999999998e-06 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.971087+0000", 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002679982 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.973767+0000", 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106339 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.973874+0000", 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011950100000000001 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.973993+0000", 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0041020029999999999 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.978095+0000", 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018784856999999999 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:02.996880+0000", 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012827710000000001 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:03.009708+0000", 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2175000000000001e-05 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:03.009760+0000", 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6167e-05 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:03.009806+0000", 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038733267000000002 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:159920 7.5 7:a72cdfe9:::benchmark_data_smithi033_81060_object49468:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:55:04.105624+0000", 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 182.73236418100001, 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.041119700000000002, 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:07.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 159920 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105624+0000", 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105624+0000", 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105621+0000", 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9850000000000001e-06 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105631+0000", 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1688999999999999e-05 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105643+0000", 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1580000000000002e-06 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105649+0000", 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5781000000000001e-05 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105695+0000", 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120828 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105816+0000", 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000142979 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:07.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.105959+0000", 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002346225 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.108305+0000", 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010144540000000001 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.118450+0000", 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028183440000000001 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.146633+0000", 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1953999999999998e-05 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.146685+0000", 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8334000000000001e-05 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:04.146743+0000", 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.041119700000000002 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:160866 7.5 7:adc0022e:::benchmark_data_smithi033_81060_object50414:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:55:05.432317+0000", 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.40567104600001, 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.039259240000000001, 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 160866 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432317+0000", 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432317+0000", 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432314+0000", 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3469999999999999e-06 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432323+0000", 2024-02-12T16:58:07.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5461999999999999e-05 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432338+0000", 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1350000000000003e-06 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432345+0000", 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9499000000000002e-05 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432395+0000", 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000110782 2024-02-12T16:58:07.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432505+0000", 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012337000000000001 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.432629+0000", 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037196920000000001 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.436348+0000", 2024-02-12T16:58:07.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011849133 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.448198+0000", 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023285091000000001 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.471483+0000", 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9920999999999999e-05 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.471533+0000", 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3362000000000003e-05 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:05.471576+0000", 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.039259240000000001 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:198598 7.5 7:ac91467b:::benchmark_data_smithi033_81060_object88146:head [delete] snapc 0=[] ondisk+write+known_if_redirected e161)", 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:55:48.134072+0000", 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.70391560799999, 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038717227999999999, 2024-02-12T16:58:07.025 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 198598 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.134072+0000", 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.134072+0000", 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.134071+0000", 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2040000000000002e-06 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.134077+0000", 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.134087+0000", 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1759e-05 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.134099+0000", 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022065409999999998 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.136306+0000", 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114388 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.136420+0000", 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000143335 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.136563+0000", 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025526619999999998 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T16:58:07.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.139116+0000", 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058619900000000001 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.144978+0000", 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.027733844000000001 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.172712+0000", 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7706000000000001e-05 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.172750+0000", 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9792999999999997e-05 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:48.172789+0000", 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038717227999999999 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:82097 8.c e203/176 8:3316d78f:::benchmark_data_smithi033_105085_object82096:head v 203'5192, mlcod=203'5187)", 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:43.240223+0000", 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.597764433999998, 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048025215000000003, 2024-02-12T16:58:07.031 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240223+0000", 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240223+0000", 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240223+0000", 2024-02-12T16:58:07.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4352e-05 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240247+0000", 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7869999999999999e-06 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240251+0000", 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6129999999999998e-06 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240254+0000", 2024-02-12T16:58:07.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001769891 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242023+0000", 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8575000000000001e-05 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242042+0000", 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046170577999999997 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288213+0000", 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5944e-05 2024-02-12T16:58:07.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288249+0000", 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048025215000000003 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:82098 8.c e203/176 8:38294761:::benchmark_data_smithi033_105085_object82097:head v 203'5193, mlcod=203'5187)", 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:43.240257+0000", 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.597730566999999, 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048015358000000001, 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.035 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240257+0000", 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240257+0000", 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240257+0000", 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1943e-05 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240279+0000", 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.847e-06 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240282+0000", 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1069999999999999e-06 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240284+0000", 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035635609999999998 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.243847+0000", 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7524999999999999e-05 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.243865+0000", 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.044389989999999997 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288255+0000", 2024-02-12T16:58:07.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7643999999999999e-05 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288273+0000", 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048015358000000001 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:82102 8.6 e203/176 8:69d3ce3a:::benchmark_data_smithi033_105085_object82101:head v 203'5117, mlcod=203'5115)", 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:43.242736+0000", 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.595252064, 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042300243000000001, 2024-02-12T16:58:07.039 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242736+0000", 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242736+0000", 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967296 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242736+0000", 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0399999999999999e-05 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242776+0000", 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7509999999999997e-06 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242785+0000", 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0299999999999999e-06 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242791+0000", 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1863999999999998e-05 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242833+0000", 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3010000000000001e-05 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242846+0000", 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042158538000000002 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.285004+0000", 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1838999999999998e-05 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.285036+0000", 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042300243000000001 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:82101 8.c e204/176 8:3d2526b9:::benchmark_data_smithi033_105085_object82100:head v 204'5194, mlcod=203'5187)", 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:43.246251+0000", 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.591737168000002, 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042136254999999997, 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.246251+0000", 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.246251+0000", 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.246249+0000", 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8566000000000002e-05 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.246298+0000", 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038512370999999997 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.284810+0000", 2024-02-12T16:58:07.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5699999999999999e-06 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.284820+0000", 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012983800000000001 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.284949+0000", 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9561000000000001e-05 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.284969+0000", 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033974069999999999 2024-02-12T16:58:07.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288366+0000", 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0460999999999999e-05 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288387+0000", 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042136254999999997 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T16:58:07.047 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T16:58:07.048 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:58:07.048 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:05.540058+0000 mon.a (mon.0) 1068 : cluster [DBG] osdmap e226: 8 total, 8 up, 6 in 2024-02-12T16:58:07.048 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:06.173690+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e227: 8 total, 8 up, 6 in 2024-02-12T16:58:07.048 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:06.509348+0000 mon.c (mon.2) 133 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T16:58:07.048 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:06.510293+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T16:58:07.048 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:05.540058+0000 mon.a (mon.0) 1068 : cluster [DBG] osdmap e226: 8 total, 8 up, 6 in 2024-02-12T16:58:07.049 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:06.173690+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e227: 8 total, 8 up, 6 in 2024-02-12T16:58:07.049 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:06.509348+0000 mon.c (mon.2) 133 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T16:58:07.049 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:06.510293+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T16:58:07.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 74 16 99497 99481 84.0034 47.375 0.0144752 0.0118972 2024-02-12T16:58:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:06.509068+0000 mgr.y (mgr.24481) 325 : cluster [DBG] pgmap v403: 141 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 134 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 37029/290160 objects misplaced (12.762%); 25 MiB/s, 404 objects/s recovering 2024-02-12T16:58:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:07.214660+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T16:58:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:07.214732+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e228: 8 total, 8 up, 6 in 2024-02-12T16:58:08.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:06.509068+0000 mgr.y (mgr.24481) 325 : cluster [DBG] pgmap v403: 141 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 134 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 37029/290160 objects misplaced (12.762%); 25 MiB/s, 404 objects/s recovering 2024-02-12T16:58:08.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:07.214660+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T16:58:08.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:07.214732+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e228: 8 total, 8 up, 6 in 2024-02-12T16:58:08.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:06.509068+0000 mgr.y (mgr.24481) 325 : cluster [DBG] pgmap v403: 141 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 134 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 37029/290160 objects misplaced (12.762%); 25 MiB/s, 404 objects/s recovering 2024-02-12T16:58:08.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:07.214660+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T16:58:08.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:07.214732+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e228: 8 total, 8 up, 6 in 2024-02-12T16:58:08.175 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:08.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:08.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:08.356 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:08.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 75 16 100269 100253 83.5267 48.25 0.0206439 0.0119674 2024-02-12T16:58:08.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:58:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:08.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:08.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:08.786 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:09.151 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:09.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:09.250 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:58:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T16:58:09.250 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:58:09.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:09.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 76 16 100961 100945 82.9968 43.25 0.0120003 0.0120426 2024-02-12T16:58:09.523 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:09.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:08.226861+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e229: 8 total, 8 up, 6 in 2024-02-12T16:58:09.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:08.509733+0000 mgr.y (mgr.24481) 326 : cluster [DBG] pgmap v406: 141 pgs: 2 clean+premerge+peered, 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 684 op/s; 36707/298362 objects misplaced (12.303%); 23 MiB/s, 372 objects/s recovering 2024-02-12T16:58:09.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:08.226861+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e229: 8 total, 8 up, 6 in 2024-02-12T16:58:09.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:08.509733+0000 mgr.y (mgr.24481) 326 : cluster [DBG] pgmap v406: 141 pgs: 2 clean+premerge+peered, 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 684 op/s; 36707/298362 objects misplaced (12.303%); 23 MiB/s, 372 objects/s recovering 2024-02-12T16:58:09.530 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:08.226861+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e229: 8 total, 8 up, 6 in 2024-02-12T16:58:09.530 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:08.509733+0000 mgr.y (mgr.24481) 326 : cluster [DBG] pgmap v406: 141 pgs: 2 clean+premerge+peered, 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 684 op/s; 36707/298362 objects misplaced (12.303%); 23 MiB/s, 372 objects/s recovering 2024-02-12T16:58:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:09.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:09.632 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:09.672 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:10.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:58:10.478 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 77 16 101652 101636 82.4798 43.1875 0.00645954 0.0121193 2024-02-12T16:58:10.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:09.238000+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e230: 8 total, 8 up, 6 in 2024-02-12T16:58:10.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:09.238000+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e230: 8 total, 8 up, 6 in 2024-02-12T16:58:10.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:09.238000+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e230: 8 total, 8 up, 6 in 2024-02-12T16:58:11.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:58:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:10] "GET /metrics HTTP/1.1" 200 175916 "" "Prometheus/2.33.4" 2024-02-12T16:58:11.291 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:10.244743+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e231: 8 total, 8 up, 6 in 2024-02-12T16:58:11.291 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:10.510141+0000 mgr.y (mgr.24481) 327 : cluster [DBG] pgmap v409: 140 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 920 op/s; 36949/301197 objects misplaced (12.267%); 23 MiB/s, 372 objects/s recovering 2024-02-12T16:58:11.291 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:10.510463+0000 mon.c (mon.2) 134 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T16:58:11.291 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:10.512707+0000 mon.a (mon.0) 1076 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T16:58:11.477 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 78 16 102442 102426 82.0554 49.375 0.0279636 0.0121823 2024-02-12T16:58:11.637 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:11.641 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:10.244743+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e231: 8 total, 8 up, 6 in 2024-02-12T16:58:11.642 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:10.510141+0000 mgr.y (mgr.24481) 327 : cluster [DBG] pgmap v409: 140 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 920 op/s; 36949/301197 objects misplaced (12.267%); 23 MiB/s, 372 objects/s recovering 2024-02-12T16:58:11.642 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:10.510463+0000 mon.c (mon.2) 134 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T16:58:11.642 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:10.512707+0000 mon.a (mon.0) 1076 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T16:58:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:11.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:11.644 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:10.244743+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e231: 8 total, 8 up, 6 in 2024-02-12T16:58:11.644 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:10.510141+0000 mgr.y (mgr.24481) 327 : cluster [DBG] pgmap v409: 140 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 920 op/s; 36949/301197 objects misplaced (12.267%); 23 MiB/s, 372 objects/s recovering 2024-02-12T16:58:11.644 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:10.510463+0000 mon.c (mon.2) 134 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T16:58:11.645 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:10.512707+0000 mon.a (mon.0) 1076 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T16:58:11.744 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:11.870 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:11.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:11.976 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:12.021 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:12.160 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:12.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:12.264 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:12.287 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:12.346 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:12.376 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:12.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:12.478 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 79 16 103202 103186 81.618 47.5 0.00659423 0.0122468 2024-02-12T16:58:12.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:12.539 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:12.583 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:12.586 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:11.282153+0000 mon.a (mon.0) 1077 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T16:58:12.587 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:11.282215+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e232: 8 total, 8 up, 6 in 2024-02-12T16:58:12.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:11.282153+0000 mon.a (mon.0) 1077 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T16:58:12.587 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:11.282215+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e232: 8 total, 8 up, 6 in 2024-02-12T16:58:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:12.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:12.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:12.730 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:12.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:11.282153+0000 mon.a (mon.0) 1077 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T16:58:12.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:11.282215+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e232: 8 total, 8 up, 6 in 2024-02-12T16:58:12.744 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:58:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:58:12.835 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:12.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:12.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:13.036 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:13.065 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:13.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:13.137 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:13.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:13.233 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:13.266 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:13.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:13.313 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:12.291354+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e233: 8 total, 8 up, 6 in 2024-02-12T16:58:13.313 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:12.510644+0000 mgr.y (mgr.24481) 328 : cluster [DBG] pgmap v412: 140 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 15 MiB/s wr, 236 op/s; 36949/301197 objects misplaced (12.267%) 2024-02-12T16:58:13.313 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:12.624135+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.33:0/2536970984' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:13.325 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T16:58:13.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:13.434 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:13.467 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:13.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:13.478 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T16:58:13.477223+0000 min lat: 0.00131853 max lat: 2.15627 avg lat: 0.01231 2024-02-12T16:58:13.478 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:58:13.478 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 80 16 103975 103959 81.2016 48.3125 0.0232665 0.01231 2024-02-12T16:58:13.573 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:13.663 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:13.690 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:13.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:13.726 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:12.291354+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e233: 8 total, 8 up, 6 in 2024-02-12T16:58:13.726 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:12.510644+0000 mgr.y (mgr.24481) 328 : cluster [DBG] pgmap v412: 140 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 15 MiB/s wr, 236 op/s; 36949/301197 objects misplaced (12.267%) 2024-02-12T16:58:13.726 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:12.624135+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.33:0/2536970984' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:13.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:12.291354+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e233: 8 total, 8 up, 6 in 2024-02-12T16:58:13.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:12.510644+0000 mgr.y (mgr.24481) 328 : cluster [DBG] pgmap v412: 140 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 6 active+remapped+backfilling, 132 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 15 MiB/s wr, 236 op/s; 36949/301197 objects misplaced (12.267%) 2024-02-12T16:58:13.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:12.624135+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.33:0/2536970984' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:13.797 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:13.838 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:13.944 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:13.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:13.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:58:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:58:14.051 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:14.097 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:14.136 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:14.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:14.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:14.323 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:14.372 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:14.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:14.380 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:13.303524+0000 mon.a (mon.0) 1081 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:58:14.380 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:13.317381+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.33:0/2536970984' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:58:14.380 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:13.317431+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e234: 8 total, 8 up, 6 in 2024-02-12T16:58:14.380 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:13.825403+0000 mgr.y (mgr.24481) 329 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:14.478 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 81 16 104748 104732 80.7956 48.3125 0.0140351 0.0123722 2024-02-12T16:58:14.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:14.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:13.303524+0000 mon.a (mon.0) 1081 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:58:14.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:13.317381+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.33:0/2536970984' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:58:14.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:13.317431+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e234: 8 total, 8 up, 6 in 2024-02-12T16:58:14.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:13.825403+0000 mgr.y (mgr.24481) 329 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:13.303524+0000 mon.a (mon.0) 1081 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:58:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:13.317381+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.33:0/2536970984' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:58:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:13.317431+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e234: 8 total, 8 up, 6 in 2024-02-12T16:58:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:13.825403+0000 mgr.y (mgr.24481) 329 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:14.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:14.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:14.745 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:14.850 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:14.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:14.956 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:14.994 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:15.021 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 6, 7] out_osds: [3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:58:15.021 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:58:15.022 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-12T16:58:15.022 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T16:58:15.214 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:58:15.214 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:58:15.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.214 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:105274 8.1 8:871a6d2a:::benchmark_data_smithi033_105085_object105273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e235)", 2024-02-12T16:58:15.214 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:15.166505+0000", 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.021704867999999999, 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021746077999999999, 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 105274 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.215 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166505+0000", 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166505+0000", 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166504+0000", 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4219999999999999e-05 2024-02-12T16:58:15.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166528+0000", 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.043e-05 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166538+0000", 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0279999999999997e-06 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:15.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166544+0000", 2024-02-12T16:58:15.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026312340000000001 2024-02-12T16:58:15.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:15.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.169175+0000", 2024-02-12T16:58:15.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000126069 2024-02-12T16:58:15.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.169301+0000", 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021302699999999999 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.169515+0000", 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0064917849999999999 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.176006+0000", 2024-02-12T16:58:15.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039252319999999999 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.179932+0000", 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002286719 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.182218+0000", 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015713014000000001 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:15.220 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:105276 8.1 8:8bb1c1ca:::benchmark_data_smithi033_105085_object105275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e235)", 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:15.166577+0000", 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.021633049000000001, 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021809574000000002, 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 105276 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166577+0000", 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166577+0000", 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166576+0000", 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0049e-05 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166606+0000", 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6209999999999998e-06 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166613+0000", 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3599999999999998e-06 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.166617+0000", 2024-02-12T16:58:15.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056623109999999997 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.172279+0000", 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000169109 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.172448+0000", 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019702399999999999 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.172645+0000", 2024-02-12T16:58:15.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035889720000000002 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.176234+0000", 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006028729 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.182263+0000", 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015686130999999999 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:105289 8.1 8:88928b8a:::benchmark_data_smithi033_105085_object105288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e235)", 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:15.181739+0000", 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0064714580000000002, 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067602260000000003, 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 105289 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.181739+0000", 2024-02-12T16:58:15.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.181739+0000", 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.181737+0000", 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5239999999999999e-05 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.181762+0000", 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3582e-05 2024-02-12T16:58:15.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.181776+0000", 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6220999999999999e-05 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.181792+0000", 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049799100000000004 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.182290+0000", 2024-02-12T16:58:15.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017877299999999999 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.182469+0000", 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018098800000000001 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.182650+0000", 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00091085899999999997 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.35042.0:105286 8.9 e235/176 8:9995b676:::benchmark_data_smithi033_105085_object105285:head v 235'6528, mlcod=235'6526)", 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:15.183241+0000", 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0049692219999999997, 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053512000000000004, 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183241+0000", 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183241+0000", 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183239+0000", 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2786000000000005e-05 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183302+0000", 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2105e-05 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:15.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183314+0000", 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9710000000000001e-06 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183320+0000", 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023680110000000002 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.185688+0000", 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9752e-05 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.185708+0000", 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002467056 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.35042.0:105280 8.2 e235/176 8:45aabf53:::benchmark_data_smithi033_105085_object105279:head v 235'6516, mlcod=235'6514)", 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:15.183585+0000", 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0046246630000000002, 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.005077565, 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:15.233 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183585+0000", 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183585+0000", 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183584+0000", 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6474999999999997e-05 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183641+0000", 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3190000000000001e-06 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183650+0000", 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9040000000000002e-06 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:15.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.183654+0000", 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004306931 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187961+0000", 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1719999999999999e-05 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187983+0000", 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043973140000000003 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.236 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.35042.0:105291 8.9 e235/176 8:9c5572b9:::benchmark_data_smithi033_105085_object105290:head v 235'6529, mlcod=235'6526)", 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:15.185924+0000", 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0022865849999999998, 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028146759999999999, 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:15.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.185924+0000", 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.185924+0000", 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:15.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.185922+0000", 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4044e-05 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.185966+0000", 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2571999999999999e-05 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.185979+0000", 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9089999999999997e-06 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.185985+0000", 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021521639999999998 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.188137+0000", 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6443000000000002e-05 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.188153+0000", 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022298259999999999 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.240 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:105274 8.1 e235/208)", 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:15.187656+0000", 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.00055424900000000004, 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001150901, 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187656+0000", 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187656+0000", 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187654+0000", 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2057e-05 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187666+0000", 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2061e-05 2024-02-12T16:58:15.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187678+0000", 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2650000000000002e-06 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187684+0000", 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8430000000000001e-05 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:105276 8.1 e235/208)", 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:15.187689+0000", 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.00052087100000000001, 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012058010000000001, 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187689+0000", 2024-02-12T16:58:15.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:15.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:15.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187689+0000", 2024-02-12T16:58:15.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:58:15.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187689+0000", 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1259999999999998e-06 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187692+0000", 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2749999999999997e-06 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187696+0000", 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.399e-06 2024-02-12T16:58:15.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:15.187700+0000", 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0349999999999999e-05 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:58:15.247 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 8 2024-02-12T16:58:15.248 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:58:15.479 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 82 16 105511 105495 80.3909 47.6875 0.0278384 0.0124346 2024-02-12T16:58:15.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:14.342186+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e235: 8 total, 8 up, 6 in 2024-02-12T16:58:15.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:14.355281+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:15.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:14.355332+0000 osd.7 (osd.7) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:15.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:14.511092+0000 mgr.y (mgr.24481) 330 : cluster [DBG] pgmap v415: 139 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 931 op/s; 36509/312363 objects misplaced (11.688%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:15.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:14.511446+0000 mon.c (mon.2) 135 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T16:58:15.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:14.513337+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T16:58:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:14.342186+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e235: 8 total, 8 up, 6 in 2024-02-12T16:58:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:14.355281+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:14.355332+0000 osd.7 (osd.7) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:14.511092+0000 mgr.y (mgr.24481) 330 : cluster [DBG] pgmap v415: 139 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 931 op/s; 36509/312363 objects misplaced (11.688%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:14.511446+0000 mon.c (mon.2) 135 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T16:58:15.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:14.513337+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T16:58:15.552 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:15.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:15.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:15.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:14.342186+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e235: 8 total, 8 up, 6 in 2024-02-12T16:58:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:14.355281+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:14.355332+0000 osd.7 (osd.7) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:14.511092+0000 mgr.y (mgr.24481) 330 : cluster [DBG] pgmap v415: 139 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 931 op/s; 36509/312363 objects misplaced (11.688%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:14.511446+0000 mon.c (mon.2) 135 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T16:58:15.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:14.513337+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T16:58:15.742 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:15.792 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:15.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:15.899 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:15.947 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:15.980 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:15.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:16.089 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:16.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:16.238 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:16.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:16.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:16.392 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:16.472 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:16.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:16.479 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 83 16 106267 106251 79.9916 47.25 0.0159705 0.0124963 2024-02-12T16:58:16.494 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:15.362578+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T16:58:16.494 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:15.362631+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e236: 8 total, 8 up, 6 in 2024-02-12T16:58:16.494 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:16.153421+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e237: 8 total, 8 up, 6 in 2024-02-12T16:58:16.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:15.362578+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T16:58:16.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:15.362631+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e236: 8 total, 8 up, 6 in 2024-02-12T16:58:16.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:16.153421+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e237: 8 total, 8 up, 6 in 2024-02-12T16:58:16.577 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:16.615 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:16.644 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:16.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:16.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:15.362578+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T16:58:16.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:15.362631+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e236: 8 total, 8 up, 6 in 2024-02-12T16:58:16.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:16.153421+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e237: 8 total, 8 up, 6 in 2024-02-12T16:58:16.759 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:16.796 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:16.878 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:58:17.059 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:17.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:17.095 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:58:17.166 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:17.264 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:17.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:17.369 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:17.377 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:17.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:58:17.410 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:16.514465+0000 mgr.y (mgr.24481) 331 : cluster [DBG] pgmap v418: 139 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 930 op/s; 36509/312363 objects misplaced (11.688%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:17.410 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:17.176067+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e238: 8 total, 8 up, 6 in 2024-02-12T16:58:17.410 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:17.352218+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.33:0/3615989546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:17.410 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:17.434 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:17.440 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:58:17.462 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:17.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:17.479 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 84 16 107016 107000 79.5966 46.8125 0.0239337 0.0125583 2024-02-12T16:58:17.485 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:58:17.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:16.514465+0000 mgr.y (mgr.24481) 331 : cluster [DBG] pgmap v418: 139 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 930 op/s; 36509/312363 objects misplaced (11.688%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:17.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:17.176067+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e238: 8 total, 8 up, 6 in 2024-02-12T16:58:17.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:17.352218+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.33:0/3615989546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:17.515 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:58:17.550 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:58:17.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:17.592 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:58:17.615 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:17.623 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:58:17.665 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:58:17.668 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:58:17.668 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:58:17.669 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:58:17.669 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:58:17.669 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:58:17.669 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:58:17.669 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:58:17.669 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:58:17.680 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:16.514465+0000 mgr.y (mgr.24481) 331 : cluster [DBG] pgmap v418: 139 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 930 op/s; 36509/312363 objects misplaced (11.688%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:17.680 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:17.176067+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e238: 8 total, 8 up, 6 in 2024-02-12T16:58:17.680 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:17.352218+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.33:0/3615989546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:17.684 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:17.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:17.792 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:17.898 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:17.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:18.007 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:18.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:18.216 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:18.246 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:18.451 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:18.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:18.479 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 85 16 107747 107731 79.1977 45.6875 0.0284952 0.0126212 2024-02-12T16:58:18.499 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:17.486033+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:18.499 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:18.177433+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e239: 8 total, 8 up, 6 in 2024-02-12T16:58:18.557 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:18.693 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:18.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:17.486033+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:18.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:18.177433+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e239: 8 total, 8 up, 6 in 2024-02-12T16:58:18.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:17.486033+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:18.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:18.177433+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e239: 8 total, 8 up, 6 in 2024-02-12T16:58:19.479 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 86 16 108546 108530 78.8574 49.9375 0.0475871 0.0126763 2024-02-12T16:58:19.690 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:18.515236+0000 mgr.y (mgr.24481) 332 : cluster [DBG] pgmap v421: 138 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 131 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 711 op/s; 35989/320895 objects misplaced (11.215%); 25 MiB/s, 394 objects/s recovering 2024-02-12T16:58:19.695 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:18.515236+0000 mgr.y (mgr.24481) 332 : cluster [DBG] pgmap v421: 138 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 131 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 711 op/s; 35989/320895 objects misplaced (11.215%); 25 MiB/s, 394 objects/s recovering 2024-02-12T16:58:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:18.515236+0000 mgr.y (mgr.24481) 332 : cluster [DBG] pgmap v421: 138 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 131 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 711 op/s; 35989/320895 objects misplaced (11.215%); 25 MiB/s, 394 objects/s recovering 2024-02-12T16:58:20.480 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 87 16 109362 109346 78.5362 51 0.0239939 0.0127281 2024-02-12T16:58:20.561 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:20.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:20.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:20.677 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:20.739 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:20.520727+0000 mon.a (mon.0) 1092 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:58:20.739 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:20.520754+0000 mon.a (mon.0) 1093 : cluster [INF] Cluster is now healthy 2024-02-12T16:58:20.742 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:20.520727+0000 mon.a (mon.0) 1092 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:58:20.742 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:20.520754+0000 mon.a (mon.0) 1093 : cluster [INF] Cluster is now healthy 2024-02-12T16:58:20.755 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:20.790 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:20.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:20.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:20.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:20.968 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:20.520727+0000 mon.a (mon.0) 1092 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:58:20.968 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:20.520754+0000 mon.a (mon.0) 1093 : cluster [INF] Cluster is now healthy 2024-02-12T16:58:20.970 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:21.013 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:21.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:21.030 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:58:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:20] "GET /metrics HTTP/1.1" 200 175927 "" "Prometheus/2.33.4" 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 239, 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2024-02-12T16:43:50.282468+0000", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2024-02-12T16:58:18.169770+0000", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2024-02-12T16:49:30.933575+0000", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2024-02-12T16:57:46.137266+0000", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5799936, 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2024-02-12T16:58:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 18, 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 8, 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 8, 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "luminous", 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "pacific", 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "device_health_metrics", 2024-02-12T16:58:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:47:33.056945+0000", 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2024-02-12T16:58:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "71", 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:58:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T16:58:21.109 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:58:21.110 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_devicehealth": {} 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2024-02-12T16:58:21.111 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".rgw.root", 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:37.647343+0000", 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:58:21.112 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "59", 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T16:58:21.113 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:58:21.115 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "iscsi", 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:38.076200+0000", 2024-02-12T16:58:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 3, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 3, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 3, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 3, 2024-02-12T16:58:21.117 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 3, 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "60", 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T16:58:21.118 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T16:58:21.119 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T16:58:21.120 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 4, 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.log", 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:39.589565+0000", 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T16:58:21.121 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T16:58:21.122 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "61", 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T16:58:21.123 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T16:58:21.124 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T16:58:21.125 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 5, 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.control", 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:41.623856+0000", 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T16:58:21.126 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "5.16", 2024-02-12T16:58:21.127 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 238, 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 237, 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 237, 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "238", 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "238", 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "236", 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T16:58:21.128 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T16:58:21.129 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T16:58:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 6, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.meta", 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:43.693806+0000", 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:58:21.131 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T16:58:21.132 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "65", 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T16:58:21.133 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.134 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_bias": 4 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T16:58:21.135 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 8, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_1", 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:56:35.384129+0000", 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:58:21.136 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T16:58:21.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "183", 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T16:58:21.138 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: "rados": {} 2024-02-12T16:58:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7", 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 8, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 208, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T16:58:21.141 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2024-02-12T16:58:21.142 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T16:58:21.143 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/2545411201", 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6802/2545411201", 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6804/2545411201", 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6803/2545411201", 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T16:58:21.144 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24", 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 13, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 198, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T16:58:21.145 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.146 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.147 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6805/1723231708", 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6806/1723231708", 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6808/1723231708", 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6807/1723231708", 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c", 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 18, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 238, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.150 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3126580659", 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6810/3126580659", 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6812/3126580659", 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6811/3126580659", 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9", 2024-02-12T16:58:21.152 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 26, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 97, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T16:58:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6813/3941314957", 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6814/3941314957", 2024-02-12T16:58:21.155 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6816/3941314957", 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6815/3941314957", 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0498129c-861c-4f7e-b8ba-219949da3130", 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T16:58:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.54083251953125, 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 32, 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 214, 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6800", 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T16:58:21.157 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6801", 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6803", 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T16:58:21.158 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6802", 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6800/544126779", 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6801/544126779", 2024-02-12T16:58:21.159 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6803/544126779", 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6802/544126779", 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50", 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T16:58:21.160 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 38, 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 234, 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6804", 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6805", 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6807", 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T16:58:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6806", 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6804/3229884005", 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6805/3229884005", 2024-02-12T16:58:21.163 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6807/3229884005", 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6806/3229884005", 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821", 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T16:58:21.164 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 45, 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 208, 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6808", 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.165 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6809", 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6811", 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T16:58:21.166 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6810", 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6808/507305558", 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6809/507305558", 2024-02-12T16:58:21.167 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6811/507305558", 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6810/507305558", 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66", 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T16:58:21.168 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 54, 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 226, 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6812", 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T16:58:21.169 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6813", 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6815", 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.170 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6814", 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6812/3401977873", 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6813/3401977873", 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6815/3401977873", 2024-02-12T16:58:21.171 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6814/3401977873", 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T16:57:37.199492+0000", 2024-02-12T16:58:21.172 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T16:57:57.302471+0000", 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T16:58:21.173 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T16:58:05.347926+0000", 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T16:57:50.686389+0000", 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T16:58:21.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T16:57:48.349775+0000", 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T16:58:21.175 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T16:57:18.049652+0000", 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T16:57:21.933694+0000", 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T16:58:21.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T16:58:14.355334+0000", 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [ 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.8", 2024-02-12T16:58:21.177 INFO:teuthology.orchestra.run.smithi033.stdout: "mappings": [ 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: "from": 7, 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: "to": 2 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.9", 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: "mappings": [ 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: "from": 7, 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: "to": 4 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:58:21.178 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [ 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "8.0", 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: 1 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "8.1", 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: 0, 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: 1 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "8.3", 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: 1 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "8.7", 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: 1 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "8.a", 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: 1 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "8.b", 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: 1 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "8.f", 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: 1 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": { 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/2829464018": "2024-02-13T16:50:02.257936+0000", 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/1835128904": "2024-02-13T16:50:02.257936+0000", 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2277321750": "2024-02-13T16:50:02.257936+0000", 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2116602769": "2024-02-13T16:50:02.257936+0000", 2024-02-12T16:58:21.182 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3967118778": "2024-02-13T16:44:46.313346+0000", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2298361247": "2024-02-13T16:44:46.313346+0000", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/218227980": "2024-02-13T16:44:22.727241+0000", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2070582473": "2024-02-13T16:44:22.727241+0000", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/4291575956": "2024-02-13T16:44:46.313346+0000", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3157562110": "2024-02-13T16:44:22.727241+0000" 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "ruleset-failure-domain": "osd", 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2024-02-12T16:58:21.183 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [], 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:21.184 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:58:21.185 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:21.230 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:21.439 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:21.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:21.480 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 88 16 110154 110138 78.2063 49.5 0.00617273 0.0127814 2024-02-12T16:58:21.544 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:21.649 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:21.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:21.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:20.515758+0000 mgr.y (mgr.24481) 333 : cluster [DBG] pgmap v422: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 993 B/s rd, 46 MiB/s wr, 731 op/s; 36224/323649 objects misplaced (11.192%); 19 MiB/s, 307 objects/s recovering 2024-02-12T16:58:21.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:21.101928+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.33:0/911840836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:58:21.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:20.515758+0000 mgr.y (mgr.24481) 333 : cluster [DBG] pgmap v422: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 993 B/s rd, 46 MiB/s wr, 731 op/s; 36224/323649 objects misplaced (11.192%); 19 MiB/s, 307 objects/s recovering 2024-02-12T16:58:21.704 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:21.101928+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.33:0/911840836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:58:21.722 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:58:21.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:21.828 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:21.864 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:21.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:21.876 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:20.515758+0000 mgr.y (mgr.24481) 333 : cluster [DBG] pgmap v422: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 993 B/s rd, 46 MiB/s wr, 731 op/s; 36224/323649 objects misplaced (11.192%); 19 MiB/s, 307 objects/s recovering 2024-02-12T16:58:21.876 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:21.101928+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.33:0/911840836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:58:21.977 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:22.026 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:22.166 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:22.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:22.193 INFO:tasks.thrashosds.thrasher:j is {'epoch': 239, 'fsid': 'cf6ad642-c9c5-11ee-95b9-87774f69a715', 'created': '2024-02-12T16:43:50.282468+0000', 'modified': '2024-02-12T16:58:18.169770+0000', 'last_up_change': '2024-02-12T16:49:30.933575+0000', 'last_in_change': '2024-02-12T16:57:46.137266+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T16:47:33.056945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-12T16:49:37.647343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-12T16:49:38.076200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-12T16:49:39.589565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-12T16:49:41.623856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 238, 'last_epoch_started': 237, 'last_epoch_clean': 237, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '238', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '238', 'last_force_op_resend_preluminous': '236', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-12T16:49:43.693806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-02-12T16:56:35.384129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '183', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '5dd172eb-ee9d-4700-9b80-60f91c5f2cb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 2545411201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6802', 'nonce': 2545411201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6804', 'nonce': 2545411201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 2545411201}]}, 'public_addr': '172.21.15.33:6801/2545411201', 'cluster_addr': '172.21.15.33:6802/2545411201', 'heartbeat_back_addr': '172.21.15.33:6804/2545411201', 'heartbeat_front_addr': '172.21.15.33:6803/2545411201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71231bc4-1716-431d-bd88-f6a3cad0bd24', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1723231708}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6806', 'nonce': 1723231708}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6808', 'nonce': 1723231708}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1723231708}]}, 'public_addr': '172.21.15.33:6805/1723231708', 'cluster_addr': '172.21.15.33:6806/1723231708', 'heartbeat_back_addr': '172.21.15.33:6808/1723231708', 'heartbeat_front_addr': '172.21.15.33:6807/1723231708', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8a9c7127-d17c-4c44-b666-4211006de47c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3126580659}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6810', 'nonce': 3126580659}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6812', 'nonce': 3126580659}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3126580659}]}, 'public_addr': '172.21.15.33:6809/3126580659', 'cluster_addr': '172.21.15.33:6810/3126580659', 'heartbeat_back_addr': '172.21.15.33:6812/3126580659', 'heartbeat_front_addr': '172.21.15.33:6811/3126580659', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3cc40671-c8f7-412c-a0f1-987f48a016b9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3941314957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6814', 'nonce': 3941314957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6816', 'nonce': 3941314957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3941314957}]}, 'public_addr': '172.21.15.33:6813/3941314957', 'cluster_addr': '172.21.15.33:6814/3941314957', 'heartbeat_back_addr': '172.21.15.33:6816/3941314957', 'heartbeat_front_addr': '172.21.15.33:6815/3941314957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0498129c-861c-4f7e-b8ba-219949da3130', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.54083251953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6800', 'nonce': 544126779}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 544126779}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 544126779}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 544126779}]}, 'public_addr': '172.21.15.94:6800/544126779', 'cluster_addr': '172.21.15.94:6801/544126779', 'heartbeat_back_addr': '172.21.15.94:6803/544126779', 'heartbeat_front_addr': '172.21.15.94:6802/544126779', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '05718a1e-1164-4b54-a493-23c1cd3adb50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 234, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 3229884005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 3229884005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 3229884005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 3229884005}]}, 'public_addr': '172.21.15.94:6804/3229884005', 'cluster_addr': '172.21.15.94:6805/3229884005', 'heartbeat_back_addr': '172.21.15.94:6807/3229884005', 'heartbeat_front_addr': '172.21.15.94:6806/3229884005', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dc48640f-8f27-405c-a9a4-cf1e3e6c8821', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 507305558}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 507305558}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 507305558}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 507305558}]}, 'public_addr': '172.21.15.94:6808/507305558', 'cluster_addr': '172.21.15.94:6809/507305558', 'heartbeat_back_addr': '172.21.15.94:6811/507305558', 'heartbeat_front_addr': '172.21.15.94:6810/507305558', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7d55aa3f-06b6-4fc3-97fd-62b535f78b66', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 226, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3401977873}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 3401977873}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 3401977873}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 3401977873}]}, 'public_addr': '172.21.15.94:6812/3401977873', 'cluster_addr': '172.21.15.94:6813/3401977873', 'heartbeat_back_addr': '172.21.15.94:6815/3401977873', 'heartbeat_front_addr': '172.21.15.94:6814/3401977873', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:57:37.199492+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T16:57:57.302471+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:58:05.347926+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T16:57:50.686389+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:57:48.349775+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:57:18.049652+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:57:21.933694+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:58:14.355334+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.8', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.9', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [{'pgid': '8.0', 'osds': [5, 4, 1]}, {'pgid': '8.1', 'osds': [0, 2, 1]}, {'pgid': '8.3', 'osds': [5, 6, 1]}, {'pgid': '8.7', 'osds': [6, 2, 1]}, {'pgid': '8.a', 'osds': [2, 5, 1]}, {'pgid': '8.b', 'osds': [4, 2, 1]}, {'pgid': '8.f', 'osds': [2, 4, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.33:6800/2829464018': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/1835128904': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2277321750': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2116602769': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:6800/3967118778': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/2298361247': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/218227980': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/2070582473': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/4291575956': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:6800/3157562110': '2024-02-13T16:44:22.727241+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T16:58:22.193 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:58:22.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:22.327 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:22.364 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:22.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:22.473 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:22.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 89 16 110898 110882 77.8498 46.5 0.0235166 0.0128402 2024-02-12T16:58:22.539 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:22.743 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:58:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:58:23.302 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:23.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:23.314 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:23.340 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:23.409 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:23.471 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:23.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 90 4 111655 111651 77.5189 48.0625 0.0188276 0.0128949 2024-02-12T16:58:23.501 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:23.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:23.601 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:58:23.603 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:22.516714+0000 mgr.y (mgr.24481) 334 : cluster [DBG] pgmap v423: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 804 B/s rd, 37 MiB/s wr, 591 op/s; 36224/323649 objects misplaced (11.192%); 16 MiB/s, 248 objects/s recovering 2024-02-12T16:58:23.603 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:23.084481+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? v1:172.21.15.33:0/686920107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:58:23.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:22.516714+0000 mgr.y (mgr.24481) 334 : cluster [DBG] pgmap v423: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 804 B/s rd, 37 MiB/s wr, 591 op/s; 36224/323649 objects misplaced (11.192%); 16 MiB/s, 248 objects/s recovering 2024-02-12T16:58:23.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:23.084481+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? v1:172.21.15.33:0/686920107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:58:23.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:23.644 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:23.671 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:23.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:22.516714+0000 mgr.y (mgr.24481) 334 : cluster [DBG] pgmap v423: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 804 B/s rd, 37 MiB/s wr, 591 op/s; 36224/323649 objects misplaced (11.192%); 16 MiB/s, 248 objects/s recovering 2024-02-12T16:58:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:23.084481+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? v1:172.21.15.33:0/686920107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:58:23.776 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:23.815 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:23.838 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:23.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:23.943 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:23.977 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:24.128 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:24.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:24.233 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:24.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:58:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:58:24.268 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:24.345 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:24.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:24.449 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total time run: 90.0209 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total writes made: 111655 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Write size: 65536 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object size: 65536 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Bandwidth (MB/sec): 77.5202 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Bandwidth: 41.7063 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max bandwidth (MB/sec): 233.812 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min bandwidth (MB/sec): 0 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average IOPS: 1240 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev IOPS: 667.301 2024-02-12T16:58:24.481 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max IOPS: 3741 2024-02-12T16:58:24.482 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min IOPS: 0 2024-02-12T16:58:24.482 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average Latency(s): 0.0128955 2024-02-12T16:58:24.482 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Latency(s): 0.0266991 2024-02-12T16:58:24.482 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max latency(s): 2.15627 2024-02-12T16:58:24.482 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min latency(s): 0.00131853 2024-02-12T16:58:24.482 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:24.485 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:58:24.599 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:24.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:24.705 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:24.755 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:23.463665+0000 osd.5 (osd.5) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:24.755 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:23.463718+0000 osd.5 (osd.5) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:24.755 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:23.588647+0000 mon.a (mon.0) 1095 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:23.593701+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? v1:172.21.15.33:0/686920107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:23.593758+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e240: 8 total, 8 up, 6 in 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:23.836427+0000 mgr.y (mgr.24481) 335 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:23.463665+0000 osd.5 (osd.5) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:23.463718+0000 osd.5 (osd.5) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:23.588647+0000 mon.a (mon.0) 1095 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:23.593701+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? v1:172.21.15.33:0/686920107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:23.593758+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e240: 8 total, 8 up, 6 in 2024-02-12T16:58:24.756 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:23.836427+0000 mgr.y (mgr.24481) 335 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:24.758 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:24.890 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:24.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:24.996 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:24.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:23.463665+0000 osd.5 (osd.5) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:24.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:23.463718+0000 osd.5 (osd.5) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:24.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:23.588647+0000 mon.a (mon.0) 1095 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:24.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:23.593701+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? v1:172.21.15.33:0/686920107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:58:24.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:23.593758+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e240: 8 total, 8 up, 6 in 2024-02-12T16:58:24.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:23.836427+0000 mgr.y (mgr.24481) 335 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:25.055 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:25.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:25.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:25.296 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:25.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:25.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:25.611 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:25.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:25.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:25.751 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:24.518610+0000 mgr.y (mgr.24481) 336 : cluster [DBG] pgmap v425: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 35886/335400 objects misplaced (10.699%); 27 MiB/s, 438 objects/s recovering 2024-02-12T16:58:25.751 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:24.518610+0000 mgr.y (mgr.24481) 336 : cluster [DBG] pgmap v425: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 35886/335400 objects misplaced (10.699%); 27 MiB/s, 438 objects/s recovering 2024-02-12T16:58:25.753 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:25.894 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:25.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:24.518610+0000 mgr.y (mgr.24481) 336 : cluster [DBG] pgmap v425: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 35886/335400 objects misplaced (10.699%); 27 MiB/s, 438 objects/s recovering 2024-02-12T16:58:26.008 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:26.058 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:26.177 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:26.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:26.200 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:58:26.200 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:58:26.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:26.384 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:26.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:26.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:26.534 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:26.582 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:26.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:26.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:26.739 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:27.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:26.157948+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e241: 8 total, 8 up, 6 in 2024-02-12T16:58:27.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:26.519416+0000 mgr.y (mgr.24481) 337 : cluster [DBG] pgmap v427: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 38 MiB/s wr, 604 op/s; 35886/335400 objects misplaced (10.699%); 13 MiB/s, 204 objects/s recovering 2024-02-12T16:58:27.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:26.157948+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e241: 8 total, 8 up, 6 in 2024-02-12T16:58:27.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:26.519416+0000 mgr.y (mgr.24481) 337 : cluster [DBG] pgmap v427: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 38 MiB/s wr, 604 op/s; 35886/335400 objects misplaced (10.699%); 13 MiB/s, 204 objects/s recovering 2024-02-12T16:58:27.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:26.157948+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e241: 8 total, 8 up, 6 in 2024-02-12T16:58:27.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:26.519416+0000 mgr.y (mgr.24481) 337 : cluster [DBG] pgmap v427: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 38 MiB/s wr, 604 op/s; 35886/335400 objects misplaced (10.699%); 13 MiB/s, 204 objects/s recovering 2024-02-12T16:58:27.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:58:27.930 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:58:28.205 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:28.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:28.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:28.312 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:28.360 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:28.394 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:28.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:28.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:28.569 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:28.604 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:28.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:28.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:28.755 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:28.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:28.916 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:28.969 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:29.003 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:29.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:29.110 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:29.190 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:29.235 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:29.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:29.286 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T16:58:29.290 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T16:58:29.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:29.435 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:29.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:29.577 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:29.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:28.525021+0000 mgr.y (mgr.24481) 338 : cluster [DBG] pgmap v428: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 800 op/s; 32191/328242 objects misplaced (9.807%); 36 MiB/s, 570 objects/s recovering 2024-02-12T16:58:29.603 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:28.525021+0000 mgr.y (mgr.24481) 338 : cluster [DBG] pgmap v428: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 800 op/s; 32191/328242 objects misplaced (9.807%); 36 MiB/s, 570 objects/s recovering 2024-02-12T16:58:29.669 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:29.714 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:29.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:29.746 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:28.525021+0000 mgr.y (mgr.24481) 338 : cluster [DBG] pgmap v428: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 800 op/s; 32191/328242 objects misplaced (9.807%); 36 MiB/s, 570 objects/s recovering 2024-02-12T16:58:29.822 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:29.873 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:30.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:30.370 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-02-12T16:58:28.524432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7157913276,"num_objects":109414,"num_object_clones":0,"num_object_copies":328242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32191,"num_objects_unfound":0,"num_objects_dirty":109414,"num_whiteouts":0,"num_read":754,"num_read_kb":506,"num_write":114518,"num_write_kb":7145962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14305,"num_bytes_recovered":923342450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70280,"ondisk_log_size":70280,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":20729048,"kb_used_data":18916712,"kb_used_omap":0,"kb_used_meta":1811968,"kb_avail":541741864,"statfs":{"total":575970213888,"available":554743668736,"internally_reserved":0,"allocated":19370713088,"data_stored":19366084348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1855455232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":100270104,"num_objects":1531,"num_object_clones":0,"num_object_copies":4593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4033,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":6399,"num_write_kb":253697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4567,"num_bytes_recovered":299302912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.009026"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":129,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.605367+0000","last_change":"2024-02-12T16:57:47.187726+0000","last_active":"2024-02-12T16:58:23.605367+0000","last_peered":"2024-02-12T16:58:23.605367+0000","last_clean":"2024-02-12T16:58:23.605367+0000","last_became_active":"2024-02-12T16:57:47.187290+0000","last_became_peered":"2024-02-12T16:57:47.187290+0000","last_unstale":"2024-02-12T16:58:23.605367+0000","last_undegraded":"2024-02-12T16:58:23.605367+0000","last_fullsized":"2024-02-12T16:58:23.605367+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":203,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.190821+0000","last_change":"2024-02-12T16:56:35.853405+0000","last_active":"2024-02-12T16:58:17.190821+0000","last_peered":"2024-02-12T16:58:17.190821+0000","last_clean":"2024-02-12T16:58:17.190821+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T16:58:17.190821+0000","last_undegraded":"2024-02-12T16:58:17.190821+0000","last_fullsized":"2024-02-12T16:58:17.190821+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.853330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.853330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":173,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596673+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T16:58:23.596673+0000","last_peered":"2024-02-12T16:58:23.596673+0000","last_clean":"2024-02-12T16:58:23.596673+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T16:58:23.596673+0000","last_undegraded":"2024-02-12T16:58:23.596673+0000","last_fullsized":"2024-02-12T16:58:23.596673+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":169,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.637529+0000","last_change":"2024-02-12T16:57:47.192069+0000","last_active":"2024-02-12T16:58:23.637529+0000","last_peered":"2024-02-12T16:58:23.637529+0000","last_clean":"2024-02-12T16:58:23.637529+0000","last_became_active":"2024-02-12T16:57:47.188886+0000","last_became_peered":"2024-02-12T16:57:47.188886+0000","last_unstale":"2024-02-12T16:58:23.637529+0000","last_undegraded":"2024-02-12T16:58:23.637529+0000","last_fullsized":"2024-02-12T16:58:23.637529+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":118,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.192411+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T16:58:17.192411+0000","last_peered":"2024-02-12T16:58:17.192411+0000","last_clean":"2024-02-12T16:58:17.192411+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T16:58:17.192411+0000","last_undegraded":"2024-02-12T16:58:17.192411+0000","last_fullsized":"2024-02-12T16:58:17.192411+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":246,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.593262+0000","last_change":"2024-02-12T16:57:47.214989+0000","last_active":"2024-02-12T16:58:23.593262+0000","last_peered":"2024-02-12T16:58:23.593262+0000","last_clean":"2024-02-12T16:58:23.593262+0000","last_became_active":"2024-02-12T16:57:47.182083+0000","last_became_peered":"2024-02-12T16:57:47.182083+0000","last_unstale":"2024-02-12T16:58:23.593262+0000","last_undegraded":"2024-02-12T16:58:23.593262+0000","last_fullsized":"2024-02-12T16:58:23.593262+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:31.812644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.812644+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":219,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.635978+0000","last_change":"2024-02-12T16:57:47.176812+0000","last_active":"2024-02-12T16:58:23.635978+0000","last_peered":"2024-02-12T16:58:23.635978+0000","last_clean":"2024-02-12T16:58:23.635978+0000","last_became_active":"2024-02-12T16:57:47.176448+0000","last_became_peered":"2024-02-12T16:57:47.176448+0000","last_unstale":"2024-02-12T16:58:23.635978+0000","last_undegraded":"2024-02-12T16:58:23.635978+0000","last_fullsized":"2024-02-12T16:58:23.635978+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:42.174992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:42.174992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":213,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604718+0000","last_change":"2024-02-12T16:54:38.784259+0000","last_active":"2024-02-12T16:58:23.604718+0000","last_peered":"2024-02-12T16:58:23.604718+0000","last_clean":"2024-02-12T16:58:23.604718+0000","last_became_active":"2024-02-12T16:49:39.104047+0000","last_became_peered":"2024-02-12T16:49:39.104047+0000","last_unstale":"2024-02-12T16:58:23.604718+0000","last_undegraded":"2024-02-12T16:58:23.604718+0000","last_fullsized":"2024-02-12T16:58:23.604718+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":76,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596061+0000","last_change":"2024-02-12T16:57:47.319602+0000","last_active":"2024-02-12T16:58:23.596061+0000","last_peered":"2024-02-12T16:58:23.596061+0000","last_clean":"2024-02-12T16:58:23.596061+0000","last_became_active":"2024-02-12T16:57:47.186061+0000","last_became_peered":"2024-02-12T16:57:47.186061+0000","last_unstale":"2024-02-12T16:58:23.596061+0000","last_undegraded":"2024-02-12T16:58:23.596061+0000","last_fullsized":"2024-02-12T16:58:23.596061+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":219,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.190022+0000","last_change":"2024-02-12T16:58:16.945996+0000","last_active":"2024-02-12T16:58:17.190022+0000","last_peered":"2024-02-12T16:58:17.190022+0000","last_clean":"2024-02-12T16:58:17.190022+0000","last_became_active":"2024-02-12T16:57:47.185903+0000","last_became_peered":"2024-02-12T16:57:47.185903+0000","last_unstale":"2024-02-12T16:58:17.190022+0000","last_undegraded":"2024-02-12T16:58:17.190022+0000","last_fullsized":"2024-02-12T16:58:17.190022+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":205,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.603968+0000","last_change":"2024-02-12T16:56:35.897211+0000","last_active":"2024-02-12T16:58:23.603968+0000","last_peered":"2024-02-12T16:58:23.603968+0000","last_clean":"2024-02-12T16:58:23.603968+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T16:58:23.603968+0000","last_undegraded":"2024-02-12T16:58:23.603968+0000","last_fullsized":"2024-02-12T16:58:23.603968+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.897088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.897088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":156,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.191254+0000","last_change":"2024-02-12T16:57:18.865914+0000","last_active":"2024-02-12T16:58:17.191254+0000","last_peered":"2024-02-12T16:58:17.191254+0000","last_clean":"2024-02-12T16:58:17.191254+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T16:58:17.191254+0000","last_undegraded":"2024-02-12T16:58:17.191254+0000","last_fullsized":"2024-02-12T16:58:17.191254+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:18.865807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.865807+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":131,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597752+0000","last_change":"2024-02-12T16:57:47.200055+0000","last_active":"2024-02-12T16:58:23.597752+0000","last_peered":"2024-02-12T16:58:23.597752+0000","last_clean":"2024-02-12T16:58:23.597752+0000","last_became_active":"2024-02-12T16:57:47.199728+0000","last_became_peered":"2024-02-12T16:57:47.199728+0000","last_unstale":"2024-02-12T16:58:23.597752+0000","last_undegraded":"2024-02-12T16:58:23.597752+0000","last_fullsized":"2024-02-12T16:58:23.597752+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":120,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.605576+0000","last_change":"2024-02-12T16:57:33.724990+0000","last_active":"2024-02-12T16:58:23.605576+0000","last_peered":"2024-02-12T16:58:23.605576+0000","last_clean":"2024-02-12T16:58:23.605576+0000","last_became_active":"2024-02-12T16:54:40.633129+0000","last_became_peered":"2024-02-12T16:54:40.633129+0000","last_unstale":"2024-02-12T16:58:23.605576+0000","last_undegraded":"2024-02-12T16:58:23.605576+0000","last_fullsized":"2024-02-12T16:58:23.605576+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":92,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.594355+0000","last_change":"2024-02-12T16:57:47.645464+0000","last_active":"2024-02-12T16:58:23.594355+0000","last_peered":"2024-02-12T16:58:23.594355+0000","last_clean":"2024-02-12T16:58:23.594355+0000","last_became_active":"2024-02-12T16:57:47.183656+0000","last_became_peered":"2024-02-12T16:57:47.183656+0000","last_unstale":"2024-02-12T16:58:23.594355+0000","last_undegraded":"2024-02-12T16:58:23.594355+0000","last_fullsized":"2024-02-12T16:58:23.594355+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:56:45.033441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.033441+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":206,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.594368+0000","last_change":"2024-02-12T16:56:18.464298+0000","last_active":"2024-02-12T16:58:23.594368+0000","last_peered":"2024-02-12T16:58:23.594368+0000","last_clean":"2024-02-12T16:58:23.594368+0000","last_became_active":"2024-02-12T16:49:45.223280+0000","last_became_peered":"2024-02-12T16:49:45.223280+0000","last_unstale":"2024-02-12T16:58:23.594368+0000","last_undegraded":"2024-02-12T16:58:23.594368+0000","last_fullsized":"2024-02-12T16:58:23.594368+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":211,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596571+0000","last_change":"2024-02-12T16:56:25.339812+0000","last_active":"2024-02-12T16:58:23.596571+0000","last_peered":"2024-02-12T16:58:23.596571+0000","last_clean":"2024-02-12T16:58:23.596571+0000","last_became_active":"2024-02-12T16:49:39.108943+0000","last_became_peered":"2024-02-12T16:49:39.108943+0000","last_unstale":"2024-02-12T16:58:23.596571+0000","last_undegraded":"2024-02-12T16:58:23.596571+0000","last_fullsized":"2024-02-12T16:58:23.596571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":289,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596753+0000","last_change":"2024-02-12T16:57:47.210170+0000","last_active":"2024-02-12T16:58:23.596753+0000","last_peered":"2024-02-12T16:58:23.596753+0000","last_clean":"2024-02-12T16:58:23.596753+0000","last_became_active":"2024-02-12T16:57:47.178386+0000","last_became_peered":"2024-02-12T16:57:47.178386+0000","last_unstale":"2024-02-12T16:58:23.596753+0000","last_undegraded":"2024-02-12T16:58:23.596753+0000","last_fullsized":"2024-02-12T16:58:23.596753+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:57:24.750245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.750245+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.638560+0000","last_change":"2024-02-12T16:57:47.179029+0000","last_active":"2024-02-12T16:58:23.638560+0000","last_peered":"2024-02-12T16:58:23.638560+0000","last_clean":"2024-02-12T16:58:23.638560+0000","last_became_active":"2024-02-12T16:57:47.178240+0000","last_became_peered":"2024-02-12T16:57:47.178240+0000","last_unstale":"2024-02-12T16:58:23.638560+0000","last_undegraded":"2024-02-12T16:58:23.638560+0000","last_fullsized":"2024-02-12T16:58:23.638560+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":120,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596486+0000","last_change":"2024-02-12T16:56:33.292072+0000","last_active":"2024-02-12T16:58:23.596486+0000","last_peered":"2024-02-12T16:58:23.596486+0000","last_clean":"2024-02-12T16:58:23.596486+0000","last_became_active":"2024-02-12T16:54:40.640452+0000","last_became_peered":"2024-02-12T16:54:40.640452+0000","last_unstale":"2024-02-12T16:58:23.596486+0000","last_undegraded":"2024-02-12T16:58:23.596486+0000","last_fullsized":"2024-02-12T16:58:23.596486+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":244,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597656+0000","last_change":"2024-02-12T16:57:47.662042+0000","last_active":"2024-02-12T16:58:23.597656+0000","last_peered":"2024-02-12T16:58:23.597656+0000","last_clean":"2024-02-12T16:58:23.597656+0000","last_became_active":"2024-02-12T16:57:47.200126+0000","last_became_peered":"2024-02-12T16:57:47.200126+0000","last_unstale":"2024-02-12T16:58:23.597656+0000","last_undegraded":"2024-02-12T16:58:23.597656+0000","last_fullsized":"2024-02-12T16:58:23.597656+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":201,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604440+0000","last_change":"2024-02-12T16:57:37.670804+0000","last_active":"2024-02-12T16:58:23.604440+0000","last_peered":"2024-02-12T16:58:23.604440+0000","last_clean":"2024-02-12T16:58:23.604440+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T16:58:23.604440+0000","last_undegraded":"2024-02-12T16:58:23.604440+0000","last_fullsized":"2024-02-12T16:58:23.604440+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:57:37.670713+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.670713+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":207,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.635509+0000","last_change":"2024-02-12T16:57:26.790389+0000","last_active":"2024-02-12T16:58:23.635509+0000","last_peered":"2024-02-12T16:58:23.635509+0000","last_clean":"2024-02-12T16:58:23.635509+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T16:58:23.635509+0000","last_undegraded":"2024-02-12T16:58:23.635509+0000","last_fullsized":"2024-02-12T16:58:23.635509+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":258,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.192879+0000","last_change":"2024-02-12T16:57:47.209446+0000","last_active":"2024-02-12T16:58:17.192879+0000","last_peered":"2024-02-12T16:58:17.192879+0000","last_clean":"2024-02-12T16:58:17.192879+0000","last_became_active":"2024-02-12T16:57:47.189652+0000","last_became_peered":"2024-02-12T16:57:47.189652+0000","last_unstale":"2024-02-12T16:58:17.192879+0000","last_undegraded":"2024-02-12T16:58:17.192879+0000","last_fullsized":"2024-02-12T16:58:17.192879+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":118,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.189394+0000","last_change":"2024-02-12T16:57:31.978946+0000","last_active":"2024-02-12T16:58:17.189394+0000","last_peered":"2024-02-12T16:58:17.189394+0000","last_clean":"2024-02-12T16:58:17.189394+0000","last_became_active":"2024-02-12T16:54:40.650168+0000","last_became_peered":"2024-02-12T16:54:40.650168+0000","last_unstale":"2024-02-12T16:58:17.189394+0000","last_undegraded":"2024-02-12T16:58:17.189394+0000","last_fullsized":"2024-02-12T16:58:17.189394+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":196,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.603651+0000","last_change":"2024-02-12T16:53:05.141324+0000","last_active":"2024-02-12T16:58:23.603651+0000","last_peered":"2024-02-12T16:58:23.603651+0000","last_clean":"2024-02-12T16:58:23.603651+0000","last_became_active":"2024-02-12T16:49:43.198447+0000","last_became_peered":"2024-02-12T16:49:43.198447+0000","last_unstale":"2024-02-12T16:58:23.603651+0000","last_undegraded":"2024-02-12T16:58:23.603651+0000","last_fullsized":"2024-02-12T16:58:23.603651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":213,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597546+0000","last_change":"2024-02-12T16:57:25.278494+0000","last_active":"2024-02-12T16:58:23.597546+0000","last_peered":"2024-02-12T16:58:23.597546+0000","last_clean":"2024-02-12T16:58:23.597546+0000","last_became_active":"2024-02-12T16:49:39.107383+0000","last_became_peered":"2024-02-12T16:49:39.107383+0000","last_unstale":"2024-02-12T16:58:23.597546+0000","last_undegraded":"2024-02-12T16:58:23.597546+0000","last_fullsized":"2024-02-12T16:58:23.597546+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":163,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.603572+0000","last_change":"2024-02-12T16:56:24.854608+0000","last_active":"2024-02-12T16:58:23.603572+0000","last_peered":"2024-02-12T16:58:23.603572+0000","last_clean":"2024-02-12T16:58:23.603572+0000","last_became_active":"2024-02-12T16:54:40.632501+0000","last_became_peered":"2024-02-12T16:54:40.632501+0000","last_unstale":"2024-02-12T16:58:23.603572+0000","last_undegraded":"2024-02-12T16:58:23.603572+0000","last_fullsized":"2024-02-12T16:58:23.603572+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":199,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604165+0000","last_change":"2024-02-12T16:57:28.846844+0000","last_active":"2024-02-12T16:58:23.604165+0000","last_peered":"2024-02-12T16:58:23.604165+0000","last_clean":"2024-02-12T16:58:23.604165+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T16:58:23.604165+0000","last_undegraded":"2024-02-12T16:58:23.604165+0000","last_fullsized":"2024-02-12T16:58:23.604165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":207,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596569+0000","last_change":"2024-02-12T16:56:45.226096+0000","last_active":"2024-02-12T16:58:23.596569+0000","last_peered":"2024-02-12T16:58:23.596569+0000","last_clean":"2024-02-12T16:58:23.596569+0000","last_became_active":"2024-02-12T16:49:43.198078+0000","last_became_peered":"2024-02-12T16:49:43.198078+0000","last_unstale":"2024-02-12T16:58:23.596569+0000","last_undegraded":"2024-02-12T16:58:23.596569+0000","last_fullsized":"2024-02-12T16:58:23.596569+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.225996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":207,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.639686+0000","last_change":"2024-02-12T16:57:24.805336+0000","last_active":"2024-02-12T16:58:23.639686+0000","last_peered":"2024-02-12T16:58:23.639686+0000","last_clean":"2024-02-12T16:58:23.639686+0000","last_became_active":"2024-02-12T16:49:39.101028+0000","last_became_peered":"2024-02-12T16:49:39.101028+0000","last_unstale":"2024-02-12T16:58:23.639686+0000","last_undegraded":"2024-02-12T16:58:23.639686+0000","last_fullsized":"2024-02-12T16:58:23.639686+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":69,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596341+0000","last_change":"2024-02-12T16:57:48.060186+0000","last_active":"2024-02-12T16:58:23.596341+0000","last_peered":"2024-02-12T16:58:23.596341+0000","last_clean":"2024-02-12T16:58:23.596341+0000","last_became_active":"2024-02-12T16:57:47.200637+0000","last_became_peered":"2024-02-12T16:57:47.200637+0000","last_unstale":"2024-02-12T16:58:23.596341+0000","last_undegraded":"2024-02-12T16:58:23.596341+0000","last_fullsized":"2024-02-12T16:58:23.596341+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":118,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.190788+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T16:58:17.190788+0000","last_peered":"2024-02-12T16:58:17.190788+0000","last_clean":"2024-02-12T16:58:17.190788+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T16:58:17.190788+0000","last_undegraded":"2024-02-12T16:58:17.190788+0000","last_fullsized":"2024-02-12T16:58:17.190788+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":145,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.637948+0000","last_change":"2024-02-12T16:57:47.193696+0000","last_active":"2024-02-12T16:58:23.637948+0000","last_peered":"2024-02-12T16:58:23.637948+0000","last_clean":"2024-02-12T16:58:23.637948+0000","last_became_active":"2024-02-12T16:57:47.192310+0000","last_became_peered":"2024-02-12T16:57:47.192310+0000","last_unstale":"2024-02-12T16:58:23.637948+0000","last_undegraded":"2024-02-12T16:58:23.637948+0000","last_fullsized":"2024-02-12T16:58:23.637948+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":45,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.638008+0000","last_change":"2024-02-12T16:57:47.190143+0000","last_active":"2024-02-12T16:58:23.638008+0000","last_peered":"2024-02-12T16:58:23.638008+0000","last_clean":"2024-02-12T16:58:23.638008+0000","last_became_active":"2024-02-12T16:57:47.189816+0000","last_became_peered":"2024-02-12T16:57:47.189816+0000","last_unstale":"2024-02-12T16:58:23.638008+0000","last_undegraded":"2024-02-12T16:58:23.638008+0000","last_fullsized":"2024-02-12T16:58:23.638008+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":253,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.593958+0000","last_change":"2024-02-12T16:57:47.627590+0000","last_active":"2024-02-12T16:58:23.593958+0000","last_peered":"2024-02-12T16:58:23.593958+0000","last_clean":"2024-02-12T16:58:23.593958+0000","last_became_active":"2024-02-12T16:57:47.182807+0000","last_became_peered":"2024-02-12T16:57:47.182807+0000","last_unstale":"2024-02-12T16:58:23.593958+0000","last_undegraded":"2024-02-12T16:58:23.593958+0000","last_fullsized":"2024-02-12T16:58:23.593958+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:37.895287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.895287+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":201,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.189548+0000","last_change":"2024-02-12T16:57:47.190745+0000","last_active":"2024-02-12T16:58:17.189548+0000","last_peered":"2024-02-12T16:58:17.189548+0000","last_clean":"2024-02-12T16:58:17.189548+0000","last_became_active":"2024-02-12T16:57:47.188892+0000","last_became_peered":"2024-02-12T16:57:47.188892+0000","last_unstale":"2024-02-12T16:58:17.189548+0000","last_undegraded":"2024-02-12T16:58:17.189548+0000","last_fullsized":"2024-02-12T16:58:17.189548+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":143,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.189496+0000","last_change":"2024-02-12T16:57:47.189653+0000","last_active":"2024-02-12T16:58:17.189496+0000","last_peered":"2024-02-12T16:58:17.189496+0000","last_clean":"2024-02-12T16:58:17.189496+0000","last_became_active":"2024-02-12T16:57:47.188909+0000","last_became_peered":"2024-02-12T16:57:47.188909+0000","last_unstale":"2024-02-12T16:58:17.189496+0000","last_undegraded":"2024-02-12T16:58:17.189496+0000","last_fullsized":"2024-02-12T16:58:17.189496+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":215,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597098+0000","last_change":"2024-02-12T16:56:39.248638+0000","last_active":"2024-02-12T16:58:23.597098+0000","last_peered":"2024-02-12T16:58:23.597098+0000","last_clean":"2024-02-12T16:58:23.597098+0000","last_became_active":"2024-02-12T16:54:40.629419+0000","last_became_peered":"2024-02-12T16:54:40.629419+0000","last_unstale":"2024-02-12T16:58:23.597098+0000","last_undegraded":"2024-02-12T16:58:23.597098+0000","last_fullsized":"2024-02-12T16:58:23.597098+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:39.248562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.248562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"80'13","reported_seq":101,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596705+0000","last_change":"2024-02-12T16:57:47.213554+0000","last_active":"2024-02-12T16:58:23.596705+0000","last_peered":"2024-02-12T16:58:23.596705+0000","last_clean":"2024-02-12T16:58:23.596705+0000","last_became_active":"2024-02-12T16:57:47.181467+0000","last_became_peered":"2024-02-12T16:57:47.181467+0000","last_unstale":"2024-02-12T16:58:23.596705+0000","last_undegraded":"2024-02-12T16:58:23.596705+0000","last_fullsized":"2024-02-12T16:58:23.596705+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":165,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597515+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T16:58:23.597515+0000","last_peered":"2024-02-12T16:58:23.597515+0000","last_clean":"2024-02-12T16:58:23.597515+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T16:58:23.597515+0000","last_undegraded":"2024-02-12T16:58:23.597515+0000","last_fullsized":"2024-02-12T16:58:23.597515+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":223,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:26.172271+0000","last_change":"2024-02-12T16:57:47.312130+0000","last_active":"2024-02-12T16:58:26.172271+0000","last_peered":"2024-02-12T16:58:26.172271+0000","last_clean":"2024-02-12T16:58:26.172271+0000","last_became_active":"2024-02-12T16:57:47.185109+0000","last_became_peered":"2024-02-12T16:57:47.185109+0000","last_unstale":"2024-02-12T16:58:26.172271+0000","last_undegraded":"2024-02-12T16:58:26.172271+0000","last_fullsized":"2024-02-12T16:58:26.172271+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":207,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.189741+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T16:58:17.189741+0000","last_peered":"2024-02-12T16:58:17.189741+0000","last_clean":"2024-02-12T16:58:17.189741+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T16:58:17.189741+0000","last_undegraded":"2024-02-12T16:58:17.189741+0000","last_fullsized":"2024-02-12T16:58:17.189741+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":235,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.637494+0000","last_change":"2024-02-12T16:54:40.641346+0000","last_active":"2024-02-12T16:58:23.637494+0000","last_peered":"2024-02-12T16:58:23.637494+0000","last_clean":"2024-02-12T16:58:23.637494+0000","last_became_active":"2024-02-12T16:54:40.626031+0000","last_became_peered":"2024-02-12T16:54:40.626031+0000","last_unstale":"2024-02-12T16:58:23.637494+0000","last_undegraded":"2024-02-12T16:58:23.637494+0000","last_fullsized":"2024-02-12T16:58:23.637494+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":206,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.192063+0000","last_change":"2024-02-12T16:57:25.835979+0000","last_active":"2024-02-12T16:58:17.192063+0000","last_peered":"2024-02-12T16:58:17.192063+0000","last_clean":"2024-02-12T16:58:17.192063+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T16:58:17.192063+0000","last_undegraded":"2024-02-12T16:58:17.192063+0000","last_fullsized":"2024-02-12T16:58:17.192063+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.835883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.835883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"208'9","reported_seq":327,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:26.174503+0000","last_change":"2024-02-12T16:57:47.225402+0000","last_active":"2024-02-12T16:58:26.174503+0000","last_peered":"2024-02-12T16:58:26.174503+0000","last_clean":"2024-02-12T16:58:26.174503+0000","last_became_active":"2024-02-12T16:57:47.178688+0000","last_became_peered":"2024-02-12T16:57:47.178688+0000","last_unstale":"2024-02-12T16:58:26.174503+0000","last_undegraded":"2024-02-12T16:58:26.174503+0000","last_fullsized":"2024-02-12T16:58:26.174503+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":211,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.637677+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T16:58:23.637677+0000","last_peered":"2024-02-12T16:58:23.637677+0000","last_clean":"2024-02-12T16:58:23.637677+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T16:58:23.637677+0000","last_undegraded":"2024-02-12T16:58:23.637677+0000","last_fullsized":"2024-02-12T16:58:23.637677+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":84,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604058+0000","last_change":"2024-02-12T16:57:48.058146+0000","last_active":"2024-02-12T16:58:23.604058+0000","last_peered":"2024-02-12T16:58:23.604058+0000","last_clean":"2024-02-12T16:58:23.604058+0000","last_became_active":"2024-02-12T16:57:47.199140+0000","last_became_peered":"2024-02-12T16:57:47.199140+0000","last_unstale":"2024-02-12T16:58:23.604058+0000","last_undegraded":"2024-02-12T16:58:23.604058+0000","last_fullsized":"2024-02-12T16:58:23.604058+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":198,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604646+0000","last_change":"2024-02-12T16:53:51.532654+0000","last_active":"2024-02-12T16:58:23.604646+0000","last_peered":"2024-02-12T16:58:23.604646+0000","last_clean":"2024-02-12T16:58:23.604646+0000","last_became_active":"2024-02-12T16:49:45.224682+0000","last_became_peered":"2024-02-12T16:49:45.224682+0000","last_unstale":"2024-02-12T16:58:23.604646+0000","last_undegraded":"2024-02-12T16:58:23.604646+0000","last_fullsized":"2024-02-12T16:58:23.604646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":215,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596579+0000","last_change":"2024-02-12T16:57:47.200705+0000","last_active":"2024-02-12T16:58:23.596579+0000","last_peered":"2024-02-12T16:58:23.596579+0000","last_clean":"2024-02-12T16:58:23.596579+0000","last_became_active":"2024-02-12T16:57:47.200448+0000","last_became_peered":"2024-02-12T16:57:47.200448+0000","last_unstale":"2024-02-12T16:58:23.596579+0000","last_undegraded":"2024-02-12T16:58:23.596579+0000","last_fullsized":"2024-02-12T16:58:23.596579+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.218031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.218031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":161,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604682+0000","last_change":"2024-02-12T16:57:47.482249+0000","last_active":"2024-02-12T16:58:23.604682+0000","last_peered":"2024-02-12T16:58:23.604682+0000","last_clean":"2024-02-12T16:58:23.604682+0000","last_became_active":"2024-02-12T16:57:47.186190+0000","last_became_peered":"2024-02-12T16:57:47.186190+0000","last_unstale":"2024-02-12T16:58:23.604682+0000","last_undegraded":"2024-02-12T16:58:23.604682+0000","last_fullsized":"2024-02-12T16:58:23.604682+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":201,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596461+0000","last_change":"2024-02-12T16:56:38.699688+0000","last_active":"2024-02-12T16:58:23.596461+0000","last_peered":"2024-02-12T16:58:23.596461+0000","last_clean":"2024-02-12T16:58:23.596461+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T16:58:23.596461+0000","last_undegraded":"2024-02-12T16:58:23.596461+0000","last_fullsized":"2024-02-12T16:58:23.596461+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.699602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":216,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.595717+0000","last_change":"2024-02-12T16:57:47.171068+0000","last_active":"2024-02-12T16:58:23.595717+0000","last_peered":"2024-02-12T16:58:23.595717+0000","last_clean":"2024-02-12T16:58:23.595717+0000","last_became_active":"2024-02-12T16:57:47.170667+0000","last_became_peered":"2024-02-12T16:57:47.170667+0000","last_unstale":"2024-02-12T16:58:23.595717+0000","last_undegraded":"2024-02-12T16:58:23.595717+0000","last_fullsized":"2024-02-12T16:58:23.595717+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":235,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.192302+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T16:58:17.192302+0000","last_peered":"2024-02-12T16:58:17.192302+0000","last_clean":"2024-02-12T16:58:17.192302+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T16:58:17.192302+0000","last_undegraded":"2024-02-12T16:58:17.192302+0000","last_fullsized":"2024-02-12T16:58:17.192302+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"241'7190","reported_seq":7263,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:28.337692+0000","last_change":"2024-02-12T16:56:36.789085+0000","last_active":"2024-02-12T16:58:28.337692+0000","last_peered":"2024-02-12T16:58:28.337692+0000","last_clean":"2024-02-12T16:58:28.337692+0000","last_became_active":"2024-02-12T16:56:36.788797+0000","last_became_peered":"2024-02-12T16:56:36.788797+0000","last_unstale":"2024-02-12T16:58:28.337692+0000","last_undegraded":"2024-02-12T16:58:28.337692+0000","last_fullsized":"2024-02-12T16:58:28.337692+0000","mapping_epoch":176,"log_start":"189'2876","ondisk_log_start":"189'2876","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4314,"ondisk_log_size":4314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":442368000,"num_objects":6750,"num_object_clones":0,"num_object_copies":20250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7190,"num_write_kb":446080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":124,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.605507+0000","last_change":"2024-02-12T16:57:47.187592+0000","last_active":"2024-02-12T16:58:23.605507+0000","last_peered":"2024-02-12T16:58:23.605507+0000","last_clean":"2024-02-12T16:58:23.605507+0000","last_became_active":"2024-02-12T16:57:47.186880+0000","last_became_peered":"2024-02-12T16:57:47.186880+0000","last_unstale":"2024-02-12T16:58:23.605507+0000","last_undegraded":"2024-02-12T16:58:23.605507+0000","last_fullsized":"2024-02-12T16:58:23.605507+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":232,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.594346+0000","last_change":"2024-02-12T16:57:22.679795+0000","last_active":"2024-02-12T16:58:23.594346+0000","last_peered":"2024-02-12T16:58:23.594346+0000","last_clean":"2024-02-12T16:58:23.594346+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T16:58:23.594346+0000","last_undegraded":"2024-02-12T16:58:23.594346+0000","last_fullsized":"2024-02-12T16:58:23.594346+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:57:22.679705+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.679705+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":149,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596334+0000","last_change":"2024-02-12T16:57:47.328801+0000","last_active":"2024-02-12T16:58:23.596334+0000","last_peered":"2024-02-12T16:58:23.596334+0000","last_clean":"2024-02-12T16:58:23.596334+0000","last_became_active":"2024-02-12T16:57:47.194749+0000","last_became_peered":"2024-02-12T16:57:47.194749+0000","last_unstale":"2024-02-12T16:58:23.596334+0000","last_undegraded":"2024-02-12T16:58:23.596334+0000","last_fullsized":"2024-02-12T16:58:23.596334+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:22.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.764984+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":230,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.595869+0000","last_change":"2024-02-12T16:58:10.280441+0000","last_active":"2024-02-12T16:58:23.595869+0000","last_peered":"2024-02-12T16:58:23.595869+0000","last_clean":"2024-02-12T16:58:23.595869+0000","last_became_active":"2024-02-12T16:58:10.280137+0000","last_became_peered":"2024-02-12T16:58:10.280137+0000","last_unstale":"2024-02-12T16:58:23.595869+0000","last_undegraded":"2024-02-12T16:58:23.595869+0000","last_fullsized":"2024-02-12T16:58:23.595869+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.647627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.647627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"239'6967","reported_seq":7038,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.415247+0000","last_change":"2024-02-12T16:56:36.446431+0000","last_active":"2024-02-12T16:58:23.415247+0000","last_peered":"2024-02-12T16:58:23.415247+0000","last_clean":"2024-02-12T16:58:23.415247+0000","last_became_active":"2024-02-12T16:56:36.446107+0000","last_became_peered":"2024-02-12T16:56:36.446107+0000","last_unstale":"2024-02-12T16:58:23.415247+0000","last_undegraded":"2024-02-12T16:58:23.415247+0000","last_fullsized":"2024-02-12T16:58:23.415247+0000","mapping_epoch":176,"log_start":"189'2312","ondisk_log_start":"189'2312","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4655,"ondisk_log_size":4655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":456589312,"num_objects":6967,"num_object_clones":0,"num_object_copies":20901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":445888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":206,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.189635+0000","last_change":"2024-02-12T16:57:29.934046+0000","last_active":"2024-02-12T16:58:17.189635+0000","last_peered":"2024-02-12T16:58:17.189635+0000","last_clean":"2024-02-12T16:58:17.189635+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T16:58:17.189635+0000","last_undegraded":"2024-02-12T16:58:17.189635+0000","last_fullsized":"2024-02-12T16:58:17.189635+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.191709+0000","last_change":"2024-02-12T16:57:47.187057+0000","last_active":"2024-02-12T16:58:17.191709+0000","last_peered":"2024-02-12T16:58:17.191709+0000","last_clean":"2024-02-12T16:58:17.191709+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T16:58:17.191709+0000","last_undegraded":"2024-02-12T16:58:17.191709+0000","last_fullsized":"2024-02-12T16:58:17.191709+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.009094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.009094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":84,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.638032+0000","last_change":"2024-02-12T16:57:48.006795+0000","last_active":"2024-02-12T16:58:23.638032+0000","last_peered":"2024-02-12T16:58:23.638032+0000","last_clean":"2024-02-12T16:58:23.638032+0000","last_became_active":"2024-02-12T16:57:47.202531+0000","last_became_peered":"2024-02-12T16:57:47.202531+0000","last_unstale":"2024-02-12T16:58:23.638032+0000","last_undegraded":"2024-02-12T16:58:23.638032+0000","last_fullsized":"2024-02-12T16:58:23.638032+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":307,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-02-12T16:58:18.181587+0000","last_change":"2024-02-12T16:53:59.173126+0000","last_active":"2024-02-12T16:58:18.181587+0000","last_peered":"2024-02-12T16:58:18.181587+0000","last_clean":"2024-02-12T16:58:18.181587+0000","last_became_active":"2024-02-12T16:49:43.206033+0000","last_became_peered":"2024-02-12T16:49:43.206033+0000","last_unstale":"2024-02-12T16:58:18.181587+0000","last_undegraded":"2024-02-12T16:58:18.181587+0000","last_fullsized":"2024-02-12T16:58:18.181587+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"241'7266","reported_seq":7339,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:28.283532+0000","last_change":"2024-02-12T16:56:36.788519+0000","last_active":"2024-02-12T16:58:28.283532+0000","last_peered":"2024-02-12T16:58:28.283532+0000","last_clean":"2024-02-12T16:58:28.283532+0000","last_became_active":"2024-02-12T16:56:36.788101+0000","last_became_peered":"2024-02-12T16:56:36.788101+0000","last_unstale":"2024-02-12T16:58:28.283532+0000","last_undegraded":"2024-02-12T16:58:28.283532+0000","last_fullsized":"2024-02-12T16:58:28.283532+0000","mapping_epoch":176,"log_start":"190'3239","ondisk_log_start":"190'3239","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4027,"ondisk_log_size":4027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":450232320,"num_objects":6870,"num_object_clones":0,"num_object_copies":20610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":452352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":124,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.594598+0000","last_change":"2024-02-12T16:57:21.653397+0000","last_active":"2024-02-12T16:58:23.594598+0000","last_peered":"2024-02-12T16:58:23.594598+0000","last_clean":"2024-02-12T16:58:23.594598+0000","last_became_active":"2024-02-12T16:54:40.625901+0000","last_became_peered":"2024-02-12T16:54:40.625901+0000","last_unstale":"2024-02-12T16:58:23.594598+0000","last_undegraded":"2024-02-12T16:58:23.594598+0000","last_fullsized":"2024-02-12T16:58:23.594598+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.653301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.653301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":216,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.637753+0000","last_change":"2024-02-12T16:57:47.193874+0000","last_active":"2024-02-12T16:58:23.637753+0000","last_peered":"2024-02-12T16:58:23.637753+0000","last_clean":"2024-02-12T16:58:23.637753+0000","last_became_active":"2024-02-12T16:57:47.192461+0000","last_became_peered":"2024-02-12T16:57:47.192461+0000","last_unstale":"2024-02-12T16:58:23.637753+0000","last_undegraded":"2024-02-12T16:58:23.637753+0000","last_fullsized":"2024-02-12T16:58:23.637753+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"208'40","reported_seq":381,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:26.181884+0000","last_change":"2024-02-12T16:57:47.215971+0000","last_active":"2024-02-12T16:58:26.181884+0000","last_peered":"2024-02-12T16:58:26.181884+0000","last_clean":"2024-02-12T16:58:26.181884+0000","last_became_active":"2024-02-12T16:57:47.182858+0000","last_became_peered":"2024-02-12T16:57:47.182858+0000","last_unstale":"2024-02-12T16:58:26.181884+0000","last_undegraded":"2024-02-12T16:58:26.181884+0000","last_fullsized":"2024-02-12T16:58:26.181884+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'39","last_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.083656+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":208,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.190459+0000","last_change":"2024-02-12T16:57:47.187774+0000","last_active":"2024-02-12T16:58:17.190459+0000","last_peered":"2024-02-12T16:58:17.190459+0000","last_clean":"2024-02-12T16:58:17.190459+0000","last_became_active":"2024-02-12T16:57:47.187410+0000","last_became_peered":"2024-02-12T16:57:47.187410+0000","last_unstale":"2024-02-12T16:58:17.190459+0000","last_undegraded":"2024-02-12T16:58:17.190459+0000","last_fullsized":"2024-02-12T16:58:17.190459+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:46.719706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:46.719706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"241'7101","reported_seq":8116,"reported_epoch":241,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:58:28.331873+0000","last_change":"2024-02-12T16:57:48.371742+0000","last_active":"2024-02-12T16:58:28.331873+0000","last_peered":"2024-02-12T16:58:28.331873+0000","last_clean":"2024-02-12T16:57:46.130664+0000","last_became_active":"2024-02-12T16:57:48.242613+0000","last_became_peered":"2024-02-12T16:57:48.242613+0000","last_unstale":"2024-02-12T16:58:28.331873+0000","last_undegraded":"2024-02-12T16:58:28.331873+0000","last_fullsized":"2024-02-12T16:58:28.331873+0000","mapping_epoch":208,"log_start":"189'2799","ondisk_log_start":"189'2799","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4302,"ondisk_log_size":4302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436666368,"num_objects":6663,"num_object_clones":0,"num_object_copies":19989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6157,"num_objects_unfound":0,"num_objects_dirty":6663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":440448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":30474240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":6663}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":205,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604848+0000","last_change":"2024-02-12T16:54:40.640502+0000","last_active":"2024-02-12T16:58:23.604848+0000","last_peered":"2024-02-12T16:58:23.604848+0000","last_clean":"2024-02-12T16:58:23.604848+0000","last_became_active":"2024-02-12T16:54:40.640105+0000","last_became_peered":"2024-02-12T16:54:40.640105+0000","last_unstale":"2024-02-12T16:58:23.604848+0000","last_undegraded":"2024-02-12T16:58:23.604848+0000","last_fullsized":"2024-02-12T16:58:23.604848+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":114,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:27.369527+0000","last_change":"2024-02-12T16:57:47.200645+0000","last_active":"2024-02-12T16:58:27.369527+0000","last_peered":"2024-02-12T16:58:27.369527+0000","last_clean":"2024-02-12T16:58:27.369527+0000","last_became_active":"2024-02-12T16:57:47.178352+0000","last_became_peered":"2024-02-12T16:57:47.178352+0000","last_unstale":"2024-02-12T16:58:27.369527+0000","last_undegraded":"2024-02-12T16:58:27.369527+0000","last_fullsized":"2024-02-12T16:58:27.369527+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":43,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":225,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596827+0000","last_change":"2024-02-12T16:57:47.178799+0000","last_active":"2024-02-12T16:58:23.596827+0000","last_peered":"2024-02-12T16:58:23.596827+0000","last_clean":"2024-02-12T16:58:23.596827+0000","last_became_active":"2024-02-12T16:57:47.178469+0000","last_became_peered":"2024-02-12T16:57:47.178469+0000","last_unstale":"2024-02-12T16:58:23.596827+0000","last_undegraded":"2024-02-12T16:58:23.596827+0000","last_fullsized":"2024-02-12T16:58:23.596827+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":120,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.640015+0000","last_change":"2024-02-12T16:57:47.220507+0000","last_active":"2024-02-12T16:58:23.640015+0000","last_peered":"2024-02-12T16:58:23.640015+0000","last_clean":"2024-02-12T16:58:23.640015+0000","last_became_active":"2024-02-12T16:57:47.190968+0000","last_became_peered":"2024-02-12T16:57:47.190968+0000","last_unstale":"2024-02-12T16:58:23.640015+0000","last_undegraded":"2024-02-12T16:58:23.640015+0000","last_fullsized":"2024-02-12T16:58:23.640015+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":221,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596599+0000","last_change":"2024-02-12T16:58:18.266322+0000","last_active":"2024-02-12T16:58:23.596599+0000","last_peered":"2024-02-12T16:58:23.596599+0000","last_clean":"2024-02-12T16:58:23.596599+0000","last_became_active":"2024-02-12T16:58:18.265999+0000","last_became_peered":"2024-02-12T16:58:18.265999+0000","last_unstale":"2024-02-12T16:58:23.596599+0000","last_undegraded":"2024-02-12T16:58:23.596599+0000","last_fullsized":"2024-02-12T16:58:23.596599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"241'7124","reported_seq":7197,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:28.277702+0000","last_change":"2024-02-12T16:56:36.447717+0000","last_active":"2024-02-12T16:58:28.277702+0000","last_peered":"2024-02-12T16:58:28.277702+0000","last_clean":"2024-02-12T16:58:28.277702+0000","last_became_active":"2024-02-12T16:56:36.447417+0000","last_became_peered":"2024-02-12T16:56:36.447417+0000","last_unstale":"2024-02-12T16:58:28.277702+0000","last_undegraded":"2024-02-12T16:58:28.277702+0000","last_fullsized":"2024-02-12T16:58:28.277702+0000","mapping_epoch":176,"log_start":"189'2773","ondisk_log_start":"189'2773","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4351,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":440139776,"num_objects":6716,"num_object_clones":0,"num_object_copies":20148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7124,"num_write_kb":442880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":214,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.640298+0000","last_change":"2024-02-12T16:57:47.191748+0000","last_active":"2024-02-12T16:58:23.640298+0000","last_peered":"2024-02-12T16:58:23.640298+0000","last_clean":"2024-02-12T16:58:23.640298+0000","last_became_active":"2024-02-12T16:57:47.191375+0000","last_became_peered":"2024-02-12T16:57:47.191375+0000","last_unstale":"2024-02-12T16:58:23.640298+0000","last_undegraded":"2024-02-12T16:58:23.640298+0000","last_fullsized":"2024-02-12T16:58:23.640298+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":120,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.637876+0000","last_change":"2024-02-12T16:56:45.218102+0000","last_active":"2024-02-12T16:58:23.637876+0000","last_peered":"2024-02-12T16:58:23.637876+0000","last_clean":"2024-02-12T16:58:23.637876+0000","last_became_active":"2024-02-12T16:54:40.628365+0000","last_became_peered":"2024-02-12T16:54:40.628365+0000","last_unstale":"2024-02-12T16:58:23.637876+0000","last_undegraded":"2024-02-12T16:58:23.637876+0000","last_fullsized":"2024-02-12T16:58:23.637876+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":207,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.636512+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T16:58:23.636512+0000","last_peered":"2024-02-12T16:58:23.636512+0000","last_clean":"2024-02-12T16:58:23.636512+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T16:58:23.636512+0000","last_undegraded":"2024-02-12T16:58:23.636512+0000","last_fullsized":"2024-02-12T16:58:23.636512+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":208,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.603545+0000","last_change":"2024-02-12T16:54:30.770032+0000","last_active":"2024-02-12T16:58:23.603545+0000","last_peered":"2024-02-12T16:58:23.603545+0000","last_clean":"2024-02-12T16:58:23.603545+0000","last_became_active":"2024-02-12T16:49:33.230457+0000","last_became_peered":"2024-02-12T16:49:33.230457+0000","last_unstale":"2024-02-12T16:58:23.603545+0000","last_undegraded":"2024-02-12T16:58:23.603545+0000","last_fullsized":"2024-02-12T16:58:23.603545+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":231,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597862+0000","last_change":"2024-02-12T16:57:19.280994+0000","last_active":"2024-02-12T16:58:23.597862+0000","last_peered":"2024-02-12T16:58:23.597862+0000","last_clean":"2024-02-12T16:58:23.597862+0000","last_became_active":"2024-02-12T16:49:41.669322+0000","last_became_peered":"2024-02-12T16:49:41.669322+0000","last_unstale":"2024-02-12T16:58:23.597862+0000","last_undegraded":"2024-02-12T16:58:23.597862+0000","last_fullsized":"2024-02-12T16:58:23.597862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:19.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.280894+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":201,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604203+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T16:58:23.604203+0000","last_peered":"2024-02-12T16:58:23.604203+0000","last_clean":"2024-02-12T16:58:23.604203+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T16:58:23.604203+0000","last_undegraded":"2024-02-12T16:58:23.604203+0000","last_fullsized":"2024-02-12T16:58:23.604203+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"241'7185","reported_seq":15163,"reported_epoch":241,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:58:28.447106+0000","last_change":"2024-02-12T16:57:48.276877+0000","last_active":"2024-02-12T16:58:28.447106+0000","last_peered":"2024-02-12T16:58:28.447106+0000","last_clean":"2024-02-12T16:57:46.133614+0000","last_became_active":"2024-02-12T16:57:48.264134+0000","last_became_peered":"2024-02-12T16:57:48.264134+0000","last_unstale":"2024-02-12T16:58:28.447106+0000","last_undegraded":"2024-02-12T16:58:28.447106+0000","last_fullsized":"2024-02-12T16:58:28.447106+0000","mapping_epoch":208,"log_start":"189'2543","ondisk_log_start":"189'2543","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4642,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":445186048,"num_objects":6793,"num_object_clones":0,"num_object_copies":20379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2444,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3950,"num_bytes_recovered":258867200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,1],"avail_no_missing":["2","1","5"],"object_location_counts":[{"shards":"1,2,5","objects":6793}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":138,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.637603+0000","last_change":"2024-02-12T16:57:47.192615+0000","last_active":"2024-02-12T16:58:23.637603+0000","last_peered":"2024-02-12T16:58:23.637603+0000","last_clean":"2024-02-12T16:58:23.637603+0000","last_became_active":"2024-02-12T16:57:47.191916+0000","last_became_peered":"2024-02-12T16:57:47.191916+0000","last_unstale":"2024-02-12T16:58:23.637603+0000","last_undegraded":"2024-02-12T16:58:23.637603+0000","last_fullsized":"2024-02-12T16:58:23.637603+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":207,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.190711+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T16:58:17.190711+0000","last_peered":"2024-02-12T16:58:17.190711+0000","last_clean":"2024-02-12T16:58:17.190711+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T16:58:17.190711+0000","last_undegraded":"2024-02-12T16:58:17.190711+0000","last_fullsized":"2024-02-12T16:58:17.190711+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.603492+0000","last_change":"2024-02-12T16:57:48.350132+0000","last_active":"2024-02-12T16:58:23.603492+0000","last_peered":"2024-02-12T16:58:23.603492+0000","last_clean":"2024-02-12T16:58:23.603492+0000","last_became_active":"2024-02-12T16:57:47.200347+0000","last_became_peered":"2024-02-12T16:57:47.200347+0000","last_unstale":"2024-02-12T16:58:23.603492+0000","last_undegraded":"2024-02-12T16:58:23.603492+0000","last_fullsized":"2024-02-12T16:58:23.603492+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":242,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.192138+0000","last_change":"2024-02-12T16:57:47.204108+0000","last_active":"2024-02-12T16:58:17.192138+0000","last_peered":"2024-02-12T16:58:17.192138+0000","last_clean":"2024-02-12T16:58:17.192138+0000","last_became_active":"2024-02-12T16:57:47.184456+0000","last_became_peered":"2024-02-12T16:57:47.184456+0000","last_unstale":"2024-02-12T16:58:17.192138+0000","last_undegraded":"2024-02-12T16:58:17.192138+0000","last_fullsized":"2024-02-12T16:58:17.192138+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":225,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.640226+0000","last_change":"2024-02-12T16:58:14.391518+0000","last_active":"2024-02-12T16:58:23.640226+0000","last_peered":"2024-02-12T16:58:23.640226+0000","last_clean":"2024-02-12T16:58:23.640226+0000","last_became_active":"2024-02-12T16:58:14.391221+0000","last_became_peered":"2024-02-12T16:58:14.391221+0000","last_unstale":"2024-02-12T16:58:23.640226+0000","last_undegraded":"2024-02-12T16:58:23.640226+0000","last_fullsized":"2024-02-12T16:58:23.640226+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"241'7195","reported_seq":16593,"reported_epoch":241,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:58:28.455266+0000","last_change":"2024-02-12T16:57:48.320519+0000","last_active":"2024-02-12T16:58:28.455266+0000","last_peered":"2024-02-12T16:58:28.455266+0000","last_clean":"2024-02-12T16:57:46.109656+0000","last_became_active":"2024-02-12T16:57:48.279326+0000","last_became_peered":"2024-02-12T16:57:48.279326+0000","last_unstale":"2024-02-12T16:58:28.455266+0000","last_undegraded":"2024-02-12T16:58:28.455266+0000","last_fullsized":"2024-02-12T16:58:28.455266+0000","mapping_epoch":208,"log_start":"189'2606","ondisk_log_start":"189'2606","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4589,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":446103552,"num_objects":6807,"num_object_clones":0,"num_object_copies":20421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":6807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7195,"num_write_kb":448064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4657,"num_bytes_recovered":305201152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,1],"avail_no_missing":["2","1","4"],"object_location_counts":[{"shards":"1,2,4","objects":6807}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596426+0000","last_change":"2024-02-12T16:57:47.195886+0000","last_active":"2024-02-12T16:58:23.596426+0000","last_peered":"2024-02-12T16:58:23.596426+0000","last_clean":"2024-02-12T16:58:23.596426+0000","last_became_active":"2024-02-12T16:57:47.195384+0000","last_became_peered":"2024-02-12T16:57:47.195384+0000","last_unstale":"2024-02-12T16:58:23.596426+0000","last_undegraded":"2024-02-12T16:58:23.596426+0000","last_fullsized":"2024-02-12T16:58:23.596426+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.053587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.053587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":205,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604924+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T16:58:23.604924+0000","last_peered":"2024-02-12T16:58:23.604924+0000","last_clean":"2024-02-12T16:58:23.604924+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T16:58:23.604924+0000","last_undegraded":"2024-02-12T16:58:23.604924+0000","last_fullsized":"2024-02-12T16:58:23.604924+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":237,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.189654+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T16:58:17.189654+0000","last_peered":"2024-02-12T16:58:17.189654+0000","last_clean":"2024-02-12T16:58:17.189654+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T16:58:17.189654+0000","last_undegraded":"2024-02-12T16:58:17.189654+0000","last_fullsized":"2024-02-12T16:58:17.189654+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":207,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596500+0000","last_change":"2024-02-12T16:57:17.283687+0000","last_active":"2024-02-12T16:58:23.596500+0000","last_peered":"2024-02-12T16:58:23.596500+0000","last_clean":"2024-02-12T16:58:23.596500+0000","last_became_active":"2024-02-12T16:49:43.198478+0000","last_became_peered":"2024-02-12T16:49:43.198478+0000","last_unstale":"2024-02-12T16:58:23.596500+0000","last_undegraded":"2024-02-12T16:58:23.596500+0000","last_fullsized":"2024-02-12T16:58:23.596500+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"239'6985","reported_seq":7056,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.449888+0000","last_change":"2024-02-12T16:56:36.789146+0000","last_active":"2024-02-12T16:58:23.449888+0000","last_peered":"2024-02-12T16:58:23.449888+0000","last_clean":"2024-02-12T16:58:23.449888+0000","last_became_active":"2024-02-12T16:56:36.788871+0000","last_became_peered":"2024-02-12T16:56:36.788871+0000","last_unstale":"2024-02-12T16:58:23.449888+0000","last_undegraded":"2024-02-12T16:58:23.449888+0000","last_fullsized":"2024-02-12T16:58:23.449888+0000","mapping_epoch":176,"log_start":"189'2356","ondisk_log_start":"189'2356","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4629,"ondisk_log_size":4629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":457768960,"num_objects":6985,"num_object_clones":0,"num_object_copies":20955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":447040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":123,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596814+0000","last_change":"2024-02-12T16:57:32.780213+0000","last_active":"2024-02-12T16:58:23.596814+0000","last_peered":"2024-02-12T16:58:23.596814+0000","last_clean":"2024-02-12T16:58:23.596814+0000","last_became_active":"2024-02-12T16:54:40.631423+0000","last_became_peered":"2024-02-12T16:54:40.631423+0000","last_unstale":"2024-02-12T16:58:23.596814+0000","last_undegraded":"2024-02-12T16:58:23.596814+0000","last_fullsized":"2024-02-12T16:58:23.596814+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.598203+0000","last_change":"2024-02-12T16:57:47.196203+0000","last_active":"2024-02-12T16:58:23.598203+0000","last_peered":"2024-02-12T16:58:23.598203+0000","last_clean":"2024-02-12T16:58:23.598203+0000","last_became_active":"2024-02-12T16:57:47.195926+0000","last_became_peered":"2024-02-12T16:57:47.195926+0000","last_unstale":"2024-02-12T16:58:23.598203+0000","last_undegraded":"2024-02-12T16:58:23.598203+0000","last_fullsized":"2024-02-12T16:58:23.598203+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":209,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.603699+0000","last_change":"2024-02-12T16:57:48.521837+0000","last_active":"2024-02-12T16:58:23.603699+0000","last_peered":"2024-02-12T16:58:23.603699+0000","last_clean":"2024-02-12T16:58:23.603699+0000","last_became_active":"2024-02-12T16:57:47.199763+0000","last_became_peered":"2024-02-12T16:57:47.199763+0000","last_unstale":"2024-02-12T16:58:23.603699+0000","last_undegraded":"2024-02-12T16:58:23.603699+0000","last_fullsized":"2024-02-12T16:58:23.603699+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":213,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.593625+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T16:58:23.593625+0000","last_peered":"2024-02-12T16:58:23.593625+0000","last_clean":"2024-02-12T16:58:23.593625+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T16:58:23.593625+0000","last_undegraded":"2024-02-12T16:58:23.593625+0000","last_fullsized":"2024-02-12T16:58:23.593625+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"241'7279","reported_seq":7353,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:28.327043+0000","last_change":"2024-02-12T16:56:36.788810+0000","last_active":"2024-02-12T16:58:28.327043+0000","last_peered":"2024-02-12T16:58:28.327043+0000","last_clean":"2024-02-12T16:58:28.327043+0000","last_became_active":"2024-02-12T16:56:36.788518+0000","last_became_peered":"2024-02-12T16:56:36.788518+0000","last_unstale":"2024-02-12T16:58:28.327043+0000","last_undegraded":"2024-02-12T16:58:28.327043+0000","last_fullsized":"2024-02-12T16:58:28.327043+0000","mapping_epoch":176,"log_start":"189'3185","ondisk_log_start":"189'3185","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4094,"ondisk_log_size":4094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":453509144,"num_objects":6921,"num_object_clones":0,"num_object_copies":20763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7279,"num_write_kb":454337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":200,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.605749+0000","last_change":"2024-02-12T16:57:19.955408+0000","last_active":"2024-02-12T16:58:23.605749+0000","last_peered":"2024-02-12T16:58:23.605749+0000","last_clean":"2024-02-12T16:58:23.605749+0000","last_became_active":"2024-02-12T16:49:45.223231+0000","last_became_peered":"2024-02-12T16:49:45.223231+0000","last_unstale":"2024-02-12T16:58:23.605749+0000","last_undegraded":"2024-02-12T16:58:23.605749+0000","last_fullsized":"2024-02-12T16:58:23.605749+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:57:19.955311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.955311+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597576+0000","last_change":"2024-02-12T16:56:26.337548+0000","last_active":"2024-02-12T16:58:23.597576+0000","last_peered":"2024-02-12T16:58:23.597576+0000","last_clean":"2024-02-12T16:58:23.597576+0000","last_became_active":"2024-02-12T16:49:39.106969+0000","last_became_peered":"2024-02-12T16:49:39.106969+0000","last_unstale":"2024-02-12T16:58:23.597576+0000","last_undegraded":"2024-02-12T16:58:23.597576+0000","last_fullsized":"2024-02-12T16:58:23.597576+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":198,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.635759+0000","last_change":"2024-02-12T16:57:18.681588+0000","last_active":"2024-02-12T16:58:23.635759+0000","last_peered":"2024-02-12T16:58:23.635759+0000","last_clean":"2024-02-12T16:58:23.635759+0000","last_became_active":"2024-02-12T16:52:20.563744+0000","last_became_peered":"2024-02-12T16:52:20.563744+0000","last_unstale":"2024-02-12T16:58:23.635759+0000","last_undegraded":"2024-02-12T16:58:23.635759+0000","last_fullsized":"2024-02-12T16:58:23.635759+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"208'10","reported_seq":77,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:26.160960+0000","last_change":"2024-02-12T16:57:47.193205+0000","last_active":"2024-02-12T16:58:26.160960+0000","last_peered":"2024-02-12T16:58:26.160960+0000","last_clean":"2024-02-12T16:58:26.160960+0000","last_became_active":"2024-02-12T16:57:47.167628+0000","last_became_peered":"2024-02-12T16:57:47.167628+0000","last_unstale":"2024-02-12T16:58:26.160960+0000","last_undegraded":"2024-02-12T16:58:26.160960+0000","last_fullsized":"2024-02-12T16:58:26.160960+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"241'7077","reported_seq":7150,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:28.355740+0000","last_change":"2024-02-12T16:56:36.445962+0000","last_active":"2024-02-12T16:58:28.355740+0000","last_peered":"2024-02-12T16:58:28.355740+0000","last_clean":"2024-02-12T16:58:28.355740+0000","last_became_active":"2024-02-12T16:56:36.445638+0000","last_became_peered":"2024-02-12T16:56:36.445638+0000","last_unstale":"2024-02-12T16:58:28.355740+0000","last_undegraded":"2024-02-12T16:58:28.355740+0000","last_fullsized":"2024-02-12T16:58:28.355740+0000","mapping_epoch":176,"log_start":"189'2773","ondisk_log_start":"189'2773","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4304,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":439681024,"num_objects":6709,"num_object_clones":0,"num_object_copies":20127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":115,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597183+0000","last_change":"2024-02-12T16:54:40.629725+0000","last_active":"2024-02-12T16:58:23.597183+0000","last_peered":"2024-02-12T16:58:23.597183+0000","last_clean":"2024-02-12T16:58:23.597183+0000","last_became_active":"2024-02-12T16:54:40.629348+0000","last_became_peered":"2024-02-12T16:54:40.629348+0000","last_unstale":"2024-02-12T16:58:23.597183+0000","last_undegraded":"2024-02-12T16:58:23.597183+0000","last_fullsized":"2024-02-12T16:58:23.597183+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"203'11","reported_seq":232,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:26.176088+0000","last_change":"2024-02-12T16:57:42.225984+0000","last_active":"2024-02-12T16:58:26.176088+0000","last_peered":"2024-02-12T16:58:26.176088+0000","last_clean":"2024-02-12T16:58:26.176088+0000","last_became_active":"2024-02-12T16:57:42.225675+0000","last_became_peered":"2024-02-12T16:57:42.225675+0000","last_unstale":"2024-02-12T16:58:26.176088+0000","last_undegraded":"2024-02-12T16:58:26.176088+0000","last_fullsized":"2024-02-12T16:58:26.176088+0000","mapping_epoch":61,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":193,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.595625+0000","last_change":"2024-02-12T16:57:47.171714+0000","last_active":"2024-02-12T16:58:23.595625+0000","last_peered":"2024-02-12T16:58:23.595625+0000","last_clean":"2024-02-12T16:58:23.595625+0000","last_became_active":"2024-02-12T16:57:47.171128+0000","last_became_peered":"2024-02-12T16:57:47.171128+0000","last_unstale":"2024-02-12T16:58:23.595625+0000","last_undegraded":"2024-02-12T16:58:23.595625+0000","last_fullsized":"2024-02-12T16:58:23.595625+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":243,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.598326+0000","last_change":"2024-02-12T16:57:47.656220+0000","last_active":"2024-02-12T16:58:23.598326+0000","last_peered":"2024-02-12T16:58:23.598326+0000","last_clean":"2024-02-12T16:58:23.598326+0000","last_became_active":"2024-02-12T16:57:47.199648+0000","last_became_peered":"2024-02-12T16:57:47.199648+0000","last_unstale":"2024-02-12T16:58:23.598326+0000","last_undegraded":"2024-02-12T16:58:23.598326+0000","last_fullsized":"2024-02-12T16:58:23.598326+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"241'7215","reported_seq":8230,"reported_epoch":241,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:58:28.319606+0000","last_change":"2024-02-12T16:57:48.265426+0000","last_active":"2024-02-12T16:58:28.319606+0000","last_peered":"2024-02-12T16:58:28.319606+0000","last_clean":"2024-02-12T16:57:46.091774+0000","last_became_active":"2024-02-12T16:57:48.228369+0000","last_became_peered":"2024-02-12T16:57:48.228369+0000","last_unstale":"2024-02-12T16:58:28.319606+0000","last_undegraded":"2024-02-12T16:58:28.319606+0000","last_fullsized":"2024-02-12T16:58:28.319606+0000","mapping_epoch":208,"log_start":"190'3134","ondisk_log_start":"190'3134","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4081,"ondisk_log_size":4081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":448069632,"num_objects":6837,"num_object_clones":0,"num_object_copies":20511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6321,"num_objects_unfound":0,"num_objects_dirty":6837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7215,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":30539776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5,6","objects":6837}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":210,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.638415+0000","last_change":"2024-02-12T16:57:47.177662+0000","last_active":"2024-02-12T16:58:23.638415+0000","last_peered":"2024-02-12T16:58:23.638415+0000","last_clean":"2024-02-12T16:58:23.638415+0000","last_became_active":"2024-02-12T16:57:47.177329+0000","last_became_peered":"2024-02-12T16:57:47.177329+0000","last_unstale":"2024-02-12T16:58:23.638415+0000","last_undegraded":"2024-02-12T16:58:23.638415+0000","last_fullsized":"2024-02-12T16:58:23.638415+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.152265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"208'12","reported_seq":304,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:26.172938+0000","last_change":"2024-02-12T16:57:46.177864+0000","last_active":"2024-02-12T16:58:26.172938+0000","last_peered":"2024-02-12T16:58:26.172938+0000","last_clean":"2024-02-12T16:58:26.172938+0000","last_became_active":"2024-02-12T16:57:46.177517+0000","last_became_peered":"2024-02-12T16:57:46.177517+0000","last_unstale":"2024-02-12T16:58:26.172938+0000","last_undegraded":"2024-02-12T16:58:26.172938+0000","last_fullsized":"2024-02-12T16:58:26.172938+0000","mapping_epoch":97,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":187,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596582+0000","last_change":"2024-02-12T16:57:47.178821+0000","last_active":"2024-02-12T16:58:23.596582+0000","last_peered":"2024-02-12T16:58:23.596582+0000","last_clean":"2024-02-12T16:58:23.596582+0000","last_became_active":"2024-02-12T16:57:47.178496+0000","last_became_peered":"2024-02-12T16:57:47.178496+0000","last_unstale":"2024-02-12T16:58:23.596582+0000","last_undegraded":"2024-02-12T16:58:23.596582+0000","last_fullsized":"2024-02-12T16:58:23.596582+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"86'22","reported_seq":195,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.604452+0000","last_change":"2024-02-12T16:57:48.169827+0000","last_active":"2024-02-12T16:58:23.604452+0000","last_peered":"2024-02-12T16:58:23.604452+0000","last_clean":"2024-02-12T16:58:23.604452+0000","last_became_active":"2024-02-12T16:57:47.199008+0000","last_became_peered":"2024-02-12T16:57:47.199008+0000","last_unstale":"2024-02-12T16:58:23.604452+0000","last_undegraded":"2024-02-12T16:58:23.604452+0000","last_fullsized":"2024-02-12T16:58:23.604452+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"241'7208","reported_seq":8224,"reported_epoch":241,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:58:28.372593+0000","last_change":"2024-02-12T16:57:48.319452+0000","last_active":"2024-02-12T16:58:28.372593+0000","last_peered":"2024-02-12T16:58:28.372593+0000","last_clean":"2024-02-12T16:57:46.133812+0000","last_became_active":"2024-02-12T16:57:48.272343+0000","last_became_peered":"2024-02-12T16:57:48.272343+0000","last_unstale":"2024-02-12T16:58:28.372593+0000","last_undegraded":"2024-02-12T16:58:28.372593+0000","last_fullsized":"2024-02-12T16:58:28.372593+0000","mapping_epoch":208,"log_start":"189'3166","ondisk_log_start":"189'3166","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4042,"ondisk_log_size":4042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":444727296,"num_objects":6786,"num_object_clones":0,"num_object_copies":20358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6275,"num_objects_unfound":0,"num_objects_dirty":6786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7208,"num_write_kb":447808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":30539776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":6786}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":48,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596688+0000","last_change":"2024-02-12T16:57:47.176436+0000","last_active":"2024-02-12T16:58:23.596688+0000","last_peered":"2024-02-12T16:58:23.596688+0000","last_clean":"2024-02-12T16:58:23.596688+0000","last_became_active":"2024-02-12T16:57:47.176041+0000","last_became_peered":"2024-02-12T16:57:47.176041+0000","last_unstale":"2024-02-12T16:58:23.596688+0000","last_undegraded":"2024-02-12T16:58:23.596688+0000","last_fullsized":"2024-02-12T16:58:23.596688+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"211'10","reported_seq":336,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:26.183530+0000","last_change":"2024-02-12T16:57:50.205871+0000","last_active":"2024-02-12T16:58:26.183530+0000","last_peered":"2024-02-12T16:58:26.183530+0000","last_clean":"2024-02-12T16:58:26.183530+0000","last_became_active":"2024-02-12T16:57:50.205330+0000","last_became_peered":"2024-02-12T16:57:50.205330+0000","last_unstale":"2024-02-12T16:58:26.183530+0000","last_undegraded":"2024-02-12T16:58:26.183530+0000","last_fullsized":"2024-02-12T16:58:26.183530+0000","mapping_epoch":61,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":149,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.603942+0000","last_change":"2024-02-12T16:57:47.191627+0000","last_active":"2024-02-12T16:58:23.603942+0000","last_peered":"2024-02-12T16:58:23.603942+0000","last_clean":"2024-02-12T16:58:23.603942+0000","last_became_active":"2024-02-12T16:57:47.191231+0000","last_became_peered":"2024-02-12T16:57:47.191231+0000","last_unstale":"2024-02-12T16:58:23.603942+0000","last_undegraded":"2024-02-12T16:58:23.603942+0000","last_fullsized":"2024-02-12T16:58:23.603942+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":180,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597375+0000","last_change":"2024-02-12T16:57:47.216753+0000","last_active":"2024-02-12T16:58:23.597375+0000","last_peered":"2024-02-12T16:58:23.597375+0000","last_clean":"2024-02-12T16:58:23.597375+0000","last_became_active":"2024-02-12T16:57:47.195358+0000","last_became_peered":"2024-02-12T16:57:47.195358+0000","last_unstale":"2024-02-12T16:58:23.597375+0000","last_undegraded":"2024-02-12T16:58:23.597375+0000","last_fullsized":"2024-02-12T16:58:23.597375+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"239'6869","reported_seq":7074,"reported_epoch":239,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:58:23.471616+0000","last_change":"2024-02-12T16:58:18.303487+0000","last_active":"2024-02-12T16:58:23.471616+0000","last_peered":"2024-02-12T16:58:23.471616+0000","last_clean":"2024-02-12T16:57:46.113428+0000","last_became_active":"2024-02-12T16:57:48.262168+0000","last_became_peered":"2024-02-12T16:57:48.262168+0000","last_unstale":"2024-02-12T16:58:23.471616+0000","last_undegraded":"2024-02-12T16:58:23.471616+0000","last_fullsized":"2024-02-12T16:58:23.471616+0000","mapping_epoch":208,"log_start":"189'2192","ondisk_log_start":"189'2192","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4677,"ondisk_log_size":4677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":450166784,"num_objects":6869,"num_object_clones":0,"num_object_copies":20607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6807,"num_objects_unfound":0,"num_objects_dirty":6869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6869,"num_write_kb":439616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":6869}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":215,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.597463+0000","last_change":"2024-02-12T16:57:36.798347+0000","last_active":"2024-02-12T16:58:23.597463+0000","last_peered":"2024-02-12T16:58:23.597463+0000","last_clean":"2024-02-12T16:58:23.597463+0000","last_became_active":"2024-02-12T16:54:40.630855+0000","last_became_peered":"2024-02-12T16:54:40.630855+0000","last_unstale":"2024-02-12T16:58:23.597463+0000","last_undegraded":"2024-02-12T16:58:23.597463+0000","last_fullsized":"2024-02-12T16:58:23.597463+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":63,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.594197+0000","last_change":"2024-02-12T16:57:54.275121+0000","last_active":"2024-02-12T16:58:23.594197+0000","last_peered":"2024-02-12T16:58:23.594197+0000","last_clean":"2024-02-12T16:58:23.594197+0000","last_became_active":"2024-02-12T16:57:54.274817+0000","last_became_peered":"2024-02-12T16:57:54.274817+0000","last_unstale":"2024-02-12T16:58:23.594197+0000","last_undegraded":"2024-02-12T16:58:23.594197+0000","last_fullsized":"2024-02-12T16:58:23.594197+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":205,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.605436+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T16:58:23.605436+0000","last_peered":"2024-02-12T16:58:23.605436+0000","last_clean":"2024-02-12T16:58:23.605436+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T16:58:23.605436+0000","last_undegraded":"2024-02-12T16:58:23.605436+0000","last_fullsized":"2024-02-12T16:58:23.605436+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":259,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596765+0000","last_change":"2024-02-12T16:57:47.223949+0000","last_active":"2024-02-12T16:58:23.596765+0000","last_peered":"2024-02-12T16:58:23.596765+0000","last_clean":"2024-02-12T16:58:23.596765+0000","last_became_active":"2024-02-12T16:57:47.181665+0000","last_became_peered":"2024-02-12T16:57:47.181665+0000","last_unstale":"2024-02-12T16:58:23.596765+0000","last_undegraded":"2024-02-12T16:58:23.596765+0000","last_fullsized":"2024-02-12T16:58:23.596765+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"239'6987","reported_seq":7058,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.456823+0000","last_change":"2024-02-12T16:56:36.788521+0000","last_active":"2024-02-12T16:58:23.456823+0000","last_peered":"2024-02-12T16:58:23.456823+0000","last_clean":"2024-02-12T16:58:23.456823+0000","last_became_active":"2024-02-12T16:56:36.788231+0000","last_became_peered":"2024-02-12T16:56:36.788231+0000","last_unstale":"2024-02-12T16:58:23.456823+0000","last_undegraded":"2024-02-12T16:58:23.456823+0000","last_fullsized":"2024-02-12T16:58:23.456823+0000","mapping_epoch":176,"log_start":"189'2283","ondisk_log_start":"189'2283","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4704,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":457900032,"num_objects":6987,"num_object_clones":0,"num_object_copies":20961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6987,"num_write_kb":447168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":208,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.603407+0000","last_change":"2024-02-12T16:57:26.862173+0000","last_active":"2024-02-12T16:58:23.603407+0000","last_peered":"2024-02-12T16:58:23.603407+0000","last_clean":"2024-02-12T16:58:23.603407+0000","last_became_active":"2024-02-12T16:54:40.645556+0000","last_became_peered":"2024-02-12T16:54:40.645556+0000","last_unstale":"2024-02-12T16:58:23.603407+0000","last_undegraded":"2024-02-12T16:58:23.603407+0000","last_fullsized":"2024-02-12T16:58:23.603407+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":226,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.595971+0000","last_change":"2024-02-12T16:57:58.203843+0000","last_active":"2024-02-12T16:58:23.595971+0000","last_peered":"2024-02-12T16:58:23.595971+0000","last_clean":"2024-02-12T16:58:23.595971+0000","last_became_active":"2024-02-12T16:57:58.203578+0000","last_became_peered":"2024-02-12T16:57:58.203578+0000","last_unstale":"2024-02-12T16:58:23.595971+0000","last_undegraded":"2024-02-12T16:58:23.595971+0000","last_fullsized":"2024-02-12T16:58:23.595971+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":207,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596662+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T16:58:23.596662+0000","last_peered":"2024-02-12T16:58:23.596662+0000","last_clean":"2024-02-12T16:58:23.596662+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T16:58:23.596662+0000","last_undegraded":"2024-02-12T16:58:23.596662+0000","last_fullsized":"2024-02-12T16:58:23.596662+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":298,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596423+0000","last_change":"2024-02-12T16:57:47.562707+0000","last_active":"2024-02-12T16:58:23.596423+0000","last_peered":"2024-02-12T16:58:23.596423+0000","last_clean":"2024-02-12T16:58:23.596423+0000","last_became_active":"2024-02-12T16:57:47.199758+0000","last_became_peered":"2024-02-12T16:57:47.199758+0000","last_unstale":"2024-02-12T16:58:23.596423+0000","last_undegraded":"2024-02-12T16:58:23.596423+0000","last_fullsized":"2024-02-12T16:58:23.596423+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:41.290236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.290236+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"241'7184","reported_seq":15316,"reported_epoch":241,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:58:28.380718+0000","last_change":"2024-02-12T16:57:48.313088+0000","last_active":"2024-02-12T16:58:28.380718+0000","last_peered":"2024-02-12T16:58:28.380718+0000","last_clean":"2024-02-12T16:57:46.089336+0000","last_became_active":"2024-02-12T16:57:48.267297+0000","last_became_peered":"2024-02-12T16:57:48.267297+0000","last_unstale":"2024-02-12T16:58:28.380718+0000","last_undegraded":"2024-02-12T16:58:28.380718+0000","last_fullsized":"2024-02-12T16:58:28.380718+0000","mapping_epoch":208,"log_start":"189'2865","ondisk_log_start":"189'2865","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4319,"ondisk_log_size":4319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":440795136,"num_objects":6726,"num_object_clones":0,"num_object_copies":20178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2424,"num_objects_unfound":0,"num_objects_dirty":6726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7184,"num_write_kb":445120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":263782400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,1],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,2,6","objects":6726}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":192,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.191020+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T16:58:17.191020+0000","last_peered":"2024-02-12T16:58:17.191020+0000","last_clean":"2024-02-12T16:58:17.191020+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T16:58:17.191020+0000","last_undegraded":"2024-02-12T16:58:17.191020+0000","last_fullsized":"2024-02-12T16:58:17.191020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":230,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596610+0000","last_change":"2024-02-12T16:58:02.376971+0000","last_active":"2024-02-12T16:58:23.596610+0000","last_peered":"2024-02-12T16:58:23.596610+0000","last_clean":"2024-02-12T16:58:23.596610+0000","last_became_active":"2024-02-12T16:58:02.376628+0000","last_became_peered":"2024-02-12T16:58:02.376628+0000","last_unstale":"2024-02-12T16:58:23.596610+0000","last_undegraded":"2024-02-12T16:58:23.596610+0000","last_fullsized":"2024-02-12T16:58:23.596610+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:19.703739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.703739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":158,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596672+0000","last_change":"2024-02-12T16:57:47.198002+0000","last_active":"2024-02-12T16:58:23.596672+0000","last_peered":"2024-02-12T16:58:23.596672+0000","last_clean":"2024-02-12T16:58:23.596672+0000","last_became_active":"2024-02-12T16:57:47.181619+0000","last_became_peered":"2024-02-12T16:57:47.181619+0000","last_unstale":"2024-02-12T16:58:23.596672+0000","last_undegraded":"2024-02-12T16:58:23.596672+0000","last_fullsized":"2024-02-12T16:58:23.596672+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":236,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-02-12T16:58:17.189831+0000","last_change":"2024-02-12T16:57:47.553993+0000","last_active":"2024-02-12T16:58:17.189831+0000","last_peered":"2024-02-12T16:58:17.189831+0000","last_clean":"2024-02-12T16:58:17.189831+0000","last_became_active":"2024-02-12T16:57:47.189672+0000","last_became_peered":"2024-02-12T16:57:47.189672+0000","last_unstale":"2024-02-12T16:58:17.189831+0000","last_undegraded":"2024-02-12T16:58:17.189831+0000","last_fullsized":"2024-02-12T16:58:17.189831+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"241'7258","reported_seq":7331,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:28.302730+0000","last_change":"2024-02-12T16:56:36.445585+0000","last_active":"2024-02-12T16:58:28.302730+0000","last_peered":"2024-02-12T16:58:28.302730+0000","last_clean":"2024-02-12T16:58:28.302730+0000","last_became_active":"2024-02-12T16:56:36.445291+0000","last_became_peered":"2024-02-12T16:56:36.445291+0000","last_unstale":"2024-02-12T16:58:28.302730+0000","last_undegraded":"2024-02-12T16:58:28.302730+0000","last_fullsized":"2024-02-12T16:58:28.302730+0000","mapping_epoch":176,"log_start":"189'3198","ondisk_log_start":"189'3198","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":4060,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":448004096,"num_objects":6836,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7258,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":196,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.637605+0000","last_change":"2024-02-12T16:54:03.996224+0000","last_active":"2024-02-12T16:58:23.637605+0000","last_peered":"2024-02-12T16:58:23.637605+0000","last_clean":"2024-02-12T16:58:23.637605+0000","last_became_active":"2024-02-12T16:49:45.221647+0000","last_became_peered":"2024-02-12T16:49:45.221647+0000","last_unstale":"2024-02-12T16:58:23.637605+0000","last_undegraded":"2024-02-12T16:58:23.637605+0000","last_fullsized":"2024-02-12T16:58:23.637605+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":221,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.596078+0000","last_change":"2024-02-12T16:56:32.717219+0000","last_active":"2024-02-12T16:58:23.596078+0000","last_peered":"2024-02-12T16:58:23.596078+0000","last_clean":"2024-02-12T16:58:23.596078+0000","last_became_active":"2024-02-12T16:54:40.630768+0000","last_became_peered":"2024-02-12T16:54:40.630768+0000","last_unstale":"2024-02-12T16:58:23.596078+0000","last_undegraded":"2024-02-12T16:58:23.596078+0000","last_fullsized":"2024-02-12T16:58:23.596078+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"178'22","reported_seq":237,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-02-12T16:58:23.635510+0000","last_change":"2024-02-12T16:56:39.138852+0000","last_active":"2024-02-12T16:58:23.635510+0000","last_peered":"2024-02-12T16:58:23.635510+0000","last_clean":"2024-02-12T16:58:23.635510+0000","last_became_active":"2024-02-12T16:49:41.669816+0000","last_became_peered":"2024-02-12T16:49:41.669816+0000","last_unstale":"2024-02-12T16:58:23.635510+0000","last_undegraded":"2024-02-12T16:58:23.635510+0000","last_fullsized":"2024-02-12T16:58:23.635510+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'22","last_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.138758+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"228'10","reported_seq":329,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-12T16:58:26.169945+0000","last_change":"2024-02-12T16:58:06.225276+0000","last_active":"2024-02-12T16:58:26.169945+0000","last_peered":"2024-02-12T16:58:26.169945+0000","last_clean":"2024-02-12T16:58:26.169945+0000","last_became_active":"2024-02-12T16:58:06.224889+0000","last_became_peered":"2024-02-12T16:58:06.224889+0000","last_unstale":"2024-02-12T16:58:26.169945+0000","last_undegraded":"2024-02-12T16:58:26.169945+0000","last_fullsized":"2024-02-12T16:58:26.169945+0000","mapping_epoch":61,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7157907480,"num_objects":109222,"num_object_clones":0,"num_object_copies":327666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32191,"num_objects_unfound":0,"num_objects_dirty":109222,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114090,"num_write_kb":7145921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14089,"num_bytes_recovered":923336704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22505533440,"data_stored":22505521224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69790,"ondisk_log_size":69790,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1231,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":43,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":683,"num_read_kb":440,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":4126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234093,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2526908,"kb_used_data":2229620,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":91218244,"statfs":{"total":95995035648,"available":93407481856,"internally_reserved":0,"allocated":2283130880,"data_stored":2282337275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:57:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.9199999999999999,"15min":0.95999999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.8869999999999996,"5min":36.439,"15min":36.439},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.748,"15min":0.92900000000000005},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.5030000000000001,"5min":36.490000000000002,"15min":36.490000000000002},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:57:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.212,"15min":0.76100000000000001},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":6.9729999999999999,"5min":12.032,"15min":12.032},"last":1.155},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.1830000000000001,"15min":0.752},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":7.4509999999999996,"5min":10.054,"15min":10.054},"last":0.68300000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.2470000000000001,"15min":0.74299999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.7679999999999998,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":1.21},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.4319999999999999,"15min":0.82299999999999995},"min":{"1min":0.27600000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.008,"5min":20.338000000000001,"15min":20.338000000000001},"last":1.127}]},{"osd":3,"last update":"Mon Feb 12 16:57:43 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.8720000000000001,"15min":1.0129999999999999},"min":{"1min":0.35899999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.6360000000000001,"5min":20.433,"15min":20.433},"last":1.27},{"interface":"front","average":{"1min":1.48,"5min":1.954,"15min":1.0249999999999999},"min":{"1min":0.30099999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":7.0380000000000003,"5min":20.393000000000001,"15min":20.393000000000001},"last":1.179}]},{"osd":4,"last update":"Mon Feb 12 16:57:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9729999999999999,"5min":1.3500000000000001,"15min":0.77500000000000002},"min":{"1min":0.27700000000000002,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":50.406999999999996,"5min":50.406999999999996,"15min":50.406999999999996},"last":1.4379999999999999},{"interface":"front","average":{"1min":4.585,"5min":1.4219999999999999,"15min":0.81699999999999995},"min":{"1min":0.34300000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":50.168999999999997,"5min":50.168999999999997,"15min":50.168999999999997},"last":3.9209999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.65700000000000003,"15min":0.56999999999999995},"min":{"1min":0.29299999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.8849999999999998,"5min":5.3390000000000004,"15min":5.3390000000000004},"last":1.587},{"interface":"front","average":{"1min":1.409,"5min":0.76800000000000002,"15min":0.61499999999999999},"min":{"1min":0.33000000000000002,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":7.0659999999999998,"5min":7.0659999999999998,"15min":7.0659999999999998},"last":1.4530000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.74399999999999999,"15min":0.58499999999999996},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.5309999999999997,"5min":7.5309999999999997,"15min":7.5309999999999997},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.143,"5min":0.85599999999999998,"15min":0.66300000000000003},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.7240000000000002,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":1.2509999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528434,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2832108,"kb_used_data":2530724,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":90913044,"statfs":{"total":95995035648,"available":93094957056,"internally_reserved":0,"allocated":2591461376,"data_stored":2590684464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:58:18 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.7290000000000001,"15min":0.93600000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":7.3659999999999997,"5min":10.949,"15min":10.949},"last":2.504},{"interface":"front","average":{"1min":2.431,"5min":1.8979999999999999,"15min":0.997},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.5570000000000004,"5min":10.56,"15min":10.56},"last":2.738}]},{"osd":1,"last update":"Mon Feb 12 16:58:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":1.8200000000000001,"15min":1.0169999999999999},"min":{"1min":0.26700000000000002,"5min":0.251,"15min":0.251},"max":{"1min":10.651,"5min":10.651,"15min":15.987},"last":1.873},{"interface":"front","average":{"1min":2.1899999999999999,"5min":1.827,"15min":1.0289999999999999},"min":{"1min":0.251,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":15.997},"last":2.7650000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:58:18 2024","interfaces":[{"interface":"back","average":{"1min":2.1699999999999999,"5min":1.728,"15min":0.92800000000000005},"min":{"1min":0.39800000000000002,"5min":0.22900000000000001,"15min":0.21099999999999999},"max":{"1min":6.2939999999999996,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":3.4580000000000002},{"interface":"front","average":{"1min":2.0179999999999998,"5min":1.6839999999999999,"15min":0.90400000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.23699999999999999},"max":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"last":2.4849999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:58:19 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.675,"15min":0.93100000000000005},"min":{"1min":0.29799999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":10.683999999999999,"5min":11.212,"15min":11.212},"last":1.915},{"interface":"front","average":{"1min":2.5699999999999998,"5min":1.958,"15min":1.052},"min":{"1min":0.34300000000000003,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":6.7160000000000002,"5min":10.875999999999999,"15min":10.875999999999999},"last":2.7970000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:58:19 2024","interfaces":[{"interface":"back","average":{"1min":2.9569999999999999,"5min":1.171,"15min":0.65300000000000002},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.217},"max":{"1min":11.276999999999999,"5min":11.276999999999999,"15min":11.276999999999999},"last":2.4529999999999998},{"interface":"front","average":{"1min":2.75,"5min":1.212,"15min":0.67900000000000005},"min":{"1min":0.39300000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":10.706,"5min":10.706,"15min":10.706},"last":0.89400000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:58:19 2024","interfaces":[{"interface":"back","average":{"1min":4.5069999999999997,"5min":1.4850000000000001,"15min":0.78100000000000003},"min":{"1min":0.34499999999999997,"5min":0.245,"15min":0.245},"max":{"1min":17.219999999999999,"5min":17.219999999999999,"15min":17.219999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":3.5830000000000002,"5min":1.4119999999999999,"15min":0.76600000000000001},"min":{"1min":0.34799999999999998,"5min":0.26700000000000002,"15min":0.247},"max":{"1min":15.388,"5min":15.388,"15min":15.388},"last":2.548}]},{"osd":7,"last update":"Mon Feb 12 16:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.92100000000000004,"15min":0.61899999999999999},"min":{"1min":0.28899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.9470000000000001,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":2.8450000000000002},{"interface":"front","average":{"1min":1.0620000000000001,"5min":0.83799999999999997,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.27900000000000003},"max":{"1min":3.2679999999999998,"5min":3.9049999999999998,"15min":3.9049999999999998},"last":2.7799999999999998}]}]},{"osd":1,"up_from":0,"seq":55834574986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738510,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3404752,"kb_used_data":3107528,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":90340400,"statfs":{"total":95995035648,"available":92508569600,"internally_reserved":0,"allocated":3182108672,"data_stored":3181345896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:57:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":2.2320000000000002,"15min":0.96599999999999997},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":8.7010000000000005,"5min":23.510999999999999,"15min":23.510999999999999},"last":0.217},{"interface":"front","average":{"1min":1.0609999999999999,"5min":2.1539999999999999,"15min":0.93700000000000006},"min":{"1min":0.21099999999999999,"5min":0.193,"15min":0.193},"max":{"1min":4.1749999999999998,"5min":10.186999999999999,"15min":10.186999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:57:56 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":2.6280000000000001,"15min":1.1859999999999999},"min":{"1min":0.216,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.9370000000000003,"5min":16.521000000000001,"15min":16.521000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.8169999999999999,"5min":2.3100000000000001,"15min":1.0509999999999999},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.149,"5min":10.340999999999999,"15min":10.340999999999999},"last":0.27000000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:58:20 2024","interfaces":[{"interface":"back","average":{"1min":2.3879999999999999,"5min":2.8860000000000001,"15min":1.2789999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":16.27,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.5660000000000001,"5min":2.5430000000000001,"15min":1.2569999999999999},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.27600000000000002},"max":{"1min":5.9059999999999997,"5min":16.446000000000002,"15min":16.446000000000002},"last":5.9059999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:57:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.756,"15min":0.96699999999999997},"min":{"1min":0.33700000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.4239999999999999,"5min":16.027000000000001,"15min":16.027000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.032,"5min":1.5169999999999999,"15min":0.89300000000000002},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.5910000000000002,"5min":10.141,"15min":10.141},"last":0.81100000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:58:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":2.4540000000000002,"15min":1.2190000000000001},"min":{"1min":0.38,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":16.190999999999999,"5min":16.190999999999999,"15min":16.190999999999999},"last":0.372},{"interface":"front","average":{"1min":3.089,"5min":2.2349999999999999,"15min":1.1639999999999999},"min":{"1min":0.40000000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":18.170999999999999,"5min":18.170999999999999,"15min":18.170999999999999},"last":0.93400000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.7669999999999999,"15min":0.97299999999999998},"min":{"1min":0.35699999999999998,"5min":0.314,"15min":0.314},"max":{"1min":3.1099999999999999,"5min":12.135,"15min":12.135},"last":4.875},{"interface":"front","average":{"1min":1.119,"5min":2.0830000000000002,"15min":1.081},"min":{"1min":0.40999999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.9500000000000002,"5min":21.062000000000001,"15min":21.062000000000001},"last":0.47599999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:57:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":1.9430000000000001,"15min":1.0640000000000001},"min":{"1min":0.44900000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":16.021000000000001,"5min":16.175000000000001,"15min":16.175000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.9950000000000001,"5min":1.6539999999999999,"15min":0.96699999999999997},"min":{"1min":0.375,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.6920000000000002,"5min":10.025,"15min":10.025},"last":8.9199999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411460,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3813272,"kb_used_data":3511952,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":89931880,"statfs":{"total":95995035648,"available":92090245120,"internally_reserved":0,"allocated":3596238848,"data_stored":3595481995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":2.1000000000000001,"5min":2.9409999999999998,"15min":1.2410000000000001},"min":{"1min":0.192,"5min":0.19,"15min":0.19},"max":{"1min":13.459,"5min":15.808,"15min":15.808},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.52,"5min":2.3039999999999998,"15min":1.0449999999999999},"min":{"1min":0.249,"5min":0.158,"15min":0.158},"max":{"1min":4.5910000000000002,"5min":19.893000000000001,"15min":19.893000000000001},"last":2.7759999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":2.226,"5min":2.9260000000000002,"15min":1.274},"min":{"1min":0.20599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":12.433999999999999,"5min":17.739000000000001,"15min":17.739000000000001},"last":2.2890000000000001},{"interface":"front","average":{"1min":1.956,"5min":3.0350000000000001,"15min":1.2989999999999999},"min":{"1min":0.28000000000000003,"5min":0.27300000000000002,"15min":0.23100000000000001},"max":{"1min":8.2260000000000009,"5min":15.766999999999999,"15min":15.766999999999999},"last":0.437}]},{"osd":3,"last update":"Mon Feb 12 16:58:21 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":3.2490000000000001,"15min":1.4810000000000001},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.23499999999999999},"max":{"1min":5.0670000000000002,"5min":51.579999999999998,"15min":51.579999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.407,"5min":2.6619999999999999,"15min":1.232},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.5800000000000001,"5min":48.988999999999997,"15min":48.988999999999997},"last":2.7370000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:57:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":2.0859999999999999,"15min":1.105},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.1100000000000003,"5min":15.737,"15min":15.737},"last":3.1930000000000001},{"interface":"front","average":{"1min":1.569,"5min":1.7450000000000001,"15min":0.98499999999999999},"min":{"1min":0.34799999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.3090000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.202}]},{"osd":5,"last update":"Mon Feb 12 16:57:57 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":2.3730000000000002,"15min":1.2849999999999999},"min":{"1min":0.318,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.0899999999999999,"5min":15.699,"15min":15.699},"last":3.339},{"interface":"front","average":{"1min":2.4359999999999999,"5min":2.4940000000000002,"15min":1.2370000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.8029999999999999,"5min":12.239000000000001,"15min":12.239000000000001},"last":1.2649999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:58:21 2024","interfaces":[{"interface":"back","average":{"1min":3.5339999999999998,"5min":2.8839999999999999,"15min":1.3959999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.31},"max":{"1min":7.8360000000000003,"5min":17.782,"15min":17.782},"last":0.98699999999999999},{"interface":"front","average":{"1min":3.351,"5min":2.343,"15min":1.258},"min":{"1min":0.373,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":8.9459999999999997,"5min":13.446999999999999,"15min":13.446999999999999},"last":2.895}]},{"osd":7,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":2.226,"5min":2.1190000000000002,"15min":1.3049999999999999},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":8.1150000000000002,"5min":13.473000000000001,"15min":13.473000000000001},"last":2.823},{"interface":"front","average":{"1min":2.3069999999999999,"5min":2.4649999999999999,"15min":1.242},"min":{"1min":0.38900000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":8.0549999999999997,"5min":12.294,"15min":12.294},"last":2.7210000000000001}]}]},{"osd":3,"up_from":0,"seq":111669149824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953596,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3897496,"kb_used_data":3587920,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":89847656,"statfs":{"total":95995035648,"available":92003999744,"internally_reserved":0,"allocated":3674030080,"data_stored":3673272870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.72,"15min":0.93200000000000005},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.3849999999999998,"5min":38.179000000000002,"15min":38.179000000000002},"last":3.2959999999999998},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.734,"15min":0.91800000000000004},"min":{"1min":0.29799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8340000000000001,"5min":38.151000000000003,"15min":38.151000000000003},"last":2.6749999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.3859999999999999,"15min":0.80800000000000005},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":3.2730000000000001,"5min":7.673,"15min":7.673},"last":5.6120000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.3939999999999999,"15min":0.82099999999999995},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.4169999999999998,"5min":6.367,"15min":6.367},"last":3.6549999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.4370000000000001,"15min":0.85099999999999998},"min":{"1min":0.26800000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.4429999999999996,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":3.629},{"interface":"front","average":{"1min":1.327,"5min":1.51,"15min":0.879},"min":{"1min":0.35999999999999999,"5min":0.224,"15min":0.224},"max":{"1min":5.875,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":4.9160000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":2.3479999999999999,"15min":1.1479999999999999},"min":{"1min":0.36399999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.891,"5min":38.170999999999999,"15min":38.170999999999999},"last":4.8540000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.9610000000000001,"15min":1.018},"min":{"1min":0.39800000000000002,"5min":0.32400000000000001,"15min":0.32000000000000001},"max":{"1min":2.7549999999999999,"5min":38.311999999999998,"15min":38.311999999999998},"last":5.5439999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:57:58 2024","interfaces":[{"interface":"back","average":{"1min":6.0309999999999997,"5min":1.897,"15min":0.877},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":79.573999999999998,"5min":79.573999999999998,"15min":79.573999999999998},"last":5.5839999999999996},{"interface":"front","average":{"1min":5.3860000000000001,"5min":1.6639999999999999,"15min":0.79500000000000004},"min":{"1min":0.27900000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":80.497,"5min":80.497,"15min":80.497},"last":3.7669999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:57:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.847,"15min":0.872},"min":{"1min":0.27400000000000002,"5min":0.255,"15min":0.255},"max":{"1min":3.0950000000000002,"5min":104.907,"15min":104.907},"last":7.7919999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":2.016,"15min":0.93600000000000005},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.2309999999999999,"5min":105.395,"15min":105.395},"last":5.6719999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:57:44 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.83599999999999997,"15min":0.56999999999999995},"min":{"1min":0.33300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":7.3630000000000004,"5min":7.3630000000000004,"15min":7.3630000000000004},"last":4.8460000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":0.90700000000000003,"15min":0.61199999999999999},"min":{"1min":0.40999999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.0609999999999999,"5min":5.7329999999999997,"15min":5.7329999999999997},"last":4.883}]}]},{"osd":5,"up_from":38,"seq":163208757368,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4254512,"kb_used_data":3948968,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":89490640,"statfs":{"total":95995035648,"available":91638415360,"internally_reserved":0,"allocated":4043743232,"data_stored":4042961848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.712,"15min":0.91800000000000004},"min":{"1min":0.215,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":5.0199999999999996,"5min":36.880000000000003,"15min":36.880000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.6910000000000001,"5min":2.024,"15min":1.0029999999999999},"min":{"1min":0.28399999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.1280000000000001,"5min":36.954000000000001,"15min":36.954000000000001},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.9390000000000001,"15min":0.98599999999999999},"min":{"1min":0.27300000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.6319999999999997,"5min":16.655000000000001,"15min":16.655000000000001},"last":5.7430000000000003},{"interface":"front","average":{"1min":1.3640000000000001,"5min":2.0219999999999998,"15min":1.014},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.8200000000000003,"5min":16.712,"15min":16.712},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":2.0179999999999998,"15min":1.0549999999999999},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.056,"5min":18.218,"15min":18.218},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.468,"5min":2.1190000000000002,"15min":1.044},"min":{"1min":0.29199999999999998,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":5.6479999999999997,"5min":18.327000000000002,"15min":18.327000000000002},"last":5.5129999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6859999999999999,"5min":2.0270000000000001,"15min":1.038},"min":{"1min":0.34300000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.5019999999999998,"5min":18.251000000000001,"15min":18.251000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.554,"5min":1.9079999999999999,"15min":0.98699999999999999},"min":{"1min":0.316,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":7.048,"5min":20.263000000000002,"15min":20.263000000000002},"last":4.8440000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":0.78800000000000003,"15min":0.52900000000000003},"min":{"1min":0.27300000000000002,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":4.8300000000000001,"5min":4.8300000000000001,"15min":4.8300000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.982,"5min":0.94699999999999995,"15min":0.58899999999999997},"min":{"1min":0.40200000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6859999999999999,"5min":5.6859999999999999,"15min":5.6859999999999999},"last":0.84499999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:58:23 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.2,"15min":0.69699999999999995},"min":{"1min":0.309,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.8070000000000004,"5min":5.8070000000000004,"15min":5.8070000000000004},"last":0.69699999999999995},{"interface":"front","average":{"1min":3.7069999999999999,"5min":1.391,"15min":0.76900000000000002},"min":{"1min":0.33100000000000002,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":22.035,"5min":22.035,"15min":22.035},"last":0.67700000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":0.93400000000000005,"15min":0.63500000000000001},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.2560000000000002,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":4.8109999999999999},{"interface":"front","average":{"1min":1.827,"5min":0.98699999999999999,"15min":0.66400000000000003},"min":{"1min":0.32400000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.8630000000000004,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":0.434}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3190882304,"data_stored":3190882304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3192848384,"data_stored":3192848384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3583184896,"data_stored":3583180824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3667722240,"data_stored":3667722240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4021424128,"data_stored":4021420056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2580480000,"data_stored":2580480000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2268991488,"data_stored":2268987416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:58:30.376 INFO:tasks.thrashosds.thrasher:Setting 2.16 to [5, 1, 0, 6, 4, 2] 2024-02-12T16:58:30.377 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.16', '5', '1', '0', '6', '4', '2'] 2024-02-12T16:58:30.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pg-upmap-items 2.16 5 1 0 6 4 2 2024-02-12T16:58:30.464 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:30.525 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:30.537 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:58:30.537 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:58:30.537 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:58:30.537 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:58:30.537 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:58:30.537 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:58:30.537 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:58:30.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:58:30.557 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:30.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:30.631 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:29.259067+0000 mgr.y (mgr.24481) 339 : audit [DBG] from='client.25165 v1:172.21.15.33:0/1487403576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:58:30.632 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:30.205557+0000 mon.c (mon.2) 138 : audit [DBG] from='client.? v1:172.21.15.33:0/4287865967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:30.662 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:30.707 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:30.753 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:30.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:30.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:30.883 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:58:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:30] "GET /metrics HTTP/1.1" 200 175927 "" "Prometheus/2.33.4" 2024-02-12T16:58:30.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:29.259067+0000 mgr.y (mgr.24481) 339 : audit [DBG] from='client.25165 v1:172.21.15.33:0/1487403576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:58:30.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:30.205557+0000 mon.c (mon.2) 138 : audit [DBG] from='client.? v1:172.21.15.33:0/4287865967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:30.920 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:29.259067+0000 mgr.y (mgr.24481) 339 : audit [DBG] from='client.25165 v1:172.21.15.33:0/1487403576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:58:30.920 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:30.205557+0000 mon.c (mon.2) 138 : audit [DBG] from='client.? v1:172.21.15.33:0/4287865967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:30.934 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:30.969 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:30.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:30.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:31.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:31.137 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:31.175 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:31.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:31.287 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:31.363 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:31.401 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:31.402 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:31.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:31.414 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:31.417 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:31.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:31.560 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:31.606 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:31.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:31.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:31.808 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:30.525610+0000 mgr.y (mgr.24481) 340 : cluster [DBG] pgmap v429: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 903 op/s; 31931/325758 objects misplaced (9.802%); 36 MiB/s, 577 objects/s recovering 2024-02-12T16:58:31.808 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:31.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:31.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:32.016 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:30.525610+0000 mgr.y (mgr.24481) 340 : cluster [DBG] pgmap v429: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 903 op/s; 31931/325758 objects misplaced (9.802%); 36 MiB/s, 577 objects/s recovering 2024-02-12T16:58:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:30.525610+0000 mgr.y (mgr.24481) 340 : cluster [DBG] pgmap v429: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.4 KiB/s rd, 31 MiB/s wr, 903 op/s; 31931/325758 objects misplaced (9.802%); 36 MiB/s, 577 objects/s recovering 2024-02-12T16:58:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:32.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:32.162 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:32.200 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:32.419 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:32.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:32.527 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:32.589 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:32.653 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:32.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:32.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:32.829 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:58:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:58:32.846 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:32.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:32.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:33.056 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:33.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:33.190 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:33.257 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:58:33.283 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:33.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:33.388 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:33.456 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:33.492 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:33.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:33.560 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:32.483061+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:33.561 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:32.526669+0000 mgr.y (mgr.24481) 341 : cluster [DBG] pgmap v430: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 687 B/s rd, 337 KiB/s wr, 371 op/s; 31931/325758 objects misplaced (9.802%); 21 MiB/s, 334 objects/s recovering 2024-02-12T16:58:33.561 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:32.483061+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:33.561 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:32.526669+0000 mgr.y (mgr.24481) 341 : cluster [DBG] pgmap v430: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 687 B/s rd, 337 KiB/s wr, 371 op/s; 31931/325758 objects misplaced (9.802%); 21 MiB/s, 334 objects/s recovering 2024-02-12T16:58:33.596 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:33.633 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:33.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:32.483061+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:33.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:32.526669+0000 mgr.y (mgr.24481) 341 : cluster [DBG] pgmap v430: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 687 B/s rd, 337 KiB/s wr, 371 op/s; 31931/325758 objects misplaced (9.802%); 21 MiB/s, 334 objects/s recovering 2024-02-12T16:58:34.094 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:34.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:34.200 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:34.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:58:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:58:34.282 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:34.319 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:34.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:34.425 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:34.471 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:34.583 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:34.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:34.686 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:34.730 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:33.846449+0000 mgr.y (mgr.24481) 342 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:34.731 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:33.846449+0000 mgr.y (mgr.24481) 342 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:34.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:33.846449+0000 mgr.y (mgr.24481) 342 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:34.775 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:34.806 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:34.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:34.918 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:34.957 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:35.518 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:34.528060+0000 mgr.y (mgr.24481) 343 : cluster [DBG] pgmap v431: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 301 KiB/s wr, 726 op/s; 29154/313905 objects misplaced (9.288%); 29 MiB/s, 470 objects/s recovering 2024-02-12T16:58:35.716 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:35.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:35.789 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:34.528060+0000 mgr.y (mgr.24481) 343 : cluster [DBG] pgmap v431: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 301 KiB/s wr, 726 op/s; 29154/313905 objects misplaced (9.288%); 29 MiB/s, 470 objects/s recovering 2024-02-12T16:58:35.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:35.864 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:34.528060+0000 mgr.y (mgr.24481) 343 : cluster [DBG] pgmap v431: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 301 KiB/s wr, 726 op/s; 29154/313905 objects misplaced (9.288%); 29 MiB/s, 470 objects/s recovering 2024-02-12T16:58:35.878 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:35.910 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:35.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:36.018 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:36.076 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:36.283 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:36.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:36.393 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:36.454 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:36.493 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:36.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:36.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:36.658 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:36.780 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:58:36.780 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T16:58:36.780 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79151 7.6 7:69fe64ad:::benchmark_data_smithi033_81060_object79150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.435509+0000", 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 309.226997686, 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.080437704999999998, 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79151 2024-02-12T16:58:36.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435509+0000", 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435509+0000", 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435508+0000", 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5213999999999999e-05 2024-02-12T16:58:36.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435534+0000", 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0950000000000001e-06 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435540+0000", 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2889999999999999e-06 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.435544+0000", 2024-02-12T16:58:36.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000888702 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.436433+0000", 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105937 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.436539+0000", 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013627600000000001 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:58:36.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.436675+0000", 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048124750000000001 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441487+0000", 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000209242 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441697+0000", 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074153001999999996 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.785 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515850+0000", 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6749000000000002e-05 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515906+0000", 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0556000000000003e-05 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515947+0000", 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.080437704999999998 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79160 7.f 7:f96ee9b4:::benchmark_data_smithi033_81060_object79159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.445660+0000", 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 309.216847105, 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077457349999999994, 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79160 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445660+0000", 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445660+0000", 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445658+0000", 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5193000000000001e-05 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445683+0000", 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9250000000000001e-06 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445692+0000", 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6190000000000002e-06 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445697+0000", 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4807e-05 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445741+0000", 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012875000000000001 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445870+0000", 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066321670999999999 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,7", 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512192+0000", 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0033652640000000002 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515557+0000", 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011440129999999999 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.516701+0000", 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0063113459999999998 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.523012+0000", 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4727000000000006e-05 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.523077+0000", 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.998e-05 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.523117+0000", 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077457349999999994 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82089 8.c 8:31b6549c:::benchmark_data_smithi033_105085_object82088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e203)", 2024-02-12T16:58:36.792 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:43.226632+0000", 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.435875285000002, 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077315903000000005, 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82089 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226632+0000", 2024-02-12T16:58:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226632+0000", 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226631+0000", 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7044999999999999e-05 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226658+0000", 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0825999999999999e-05 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226669+0000", 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6289999999999998e-06 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226674+0000", 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0030726730000000002 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.229747+0000", 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017369799999999999 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.229921+0000", 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021196600000000001 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.230132+0000", 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085584570000000002 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.238691+0000", 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051109399999999996 2024-02-12T16:58:36.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.243802+0000", 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.060073427999999998 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.303875+0000", 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8711000000000003e-05 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.303914+0000", 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3453e-05 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.303947+0000", 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077315903000000005 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82091 8.c 8:384a84d5:::benchmark_data_smithi033_105085_object82090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e203)", 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:43.226680+0000", 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.435826538000001, 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.087238804000000003, 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.798 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82091 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226680+0000", 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226680+0000", 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226679+0000", 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6044e-05 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226705+0000", 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8340000000000001e-06 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226712+0000", 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9559999999999996e-06 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.226717+0000", 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061355840000000004 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.232853+0000", 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014031699999999999 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.232993+0000", 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018073500000000001 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:58:36.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.233174+0000", 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013908406999999999 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.247082+0000", 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001469649 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.248552+0000", 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065291842000000003 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.313844+0000", 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8186000000000002e-05 2024-02-12T16:58:36.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.313892+0000", 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7109000000000001e-05 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.313919+0000", 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.087238804000000003 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82097 8.c 8:3316d78f:::benchmark_data_smithi033_105085_object82096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e203)", 2024-02-12T16:58:36.803 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:43.234742+0000", 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.427764815000003, 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.079242409999999999, 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82097 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234742+0000", 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234742+0000", 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999967 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234739+0000", 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3455000000000003e-05 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234782+0000", 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6073999999999999e-05 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234798+0000", 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3019999999999995e-06 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234806+0000", 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061370399999999996 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235420+0000", 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00050650000000000001 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235927+0000", 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018109199999999999 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.236108+0000", 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012452282 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.248560+0000", 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042065512999999999 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.290625+0000", 2024-02-12T16:58:36.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023316232999999999 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.313942+0000", 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3059000000000001e-05 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.313965+0000", 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9677000000000002e-05 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.313984+0000", 2024-02-12T16:58:36.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.079242409999999999 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82098 8.c 8:38294761:::benchmark_data_smithi033_105085_object82097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e203)", 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:43.234820+0000", 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.427687212000002, 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.083448426000000006, 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82098 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.809 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234820+0000", 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234820+0000", 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234819+0000", 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1410999999999999e-05 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234840+0000", 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2689999999999996e-06 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234846+0000", 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2910000000000001e-06 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.234851+0000", 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0046439849999999998 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.239495+0000", 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000144394 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.239639+0000", 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018750799999999999 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.239827+0000", 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008739999 2024-02-12T16:58:36.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.248567+0000", 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042088329000000001 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.290655+0000", 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027525197000000001 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.318180+0000", 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8711999999999997e-05 2024-02-12T16:58:36.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.318229+0000", 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9196000000000002e-05 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.318268+0000", 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.083448426000000006 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82973 8.c 8:32ba51af:::benchmark_data_smithi033_105085_object82972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:44.096689+0000", 2024-02-12T16:58:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.565817951, 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077955485000000005, 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82973 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096689+0000", 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.815 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096689+0000", 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096688+0000", 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6183000000000001e-05 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096705+0000", 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1099999999999992e-06 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096714+0000", 2024-02-12T16:58:36.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5519999999999999e-06 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096716+0000", 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.142e-05 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096758+0000", 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015190200000000001 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096910+0000", 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000210001 2024-02-12T16:58:36.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.097120+0000", 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011942843999999999 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.109062+0000", 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036179099999999998 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.109424+0000", 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065143115000000001 2024-02-12T16:58:36.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.174567+0000", 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6424999999999998e-05 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.174604+0000", 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0572999999999999e-05 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.174644+0000", 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077955485000000005 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.819 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:85073 8.1 8:885b575d:::benchmark_data_smithi033_105085_object85072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e207)", 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:46.171248+0000", 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 50.491258934999998, 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1227682410000002, 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 85073 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.820 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.171248+0000", 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.171248+0000", 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.171246+0000", 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8768e-05 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.171275+0000", 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5310000000000001e-06 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.171283+0000", 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.981e-06 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.171289+0000", 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0033448169999999999 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174634+0000", 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2972999999999999e-05 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174647+0000", 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99182255900000005 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.166470+0000", 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6780000000000003e-06 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.166476+0000", 2024-02-12T16:58:36.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1005852009999999 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.267061+0000", 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018182540000000001 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.268880+0000", 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024929200000000002 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.269129+0000", 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010905300999999999 2024-02-12T16:58:36.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.280034+0000", 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024238499999999999 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.280277+0000", 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044475549999999997 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.284724+0000", 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091033829999999996 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.293828+0000", 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2111000000000002e-05 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.293890+0000", 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012646099999999999 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.294016+0000", 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1227682410000002 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:85074 8.1 8:80afb67f:::benchmark_data_smithi033_105085_object85073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:46.174586+0000", 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 50.487920482, 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.119628423, 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 85074 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174586+0000", 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174586+0000", 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174584+0000", 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9967e-05 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174614+0000", 2024-02-12T16:58:36.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7950000000000008e-06 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174622+0000", 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5799999999999997e-06 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174628+0000", 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1461000000000002e-05 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174670+0000", 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5839999999999997e-06 2024-02-12T16:58:36.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.174678+0000", 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99181719899999998 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.166495+0000", 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4599999999999997e-06 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.166502+0000", 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1087985199999999 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.275301+0000", 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015669499999999999 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.275457+0000", 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020958500000000001 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.275667+0000", 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043742770000000002 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.280041+0000", 2024-02-12T16:58:36.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047150810000000003 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.284756+0000", 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060314519999999996 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.290788+0000", 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0032670920000000001 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.294055+0000", 2024-02-12T16:58:36.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3306000000000002e-05 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.294098+0000", 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116601 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.294215+0000", 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.119628423 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:101419 8.1 8:8a471f6b:::benchmark_data_smithi033_105085_object101418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-02-12T16:58:36.833 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.088610+0000", 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.573896394999998, 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.13367296300000001, 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 101419 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.088610+0000", 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.088610+0000", 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.088610+0000", 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0000000000000002e-05 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.088630+0000", 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6819999999999997e-06 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.088636+0000", 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6349999999999999e-06 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.088640+0000", 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043557830000000002 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.092996+0000", 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000132865 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.093129+0000", 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000197319 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.093326+0000", 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031646489999999998 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096491+0000", 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4777000000000005e-05 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096575+0000", 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2790000000000004e-06 2024-02-12T16:58:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096585+0000", 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050685319999999997 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101653+0000", 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8788999999999997e-05 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101692+0000", 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12059152099999999 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222283+0000", 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.13367296300000001 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:101424 8.1 8:835c9701:::benchmark_data_smithi033_105085_object101423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.095975+0000", 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.566531696999999, 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12681148, 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 101424 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.095975+0000", 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.095975+0000", 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.095973+0000", 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5443999999999999e-05 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.095998+0000", 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4450000000000002e-06 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096008+0000", 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4749999999999998e-06 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096014+0000", 2024-02-12T16:58:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00062023099999999995 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096635+0000", 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018524700000000001 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096820+0000", 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015736 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096977+0000", 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023260059999999998 2024-02-12T16:58:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.099303+0000", 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.123146323 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222449+0000", 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3350000000000007e-05 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222533+0000", 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0803999999999994e-05 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222614+0000", 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1131999999999998e-05 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222665+0000", 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000121897 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222787+0000", 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12681148 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:101426 8.c 8:3b0468b9:::benchmark_data_smithi033_105085_object101425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.096143+0000", 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.566363443, 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12616466100000001, 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 101426 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096143+0000", 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096143+0000", 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096142+0000", 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3898999999999999e-05 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096166+0000", 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8580000000000006e-06 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096175+0000", 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.883e-06 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096182+0000", 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000140603 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096322+0000", 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000145959 2024-02-12T16:58:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096468+0000", 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015257899999999999 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.096621+0000", 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025650030000000002 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.099186+0000", 2024-02-12T16:58:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.123011603 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222197+0000", 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8940000000000003e-05 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222236+0000", 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8878000000000003e-05 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222275+0000", 2024-02-12T16:58:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2886999999999999e-05 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222308+0000", 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12616466100000001 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.35042.0:101427 8.2 e230/176 8:40d3d2d5:::benchmark_data_smithi033_105085_object101426:head v 230'6282, mlcod=230'6278)", 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.099228+0000", 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.563279217000002, 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12298790599999999, 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.099228+0000", 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.099228+0000", 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.099226+0000", 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8573000000000003e-05 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.099285+0000", 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0626e-05 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.099295+0000", 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.883e-06 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.099302+0000", 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010927199999999999 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.100395+0000", 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3431999999999999e-05 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.100408+0000", 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.121767371 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222176+0000", 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9833999999999998e-05 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222216+0000", 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12298790599999999 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:101419 8.1 e230/208 ondisk, result = 0)", 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.100475+0000", 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.562031384000001, 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.121817857, 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.100475+0000", 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.100475+0000", 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.100474+0000", 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.801e-06 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.100480+0000", 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0070000000000003e-06 2024-02-12T16:58:36.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.100489+0000", 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2739999999999998e-06 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.100494+0000", 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011440090000000001 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101638+0000", 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0710000000000001e-06 2024-02-12T16:58:36.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101647+0000", 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.120646087 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222293+0000", 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.121817857 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:101424 8.1 e230/208 ondisk, result = 0)", 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.101345+0000", 2024-02-12T16:58:36.857 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.561161798000001, 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.121109649, 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101345+0000", 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101345+0000", 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101343+0000", 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0959999999999997e-06 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101349+0000", 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8399000000000001e-05 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101368+0000", 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2310000000000001e-06 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101374+0000", 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12105792 2024-02-12T16:58:36.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222432+0000", 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3009999999999995e-06 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222441+0000", 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3409e-05 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222455+0000", 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.121109649 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.860 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:101424 8.1 e230/208 ondisk, result = 0)", 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.101545+0000", 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.560962249999999, 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.120991948, 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101545+0000", 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101545+0000", 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101544+0000", 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3129999999999998e-06 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101549+0000", 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6490000000000003e-06 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.862 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101559+0000", 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.728e-06 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101564+0000", 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.120952773 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222517+0000", 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5350000000000004e-06 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222526+0000", 2024-02-12T16:58:36.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0873000000000001e-05 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222537+0000", 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.120991948 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:101424 8.1 e230/208 ondisk, result = 0)", 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.101597+0000", 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.560910148000001, 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.121200872, 2024-02-12T16:58:36.864 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101597+0000", 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101597+0000", 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101596+0000", 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1829999999999996e-06 2024-02-12T16:58:36.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101600+0000", 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9169999999999997e-06 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101605+0000", 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8509999999999999e-06 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101608+0000", 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12099164 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222599+0000", 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7160000000000003e-06 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222607+0000", 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019055100000000001 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222798+0000", 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.121200872 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:101426 8.c e230/176 ondisk, result = 0)", 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.101680+0000", 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.560827113999999, 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.120522974, 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101680+0000", 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101680+0000", 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101679+0000", 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.717e-06 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101682+0000", 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1819999999999999e-06 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101686+0000", 2024-02-12T16:58:36.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.88e-06 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.101689+0000", 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.120479974 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222169+0000", 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8592000000000001e-05 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222188+0000", 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5085999999999999e-05 2024-02-12T16:58:36.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222203+0000", 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.120522974 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:101430 8.6 8:618a7f7e:::benchmark_data_smithi033_105085_object101429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.102393+0000", 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.560114092999999, 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.13566904399999999, 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:58:36.871 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 101430 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.102393+0000", 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.102393+0000", 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.102390+0000", 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9428999999999998e-05 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.102430+0000", 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.11995488 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222384+0000", 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6670000000000002e-06 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222393+0000", 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7613000000000001e-05 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222421+0000", 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000129037 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222550+0000", 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014026899999999999 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 6,7", 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222690+0000", 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093814139999999994 2024-02-12T16:58:36.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.232071+0000", 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.005612522 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.237684+0000", 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000286712 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.237971+0000", 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3273000000000003e-05 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:58:36.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.238024+0000", 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7824999999999998e-05 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.238062+0000", 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.13566904399999999 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:101426 8.c e230/176 ondisk, result = 0)", 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:10.108308+0000", 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.554199319999999, 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.11400655799999999, 2024-02-12T16:58:36.876 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.108308+0000", 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.108308+0000", 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.108306+0000", 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0800000000000002e-06 2024-02-12T16:58:36.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.108312+0000", 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.11385007900000001 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222162+0000", 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2037e-05 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222174+0000", 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7500999999999998e-05 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222222+0000", 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8609999999999998e-06 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222230+0000", 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4343000000000002e-05 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:10.222314+0000", 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.11400655799999999 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:58:36.879 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:58:36.880 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:58:36.993 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:37.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:37.100 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:37.140 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:37.251 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:37.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:37.286 INFO:teuthology.orchestra.run.smithi033.stderr:set 2.16 pg_upmap_items mapping to [5->1,0->6,4->2] 2024-02-12T16:58:37.358 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:37.408 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:37.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:36.528468+0000 mgr.y (mgr.24481) 344 : cluster [DBG] pgmap v432: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 290 KiB/s wr, 701 op/s; 29154/313905 objects misplaced (9.288%); 28 MiB/s, 454 objects/s recovering 2024-02-12T16:58:37.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:36.882835+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.33:0/151955887' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]: dispatch 2024-02-12T16:58:37.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:36.883167+0000 mon.a (mon.0) 1100 : audit [INF] from='client.35174 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]: dispatch 2024-02-12T16:58:37.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:36.528468+0000 mgr.y (mgr.24481) 344 : cluster [DBG] pgmap v432: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 290 KiB/s wr, 701 op/s; 29154/313905 objects misplaced (9.288%); 28 MiB/s, 454 objects/s recovering 2024-02-12T16:58:37.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:36.882835+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.33:0/151955887' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]: dispatch 2024-02-12T16:58:37.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:36.883167+0000 mon.a (mon.0) 1100 : audit [INF] from='client.35174 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]: dispatch 2024-02-12T16:58:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:36.528468+0000 mgr.y (mgr.24481) 344 : cluster [DBG] pgmap v432: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 290 KiB/s wr, 701 op/s; 29154/313905 objects misplaced (9.288%); 28 MiB/s, 454 objects/s recovering 2024-02-12T16:58:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:36.882835+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.33:0/151955887' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]: dispatch 2024-02-12T16:58:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:36.883167+0000 mon.a (mon.0) 1100 : audit [INF] from='client.35174 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]: dispatch 2024-02-12T16:58:38.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:37.068249+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:37.068291+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:37.269893+0000 mon.a (mon.0) 1101 : audit [INF] from='client.35174 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]': finished 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:37.269962+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e242: 8 total, 8 up, 6 in 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:38.033975+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.33:0/3884871025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:38.035800+0000 mon.a (mon.0) 1103 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:37.068249+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:37.068291+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:37.269893+0000 mon.a (mon.0) 1101 : audit [INF] from='client.35174 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]': finished 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:37.269962+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e242: 8 total, 8 up, 6 in 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:38.033975+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.33:0/3884871025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:38.035800+0000 mon.a (mon.0) 1103 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:37.068249+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:37.068291+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:37.269893+0000 mon.a (mon.0) 1101 : audit [INF] from='client.35174 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.16", "id": [5, 1, 0, 6, 4, 2]}]': finished 2024-02-12T16:58:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:37.269962+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e242: 8 total, 8 up, 6 in 2024-02-12T16:58:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:38.033975+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.33:0/3884871025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:38.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:38.035800+0000 mon.a (mon.0) 1103 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:58:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:38.267970+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:38.293336+0000 mon.a (mon.0) 1105 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:58:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:38.293407+0000 mon.a (mon.0) 1106 : cluster [DBG] osdmap e243: 8 total, 8 up, 6 in 2024-02-12T16:58:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:38.529043+0000 mgr.y (mgr.24481) 345 : cluster [DBG] pgmap v435: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 869 op/s; 26634/304920 objects misplaced (8.735%); 26 MiB/s, 422 objects/s recovering 2024-02-12T16:58:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:38.267970+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:38.293336+0000 mon.a (mon.0) 1105 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:58:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:38.293407+0000 mon.a (mon.0) 1106 : cluster [DBG] osdmap e243: 8 total, 8 up, 6 in 2024-02-12T16:58:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:38.529043+0000 mgr.y (mgr.24481) 345 : cluster [DBG] pgmap v435: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 869 op/s; 26634/304920 objects misplaced (8.735%); 26 MiB/s, 422 objects/s recovering 2024-02-12T16:58:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:38.267970+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:38.293336+0000 mon.a (mon.0) 1105 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:58:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:38.293407+0000 mon.a (mon.0) 1106 : cluster [DBG] osdmap e243: 8 total, 8 up, 6 in 2024-02-12T16:58:39.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:38.529043+0000 mgr.y (mgr.24481) 345 : cluster [DBG] pgmap v435: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 869 op/s; 26634/304920 objects misplaced (8.735%); 26 MiB/s, 422 objects/s recovering 2024-02-12T16:58:41.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:58:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:40] "GET /metrics HTTP/1.1" 200 175931 "" "Prometheus/2.33.4" 2024-02-12T16:58:42.008 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:40.529558+0000 mgr.y (mgr.24481) 346 : cluster [DBG] pgmap v436: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 26331/301797 objects misplaced (8.725%); 27 MiB/s, 429 objects/s recovering 2024-02-12T16:58:42.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:40.529558+0000 mgr.y (mgr.24481) 346 : cluster [DBG] pgmap v436: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 26331/301797 objects misplaced (8.725%); 27 MiB/s, 429 objects/s recovering 2024-02-12T16:58:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:40.529558+0000 mgr.y (mgr.24481) 346 : cluster [DBG] pgmap v436: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 26331/301797 objects misplaced (8.725%); 27 MiB/s, 429 objects/s recovering 2024-02-12T16:58:42.098 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:42.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:58:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:58:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:41.728650+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:41.728693+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:43.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:41.728650+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:43.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:41.728693+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:41.728650+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:41.728693+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:43.133 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:58:43.861 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:42.529958+0000 mgr.y (mgr.24481) 347 : cluster [DBG] pgmap v437: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 504 op/s; 26331/301797 objects misplaced (8.725%); 13 MiB/s, 215 objects/s recovering 2024-02-12T16:58:43.861 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:42.604270+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e244: 8 total, 8 up, 6 in 2024-02-12T16:58:43.861 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:58:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:58:44.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:42.529958+0000 mgr.y (mgr.24481) 347 : cluster [DBG] pgmap v437: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 504 op/s; 26331/301797 objects misplaced (8.725%); 13 MiB/s, 215 objects/s recovering 2024-02-12T16:58:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:42.604270+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e244: 8 total, 8 up, 6 in 2024-02-12T16:58:44.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:42.529958+0000 mgr.y (mgr.24481) 347 : cluster [DBG] pgmap v437: 138 pgs: 7 active+remapped+backfilling, 131 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 504 op/s; 26331/301797 objects misplaced (8.725%); 13 MiB/s, 215 objects/s recovering 2024-02-12T16:58:44.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:42.604270+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e244: 8 total, 8 up, 6 in 2024-02-12T16:58:44.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:43.616638+0000 mon.a (mon.0) 1108 : cluster [DBG] osdmap e245: 8 total, 8 up, 6 in 2024-02-12T16:58:44.997 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:43.858322+0000 mgr.y (mgr.24481) 348 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:43.616638+0000 mon.a (mon.0) 1108 : cluster [DBG] osdmap e245: 8 total, 8 up, 6 in 2024-02-12T16:58:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:43.858322+0000 mgr.y (mgr.24481) 348 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:43.616638+0000 mon.a (mon.0) 1108 : cluster [DBG] osdmap e245: 8 total, 8 up, 6 in 2024-02-12T16:58:45.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:43.858322+0000 mgr.y (mgr.24481) 348 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:45.557 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:58:45.837 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:44.530306+0000 mgr.y (mgr.24481) 349 : cluster [DBG] pgmap v440: 138 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 131 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.30k op/s; 23554/289656 objects misplaced (8.132%); 35 MiB/s, 561 objects/s recovering 2024-02-12T16:58:45.838 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:44.623869+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e246: 8 total, 8 up, 6 in 2024-02-12T16:58:45.838 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:44.530306+0000 mgr.y (mgr.24481) 349 : cluster [DBG] pgmap v440: 138 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 131 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.30k op/s; 23554/289656 objects misplaced (8.132%); 35 MiB/s, 561 objects/s recovering 2024-02-12T16:58:45.838 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:44.623869+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e246: 8 total, 8 up, 6 in 2024-02-12T16:58:45.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:44.530306+0000 mgr.y (mgr.24481) 349 : cluster [DBG] pgmap v440: 138 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 131 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.30k op/s; 23554/289656 objects misplaced (8.132%); 35 MiB/s, 561 objects/s recovering 2024-02-12T16:58:45.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:44.623869+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e246: 8 total, 8 up, 6 in 2024-02-12T16:58:46.383 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:46.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:46.488 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:46.558 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:46.598 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:46.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:46.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:46.753 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:46.913 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:46.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:47.022 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:47.063 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:47.342 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:46.166400+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e247: 8 total, 8 up, 6 in 2024-02-12T16:58:47.342 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:46.530702+0000 mgr.y (mgr.24481) 350 : cluster [DBG] pgmap v443: 138 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 131 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23554/289656 objects misplaced (8.132%); 28 MiB/s, 444 objects/s recovering 2024-02-12T16:58:47.342 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:46.166400+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e247: 8 total, 8 up, 6 in 2024-02-12T16:58:47.342 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:46.530702+0000 mgr.y (mgr.24481) 350 : cluster [DBG] pgmap v443: 138 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 131 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23554/289656 objects misplaced (8.132%); 28 MiB/s, 444 objects/s recovering 2024-02-12T16:58:47.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:46.166400+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e247: 8 total, 8 up, 6 in 2024-02-12T16:58:47.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:46.530702+0000 mgr.y (mgr.24481) 350 : cluster [DBG] pgmap v443: 138 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 131 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23554/289656 objects misplaced (8.132%); 28 MiB/s, 444 objects/s recovering 2024-02-12T16:58:47.868 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:48.470 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:58:48.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:47.199263+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e248: 8 total, 8 up, 6 in 2024-02-12T16:58:48.493 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:58:48.514 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:58:48.521 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:48.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:48.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:47.199263+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e248: 8 total, 8 up, 6 in 2024-02-12T16:58:48.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:47.199263+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e248: 8 total, 8 up, 6 in 2024-02-12T16:58:48.545 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:58:48.574 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:58:48.599 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:58:48.623 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:58:48.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:48.652 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:58:48.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:58:48.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:58:48.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:58:48.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:58:48.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:58:48.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:58:48.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:58:48.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:58:48.667 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:49.370 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:58:49.416 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:48.210254+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e249: 8 total, 8 up, 6 in 2024-02-12T16:58:49.416 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:48.445650+0000 mon.c (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.33:0/3098911462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:49.416 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:48.531427+0000 mgr.y (mgr.24481) 351 : cluster [DBG] pgmap v446: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 556 op/s; 21212/282993 objects misplaced (7.496%); 28 MiB/s, 448 objects/s recovering 2024-02-12T16:58:49.416 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:48.210254+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e249: 8 total, 8 up, 6 in 2024-02-12T16:58:49.416 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:48.445650+0000 mon.c (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.33:0/3098911462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:49.416 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:48.531427+0000 mgr.y (mgr.24481) 351 : cluster [DBG] pgmap v446: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 556 op/s; 21212/282993 objects misplaced (7.496%); 28 MiB/s, 448 objects/s recovering 2024-02-12T16:58:49.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:48.210254+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e249: 8 total, 8 up, 6 in 2024-02-12T16:58:49.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:48.445650+0000 mon.c (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.33:0/3098911462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:49.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:48.531427+0000 mgr.y (mgr.24481) 351 : cluster [DBG] pgmap v446: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 556 op/s; 21212/282993 objects misplaced (7.496%); 28 MiB/s, 448 objects/s recovering 2024-02-12T16:58:50.223 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:49.212236+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e250: 8 total, 8 up, 6 in 2024-02-12T16:58:50.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:49.212236+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e250: 8 total, 8 up, 6 in 2024-02-12T16:58:50.518 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:49.212236+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e250: 8 total, 8 up, 6 in 2024-02-12T16:58:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:50.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:50.556 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:58:50.557 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:58:50.632 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:50.682 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:51.222 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:58:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:50] "GET /metrics HTTP/1.1" 200 175934 "" "Prometheus/2.33.4" 2024-02-12T16:58:51.222 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:50.531800+0000 mgr.y (mgr.24481) 352 : cluster [DBG] pgmap v448: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 677 op/s; 20998/280800 objects misplaced (7.478%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:51.222 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:51.167076+0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e251: 8 total, 8 up, 6 in 2024-02-12T16:58:51.338 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:51.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:51.380 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 6, 7] out_osds: [3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:58:51.380 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:58:51.380 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-02-12T16:58:51.380 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:58:51.380 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:58:51.381 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-02-12T16:58:51.443 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:51.481 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:50.531800+0000 mgr.y (mgr.24481) 352 : cluster [DBG] pgmap v448: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 677 op/s; 20998/280800 objects misplaced (7.478%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:51.481 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:51.167076+0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e251: 8 total, 8 up, 6 in 2024-02-12T16:58:51.484 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:50.531800+0000 mgr.y (mgr.24481) 352 : cluster [DBG] pgmap v448: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 677 op/s; 20998/280800 objects misplaced (7.478%); 27 MiB/s, 424 objects/s recovering 2024-02-12T16:58:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:51.167076+0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e251: 8 total, 8 up, 6 in 2024-02-12T16:58:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:58:51.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:58:52.141 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:52.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:52.246 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:52.346 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:52.741 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:58:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:58:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:58:53.104 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:58:53.630 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:53.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:53.734 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:53.749 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:53.803 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:53.859 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:53.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:52.532221+0000 mgr.y (mgr.24481) 353 : cluster [DBG] pgmap v450: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 960 B/s rd, 0 B/s wr, 554 op/s; 20998/280800 objects misplaced (7.478%); 22 MiB/s, 347 objects/s recovering 2024-02-12T16:58:53.865 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:52.532221+0000 mgr.y (mgr.24481) 353 : cluster [DBG] pgmap v450: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 960 B/s rd, 0 B/s wr, 554 op/s; 20998/280800 objects misplaced (7.478%); 22 MiB/s, 347 objects/s recovering 2024-02-12T16:58:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:53.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:53.878 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:52.532221+0000 mgr.y (mgr.24481) 353 : cluster [DBG] pgmap v450: 138 pgs: 1 peering, 4 active+remapped+backfilling, 133 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 960 B/s rd, 0 B/s wr, 554 op/s; 20998/280800 objects misplaced (7.478%); 22 MiB/s, 347 objects/s recovering 2024-02-12T16:58:53.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:58:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:58:53.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:54.000 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:54.145 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:54.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:54.254 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:54.287 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:54.620 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:53.873622+0000 mgr.y (mgr.24481) 354 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:54.620 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:54.163665+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.33:0/73498324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:58:54.626 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:58:54.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:53.873622+0000 mgr.y (mgr.24481) 354 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:54.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:54.163665+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.33:0/73498324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:58:54.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:53.873622+0000 mgr.y (mgr.24481) 354 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:58:54.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:54.163665+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.33:0/73498324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:58:54.933 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:54.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:55.051 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:55.199 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:55.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:55.309 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:58:55.393 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T16:58:55.393 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "" 2024-02-12T16:58:55.393 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T16:58:55.393 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:55.675 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:58:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:55.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:58:55.682 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:54.532881+0000 mgr.y (mgr.24481) 355 : cluster [DBG] pgmap v451: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.05k op/s; 19948/269733 objects misplaced (7.395%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:58:55.682 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:54.590861+0000 mon.a (mon.0) 1116 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:55.682 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:54.614339+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? v1:172.21.15.33:0/73498324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:58:55.682 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:54.614398+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e252: 8 total, 8 up, 6 in 2024-02-12T16:58:55.682 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:54.909194+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:55.682 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:54.909247+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:54.532881+0000 mgr.y (mgr.24481) 355 : cluster [DBG] pgmap v451: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.05k op/s; 19948/269733 objects misplaced (7.395%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:54.590861+0000 mon.a (mon.0) 1116 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:54.614339+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? v1:172.21.15.33:0/73498324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:54.614398+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e252: 8 total, 8 up, 6 in 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:54.909194+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:54.909247+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:54.532881+0000 mgr.y (mgr.24481) 355 : cluster [DBG] pgmap v451: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.05k op/s; 19948/269733 objects misplaced (7.395%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:54.590861+0000 mon.a (mon.0) 1116 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:54.614339+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? v1:172.21.15.33:0/73498324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:54.614398+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e252: 8 total, 8 up, 6 in 2024-02-12T16:58:55.749 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:54.909194+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:58:55.750 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:54.909247+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:58:55.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:55.841 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:55.864 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:55.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:55.978 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:58:56.023 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:56.174 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:58:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:56.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:58:56.290 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:56.365 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:56.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:56.476 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:56.518 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:56.634 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:55.630101+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e253: 8 total, 8 up, 6 in 2024-02-12T16:58:56.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:55.630101+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e253: 8 total, 8 up, 6 in 2024-02-12T16:58:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:55.630101+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e253: 8 total, 8 up, 6 in 2024-02-12T16:58:56.875 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:56.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:56.983 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:57.035 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:57.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:57.200 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:58:57.241 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:57.326 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:58:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:57.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:58:57.430 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:57.477 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:57.502 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:58:57.503 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:58:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:57.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:58:57.515 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:58:57.545 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:58:57.583 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:58:57.609 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:57.617 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:58:57.652 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:57.663 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:58:57.717 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:58:57.754 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:58:57.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:56.533340+0000 mgr.y (mgr.24481) 356 : cluster [DBG] pgmap v454: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 615 op/s; 19948/269733 objects misplaced (7.395%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:58:57.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:57.484019+0000 mon.c (mon.2) 142 : audit [DBG] from='client.? v1:172.21.15.33:0/3888169405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:56.533340+0000 mgr.y (mgr.24481) 356 : cluster [DBG] pgmap v454: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 615 op/s; 19948/269733 objects misplaced (7.395%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:58:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:57.484019+0000 mon.c (mon.2) 142 : audit [DBG] from='client.? v1:172.21.15.33:0/3888169405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:57.780 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:58:57.785 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:58:57.785 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:58:57.785 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:58:57.786 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:58:57.786 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:58:57.786 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:58:57.786 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:58:57.786 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:58:57.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:56.533340+0000 mgr.y (mgr.24481) 356 : cluster [DBG] pgmap v454: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 615 op/s; 19948/269733 objects misplaced (7.395%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:58:57.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:57.484019+0000 mon.c (mon.2) 142 : audit [DBG] from='client.? v1:172.21.15.33:0/3888169405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:58:58.009 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:58.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:58.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:58.168 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:58.207 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:58.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:58.320 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:58.385 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:58.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:58.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:58.644 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:58.687 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:58.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:58.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:58:58.867 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:58:58.902 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:57.932596+0000 mon.c (mon.2) 143 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:58:58.902 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:58.406976+0000 mon.c (mon.2) 144 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:58:58.903 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:58.407838+0000 mon.c (mon.2) 145 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:58:58.903 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:58:58.415599+0000 mon.a (mon.0) 1120 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:58.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:58:58.908 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:57.932596+0000 mon.c (mon.2) 143 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:58:58.908 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:58.406976+0000 mon.c (mon.2) 144 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:58:58.908 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:58.407838+0000 mon.c (mon.2) 145 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:58:58.908 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:58:58.415599+0000 mon.a (mon.0) 1120 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:58.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:57.932596+0000 mon.c (mon.2) 143 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:58:58.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:58.406976+0000 mon.c (mon.2) 144 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:58:58.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:58.407838+0000 mon.c (mon.2) 145 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:58:58.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:58:58.415599+0000 mon.a (mon.0) 1120 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:58:59.004 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:58:59.043 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:59.080 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:58:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:59.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:58:59.185 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:58:59.194 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:58:59.248 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:58:59.248 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T16:58:59.248 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T16:58:59.248 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T16:58:59.248 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:58:59.256 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:59.293 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:58:59.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:59.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:58:59.402 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:58:59.443 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:59.471 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:58:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:58:59.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:58:59.575 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:58:59.608 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:58:59.647 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:58:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:58:58.534010+0000 mgr.y (mgr.24481) 357 : cluster [DBG] pgmap v455: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 916 op/s; 19124/260562 objects misplaced (7.340%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:58:59.867 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:58:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:58:58.534010+0000 mgr.y (mgr.24481) 357 : cluster [DBG] pgmap v455: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 916 op/s; 19124/260562 objects misplaced (7.340%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:58:59.867 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:58:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:58:58.534010+0000 mgr.y (mgr.24481) 357 : cluster [DBG] pgmap v455: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 916 op/s; 19124/260562 objects misplaced (7.340%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:59:00.269 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:00.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:00.374 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:00.439 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:00.484 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:00.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:00.596 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:00.634 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:00.677 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:59:00.841 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:00.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:00.875 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:59:00.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:01.045 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:01.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:01.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:01.151 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:01.192 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 6, 7] out_osds: [3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:59:01.192 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:59:01.192 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 2, 4, 5, 6, 7] 2024-02-12T16:59:01.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd out 6 2024-02-12T16:59:01.218 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:01.242 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:59:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:00] "GET /metrics HTTP/1.1" 200 175934 "" "Prometheus/2.33.4" 2024-02-12T16:59:01.270 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:01.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:01.374 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:01.418 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:01.445 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:01.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:01.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:01.623 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:01.672 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:01.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:01.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:01.830 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:01.867 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:01.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:01.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:01.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:00.534411+0000 mgr.y (mgr.24481) 358 : cluster [DBG] pgmap v456: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 969 op/s; 18818/257541 objects misplaced (7.307%); 3.6 MiB/s, 58 objects/s recovering 2024-02-12T16:59:02.014 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:00.534411+0000 mgr.y (mgr.24481) 358 : cluster [DBG] pgmap v456: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 969 op/s; 18818/257541 objects misplaced (7.307%); 3.6 MiB/s, 58 objects/s recovering 2024-02-12T16:59:02.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:00.534411+0000 mgr.y (mgr.24481) 358 : cluster [DBG] pgmap v456: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 969 op/s; 18818/257541 objects misplaced (7.307%); 3.6 MiB/s, 58 objects/s recovering 2024-02-12T16:59:02.016 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:02.175 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:02.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:02.236 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:02.259 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:02.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:02.354 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:02.406 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:02.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:02.511 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:02.603 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:59:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:59:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:02.652 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:02.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:02.705 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T16:59:02.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:02.833 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:02.861 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:02.863 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:02.448781+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.33:0/1641775500' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:02.863 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:02.449107+0000 mon.a (mon.0) 1121 : audit [INF] from='client.35201 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:02.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:02.898 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:02.448781+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.33:0/1641775500' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:02.898 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:02.449107+0000 mon.a (mon.0) 1121 : audit [INF] from='client.35201 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:02.899 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:02.448781+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.33:0/1641775500' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:02.899 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:02.449107+0000 mon.a (mon.0) 1121 : audit [INF] from='client.35201 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:02.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:03.045 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:03.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:03.193 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:03.238 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:03.283 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:03.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:03.397 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:03.477 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:03.523 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:03.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:03.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:03.664 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:02.534782+0000 mgr.y (mgr.24481) 359 : cluster [DBG] pgmap v457: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 508 op/s; 18818/257541 objects misplaced (7.307%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:59:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:02.651429+0000 mon.a (mon.0) 1122 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:59:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:02.651461+0000 mon.a (mon.0) 1123 : cluster [INF] Cluster is now healthy 2024-02-12T16:59:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:02.671502+0000 mon.a (mon.0) 1124 : audit [INF] from='client.35201 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:59:03.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:02.671558+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e254: 8 total, 8 up, 6 in 2024-02-12T16:59:04.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:02.534782+0000 mgr.y (mgr.24481) 359 : cluster [DBG] pgmap v457: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 508 op/s; 18818/257541 objects misplaced (7.307%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:59:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:02.651429+0000 mon.a (mon.0) 1122 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:59:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:02.651461+0000 mon.a (mon.0) 1123 : cluster [INF] Cluster is now healthy 2024-02-12T16:59:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:02.671502+0000 mon.a (mon.0) 1124 : audit [INF] from='client.35201 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:59:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:02.671558+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e254: 8 total, 8 up, 6 in 2024-02-12T16:59:04.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:02.534782+0000 mgr.y (mgr.24481) 359 : cluster [DBG] pgmap v457: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 508 op/s; 18818/257541 objects misplaced (7.307%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:59:04.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:02.651429+0000 mon.a (mon.0) 1122 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:59:04.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:02.651461+0000 mon.a (mon.0) 1123 : cluster [INF] Cluster is now healthy 2024-02-12T16:59:04.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:02.671502+0000 mon.a (mon.0) 1124 : audit [INF] from='client.35201 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:59:04.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:02.671558+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e254: 8 total, 8 up, 6 in 2024-02-12T16:59:04.029 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:04.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:04.135 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:04.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:59:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:59:04.227 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:04.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:04.380 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:04.420 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:04.688 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:03.894704+0000 mgr.y (mgr.24481) 360 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:04.689 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:03.894704+0000 mgr.y (mgr.24481) 360 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:04.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:03.894704+0000 mgr.y (mgr.24481) 360 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:05.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:05.183 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:05.220 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:05.877 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:05.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:05.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:05.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:04.535583+0000 mgr.y (mgr.24481) 361 : cluster [DBG] pgmap v459: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 937 op/s; 17645/244707 objects misplaced (7.211%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:59:06.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:04.535583+0000 mgr.y (mgr.24481) 361 : cluster [DBG] pgmap v459: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 937 op/s; 17645/244707 objects misplaced (7.211%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:59:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:04.535583+0000 mgr.y (mgr.24481) 361 : cluster [DBG] pgmap v459: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 937 op/s; 17645/244707 objects misplaced (7.211%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:59:06.031 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:35215 8.7 8:e3f465d8:::benchmark_data_smithi033_105085_object35214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:07.612424+0000", 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 118.414882474, 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.067872991999999993, 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 35215 2024-02-12T16:59:06.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612424+0000", 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612424+0000", 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612423+0000", 2024-02-12T16:59:06.033 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8495000000000002e-05 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612441+0000", 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4300000000000003e-06 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612447+0000", 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5310000000000002e-06 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612451+0000", 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3476999999999998e-05 2024-02-12T16:59:06.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612484+0000", 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012479 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612609+0000", 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000164246 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,6", 2024-02-12T16:59:06.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.612773+0000", 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002019299 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.614793+0000", 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064945499999999995 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.615442+0000", 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064772947999999997 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.680215+0000", 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3925999999999999e-05 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.680259+0000", 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7438999999999998e-05 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.680297+0000", 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.067872991999999993 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:35225 8.7 8:e8e137a3:::benchmark_data_smithi033_105085_object35224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:07.615910+0000", 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 118.41139563500001, 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071371525000000005, 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 35225 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.615910+0000", 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.615910+0000", 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.615908+0000", 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6007999999999999e-05 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.615934+0000", 2024-02-12T16:59:06.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0616e-05 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.615945+0000", 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7750000000000004e-06 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.615952+0000", 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7752e-05 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.615999+0000", 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013061 2024-02-12T16:59:06.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.616130+0000", 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.06233868 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,6", 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.678469+0000", 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040462609999999998 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.682515+0000", 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1757000000000001e-05 2024-02-12T16:59:06.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.682547+0000", 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0046444149999999998 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.687191+0000", 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9344000000000001e-05 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.687241+0000", 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1406999999999998e-05 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.687282+0000", 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071371525000000005 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.35042.0:35224 8.f e189/176 8:f4084943:::benchmark_data_smithi033_105085_object35223:head v 189'2146, mlcod=189'2144)", 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:07.616442+0000", 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 118.41086375099999, 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070880979999999996, 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.043 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.616442+0000", 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.616442+0000", 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.044 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.616441+0000", 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3912000000000001e-05 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.616465+0000", 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062130999999999999 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.678596+0000", 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9850000000000003e-06 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.678602+0000", 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013787700000000001 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.679981+0000", 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.999e-05 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.680001+0000", 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0072856889999999997 2024-02-12T16:59:06.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.687286+0000", 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6915000000000001e-05 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:07.687323+0000", 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070880979999999996 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:37077 8.1 8:8f31c260:::benchmark_data_smithi033_105085_object37076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.047 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.650516+0000", 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.376790325, 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072565274999999999, 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 37077 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650516+0000", 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650516+0000", 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650514+0000", 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6004999999999997e-05 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650560+0000", 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1009999999999998e-06 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650569+0000", 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6820000000000002e-06 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650574+0000", 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3896999999999997e-05 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650618+0000", 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1044000000000005e-05 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650699+0000", 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9205999999999999e-05 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650798+0000", 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001516882 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652315+0000", 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022235399999999999 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652537+0000", 2024-02-12T16:59:06.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070450178000000002 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722988+0000", 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2148000000000003e-05 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723040+0000", 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1325999999999997e-05 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723081+0000", 2024-02-12T16:59:06.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072565274999999999 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:37078 8.a 8:5f8a1fc6:::benchmark_data_smithi033_105085_object37077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.650580+0000", 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.376726479, 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071634525000000004, 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 37078 2024-02-12T16:59:06.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650580+0000", 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650580+0000", 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650579+0000", 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2421e-05 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650611+0000", 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7309999999999996e-06 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650618+0000", 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0130000000000001e-06 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650623+0000", 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3136999999999998e-05 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650656+0000", 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000107385 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650764+0000", 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013291299999999999 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.650896+0000", 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001668442 2024-02-12T16:59:06.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652565+0000", 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003280317 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.655845+0000", 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066272864000000001 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722118+0000", 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4636999999999999e-05 2024-02-12T16:59:06.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722173+0000", 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1384000000000002e-05 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722214+0000", 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071634525000000004 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:37079 8.3 8:c92e79c4:::benchmark_data_smithi033_105085_object37078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.652444+0000", 2024-02-12T16:59:06.058 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.374861782, 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070643974999999998, 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 37079 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652444+0000", 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652444+0000", 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652443+0000", 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4762999999999998e-05 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652488+0000", 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2035000000000001e-05 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652500+0000", 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2887999999999999e-05 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652512+0000", 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7251999999999998e-05 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652590+0000", 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4964999999999998e-05 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652665+0000", 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8349000000000003e-05 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652763+0000", 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068116887000000001 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.720880+0000", 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020195270000000001 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722899+0000", 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110677 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723010+0000", 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3627000000000001e-05 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723054+0000", 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4511000000000002e-05 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723088+0000", 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070643974999999998 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:37082 8.3 8:c85354ce:::benchmark_data_smithi033_105085_object37081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.652527+0000", 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.374778663, 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072068463999999999, 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.064 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 37082 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652527+0000", 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652527+0000", 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652527+0000", 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0934999999999999e-05 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652547+0000", 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5857999999999999e-05 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652573+0000", 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3079999999999998e-06 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652579+0000", 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026315650000000002 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.655210+0000", 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8417000000000004e-05 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.655299+0000", 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116556 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:59:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.655415+0000", 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065565430999999993 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.720981+0000", 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002543619 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723524+0000", 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0009792589999999999 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.724503+0000", 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2296999999999998e-05 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.724556+0000", 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0049999999999998e-05 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.724596+0000", 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072068463999999999 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:37083 8.1 8:8ad81001:::benchmark_data_smithi033_105085_object37082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.652653+0000", 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.374652901, 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070555371000000006, 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 37083 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652653+0000", 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652653+0000", 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652652+0000", 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5402999999999998e-05 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652707+0000", 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.533e-06 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652716+0000", 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8690000000000001e-06 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652722+0000", 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0467999999999999e-05 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652763+0000", 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116753 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652880+0000", 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015455500000000001 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.653034+0000", 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017235950000000001 2024-02-12T16:59:06.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.654758+0000", 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068186073 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722944+0000", 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018359800000000001 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723127+0000", 2024-02-12T16:59:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1795999999999998e-05 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723169+0000", 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9328e-05 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723208+0000", 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070555371000000006 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:37084 8.7 8:eee1a8a1:::benchmark_data_smithi033_105085_object37083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.652731+0000", 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.37457546500001, 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068524189999999999, 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 37084 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652731+0000", 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652731+0000", 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652730+0000", 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9427e-05 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652769+0000", 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0100000000000001e-06 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652775+0000", 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1690000000000002e-06 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652779+0000", 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6773000000000001e-05 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652826+0000", 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011325800000000001 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652939+0000", 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117527 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,6", 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.653057+0000", 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0027418849999999999 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.655799+0000", 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032755499999999999 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656126+0000", 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065046178999999996 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.721172+0000", 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5241000000000003e-05 2024-02-12T16:59:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.721218+0000", 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7024999999999999e-05 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.721255+0000", 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068524189999999999 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.35042.0:37080 8.b e189/176 8:d2498496:::benchmark_data_smithi033_105085_object37079:head v 189'2235, mlcod=189'2233)", 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.652866+0000", 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.374439975, 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068091714999999997, 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652866+0000", 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652866+0000", 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652865+0000", 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111952 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652977+0000", 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7579999999999998e-06 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652986+0000", 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3990000000000003e-06 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.652991+0000", 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011194499999999999 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.654111+0000", 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7541e-05 2024-02-12T16:59:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.654128+0000", 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066755497999999996 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.720884+0000", 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4033000000000001e-05 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.720958+0000", 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068091714999999997 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:37087 8.1 8:8ce7618b:::benchmark_data_smithi033_105085_object37086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.655967+0000", 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.37133941899999, 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070511317000000004, 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 37087 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.655967+0000", 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.655967+0000", 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.655965+0000", 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8010000000000003e-05 2024-02-12T16:59:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656013+0000", 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0344e-05 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656023+0000", 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1279999999999996e-06 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656029+0000", 2024-02-12T16:59:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2511e-05 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656072+0000", 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114559 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656186+0000", 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064896130999999996 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,2", 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.721082+0000", 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038064399999999999 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.724889+0000", 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00090134299999999998 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.725790+0000", 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00059239299999999998 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.726383+0000", 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4147999999999997e-05 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.726437+0000", 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1273000000000003e-05 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.726478+0000", 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070511317000000004 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:37083 8.1 e189/176 ondisk, result = 0)", 2024-02-12T16:59:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.656046+0000", 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.37126047300001, 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066905656999999993, 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656046+0000", 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656046+0000", 2024-02-12T16:59:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656045+0000", 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9310000000000001e-06 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656049+0000", 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7289999999999997e-06 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656054+0000", 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1139999999999999e-06 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656059+0000", 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066857982999999996 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722917+0000", 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9780000000000005e-06 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722925+0000", 2024-02-12T16:59:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6670999999999999e-05 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722951+0000", 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066905656999999993 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.35042.0:37077 8.1 e189/176 ondisk, result = 0)", 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.656558+0000", 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.370747905, 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066532174999999999, 2024-02-12T16:59:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656558+0000", 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656558+0000", 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656557+0000", 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9010000000000002e-06 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.656562+0000", 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064230449999999994 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.720792+0000", 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5986000000000002e-05 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.720818+0000", 2024-02-12T16:59:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002156598 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722975+0000", 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1929999999999998e-06 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.722981+0000", 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000109034 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.723090+0000", 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066532174999999999 2024-02-12T16:59:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:37092 8.3 8:cb036e26:::benchmark_data_smithi033_105085_object37091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:08.657785+0000", 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 117.369521507, 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068235566999999997, 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 37092 2024-02-12T16:59:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.657785+0000", 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.657785+0000", 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.657783+0000", 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8976e-05 2024-02-12T16:59:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.657832+0000", 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063121372999999995 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.720953+0000", 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4136999999999999e-05 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.720977+0000", 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012895800000000001 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.721106+0000", 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015712899999999999 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.721263+0000", 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017453600000000001 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.721438+0000", 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031920350000000002 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.724630+0000", 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4473000000000001e-05 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.724655+0000", 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012638 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.725918+0000", 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1964999999999994e-05 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.725980+0000", 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9801000000000001e-05 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:08.726020+0000", 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068235566999999997 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82100 8.a 8:56f03a1b:::benchmark_data_smithi033_105085_object82099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e203)", 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:43.235672+0000", 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 82.791633564999998, 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074799434999999997, 2024-02-12T16:59:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82100 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235672+0000", 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235672+0000", 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235671+0000", 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.709e-05 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235698+0000", 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2264e-05 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235710+0000", 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9160000000000004e-06 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235717+0000", 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8940000000000002e-05 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235776+0000", 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015033000000000001 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.235926+0000", 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018243400000000001 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.236109+0000", 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00358626 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.239695+0000", 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001765668 2024-02-12T16:59:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.241461+0000", 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068934595000000001 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.310395+0000", 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8470000000000002e-05 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.310444+0000", 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8319000000000001e-05 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:43.310472+0000", 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074799434999999997 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82968 8.a 8:59b8eb47:::benchmark_data_smithi033_105085_object82967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:44.096756+0000", 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 81.930550084000004, 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.083323228999999999, 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82968 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096756+0000", 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096756+0000", 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096754+0000", 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6475e-05 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096780+0000", 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7899999999999994e-06 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096790+0000", 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0789999999999997e-06 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096796+0000", 2024-02-12T16:59:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000162249 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096958+0000", 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012408699999999999 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.097082+0000", 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015761900000000001 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.097240+0000", 2024-02-12T16:59:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011788710000000001 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.109029+0000", 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002903989 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.111933+0000", 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.068052185000000001 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.179985+0000", 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3319000000000002e-05 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.180038+0000", 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1069000000000003e-05 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.180079+0000", 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.083323228999999999 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82972 8.a 8:5d89183f:::benchmark_data_smithi033_105085_object82971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:44.096882+0000", 2024-02-12T16:59:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 81.930424197999997, 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.096931437999999995, 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82972 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096882+0000", 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096882+0000", 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096881+0000", 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2204e-05 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096903+0000", 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2370000000000003e-06 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096910+0000", 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0289999999999996e-06 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.096915+0000", 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029561029999999999 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.099871+0000", 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120623 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.099992+0000", 2024-02-12T16:59:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000156322 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.100148+0000", 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089154920000000006 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.109064+0000", 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028808940000000002 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.111945+0000", 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.081762007999999997 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.193707+0000", 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6660000000000003e-05 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.193763+0000", 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9833999999999998e-05 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.193813+0000", 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.096931437999999995 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:82979 8.a 8:5a4e1783:::benchmark_data_smithi033_105085_object82978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:44.102647+0000", 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 81.924658726000004, 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.091278082999999996, 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 82979 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.102647+0000", 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.102647+0000", 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.102645+0000", 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1271999999999999e-05 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.102666+0000", 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0797e-05 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.102677+0000", 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8940000000000004e-06 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.102684+0000", 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0760000000000006e-05 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.102765+0000", 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000142368 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.102907+0000", 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000170593 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.103078+0000", 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088722490000000005 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.111950+0000", 2024-02-12T16:59:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013570089999999999 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.113307+0000", 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.080559486999999999 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.193867+0000", 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0241000000000001e-05 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.193897+0000", 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8585000000000002e-05 2024-02-12T16:59:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.193925+0000", 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.091278082999999996 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:83380 8.a 8:5d8feba9:::benchmark_data_smithi033_105085_object83379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:44.499331+0000", 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 81.527974759000003, 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.091219944999999997, 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 83380 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499331+0000", 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499331+0000", 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499330+0000", 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2237999999999999e-05 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499362+0000", 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.377e-05 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499376+0000", 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3419999999999996e-06 2024-02-12T16:59:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499382+0000", 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1230999999999999e-05 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499423+0000", 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105485 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499529+0000", 2024-02-12T16:59:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013905699999999999 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499668+0000", 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039576359999999996 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.503626+0000", 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5290999999999998e-05 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.503661+0000", 2024-02-12T16:59:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.086792365999999996 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.590453+0000", 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5080000000000001e-05 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.590508+0000", 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2852000000000002e-05 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.590551+0000", 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.091219944999999997 2024-02-12T16:59:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:83381 8.a 8:513c3e20:::benchmark_data_smithi033_105085_object83380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:44.499687+0000", 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 81.527619127999998, 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.090953418999999994, 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 83381 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499687+0000", 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499687+0000", 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499686+0000", 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9139000000000001e-05 2024-02-12T16:59:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499705+0000", 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7880000000000001e-06 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499712+0000", 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6909999999999999e-06 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.499715+0000", 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019666940000000002 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.501682+0000", 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8306999999999993e-05 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.501770+0000", 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013118599999999999 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.501902+0000", 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018846729999999999 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.503786+0000", 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024634899999999998 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.504033+0000", 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.086560740999999997 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.590593+0000", 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6098999999999999e-05 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.590619+0000", 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.092e-05 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:44.590640+0000", 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.090953418999999994 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:06.137 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:59:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:06.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:06.240 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:06.278 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:06.413 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:06.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:06.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:06.620 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:06.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:06.771 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:06.805 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:06.910 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:06.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:07.018 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:07.103 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:07.141 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:07.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:07.245 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:07.302 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:07.392 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:06.175034+0000 mon.a (mon.0) 1126 : cluster [DBG] osdmap e255: 8 total, 8 up, 6 in 2024-02-12T16:59:07.392 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:06.369967+0000 osd.1 (osd.1) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:07.392 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:06.370019+0000 osd.1 (osd.1) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:07.393 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:06.535984+0000 mgr.y (mgr.24481) 362 : cluster [DBG] pgmap v461: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 661 op/s; 17645/244707 objects misplaced (7.211%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:59:07.393 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:06.175034+0000 mon.a (mon.0) 1126 : cluster [DBG] osdmap e255: 8 total, 8 up, 6 in 2024-02-12T16:59:07.393 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:06.369967+0000 osd.1 (osd.1) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:07.393 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:06.370019+0000 osd.1 (osd.1) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:07.393 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:06.535984+0000 mgr.y (mgr.24481) 362 : cluster [DBG] pgmap v461: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 661 op/s; 17645/244707 objects misplaced (7.211%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:59:07.396 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:07.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:07.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:06.175034+0000 mon.a (mon.0) 1126 : cluster [DBG] osdmap e255: 8 total, 8 up, 6 in 2024-02-12T16:59:07.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:06.369967+0000 osd.1 (osd.1) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:07.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:06.370019+0000 osd.1 (osd.1) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:07.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:06.535984+0000 mgr.y (mgr.24481) 362 : cluster [DBG] pgmap v461: 138 pgs: 4 active+remapped+backfilling, 134 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 661 op/s; 17645/244707 objects misplaced (7.211%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:59:07.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:07.573 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:07.598 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:07.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:07.706 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:59:07.750 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T16:59:07.617+0000 7fd96a9de700 -1 mon.a@0(leader).osd e256 definitely_dead 0 2024-02-12T16:59:07.758 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:07.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:59:07.782 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:59:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:07.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:59:07.889 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:07.980 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:07.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:08.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:08.185 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:08.209 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:08.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:08.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:08.229 INFO:teuthology.orchestra.run.smithi033.stderr:marked out osd.6. 2024-02-12T16:59:08.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:08.338 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:08.356 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:08.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:08.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:08.411 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:07.205604+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e256: 8 total, 8 up, 6 in 2024-02-12T16:59:08.411 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:07.618860+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.33:0/2172554614' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:59:08.412 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:07.619212+0000 mon.a (mon.0) 1128 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:59:08.412 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:07.619317+0000 mon.a (mon.0) 1129 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-12T16:59:08.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:07.205604+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e256: 8 total, 8 up, 6 in 2024-02-12T16:59:08.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:07.618860+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.33:0/2172554614' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:59:08.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:07.619212+0000 mon.a (mon.0) 1128 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:59:08.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:07.619317+0000 mon.a (mon.0) 1129 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-12T16:59:08.460 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:08.488 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:07.205604+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e256: 8 total, 8 up, 6 in 2024-02-12T16:59:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:07.618860+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.33:0/2172554614' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:59:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:07.619212+0000 mon.a (mon.0) 1128 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:59:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:07.619317+0000 mon.a (mon.0) 1129 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-12T16:59:09.428 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:08.219990+0000 mon.a (mon.0) 1130 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-12T16:59:09.428 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:08.220065+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e257: 8 total, 8 up, 5 in 2024-02-12T16:59:09.428 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:08.536703+0000 mgr.y (mgr.24481) 363 : cluster [DBG] pgmap v464: 138 pgs: 2 remapped+peering, 60 peering, 2 active+remapped+backfilling, 74 active+clean; 4.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 8616/235458 objects misplaced (3.659%); 4.3 MiB/s, 62 objects/s recovering 2024-02-12T16:59:09.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:08.219990+0000 mon.a (mon.0) 1130 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-12T16:59:09.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:08.220065+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e257: 8 total, 8 up, 5 in 2024-02-12T16:59:09.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:08.536703+0000 mgr.y (mgr.24481) 363 : cluster [DBG] pgmap v464: 138 pgs: 2 remapped+peering, 60 peering, 2 active+remapped+backfilling, 74 active+clean; 4.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 8616/235458 objects misplaced (3.659%); 4.3 MiB/s, 62 objects/s recovering 2024-02-12T16:59:09.438 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:09.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:08.219990+0000 mon.a (mon.0) 1130 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-12T16:59:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:08.220065+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e257: 8 total, 8 up, 5 in 2024-02-12T16:59:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:08.536703+0000 mgr.y (mgr.24481) 363 : cluster [DBG] pgmap v464: 138 pgs: 2 remapped+peering, 60 peering, 2 active+remapped+backfilling, 74 active+clean; 4.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 8616/235458 objects misplaced (3.659%); 4.3 MiB/s, 62 objects/s recovering 2024-02-12T16:59:09.542 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:59:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:09.596 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:59:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:09.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:59:09.675 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:59:09.678 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:09.696 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:59:09.700 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:09.718 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:59:09.732 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:09.738 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:59:09.763 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:59:09.818 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:59:09.818 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:59:09.818 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:59:09.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:59:09.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:59:09.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:59:09.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:59:09.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:59:09.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:59:09.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:59:09.820 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:59:10.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:09.217183+0000 mon.a (mon.0) 1132 : cluster [WRN] Health check failed: Reduced data availability: 13 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-02-12T16:59:10.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:09.224382+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e258: 8 total, 8 up, 5 in 2024-02-12T16:59:10.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:09.226334+0000 osd.5 (osd.5) 120 : cluster [DBG] 8.0 starting backfill to osd.0 from (0'0,0'0] MAX to 256'9143 2024-02-12T16:59:10.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:09.238428+0000 osd.5 (osd.5) 121 : cluster [INF] 8.3 continuing backfill to osd.2 from (202'5034,256'9130] 8:c27ca50b:::benchmark_data_smithi033_105085_object78236:head to 256'9130 2024-02-12T16:59:10.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:09.238460+0000 osd.5 (osd.5) 122 : cluster [DBG] 8.3 starting backfill to osd.4 from (0'0,0'0] MAX to 256'9130 2024-02-12T16:59:10.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:09.654480+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.33:0/322759273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:09.217183+0000 mon.a (mon.0) 1132 : cluster [WRN] Health check failed: Reduced data availability: 13 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:09.224382+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e258: 8 total, 8 up, 5 in 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:09.226334+0000 osd.5 (osd.5) 120 : cluster [DBG] 8.0 starting backfill to osd.0 from (0'0,0'0] MAX to 256'9143 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:09.238428+0000 osd.5 (osd.5) 121 : cluster [INF] 8.3 continuing backfill to osd.2 from (202'5034,256'9130] 8:c27ca50b:::benchmark_data_smithi033_105085_object78236:head to 256'9130 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:09.238460+0000 osd.5 (osd.5) 122 : cluster [DBG] 8.3 starting backfill to osd.4 from (0'0,0'0] MAX to 256'9130 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:09.654480+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.33:0/322759273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:09.217183+0000 mon.a (mon.0) 1132 : cluster [WRN] Health check failed: Reduced data availability: 13 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:09.224382+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e258: 8 total, 8 up, 5 in 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:09.226334+0000 osd.5 (osd.5) 120 : cluster [DBG] 8.0 starting backfill to osd.0 from (0'0,0'0] MAX to 256'9143 2024-02-12T16:59:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:09.238428+0000 osd.5 (osd.5) 121 : cluster [INF] 8.3 continuing backfill to osd.2 from (202'5034,256'9130] 8:c27ca50b:::benchmark_data_smithi033_105085_object78236:head to 256'9130 2024-02-12T16:59:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:09.238460+0000 osd.5 (osd.5) 122 : cluster [DBG] 8.3 starting backfill to osd.4 from (0'0,0'0] MAX to 256'9130 2024-02-12T16:59:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:09.654480+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.33:0/322759273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:10.589 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:10.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:10.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:59:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:59:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:10.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:59:10.876 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:10.880 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:59:10.923 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.231473+0000 osd.5 (osd.5) 123 : cluster [DBG] 8.9 starting backfill to osd.4 from (0'0,0'0] MAX to 256'9025 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.232867+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e259: 8 total, 8 up, 5 in 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.233010+0000 osd.2 (osd.2) 137 : cluster [DBG] 8.f starting backfill to osd.5 from (0'0,0'0] MAX to 256'9093 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.233077+0000 osd.2 (osd.2) 138 : cluster [DBG] 8.7 starting backfill to osd.7 from (0'0,0'0] MAX to 256'9086 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.238100+0000 osd.0 (osd.0) 118 : cluster [DBG] 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 256'9159 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.238994+0000 osd.7 (osd.7) 117 : cluster [DBG] 5.9 starting backfill to osd.2 from (0'0,0'0] MAX to 228'10 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.239853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 203'11 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.242952+0000 osd.7 (osd.7) 118 : cluster [DBG] 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 256'9125 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:10.537205+0000 mgr.y (mgr.24481) 364 : cluster [DBG] pgmap v467: 138 pgs: 2 remapped+peering, 68 peering, 2 active+remapped+backfilling, 66 active+clean; 4.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 255 B/s rd, 0 B/s wr, 1.03k op/s; 8335/232293 objects misplaced (3.588%); 3.6 MiB/s, 45 objects/s recovering 2024-02-12T16:59:11.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:11.177085+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e260: 8 total, 8 up, 5 in 2024-02-12T16:59:11.243 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.231473+0000 osd.5 (osd.5) 123 : cluster [DBG] 8.9 starting backfill to osd.4 from (0'0,0'0] MAX to 256'9025 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.232867+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e259: 8 total, 8 up, 5 in 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.233010+0000 osd.2 (osd.2) 137 : cluster [DBG] 8.f starting backfill to osd.5 from (0'0,0'0] MAX to 256'9093 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.233077+0000 osd.2 (osd.2) 138 : cluster [DBG] 8.7 starting backfill to osd.7 from (0'0,0'0] MAX to 256'9086 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.238100+0000 osd.0 (osd.0) 118 : cluster [DBG] 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 256'9159 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.238994+0000 osd.7 (osd.7) 117 : cluster [DBG] 5.9 starting backfill to osd.2 from (0'0,0'0] MAX to 228'10 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.239853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 203'11 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.242952+0000 osd.7 (osd.7) 118 : cluster [DBG] 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 256'9125 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:10.537205+0000 mgr.y (mgr.24481) 364 : cluster [DBG] pgmap v467: 138 pgs: 2 remapped+peering, 68 peering, 2 active+remapped+backfilling, 66 active+clean; 4.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 255 B/s rd, 0 B/s wr, 1.03k op/s; 8335/232293 objects misplaced (3.588%); 3.6 MiB/s, 45 objects/s recovering 2024-02-12T16:59:11.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:11.177085+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e260: 8 total, 8 up, 5 in 2024-02-12T16:59:11.245 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:59:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:10] "GET /metrics HTTP/1.1" 200 175938 "" "Prometheus/2.33.4" 2024-02-12T16:59:11.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.231473+0000 osd.5 (osd.5) 123 : cluster [DBG] 8.9 starting backfill to osd.4 from (0'0,0'0] MAX to 256'9025 2024-02-12T16:59:11.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.232867+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e259: 8 total, 8 up, 5 in 2024-02-12T16:59:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.233010+0000 osd.2 (osd.2) 137 : cluster [DBG] 8.f starting backfill to osd.5 from (0'0,0'0] MAX to 256'9093 2024-02-12T16:59:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.233077+0000 osd.2 (osd.2) 138 : cluster [DBG] 8.7 starting backfill to osd.7 from (0'0,0'0] MAX to 256'9086 2024-02-12T16:59:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.238100+0000 osd.0 (osd.0) 118 : cluster [DBG] 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 256'9159 2024-02-12T16:59:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.238994+0000 osd.7 (osd.7) 117 : cluster [DBG] 5.9 starting backfill to osd.2 from (0'0,0'0] MAX to 228'10 2024-02-12T16:59:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.239853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 203'11 2024-02-12T16:59:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.242952+0000 osd.7 (osd.7) 118 : cluster [DBG] 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 256'9125 2024-02-12T16:59:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:10.537205+0000 mgr.y (mgr.24481) 364 : cluster [DBG] pgmap v467: 138 pgs: 2 remapped+peering, 68 peering, 2 active+remapped+backfilling, 66 active+clean; 4.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 255 B/s rd, 0 B/s wr, 1.03k op/s; 8335/232293 objects misplaced (3.588%); 3.6 MiB/s, 45 objects/s recovering 2024-02-12T16:59:11.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:11.177085+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e260: 8 total, 8 up, 5 in 2024-02-12T16:59:11.683 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:11.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:11.787 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:11.829 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:12.767 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:12.784 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:59:12.784 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:59:12.785 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:12.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:12.892 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:12.936 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:12.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:59:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:59:13.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:11.800748+0000 osd.3 (osd.3) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:13.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:11.800807+0000 osd.3 (osd.3) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:13.466 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:12.190425+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e261: 8 total, 8 up, 5 in 2024-02-12T16:59:13.467 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:12.537801+0000 mgr.y (mgr.24481) 365 : cluster [DBG] pgmap v470: 138 pgs: 2 remapped+peering, 68 peering, 2 active+remapped+backfilling, 66 active+clean; 4.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 263 op/s; 8335/232293 objects misplaced (3.588%); 928 KiB/s, 13 objects/s recovering 2024-02-12T16:59:13.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:11.800748+0000 osd.3 (osd.3) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:13.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:11.800807+0000 osd.3 (osd.3) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:13.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:12.190425+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e261: 8 total, 8 up, 5 in 2024-02-12T16:59:13.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:12.537801+0000 mgr.y (mgr.24481) 365 : cluster [DBG] pgmap v470: 138 pgs: 2 remapped+peering, 68 peering, 2 active+remapped+backfilling, 66 active+clean; 4.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 263 op/s; 8335/232293 objects misplaced (3.588%); 928 KiB/s, 13 objects/s recovering 2024-02-12T16:59:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:11.800748+0000 osd.3 (osd.3) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:13.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:11.800807+0000 osd.3 (osd.3) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:13.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:12.190425+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e261: 8 total, 8 up, 5 in 2024-02-12T16:59:13.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:12.537801+0000 mgr.y (mgr.24481) 365 : cluster [DBG] pgmap v470: 138 pgs: 2 remapped+peering, 68 peering, 2 active+remapped+backfilling, 66 active+clean; 4.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 263 op/s; 8335/232293 objects misplaced (3.588%); 928 KiB/s, 13 objects/s recovering 2024-02-12T16:59:14.196 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7] out_osds: [3, 1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T16:59:14.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T16:59:14.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:59:14.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:59:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:59:14.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:13.909595+0000 mgr.y (mgr.24481) 366 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:14.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:13.909595+0000 mgr.y (mgr.24481) 366 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:14.405 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:14.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:14.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:13.909595+0000 mgr.y (mgr.24481) 366 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:14.512 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:14.551 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:14.760 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:14.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:14.782 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:59:14.867 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:15.084 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:15.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:15.189 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:15.269 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:15.309 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:15.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:15.408 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:15.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:15.454 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:14.538680+0000 mgr.y (mgr.24481) 367 : cluster [DBG] pgmap v471: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 963 B/s rd, 0 B/s wr, 444 op/s; 43392/225213 objects misplaced (19.267%); 20 MiB/s, 343 objects/s recovering 2024-02-12T16:59:15.505 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:14.538680+0000 mgr.y (mgr.24481) 367 : cluster [DBG] pgmap v471: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 963 B/s rd, 0 B/s wr, 444 op/s; 43392/225213 objects misplaced (19.267%); 20 MiB/s, 343 objects/s recovering 2024-02-12T16:59:15.505 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:14.538680+0000 mgr.y (mgr.24481) 367 : cluster [DBG] pgmap v471: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 963 B/s rd, 0 B/s wr, 444 op/s; 43392/225213 objects misplaced (19.267%); 20 MiB/s, 343 objects/s recovering 2024-02-12T16:59:15.514 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:15.684 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:15.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:15.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:15.903 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:15.947 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:15.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:16.057 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:16.101 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:16.333 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T16:59:16.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:15.220877+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:59:16.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:15.220902+0000 mon.a (mon.0) 1138 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 20 pgs peering) 2024-02-12T16:59:16.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:15.769788+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.33:0/453294355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:59:16.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:15.770150+0000 mon.a (mon.0) 1139 : audit [INF] from='client.25192 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:59:16.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:15.220877+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:59:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:15.220902+0000 mon.a (mon.0) 1138 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 20 pgs peering) 2024-02-12T16:59:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:15.769788+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.33:0/453294355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:59:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:15.770150+0000 mon.a (mon.0) 1139 : audit [INF] from='client.25192 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:59:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:15.220877+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:59:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:15.220902+0000 mon.a (mon.0) 1138 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 20 pgs peering) 2024-02-12T16:59:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:15.769788+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.33:0/453294355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:59:16.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:15.770150+0000 mon.a (mon.0) 1139 : audit [INF] from='client.25192 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:59:17.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:16.269727+0000 mon.a (mon.0) 1140 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:17.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:16.326892+0000 mon.a (mon.0) 1141 : audit [INF] from='client.25192 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:59:17.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:16.326959+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e262: 8 total, 8 up, 5 in 2024-02-12T16:59:17.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:16.539097+0000 mgr.y (mgr.24481) 368 : cluster [DBG] pgmap v473: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 394 op/s; 43392/225213 objects misplaced (19.267%); 18 MiB/s, 304 objects/s recovering 2024-02-12T16:59:17.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:16.269727+0000 mon.a (mon.0) 1140 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:17.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:16.326892+0000 mon.a (mon.0) 1141 : audit [INF] from='client.25192 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:59:17.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:16.326959+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e262: 8 total, 8 up, 5 in 2024-02-12T16:59:17.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:16.539097+0000 mgr.y (mgr.24481) 368 : cluster [DBG] pgmap v473: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 394 op/s; 43392/225213 objects misplaced (19.267%); 18 MiB/s, 304 objects/s recovering 2024-02-12T16:59:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:16.269727+0000 mon.a (mon.0) 1140 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:16.326892+0000 mon.a (mon.0) 1141 : audit [INF] from='client.25192 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:59:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:16.326959+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e262: 8 total, 8 up, 5 in 2024-02-12T16:59:17.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:16.539097+0000 mgr.y (mgr.24481) 368 : cluster [DBG] pgmap v473: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 394 op/s; 43392/225213 objects misplaced (19.267%); 18 MiB/s, 304 objects/s recovering 2024-02-12T16:59:18.775 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:18.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:17.488394+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:59:18.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:17.488394+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:59:18.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:17.488394+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:59:19.770 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:18.539710+0000 mgr.y (mgr.24481) 369 : cluster [DBG] pgmap v474: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 710 op/s; 40272/216627 objects misplaced (18.590%); 26 MiB/s, 422 objects/s recovering 2024-02-12T16:59:19.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:18.539710+0000 mgr.y (mgr.24481) 369 : cluster [DBG] pgmap v474: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 710 op/s; 40272/216627 objects misplaced (18.590%); 26 MiB/s, 422 objects/s recovering 2024-02-12T16:59:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:19.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:19.881 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:19.925 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:19.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:18.539710+0000 mgr.y (mgr.24481) 369 : cluster [DBG] pgmap v474: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 710 op/s; 40272/216627 objects misplaced (18.590%); 26 MiB/s, 422 objects/s recovering 2024-02-12T16:59:21.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:59:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:20] "GET /metrics HTTP/1.1" 200 175947 "" "Prometheus/2.33.4" 2024-02-12T16:59:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:20.540267+0000 mgr.y (mgr.24481) 370 : cluster [DBG] pgmap v475: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 746 op/s; 38553/213630 objects misplaced (18.047%); 32 MiB/s, 532 objects/s recovering 2024-02-12T16:59:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:21.183185+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e263: 8 total, 8 up, 5 in 2024-02-12T16:59:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:20.540267+0000 mgr.y (mgr.24481) 370 : cluster [DBG] pgmap v475: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 746 op/s; 38553/213630 objects misplaced (18.047%); 32 MiB/s, 532 objects/s recovering 2024-02-12T16:59:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:21.183185+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e263: 8 total, 8 up, 5 in 2024-02-12T16:59:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:20.540267+0000 mgr.y (mgr.24481) 370 : cluster [DBG] pgmap v475: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 746 op/s; 38553/213630 objects misplaced (18.047%); 32 MiB/s, 532 objects/s recovering 2024-02-12T16:59:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:21.183185+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e263: 8 total, 8 up, 5 in 2024-02-12T16:59:22.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:59:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:59:23.933 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:22.540681+0000 mgr.y (mgr.24481) 371 : cluster [DBG] pgmap v477: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 483 op/s; 38553/213630 objects misplaced (18.047%); 20 MiB/s, 327 objects/s recovering 2024-02-12T16:59:23.933 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:59:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:59:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:22.540681+0000 mgr.y (mgr.24481) 371 : cluster [DBG] pgmap v477: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 483 op/s; 38553/213630 objects misplaced (18.047%); 20 MiB/s, 327 objects/s recovering 2024-02-12T16:59:24.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:22.540681+0000 mgr.y (mgr.24481) 371 : cluster [DBG] pgmap v477: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 483 op/s; 38553/213630 objects misplaced (18.047%); 20 MiB/s, 327 objects/s recovering 2024-02-12T16:59:24.651 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:23.924534+0000 mgr.y (mgr.24481) 372 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:24.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:23.924534+0000 mgr.y (mgr.24481) 372 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:25.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:23.924534+0000 mgr.y (mgr.24481) 372 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:25.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:24.541396+0000 mgr.y (mgr.24481) 373 : cluster [DBG] pgmap v478: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 934 op/s; 33831/202281 objects misplaced (16.725%); 41 MiB/s, 652 objects/s recovering 2024-02-12T16:59:25.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:25.454655+0000 osd.5 (osd.5) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:25.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:25.454705+0000 osd.5 (osd.5) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:25.993 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:25.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:24.541396+0000 mgr.y (mgr.24481) 373 : cluster [DBG] pgmap v478: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 934 op/s; 33831/202281 objects misplaced (16.725%); 41 MiB/s, 652 objects/s recovering 2024-02-12T16:59:25.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:25.454655+0000 osd.5 (osd.5) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:25.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:25.454705+0000 osd.5 (osd.5) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:25.998 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:24.541396+0000 mgr.y (mgr.24481) 373 : cluster [DBG] pgmap v478: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 934 op/s; 33831/202281 objects misplaced (16.725%); 41 MiB/s, 652 objects/s recovering 2024-02-12T16:59:25.998 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:25.454655+0000 osd.5 (osd.5) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:25.998 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:25.454705+0000 osd.5 (osd.5) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:25.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:26.099 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:26.158 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:26.215 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:26.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:26.328 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:59:26.382 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:26.430 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:59:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:26.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:59:26.535 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:26.579 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:25.664420+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e264: 8 total, 8 up, 5 in 2024-02-12T16:59:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:26.181582+0000 mon.a (mon.0) 1146 : cluster [DBG] osdmap e265: 8 total, 8 up, 5 in 2024-02-12T16:59:27.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:25.664420+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e264: 8 total, 8 up, 5 in 2024-02-12T16:59:27.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:26.181582+0000 mon.a (mon.0) 1146 : cluster [DBG] osdmap e265: 8 total, 8 up, 5 in 2024-02-12T16:59:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:25.664420+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e264: 8 total, 8 up, 5 in 2024-02-12T16:59:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:26.181582+0000 mon.a (mon.0) 1146 : cluster [DBG] osdmap e265: 8 total, 8 up, 5 in 2024-02-12T16:59:27.535 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:27.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:27.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:27.689 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:27.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:27.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:59:27.910 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:27.945 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:59:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:27.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:59:27.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:26.541808+0000 mgr.y (mgr.24481) 374 : cluster [DBG] pgmap v481: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 631 op/s; 33831/202281 objects misplaced (16.725%); 28 MiB/s, 453 objects/s recovering 2024-02-12T16:59:27.952 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:27.198133+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e266: 8 total, 8 up, 5 in 2024-02-12T16:59:28.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:26.541808+0000 mgr.y (mgr.24481) 374 : cluster [DBG] pgmap v481: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 631 op/s; 33831/202281 objects misplaced (16.725%); 28 MiB/s, 453 objects/s recovering 2024-02-12T16:59:28.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:27.198133+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e266: 8 total, 8 up, 5 in 2024-02-12T16:59:28.009 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:26.541808+0000 mgr.y (mgr.24481) 374 : cluster [DBG] pgmap v481: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 631 op/s; 33831/202281 objects misplaced (16.725%); 28 MiB/s, 453 objects/s recovering 2024-02-12T16:59:28.009 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:27.198133+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e266: 8 total, 8 up, 5 in 2024-02-12T16:59:28.051 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:28.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:59:28.124 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:28.167 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:28.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:28.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:28.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:28.353 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:28.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:28.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:28.538 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:28.976 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:59:28.976 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:59:29.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:27.808020+0000 mon.a (mon.0) 1148 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:59:29.264 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:59:29.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:27.808020+0000 mon.a (mon.0) 1148 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:59:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:27.808020+0000 mon.a (mon.0) 1148 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:59:29.295 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:59:29.321 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:59:29.347 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:59:29.388 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:59:29.421 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:59:29.467 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:59:29.508 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:59:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:59:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:59:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:59:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:59:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:59:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:59:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:59:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:59:29.846 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:28.543058+0000 mgr.y (mgr.24481) 375 : cluster [DBG] pgmap v483: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.0 GiB data, 12 GiB used, 436 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 30600/194955 objects misplaced (15.696%); 47 MiB/s, 758 objects/s recovering 2024-02-12T16:59:29.846 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:29.224193+0000 mon.a (mon.0) 1149 : audit [DBG] from='client.? v1:172.21.15.33:0/2382133675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:29.857 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T16:59:30.210 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:30.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:30.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:28.543058+0000 mgr.y (mgr.24481) 375 : cluster [DBG] pgmap v483: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.0 GiB data, 12 GiB used, 436 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 30600/194955 objects misplaced (15.696%); 47 MiB/s, 758 objects/s recovering 2024-02-12T16:59:30.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:29.224193+0000 mon.a (mon.0) 1149 : audit [DBG] from='client.? v1:172.21.15.33:0/2382133675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:30.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:29.809709+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.33:0/1944824756' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:59:30.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:29.810079+0000 mon.a (mon.0) 1150 : audit [INF] from='client.35228 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:59:30.215 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:29.809709+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.33:0/1944824756' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:59:30.215 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:29.810079+0000 mon.a (mon.0) 1150 : audit [INF] from='client.35228 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:59:30.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:28.543058+0000 mgr.y (mgr.24481) 375 : cluster [DBG] pgmap v483: 138 pgs: 1 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 129 active+clean; 4.0 GiB data, 12 GiB used, 436 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 30600/194955 objects misplaced (15.696%); 47 MiB/s, 758 objects/s recovering 2024-02-12T16:59:30.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:29.224193+0000 mon.a (mon.0) 1149 : audit [DBG] from='client.? v1:172.21.15.33:0/2382133675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:30.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:29.809709+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.33:0/1944824756' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:59:30.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:29.810079+0000 mon.a (mon.0) 1150 : audit [INF] from='client.35228 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:59:30.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:30.872 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:59:30.919 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:30.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:30.930 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:31.026 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:31.081 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:31.129 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:29.837011+0000 mon.a (mon.0) 1151 : audit [INF] from='client.35228 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:59:31.129 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:29.837063+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e267: 8 total, 8 up, 5 in 2024-02-12T16:59:31.129 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:59:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:30] "GET /metrics HTTP/1.1" 200 175947 "" "Prometheus/2.33.4" 2024-02-12T16:59:31.130 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:31.130 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:29.837011+0000 mon.a (mon.0) 1151 : audit [INF] from='client.35228 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:59:31.130 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:29.837063+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e267: 8 total, 8 up, 5 in 2024-02-12T16:59:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:31.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:31.239 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:31.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:29.837011+0000 mon.a (mon.0) 1151 : audit [INF] from='client.35228 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:59:31.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:29.837063+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e267: 8 total, 8 up, 5 in 2024-02-12T16:59:31.273 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:31.482 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:31.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:31.588 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:31.695 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:31.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:31.799 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:31.903 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:31.914 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:30.543534+0000 mgr.y (mgr.24481) 376 : cluster [DBG] pgmap v485: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 677 op/s; 29586/192369 objects misplaced (15.380%); 34 MiB/s, 543 objects/s recovering 2024-02-12T16:59:31.914 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:31.171935+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:31.914 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:31.171988+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:31.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:32.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:30.543534+0000 mgr.y (mgr.24481) 376 : cluster [DBG] pgmap v485: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 677 op/s; 29586/192369 objects misplaced (15.380%); 34 MiB/s, 543 objects/s recovering 2024-02-12T16:59:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:31.171935+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:31.171988+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:30.543534+0000 mgr.y (mgr.24481) 376 : cluster [DBG] pgmap v485: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 677 op/s; 29586/192369 objects misplaced (15.380%); 34 MiB/s, 543 objects/s recovering 2024-02-12T16:59:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:31.171935+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:31.171988+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:32.057 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:32.090 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:32.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:32.099 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T16:59:32.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:32.248 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:32.619 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:32.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:32.726 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:32.766 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:32.827 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:32.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:32.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:31.892466+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e268: 8 total, 8 up, 5 in 2024-02-12T16:59:32.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:31.970657+0000 osd.7 (osd.7) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:32.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:31.970712+0000 osd.7 (osd.7) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:32.907 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:31.892466+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e268: 8 total, 8 up, 5 in 2024-02-12T16:59:32.907 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:31.970657+0000 osd.7 (osd.7) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:32.907 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:31.970712+0000 osd.7 (osd.7) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:32.911 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:59:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:59:32.911 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:31.892466+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e268: 8 total, 8 up, 5 in 2024-02-12T16:59:32.911 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:31.970657+0000 osd.7 (osd.7) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:32.911 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:31.970712+0000 osd.7 (osd.7) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:32.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:32.999 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:33.036 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:33.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:33.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:33.250 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:33.255 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:33.274 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:33.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:33.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:33.447 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:33.473 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:33.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:33.501 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:59:33.576 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:33.678 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:33.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:33.789 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:33.843 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:34.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:32.543955+0000 mgr.y (mgr.24481) 377 : cluster [DBG] pgmap v487: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 551 op/s; 29586/192369 objects misplaced (15.380%); 28 MiB/s, 441 objects/s recovering 2024-02-12T16:59:34.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:32.907073+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e269: 8 total, 8 up, 5 in 2024-02-12T16:59:34.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:59:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:59:34.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:32.543955+0000 mgr.y (mgr.24481) 377 : cluster [DBG] pgmap v487: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 551 op/s; 29586/192369 objects misplaced (15.380%); 28 MiB/s, 441 objects/s recovering 2024-02-12T16:59:34.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:32.907073+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e269: 8 total, 8 up, 5 in 2024-02-12T16:59:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:32.543955+0000 mgr.y (mgr.24481) 377 : cluster [DBG] pgmap v487: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 551 op/s; 29586/192369 objects misplaced (15.380%); 28 MiB/s, 441 objects/s recovering 2024-02-12T16:59:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:32.907073+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e269: 8 total, 8 up, 5 in 2024-02-12T16:59:34.806 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:34.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:34.913 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:34.976 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:35.110 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:35.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:35.116 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:33.937699+0000 mgr.y (mgr.24481) 378 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:35.116 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:33.937699+0000 mgr.y (mgr.24481) 378 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:35.216 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:35.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:33.937699+0000 mgr.y (mgr.24481) 378 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:35.254 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:35.721 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:35.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:35.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:59:35.895 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:35.940 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:59:35.945 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T16:59:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:35.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:59:35.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:34.544826+0000 mgr.y (mgr.24481) 379 : cluster [DBG] pgmap v489: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 26457/180645 objects misplaced (14.646%); 23 MiB/s, 368 objects/s recovering 2024-02-12T16:59:35.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:35.028279+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.33:0/3103532503' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:35.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:35.028669+0000 mon.a (mon.0) 1155 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:35.973 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:34.544826+0000 mgr.y (mgr.24481) 379 : cluster [DBG] pgmap v489: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 26457/180645 objects misplaced (14.646%); 23 MiB/s, 368 objects/s recovering 2024-02-12T16:59:35.973 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:35.028279+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.33:0/3103532503' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:35.973 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:35.028669+0000 mon.a (mon.0) 1155 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:36.047 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:36.093 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:34.544826+0000 mgr.y (mgr.24481) 379 : cluster [DBG] pgmap v489: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 26457/180645 objects misplaced (14.646%); 23 MiB/s, 368 objects/s recovering 2024-02-12T16:59:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:35.028279+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.33:0/3103532503' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:35.028669+0000 mon.a (mon.0) 1155 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:59:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:36.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:36.402 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:36.450 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:36.935 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:35.924973+0000 mon.a (mon.0) 1156 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:36.935 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:35.937401+0000 mon.a (mon.0) 1157 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:59:36.935 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:35.937468+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e270: 8 total, 8 up, 5 in 2024-02-12T16:59:36.935 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:36.186494+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e271: 8 total, 8 up, 5 in 2024-02-12T16:59:36.936 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:35.924973+0000 mon.a (mon.0) 1156 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:36.936 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:35.937401+0000 mon.a (mon.0) 1157 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:59:36.936 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:35.937468+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e270: 8 total, 8 up, 5 in 2024-02-12T16:59:36.936 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:36.186494+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e271: 8 total, 8 up, 5 in 2024-02-12T16:59:37.012 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:37.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:37.128 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:37.180 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:37.209 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:37.221 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:35.924973+0000 mon.a (mon.0) 1156 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:37.222 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:35.937401+0000 mon.a (mon.0) 1157 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:59:37.222 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:35.937468+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e270: 8 total, 8 up, 5 in 2024-02-12T16:59:37.222 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:36.186494+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e271: 8 total, 8 up, 5 in 2024-02-12T16:59:37.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:37.323 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:37.369 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:37.784 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:59:37.784 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:85064 8.a 8:5639ee32:::benchmark_data_smithi033_105085_object85063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e207)", 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:46.173132+0000", 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 110.411311817, 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0964025579999999, 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 85064 2024-02-12T16:59:37.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.173132+0000", 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.173132+0000", 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.173129+0000", 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7231000000000001e-05 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.173157+0000", 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0399e-05 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.173167+0000", 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0190000000000003e-06 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.173173+0000", 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002949424 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176122+0000", 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2811e-05 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176135+0000", 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98742135600000003 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.163557+0000", 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0857e-05 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.163567+0000", 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1045758430000001 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.268143+0000", 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001391059 2024-02-12T16:59:37.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.269534+0000", 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0964025579999999 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:85065 8.a 8:56b38310:::benchmark_data_smithi033_105085_object85064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e207)", 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:46.176076+0000", 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 110.40836759299999, 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0938254110000001, 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T16:59:37.790 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 85065 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176076+0000", 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176076+0000", 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:59:37.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176073+0000", 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9279000000000001e-05 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176113+0000", 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3995999999999999e-05 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176127+0000", 2024-02-12T16:59:37.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7600000000000002e-06 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176134+0000", 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3229999999999997e-05 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176188+0000", 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2561000000000001e-05 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176200+0000", 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.987393348 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.163594+0000", 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6310000000000002e-06 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.163598+0000", 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1062310870000001 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.269829+0000", 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2028999999999998e-05 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.269901+0000", 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0938254110000001 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:85066 8.a 8:57eb6a82:::benchmark_data_smithi033_105085_object85065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e207)", 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:46.176144+0000", 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 110.40829926799999, 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0938639320000001, 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.795 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 85066 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176144+0000", 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176144+0000", 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176143+0000", 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8047000000000001e-05 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176171+0000", 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2456000000000001e-05 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176184+0000", 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0549999999999999e-06 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176189+0000", 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7558000000000002e-05 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176226+0000", 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9369999999999999e-06 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.176236+0000", 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98784126900000002 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.164078+0000", 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4600000000000003e-06 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.164086+0000", 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1058394309999999 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.269926+0000", 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2688999999999994e-05 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.270008+0000", 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0938639320000001 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:85087 8.f 8:fdca37a4:::benchmark_data_smithi033_105085_object85086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:46.199909+0000", 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 110.384534692, 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1271647699999998, 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.800 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 85087 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.199909+0000", 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.199909+0000", 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999967 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.199905+0000", 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2447999999999998e-05 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.199938+0000", 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7776000000000001e-05 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.199956+0000", 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0019999999999995e-06 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.199965+0000", 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9543999999999999e-05 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.200014+0000", 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0438999999999999e-05 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.200025+0000", 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96750856500000004 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.167533+0000", 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8470000000000008e-06 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.167543+0000", 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.116876148 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.284419+0000", 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000139581 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.804 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.284559+0000", 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00046330500000000001 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.285022+0000", 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012296858000000001 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.297319+0000", 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002644141 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.299963+0000", 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020682755000000001 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.320646+0000", 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061621590000000004 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.326808+0000", 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000139973 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.326948+0000", 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125714 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.327074+0000", 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1271647699999998 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:85092 8.f 8:f93f1282:::benchmark_data_smithi033_105085_object85091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:46.240750+0000", 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 110.343693696, 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.086537506, 2024-02-12T16:59:37.807 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 85092 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.240750+0000", 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.240750+0000", 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.240747+0000", 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8775000000000001e-05 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.240776+0000", 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.647e-05 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.240792+0000", 2024-02-12T16:59:37.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8820000000000003e-06 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.240799+0000", 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5718999999999999e-05 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.240865+0000", 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7982000000000002e-05 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:46.240883+0000", 2024-02-12T16:59:37.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.92903756900000001 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.169920+0000", 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.662e-06 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.169930+0000", 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1225726220000001 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.292502+0000", 2024-02-12T16:59:37.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016017599999999999 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.292663+0000", 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022976800000000001 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.292892+0000", 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044345069999999999 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.297327+0000", 2024-02-12T16:59:37.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026881829999999998 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.300015+0000", 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020660584999999999 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.320676+0000", 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0064575459999999998 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.327133+0000", 2024-02-12T16:59:37.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3921999999999997e-05 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.327177+0000", 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110179 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.327287+0000", 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.086537506 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.814 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:1275 5.d 5.a204812d (undecoded) ondisk+write+known_if_redirected e203)", 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:47.337562+0000", 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 109.246881398, 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.84305708599999996, 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 1275 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.337562+0000", 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.337562+0000", 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999962 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.337558+0000", 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2091e-05 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.337570+0000", 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3244e-05 2024-02-12T16:59:37.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.337594+0000", 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.062e-05 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.337604+0000", 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101217 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.337706+0000", 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4202999999999999e-05 2024-02-12T16:59:37.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.337720+0000", 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.84287574399999998 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.180595+0000", 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3771e-05 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.180619+0000", 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.84305708599999996 2024-02-12T16:59:37.818 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:1287 5.d 5.a204812d (undecoded) ondisk+write+known_if_redirected e208)", 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:47.338063+0000", 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 109.246380355, 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.84261815200000001, 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 1287 2024-02-12T16:59:37.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.338063+0000", 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.338063+0000", 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.338062+0000", 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3611000000000001e-05 2024-02-12T16:59:37.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.338076+0000", 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3550000000000001e-06 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.338082+0000", 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.746e-06 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.338087+0000", 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3003000000000002e-05 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.338120+0000", 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5110000000000003e-06 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:47.338126+0000", 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.84254599100000005 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.180672+0000", 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6439999999999995e-06 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:48.180681+0000", 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.84261815200000001 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:1275 5.d 5:b4812045:::notify.4:head [watch ping cookie 94508044030464] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e209)", 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:49.166150+0000", 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 107.418293385, 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0396216119999999, 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.823 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 1275 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166150+0000", 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166150+0000", 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166148+0000", 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5799999999999999e-06 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166154+0000", 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1300000000000001e-06 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166162+0000", 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7600000000000002e-06 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166167+0000", 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000465036 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166632+0000", 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.968e-06 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166639+0000", 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0389667010000001 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.205605+0000", 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118421 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.205724+0000", 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7876e-05 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.205772+0000", 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0396216119999999 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:1287 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94508044030464 gen 1] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e209)", 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:49.166170+0000", 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 107.418273321, 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.070504466, 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 1287 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.828 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166170+0000", 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166170+0000", 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166170+0000", 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.891e-06 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166172+0000", 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.963e-06 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166176+0000", 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8600000000000003e-06 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166180+0000", 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00048226299999999997 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166662+0000", 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6669999999999996e-06 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166669+0000", 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.039326727 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.205996+0000", 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0630000000000006e-05 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.206077+0000", 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030597889999999999 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.236675+0000", 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.070504466 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:1288 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94508044030464 gen 2] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:57:49.166824+0000", 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 107.417619294, 2024-02-12T16:59:37.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0699470790000001, 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 1288 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166824+0000", 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166824+0000", 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166823+0000", 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3470000000000004e-06 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166829+0000", 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1109999999999997e-06 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166838+0000", 2024-02-12T16:59:37.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4700000000000004e-06 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166843+0000", 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8444e-05 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166881+0000", 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3770000000000004e-06 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:49.166886+0000", 2024-02-12T16:59:37.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0395154090000001 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.206402+0000", 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2353000000000004e-05 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.206464+0000", 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030307094999999999 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:57:50.236771+0000", 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0699470790000001 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127146 8.f 8:f9f7d3f8:::benchmark_data_smithi033_105085_object15488:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.649280+0000", 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.935163342999999, 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99453172400000001, 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127146 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.649280+0000", 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.649280+0000", 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999871 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.649267+0000", 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8474e-05 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.649286+0000", 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6047000000000001e-05 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.649302+0000", 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.596e-06 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.649308+0000", 2024-02-12T16:59:37.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2561999999999997e-05 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.649351+0000", 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2539e-05 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.649363+0000", 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98950283699999997 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.638866+0000", 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013291799999999999 2024-02-12T16:59:37.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.638999+0000", 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015307799999999999 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.639152+0000", 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002742179 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.641894+0000", 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7178e-05 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.641952+0000", 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00178527 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.643737+0000", 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2991999999999999e-05 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.643770+0000", 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2012999999999999e-05 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.643812+0000", 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99453172400000001 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127147 8.f 8:f34b2806:::benchmark_data_smithi033_105085_object15489:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.653013+0000", 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.931430763000002, 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.996295033, 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.843 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127147 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.653013+0000", 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.653013+0000", 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.653011+0000", 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9299999999999997e-06 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.653018+0000", 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.465e-06 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.653027+0000", 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2849999999999996e-06 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.653031+0000", 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0839e-05 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.653072+0000", 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5979999999999997e-06 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.653081+0000", 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98889592999999998 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.641977+0000", 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102126 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.642079+0000", 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000119175 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.642198+0000", 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069105989999999999 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649109+0000", 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6005000000000003e-05 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649165+0000", 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1892999999999999e-05 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649217+0000", 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8903999999999998e-05 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649265+0000", 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2360000000000001e-05 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649308+0000", 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.996295033 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127149 8.f 8:f057e531:::benchmark_data_smithi033_105085_object15491:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.654696+0000", 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.929747378000002, 2024-02-12T16:59:37.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.000775328, 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127149 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.654696+0000", 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.654696+0000", 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.654695+0000", 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7440000000000004e-06 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.654701+0000", 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.096e-05 2024-02-12T16:59:37.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.654711+0000", 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1410000000000002e-06 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.654718+0000", 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6030999999999999e-05 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.654744+0000", 2024-02-12T16:59:37.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8860000000000003e-06 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.654749+0000", 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98909953699999997 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.643848+0000", 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011370399999999999 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.643962+0000", 2024-02-12T16:59:37.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000142211 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.644104+0000", 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052159729999999996 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649320+0000", 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029902890000000001 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.652310+0000", 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003070503 2024-02-12T16:59:37.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.655381+0000", 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8176000000000001e-05 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.655429+0000", 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2567000000000001e-05 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.655471+0000", 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.000775328 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.855 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127182 8.f 8:f323a601:::benchmark_data_smithi033_105085_object15524:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.693797+0000", 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.890646113999999, 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96760486400000001, 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127182 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.693797+0000", 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.693797+0000", 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.693796+0000", 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5989999999999998e-06 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.693802+0000", 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2267e-05 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.693815+0000", 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.212e-06 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.693821+0000", 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4613000000000001e-05 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.693865+0000", 2024-02-12T16:59:37.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9870000000000005e-06 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.693874+0000", 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.95191449100000003 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.645789+0000", 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104176 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.645893+0000", 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000124299 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.646017+0000", 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0033118190000000001 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649329+0000", 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0030126520000000002 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.652342+0000", 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089589400000000003 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.661301+0000", 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7818000000000002e-05 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.661349+0000", 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3609000000000001e-05 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.661402+0000", 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96760486400000001 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.861 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127210 8.f 8:fdefbf56:::benchmark_data_smithi033_105085_object15552:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.729636+0000", 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.854807893999997, 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.93192830699999996, 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127210 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.862 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.729636+0000", 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.729636+0000", 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.729634+0000", 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9650000000000002e-06 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.729641+0000", 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4550000000000001e-06 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.729650+0000", 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.656e-06 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.729655+0000", 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8183999999999999e-05 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.729693+0000", 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7560000000000002e-06 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.729700+0000", 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.91787303600000003 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.647573+0000", 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011680699999999999 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.647689+0000", 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014455499999999999 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.647834+0000", 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044402749999999996 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.652274+0000", 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104503 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.652379+0000", 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0090868689999999992 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.661466+0000", 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1227000000000003e-05 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.661517+0000", 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7094e-05 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.661564+0000", 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.93192830699999996 2024-02-12T16:59:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127227 8.f 8:f364a8d4:::benchmark_data_smithi033_105085_object15569:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.748511+0000", 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.835932755000002, 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.92295632699999997, 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127227 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.748511+0000", 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.748511+0000", 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.748509+0000", 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.216e-06 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.748516+0000", 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1230000000000006e-06 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.748525+0000", 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.865e-06 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.748530+0000", 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3293000000000001e-05 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.748563+0000", 2024-02-12T16:59:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.037e-06 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.748569+0000", 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90080697700000001 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649376+0000", 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102365 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649478+0000", 2024-02-12T16:59:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6550000000000004e-05 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.649555+0000", 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0027912089999999998 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.652346+0000", 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011938516 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.664285+0000", 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0071141160000000002 2024-02-12T16:59:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671399+0000", 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6976e-05 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671436+0000", 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1300999999999997e-05 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671467+0000", 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.92295632699999997 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127229 8.f 8:f1c012ad:::benchmark_data_smithi033_105085_object15571:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.750677+0000", 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.833766763, 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.92090182799999998, 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127229 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.750677+0000", 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.750677+0000", 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.750676+0000", 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1760000000000001e-06 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.750681+0000", 2024-02-12T16:59:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5379999999999997e-06 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.750688+0000", 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9979999999999997e-06 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.750692+0000", 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3742000000000003e-05 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.750726+0000", 2024-02-12T16:59:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7090000000000002e-06 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.750734+0000", 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.89992632400000006 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.650660+0000", 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102957 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.650763+0000", 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000132459 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.650895+0000", 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044140430000000003 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.655310+0000", 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0090129419999999995 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.664322+0000", 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0071849959999999999 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671507+0000", 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3056000000000002e-05 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671550+0000", 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8102999999999998e-05 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671579+0000", 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.92090182799999998 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127238 8.f 8:fba6dd37:::benchmark_data_smithi033_105085_object15580:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.764619+0000", 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.819824877000002, 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.907043355, 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127238 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.764619+0000", 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.764619+0000", 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.764616+0000", 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3289999999999994e-06 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.764624+0000", 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5165e-05 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.764639+0000", 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5179999999999998e-06 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.764647+0000", 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9632999999999997e-05 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.764697+0000", 2024-02-12T16:59:37.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9180000000000006e-06 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.764707+0000", 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88769449300000003 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.652401+0000", 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2915000000000003e-05 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.652464+0000", 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4653999999999994e-05 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.652539+0000", 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042483770000000002 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.656787+0000", 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0075768600000000004 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.664364+0000", 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0072496239999999997 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671613+0000", 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5205999999999999e-05 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671639+0000", 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3323999999999999e-05 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671662+0000", 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.907043355 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127243 8.f 8:ff68d556:::benchmark_data_smithi033_105085_object15585:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.769344+0000", 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.815099302, 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90241916499999997, 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127243 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.769344+0000", 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.769344+0000", 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.769342+0000", 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2749999999999998e-06 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.769349+0000", 2024-02-12T16:59:37.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8309999999999999e-05 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.769368+0000", 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0650000000000004e-06 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.769374+0000", 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5301e-05 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.769409+0000", 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3199999999999999e-06 2024-02-12T16:59:37.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.769414+0000", 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88421659600000002 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.653631+0000", 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6805999999999994e-05 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.653728+0000", 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014174000000000001 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.653870+0000", 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0046943139999999998 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.658564+0000", 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010341233 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.668905+0000", 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002796523 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671702+0000", 2024-02-12T16:59:37.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6023000000000001e-05 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671728+0000", 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5796000000000001e-05 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671763+0000", 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90241916499999997 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.35042.0:127258 8.f 8:f1912541:::benchmark_data_smithi033_105085_object15600:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-02-12T16:59:37.891 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:58:43.791456+0000", 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.792987385000004, 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88045005600000004, 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.35042", 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 127258 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.791456+0000", 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.791456+0000", 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.791454+0000", 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5969999999999997e-06 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.791461+0000", 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7890000000000006e-06 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.791471+0000", 2024-02-12T16:59:37.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2240000000000003e-06 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.791476+0000", 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4603000000000001e-05 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.791510+0000", 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0020000000000006e-06 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:43.791518+0000", 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.86400731900000005 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.655526+0000", 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2849000000000006e-05 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.655609+0000", 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7937999999999998e-05 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 4,6", 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.655686+0000", 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0058901409999999998 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T16:59:37.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.661577+0000", 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073635979999999998 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.668940+0000", 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028631920000000001 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671803+0000", 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3214000000000003e-05 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671847+0000", 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9596000000000003e-05 2024-02-12T16:59:37.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:58:44.671906+0000", 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88045005600000004 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T16:59:37.897 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:59:38.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:36.545258+0000 mgr.y (mgr.24481) 380 : cluster [DBG] pgmap v492: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 841 op/s; 26457/180645 objects misplaced (14.646%); 19 MiB/s, 298 objects/s recovering 2024-02-12T16:59:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:36.545258+0000 mgr.y (mgr.24481) 380 : cluster [DBG] pgmap v492: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 841 op/s; 26457/180645 objects misplaced (14.646%); 19 MiB/s, 298 objects/s recovering 2024-02-12T16:59:38.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:36.545258+0000 mgr.y (mgr.24481) 380 : cluster [DBG] pgmap v492: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 841 op/s; 26457/180645 objects misplaced (14.646%); 19 MiB/s, 298 objects/s recovering 2024-02-12T16:59:38.476 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:59:38.735 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:38.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:38.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:38.890 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:39.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:59:39.046 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:39.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:39.122 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:39.132 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:39.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:39.217 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:39.263 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:39.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:39.366 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:39.415 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:39.443 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:39.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:39.547 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:39.599 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:39.760 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:39.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:39.865 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:39.899 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:39.967 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T16:59:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:38.545999+0000 mgr.y (mgr.24481) 381 : cluster [DBG] pgmap v493: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 23517/171408 objects misplaced (13.720%); 29 MiB/s, 458 objects/s recovering 2024-02-12T16:59:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:39.376525+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.33:0/647944846' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:59:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:39.377076+0000 mon.a (mon.0) 1160 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:59:40.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:38.545999+0000 mgr.y (mgr.24481) 381 : cluster [DBG] pgmap v493: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 23517/171408 objects misplaced (13.720%); 29 MiB/s, 458 objects/s recovering 2024-02-12T16:59:40.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:39.376525+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.33:0/647944846' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:59:40.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:39.377076+0000 mon.a (mon.0) 1160 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:59:40.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:38.545999+0000 mgr.y (mgr.24481) 381 : cluster [DBG] pgmap v493: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 23517/171408 objects misplaced (13.720%); 29 MiB/s, 458 objects/s recovering 2024-02-12T16:59:40.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:39.376525+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.33:0/647944846' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:59:40.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:39.377076+0000 mon.a (mon.0) 1160 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:59:40.866 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:40.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:40.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:41.090 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:41.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:41.200 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:41.237 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:59:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:40] "GET /metrics HTTP/1.1" 200 175947 "" "Prometheus/2.33.4" 2024-02-12T16:59:41.238 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:39.957831+0000 mon.a (mon.0) 1161 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:59:41.238 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:39.957901+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e272: 8 total, 8 up, 5 in 2024-02-12T16:59:41.238 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:39.957831+0000 mon.a (mon.0) 1161 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:59:41.238 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:39.957901+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e272: 8 total, 8 up, 5 in 2024-02-12T16:59:41.241 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:41.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:39.957831+0000 mon.a (mon.0) 1161 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:59:41.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:39.957901+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e272: 8 total, 8 up, 5 in 2024-02-12T16:59:41.287 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:41.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:41.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:41.479 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:41.520 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:41.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:41.631 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:41.670 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:42.029 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:42.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:42.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:42.178 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:40.546345+0000 mgr.y (mgr.24481) 382 : cluster [DBG] pgmap v495: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 681 op/s; 23264/168387 objects misplaced (13.816%); 15 MiB/s, 236 objects/s recovering 2024-02-12T16:59:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:41.194345+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e273: 8 total, 8 up, 5 in 2024-02-12T16:59:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:40.546345+0000 mgr.y (mgr.24481) 382 : cluster [DBG] pgmap v495: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 681 op/s; 23264/168387 objects misplaced (13.816%); 15 MiB/s, 236 objects/s recovering 2024-02-12T16:59:42.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:41.194345+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e273: 8 total, 8 up, 5 in 2024-02-12T16:59:42.292 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:42.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:42.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:42.477 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:40.546345+0000 mgr.y (mgr.24481) 382 : cluster [DBG] pgmap v495: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 681 op/s; 23264/168387 objects misplaced (13.816%); 15 MiB/s, 236 objects/s recovering 2024-02-12T16:59:42.477 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:41.194345+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e273: 8 total, 8 up, 5 in 2024-02-12T16:59:42.481 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:42.511 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:42.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:42.616 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:42.675 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:42.731 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:42.741 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:59:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:59:42.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:42.779 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T16:59:42.779 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T16:59:42.779 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-02-12T16:59:42.779 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T16:59:42.842 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:42.926 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:42.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:43.068 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:43.111 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:43.581 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T16:59:43.775 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:43.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:43.856 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:59:43.882 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:43.891 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:59:43.922 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:59:43.955 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:43.956 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:59:44.004 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:44.011 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:59:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:44.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:44.064 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:59:44.110 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:59:44.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:44.137 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:59:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:59:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:59:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:59:44.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:59:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:59:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:59:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:59:44.142 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:59:44.162 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:44.247 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:42.546821+0000 mgr.y (mgr.24481) 383 : cluster [DBG] pgmap v497: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 804 B/s rd, 0 B/s wr, 642 op/s; 23264/168387 objects misplaced (13.816%); 14 MiB/s, 223 objects/s recovering 2024-02-12T16:59:44.248 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:43.813596+0000 mon.a (mon.0) 1164 : audit [DBG] from='client.? v1:172.21.15.33:0/1090905879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:44.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:59:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:59:44.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:42.546821+0000 mgr.y (mgr.24481) 383 : cluster [DBG] pgmap v497: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 804 B/s rd, 0 B/s wr, 642 op/s; 23264/168387 objects misplaced (13.816%); 14 MiB/s, 223 objects/s recovering 2024-02-12T16:59:44.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:43.813596+0000 mon.a (mon.0) 1164 : audit [DBG] from='client.? v1:172.21.15.33:0/1090905879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:42.546821+0000 mgr.y (mgr.24481) 383 : cluster [DBG] pgmap v497: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 804 B/s rd, 0 B/s wr, 642 op/s; 23264/168387 objects misplaced (13.816%); 14 MiB/s, 223 objects/s recovering 2024-02-12T16:59:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:43.813596+0000 mon.a (mon.0) 1164 : audit [DBG] from='client.? v1:172.21.15.33:0/1090905879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:44.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:59:44.514 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:44.516 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:44.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:44.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:59:44.647 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:44.667 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:59:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:44.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:59:44.771 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:44.799 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:44.818 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:44.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:44.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:44.923 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:44.956 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:44.983 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:44.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:45.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:59:45.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:43.947433+0000 mgr.y (mgr.24481) 384 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:45.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:44.014543+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e274: 8 total, 8 up, 5 in 2024-02-12T16:59:45.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:43.947433+0000 mgr.y (mgr.24481) 384 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:45.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:44.014543+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e274: 8 total, 8 up, 5 in 2024-02-12T16:59:45.167 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:45.198 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T16:59:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:45.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:59:45.208 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:43.947433+0000 mgr.y (mgr.24481) 384 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:45.208 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:44.014543+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e274: 8 total, 8 up, 5 in 2024-02-12T16:59:45.306 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:45.341 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:45.453 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:45.465 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:45.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:45.570 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:45.596 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:45.794 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:45.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:45.897 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:45.926 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:46.052 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:44.547270+0000 mgr.y (mgr.24481) 385 : cluster [DBG] pgmap v499: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 20331/156363 objects misplaced (13.002%); 14 MiB/s, 229 objects/s recovering 2024-02-12T16:59:46.052 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:45.021200+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e275: 8 total, 8 up, 5 in 2024-02-12T16:59:46.052 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:44.547270+0000 mgr.y (mgr.24481) 385 : cluster [DBG] pgmap v499: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 20331/156363 objects misplaced (13.002%); 14 MiB/s, 229 objects/s recovering 2024-02-12T16:59:46.052 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:45.021200+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e275: 8 total, 8 up, 5 in 2024-02-12T16:59:46.054 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:46.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:46.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:46.200 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:46.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:44.547270+0000 mgr.y (mgr.24481) 385 : cluster [DBG] pgmap v499: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 20331/156363 objects misplaced (13.002%); 14 MiB/s, 229 objects/s recovering 2024-02-12T16:59:46.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:45.021200+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e275: 8 total, 8 up, 5 in 2024-02-12T16:59:46.890 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:47.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:46.021889+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e276: 8 total, 8 up, 5 in 2024-02-12T16:59:47.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:46.186973+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e277: 8 total, 8 up, 5 in 2024-02-12T16:59:47.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:46.021889+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e276: 8 total, 8 up, 5 in 2024-02-12T16:59:47.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:46.186973+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e277: 8 total, 8 up, 5 in 2024-02-12T16:59:47.478 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:59:47.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:46.021889+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e276: 8 total, 8 up, 5 in 2024-02-12T16:59:47.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:46.186973+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e277: 8 total, 8 up, 5 in 2024-02-12T16:59:47.763 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:47.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:47.873 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:47.908 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:47.991 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:48.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:48.100 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:48.145 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:48.272 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:46.547675+0000 mgr.y (mgr.24481) 386 : cluster [DBG] pgmap v503: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 20331/156363 objects misplaced (13.002%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:59:48.272 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:46.547675+0000 mgr.y (mgr.24481) 386 : cluster [DBG] pgmap v503: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 20331/156363 objects misplaced (13.002%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:59:48.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:46.547675+0000 mgr.y (mgr.24481) 386 : cluster [DBG] pgmap v503: 138 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 130 active+clean; 3.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 20331/156363 objects misplaced (13.002%); 21 MiB/s, 329 objects/s recovering 2024-02-12T16:59:49.395 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:47.904064+0000 osd.6 (osd.6) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:49.395 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:47.904122+0000 osd.6 (osd.6) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:49.395 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:48.217725+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e278: 8 total, 8 up, 5 in 2024-02-12T16:59:49.396 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:48.548278+0000 mgr.y (mgr.24481) 387 : cluster [DBG] pgmap v505: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 49 KiB/s rd, 0 B/s wr, 648 op/s; 18789/149544 objects misplaced (12.564%); 8.8 MiB/s, 140 objects/s recovering 2024-02-12T16:59:49.396 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:47.904064+0000 osd.6 (osd.6) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:49.396 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:47.904122+0000 osd.6 (osd.6) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:49.396 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:48.217725+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e278: 8 total, 8 up, 5 in 2024-02-12T16:59:49.396 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:48.548278+0000 mgr.y (mgr.24481) 387 : cluster [DBG] pgmap v505: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 49 KiB/s rd, 0 B/s wr, 648 op/s; 18789/149544 objects misplaced (12.564%); 8.8 MiB/s, 140 objects/s recovering 2024-02-12T16:59:49.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:47.904064+0000 osd.6 (osd.6) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:59:49.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:47.904122+0000 osd.6 (osd.6) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:59:49.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:48.217725+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e278: 8 total, 8 up, 5 in 2024-02-12T16:59:49.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:48.548278+0000 mgr.y (mgr.24481) 387 : cluster [DBG] pgmap v505: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 49 KiB/s rd, 0 B/s wr, 648 op/s; 18789/149544 objects misplaced (12.564%); 8.8 MiB/s, 140 objects/s recovering 2024-02-12T16:59:49.920 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:49.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:50.031 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:50.096 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:50.127 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:50.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:50.186 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:50.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:50.268 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T16:59:50.273 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T16:59:50.308 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:50.349 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:50.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:50.355 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:49.974192+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.33:0/3810339558' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:59:50.355 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:49.974560+0000 mon.a (mon.0) 1170 : audit [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:59:50.355 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:50.204881+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.33:0/3199470604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:59:50.355 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:50.207800+0000 mon.a (mon.0) 1171 : audit [INF] from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:59:50.456 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:50.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:49.974192+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.33:0/3810339558' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:59:50.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:49.974560+0000 mon.a (mon.0) 1170 : audit [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:59:50.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:50.204881+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.33:0/3199470604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:59:50.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:50.207800+0000 mon.a (mon.0) 1171 : audit [INF] from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:59:50.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:49.974192+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.33:0/3810339558' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:59:50.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:49.974560+0000 mon.a (mon.0) 1170 : audit [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:59:50.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:50.204881+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.33:0/3199470604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:59:50.508 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:50.207800+0000 mon.a (mon.0) 1171 : audit [INF] from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:59:50.514 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:50.571 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T16:59:50.571 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:59:51.270 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 16:59:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:50] "GET /metrics HTTP/1.1" 200 175949 "" "Prometheus/2.33.4" 2024-02-12T16:59:51.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:50.235302+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:51.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:50.247592+0000 mon.a (mon.0) 1173 : audit [INF] from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:59:51.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:50.247640+0000 mon.a (mon.0) 1174 : audit [INF] from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:59:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:50.247665+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e279: 8 total, 8 up, 5 in 2024-02-12T16:59:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:50.548593+0000 mgr.y (mgr.24481) 388 : cluster [DBG] pgmap v507: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 57 KiB/s rd, 0 B/s wr, 746 op/s; 18577/147492 objects misplaced (12.595%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T16:59:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:50.235302+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:50.247592+0000 mon.a (mon.0) 1173 : audit [INF] from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:59:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:50.247640+0000 mon.a (mon.0) 1174 : audit [INF] from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:59:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:50.247665+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e279: 8 total, 8 up, 5 in 2024-02-12T16:59:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:50.548593+0000 mgr.y (mgr.24481) 388 : cluster [DBG] pgmap v507: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 57 KiB/s rd, 0 B/s wr, 746 op/s; 18577/147492 objects misplaced (12.595%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T16:59:51.614 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T16:59:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:50.235302+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:59:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:50.247592+0000 mon.a (mon.0) 1173 : audit [INF] from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:59:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:50.247640+0000 mon.a (mon.0) 1174 : audit [INF] from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:59:51.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:50.247665+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e279: 8 total, 8 up, 5 in 2024-02-12T16:59:51.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:50.548593+0000 mgr.y (mgr.24481) 388 : cluster [DBG] pgmap v507: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 57 KiB/s rd, 0 B/s wr, 746 op/s; 18577/147492 objects misplaced (12.595%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T16:59:52.690 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:52.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:52.796 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:52.856 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 16:59:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:16:59:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T16:59:52.864 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:52.906 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:52.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:53.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:53.052 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:53.140 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:53.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:53.248 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:53.289 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:53.655 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:53.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:53.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:59:53.758 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:53.815 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:53.965 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:52.549020+0000 mgr.y (mgr.24481) 389 : cluster [DBG] pgmap v508: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 40 KiB/s rd, 0 B/s wr, 531 op/s; 18577/147492 objects misplaced (12.595%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:59:53.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 16:59:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T16:59:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:52.549020+0000 mgr.y (mgr.24481) 389 : cluster [DBG] pgmap v508: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 40 KiB/s rd, 0 B/s wr, 531 op/s; 18577/147492 objects misplaced (12.595%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:59:54.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:52.549020+0000 mgr.y (mgr.24481) 389 : cluster [DBG] pgmap v508: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 3.0 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 40 KiB/s rd, 0 B/s wr, 531 op/s; 18577/147492 objects misplaced (12.595%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:59:54.166 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T16:59:54.177 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:54.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:59:54.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:54.333 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:54.370 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:54.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:54.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:59:54.550 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:54.590 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T16:59:54.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:54.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:59:54.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:54.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:53.958835+0000 mgr.y (mgr.24481) 390 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:54.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:53.958835+0000 mgr.y (mgr.24481) 390 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:54.816 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:54.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:54.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T16:59:54.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T16:59:54.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T16:59:54.897 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T16:59:54.916 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T16:59:54.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:54.944 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T16:59:54.963 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:53.958835+0000 mgr.y (mgr.24481) 390 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T16:59:54.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T16:59:54.966 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:55.008 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T16:59:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:59:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:59:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:59:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:59:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:59:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:59:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:59:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:59:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:55.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:55.114 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:55.158 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:59:55.158 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:55.368 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:55.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:55.473 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:55.523 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:55.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:54.549658+0000 mgr.y (mgr.24481) 391 : cluster [DBG] pgmap v509: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.02k op/s; 16656/134133 objects misplaced (12.418%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T16:59:55.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:54.819518+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.33:0/1431997358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:55.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:54.549658+0000 mgr.y (mgr.24481) 391 : cluster [DBG] pgmap v509: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.02k op/s; 16656/134133 objects misplaced (12.418%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T16:59:55.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:54.819518+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.33:0/1431997358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:55.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:55.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:54.549658+0000 mgr.y (mgr.24481) 391 : cluster [DBG] pgmap v509: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.02k op/s; 16656/134133 objects misplaced (12.418%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T16:59:55.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:54.819518+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.33:0/1431997358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:59:56.012 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:59:56.075 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T16:59:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:56.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:59:56.158 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:56.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:59:56.265 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:56.301 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T16:59:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:56.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:59:56.409 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:56.444 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:56.702 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:56.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:56.810 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:59:56.853 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:56.550013+0000 mgr.y (mgr.24481) 392 : cluster [DBG] pgmap v510: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 53 KiB/s rd, 0 B/s wr, 976 op/s; 16656/134133 objects misplaced (12.418%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T16:59:57.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:56.550013+0000 mgr.y (mgr.24481) 392 : cluster [DBG] pgmap v510: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 53 KiB/s rd, 0 B/s wr, 976 op/s; 16656/134133 objects misplaced (12.418%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T16:59:57.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:56.550013+0000 mgr.y (mgr.24481) 392 : cluster [DBG] pgmap v510: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 53 KiB/s rd, 0 B/s wr, 976 op/s; 16656/134133 objects misplaced (12.418%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T16:59:57.817 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T16:59:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:57.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:59:57.925 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:59:57.970 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:58.727 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:59:58.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:58.421697+0000 mon.c (mon.2) 154 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:59:58.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:58.421697+0000 mon.c (mon.2) 154 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:59:58.767 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:58.421697+0000 mon.c (mon.2) 154 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:59:58.843 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:59:59.027 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T16:59:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:59.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:59:59.144 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:59:59.186 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:59.316 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T16:59:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T16:59:59.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:59:59.425 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:59:59.478 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T16:59:59.514 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T16:59:59.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:58.550625+0000 mgr.y (mgr.24481) 393 : cluster [DBG] pgmap v511: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 25 KiB/s rd, 0 B/s wr, 873 op/s; 15059/124566 objects misplaced (12.089%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:59:59.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:58.816350+0000 mon.c (mon.2) 155 : audit [INF] from='client.? v1:172.21.15.33:0/3966895733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:59:59.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:58.816799+0000 mon.a (mon.0) 1176 : audit [INF] from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:59.051967+0000 mon.c (mon.2) 156 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:59.053030+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:59.061687+0000 mon.a (mon.0) 1177 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:58.550625+0000 mgr.y (mgr.24481) 393 : cluster [DBG] pgmap v511: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 25 KiB/s rd, 0 B/s wr, 873 op/s; 15059/124566 objects misplaced (12.089%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:58.816350+0000 mon.c (mon.2) 155 : audit [INF] from='client.? v1:172.21.15.33:0/3966895733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:58.816799+0000 mon.a (mon.0) 1176 : audit [INF] from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:59.051967+0000 mon.c (mon.2) 156 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:59.053030+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:59:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 16:59:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:59.061687+0000 mon.a (mon.0) 1177 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T16:59:59.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:58.550625+0000 mgr.y (mgr.24481) 393 : cluster [DBG] pgmap v511: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 25 KiB/s rd, 0 B/s wr, 873 op/s; 15059/124566 objects misplaced (12.089%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:59:59.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:58.816350+0000 mon.c (mon.2) 155 : audit [INF] from='client.? v1:172.21.15.33:0/3966895733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:59:59.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:58.816799+0000 mon.a (mon.0) 1176 : audit [INF] from='client.35261 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:59:59.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:59.051967+0000 mon.c (mon.2) 156 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:59:59.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:59.053030+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:59:59.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 16:59:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:59.061687+0000 mon.a (mon.0) 1177 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:00:00.636 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T17:00:00.637 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:00.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:00.741 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T16:59:59.497428+0000 mon.a (mon.0) 1178 : audit [INF] from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:00:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T16:59:59.497469+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e280: 8 total, 8 up, 5 in 2024-02-12T17:00:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:00.000428+0000 mon.a (mon.0) 1180 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-12T17:00:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:00.000460+0000 mon.a (mon.0) 1181 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T17:00:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:00.000470+0000 mon.a (mon.0) 1182 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-12T17:00:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:00.000482+0000 mon.a (mon.0) 1183 : cluster [WRN] pg 8.9 is active+remapped+backfill_toofull, acting [5,0,6] 2024-02-12T17:00:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T16:59:59.497428+0000 mon.a (mon.0) 1178 : audit [INF] from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:00:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T16:59:59.497469+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e280: 8 total, 8 up, 5 in 2024-02-12T17:00:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:00.000428+0000 mon.a (mon.0) 1180 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-12T17:00:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:00.000460+0000 mon.a (mon.0) 1181 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T17:00:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:00.000470+0000 mon.a (mon.0) 1182 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-12T17:00:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:00.000482+0000 mon.a (mon.0) 1183 : cluster [WRN] pg 8.9 is active+remapped+backfill_toofull, acting [5,0,6] 2024-02-12T17:00:00.790 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:00.819 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:00.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:00.925 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:00.968 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T16:59:59.497428+0000 mon.a (mon.0) 1178 : audit [INF] from='client.35261 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:00:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T16:59:59.497469+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e280: 8 total, 8 up, 5 in 2024-02-12T17:00:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:00.000428+0000 mon.a (mon.0) 1180 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-12T17:00:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:00.000460+0000 mon.a (mon.0) 1181 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T17:00:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:00.000470+0000 mon.a (mon.0) 1182 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-12T17:00:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:00.000482+0000 mon.a (mon.0) 1183 : cluster [WRN] pg 8.9 is active+remapped+backfill_toofull, acting [5,0,6] 2024-02-12T17:00:01.124 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:01.126 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:00:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:00] "GET /metrics HTTP/1.1" 200 175949 "" "Prometheus/2.33.4" 2024-02-12T17:00:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:01.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:01.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:01.302 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:01.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:01.433 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:01.469 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:01.603 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:01.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:01.708 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:01.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:00.551061+0000 mgr.y (mgr.24481) 394 : cluster [DBG] pgmap v513: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 19 KiB/s rd, 0 B/s wr, 894 op/s; 14803/121584 objects misplaced (12.175%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T17:00:01.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:01.191228+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e281: 8 total, 8 up, 5 in 2024-02-12T17:00:01.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:00.551061+0000 mgr.y (mgr.24481) 394 : cluster [DBG] pgmap v513: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 19 KiB/s rd, 0 B/s wr, 894 op/s; 14803/121584 objects misplaced (12.175%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T17:00:01.744 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:01.191228+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e281: 8 total, 8 up, 5 in 2024-02-12T17:00:01.744 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:00.551061+0000 mgr.y (mgr.24481) 394 : cluster [DBG] pgmap v513: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 19 KiB/s rd, 0 B/s wr, 894 op/s; 14803/121584 objects misplaced (12.175%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T17:00:01.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:01.191228+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e281: 8 total, 8 up, 5 in 2024-02-12T17:00:01.748 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:01.903 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:01.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:02.010 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:02.052 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:02.187 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:02.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:02.295 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:02.342 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:02.466 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:00:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T17:00:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T17:00:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:00:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49318 7.3 7:c02d99e1:::benchmark_data_smithi033_81060_object49317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T17:00:02.466 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.938291+0000", 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 422.51408635799999, 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072684040000000005, 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49318 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.938291+0000", 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.938291+0000", 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.938289+0000", 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002156233 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.940446+0000", 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0890000000000001e-05 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.940457+0000", 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5180000000000004e-06 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.940465+0000", 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00038189400000000003 2024-02-12T17:00:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.940847+0000", 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011186 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.940959+0000", 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013825799999999999 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.941097+0000", 2024-02-12T17:00:02.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0034128819999999999 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944510+0000", 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066306871000000003 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.010817+0000", 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2234999999999997e-05 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.010869+0000", 2024-02-12T17:00:02.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6830999999999998e-05 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.010926+0000", 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8610999999999999e-05 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.010975+0000", 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072684040000000005 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49328 7.a 7:554bcf16:::benchmark_data_smithi033_81060_object49327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.944178+0000", 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 422.50819916799998, 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071120117999999996, 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49328 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944178+0000", 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944178+0000", 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944175+0000", 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7192e-05 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944202+0000", 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2371000000000001e-05 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944215+0000", 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5540000000000003e-06 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944221+0000", 2024-02-12T17:00:02.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0074999999999997e-05 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944271+0000", 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000128465 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944400+0000", 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000148016 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.944548+0000", 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003352636 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947900+0000", 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.067270859000000002 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015171+0000", 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5435999999999998e-05 2024-02-12T17:00:02.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015207+0000", 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1573000000000002e-05 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015258+0000", 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9558000000000003e-05 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015298+0000", 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071120117999999996 2024-02-12T17:00:02.478 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49331 7.3 7:c343f2a6:::benchmark_data_smithi033_81060_object49330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.947657+0000", 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 422.504719696, 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065992717000000006, 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49331 2024-02-12T17:00:02.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947657+0000", 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947657+0000", 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947656+0000", 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.472e-05 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947680+0000", 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3690000000000001e-06 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947688+0000", 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.339e-06 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947693+0000", 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7277999999999999e-05 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947790+0000", 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014679100000000001 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.947937+0000", 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000165241 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:00:02.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.948102+0000", 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062742293000000005 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.010845+0000", 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014262300000000001 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.010987+0000", 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025789680000000001 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.013566+0000", 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7488999999999999e-05 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.013614+0000", 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6235999999999997e-05 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.013650+0000", 2024-02-12T17:00:02.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065992717000000006 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:49333 7.a 7:57813081:::benchmark_data_smithi033_81060_object49332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.949482+0000", 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 422.50289501200001, 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071148713000000002, 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.485 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 49333 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949482+0000", 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949482+0000", 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949480+0000", 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5533000000000001e-05 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949505+0000", 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0988999999999999e-05 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949516+0000", 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2179999999999996e-06 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949522+0000", 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8251999999999998e-05 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949571+0000", 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011555199999999999 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.949686+0000", 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063704312999999999 2024-02-12T17:00:02.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.013391+0000", 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.005831976 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.019223+0000", 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00095663400000000004 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.020179+0000", 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00035717600000000001 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.020536+0000", 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3053000000000002e-05 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.020589+0000", 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1205999999999998e-05 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.020631+0000", 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071148713000000002 2024-02-12T17:00:02.490 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.14838.0:49328 7.a e111/93 ondisk, result = 0)", 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:52:59.950139+0000", 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 422.50223822100003, 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065036161999999995, 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.950139+0000", 2024-02-12T17:00:02.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.950139+0000", 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.950137+0000", 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8160000000000005e-06 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:52:59.950147+0000", 2024-02-12T17:00:02.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.058096591000000003 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.008244+0000", 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5380000000000001e-05 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.008269+0000", 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068886930000000004 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015158+0000", 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4359999999999999e-06 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015164+0000", 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0582e-05 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:00.015175+0000", 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065036161999999995 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.494 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79130 7.a 7:56c3b3b4:::benchmark_data_smithi033_81060_object79129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.412738+0000", 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.03963897099999, 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.099741402000000007, 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79130 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.495 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412738+0000", 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412738+0000", 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412736+0000", 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6786000000000003e-05 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412772+0000", 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2577e-05 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412785+0000", 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4540000000000001e-06 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412792+0000", 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016494 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412957+0000", 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013535000000000001 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413093+0000", 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000169612 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413262+0000", 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014261098 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.427523+0000", 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013991546000000001 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441515+0000", 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070881793999999998 2024-02-12T17:00:02.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512397+0000", 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4039999999999998e-05 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512441+0000", 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8479000000000002e-05 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512479+0000", 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.099741402000000007 2024-02-12T17:00:02.500 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79131 7.a 7:5c3e7aed:::benchmark_data_smithi033_81060_object79130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.412923+0000", 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.039453407, 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.099724977000000006, 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79131 2024-02-12T17:00:02.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412923+0000", 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412923+0000", 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412923+0000", 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0502000000000001e-05 2024-02-12T17:00:02.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412943+0000", 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8349999999999998e-06 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412950+0000", 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4730000000000002e-06 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.412954+0000", 2024-02-12T17:00:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088317369999999992 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.421786+0000", 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012325699999999999 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.421909+0000", 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015588099999999999 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:00:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.422065+0000", 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01197314 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.434038+0000", 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074904250000000002 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441529+0000", 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071041235999999994 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512570+0000", 2024-02-12T17:00:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1928000000000002e-05 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512612+0000", 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6377000000000001e-05 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512648+0000", 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.099724977000000006 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79136 7.2 7:41be778c:::benchmark_data_smithi033_81060_object79135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.413639+0000", 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.038737674, 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.098965773000000007, 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79136 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413639+0000", 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413639+0000", 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413638+0000", 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.26e-05 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413660+0000", 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0091e-05 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413670+0000", 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2700000000000001e-06 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413677+0000", 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002277346 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.415954+0000", 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010863599999999999 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.416063+0000", 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000144277 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.416207+0000", 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011346528999999999 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.427553+0000", 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013970231 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441524+0000", 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070995239000000002 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512519+0000", 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8962999999999999e-05 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512568+0000", 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7054000000000001e-05 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512605+0000", 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.098965773000000007 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79138 7.a 7:530077c3:::benchmark_data_smithi033_81060_object79137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.413682+0000", 2024-02-12T17:00:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.038695281, 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.099064142999999993, 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79138 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413682+0000", 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413682+0000", 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413681+0000", 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5566999999999999e-05 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413696+0000", 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8800000000000001e-06 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413700+0000", 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.762e-06 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413703+0000", 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010876866000000001 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424580+0000", 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014050400000000001 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424720+0000", 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000202797 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424923+0000", 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091431220000000001 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.434066+0000", 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074679710000000003 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441534+0000", 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071145349999999996 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512680+0000", 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5098999999999997e-05 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512715+0000", 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0932000000000001e-05 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512746+0000", 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.099064142999999993 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79139 7.a 7:56b09edd:::benchmark_data_smithi033_81060_object79138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.413706+0000", 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.03867073599997, 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.099206076000000004, 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79139 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413706+0000", 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413706+0000", 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413706+0000", 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5364000000000001e-05 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413721+0000", 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1200000000000002e-06 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413724+0000", 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5339999999999998e-06 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.413727+0000", 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013849016 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.427576+0000", 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012771399999999999 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.427703+0000", 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036661699999999998 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.431370+0000", 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010176068999999999 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441546+0000", 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070806050999999995 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512352+0000", 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047339799999999998 2024-02-12T17:00:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512825+0000", 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2049999999999999e-05 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512867+0000", 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5003999999999998e-05 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512912+0000", 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.099206076000000004 2024-02-12T17:00:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79143 7.2 7:46065d23:::benchmark_data_smithi033_81060_object79142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.424743+0000", 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.02763364999998, 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.087965161, 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79143 2024-02-12T17:00:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424743+0000", 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424743+0000", 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424741+0000", 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0287999999999999e-05 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424771+0000", 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0242e-05 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424781+0000", 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.657e-06 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.424788+0000", 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003280196 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.428068+0000", 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104509 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.428173+0000", 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014772400000000001 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.428321+0000", 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011966291 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.440287+0000", 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001254212 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441541+0000", 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.071097629999999995 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512639+0000", 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7129000000000003e-05 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512676+0000", 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2591e-05 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512708+0000", 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.087965161 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79154 7.3 7:c5fee9d6:::benchmark_data_smithi033_81060_object79153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.441631+0000", 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.01074617, 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.073696696000000006, 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79154 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441631+0000", 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441631+0000", 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441628+0000", 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1230000000000003e-05 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441669+0000", 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3557000000000001e-05 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441683+0000", 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7259999999999996e-06 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441690+0000", 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022743700000000001 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441917+0000", 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012882700000000001 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.442046+0000", 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015096400000000001 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.442197+0000", 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070056303 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512253+0000", 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029334700000000001 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515187+0000", 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0164999999999997e-05 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515247+0000", 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5896000000000002e-05 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515293+0000", 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4829999999999997e-05 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515327+0000", 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.073696696000000006 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79157 7.3 7:cdb8a9ef:::benchmark_data_smithi033_81060_object79156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.441706+0000", 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.01067068700002, 2024-02-12T17:00:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.073741685000000001, 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79157 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441706+0000", 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441706+0000", 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441705+0000", 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5346999999999999e-05 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441740+0000", 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3479999999999993e-06 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441750+0000", 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5650000000000001e-06 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441755+0000", 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029208239999999998 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.444676+0000", 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125878 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.444802+0000", 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000152798 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.444955+0000", 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.067304517999999994 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512259+0000", 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002959415 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515219+0000", 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000161392 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515380+0000", 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6477000000000003e-05 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515416+0000", 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1457000000000001e-05 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515448+0000", 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.073741685000000001 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79159 7.3 7:c15b4a7b:::benchmark_data_smithi033_81060_object79158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.441761+0000", 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.01061591500002, 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.083795318999999993, 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79159 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441761+0000", 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441761+0000", 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441760+0000", 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2039999999999999e-05 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441782+0000", 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8959999999999997e-06 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441789+0000", 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8989999999999998e-06 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441795+0000", 2024-02-12T17:00:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070488432000000004 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512283+0000", 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000173945 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512457+0000", 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020397199999999999 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:00:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512661+0000", 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0030500119999999999 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.515711+0000", 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00079335899999999999 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.516505+0000", 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089524500000000007 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.525457+0000", 2024-02-12T17:00:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4543999999999998e-05 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.525512+0000", 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4616000000000001e-05 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.525556+0000", 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.083795318999999993 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:79151 7.6 e120/93 7:69fe64ad:::benchmark_data_smithi033_81060_object79150:head v 120'4931, mlcod=120'4929)", 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.441817+0000", 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.01056019399999, 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072286176999999993, 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441817+0000", 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441817+0000", 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441816+0000", 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3142999999999998e-05 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441839+0000", 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7600000000000002e-06 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441846+0000", 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7620000000000001e-06 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441852+0000", 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4926999999999999e-05 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441897+0000", 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2123000000000001e-05 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.441919+0000", 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056076499999999996 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.447527+0000", 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066575898999999994 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.514103+0000", 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072286176999999993 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.14838.0:79139 7.a e120/93 ondisk, result = 0)", 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.445540+0000", 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.00683670199999, 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066824146000000001, 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445540+0000", 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445540+0000", 2024-02-12T17:00:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445539+0000", 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5089999999999999e-06 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445545+0000", 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9089999999999995e-06 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445555+0000", 2024-02-12T17:00:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9240000000000001e-06 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.445562+0000", 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066769838999999997 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512332+0000", 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1586000000000001e-05 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512343+0000", 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0835e-05 2024-02-12T17:00:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512364+0000", 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.066824146000000001 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop_reply(client.14838.0:79130 7.a e120/93 ondisk, result = 0)", 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.446587+0000", 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.005789515, 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065899446, 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:00:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.446587+0000", 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.446587+0000", 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.446586+0000", 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.716e-06 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.446592+0000", 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065568790000000002 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512161+0000", 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.216e-05 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512173+0000", 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000213604 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512386+0000", 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1240000000000004e-06 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512391+0000", 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5372999999999996e-05 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512487+0000", 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.065899446 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:79162 7.2 7:4f2ee7e3:::benchmark_data_smithi033_81060_object79161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.448026+0000", 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.00435064599998, 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.079890828999999997, 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 79162 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.448026+0000", 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.448026+0000", 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.448024+0000", 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2904999999999998e-05 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.448047+0000", 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064131896999999993 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512179+0000", 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8110000000000003e-06 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512186+0000", 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00058763499999999998 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512774+0000", 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012458599999999999 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512899+0000", 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017414500000000001 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.513073+0000", 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092326049999999996 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.522305+0000", 2024-02-12T17:00:02.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031838550000000002 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.525489+0000", 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023478650000000002 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.527837+0000", 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1972999999999999e-05 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.527879+0000", 2024-02-12T17:00:02.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8022000000000003e-05 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.527917+0000", 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.079890828999999997 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.14838.0:79163 7.c e120/93 7:3e687bab:::benchmark_data_smithi033_81060_object79162:head v 120'4966, mlcod=120'4964)", 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:27.449354+0000", 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 395.003023063, 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.069593822, 2024-02-12T17:00:02.563 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.449354+0000", 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.449354+0000", 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.449353+0000", 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7880000000000001e-05 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.449410+0000", 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062751984999999996 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512162+0000", 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0769e-05 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512173+0000", 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0383000000000003e-05 2024-02-12T17:00:02.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512214+0000", 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3244e-05 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.512237+0000", 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0066648749999999998 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.518902+0000", 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5936000000000002e-05 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:27.518948+0000", 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.069593822 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.14838.0:80693 7.2 7:4126d7be:::benchmark_data_smithi033_81060_object80692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T16:53:29.037607+0000", 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 393.41477006700001, 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064764652000000006, 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.14838", 2024-02-12T17:00:02.567 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 80693 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.037607+0000", 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.037607+0000", 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.037605+0000", 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5592000000000002e-05 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.037630+0000", 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0473e-05 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.037641+0000", 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9950000000000002e-06 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.037647+0000", 2024-02-12T17:00:02.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2098999999999998e-05 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.037689+0000", 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013699799999999999 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.037826+0000", 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000183547 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.038009+0000", 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0058546379999999997 2024-02-12T17:00:02.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.043864+0000", 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047303099999999997 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.048594+0000", 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053643248999999997 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102237+0000", 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5119999999999999e-05 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102313+0000", 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8851000000000003e-05 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T16:53:29.102371+0000", 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.064764652000000006 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:00:02.572 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:00:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:02.573 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:02.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:02.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:02.735 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:02.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:02.777 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:00:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:00:02.875 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:02.934 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:02.968 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:02.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:03.073 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:03.119 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:03.170 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:03.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:03.280 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:03.338 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:03.373 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:03.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:03.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:03.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T17:00:03.482 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:03.546 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:03.560 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:03.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:03.630 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:00:03.727 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:03.746 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:02.551483+0000 mgr.y (mgr.24481) 395 : cluster [DBG] pgmap v515: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 523 op/s; 14803/121584 objects misplaced (12.175%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:03.746 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:02.786584+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.33:0/2285522651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:03.746 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:02.787668+0000 mon.a (mon.0) 1185 : audit [INF] from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:03.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:02.551483+0000 mgr.y (mgr.24481) 395 : cluster [DBG] pgmap v515: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 523 op/s; 14803/121584 objects misplaced (12.175%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:03.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:02.786584+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.33:0/2285522651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:03.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:02.787668+0000 mon.a (mon.0) 1185 : audit [INF] from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:03.762 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:02.551483+0000 mgr.y (mgr.24481) 395 : cluster [DBG] pgmap v515: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.5 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 523 op/s; 14803/121584 objects misplaced (12.175%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:03.762 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:02.786584+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.33:0/2285522651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:03.762 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:02.787668+0000 mon.a (mon.0) 1185 : audit [INF] from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:03.837 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:03.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:03.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:03.943 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:03.981 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:03.995 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:04.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:00:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:00:04.441 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:04.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:04.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:03.611174+0000 mon.a (mon.0) 1186 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:00:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:03.615198+0000 mon.a (mon.0) 1187 : audit [INF] from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:03.615257+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e282: 8 total, 8 up, 5 in 2024-02-12T17:00:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:03.974481+0000 mgr.y (mgr.24481) 396 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:03.611174+0000 mon.a (mon.0) 1186 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:00:04.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:03.615198+0000 mon.a (mon.0) 1187 : audit [INF] from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:04.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:03.615257+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e282: 8 total, 8 up, 5 in 2024-02-12T17:00:04.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:03.974481+0000 mgr.y (mgr.24481) 396 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:04.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:03.611174+0000 mon.a (mon.0) 1186 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:00:04.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:03.615198+0000 mon.a (mon.0) 1187 : audit [INF] from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:04.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:03.615257+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e282: 8 total, 8 up, 5 in 2024-02-12T17:00:04.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:03.974481+0000 mgr.y (mgr.24481) 396 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:05.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:05.640 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:04.551863+0000 mgr.y (mgr.24481) 397 : cluster [DBG] pgmap v517: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 871 op/s; 12924/108897 objects misplaced (11.868%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:00:05.717 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:05.757 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:05.788 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:05.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:05.891 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:05.934 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:04.551863+0000 mgr.y (mgr.24481) 397 : cluster [DBG] pgmap v517: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 871 op/s; 12924/108897 objects misplaced (11.868%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:00:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:05.956 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:04.551863+0000 mgr.y (mgr.24481) 397 : cluster [DBG] pgmap v517: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 871 op/s; 12924/108897 objects misplaced (11.868%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:00:06.012 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:06.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:06.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:06.259 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:06.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:06.332 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:189415 8.5 e283/258 8:a166f1a2:::benchmark_data_smithi033_105085_object77757:head v 283'11779, mlcod=283'11776)", 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:06.312767+0000", 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0097712630000000005, 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0097992819999999994, 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312767+0000", 2024-02-12T17:00:06.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312767+0000", 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312765+0000", 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5820000000000004e-06 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:06.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312773+0000", 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1214e-05 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312784+0000", 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9070000000000004e-06 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312790+0000", 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1236999999999997e-05 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:06.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312841+0000", 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0014000000000001e-05 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312871+0000", 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000104594 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:189416 8.5 e283/258 8:a575fed5:::benchmark_data_smithi033_105085_object77758:head v 283'11780, mlcod=283'11776)", 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:06.312799+0000", 2024-02-12T17:00:06.336 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0097384120000000001, 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0098428249999999995, 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312799+0000", 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312799+0000", 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312799+0000", 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6689999999999997e-06 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312804+0000", 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9910000000000002e-06 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312809+0000", 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8590000000000004e-06 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312813+0000", 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001797421 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.314611+0000", 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.961e-05 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.314630+0000", 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018309229999999999 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:189423 8.e e283/176 8:7ecb80d9:::benchmark_data_smithi033_105085_object77765:head v 283'11851, mlcod=282'11849)", 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:06.319220+0000", 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.003318016, 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034915459999999999, 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319220+0000", 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319220+0000", 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319218+0000", 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3189999999999995e-06 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319227+0000", 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0757999999999999e-05 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319237+0000", 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3619999999999997e-06 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319243+0000", 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000123838 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319367+0000", 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7668000000000001e-05 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319384+0000", 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016439699999999999 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:189425 8.e e283/176 8:7e82712e:::benchmark_data_smithi033_105085_object77767:head v 283'11852, mlcod=282'11849)", 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:06.319252+0000", 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0032861539999999999, 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035285910000000002, 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:06.343 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319252+0000", 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319252+0000", 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319251+0000", 2024-02-12T17:00:06.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5590000000000001e-06 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319255+0000", 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319259+0000", 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.19e-06 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319262+0000", 2024-02-12T17:00:06.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019491300000000001 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.321212+0000", 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8068e-05 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.321230+0000", 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019779139999999999 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.346 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:189424 8.5 e283/258)", 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:06.319280+0000", 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0032575299999999998, 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035674309999999998, 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "queued for pg", 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319280+0000", 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319280+0000", 2024-02-12T17:00:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319280+0000", 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5369999999999999e-06 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319284+0000", 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.87e-06 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319289+0000", 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2930000000000001e-06 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.319293+0000", 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2143999999999999e-05 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:189412 8.b 8:d94f9a04:::benchmark_data_smithi033_105085_object77754:head [delete] snapc 0=[] ondisk+write+known_if_redirected e283)", 2024-02-12T17:00:06.349 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:06.301999+0000", 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.020539017, 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020909093, 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 189412 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.301999+0000", 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.301999+0000", 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.301997+0000", 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1060000000000002e-06 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.302004+0000", 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2864000000000001e-05 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.302017+0000", 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6359999999999997e-06 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.302023+0000", 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001845326 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.303869+0000", 2024-02-12T17:00:06.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000128866 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.303998+0000", 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015608199999999999 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,7", 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.304154+0000", 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063064430000000001 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.310460+0000", 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001856494 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312317+0000", 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3884000000000003e-05 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:06.312351+0000", 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010351711 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T17:00:06.354 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 6 2024-02-12T17:00:06.355 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:00:06.365 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:06.418 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:06.975 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:06.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:07.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T17:00:07.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:07.183 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:07.229 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:07.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:07.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:07.337 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:07.374 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:05.711106+0000 osd.0 (osd.0) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:07.374 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:05.711157+0000 osd.0 (osd.0) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:07.374 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:05.914507+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:07.374 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:05.914564+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:07.375 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:06.192419+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e283: 8 total, 8 up, 5 in 2024-02-12T17:00:07.375 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:06.552287+0000 mgr.y (mgr.24481) 398 : cluster [DBG] pgmap v519: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 705 op/s; 12924/108897 objects misplaced (11.868%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:00:07.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:05.711106+0000 osd.0 (osd.0) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:07.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:05.711157+0000 osd.0 (osd.0) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:07.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:05.914507+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:07.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:05.914564+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:07.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:06.192419+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e283: 8 total, 8 up, 5 in 2024-02-12T17:00:07.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:06.552287+0000 mgr.y (mgr.24481) 398 : cluster [DBG] pgmap v519: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 705 op/s; 12924/108897 objects misplaced (11.868%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:00:07.382 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:07.487 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:07.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:07.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:05.711106+0000 osd.0 (osd.0) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:07.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:05.711157+0000 osd.0 (osd.0) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:07.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:05.914507+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:07.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:05.914564+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:07.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:06.192419+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e283: 8 total, 8 up, 5 in 2024-02-12T17:00:07.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:06.552287+0000 mgr.y (mgr.24481) 398 : cluster [DBG] pgmap v519: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 705 op/s; 12924/108897 objects misplaced (11.868%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:00:07.553 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:07.593 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:07.659 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:07.700 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:00:07.700 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:00:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:07.732 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:00:07.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:07.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:07.763 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:00:07.812 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:00:07.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:07.841 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:00:07.871 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:00:07.889 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:07.910 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:00:07.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:00:07.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:00:07.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:00:07.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:00:07.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:00:07.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:00:07.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:00:07.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:00:07.925 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:07.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:08.034 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:08.076 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:08.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:07.641545+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? v1:172.21.15.33:0/2139057160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:08.238 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:08.011916+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.33:0/947721781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:00:08.238 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:08.012400+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25225 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:00:08.340 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T17:00:08.430 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:08.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:08.498 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:07.641545+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? v1:172.21.15.33:0/2139057160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:08.498 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:08.011916+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.33:0/947721781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:00:08.498 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:08.012400+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25225 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:00:08.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:07.641545+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? v1:172.21.15.33:0/2139057160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:08.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:08.011916+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.33:0/947721781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:00:08.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:08.012400+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25225 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:00:08.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:08.575 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:08.876 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:00:09.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:08.222727+0000 mon.a (mon.0) 1191 : audit [INF] from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:00:09.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:08.222810+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e284: 8 total, 8 up, 5 in 2024-02-12T17:00:09.252 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:08.552838+0000 mgr.y (mgr.24481) 399 : cluster [DBG] pgmap v521: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s; 11427/100044 objects misplaced (11.422%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:00:09.337 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:09.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:09.443 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:09.503 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:09.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:08.222727+0000 mon.a (mon.0) 1191 : audit [INF] from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:00:09.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:08.222810+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e284: 8 total, 8 up, 5 in 2024-02-12T17:00:09.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:08.552838+0000 mgr.y (mgr.24481) 399 : cluster [DBG] pgmap v521: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s; 11427/100044 objects misplaced (11.422%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:00:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:09.612 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:09.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:09.612 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:08.222727+0000 mon.a (mon.0) 1191 : audit [INF] from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:00:09.612 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:08.222810+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e284: 8 total, 8 up, 5 in 2024-02-12T17:00:09.612 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:08.552838+0000 mgr.y (mgr.24481) 399 : cluster [DBG] pgmap v521: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s; 11427/100044 objects misplaced (11.422%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:00:09.665 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:00:09.665 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:00:09.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:09.746 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:09.946 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T17:00:09.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:00:09.957 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:09.962 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:09.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:10.067 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:10.117 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:10.218 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:10.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:10.324 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:10.425 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:10.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T17:00:10.461 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:10.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:10.569 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:10.607 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:10.812 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:10.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:10.915 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:10.919 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:10.978 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:11.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:00:11.165 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:00:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:10] "GET /metrics HTTP/1.1" 200 175968 "" "Prometheus/2.33.4" 2024-02-12T17:00:11.223 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:11.293 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:11.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:11.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:11.451 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:11.482 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:11.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:11.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:10.553266+0000 mgr.y (mgr.24481) 400 : cluster [DBG] pgmap v522: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 590 B/s rd, 0 B/s wr, 567 op/s; 11192/97086 objects misplaced (11.528%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:00:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:11.203361+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e285: 8 total, 8 up, 5 in 2024-02-12T17:00:11.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:10.553266+0000 mgr.y (mgr.24481) 400 : cluster [DBG] pgmap v522: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 590 B/s rd, 0 B/s wr, 567 op/s; 11192/97086 objects misplaced (11.528%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:00:11.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:11.203361+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e285: 8 total, 8 up, 5 in 2024-02-12T17:00:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:10.553266+0000 mgr.y (mgr.24481) 400 : cluster [DBG] pgmap v522: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 590 B/s rd, 0 B/s wr, 567 op/s; 11192/97086 objects misplaced (11.528%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:00:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:11.203361+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e285: 8 total, 8 up, 5 in 2024-02-12T17:00:11.781 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:11.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:11.889 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:11.927 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:12.391 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:12.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:12.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:12.586 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:12.614 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:12.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:12.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:12.900 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:00:12.901 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:00:12.933 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:00:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:00:13.111 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:13.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:13.119 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:13.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:13.311 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:13.608 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:13.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:13.630 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T17:00:13.669 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:00:13.671 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:12.553682+0000 mgr.y (mgr.24481) 401 : cluster [DBG] pgmap v524: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 643 B/s rd, 0 B/s wr, 619 op/s; 11192/97086 objects misplaced (11.528%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:00:13.672 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:12.802501+0000 mon.c (mon.2) 160 : audit [INF] from='client.? v1:172.21.15.33:0/1502920854' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:00:13.672 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:12.802974+0000 mon.a (mon.0) 1194 : audit [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:00:13.672 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:12.553682+0000 mgr.y (mgr.24481) 401 : cluster [DBG] pgmap v524: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 643 B/s rd, 0 B/s wr, 619 op/s; 11192/97086 objects misplaced (11.528%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:00:13.672 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:12.802501+0000 mon.c (mon.2) 160 : audit [INF] from='client.? v1:172.21.15.33:0/1502920854' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:00:13.672 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:12.802974+0000 mon.a (mon.0) 1194 : audit [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:00:13.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:13.813 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:13.851 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:13.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:13.859 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:12.553682+0000 mgr.y (mgr.24481) 401 : cluster [DBG] pgmap v524: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 2.0 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 643 B/s rd, 0 B/s wr, 619 op/s; 11192/97086 objects misplaced (11.528%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:00:13.859 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:12.802501+0000 mon.c (mon.2) 160 : audit [INF] from='client.? v1:172.21.15.33:0/1502920854' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:00:13.859 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:12.802974+0000 mon.a (mon.0) 1194 : audit [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:00:13.960 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:13.998 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:00:14.001 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:14.001 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:00:14.132 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:14.242 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:00:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:00:14.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:13.647116+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:14.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:13.656107+0000 mon.a (mon.0) 1196 : audit [INF] from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:00:14.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:13.656175+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e286: 8 total, 8 up, 5 in 2024-02-12T17:00:14.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:13.963924+0000 mgr.y (mgr.24481) 402 : audit [DBG] from='client.35285 v1:172.21.15.33:0/4294492581' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:14.687 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:13.995414+0000 mgr.y (mgr.24481) 403 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:15.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:13.647116+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:15.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:13.656107+0000 mon.a (mon.0) 1196 : audit [INF] from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:00:15.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:13.656175+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e286: 8 total, 8 up, 5 in 2024-02-12T17:00:15.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:13.963924+0000 mgr.y (mgr.24481) 402 : audit [DBG] from='client.35285 v1:172.21.15.33:0/4294492581' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:15.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:13.995414+0000 mgr.y (mgr.24481) 403 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:13.647116+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:13.656107+0000 mon.a (mon.0) 1196 : audit [INF] from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:00:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:13.656175+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e286: 8 total, 8 up, 5 in 2024-02-12T17:00:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:13.963924+0000 mgr.y (mgr.24481) 402 : audit [DBG] from='client.35285 v1:172.21.15.33:0/4294492581' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:13.995414+0000 mgr.y (mgr.24481) 403 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:15.950 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:14.554248+0000 mgr.y (mgr.24481) 404 : cluster [DBG] pgmap v526: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.7 GiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 9480/85179 objects misplaced (11.130%); 5.5 MiB/s, 88 objects/s recovering 2024-02-12T17:00:15.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:15.257154+0000 osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:15.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:15.257199+0000 osd.3 (osd.3) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:14.554248+0000 mgr.y (mgr.24481) 404 : cluster [DBG] pgmap v526: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.7 GiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 9480/85179 objects misplaced (11.130%); 5.5 MiB/s, 88 objects/s recovering 2024-02-12T17:00:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:15.257154+0000 osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:15.257199+0000 osd.3 (osd.3) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:16.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:14.554248+0000 mgr.y (mgr.24481) 404 : cluster [DBG] pgmap v526: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.7 GiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 9480/85179 objects misplaced (11.130%); 5.5 MiB/s, 88 objects/s recovering 2024-02-12T17:00:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:15.257154+0000 osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:15.257199+0000 osd.3 (osd.3) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:16.279 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:00:16.279 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:00:16.659 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:16.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:16.767 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:16.814 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:16.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:15.696834+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e287: 8 total, 8 up, 5 in 2024-02-12T17:00:17.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:15.696834+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e287: 8 total, 8 up, 5 in 2024-02-12T17:00:17.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:15.696834+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e287: 8 total, 8 up, 5 in 2024-02-12T17:00:17.041 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T17:00:17.530 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:17.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:16.554647+0000 mgr.y (mgr.24481) 405 : cluster [DBG] pgmap v528: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.7 GiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 9480/85179 objects misplaced (11.130%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:00:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:16.554647+0000 mgr.y (mgr.24481) 405 : cluster [DBG] pgmap v528: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.7 GiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 9480/85179 objects misplaced (11.130%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:00:18.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:16.554647+0000 mgr.y (mgr.24481) 405 : cluster [DBG] pgmap v528: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.7 GiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 9480/85179 objects misplaced (11.130%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:00:18.373 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:18.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:18.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:18.561 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:19.068 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:00:19.069 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:00:19.069 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:00:19.081 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:00:19.121 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:00:19.148 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:00:19.181 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:00:19.210 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:00:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:00:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:00:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:00:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:00:19.212 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:00:19.213 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:00:19.213 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:00:19.213 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:00:19.332 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:19.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:19.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:19.542 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:19.585 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:19.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:19.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:19.604 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-02-12T17:00:13.658740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1848121020,"num_objects":28393,"num_object_clones":0,"num_object_copies":85179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9480,"num_objects_unfound":0,"num_objects_dirty":28393,"num_whiteouts":0,"num_read":1282,"num_read_kb":1034,"num_write":195825,"num_write_kb":7145962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38285,"num_bytes_recovered":2490835055,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58878,"ondisk_log_size":58878,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":6403112,"kb_used_data":4888192,"kb_used_omap":0,"kb_used_meta":1514752,"kb_avail":462322648,"statfs":{"total":479975178240,"available":473418391552,"internally_reserved":0,"allocated":5005508608,"data_stored":5002091872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1551106048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-518127616,"num_objects":-7906,"num_object_clones":0,"num_object_copies":-23718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3444,"num_objects_unfound":0,"num_objects_dirty":-7906,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":7906,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":36634624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.106715"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":182,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.271424+0000","last_change":"2024-02-12T16:59:09.246682+0000","last_active":"2024-02-12T17:00:08.271424+0000","last_peered":"2024-02-12T17:00:08.271424+0000","last_clean":"2024-02-12T17:00:08.271424+0000","last_became_active":"2024-02-12T16:59:09.246207+0000","last_became_peered":"2024-02-12T16:59:09.246207+0000","last_unstale":"2024-02-12T17:00:08.271424+0000","last_undegraded":"2024-02-12T17:00:08.271424+0000","last_fullsized":"2024-02-12T17:00:08.271424+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":249,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229124+0000","last_change":"2024-02-12T16:56:35.853405+0000","last_active":"2024-02-12T17:00:08.229124+0000","last_peered":"2024-02-12T17:00:08.229124+0000","last_clean":"2024-02-12T17:00:08.229124+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T17:00:08.229124+0000","last_undegraded":"2024-02-12T17:00:08.229124+0000","last_fullsized":"2024-02-12T17:00:08.229124+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.853330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.853330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":235,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225375+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T17:00:08.225375+0000","last_peered":"2024-02-12T17:00:08.225375+0000","last_clean":"2024-02-12T17:00:08.225375+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T17:00:08.225375+0000","last_undegraded":"2024-02-12T17:00:08.225375+0000","last_fullsized":"2024-02-12T17:00:08.225375+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":213,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240302+0000","last_change":"2024-02-12T16:57:47.192069+0000","last_active":"2024-02-12T17:00:08.240302+0000","last_peered":"2024-02-12T17:00:08.240302+0000","last_clean":"2024-02-12T17:00:08.240302+0000","last_became_active":"2024-02-12T16:57:47.188886+0000","last_became_peered":"2024-02-12T16:57:47.188886+0000","last_unstale":"2024-02-12T17:00:08.240302+0000","last_undegraded":"2024-02-12T17:00:08.240302+0000","last_fullsized":"2024-02-12T17:00:08.240302+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":164,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229159+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T17:00:08.229159+0000","last_peered":"2024-02-12T17:00:08.229159+0000","last_clean":"2024-02-12T17:00:08.229159+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T17:00:08.229159+0000","last_undegraded":"2024-02-12T17:00:08.229159+0000","last_fullsized":"2024-02-12T17:00:08.229159+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"277'15","reported_seq":318,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240419+0000","last_change":"2024-02-12T16:59:09.351570+0000","last_active":"2024-02-12T17:00:08.240419+0000","last_peered":"2024-02-12T17:00:08.240419+0000","last_clean":"2024-02-12T17:00:08.240419+0000","last_became_active":"2024-02-12T16:59:09.243960+0000","last_became_peered":"2024-02-12T16:59:09.243960+0000","last_unstale":"2024-02-12T17:00:08.240419+0000","last_undegraded":"2024-02-12T17:00:08.240419+0000","last_fullsized":"2024-02-12T17:00:08.240419+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:31.812644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.812644+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":263,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.237500+0000","last_change":"2024-02-12T16:57:47.176812+0000","last_active":"2024-02-12T17:00:08.237500+0000","last_peered":"2024-02-12T17:00:08.237500+0000","last_clean":"2024-02-12T17:00:08.237500+0000","last_became_active":"2024-02-12T16:57:47.176448+0000","last_became_peered":"2024-02-12T16:57:47.176448+0000","last_unstale":"2024-02-12T17:00:08.237500+0000","last_undegraded":"2024-02-12T17:00:08.237500+0000","last_fullsized":"2024-02-12T17:00:08.237500+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:42.174992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:42.174992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":266,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.263551+0000","last_change":"2024-02-12T16:59:09.248489+0000","last_active":"2024-02-12T17:00:08.263551+0000","last_peered":"2024-02-12T17:00:08.263551+0000","last_clean":"2024-02-12T17:00:08.263551+0000","last_became_active":"2024-02-12T16:59:09.248262+0000","last_became_peered":"2024-02-12T16:59:09.248262+0000","last_unstale":"2024-02-12T17:00:08.263551+0000","last_undegraded":"2024-02-12T17:00:08.263551+0000","last_fullsized":"2024-02-12T17:00:08.263551+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":123,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226256+0000","last_change":"2024-02-12T16:57:47.319602+0000","last_active":"2024-02-12T17:00:08.226256+0000","last_peered":"2024-02-12T17:00:08.226256+0000","last_clean":"2024-02-12T17:00:08.226256+0000","last_became_active":"2024-02-12T16:57:47.186061+0000","last_became_peered":"2024-02-12T16:57:47.186061+0000","last_unstale":"2024-02-12T17:00:08.226256+0000","last_undegraded":"2024-02-12T17:00:08.226256+0000","last_fullsized":"2024-02-12T17:00:08.226256+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":265,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.230290+0000","last_change":"2024-02-12T16:58:16.945996+0000","last_active":"2024-02-12T17:00:08.230290+0000","last_peered":"2024-02-12T17:00:08.230290+0000","last_clean":"2024-02-12T17:00:08.230290+0000","last_became_active":"2024-02-12T16:57:47.185903+0000","last_became_peered":"2024-02-12T16:57:47.185903+0000","last_unstale":"2024-02-12T17:00:08.230290+0000","last_undegraded":"2024-02-12T17:00:08.230290+0000","last_fullsized":"2024-02-12T17:00:08.230290+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":249,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.265937+0000","last_change":"2024-02-12T16:56:35.897211+0000","last_active":"2024-02-12T17:00:08.265937+0000","last_peered":"2024-02-12T17:00:08.265937+0000","last_clean":"2024-02-12T17:00:08.265937+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:00:08.265937+0000","last_undegraded":"2024-02-12T17:00:08.265937+0000","last_fullsized":"2024-02-12T17:00:08.265937+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.897088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.897088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":211,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229405+0000","last_change":"2024-02-12T16:57:18.865914+0000","last_active":"2024-02-12T17:00:08.229405+0000","last_peered":"2024-02-12T17:00:08.229405+0000","last_clean":"2024-02-12T17:00:08.229405+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T17:00:08.229405+0000","last_undegraded":"2024-02-12T17:00:08.229405+0000","last_fullsized":"2024-02-12T17:00:08.229405+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:18.865807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.865807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":128,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.242514+0000","last_change":"2024-02-12T16:59:09.260070+0000","last_active":"2024-02-12T17:00:08.242514+0000","last_peered":"2024-02-12T17:00:08.242514+0000","last_clean":"2024-02-12T17:00:08.242514+0000","last_became_active":"2024-02-12T16:59:09.259293+0000","last_became_peered":"2024-02-12T16:59:09.259293+0000","last_unstale":"2024-02-12T17:00:08.242514+0000","last_undegraded":"2024-02-12T17:00:08.242514+0000","last_fullsized":"2024-02-12T17:00:08.242514+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":173,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.271676+0000","last_change":"2024-02-12T16:59:09.276561+0000","last_active":"2024-02-12T17:00:08.271676+0000","last_peered":"2024-02-12T17:00:08.271676+0000","last_clean":"2024-02-12T17:00:08.271676+0000","last_became_active":"2024-02-12T16:59:09.272423+0000","last_became_peered":"2024-02-12T16:59:09.272423+0000","last_unstale":"2024-02-12T17:00:08.271676+0000","last_undegraded":"2024-02-12T17:00:08.271676+0000","last_fullsized":"2024-02-12T17:00:08.271676+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":166,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240169+0000","last_change":"2024-02-12T16:59:09.890505+0000","last_active":"2024-02-12T17:00:08.240169+0000","last_peered":"2024-02-12T17:00:08.240169+0000","last_clean":"2024-02-12T17:00:08.240169+0000","last_became_active":"2024-02-12T16:59:09.249268+0000","last_became_peered":"2024-02-12T16:59:09.249268+0000","last_unstale":"2024-02-12T17:00:08.240169+0000","last_undegraded":"2024-02-12T17:00:08.240169+0000","last_fullsized":"2024-02-12T17:00:08.240169+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:56:45.033441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.033441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":259,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.246060+0000","last_change":"2024-02-12T16:59:09.270880+0000","last_active":"2024-02-12T17:00:08.246060+0000","last_peered":"2024-02-12T17:00:08.246060+0000","last_clean":"2024-02-12T17:00:08.246060+0000","last_became_active":"2024-02-12T16:59:09.270574+0000","last_became_peered":"2024-02-12T16:59:09.270574+0000","last_unstale":"2024-02-12T17:00:08.246060+0000","last_undegraded":"2024-02-12T17:00:08.246060+0000","last_fullsized":"2024-02-12T17:00:08.246060+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":39,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.247237+0000","last_change":"2024-02-12T16:59:09.248658+0000","last_active":"2024-02-12T17:00:08.247237+0000","last_peered":"2024-02-12T17:00:08.247237+0000","last_clean":"2024-02-12T17:00:08.247237+0000","last_became_active":"2024-02-12T16:59:09.248342+0000","last_became_peered":"2024-02-12T16:59:09.248342+0000","last_unstale":"2024-02-12T17:00:08.247237+0000","last_undegraded":"2024-02-12T17:00:08.247237+0000","last_fullsized":"2024-02-12T17:00:08.247237+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":351,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226660+0000","last_change":"2024-02-12T16:57:47.210170+0000","last_active":"2024-02-12T17:00:08.226660+0000","last_peered":"2024-02-12T17:00:08.226660+0000","last_clean":"2024-02-12T17:00:08.226660+0000","last_became_active":"2024-02-12T16:57:47.178386+0000","last_became_peered":"2024-02-12T16:57:47.178386+0000","last_unstale":"2024-02-12T17:00:08.226660+0000","last_undegraded":"2024-02-12T17:00:08.226660+0000","last_fullsized":"2024-02-12T17:00:08.226660+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:57:24.750245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.750245+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":89,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.241382+0000","last_change":"2024-02-12T16:57:47.179029+0000","last_active":"2024-02-12T17:00:08.241382+0000","last_peered":"2024-02-12T17:00:08.241382+0000","last_clean":"2024-02-12T17:00:08.241382+0000","last_became_active":"2024-02-12T16:57:47.178240+0000","last_became_peered":"2024-02-12T16:57:47.178240+0000","last_unstale":"2024-02-12T17:00:08.241382+0000","last_undegraded":"2024-02-12T17:00:08.241382+0000","last_fullsized":"2024-02-12T17:00:08.241382+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":42,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228770+0000","last_change":"2024-02-12T16:59:09.242061+0000","last_active":"2024-02-12T17:00:08.228770+0000","last_peered":"2024-02-12T17:00:08.228770+0000","last_clean":"2024-02-12T17:00:08.228770+0000","last_became_active":"2024-02-12T16:59:09.241719+0000","last_became_peered":"2024-02-12T16:59:09.241719+0000","last_unstale":"2024-02-12T17:00:08.228770+0000","last_undegraded":"2024-02-12T17:00:08.228770+0000","last_fullsized":"2024-02-12T17:00:08.228770+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":94,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.242684+0000","last_change":"2024-02-12T16:59:09.533676+0000","last_active":"2024-02-12T17:00:08.242684+0000","last_peered":"2024-02-12T17:00:08.242684+0000","last_clean":"2024-02-12T17:00:08.242684+0000","last_became_active":"2024-02-12T16:59:09.259731+0000","last_became_peered":"2024-02-12T16:59:09.259731+0000","last_unstale":"2024-02-12T17:00:08.242684+0000","last_undegraded":"2024-02-12T17:00:08.242684+0000","last_fullsized":"2024-02-12T17:00:08.242684+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":245,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.266428+0000","last_change":"2024-02-12T16:57:37.670804+0000","last_active":"2024-02-12T17:00:08.266428+0000","last_peered":"2024-02-12T17:00:08.266428+0000","last_clean":"2024-02-12T17:00:08.266428+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T17:00:08.266428+0000","last_undegraded":"2024-02-12T17:00:08.266428+0000","last_fullsized":"2024-02-12T17:00:08.266428+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:57:37.670713+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.670713+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":251,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.235027+0000","last_change":"2024-02-12T16:57:26.790389+0000","last_active":"2024-02-12T17:00:08.235027+0000","last_peered":"2024-02-12T17:00:08.235027+0000","last_clean":"2024-02-12T17:00:08.235027+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T17:00:08.235027+0000","last_undegraded":"2024-02-12T17:00:08.235027+0000","last_fullsized":"2024-02-12T17:00:08.235027+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":319,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229265+0000","last_change":"2024-02-12T16:57:47.209446+0000","last_active":"2024-02-12T17:00:08.229265+0000","last_peered":"2024-02-12T17:00:08.229265+0000","last_clean":"2024-02-12T17:00:08.229265+0000","last_became_active":"2024-02-12T16:57:47.189652+0000","last_became_peered":"2024-02-12T16:57:47.189652+0000","last_unstale":"2024-02-12T17:00:08.229265+0000","last_undegraded":"2024-02-12T17:00:08.229265+0000","last_fullsized":"2024-02-12T17:00:08.229265+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":173,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228249+0000","last_change":"2024-02-12T16:59:09.264851+0000","last_active":"2024-02-12T17:00:08.228249+0000","last_peered":"2024-02-12T17:00:08.228249+0000","last_clean":"2024-02-12T17:00:08.228249+0000","last_became_active":"2024-02-12T16:59:09.264513+0000","last_became_peered":"2024-02-12T16:59:09.264513+0000","last_unstale":"2024-02-12T17:00:08.228249+0000","last_undegraded":"2024-02-12T17:00:08.228249+0000","last_fullsized":"2024-02-12T17:00:08.228249+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":249,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261743+0000","last_change":"2024-02-12T16:59:09.270847+0000","last_active":"2024-02-12T17:00:08.261743+0000","last_peered":"2024-02-12T17:00:08.261743+0000","last_clean":"2024-02-12T17:00:08.261743+0000","last_became_active":"2024-02-12T16:59:09.270516+0000","last_became_peered":"2024-02-12T16:59:09.270516+0000","last_unstale":"2024-02-12T17:00:08.261743+0000","last_undegraded":"2024-02-12T17:00:08.261743+0000","last_fullsized":"2024-02-12T17:00:08.261743+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":39,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.242587+0000","last_change":"2024-02-12T16:59:09.260879+0000","last_active":"2024-02-12T17:00:08.242587+0000","last_peered":"2024-02-12T17:00:08.242587+0000","last_clean":"2024-02-12T17:00:08.242587+0000","last_became_active":"2024-02-12T16:59:09.259058+0000","last_became_peered":"2024-02-12T16:59:09.259058+0000","last_unstale":"2024-02-12T17:00:08.242587+0000","last_undegraded":"2024-02-12T17:00:08.242587+0000","last_fullsized":"2024-02-12T17:00:08.242587+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":240,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.260835+0000","last_change":"2024-02-12T16:59:10.061885+0000","last_active":"2024-02-12T17:00:08.260835+0000","last_peered":"2024-02-12T17:00:08.260835+0000","last_clean":"2024-02-12T17:00:08.260835+0000","last_became_active":"2024-02-12T16:59:09.267628+0000","last_became_peered":"2024-02-12T16:59:09.267628+0000","last_unstale":"2024-02-12T17:00:08.260835+0000","last_undegraded":"2024-02-12T17:00:08.260835+0000","last_fullsized":"2024-02-12T17:00:08.260835+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":243,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261476+0000","last_change":"2024-02-12T16:57:28.846844+0000","last_active":"2024-02-12T17:00:08.261476+0000","last_peered":"2024-02-12T17:00:08.261476+0000","last_clean":"2024-02-12T17:00:08.261476+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T17:00:08.261476+0000","last_undegraded":"2024-02-12T17:00:08.261476+0000","last_fullsized":"2024-02-12T17:00:08.261476+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.241080+0000","last_change":"2024-02-12T16:59:09.247624+0000","last_active":"2024-02-12T17:00:08.241080+0000","last_peered":"2024-02-12T17:00:08.241080+0000","last_clean":"2024-02-12T17:00:08.241080+0000","last_became_active":"2024-02-12T16:59:09.247346+0000","last_became_peered":"2024-02-12T16:59:09.247346+0000","last_unstale":"2024-02-12T17:00:08.241080+0000","last_undegraded":"2024-02-12T17:00:08.241080+0000","last_fullsized":"2024-02-12T17:00:08.241080+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.225996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":260,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240898+0000","last_change":"2024-02-12T16:59:09.265096+0000","last_active":"2024-02-12T17:00:08.240898+0000","last_peered":"2024-02-12T17:00:08.240898+0000","last_clean":"2024-02-12T17:00:08.240898+0000","last_became_active":"2024-02-12T16:59:09.264558+0000","last_became_peered":"2024-02-12T16:59:09.264558+0000","last_unstale":"2024-02-12T17:00:08.240898+0000","last_undegraded":"2024-02-12T17:00:08.240898+0000","last_fullsized":"2024-02-12T17:00:08.240898+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"277'8","reported_seq":72,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.234703+0000","last_change":"2024-02-12T16:59:09.961214+0000","last_active":"2024-02-12T17:00:08.234703+0000","last_peered":"2024-02-12T17:00:08.234703+0000","last_clean":"2024-02-12T17:00:08.234703+0000","last_became_active":"2024-02-12T16:59:09.280196+0000","last_became_peered":"2024-02-12T16:59:09.280196+0000","last_unstale":"2024-02-12T17:00:08.234703+0000","last_undegraded":"2024-02-12T17:00:08.234703+0000","last_fullsized":"2024-02-12T17:00:08.234703+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":164,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228689+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T17:00:08.228689+0000","last_peered":"2024-02-12T17:00:08.228689+0000","last_clean":"2024-02-12T17:00:08.228689+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T17:00:08.228689+0000","last_undegraded":"2024-02-12T17:00:08.228689+0000","last_fullsized":"2024-02-12T17:00:08.228689+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":198,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.241403+0000","last_change":"2024-02-12T16:59:09.263555+0000","last_active":"2024-02-12T17:00:08.241403+0000","last_peered":"2024-02-12T17:00:08.241403+0000","last_clean":"2024-02-12T17:00:08.241403+0000","last_became_active":"2024-02-12T16:59:09.262674+0000","last_became_peered":"2024-02-12T16:59:09.262674+0000","last_unstale":"2024-02-12T17:00:08.241403+0000","last_undegraded":"2024-02-12T17:00:08.241403+0000","last_fullsized":"2024-02-12T17:00:08.241403+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":98,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.241186+0000","last_change":"2024-02-12T16:59:09.258970+0000","last_active":"2024-02-12T17:00:08.241186+0000","last_peered":"2024-02-12T17:00:08.241186+0000","last_clean":"2024-02-12T17:00:08.241186+0000","last_became_active":"2024-02-12T16:59:09.257167+0000","last_became_peered":"2024-02-12T16:59:09.257167+0000","last_unstale":"2024-02-12T17:00:08.241186+0000","last_undegraded":"2024-02-12T17:00:08.241186+0000","last_fullsized":"2024-02-12T17:00:08.241186+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":330,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.244129+0000","last_change":"2024-02-12T16:59:09.950284+0000","last_active":"2024-02-12T17:00:08.244129+0000","last_peered":"2024-02-12T17:00:08.244129+0000","last_clean":"2024-02-12T17:00:08.244129+0000","last_became_active":"2024-02-12T16:59:09.274991+0000","last_became_peered":"2024-02-12T16:59:09.274991+0000","last_unstale":"2024-02-12T17:00:08.244129+0000","last_undegraded":"2024-02-12T17:00:08.244129+0000","last_fullsized":"2024-02-12T17:00:08.244129+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:37.895287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.895287+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":256,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.230004+0000","last_change":"2024-02-12T16:59:09.269463+0000","last_active":"2024-02-12T17:00:08.230004+0000","last_peered":"2024-02-12T17:00:08.230004+0000","last_clean":"2024-02-12T17:00:08.230004+0000","last_became_active":"2024-02-12T16:59:09.269260+0000","last_became_peered":"2024-02-12T16:59:09.269260+0000","last_unstale":"2024-02-12T17:00:08.230004+0000","last_undegraded":"2024-02-12T17:00:08.230004+0000","last_fullsized":"2024-02-12T17:00:08.230004+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":189,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228395+0000","last_change":"2024-02-12T16:57:47.189653+0000","last_active":"2024-02-12T17:00:08.228395+0000","last_peered":"2024-02-12T17:00:08.228395+0000","last_clean":"2024-02-12T17:00:08.228395+0000","last_became_active":"2024-02-12T16:57:47.188909+0000","last_became_peered":"2024-02-12T16:57:47.188909+0000","last_unstale":"2024-02-12T17:00:08.228395+0000","last_undegraded":"2024-02-12T17:00:08.228395+0000","last_fullsized":"2024-02-12T17:00:08.228395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":137,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.237573+0000","last_change":"2024-02-12T16:59:09.260492+0000","last_active":"2024-02-12T17:00:08.237573+0000","last_peered":"2024-02-12T17:00:08.237573+0000","last_clean":"2024-02-12T17:00:08.237573+0000","last_became_active":"2024-02-12T16:59:09.260045+0000","last_became_peered":"2024-02-12T16:59:09.260045+0000","last_unstale":"2024-02-12T17:00:08.237573+0000","last_undegraded":"2024-02-12T17:00:08.237573+0000","last_fullsized":"2024-02-12T17:00:08.237573+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:39.248562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.248562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":104,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225654+0000","last_change":"2024-02-12T16:59:09.288225+0000","last_active":"2024-02-12T17:00:08.225654+0000","last_peered":"2024-02-12T17:00:08.225654+0000","last_clean":"2024-02-12T17:00:08.225654+0000","last_became_active":"2024-02-12T16:59:09.253231+0000","last_became_peered":"2024-02-12T16:59:09.253231+0000","last_unstale":"2024-02-12T17:00:08.225654+0000","last_undegraded":"2024-02-12T17:00:08.225654+0000","last_fullsized":"2024-02-12T17:00:08.225654+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":209,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228140+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T17:00:08.228140+0000","last_peered":"2024-02-12T17:00:08.228140+0000","last_clean":"2024-02-12T17:00:08.228140+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T17:00:08.228140+0000","last_undegraded":"2024-02-12T17:00:08.228140+0000","last_fullsized":"2024-02-12T17:00:08.228140+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":288,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.251522+0000","last_change":"2024-02-12T16:57:47.312130+0000","last_active":"2024-02-12T17:00:11.251522+0000","last_peered":"2024-02-12T17:00:11.251522+0000","last_clean":"2024-02-12T17:00:11.251522+0000","last_became_active":"2024-02-12T16:57:47.185109+0000","last_became_peered":"2024-02-12T16:57:47.185109+0000","last_unstale":"2024-02-12T17:00:11.251522+0000","last_undegraded":"2024-02-12T17:00:11.251522+0000","last_fullsized":"2024-02-12T17:00:11.251522+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":253,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228590+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T17:00:08.228590+0000","last_peered":"2024-02-12T17:00:08.228590+0000","last_clean":"2024-02-12T17:00:08.228590+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T17:00:08.228590+0000","last_undegraded":"2024-02-12T17:00:08.228590+0000","last_fullsized":"2024-02-12T17:00:08.228590+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":307,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.241138+0000","last_change":"2024-02-12T16:59:09.524334+0000","last_active":"2024-02-12T17:00:08.241138+0000","last_peered":"2024-02-12T17:00:08.241138+0000","last_clean":"2024-02-12T17:00:08.241138+0000","last_became_active":"2024-02-12T16:59:09.258515+0000","last_became_peered":"2024-02-12T16:59:09.258515+0000","last_unstale":"2024-02-12T17:00:08.241138+0000","last_undegraded":"2024-02-12T17:00:08.241138+0000","last_fullsized":"2024-02-12T17:00:08.241138+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":252,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229013+0000","last_change":"2024-02-12T16:57:25.835979+0000","last_active":"2024-02-12T17:00:08.229013+0000","last_peered":"2024-02-12T17:00:08.229013+0000","last_clean":"2024-02-12T17:00:08.229013+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T17:00:08.229013+0000","last_undegraded":"2024-02-12T17:00:08.229013+0000","last_fullsized":"2024-02-12T17:00:08.229013+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.835883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.835883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"208'9","reported_seq":392,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.219599+0000","last_change":"2024-02-12T16:57:47.225402+0000","last_active":"2024-02-12T17:00:11.219599+0000","last_peered":"2024-02-12T17:00:11.219599+0000","last_clean":"2024-02-12T17:00:11.219599+0000","last_became_active":"2024-02-12T16:57:47.178688+0000","last_became_peered":"2024-02-12T16:57:47.178688+0000","last_unstale":"2024-02-12T17:00:11.219599+0000","last_undegraded":"2024-02-12T17:00:11.219599+0000","last_fullsized":"2024-02-12T17:00:11.219599+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":255,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240451+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T17:00:08.240451+0000","last_peered":"2024-02-12T17:00:08.240451+0000","last_clean":"2024-02-12T17:00:08.240451+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T17:00:08.240451+0000","last_undegraded":"2024-02-12T17:00:08.240451+0000","last_fullsized":"2024-02-12T17:00:08.240451+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":143,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.266009+0000","last_change":"2024-02-12T16:57:48.058146+0000","last_active":"2024-02-12T17:00:08.266009+0000","last_peered":"2024-02-12T17:00:08.266009+0000","last_clean":"2024-02-12T17:00:08.266009+0000","last_became_active":"2024-02-12T16:57:47.199140+0000","last_became_peered":"2024-02-12T16:57:47.199140+0000","last_unstale":"2024-02-12T17:00:08.266009+0000","last_undegraded":"2024-02-12T17:00:08.266009+0000","last_fullsized":"2024-02-12T17:00:08.266009+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":251,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.263517+0000","last_change":"2024-02-12T16:59:09.269750+0000","last_active":"2024-02-12T17:00:08.263517+0000","last_peered":"2024-02-12T17:00:08.263517+0000","last_clean":"2024-02-12T17:00:08.263517+0000","last_became_active":"2024-02-12T16:59:09.269414+0000","last_became_peered":"2024-02-12T16:59:09.269414+0000","last_unstale":"2024-02-12T17:00:08.263517+0000","last_undegraded":"2024-02-12T17:00:08.263517+0000","last_fullsized":"2024-02-12T17:00:08.263517+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":212,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.243272+0000","last_change":"2024-02-12T16:59:09.243989+0000","last_active":"2024-02-12T17:00:08.243272+0000","last_peered":"2024-02-12T17:00:08.243272+0000","last_clean":"2024-02-12T17:00:08.243272+0000","last_became_active":"2024-02-12T16:59:09.243603+0000","last_became_peered":"2024-02-12T16:59:09.243603+0000","last_unstale":"2024-02-12T17:00:08.243272+0000","last_undegraded":"2024-02-12T17:00:08.243272+0000","last_fullsized":"2024-02-12T17:00:08.243272+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.218031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.218031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":231,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.264851+0000","last_change":"2024-02-12T16:59:09.519418+0000","last_active":"2024-02-12T17:00:08.264851+0000","last_peered":"2024-02-12T17:00:08.264851+0000","last_clean":"2024-02-12T17:00:08.264851+0000","last_became_active":"2024-02-12T16:59:09.242413+0000","last_became_peered":"2024-02-12T16:59:09.242413+0000","last_unstale":"2024-02-12T17:00:08.264851+0000","last_undegraded":"2024-02-12T17:00:08.264851+0000","last_fullsized":"2024-02-12T17:00:08.264851+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":245,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226507+0000","last_change":"2024-02-12T16:56:38.699688+0000","last_active":"2024-02-12T17:00:08.226507+0000","last_peered":"2024-02-12T17:00:08.226507+0000","last_clean":"2024-02-12T17:00:08.226507+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T17:00:08.226507+0000","last_undegraded":"2024-02-12T17:00:08.226507+0000","last_fullsized":"2024-02-12T17:00:08.226507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.699602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":260,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225327+0000","last_change":"2024-02-12T16:57:47.171068+0000","last_active":"2024-02-12T17:00:08.225327+0000","last_peered":"2024-02-12T17:00:08.225327+0000","last_clean":"2024-02-12T17:00:08.225327+0000","last_became_active":"2024-02-12T16:57:47.170667+0000","last_became_peered":"2024-02-12T16:57:47.170667+0000","last_unstale":"2024-02-12T17:00:08.225327+0000","last_undegraded":"2024-02-12T17:00:08.225327+0000","last_fullsized":"2024-02-12T17:00:08.225327+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":290,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229109+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T17:00:08.229109+0000","last_peered":"2024-02-12T17:00:08.229109+0000","last_clean":"2024-02-12T17:00:08.229109+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T17:00:08.229109+0000","last_undegraded":"2024-02-12T17:00:08.229109+0000","last_fullsized":"2024-02-12T17:00:08.229109+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"285'12123","reported_seq":13585,"reported_epoch":285,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:13.432302+0000","last_change":"2024-02-12T16:59:10.444155+0000","last_active":"2024-02-12T17:00:13.432302+0000","last_peered":"2024-02-12T17:00:13.432302+0000","last_clean":"2024-02-12T16:59:08.181528+0000","last_became_active":"2024-02-12T16:59:10.326952+0000","last_became_peered":"2024-02-12T16:59:10.326952+0000","last_unstale":"2024-02-12T17:00:13.432302+0000","last_undegraded":"2024-02-12T17:00:13.432302+0000","last_fullsized":"2024-02-12T17:00:13.432302+0000","mapping_epoch":258,"log_start":"251'8322","ondisk_log_start":"251'8322","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3801,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":119078912,"num_objects":1817,"num_object_clones":0,"num_object_copies":5451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1435,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":446080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":668,"num_bytes_recovered":43778048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":1817}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":177,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.271599+0000","last_change":"2024-02-12T16:59:09.273017+0000","last_active":"2024-02-12T17:00:08.271599+0000","last_peered":"2024-02-12T17:00:08.271599+0000","last_clean":"2024-02-12T17:00:08.271599+0000","last_became_active":"2024-02-12T16:59:09.272706+0000","last_became_peered":"2024-02-12T16:59:09.272706+0000","last_unstale":"2024-02-12T17:00:08.271599+0000","last_undegraded":"2024-02-12T17:00:08.271599+0000","last_fullsized":"2024-02-12T17:00:08.271599+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":276,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.255234+0000","last_change":"2024-02-12T16:57:22.679795+0000","last_active":"2024-02-12T17:00:08.255234+0000","last_peered":"2024-02-12T17:00:08.255234+0000","last_clean":"2024-02-12T17:00:08.255234+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T17:00:08.255234+0000","last_undegraded":"2024-02-12T17:00:08.255234+0000","last_fullsized":"2024-02-12T17:00:08.255234+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:57:22.679705+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.679705+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":155,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.227992+0000","last_change":"2024-02-12T16:59:09.952265+0000","last_active":"2024-02-12T17:00:08.227992+0000","last_peered":"2024-02-12T17:00:08.227992+0000","last_clean":"2024-02-12T17:00:08.227992+0000","last_became_active":"2024-02-12T16:59:09.264482+0000","last_became_peered":"2024-02-12T16:59:09.264482+0000","last_unstale":"2024-02-12T17:00:08.227992+0000","last_undegraded":"2024-02-12T17:00:08.227992+0000","last_fullsized":"2024-02-12T17:00:08.227992+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:22.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.764984+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":274,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225340+0000","last_change":"2024-02-12T16:58:10.280441+0000","last_active":"2024-02-12T17:00:08.225340+0000","last_peered":"2024-02-12T17:00:08.225340+0000","last_clean":"2024-02-12T17:00:08.225340+0000","last_became_active":"2024-02-12T16:58:10.280137+0000","last_became_peered":"2024-02-12T16:58:10.280137+0000","last_unstale":"2024-02-12T17:00:08.225340+0000","last_undegraded":"2024-02-12T17:00:08.225340+0000","last_fullsized":"2024-02-12T17:00:08.225340+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.647627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.647627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"285'12207","reported_seq":12324,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.513659+0000","last_change":"2024-02-12T16:56:36.446431+0000","last_active":"2024-02-12T17:00:13.513659+0000","last_peered":"2024-02-12T17:00:13.513659+0000","last_clean":"2024-02-12T17:00:13.513659+0000","last_became_active":"2024-02-12T16:56:36.446107+0000","last_became_peered":"2024-02-12T16:56:36.446107+0000","last_unstale":"2024-02-12T17:00:13.513659+0000","last_undegraded":"2024-02-12T17:00:13.513659+0000","last_fullsized":"2024-02-12T17:00:13.513659+0000","mapping_epoch":176,"log_start":"253'8626","ondisk_log_start":"253'8626","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3581,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":113180672,"num_objects":1727,"num_object_clones":0,"num_object_copies":5181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":445888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":252,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.230034+0000","last_change":"2024-02-12T16:57:29.934046+0000","last_active":"2024-02-12T17:00:08.230034+0000","last_peered":"2024-02-12T17:00:08.230034+0000","last_clean":"2024-02-12T17:00:08.230034+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T17:00:08.230034+0000","last_undegraded":"2024-02-12T17:00:08.230034+0000","last_fullsized":"2024-02-12T17:00:08.230034+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229978+0000","last_change":"2024-02-12T16:57:47.187057+0000","last_active":"2024-02-12T17:00:08.229978+0000","last_peered":"2024-02-12T17:00:08.229978+0000","last_clean":"2024-02-12T17:00:08.229978+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:00:08.229978+0000","last_undegraded":"2024-02-12T17:00:08.229978+0000","last_fullsized":"2024-02-12T17:00:08.229978+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.009094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.009094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":143,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.241483+0000","last_change":"2024-02-12T16:57:48.006795+0000","last_active":"2024-02-12T17:00:08.241483+0000","last_peered":"2024-02-12T17:00:08.241483+0000","last_clean":"2024-02-12T17:00:08.241483+0000","last_became_active":"2024-02-12T16:57:47.202531+0000","last_became_peered":"2024-02-12T16:57:47.202531+0000","last_unstale":"2024-02-12T17:00:08.241483+0000","last_undegraded":"2024-02-12T17:00:08.241483+0000","last_fullsized":"2024-02-12T17:00:08.241483+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"261'9","reported_seq":388,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.217568+0000","last_change":"2024-02-12T16:59:09.377161+0000","last_active":"2024-02-12T17:00:11.217568+0000","last_peered":"2024-02-12T17:00:11.217568+0000","last_clean":"2024-02-12T17:00:11.217568+0000","last_became_active":"2024-02-12T16:59:09.270667+0000","last_became_peered":"2024-02-12T16:59:09.270667+0000","last_unstale":"2024-02-12T17:00:11.217568+0000","last_undegraded":"2024-02-12T17:00:11.217568+0000","last_fullsized":"2024-02-12T17:00:11.217568+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"285'12354","reported_seq":12471,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.474694+0000","last_change":"2024-02-12T16:56:36.788519+0000","last_active":"2024-02-12T17:00:13.474694+0000","last_peered":"2024-02-12T17:00:13.474694+0000","last_clean":"2024-02-12T17:00:13.474694+0000","last_became_active":"2024-02-12T16:56:36.788101+0000","last_became_peered":"2024-02-12T16:56:36.788101+0000","last_unstale":"2024-02-12T17:00:13.474694+0000","last_undegraded":"2024-02-12T17:00:13.474694+0000","last_fullsized":"2024-02-12T17:00:13.474694+0000","mapping_epoch":176,"log_start":"253'8769","ondisk_log_start":"253'8769","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3585,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":116785152,"num_objects":1782,"num_object_clones":0,"num_object_copies":5346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12354,"num_write_kb":452352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":177,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.246301+0000","last_change":"2024-02-12T16:59:09.265012+0000","last_active":"2024-02-12T17:00:08.246301+0000","last_peered":"2024-02-12T17:00:08.246301+0000","last_clean":"2024-02-12T17:00:08.246301+0000","last_became_active":"2024-02-12T16:59:09.264617+0000","last_became_peered":"2024-02-12T16:59:09.264617+0000","last_unstale":"2024-02-12T17:00:08.246301+0000","last_undegraded":"2024-02-12T17:00:08.246301+0000","last_fullsized":"2024-02-12T17:00:08.246301+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.653301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.653301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":269,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240552+0000","last_change":"2024-02-12T16:59:09.263209+0000","last_active":"2024-02-12T17:00:08.240552+0000","last_peered":"2024-02-12T17:00:08.240552+0000","last_clean":"2024-02-12T17:00:08.240552+0000","last_became_active":"2024-02-12T16:59:09.261764+0000","last_became_peered":"2024-02-12T16:59:09.261764+0000","last_unstale":"2024-02-12T17:00:08.240552+0000","last_undegraded":"2024-02-12T17:00:08.240552+0000","last_fullsized":"2024-02-12T17:00:08.240552+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"278'53","reported_seq":495,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.222834+0000","last_change":"2024-02-12T16:59:09.306589+0000","last_active":"2024-02-12T17:00:11.222834+0000","last_peered":"2024-02-12T17:00:11.222834+0000","last_clean":"2024-02-12T17:00:11.222834+0000","last_became_active":"2024-02-12T16:59:09.253257+0000","last_became_peered":"2024-02-12T16:59:09.253257+0000","last_unstale":"2024-02-12T17:00:11.222834+0000","last_undegraded":"2024-02-12T17:00:11.222834+0000","last_fullsized":"2024-02-12T17:00:11.222834+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'39","last_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.083656+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":263,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.230252+0000","last_change":"2024-02-12T16:59:09.264990+0000","last_active":"2024-02-12T17:00:08.230252+0000","last_peered":"2024-02-12T17:00:08.230252+0000","last_clean":"2024-02-12T17:00:08.230252+0000","last_became_active":"2024-02-12T16:59:09.264694+0000","last_became_peered":"2024-02-12T16:59:09.264694+0000","last_unstale":"2024-02-12T17:00:08.230252+0000","last_undegraded":"2024-02-12T17:00:08.230252+0000","last_fullsized":"2024-02-12T17:00:08.230252+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:46.719706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:46.719706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"285'12029","reported_seq":15519,"reported_epoch":285,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:13.444281+0000","last_change":"2024-02-12T16:59:09.891651+0000","last_active":"2024-02-12T17:00:13.444281+0000","last_peered":"2024-02-12T17:00:13.444281+0000","last_clean":"2024-02-12T16:57:46.130664+0000","last_became_active":"2024-02-12T16:57:48.242613+0000","last_became_peered":"2024-02-12T16:57:48.242613+0000","last_unstale":"2024-02-12T17:00:13.444281+0000","last_undegraded":"2024-02-12T17:00:13.444281+0000","last_fullsized":"2024-02-12T17:00:13.444281+0000","mapping_epoch":208,"log_start":"251'8217","ondisk_log_start":"251'8217","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3812,"ondisk_log_size":3812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":113704960,"num_objects":1735,"num_object_clones":0,"num_object_copies":5205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1031,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12029,"num_write_kb":440448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1679,"num_bytes_recovered":110034944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":1735}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":258,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.263644+0000","last_change":"2024-02-12T16:59:09.269787+0000","last_active":"2024-02-12T17:00:08.263644+0000","last_peered":"2024-02-12T17:00:08.263644+0000","last_clean":"2024-02-12T17:00:08.263644+0000","last_became_active":"2024-02-12T16:59:09.269488+0000","last_became_peered":"2024-02-12T16:59:09.269488+0000","last_unstale":"2024-02-12T17:00:08.263644+0000","last_undegraded":"2024-02-12T17:00:08.263644+0000","last_fullsized":"2024-02-12T17:00:08.263644+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":274,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.531577+0000","last_change":"2024-02-12T16:59:10.016594+0000","last_active":"2024-02-12T17:00:11.531577+0000","last_peered":"2024-02-12T17:00:11.531577+0000","last_clean":"2024-02-12T17:00:11.531577+0000","last_became_active":"2024-02-12T16:59:09.273056+0000","last_became_peered":"2024-02-12T16:59:09.273056+0000","last_unstale":"2024-02-12T17:00:11.531577+0000","last_undegraded":"2024-02-12T17:00:11.531577+0000","last_fullsized":"2024-02-12T17:00:11.531577+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":145,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":278,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225460+0000","last_change":"2024-02-12T16:59:09.248652+0000","last_active":"2024-02-12T17:00:08.225460+0000","last_peered":"2024-02-12T17:00:08.225460+0000","last_clean":"2024-02-12T17:00:08.225460+0000","last_became_active":"2024-02-12T16:59:09.248320+0000","last_became_peered":"2024-02-12T16:59:09.248320+0000","last_unstale":"2024-02-12T17:00:08.225460+0000","last_undegraded":"2024-02-12T17:00:08.225460+0000","last_fullsized":"2024-02-12T17:00:08.225460+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":170,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.242632+0000","last_change":"2024-02-12T16:57:47.220507+0000","last_active":"2024-02-12T17:00:08.242632+0000","last_peered":"2024-02-12T17:00:08.242632+0000","last_clean":"2024-02-12T17:00:08.242632+0000","last_became_active":"2024-02-12T16:57:47.190968+0000","last_became_peered":"2024-02-12T16:57:47.190968+0000","last_unstale":"2024-02-12T17:00:08.242632+0000","last_undegraded":"2024-02-12T17:00:08.242632+0000","last_fullsized":"2024-02-12T17:00:08.242632+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":265,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226825+0000","last_change":"2024-02-12T16:58:18.266322+0000","last_active":"2024-02-12T17:00:08.226825+0000","last_peered":"2024-02-12T17:00:08.226825+0000","last_clean":"2024-02-12T17:00:08.226825+0000","last_became_active":"2024-02-12T16:58:18.265999+0000","last_became_peered":"2024-02-12T16:58:18.265999+0000","last_unstale":"2024-02-12T17:00:08.226825+0000","last_undegraded":"2024-02-12T17:00:08.226825+0000","last_fullsized":"2024-02-12T17:00:08.226825+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"285'12041","reported_seq":12177,"reported_epoch":285,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:00:13.495087+0000","last_change":"2024-02-12T17:00:11.190095+0000","last_active":"2024-02-12T17:00:13.495087+0000","last_peered":"2024-02-12T17:00:13.495087+0000","last_clean":"2024-02-12T16:59:08.180231+0000","last_became_active":"2024-02-12T16:59:10.299589+0000","last_became_peered":"2024-02-12T16:59:10.299589+0000","last_unstale":"2024-02-12T17:00:13.495087+0000","last_undegraded":"2024-02-12T17:00:13.495087+0000","last_fullsized":"2024-02-12T17:00:13.495087+0000","mapping_epoch":258,"log_start":"253'8537","ondisk_log_start":"253'8537","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3504,"ondisk_log_size":3504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":117899264,"num_objects":1799,"num_object_clones":0,"num_object_copies":5397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1799,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12041,"num_write_kb":442880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1799}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":258,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.242785+0000","last_change":"2024-02-12T16:57:47.191748+0000","last_active":"2024-02-12T17:00:08.242785+0000","last_peered":"2024-02-12T17:00:08.242785+0000","last_clean":"2024-02-12T17:00:08.242785+0000","last_became_active":"2024-02-12T16:57:47.191375+0000","last_became_peered":"2024-02-12T16:57:47.191375+0000","last_unstale":"2024-02-12T17:00:08.242785+0000","last_undegraded":"2024-02-12T17:00:08.242785+0000","last_fullsized":"2024-02-12T17:00:08.242785+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":173,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.242105+0000","last_change":"2024-02-12T16:59:09.263388+0000","last_active":"2024-02-12T17:00:08.242105+0000","last_peered":"2024-02-12T17:00:08.242105+0000","last_clean":"2024-02-12T17:00:08.242105+0000","last_became_active":"2024-02-12T16:59:09.262266+0000","last_became_peered":"2024-02-12T16:59:09.262266+0000","last_unstale":"2024-02-12T17:00:08.242105+0000","last_undegraded":"2024-02-12T17:00:08.242105+0000","last_fullsized":"2024-02-12T17:00:08.242105+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":251,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.238393+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:00:08.238393+0000","last_peered":"2024-02-12T17:00:08.238393+0000","last_clean":"2024-02-12T17:00:08.238393+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:00:08.238393+0000","last_undegraded":"2024-02-12T17:00:08.238393+0000","last_fullsized":"2024-02-12T17:00:08.238393+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":261,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261641+0000","last_change":"2024-02-12T16:59:09.245920+0000","last_active":"2024-02-12T17:00:08.261641+0000","last_peered":"2024-02-12T17:00:08.261641+0000","last_clean":"2024-02-12T17:00:08.261641+0000","last_became_active":"2024-02-12T16:59:09.245635+0000","last_became_peered":"2024-02-12T16:59:09.245635+0000","last_unstale":"2024-02-12T17:00:08.261641+0000","last_undegraded":"2024-02-12T17:00:08.261641+0000","last_fullsized":"2024-02-12T17:00:08.261641+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":107,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.239974+0000","last_change":"2024-02-12T16:59:10.440537+0000","last_active":"2024-02-12T17:00:08.239974+0000","last_peered":"2024-02-12T17:00:08.239974+0000","last_clean":"2024-02-12T17:00:08.239974+0000","last_became_active":"2024-02-12T16:59:09.249903+0000","last_became_peered":"2024-02-12T16:59:09.249903+0000","last_unstale":"2024-02-12T17:00:08.239974+0000","last_undegraded":"2024-02-12T17:00:08.239974+0000","last_fullsized":"2024-02-12T17:00:08.239974+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:19.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.280894+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":245,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.266140+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T17:00:08.266140+0000","last_peered":"2024-02-12T17:00:08.266140+0000","last_clean":"2024-02-12T17:00:08.266140+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:00:08.266140+0000","last_undegraded":"2024-02-12T17:00:08.266140+0000","last_fullsized":"2024-02-12T17:00:08.266140+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"285'12147","reported_seq":24648,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.433468+0000","last_change":"2024-02-12T16:58:49.227803+0000","last_active":"2024-02-12T17:00:13.433468+0000","last_peered":"2024-02-12T17:00:13.433468+0000","last_clean":"2024-02-12T17:00:13.433468+0000","last_became_active":"2024-02-12T16:58:49.217536+0000","last_became_peered":"2024-02-12T16:58:49.217536+0000","last_unstale":"2024-02-12T17:00:13.433468+0000","last_undegraded":"2024-02-12T17:00:13.433468+0000","last_fullsized":"2024-02-12T17:00:13.433468+0000","mapping_epoch":249,"log_start":"252'8437","ondisk_log_start":"252'8437","created":176,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3710,"ondisk_log_size":3710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":119996416,"num_objects":1831,"num_object_clones":0,"num_object_copies":5493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12147,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6182,"num_bytes_recovered":405143552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":191,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.237342+0000","last_change":"2024-02-12T16:59:09.252727+0000","last_active":"2024-02-12T17:00:08.237342+0000","last_peered":"2024-02-12T17:00:08.237342+0000","last_clean":"2024-02-12T17:00:08.237342+0000","last_became_active":"2024-02-12T16:59:09.252442+0000","last_became_peered":"2024-02-12T16:59:09.252442+0000","last_unstale":"2024-02-12T17:00:08.237342+0000","last_undegraded":"2024-02-12T17:00:08.237342+0000","last_fullsized":"2024-02-12T17:00:08.237342+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":253,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228604+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T17:00:08.228604+0000","last_peered":"2024-02-12T17:00:08.228604+0000","last_clean":"2024-02-12T17:00:08.228604+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T17:00:08.228604+0000","last_undegraded":"2024-02-12T17:00:08.228604+0000","last_fullsized":"2024-02-12T17:00:08.228604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":268,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261598+0000","last_change":"2024-02-12T16:57:48.350132+0000","last_active":"2024-02-12T17:00:08.261598+0000","last_peered":"2024-02-12T17:00:08.261598+0000","last_clean":"2024-02-12T17:00:08.261598+0000","last_became_active":"2024-02-12T16:57:47.200347+0000","last_became_peered":"2024-02-12T16:57:47.200347+0000","last_unstale":"2024-02-12T17:00:08.261598+0000","last_undegraded":"2024-02-12T17:00:08.261598+0000","last_fullsized":"2024-02-12T17:00:08.261598+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":306,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229071+0000","last_change":"2024-02-12T16:57:47.204108+0000","last_active":"2024-02-12T17:00:08.229071+0000","last_peered":"2024-02-12T17:00:08.229071+0000","last_clean":"2024-02-12T17:00:08.229071+0000","last_became_active":"2024-02-12T16:57:47.184456+0000","last_became_peered":"2024-02-12T16:57:47.184456+0000","last_unstale":"2024-02-12T17:00:08.229071+0000","last_undegraded":"2024-02-12T17:00:08.229071+0000","last_fullsized":"2024-02-12T17:00:08.229071+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":269,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.242740+0000","last_change":"2024-02-12T16:58:14.391518+0000","last_active":"2024-02-12T17:00:08.242740+0000","last_peered":"2024-02-12T17:00:08.242740+0000","last_clean":"2024-02-12T17:00:08.242740+0000","last_became_active":"2024-02-12T16:58:14.391221+0000","last_became_peered":"2024-02-12T16:58:14.391221+0000","last_unstale":"2024-02-12T17:00:08.242740+0000","last_undegraded":"2024-02-12T17:00:08.242740+0000","last_fullsized":"2024-02-12T17:00:08.242740+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"285'12204","reported_seq":32931,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.504385+0000","last_change":"2024-02-12T16:59:46.062924+0000","last_active":"2024-02-12T17:00:13.504385+0000","last_peered":"2024-02-12T17:00:13.504385+0000","last_clean":"2024-02-12T17:00:13.504385+0000","last_became_active":"2024-02-12T16:59:46.029014+0000","last_became_peered":"2024-02-12T16:59:46.029014+0000","last_unstale":"2024-02-12T17:00:13.504385+0000","last_undegraded":"2024-02-12T17:00:13.504385+0000","last_fullsized":"2024-02-12T17:00:13.504385+0000","mapping_epoch":275,"log_start":"253'8579","ondisk_log_start":"253'8579","created":176,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3625,"ondisk_log_size":3625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":117833728,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12204,"num_write_kb":448064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10282,"num_bytes_recovered":673841152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":42,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225480+0000","last_change":"2024-02-12T16:59:09.242340+0000","last_active":"2024-02-12T17:00:08.225480+0000","last_peered":"2024-02-12T17:00:08.225480+0000","last_clean":"2024-02-12T17:00:08.225480+0000","last_became_active":"2024-02-12T16:59:09.242012+0000","last_became_peered":"2024-02-12T16:59:09.242012+0000","last_unstale":"2024-02-12T17:00:08.225480+0000","last_undegraded":"2024-02-12T17:00:08.225480+0000","last_fullsized":"2024-02-12T17:00:08.225480+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.053587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.053587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":249,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.263731+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T17:00:08.263731+0000","last_peered":"2024-02-12T17:00:08.263731+0000","last_clean":"2024-02-12T17:00:08.263731+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T17:00:08.263731+0000","last_undegraded":"2024-02-12T17:00:08.263731+0000","last_fullsized":"2024-02-12T17:00:08.263731+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":298,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228518+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T17:00:08.228518+0000","last_peered":"2024-02-12T17:00:08.228518+0000","last_clean":"2024-02-12T17:00:08.228518+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T17:00:08.228518+0000","last_undegraded":"2024-02-12T17:00:08.228518+0000","last_fullsized":"2024-02-12T17:00:08.228518+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229315+0000","last_change":"2024-02-12T16:59:09.271420+0000","last_active":"2024-02-12T17:00:08.229315+0000","last_peered":"2024-02-12T17:00:08.229315+0000","last_clean":"2024-02-12T17:00:08.229315+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:00:08.229315+0000","last_undegraded":"2024-02-12T17:00:08.229315+0000","last_fullsized":"2024-02-12T17:00:08.229315+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"285'12265","reported_seq":12382,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.547426+0000","last_change":"2024-02-12T16:56:36.789146+0000","last_active":"2024-02-12T17:00:13.547426+0000","last_peered":"2024-02-12T17:00:13.547426+0000","last_clean":"2024-02-12T17:00:13.547426+0000","last_became_active":"2024-02-12T16:56:36.788871+0000","last_became_peered":"2024-02-12T16:56:36.788871+0000","last_unstale":"2024-02-12T17:00:13.547426+0000","last_undegraded":"2024-02-12T17:00:13.547426+0000","last_fullsized":"2024-02-12T17:00:13.547426+0000","mapping_epoch":176,"log_start":"253'8648","ondisk_log_start":"253'8648","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3617,"ondisk_log_size":3617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":111738880,"num_objects":1705,"num_object_clones":0,"num_object_copies":5115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":447040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":176,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225776+0000","last_change":"2024-02-12T16:59:09.242302+0000","last_active":"2024-02-12T17:00:08.225776+0000","last_peered":"2024-02-12T17:00:08.225776+0000","last_clean":"2024-02-12T17:00:08.225776+0000","last_became_active":"2024-02-12T16:59:09.241987+0000","last_became_peered":"2024-02-12T16:59:09.241987+0000","last_unstale":"2024-02-12T17:00:08.225776+0000","last_undegraded":"2024-02-12T17:00:08.225776+0000","last_fullsized":"2024-02-12T17:00:08.225776+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.246129+0000","last_change":"2024-02-12T16:59:09.242865+0000","last_active":"2024-02-12T17:00:08.246129+0000","last_peered":"2024-02-12T17:00:08.246129+0000","last_clean":"2024-02-12T17:00:08.246129+0000","last_became_active":"2024-02-12T16:59:09.242505+0000","last_became_peered":"2024-02-12T16:59:09.242505+0000","last_unstale":"2024-02-12T17:00:08.246129+0000","last_undegraded":"2024-02-12T17:00:08.246129+0000","last_fullsized":"2024-02-12T17:00:08.246129+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":271,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261794+0000","last_change":"2024-02-12T16:57:48.521837+0000","last_active":"2024-02-12T17:00:08.261794+0000","last_peered":"2024-02-12T17:00:08.261794+0000","last_clean":"2024-02-12T17:00:08.261794+0000","last_became_active":"2024-02-12T16:57:47.199763+0000","last_became_peered":"2024-02-12T16:57:47.199763+0000","last_unstale":"2024-02-12T17:00:08.261794+0000","last_undegraded":"2024-02-12T17:00:08.261794+0000","last_fullsized":"2024-02-12T17:00:08.261794+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":257,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.245813+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T17:00:08.245813+0000","last_peered":"2024-02-12T17:00:08.245813+0000","last_clean":"2024-02-12T17:00:08.245813+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T17:00:08.245813+0000","last_undegraded":"2024-02-12T17:00:08.245813+0000","last_fullsized":"2024-02-12T17:00:08.245813+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"285'12384","reported_seq":12502,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.464866+0000","last_change":"2024-02-12T16:56:36.788810+0000","last_active":"2024-02-12T17:00:13.464866+0000","last_peered":"2024-02-12T17:00:13.464866+0000","last_clean":"2024-02-12T17:00:13.464866+0000","last_became_active":"2024-02-12T16:56:36.788518+0000","last_became_peered":"2024-02-12T16:56:36.788518+0000","last_unstale":"2024-02-12T17:00:13.464866+0000","last_undegraded":"2024-02-12T17:00:13.464866+0000","last_fullsized":"2024-02-12T17:00:13.464866+0000","mapping_epoch":176,"log_start":"253'8812","ondisk_log_start":"253'8812","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3572,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":118947864,"num_objects":1816,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12384,"num_write_kb":454337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":257,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.271882+0000","last_change":"2024-02-12T16:59:09.340317+0000","last_active":"2024-02-12T17:00:08.271882+0000","last_peered":"2024-02-12T17:00:08.271882+0000","last_clean":"2024-02-12T17:00:08.271882+0000","last_became_active":"2024-02-12T16:59:09.246616+0000","last_became_peered":"2024-02-12T16:59:09.246616+0000","last_unstale":"2024-02-12T17:00:08.271882+0000","last_undegraded":"2024-02-12T17:00:08.271882+0000","last_fullsized":"2024-02-12T17:00:08.271882+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:57:19.955311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.955311+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.264771+0000","last_change":"2024-02-12T16:59:09.268325+0000","last_active":"2024-02-12T17:00:08.264771+0000","last_peered":"2024-02-12T17:00:08.264771+0000","last_clean":"2024-02-12T17:00:08.264771+0000","last_became_active":"2024-02-12T16:59:09.268028+0000","last_became_peered":"2024-02-12T16:59:09.268028+0000","last_unstale":"2024-02-12T17:00:08.264771+0000","last_undegraded":"2024-02-12T17:00:08.264771+0000","last_fullsized":"2024-02-12T17:00:08.264771+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":288,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.237424+0000","last_change":"2024-02-12T16:59:09.996418+0000","last_active":"2024-02-12T17:00:08.237424+0000","last_peered":"2024-02-12T17:00:08.237424+0000","last_clean":"2024-02-12T17:00:08.237424+0000","last_became_active":"2024-02-12T16:59:09.256058+0000","last_became_peered":"2024-02-12T16:59:09.256058+0000","last_unstale":"2024-02-12T17:00:08.237424+0000","last_undegraded":"2024-02-12T17:00:08.237424+0000","last_fullsized":"2024-02-12T17:00:08.237424+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"208'10","reported_seq":142,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.230485+0000","last_change":"2024-02-12T16:57:47.193205+0000","last_active":"2024-02-12T17:00:11.230485+0000","last_peered":"2024-02-12T17:00:11.230485+0000","last_clean":"2024-02-12T17:00:11.230485+0000","last_became_active":"2024-02-12T16:57:47.167628+0000","last_became_peered":"2024-02-12T16:57:47.167628+0000","last_unstale":"2024-02-12T17:00:11.230485+0000","last_undegraded":"2024-02-12T17:00:11.230485+0000","last_fullsized":"2024-02-12T17:00:11.230485+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"285'12110","reported_seq":12227,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.409527+0000","last_change":"2024-02-12T16:56:36.445962+0000","last_active":"2024-02-12T17:00:13.409527+0000","last_peered":"2024-02-12T17:00:13.409527+0000","last_clean":"2024-02-12T17:00:13.409527+0000","last_became_active":"2024-02-12T16:56:36.445638+0000","last_became_peered":"2024-02-12T16:56:36.445638+0000","last_unstale":"2024-02-12T17:00:13.409527+0000","last_undegraded":"2024-02-12T17:00:13.409527+0000","last_fullsized":"2024-02-12T17:00:13.409527+0000","mapping_epoch":176,"log_start":"251'8223","ondisk_log_start":"251'8223","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3887,"ondisk_log_size":3887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109838336,"num_objects":1676,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12110,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":42,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.237259+0000","last_change":"2024-02-12T16:59:09.260336+0000","last_active":"2024-02-12T17:00:08.237259+0000","last_peered":"2024-02-12T17:00:08.237259+0000","last_clean":"2024-02-12T17:00:08.237259+0000","last_became_active":"2024-02-12T16:59:09.259654+0000","last_became_peered":"2024-02-12T16:59:09.259654+0000","last_unstale":"2024-02-12T17:00:08.237259+0000","last_undegraded":"2024-02-12T17:00:08.237259+0000","last_fullsized":"2024-02-12T17:00:08.237259+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"261'14","reported_seq":324,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.220915+0000","last_change":"2024-02-12T16:59:12.225158+0000","last_active":"2024-02-12T17:00:11.220915+0000","last_peered":"2024-02-12T17:00:11.220915+0000","last_clean":"2024-02-12T17:00:11.220915+0000","last_became_active":"2024-02-12T16:59:12.224830+0000","last_became_peered":"2024-02-12T16:59:12.224830+0000","last_unstale":"2024-02-12T17:00:11.220915+0000","last_undegraded":"2024-02-12T17:00:11.220915+0000","last_fullsized":"2024-02-12T17:00:11.220915+0000","mapping_epoch":260,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":237,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225120+0000","last_change":"2024-02-12T16:57:47.171714+0000","last_active":"2024-02-12T17:00:08.225120+0000","last_peered":"2024-02-12T17:00:08.225120+0000","last_clean":"2024-02-12T17:00:08.225120+0000","last_became_active":"2024-02-12T16:57:47.171128+0000","last_became_peered":"2024-02-12T16:57:47.171128+0000","last_unstale":"2024-02-12T17:00:08.225120+0000","last_undegraded":"2024-02-12T17:00:08.225120+0000","last_fullsized":"2024-02-12T17:00:08.225120+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":252,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240716+0000","last_change":"2024-02-12T16:59:09.788881+0000","last_active":"2024-02-12T17:00:08.240716+0000","last_peered":"2024-02-12T17:00:08.240716+0000","last_clean":"2024-02-12T17:00:08.240716+0000","last_became_active":"2024-02-12T16:59:09.277264+0000","last_became_peered":"2024-02-12T16:59:09.277264+0000","last_unstale":"2024-02-12T17:00:08.240716+0000","last_undegraded":"2024-02-12T17:00:08.240716+0000","last_fullsized":"2024-02-12T17:00:08.240716+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"285'12256","reported_seq":15903,"reported_epoch":285,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:13.449006+0000","last_change":"2024-02-12T16:59:09.997308+0000","last_active":"2024-02-12T17:00:13.449006+0000","last_peered":"2024-02-12T17:00:13.449006+0000","last_clean":"2024-02-12T16:57:46.091774+0000","last_became_active":"2024-02-12T16:59:09.335728+0000","last_became_peered":"2024-02-12T16:59:09.335728+0000","last_unstale":"2024-02-12T17:00:13.449006+0000","last_undegraded":"2024-02-12T17:00:13.449006+0000","last_fullsized":"2024-02-12T17:00:13.449006+0000","mapping_epoch":257,"log_start":"253'8701","ondisk_log_start":"253'8701","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3555,"ondisk_log_size":3555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":117702656,"num_objects":1796,"num_object_clones":0,"num_object_copies":5388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2742,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12256,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":109248512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5,6","objects":1796}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":254,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.241339+0000","last_change":"2024-02-12T16:57:47.177662+0000","last_active":"2024-02-12T17:00:08.241339+0000","last_peered":"2024-02-12T17:00:08.241339+0000","last_clean":"2024-02-12T17:00:08.241339+0000","last_became_active":"2024-02-12T16:57:47.177329+0000","last_became_peered":"2024-02-12T16:57:47.177329+0000","last_unstale":"2024-02-12T17:00:08.241339+0000","last_undegraded":"2024-02-12T17:00:08.241339+0000","last_fullsized":"2024-02-12T17:00:08.241339+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.152265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"208'12","reported_seq":369,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.223021+0000","last_change":"2024-02-12T16:57:46.177864+0000","last_active":"2024-02-12T17:00:11.223021+0000","last_peered":"2024-02-12T17:00:11.223021+0000","last_clean":"2024-02-12T17:00:11.223021+0000","last_became_active":"2024-02-12T16:57:46.177517+0000","last_became_peered":"2024-02-12T16:57:46.177517+0000","last_unstale":"2024-02-12T17:00:11.223021+0000","last_undegraded":"2024-02-12T17:00:11.223021+0000","last_fullsized":"2024-02-12T17:00:11.223021+0000","mapping_epoch":97,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":37,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.260887+0000","last_change":"2024-02-12T16:59:09.268099+0000","last_active":"2024-02-12T17:00:08.260887+0000","last_peered":"2024-02-12T17:00:08.260887+0000","last_clean":"2024-02-12T17:00:08.260887+0000","last_became_active":"2024-02-12T16:59:09.267755+0000","last_became_peered":"2024-02-12T16:59:09.267755+0000","last_unstale":"2024-02-12T17:00:08.260887+0000","last_undegraded":"2024-02-12T17:00:08.260887+0000","last_fullsized":"2024-02-12T17:00:08.260887+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":257,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.264601+0000","last_change":"2024-02-12T16:57:48.169827+0000","last_active":"2024-02-12T17:00:08.264601+0000","last_peered":"2024-02-12T17:00:08.264601+0000","last_clean":"2024-02-12T17:00:08.264601+0000","last_became_active":"2024-02-12T16:57:47.199008+0000","last_became_peered":"2024-02-12T16:57:47.199008+0000","last_unstale":"2024-02-12T17:00:08.264601+0000","last_undegraded":"2024-02-12T17:00:08.264601+0000","last_fullsized":"2024-02-12T17:00:08.264601+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"285'12279","reported_seq":15583,"reported_epoch":285,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:13.486166+0000","last_change":"2024-02-12T16:59:10.062476+0000","last_active":"2024-02-12T17:00:13.486166+0000","last_peered":"2024-02-12T17:00:13.486166+0000","last_clean":"2024-02-12T16:57:46.133812+0000","last_became_active":"2024-02-12T16:59:09.319664+0000","last_became_peered":"2024-02-12T16:59:09.319664+0000","last_unstale":"2024-02-12T17:00:13.486166+0000","last_undegraded":"2024-02-12T17:00:13.486166+0000","last_fullsized":"2024-02-12T17:00:13.486166+0000","mapping_epoch":257,"log_start":"253'8714","ondisk_log_start":"253'8714","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3565,"ondisk_log_size":3565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":112394240,"num_objects":1715,"num_object_clones":0,"num_object_copies":5145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1351,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12279,"num_write_kb":447808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":103809024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":1715}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226905+0000","last_change":"2024-02-12T16:57:47.176436+0000","last_active":"2024-02-12T17:00:08.226905+0000","last_peered":"2024-02-12T17:00:08.226905+0000","last_clean":"2024-02-12T17:00:08.226905+0000","last_became_active":"2024-02-12T16:57:47.176041+0000","last_became_peered":"2024-02-12T16:57:47.176041+0000","last_unstale":"2024-02-12T17:00:08.226905+0000","last_undegraded":"2024-02-12T17:00:08.226905+0000","last_fullsized":"2024-02-12T17:00:08.226905+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"211'10","reported_seq":401,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.222937+0000","last_change":"2024-02-12T16:57:50.205871+0000","last_active":"2024-02-12T17:00:11.222937+0000","last_peered":"2024-02-12T17:00:11.222937+0000","last_clean":"2024-02-12T17:00:11.222937+0000","last_became_active":"2024-02-12T16:57:50.205330+0000","last_became_peered":"2024-02-12T16:57:50.205330+0000","last_unstale":"2024-02-12T17:00:11.222937+0000","last_undegraded":"2024-02-12T17:00:11.222937+0000","last_fullsized":"2024-02-12T17:00:11.222937+0000","mapping_epoch":61,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":193,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261196+0000","last_change":"2024-02-12T16:57:47.191627+0000","last_active":"2024-02-12T17:00:08.261196+0000","last_peered":"2024-02-12T17:00:08.261196+0000","last_clean":"2024-02-12T17:00:08.261196+0000","last_became_active":"2024-02-12T16:57:47.191231+0000","last_became_peered":"2024-02-12T16:57:47.191231+0000","last_unstale":"2024-02-12T17:00:08.261196+0000","last_undegraded":"2024-02-12T17:00:08.261196+0000","last_fullsized":"2024-02-12T17:00:08.261196+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":192,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261088+0000","last_change":"2024-02-12T16:59:09.782291+0000","last_active":"2024-02-12T17:00:08.261088+0000","last_peered":"2024-02-12T17:00:08.261088+0000","last_clean":"2024-02-12T17:00:08.261088+0000","last_became_active":"2024-02-12T16:59:09.268136+0000","last_became_peered":"2024-02-12T16:59:09.268136+0000","last_unstale":"2024-02-12T17:00:08.261088+0000","last_undegraded":"2024-02-12T17:00:08.261088+0000","last_fullsized":"2024-02-12T17:00:08.261088+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"285'11996","reported_seq":14799,"reported_epoch":285,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:13.554629+0000","last_change":"2024-02-12T16:59:09.891108+0000","last_active":"2024-02-12T17:00:13.554629+0000","last_peered":"2024-02-12T17:00:13.554629+0000","last_clean":"2024-02-12T16:57:46.113428+0000","last_became_active":"2024-02-12T16:57:48.262168+0000","last_became_peered":"2024-02-12T16:57:48.262168+0000","last_unstale":"2024-02-12T17:00:13.554629+0000","last_undegraded":"2024-02-12T17:00:13.554629+0000","last_fullsized":"2024-02-12T17:00:13.554629+0000","mapping_epoch":208,"log_start":"253'8436","ondisk_log_start":"253'8436","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3560,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114163712,"num_objects":1742,"num_object_clones":0,"num_object_copies":5226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1122,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11996,"num_write_kb":439616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1333,"num_bytes_recovered":87359488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":1742}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":268,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228066+0000","last_change":"2024-02-12T16:59:09.246938+0000","last_active":"2024-02-12T17:00:08.228066+0000","last_peered":"2024-02-12T17:00:08.228066+0000","last_clean":"2024-02-12T17:00:08.228066+0000","last_became_active":"2024-02-12T16:59:09.246605+0000","last_became_peered":"2024-02-12T16:59:09.246605+0000","last_unstale":"2024-02-12T17:00:08.228066+0000","last_undegraded":"2024-02-12T17:00:08.228066+0000","last_fullsized":"2024-02-12T17:00:08.228066+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":116,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240423+0000","last_change":"2024-02-12T16:59:09.247764+0000","last_active":"2024-02-12T17:00:08.240423+0000","last_peered":"2024-02-12T17:00:08.240423+0000","last_clean":"2024-02-12T17:00:08.240423+0000","last_became_active":"2024-02-12T16:59:09.247447+0000","last_became_peered":"2024-02-12T16:59:09.247447+0000","last_unstale":"2024-02-12T17:00:08.240423+0000","last_undegraded":"2024-02-12T17:00:08.240423+0000","last_fullsized":"2024-02-12T17:00:08.240423+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":249,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.271516+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T17:00:08.271516+0000","last_peered":"2024-02-12T17:00:08.271516+0000","last_clean":"2024-02-12T17:00:08.271516+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T17:00:08.271516+0000","last_undegraded":"2024-02-12T17:00:08.271516+0000","last_fullsized":"2024-02-12T17:00:08.271516+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":351,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.227052+0000","last_change":"2024-02-12T16:59:10.004273+0000","last_active":"2024-02-12T17:00:08.227052+0000","last_peered":"2024-02-12T17:00:08.227052+0000","last_clean":"2024-02-12T17:00:08.227052+0000","last_became_active":"2024-02-12T16:59:09.262475+0000","last_became_peered":"2024-02-12T16:59:09.262475+0000","last_unstale":"2024-02-12T17:00:08.227052+0000","last_undegraded":"2024-02-12T17:00:08.227052+0000","last_fullsized":"2024-02-12T17:00:08.227052+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"285'12188","reported_seq":21098,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.532350+0000","last_change":"2024-02-12T16:59:27.226608+0000","last_active":"2024-02-12T17:00:13.532350+0000","last_peered":"2024-02-12T17:00:13.532350+0000","last_clean":"2024-02-12T17:00:13.532350+0000","last_became_active":"2024-02-12T16:59:27.206365+0000","last_became_peered":"2024-02-12T16:59:27.206365+0000","last_unstale":"2024-02-12T17:00:13.532350+0000","last_undegraded":"2024-02-12T17:00:13.532350+0000","last_fullsized":"2024-02-12T17:00:13.532350+0000","mapping_epoch":265,"log_start":"253'8630","ondisk_log_start":"253'8630","created":176,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3558,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":117047296,"num_objects":1786,"num_object_clones":0,"num_object_copies":5358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12188,"num_write_kb":447168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4385,"num_bytes_recovered":287375360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":261,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261550+0000","last_change":"2024-02-12T16:59:09.242082+0000","last_active":"2024-02-12T17:00:08.261550+0000","last_peered":"2024-02-12T17:00:08.261550+0000","last_clean":"2024-02-12T17:00:08.261550+0000","last_became_active":"2024-02-12T16:59:09.241648+0000","last_became_peered":"2024-02-12T16:59:09.241648+0000","last_unstale":"2024-02-12T17:00:08.261550+0000","last_undegraded":"2024-02-12T17:00:08.261550+0000","last_fullsized":"2024-02-12T17:00:08.261550+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":270,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226412+0000","last_change":"2024-02-12T16:57:58.203843+0000","last_active":"2024-02-12T17:00:08.226412+0000","last_peered":"2024-02-12T17:00:08.226412+0000","last_clean":"2024-02-12T17:00:08.226412+0000","last_became_active":"2024-02-12T16:57:58.203578+0000","last_became_peered":"2024-02-12T16:57:58.203578+0000","last_unstale":"2024-02-12T17:00:08.226412+0000","last_undegraded":"2024-02-12T17:00:08.226412+0000","last_fullsized":"2024-02-12T17:00:08.226412+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":251,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226609+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T17:00:08.226609+0000","last_peered":"2024-02-12T17:00:08.226609+0000","last_clean":"2024-02-12T17:00:08.226609+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T17:00:08.226609+0000","last_undegraded":"2024-02-12T17:00:08.226609+0000","last_fullsized":"2024-02-12T17:00:08.226609+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":126,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.261383+0000","last_change":"2024-02-12T16:59:10.836208+0000","last_active":"2024-02-12T17:00:08.261383+0000","last_peered":"2024-02-12T17:00:08.261383+0000","last_clean":"2024-02-12T17:00:08.261383+0000","last_became_active":"2024-02-12T16:59:09.273985+0000","last_became_peered":"2024-02-12T16:59:09.273985+0000","last_unstale":"2024-02-12T17:00:08.261383+0000","last_undegraded":"2024-02-12T17:00:08.261383+0000","last_fullsized":"2024-02-12T17:00:08.261383+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:41.290236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.290236+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"285'12198","reported_seq":32809,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.491331+0000","last_change":"2024-02-12T16:59:45.048235+0000","last_active":"2024-02-12T17:00:13.491331+0000","last_peered":"2024-02-12T17:00:13.491331+0000","last_clean":"2024-02-12T17:00:13.491331+0000","last_became_active":"2024-02-12T16:59:45.029644+0000","last_became_peered":"2024-02-12T16:59:45.029644+0000","last_unstale":"2024-02-12T17:00:13.491331+0000","last_undegraded":"2024-02-12T17:00:13.491331+0000","last_fullsized":"2024-02-12T17:00:13.491331+0000","mapping_epoch":274,"log_start":"253'8572","ondisk_log_start":"253'8572","created":176,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3626,"ondisk_log_size":3626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12198,"num_write_kb":445120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10227,"num_bytes_recovered":670236672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":238,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.229214+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T17:00:08.229214+0000","last_peered":"2024-02-12T17:00:08.229214+0000","last_clean":"2024-02-12T17:00:08.229214+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T17:00:08.229214+0000","last_undegraded":"2024-02-12T17:00:08.229214+0000","last_fullsized":"2024-02-12T17:00:08.229214+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":274,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.225605+0000","last_change":"2024-02-12T16:58:02.376971+0000","last_active":"2024-02-12T17:00:08.225605+0000","last_peered":"2024-02-12T17:00:08.225605+0000","last_clean":"2024-02-12T17:00:08.225605+0000","last_became_active":"2024-02-12T16:58:02.376628+0000","last_became_peered":"2024-02-12T16:58:02.376628+0000","last_unstale":"2024-02-12T17:00:08.225605+0000","last_undegraded":"2024-02-12T17:00:08.225605+0000","last_fullsized":"2024-02-12T17:00:08.225605+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:19.703739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.703739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":215,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226996+0000","last_change":"2024-02-12T16:59:09.279916+0000","last_active":"2024-02-12T17:00:08.226996+0000","last_peered":"2024-02-12T17:00:08.226996+0000","last_clean":"2024-02-12T17:00:08.226996+0000","last_became_active":"2024-02-12T16:59:09.262401+0000","last_became_peered":"2024-02-12T16:59:09.262401+0000","last_unstale":"2024-02-12T17:00:08.226996+0000","last_undegraded":"2024-02-12T17:00:08.226996+0000","last_fullsized":"2024-02-12T17:00:08.226996+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":291,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.228677+0000","last_change":"2024-02-12T16:57:47.553993+0000","last_active":"2024-02-12T17:00:08.228677+0000","last_peered":"2024-02-12T17:00:08.228677+0000","last_clean":"2024-02-12T17:00:08.228677+0000","last_became_active":"2024-02-12T16:57:47.189672+0000","last_became_peered":"2024-02-12T16:57:47.189672+0000","last_unstale":"2024-02-12T17:00:08.228677+0000","last_undegraded":"2024-02-12T17:00:08.228677+0000","last_fullsized":"2024-02-12T17:00:08.228677+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"285'12330","reported_seq":12447,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.489043+0000","last_change":"2024-02-12T16:56:36.445585+0000","last_active":"2024-02-12T17:00:13.489043+0000","last_peered":"2024-02-12T17:00:13.489043+0000","last_clean":"2024-02-12T17:00:13.489043+0000","last_became_active":"2024-02-12T16:56:36.445291+0000","last_became_peered":"2024-02-12T16:56:36.445291+0000","last_unstale":"2024-02-12T17:00:13.489043+0000","last_undegraded":"2024-02-12T17:00:13.489043+0000","last_fullsized":"2024-02-12T17:00:13.489043+0000","mapping_epoch":176,"log_start":"253'8794","ondisk_log_start":"253'8794","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3536,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":115605504,"num_objects":1764,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12330,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":249,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.240374+0000","last_change":"2024-02-12T16:59:09.262958+0000","last_active":"2024-02-12T17:00:08.240374+0000","last_peered":"2024-02-12T17:00:08.240374+0000","last_clean":"2024-02-12T17:00:08.240374+0000","last_became_active":"2024-02-12T16:59:09.261034+0000","last_became_peered":"2024-02-12T16:59:09.261034+0000","last_unstale":"2024-02-12T17:00:08.240374+0000","last_undegraded":"2024-02-12T17:00:08.240374+0000","last_fullsized":"2024-02-12T17:00:08.240374+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":274,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.226628+0000","last_change":"2024-02-12T16:59:09.250623+0000","last_active":"2024-02-12T17:00:08.226628+0000","last_peered":"2024-02-12T17:00:08.226628+0000","last_clean":"2024-02-12T17:00:08.226628+0000","last_became_active":"2024-02-12T16:59:09.250318+0000","last_became_peered":"2024-02-12T16:59:09.250318+0000","last_unstale":"2024-02-12T17:00:08.226628+0000","last_undegraded":"2024-02-12T17:00:08.226628+0000","last_fullsized":"2024-02-12T17:00:08.226628+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":324,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-02-12T17:00:08.234953+0000","last_change":"2024-02-12T16:59:11.094393+0000","last_active":"2024-02-12T17:00:08.234953+0000","last_peered":"2024-02-12T17:00:08.234953+0000","last_clean":"2024-02-12T17:00:08.234953+0000","last_became_active":"2024-02-12T16:59:09.280858+0000","last_became_peered":"2024-02-12T16:59:09.280858+0000","last_unstale":"2024-02-12T17:00:08.234953+0000","last_undegraded":"2024-02-12T17:00:08.234953+0000","last_fullsized":"2024-02-12T17:00:08.234953+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'22","last_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.138758+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"261'13","reported_seq":421,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-12T17:00:11.248104+0000","last_change":"2024-02-12T16:59:12.243084+0000","last_active":"2024-02-12T17:00:11.248104+0000","last_peered":"2024-02-12T17:00:11.248104+0000","last_clean":"2024-02-12T17:00:11.248104+0000","last_became_active":"2024-02-12T16:59:12.242710+0000","last_became_peered":"2024-02-12T16:59:12.242710+0000","last_unstale":"2024-02-12T17:00:11.248104+0000","last_undegraded":"2024-02-12T17:00:11.248104+0000","last_fullsized":"2024-02-12T17:00:11.248104+0000","mapping_epoch":260,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1848115224,"num_objects":28201,"num_object_clones":0,"num_object_copies":84603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9480,"num_objects_unfound":0,"num_objects_dirty":28201,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195111,"num_write_kb":7145921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38007,"num_bytes_recovered":2490826752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7064465408,"data_stored":7064453192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58094,"ondisk_log_size":58094,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":145,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":5408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234114,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043116,"kb_used_data":733540,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92702036,"statfs":{"total":95995035648,"available":94926884864,"internally_reserved":0,"allocated":751144960,"data_stored":750438695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.407,"15min":1.101},"min":{"1min":0.246,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.5339999999999998,"5min":7.3239999999999998,"15min":36.439},"last":0.33500000000000002},{"interface":"front","average":{"1min":3.089,"5min":1.851,"15min":1.2529999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.987,"5min":11.231999999999999,"15min":36.490000000000002},"last":17.268999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":1.778,"15min":1.01},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.048999999999999,"5min":12.032,"15min":12.032},"last":6.1820000000000004},{"interface":"front","average":{"1min":2.456,"5min":1.6830000000000001,"15min":0.97699999999999998},"min":{"1min":0.251,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.965999999999999,"5min":11.260999999999999,"15min":11.260999999999999},"last":17.324999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.2350000000000001,"15min":0.874},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.2450000000000001,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.38},{"interface":"front","average":{"1min":2.5920000000000001,"5min":1.8959999999999999,"15min":1.131},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":11.071999999999999,"5min":11.071999999999999,"15min":20.338000000000001},"last":6.1580000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.569,"15min":1.161},"min":{"1min":0.28999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.8600000000000003,"5min":9.1069999999999993,"15min":20.433},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7040000000000002,"5min":2.0960000000000001,"15min":1.3340000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.092000000000001,"5min":11.092000000000001,"15min":20.393000000000001},"last":6.1200000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":1.9239999999999999,"15min":0.97699999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.219},"max":{"1min":5.556,"5min":50.406999999999996,"15min":50.406999999999996},"last":6.6050000000000004},{"interface":"front","average":{"1min":4.2519999999999998,"5min":2.8119999999999998,"15min":1.27},"min":{"1min":0.29499999999999998,"5min":0.23999999999999999,"15min":0.23400000000000001},"max":{"1min":14.863,"5min":50.168999999999997,"15min":50.168999999999997},"last":11.512}]},{"osd":5,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.4790000000000001,"15min":0.83299999999999996},"min":{"1min":0.45800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.5910000000000002,"5min":7.4269999999999996,"15min":7.4269999999999996},"last":10.91},{"interface":"front","average":{"1min":4.4210000000000003,"5min":2.2210000000000001,"15min":1.093},"min":{"1min":0.371,"5min":0.27400000000000002,"15min":0.26200000000000001},"max":{"1min":11.696,"5min":12.352,"15min":12.352},"last":6.2030000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.663,"15min":0.90100000000000002},"min":{"1min":0.30399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":10.929,"5min":11.282999999999999,"15min":11.282999999999999},"last":17.234999999999999},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5669999999999999,"15min":0.91100000000000003},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.30199999999999999},"max":{"1min":4.4930000000000003,"5min":12.407999999999999,"15min":12.407999999999999},"last":0.63400000000000001}]}]},{"osd":6,"up_from":0,"seq":193273528455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738532,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1138272,"kb_used_data":841048,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92606880,"statfs":{"total":95995035648,"available":94829445120,"internally_reserved":0,"allocated":861233152,"data_stored":860538270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.9199999999999999,"15min":1.0840000000000001},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.3839999999999999,"5min":23.510999999999999,"15min":23.510999999999999},"last":2.0920000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.639,"15min":1.02},"min":{"1min":0.21299999999999999,"5min":0.193,"15min":0.193},"max":{"1min":5.0449999999999999,"5min":10.186999999999999,"15min":10.186999999999999},"last":1.7390000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.841,"15min":1.3180000000000001},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.20499999999999999},"max":{"1min":11.122999999999999,"5min":16.521000000000001,"15min":16.521000000000001},"last":1.9330000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.6100000000000001,"15min":1.1459999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.984,"5min":10.340999999999999,"15min":10.340999999999999},"last":1.607}]},{"osd":3,"last update":"Mon Feb 12 16:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":2.3029999999999999,"15min":1.3149999999999999},"min":{"1min":0.26300000000000001,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":2.415,"5min":21.100000000000001,"15min":21.100000000000001},"last":1.9730000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":2.0179999999999998,"15min":1.2809999999999999},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":2.5659999999999998,"5min":16.446000000000002,"15min":16.446000000000002},"last":2.181}]},{"osd":4,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.7130000000000001,"15min":1.125},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":7.2069999999999999,"5min":16.027000000000001,"15min":16.027000000000001},"last":2.1560000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.716,"15min":1.083},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":8.4659999999999993,"5min":10.141,"15min":10.141},"last":1.8049999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:00:08 2024","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.5070000000000001,"15min":1.456},"min":{"1min":0.308,"5min":0.308,"15min":0.26400000000000001},"max":{"1min":7.1600000000000001,"5min":16.190999999999999,"15min":16.190999999999999},"last":3.0739999999999998},{"interface":"front","average":{"1min":3.028,"5min":2.2080000000000002,"15min":1.431},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":11.423,"5min":18.170999999999999,"15min":18.170999999999999},"last":3.363}]},{"osd":6,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":2.0640000000000001,"15min":1.2110000000000001},"min":{"1min":0.41599999999999998,"5min":0.32100000000000001,"15min":0.314},"max":{"1min":5.077,"5min":17.907,"15min":17.907},"last":4.46},{"interface":"front","average":{"1min":1.7030000000000001,"5min":2.3599999999999999,"15min":1.3600000000000001},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.27900000000000003},"max":{"1min":5.5880000000000001,"5min":21.062000000000001,"15min":21.062000000000001},"last":5.1890000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.472,"5min":1.9390000000000001,"15min":1.2669999999999999},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.32400000000000001},"max":{"1min":12.058,"5min":16.175000000000001,"15min":16.175000000000001},"last":5.0599999999999996},{"interface":"front","average":{"1min":3.79,"5min":2.1589999999999998,"15min":1.274},"min":{"1min":0.45600000000000002,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":34.722999999999999,"5min":34.722999999999999,"15min":34.722999999999999},"last":5.1310000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1363456,"kb_used_data":1066168,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92381696,"statfs":{"total":95995035648,"available":94598856704,"internally_reserved":0,"allocated":1091756032,"data_stored":1091093728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.9550000000000001,"15min":1.3440000000000001},"min":{"1min":0.192,"5min":0.186,"15min":0.186},"max":{"1min":3.7400000000000002,"5min":15.808,"15min":15.808},"last":0.191},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.756,"15min":1.1899999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.158},"max":{"1min":6.1070000000000002,"5min":13.497,"15min":19.893000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":2.0009999999999999,"15min":1.4099999999999999},"min":{"1min":0.23400000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":6.5819999999999999,"5min":17.739000000000001,"15min":17.739000000000001},"last":0.436},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.9399999999999999,"15min":1.4059999999999999},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":3.7629999999999999,"5min":15.766999999999999,"15min":15.766999999999999},"last":0.377}]},{"osd":3,"last update":"Mon Feb 12 16:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":2.1949999999999998,"15min":1.5389999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23499999999999999},"max":{"1min":3.702,"5min":15.787000000000001,"15min":51.579999999999998},"last":0.753},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.855,"15min":1.2909999999999999},"min":{"1min":0.249,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.8889999999999998,"5min":13.521000000000001,"15min":48.988999999999997},"last":0.41299999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:59:50 2024","interfaces":[{"interface":"back","average":{"1min":2.415,"5min":2.2930000000000001,"15min":1.361},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.26700000000000002},"max":{"1min":7.2789999999999999,"5min":15.737,"15min":15.737},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.4630000000000001,"5min":2.2080000000000002,"15min":1.252},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.26800000000000002},"max":{"1min":7.5449999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.308}]},{"osd":5,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":2.081,"15min":1.4379999999999999},"min":{"1min":0.41499999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":10.407999999999999,"5min":15.699,"15min":15.699},"last":0.54800000000000004},{"interface":"front","average":{"1min":2.9580000000000002,"5min":2.1560000000000001,"15min":1.518},"min":{"1min":0.33600000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":11.999000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:59:21 2024","interfaces":[{"interface":"back","average":{"1min":3.0529999999999999,"5min":2.754,"15min":1.5600000000000001},"min":{"1min":0.40999999999999998,"5min":0.36399999999999999,"15min":0.31},"max":{"1min":16.739000000000001,"5min":17.782,"15min":17.782},"last":4.3810000000000002},{"interface":"front","average":{"1min":1.2430000000000001,"5min":2.1960000000000002,"15min":1.292},"min":{"1min":0.36199999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.6449999999999996,"5min":13.446999999999999,"15min":13.446999999999999},"last":0.44500000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.8330000000000002,"5min":2.2290000000000001,"15min":1.5149999999999999},"min":{"1min":0.45500000000000002,"5min":0.36299999999999999,"15min":0.32600000000000001},"max":{"1min":8.8520000000000003,"5min":13.473000000000001,"15min":13.473000000000001},"last":0.5},{"interface":"front","average":{"1min":3.387,"5min":2.3340000000000001,"15min":1.546},"min":{"1min":0.36199999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":16.768000000000001,"5min":16.768000000000001,"15min":16.768000000000001},"last":0.52500000000000002}]}]},{"osd":3,"up_from":0,"seq":111669149845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953617,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1348792,"kb_used_data":1047472,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92396360,"statfs":{"total":95995035648,"available":94613872640,"internally_reserved":0,"allocated":1072611328,"data_stored":1071956639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":2.9140000000000001,"5min":1.7889999999999999,"15min":1.2170000000000001},"min":{"1min":0.20899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":38.179000000000002},"last":0.97399999999999998},{"interface":"front","average":{"1min":3.351,"5min":1.9199999999999999,"15min":1.258},"min":{"1min":0.23599999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.6760000000000002,"5min":20.244,"15min":38.151000000000003},"last":0.215}]},{"osd":1,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0510000000000002,"5min":2.004,"15min":1.1000000000000001},"min":{"1min":0.251,"5min":0.218,"15min":0.218},"max":{"1min":11.077999999999999,"5min":11.077999999999999,"15min":11.077999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":2.8069999999999999,"5min":2.0259999999999998,"15min":1.121},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"last":1.01}]},{"osd":2,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.1419999999999999,"15min":1.161},"min":{"1min":0.25800000000000001,"5min":0.252,"15min":0.22500000000000001},"max":{"1min":9.5790000000000006,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":4.2370000000000001},{"interface":"front","average":{"1min":3.2759999999999998,"5min":2.129,"15min":1.1830000000000001},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":9.6370000000000005,"5min":10.106,"15min":10.106},"last":0.245}]},{"osd":3,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":3.177,"5min":2.153,"15min":1.488},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.26700000000000002},"max":{"1min":9.6969999999999992,"5min":20.271000000000001,"15min":38.170999999999999},"last":2.3599999999999999},{"interface":"front","average":{"1min":3.496,"5min":1.9339999999999999,"15min":1.339},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32000000000000001},"max":{"1min":10.865,"5min":10.865,"15min":38.311999999999998},"last":0.95899999999999996}]},{"osd":5,"last update":"Mon Feb 12 17:00:04 2024","interfaces":[{"interface":"back","average":{"1min":6.6989999999999998,"5min":3.9649999999999999,"15min":1.589},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":42.420000000000002,"5min":79.573999999999998,"15min":79.573999999999998},"last":0.94199999999999995},{"interface":"front","average":{"1min":7.8550000000000004,"5min":3.964,"15min":1.575},"min":{"1min":0.622,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":45.204999999999998,"5min":80.497,"15min":80.497},"last":4.2069999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":4.4020000000000001,"5min":3.3130000000000002,"15min":1.3799999999999999},"min":{"1min":0.373,"5min":0.27000000000000002,"15min":0.255},"max":{"1min":10.49,"5min":104.907,"15min":104.907},"last":0.83999999999999997},{"interface":"front","average":{"1min":5.5149999999999997,"5min":3.5910000000000002,"15min":1.4850000000000001},"min":{"1min":0.54100000000000004,"5min":0.28699999999999998,"15min":0.26500000000000001},"max":{"1min":13.739000000000001,"5min":105.395,"15min":105.395},"last":2.427}]},{"osd":7,"last update":"Mon Feb 12 16:59:47 2024","interfaces":[{"interface":"back","average":{"1min":3.6819999999999999,"5min":2.1219999999999999,"15min":1.004},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":11.786,"5min":12.311999999999999,"15min":12.311999999999999},"last":6.5220000000000002},{"interface":"front","average":{"1min":5.7000000000000002,"5min":2.7170000000000001,"15min":1.2270000000000001},"min":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.28799999999999998},"max":{"1min":21.079000000000001,"5min":27.773,"15min":27.773},"last":0.81699999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757389,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509476,"kb_used_data":1199964,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92235676,"statfs":{"total":95995035648,"available":94449332224,"internally_reserved":0,"allocated":1228763136,"data_stored":1228064540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.8050000000000002,"5min":1.72,"15min":1.1839999999999999},"min":{"1min":0.29399999999999998,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":8.7530000000000001,"5min":8.7530000000000001,"15min":36.880000000000003},"last":2.7280000000000002},{"interface":"front","average":{"1min":3.4359999999999999,"5min":2.411,"15min":1.417},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.9779999999999998,"5min":10.752000000000001,"15min":36.954000000000001},"last":2.6110000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.1699999999999999,"5min":2.198,"15min":1.202},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.8380000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.1659999999999999,"5min":2.3490000000000002,"15min":1.327},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9459999999999997,"5min":16.712,"15min":16.712},"last":2.6339999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.2629999999999999,"5min":2.5009999999999999,"15min":1.399},"min":{"1min":0.40000000000000002,"5min":0.247,"15min":0.23499999999999999},"max":{"1min":8.8019999999999996,"5min":16.940999999999999,"15min":18.218},"last":2.754},{"interface":"front","average":{"1min":2.3879999999999999,"5min":2.0960000000000001,"15min":1.2889999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":6.819,"5min":17.103999999999999,"15min":18.327000000000002},"last":0.498}]},{"osd":3,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":1.9319999999999999,"15min":1.321},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.8230000000000004,"5min":8.8230000000000004,"15min":18.251000000000001},"last":5.2670000000000003},{"interface":"front","average":{"1min":2.3620000000000001,"5min":1.7490000000000001,"15min":1.2290000000000001},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.7720000000000002,"5min":7.048,"15min":20.263000000000002},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":4.2220000000000004,"5min":2.0569999999999999,"15min":0.98299999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":10.021000000000001,"5min":11.880000000000001,"15min":11.880000000000001},"last":9.5619999999999994},{"interface":"front","average":{"1min":4.7709999999999999,"5min":2.601,"15min":1.161},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":2.7010000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:59:28 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":1.6140000000000001,"15min":0.88900000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.9499999999999993,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":2.6739999999999999},{"interface":"front","average":{"1min":2.9980000000000002,"5min":1.8360000000000001,"15min":0.93999999999999995},"min":{"1min":0.32400000000000001,"5min":0.30299999999999999,"15min":0.26700000000000002},"max":{"1min":7.5789999999999997,"5min":22.035,"15min":22.035},"last":7.2380000000000004}]},{"osd":7,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":1.889,"15min":0.97899999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":6.7999999999999998,"5min":9.484,"15min":9.484},"last":0.54100000000000004},{"interface":"front","average":{"1min":3.7490000000000001,"5min":2.04,"15min":1.044},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":12.253,"5min":12.253,"15min":12.253},"last":9.5850000000000009}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":848691200,"data_stored":848691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1252392960,"data_stored":1252392960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1070600192,"data_stored":1070596120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1062600704,"data_stored":1062600704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1199312896,"data_stored":1199308824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":887357440,"data_stored":887357440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":743510016,"data_stored":743505944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:00:19.609 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:00:19.611 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:00:19.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:19.759 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:19.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:18.555327+0000 mgr.y (mgr.24481) 406 : cluster [DBG] pgmap v529: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 935 op/s; 8073/76488 objects misplaced (10.555%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:00:19.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:18.979986+0000 mon.a (mon.0) 1199 : audit [DBG] from='client.? v1:172.21.15.33:0/740696924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:20.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:18.555327+0000 mgr.y (mgr.24481) 406 : cluster [DBG] pgmap v529: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 935 op/s; 8073/76488 objects misplaced (10.555%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:00:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:18.979986+0000 mon.a (mon.0) 1199 : audit [DBG] from='client.? v1:172.21.15.33:0/740696924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:20.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:18.555327+0000 mgr.y (mgr.24481) 406 : cluster [DBG] pgmap v529: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 935 op/s; 8073/76488 objects misplaced (10.555%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:00:20.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:18.979986+0000 mon.a (mon.0) 1199 : audit [DBG] from='client.? v1:172.21.15.33:0/740696924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:20.055 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:00:20.055 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:00:20.055 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:00:20.055 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:00:20.055 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:00:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:20.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:20.199 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:20.226 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:20.246 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:20.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T17:00:20.803 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:20.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:20.907 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:20.974 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:21.004 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:21.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:21.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:00:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:20] "GET /metrics HTTP/1.1" 200 175966 "" "Prometheus/2.33.4" 2024-02-12T17:00:21.109 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:21.145 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:21.250 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:21.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:21.290 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:21.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:21.388 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:21.433 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:21.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:21.547 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:21.650 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:21.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:21.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:20.555867+0000 mgr.y (mgr.24481) 407 : cluster [DBG] pgmap v530: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 979 op/s; 7864/73605 objects misplaced (10.684%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:21.757 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:21.801 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:21.885 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:20.555867+0000 mgr.y (mgr.24481) 407 : cluster [DBG] pgmap v530: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 979 op/s; 7864/73605 objects misplaced (10.684%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:22.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:20.555867+0000 mgr.y (mgr.24481) 407 : cluster [DBG] pgmap v530: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 979 op/s; 7864/73605 objects misplaced (10.684%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:22.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:00:22.459 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:22.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:22.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:22.625 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:00:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:00:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:22.689 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:22.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:22.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:22.795 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:22.844 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:23.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:23.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:23.304 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:23.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:23.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:23.361 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:00:23.361 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:00:23.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:00:23.361 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:00:23.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.361 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220600 7.0 7:02c49166:::benchmark_data_smithi033_81060_object110148:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-02-12T17:00:23.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.423888+0000", 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.933138058, 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0364286869999999, 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220600 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423888+0000", 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423888+0000", 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423885+0000", 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0101e-05 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423895+0000", 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7076999999999999e-05 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423912+0000", 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8669999999999994e-06 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423920+0000", 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2386999999999998e-05 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.423972+0000", 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013168699999999999 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.424104+0000", 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000158741 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,3", 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.424263+0000", 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028780569999999998 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.427141+0000", 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0050549430000000001 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.432196+0000", 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1647999999999999e-05 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.432207+0000", 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028020624 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460228+0000", 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8412999999999995e-05 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460316+0000", 2024-02-12T17:00:23.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0364286869999999 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220610 7.0 7:099dd978:::benchmark_data_smithi033_81060_object110158:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.440693+0000", 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.916332307, 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0445725020000001, 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:23.367 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220610 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440693+0000", 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440693+0000", 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440691+0000", 2024-02-12T17:00:23.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2689999999999999e-06 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440699+0000", 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6155e-05 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440716+0000", 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2880000000000004e-06 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440723+0000", 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5838999999999999e-05 2024-02-12T17:00:23.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440769+0000", 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8959999999999992e-06 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.440778+0000", 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0195559999999999 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460334+0000", 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4116000000000006e-05 2024-02-12T17:00:23.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460418+0000", 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9007999999999998e-05 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.460517+0000", 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010719841000000001 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471237+0000", 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013909055 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485146+0000", 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3446000000000003e-05 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485199+0000", 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9681000000000002e-05 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485239+0000", 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7188000000000001e-05 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485266+0000", 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0445725020000001 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220613 7.0 7:04831311:::benchmark_data_smithi033_81060_object110161:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.444906+0000", 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.912120109, 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.040521985, 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:23.373 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220613 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444906+0000", 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444906+0000", 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444904+0000", 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5519999999999999e-06 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444911+0000", 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2004000000000001e-05 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444923+0000", 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.815e-06 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444930+0000", 2024-02-12T17:00:23.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1479e-05 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444972+0000", 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6400000000000002e-06 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.444977+0000", 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0176013660000001 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.462579+0000", 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000110575 2024-02-12T17:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.462689+0000", 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015559300000000001 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.462845+0000", 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0084006129999999995 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471245+0000", 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014058726000000001 2024-02-12T17:00:23.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485304+0000", 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4366e-05 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485368+0000", 2024-02-12T17:00:23.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4146999999999997e-05 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485403+0000", 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5015999999999999e-05 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485428+0000", 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.040521985 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220649 7.0 7:0e13de3b:::benchmark_data_smithi033_81060_object110197:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.474384+0000", 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.88264140800001, 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0112301079999999, 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220649 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474384+0000", 2024-02-12T17:00:23.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474384+0000", 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474382+0000", 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8539999999999997e-06 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474390+0000", 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3152999999999999e-05 2024-02-12T17:00:23.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474403+0000", 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3549999999999999e-06 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474410+0000", 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9561000000000002e-05 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474460+0000", 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0509999999999999e-05 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.474470+0000", 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99010147599999998 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.464572+0000", 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010268900000000001 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.464674+0000", 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011038 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.464785+0000", 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064674779999999996 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471252+0000", 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014089404 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485342+0000", 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018344800000000001 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485525+0000", 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6623999999999999e-05 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485572+0000", 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765999999999997e-05 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485614+0000", 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0112301079999999 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.385 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220655 7.0 7:094c2cc9:::benchmark_data_smithi033_81060_object110203:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.480491+0000", 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.87653478799999, 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.005281632, 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220655 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480491+0000", 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480491+0000", 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480489+0000", 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5780000000000001e-06 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480496+0000", 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0825000000000001e-05 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480507+0000", 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5570000000000001e-06 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480514+0000", 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2985000000000002e-05 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480567+0000", 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.003e-06 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.480576+0000", 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98622844899999995 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.466804+0000", 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012368399999999999 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.466928+0000", 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000128603 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.467056+0000", 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042049790000000002 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.471261+0000", 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014414544 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485676+0000", 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7676999999999999e-05 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485713+0000", 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5408000000000001e-05 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485739+0000", 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3747000000000001e-05 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485773+0000", 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.005281632 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220656 7.0 7:07768b72:::benchmark_data_smithi033_81060_object110204:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.481291+0000", 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.87573514900001, 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.004739396, 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:23.392 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220656 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481291+0000", 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481291+0000", 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481289+0000", 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6919999999999999e-06 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481295+0000", 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6899999999999998e-06 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481304+0000", 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4210000000000003e-06 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481309+0000", 2024-02-12T17:00:23.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6100000000000002e-05 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481355+0000", 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6790000000000005e-06 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.481363+0000", 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98715179399999997 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.468515+0000", 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6761000000000005e-05 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.468602+0000", 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4424000000000003e-05 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.468686+0000", 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062958279999999998 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.474982+0000", 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010832078 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485814+0000", 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012336199999999999 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485937+0000", 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7234999999999997e-05 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485995+0000", 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5515000000000001e-05 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486030+0000", 2024-02-12T17:00:23.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.004739396 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:220706 7.0 7:06895523:::benchmark_data_smithi033_81060_object110254:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:56:09.524105+0000", 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.832920852, 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96202424399999997, 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:23.398 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220706 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524105+0000", 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524105+0000", 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524103+0000", 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3710000000000002e-06 2024-02-12T17:00:23.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524110+0000", 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1291000000000001e-05 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524122+0000", 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0849999999999996e-06 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524127+0000", 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9991999999999998e-05 2024-02-12T17:00:23.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524187+0000", 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8079999999999999e-06 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:09.524196+0000", 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94564106599999997 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.469837+0000", 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113767 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.469951+0000", 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6594000000000005e-05 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.470038+0000", 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092142419999999992 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.479252+0000", 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065906769999999996 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.485843+0000", 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021390100000000001 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486057+0000", 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8380000000000002e-05 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486095+0000", 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4137999999999997e-05 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:56:10.486129+0000", 2024-02-12T17:00:23.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96202424399999997 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85071 8.3 8:cbed544b:::benchmark_data_smithi033_105085_object85070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e207)", 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.169767+0000", 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 157.187258493, 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.086660556, 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.404 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85071 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169767+0000", 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169767+0000", 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999919 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169759+0000", 2024-02-12T17:00:23.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1277999999999999e-05 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169811+0000", 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2286000000000001e-05 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169823+0000", 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1550000000000007e-06 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169831+0000", 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0877999999999997e-05 2024-02-12T17:00:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169872+0000", 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0469999999999996e-06 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169879+0000", 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99304094300000001 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162920+0000", 2024-02-12T17:00:23.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5599999999999994e-06 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162928+0000", 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0707229229999999 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.233651+0000", 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020893259999999999 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.235741+0000", 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000485759 2024-02-12T17:00:23.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,6", 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.236226+0000", 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014595634999999999 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.250822+0000", 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3120999999999997e-05 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.250855+0000", 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002971998 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253827+0000", 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024091210000000002 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.256236+0000", 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7637000000000003e-05 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.256304+0000", 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012393300000000001 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.256428+0000", 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.086660556 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85082 8.0 8:07fce887:::benchmark_data_smithi033_105085_object85081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.188697+0000", 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 157.168328624, 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1147261230000001, 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85082 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188697+0000", 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188697+0000", 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188695+0000", 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8671999999999997e-05 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188754+0000", 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3957999999999999e-05 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188768+0000", 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3999999999999997e-06 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188774+0000", 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8887e-05 2024-02-12T17:00:23.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188833+0000", 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.85e-06 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188842+0000", 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97349081199999998 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162333+0000", 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0910000000000001e-05 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162343+0000", 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.114529681 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.276873+0000", 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000146785 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.277020+0000", 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047932699999999999 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.277499+0000", 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013512467 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.291012+0000", 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0050795270000000003 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.296091+0000", 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013251560000000001 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.297416+0000", 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058099359999999999 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303226+0000", 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3219999999999994e-05 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303290+0000", 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013372899999999999 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303423+0000", 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1147261230000001 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85085 8.3 8:c095f601:::benchmark_data_smithi033_105085_object85084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.199890+0000", 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 157.15713597199999, 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0657017529999999, 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85085 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199890+0000", 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199890+0000", 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199888+0000", 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3984000000000001e-05 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199972+0000", 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0386e-05 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199983+0000", 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6250000000000001e-06 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199989+0000", 2024-02-12T17:00:23.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0538000000000002e-05 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.200050+0000", 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6120000000000005e-06 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.200059+0000", 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96288834499999998 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162948+0000", 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9970000000000001e-06 2024-02-12T17:00:23.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162955+0000", 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.084958568 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.247913+0000", 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015372300000000001 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248067+0000", 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022328699999999999 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,6", 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248290+0000", 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026044319999999998 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.250895+0000", 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00055957100000000003 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.251454+0000", 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024053130000000001 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253860+0000", 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011626877000000001 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.265486+0000", 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1525e-05 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.265528+0000", 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3641999999999998e-05 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.265592+0000", 2024-02-12T17:00:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0657017529999999 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85090 8.0 8:00c4cb6e:::benchmark_data_smithi033_105085_object85089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.236897+0000", 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 157.12012865299999, 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.071817995, 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85090 2024-02-12T17:00:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236897+0000", 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236897+0000", 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236895+0000", 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6634000000000001e-05 2024-02-12T17:00:23.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236942+0000", 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193999999999999e-05 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236952+0000", 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9799999999999998e-06 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236957+0000", 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7144000000000001e-05 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.237004+0000", 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0240000000000003e-06 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.237010+0000", 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.92535760499999997 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162368+0000", 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6899999999999992e-06 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162375+0000", 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.123740913 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.286116+0000", 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000141197 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.286258+0000", 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018553800000000001 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T17:00:23.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.286443+0000", 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0046879440000000003 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.291131+0000", 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052767229999999997 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.296408+0000", 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070267460000000004 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303435+0000", 2024-02-12T17:00:23.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051164299999999999 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.308551+0000", 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2330000000000002e-05 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.308603+0000", 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111811 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.308715+0000", 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.071817995 2024-02-12T17:00:23.431 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85096 8.0 8:0b744df4:::benchmark_data_smithi033_105085_object85095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.253913+0000", 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 157.10311240499999, 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.065851597, 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85096 2024-02-12T17:00:23.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253913+0000", 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253913+0000", 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253911+0000", 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4963000000000002e-05 2024-02-12T17:00:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253966+0000", 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2769999999999994e-06 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253975+0000", 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9950000000000002e-06 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253981+0000", 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.333e-05 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.254025+0000", 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2459999999999997e-06 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.254032+0000", 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.90836611599999995 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162398+0000", 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5900000000000002e-06 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162406+0000", 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.126113074 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.288519+0000", 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013297999999999999 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.288652+0000", 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019030100000000001 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T17:00:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.288842+0000", 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002323276 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.291165+0000", 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053399099999999998 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.296505+0000", 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069363690000000004 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303441+0000", 2024-02-12T17:00:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016137531 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.319579+0000", 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6953000000000002e-05 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.319636+0000", 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000129016 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.319765+0000", 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.065851597 2024-02-12T17:00:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146034 8.0 8:0210f4aa:::benchmark_data_smithi033_105085_object34376:head [delete] snapc 0=[] ondisk+write+known_if_redirected e256)", 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.200477+0000", 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.156549115000004, 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1397366739999999, 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146034 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200477+0000", 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200477+0000", 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200476+0000", 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.525e-06 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200479+0000", 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0209999999999999e-06 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200484+0000", 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.703e-06 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200487+0000", 2024-02-12T17:00:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028473560000000001 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.203335+0000", 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8774000000000002e-05 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.203414+0000", 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5388000000000002e-05 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.203509+0000", 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079368089999999995 2024-02-12T17:00:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.211446+0000", 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177801 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.211624+0000", 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000399214 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.212023+0000", 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013163431 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.225186+0000", 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2659e-05 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.225199+0000", 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.114932679 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340132+0000", 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1829000000000005e-05 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340213+0000", 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1397366739999999 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146067 8.3 8:c3c181c2:::benchmark_data_smithi033_105085_object34409:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.273536+0000", 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.083490062999999, 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1236668910000001, 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146067 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273536+0000", 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273536+0000", 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273534+0000", 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2813000000000001e-05 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273547+0000", 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2418999999999999e-05 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273559+0000", 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4520000000000002e-06 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273566+0000", 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1288999999999997e-05 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273607+0000", 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1859999999999992e-06 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273615+0000", 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0830521850000001 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.356667+0000", 2024-02-12T17:00:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2581e-05 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.356760+0000", 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00033919000000000001 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.357099+0000", 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010171671 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367271+0000", 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052693300000000005 2024-02-12T17:00:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367798+0000", 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367836+0000", 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036862040000000002 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371522+0000", 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025472465999999999 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.396994+0000", 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8136e-05 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397052+0000", 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015023900000000001 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397203+0000", 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1236668910000001 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146069 8.3 8:ccab45b3:::benchmark_data_smithi033_105085_object34411:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.277018+0000", 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.080007464999994, 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.120418567, 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146069 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277018+0000", 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277018+0000", 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277016+0000", 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1019999999999999e-06 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277024+0000", 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2464e-05 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277036+0000", 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3720000000000001e-06 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277042+0000", 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900999999999999e-05 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277093+0000", 2024-02-12T17:00:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3749999999999992e-06 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277103+0000", 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0849561649999999 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.362059+0000", 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5056999999999996e-05 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.362134+0000", 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9012999999999995e-05 2024-02-12T17:00:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.362233+0000", 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0050455099999999996 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367278+0000", 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037674560000000002 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371046+0000", 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000510445 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371556+0000", 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057177419999999996 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377274+0000", 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019977175999999999 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397251+0000", 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0614e-05 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397302+0000", 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000135062 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397437+0000", 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.120418567 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146070 8.3 8:c3719ad9:::benchmark_data_smithi033_105085_object34412:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.277252+0000", 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.079773496000001, 2024-02-12T17:00:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1204189790000001, 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146070 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277252+0000", 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277252+0000", 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277251+0000", 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6280000000000002e-06 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277257+0000", 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0902e-05 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277268+0000", 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9390000000000002e-06 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277273+0000", 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7271999999999998e-05 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277311+0000", 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8779999999999997e-06 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277316+0000", 2024-02-12T17:00:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0858321420000001 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.363148+0000", 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114474 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.363262+0000", 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014188800000000001 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.363404+0000", 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038811050000000001 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367285+0000", 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037957440000000002 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371081+0000", 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00058387300000000005 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371665+0000", 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056816669999999996 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377347+0000", 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020137176 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397484+0000", 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397531+0000", 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014016300000000001 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397671+0000", 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1204189790000001 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146071 8.3 8:c37e2b9c:::benchmark_data_smithi033_105085_object34413:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.278557+0000", 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.078468443000006, 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.11933178, 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146071 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278557+0000", 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278557+0000", 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278556+0000", 2024-02-12T17:00:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278561+0000", 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4619999999999999e-06 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278568+0000", 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.506e-06 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278572+0000", 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3637999999999999e-05 2024-02-12T17:00:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278605+0000", 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0549999999999996e-06 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278611+0000", 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.086035463 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.364647+0000", 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0767999999999997e-05 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.364718+0000", 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9730999999999999e-05 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.364807+0000", 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024845140000000002 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367292+0000", 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043985839999999997 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371690+0000", 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00045211699999999998 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.372143+0000", 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052212309999999998 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377364+0000", 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020349597000000001 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397713+0000", 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3578999999999997e-05 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397757+0000", 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013217799999999999 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397889+0000", 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.11933178 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146074 8.3 8:cb2d083d:::benchmark_data_smithi033_105085_object34416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.280792+0000", 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.076234209999996, 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1173190850000001, 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146074 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280792+0000", 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280792+0000", 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280791+0000", 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1910000000000002e-06 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280795+0000", 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0710000000000002e-06 2024-02-12T17:00:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280800+0000", 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2890000000000002e-06 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280803+0000", 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4401999999999999e-05 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280848+0000", 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2479999999999996e-06 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280856+0000", 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.085028415 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.365884+0000", 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112866 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.365997+0000", 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015273899999999999 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:00:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.366150+0000", 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020551240000000002 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.368205+0000", 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040942050000000001 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.372299+0000", 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047219779999999999 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377021+0000", 2024-02-12T17:00:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00036072399999999998 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377382+0000", 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020553024 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397935+0000", 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4613000000000001e-05 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397979+0000", 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013121699999999999 2024-02-12T17:00:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.398111+0000", 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1173190850000001 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146076 8.0 8:0aca71f6:::benchmark_data_smithi033_105085_object34418:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.283134+0000", 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.073892126000004, 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0750004909999999, 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146076 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.283134+0000", 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.283134+0000", 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.283132+0000", 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4169999999999997e-06 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.283138+0000", 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0868e-05 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.283149+0000", 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0719999999999996e-06 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.283155+0000", 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.872e-05 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.283214+0000", 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4499999999999998e-06 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.283221+0000", 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.057015086 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340236+0000", 2024-02-12T17:00:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8389999999999998e-05 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340335+0000", 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000385239 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,1,4", 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340720+0000", 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014698815 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.355419+0000", 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8954e-05 2024-02-12T17:00:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.355458+0000", 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1148e-05 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.355499+0000", 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002477529 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.357977+0000", 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3801999999999998e-05 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.358020+0000", 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113787 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.358134+0000", 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0750004909999999 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146082 8.0 8:0cd08f55:::benchmark_data_smithi033_105085_object34424:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:00:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.289053+0000", 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.067972652999998, 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0778379229999999, 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146082 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.289053+0000", 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.289053+0000", 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.289052+0000", 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8830000000000003e-06 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.289058+0000", 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0335e-05 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.289068+0000", 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.87e-06 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.289073+0000", 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5858999999999997e-05 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.289109+0000", 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8810000000000001e-06 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.289115+0000", 2024-02-12T17:00:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0644123599999999 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.353527+0000", 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892599999999999 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.353656+0000", 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016621499999999999 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,1,4", 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.353822+0000", 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003966486 2024-02-12T17:00:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.357788+0000", 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00035335199999999997 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.358142+0000", 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000392397 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.358534+0000", 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081715590000000001 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.366706+0000", 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8826000000000003e-05 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.366755+0000", 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000136432 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.366891+0000", 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0778379229999999 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:23.491 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:00:23.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:23.556 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:23.787 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:23.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:00:23.896 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:23.899 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:00:23.986 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:24.013 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:22.556289+0000 mgr.y (mgr.24481) 408 : cluster [DBG] pgmap v531: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 575 B/s rd, 0 B/s wr, 434 op/s; 7864/73605 objects misplaced (10.684%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:00:24.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:00:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:00:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:22.556289+0000 mgr.y (mgr.24481) 408 : cluster [DBG] pgmap v531: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 575 B/s rd, 0 B/s wr, 434 op/s; 7864/73605 objects misplaced (10.684%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:00:24.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:22.556289+0000 mgr.y (mgr.24481) 408 : cluster [DBG] pgmap v531: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.5 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 575 B/s rd, 0 B/s wr, 434 op/s; 7864/73605 objects misplaced (10.684%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:00:24.129 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:24.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:24.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:24.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T17:00:24.239 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:24.277 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:24.672 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:24.794 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:00:24.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:23.863982+0000 mgr.y (mgr.24481) 409 : audit [DBG] from='client.15345 v1:172.21.15.33:0/482730209' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:24.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:23.947352+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.33:0/380906160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:24.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:23.947782+0000 mon.a (mon.0) 1200 : audit [INF] from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:24.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:24.009050+0000 mgr.y (mgr.24481) 410 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:24.819 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:23.863982+0000 mgr.y (mgr.24481) 409 : audit [DBG] from='client.15345 v1:172.21.15.33:0/482730209' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:24.819 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:23.947352+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.33:0/380906160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:24.819 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:23.947782+0000 mon.a (mon.0) 1200 : audit [INF] from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:24.819 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:24.009050+0000 mgr.y (mgr.24481) 410 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:25.247 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:23.863982+0000 mgr.y (mgr.24481) 409 : audit [DBG] from='client.15345 v1:172.21.15.33:0/482730209' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:25.247 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:23.947352+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.33:0/380906160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:25.247 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:23.947782+0000 mon.a (mon.0) 1200 : audit [INF] from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:25.247 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:24.009050+0000 mgr.y (mgr.24481) 410 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:25.499 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-02-12T17:00:22.556030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1595283132,"num_objects":24535,"num_object_clones":0,"num_object_copies":73605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7864,"num_objects_unfound":0,"num_objects_dirty":24535,"num_whiteouts":0,"num_read":1287,"num_read_kb":1039,"num_write":199683,"num_write_kb":7145962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38566,"num_bytes_recovered":2509250671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58636,"ondisk_log_size":58636,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":5772640,"kb_used_data":4253368,"kb_used_omap":0,"kb_used_meta":1518848,"kb_avail":462953120,"statfs":{"total":479975178240,"available":474063994880,"internally_reserved":0,"allocated":4355448832,"data_stored":4351978476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1555300352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-252837888,"num_objects":-3858,"num_object_clones":0,"num_object_copies":-11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":-3858,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3858,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":18415616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.897289"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":184,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.674860+0000","last_change":"2024-02-12T16:59:09.246682+0000","last_active":"2024-02-12T17:00:13.674860+0000","last_peered":"2024-02-12T17:00:13.674860+0000","last_clean":"2024-02-12T17:00:13.674860+0000","last_became_active":"2024-02-12T16:59:09.246207+0000","last_became_peered":"2024-02-12T16:59:09.246207+0000","last_unstale":"2024-02-12T17:00:13.674860+0000","last_undegraded":"2024-02-12T17:00:13.674860+0000","last_fullsized":"2024-02-12T17:00:13.674860+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":251,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.672555+0000","last_change":"2024-02-12T16:56:35.853405+0000","last_active":"2024-02-12T17:00:13.672555+0000","last_peered":"2024-02-12T17:00:13.672555+0000","last_clean":"2024-02-12T17:00:13.672555+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T17:00:13.672555+0000","last_undegraded":"2024-02-12T17:00:13.672555+0000","last_fullsized":"2024-02-12T17:00:13.672555+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.853330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.853330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":237,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.660726+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T17:00:13.660726+0000","last_peered":"2024-02-12T17:00:13.660726+0000","last_clean":"2024-02-12T17:00:13.660726+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T17:00:13.660726+0000","last_undegraded":"2024-02-12T17:00:13.660726+0000","last_fullsized":"2024-02-12T17:00:13.660726+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":215,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.673716+0000","last_change":"2024-02-12T16:57:47.192069+0000","last_active":"2024-02-12T17:00:13.673716+0000","last_peered":"2024-02-12T17:00:13.673716+0000","last_clean":"2024-02-12T17:00:13.673716+0000","last_became_active":"2024-02-12T16:57:47.188886+0000","last_became_peered":"2024-02-12T16:57:47.188886+0000","last_unstale":"2024-02-12T17:00:13.673716+0000","last_undegraded":"2024-02-12T17:00:13.673716+0000","last_fullsized":"2024-02-12T17:00:13.673716+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":166,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.672860+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T17:00:13.672860+0000","last_peered":"2024-02-12T17:00:13.672860+0000","last_clean":"2024-02-12T17:00:13.672860+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T17:00:13.672860+0000","last_undegraded":"2024-02-12T17:00:13.672860+0000","last_fullsized":"2024-02-12T17:00:13.672860+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"277'15","reported_seq":320,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.670377+0000","last_change":"2024-02-12T16:59:09.351570+0000","last_active":"2024-02-12T17:00:13.670377+0000","last_peered":"2024-02-12T17:00:13.670377+0000","last_clean":"2024-02-12T17:00:13.670377+0000","last_became_active":"2024-02-12T16:59:09.243960+0000","last_became_peered":"2024-02-12T16:59:09.243960+0000","last_unstale":"2024-02-12T17:00:13.670377+0000","last_undegraded":"2024-02-12T17:00:13.670377+0000","last_fullsized":"2024-02-12T17:00:13.670377+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:31.812644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.812644+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":265,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.673108+0000","last_change":"2024-02-12T16:57:47.176812+0000","last_active":"2024-02-12T17:00:13.673108+0000","last_peered":"2024-02-12T17:00:13.673108+0000","last_clean":"2024-02-12T17:00:13.673108+0000","last_became_active":"2024-02-12T16:57:47.176448+0000","last_became_peered":"2024-02-12T16:57:47.176448+0000","last_unstale":"2024-02-12T17:00:13.673108+0000","last_undegraded":"2024-02-12T17:00:13.673108+0000","last_fullsized":"2024-02-12T17:00:13.673108+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:42.174992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:42.174992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":268,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.680625+0000","last_change":"2024-02-12T16:59:09.248489+0000","last_active":"2024-02-12T17:00:13.680625+0000","last_peered":"2024-02-12T17:00:13.680625+0000","last_clean":"2024-02-12T17:00:13.680625+0000","last_became_active":"2024-02-12T16:59:09.248262+0000","last_became_peered":"2024-02-12T16:59:09.248262+0000","last_unstale":"2024-02-12T17:00:13.680625+0000","last_undegraded":"2024-02-12T17:00:13.680625+0000","last_fullsized":"2024-02-12T17:00:13.680625+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":125,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661210+0000","last_change":"2024-02-12T16:57:47.319602+0000","last_active":"2024-02-12T17:00:13.661210+0000","last_peered":"2024-02-12T17:00:13.661210+0000","last_clean":"2024-02-12T17:00:13.661210+0000","last_became_active":"2024-02-12T16:57:47.186061+0000","last_became_peered":"2024-02-12T16:57:47.186061+0000","last_unstale":"2024-02-12T17:00:13.661210+0000","last_undegraded":"2024-02-12T17:00:13.661210+0000","last_fullsized":"2024-02-12T17:00:13.661210+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":267,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666142+0000","last_change":"2024-02-12T16:58:16.945996+0000","last_active":"2024-02-12T17:00:13.666142+0000","last_peered":"2024-02-12T17:00:13.666142+0000","last_clean":"2024-02-12T17:00:13.666142+0000","last_became_active":"2024-02-12T16:57:47.185903+0000","last_became_peered":"2024-02-12T16:57:47.185903+0000","last_unstale":"2024-02-12T17:00:13.666142+0000","last_undegraded":"2024-02-12T17:00:13.666142+0000","last_fullsized":"2024-02-12T17:00:13.666142+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":251,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.675680+0000","last_change":"2024-02-12T16:56:35.897211+0000","last_active":"2024-02-12T17:00:13.675680+0000","last_peered":"2024-02-12T17:00:13.675680+0000","last_clean":"2024-02-12T17:00:13.675680+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:00:13.675680+0000","last_undegraded":"2024-02-12T17:00:13.675680+0000","last_fullsized":"2024-02-12T17:00:13.675680+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.897088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.897088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":213,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666266+0000","last_change":"2024-02-12T16:57:18.865914+0000","last_active":"2024-02-12T17:00:13.666266+0000","last_peered":"2024-02-12T17:00:13.666266+0000","last_clean":"2024-02-12T17:00:13.666266+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T17:00:13.666266+0000","last_undegraded":"2024-02-12T17:00:13.666266+0000","last_fullsized":"2024-02-12T17:00:13.666266+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:18.865807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.865807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":130,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.677231+0000","last_change":"2024-02-12T16:59:09.260070+0000","last_active":"2024-02-12T17:00:13.677231+0000","last_peered":"2024-02-12T17:00:13.677231+0000","last_clean":"2024-02-12T17:00:13.677231+0000","last_became_active":"2024-02-12T16:59:09.259293+0000","last_became_peered":"2024-02-12T16:59:09.259293+0000","last_unstale":"2024-02-12T17:00:13.677231+0000","last_undegraded":"2024-02-12T17:00:13.677231+0000","last_fullsized":"2024-02-12T17:00:13.677231+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":175,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.675075+0000","last_change":"2024-02-12T16:59:09.276561+0000","last_active":"2024-02-12T17:00:13.675075+0000","last_peered":"2024-02-12T17:00:13.675075+0000","last_clean":"2024-02-12T17:00:13.675075+0000","last_became_active":"2024-02-12T16:59:09.272423+0000","last_became_peered":"2024-02-12T16:59:09.272423+0000","last_unstale":"2024-02-12T17:00:13.675075+0000","last_undegraded":"2024-02-12T17:00:13.675075+0000","last_fullsized":"2024-02-12T17:00:13.675075+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":168,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.679611+0000","last_change":"2024-02-12T16:59:09.890505+0000","last_active":"2024-02-12T17:00:13.679611+0000","last_peered":"2024-02-12T17:00:13.679611+0000","last_clean":"2024-02-12T17:00:13.679611+0000","last_became_active":"2024-02-12T16:59:09.249268+0000","last_became_peered":"2024-02-12T16:59:09.249268+0000","last_unstale":"2024-02-12T17:00:13.679611+0000","last_undegraded":"2024-02-12T17:00:13.679611+0000","last_fullsized":"2024-02-12T17:00:13.679611+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:56:45.033441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.033441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":261,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.678154+0000","last_change":"2024-02-12T16:59:09.270880+0000","last_active":"2024-02-12T17:00:13.678154+0000","last_peered":"2024-02-12T17:00:13.678154+0000","last_clean":"2024-02-12T17:00:13.678154+0000","last_became_active":"2024-02-12T16:59:09.270574+0000","last_became_peered":"2024-02-12T16:59:09.270574+0000","last_unstale":"2024-02-12T17:00:13.678154+0000","last_undegraded":"2024-02-12T17:00:13.678154+0000","last_fullsized":"2024-02-12T17:00:13.678154+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":41,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.676242+0000","last_change":"2024-02-12T16:59:09.248658+0000","last_active":"2024-02-12T17:00:13.676242+0000","last_peered":"2024-02-12T17:00:13.676242+0000","last_clean":"2024-02-12T17:00:13.676242+0000","last_became_active":"2024-02-12T16:59:09.248342+0000","last_became_peered":"2024-02-12T16:59:09.248342+0000","last_unstale":"2024-02-12T17:00:13.676242+0000","last_undegraded":"2024-02-12T17:00:13.676242+0000","last_fullsized":"2024-02-12T17:00:13.676242+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":353,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.662407+0000","last_change":"2024-02-12T16:57:47.210170+0000","last_active":"2024-02-12T17:00:13.662407+0000","last_peered":"2024-02-12T17:00:13.662407+0000","last_clean":"2024-02-12T17:00:13.662407+0000","last_became_active":"2024-02-12T16:57:47.178386+0000","last_became_peered":"2024-02-12T16:57:47.178386+0000","last_unstale":"2024-02-12T17:00:13.662407+0000","last_undegraded":"2024-02-12T17:00:13.662407+0000","last_fullsized":"2024-02-12T17:00:13.662407+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:57:24.750245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.750245+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":91,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.683267+0000","last_change":"2024-02-12T16:57:47.179029+0000","last_active":"2024-02-12T17:00:13.683267+0000","last_peered":"2024-02-12T17:00:13.683267+0000","last_clean":"2024-02-12T17:00:13.683267+0000","last_became_active":"2024-02-12T16:57:47.178240+0000","last_became_peered":"2024-02-12T16:57:47.178240+0000","last_unstale":"2024-02-12T17:00:13.683267+0000","last_undegraded":"2024-02-12T17:00:13.683267+0000","last_fullsized":"2024-02-12T17:00:13.683267+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":44,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666688+0000","last_change":"2024-02-12T16:59:09.242061+0000","last_active":"2024-02-12T17:00:13.666688+0000","last_peered":"2024-02-12T17:00:13.666688+0000","last_clean":"2024-02-12T17:00:13.666688+0000","last_became_active":"2024-02-12T16:59:09.241719+0000","last_became_peered":"2024-02-12T16:59:09.241719+0000","last_unstale":"2024-02-12T17:00:13.666688+0000","last_undegraded":"2024-02-12T17:00:13.666688+0000","last_fullsized":"2024-02-12T17:00:13.666688+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.683916+0000","last_change":"2024-02-12T16:59:09.533676+0000","last_active":"2024-02-12T17:00:13.683916+0000","last_peered":"2024-02-12T17:00:13.683916+0000","last_clean":"2024-02-12T17:00:13.683916+0000","last_became_active":"2024-02-12T16:59:09.259731+0000","last_became_peered":"2024-02-12T16:59:09.259731+0000","last_unstale":"2024-02-12T17:00:13.683916+0000","last_undegraded":"2024-02-12T17:00:13.683916+0000","last_fullsized":"2024-02-12T17:00:13.683916+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":247,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.687518+0000","last_change":"2024-02-12T16:57:37.670804+0000","last_active":"2024-02-12T17:00:13.687518+0000","last_peered":"2024-02-12T17:00:13.687518+0000","last_clean":"2024-02-12T17:00:13.687518+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T17:00:13.687518+0000","last_undegraded":"2024-02-12T17:00:13.687518+0000","last_fullsized":"2024-02-12T17:00:13.687518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:57:37.670713+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.670713+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":253,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.670747+0000","last_change":"2024-02-12T16:57:26.790389+0000","last_active":"2024-02-12T17:00:13.670747+0000","last_peered":"2024-02-12T17:00:13.670747+0000","last_clean":"2024-02-12T17:00:13.670747+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T17:00:13.670747+0000","last_undegraded":"2024-02-12T17:00:13.670747+0000","last_fullsized":"2024-02-12T17:00:13.670747+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":321,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.673021+0000","last_change":"2024-02-12T16:57:47.209446+0000","last_active":"2024-02-12T17:00:13.673021+0000","last_peered":"2024-02-12T17:00:13.673021+0000","last_clean":"2024-02-12T17:00:13.673021+0000","last_became_active":"2024-02-12T16:57:47.189652+0000","last_became_peered":"2024-02-12T16:57:47.189652+0000","last_unstale":"2024-02-12T17:00:13.673021+0000","last_undegraded":"2024-02-12T17:00:13.673021+0000","last_fullsized":"2024-02-12T17:00:13.673021+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":175,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666015+0000","last_change":"2024-02-12T16:59:09.264851+0000","last_active":"2024-02-12T17:00:13.666015+0000","last_peered":"2024-02-12T17:00:13.666015+0000","last_clean":"2024-02-12T17:00:13.666015+0000","last_became_active":"2024-02-12T16:59:09.264513+0000","last_became_peered":"2024-02-12T16:59:09.264513+0000","last_unstale":"2024-02-12T17:00:13.666015+0000","last_undegraded":"2024-02-12T17:00:13.666015+0000","last_fullsized":"2024-02-12T17:00:13.666015+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":251,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.677806+0000","last_change":"2024-02-12T16:59:09.270847+0000","last_active":"2024-02-12T17:00:13.677806+0000","last_peered":"2024-02-12T17:00:13.677806+0000","last_clean":"2024-02-12T17:00:13.677806+0000","last_became_active":"2024-02-12T16:59:09.270516+0000","last_became_peered":"2024-02-12T16:59:09.270516+0000","last_unstale":"2024-02-12T17:00:13.677806+0000","last_undegraded":"2024-02-12T17:00:13.677806+0000","last_fullsized":"2024-02-12T17:00:13.677806+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":41,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.677358+0000","last_change":"2024-02-12T16:59:09.260879+0000","last_active":"2024-02-12T17:00:13.677358+0000","last_peered":"2024-02-12T17:00:13.677358+0000","last_clean":"2024-02-12T17:00:13.677358+0000","last_became_active":"2024-02-12T16:59:09.259058+0000","last_became_peered":"2024-02-12T16:59:09.259058+0000","last_unstale":"2024-02-12T17:00:13.677358+0000","last_undegraded":"2024-02-12T17:00:13.677358+0000","last_fullsized":"2024-02-12T17:00:13.677358+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":242,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.674344+0000","last_change":"2024-02-12T16:59:10.061885+0000","last_active":"2024-02-12T17:00:13.674344+0000","last_peered":"2024-02-12T17:00:13.674344+0000","last_clean":"2024-02-12T17:00:13.674344+0000","last_became_active":"2024-02-12T16:59:09.267628+0000","last_became_peered":"2024-02-12T16:59:09.267628+0000","last_unstale":"2024-02-12T17:00:13.674344+0000","last_undegraded":"2024-02-12T17:00:13.674344+0000","last_fullsized":"2024-02-12T17:00:13.674344+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":245,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.680144+0000","last_change":"2024-02-12T16:57:28.846844+0000","last_active":"2024-02-12T17:00:13.680144+0000","last_peered":"2024-02-12T17:00:13.680144+0000","last_clean":"2024-02-12T17:00:13.680144+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T17:00:13.680144+0000","last_undegraded":"2024-02-12T17:00:13.680144+0000","last_fullsized":"2024-02-12T17:00:13.680144+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":41,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.676051+0000","last_change":"2024-02-12T16:59:09.247624+0000","last_active":"2024-02-12T17:00:13.676051+0000","last_peered":"2024-02-12T17:00:13.676051+0000","last_clean":"2024-02-12T17:00:13.676051+0000","last_became_active":"2024-02-12T16:59:09.247346+0000","last_became_peered":"2024-02-12T16:59:09.247346+0000","last_unstale":"2024-02-12T17:00:13.676051+0000","last_undegraded":"2024-02-12T17:00:13.676051+0000","last_fullsized":"2024-02-12T17:00:13.676051+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.225996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":262,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.676867+0000","last_change":"2024-02-12T16:59:09.265096+0000","last_active":"2024-02-12T17:00:13.676867+0000","last_peered":"2024-02-12T17:00:13.676867+0000","last_clean":"2024-02-12T17:00:13.676867+0000","last_became_active":"2024-02-12T16:59:09.264558+0000","last_became_peered":"2024-02-12T16:59:09.264558+0000","last_unstale":"2024-02-12T17:00:13.676867+0000","last_undegraded":"2024-02-12T17:00:13.676867+0000","last_fullsized":"2024-02-12T17:00:13.676867+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"277'8","reported_seq":74,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.670382+0000","last_change":"2024-02-12T16:59:09.961214+0000","last_active":"2024-02-12T17:00:13.670382+0000","last_peered":"2024-02-12T17:00:13.670382+0000","last_clean":"2024-02-12T17:00:13.670382+0000","last_became_active":"2024-02-12T16:59:09.280196+0000","last_became_peered":"2024-02-12T16:59:09.280196+0000","last_unstale":"2024-02-12T17:00:13.670382+0000","last_undegraded":"2024-02-12T17:00:13.670382+0000","last_fullsized":"2024-02-12T17:00:13.670382+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":166,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666602+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T17:00:13.666602+0000","last_peered":"2024-02-12T17:00:13.666602+0000","last_clean":"2024-02-12T17:00:13.666602+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T17:00:13.666602+0000","last_undegraded":"2024-02-12T17:00:13.666602+0000","last_fullsized":"2024-02-12T17:00:13.666602+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":200,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.674092+0000","last_change":"2024-02-12T16:59:09.263555+0000","last_active":"2024-02-12T17:00:13.674092+0000","last_peered":"2024-02-12T17:00:13.674092+0000","last_clean":"2024-02-12T17:00:13.674092+0000","last_became_active":"2024-02-12T16:59:09.262674+0000","last_became_peered":"2024-02-12T16:59:09.262674+0000","last_unstale":"2024-02-12T17:00:13.674092+0000","last_undegraded":"2024-02-12T17:00:13.674092+0000","last_fullsized":"2024-02-12T17:00:13.674092+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":100,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.682942+0000","last_change":"2024-02-12T16:59:09.258970+0000","last_active":"2024-02-12T17:00:13.682942+0000","last_peered":"2024-02-12T17:00:13.682942+0000","last_clean":"2024-02-12T17:00:13.682942+0000","last_became_active":"2024-02-12T16:59:09.257167+0000","last_became_peered":"2024-02-12T16:59:09.257167+0000","last_unstale":"2024-02-12T17:00:13.682942+0000","last_undegraded":"2024-02-12T17:00:13.682942+0000","last_fullsized":"2024-02-12T17:00:13.682942+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":332,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.682260+0000","last_change":"2024-02-12T16:59:09.950284+0000","last_active":"2024-02-12T17:00:13.682260+0000","last_peered":"2024-02-12T17:00:13.682260+0000","last_clean":"2024-02-12T17:00:13.682260+0000","last_became_active":"2024-02-12T16:59:09.274991+0000","last_became_peered":"2024-02-12T16:59:09.274991+0000","last_unstale":"2024-02-12T17:00:13.682260+0000","last_undegraded":"2024-02-12T17:00:13.682260+0000","last_fullsized":"2024-02-12T17:00:13.682260+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:37.895287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.895287+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":258,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.665727+0000","last_change":"2024-02-12T16:59:09.269463+0000","last_active":"2024-02-12T17:00:13.665727+0000","last_peered":"2024-02-12T17:00:13.665727+0000","last_clean":"2024-02-12T17:00:13.665727+0000","last_became_active":"2024-02-12T16:59:09.269260+0000","last_became_peered":"2024-02-12T16:59:09.269260+0000","last_unstale":"2024-02-12T17:00:13.665727+0000","last_undegraded":"2024-02-12T17:00:13.665727+0000","last_fullsized":"2024-02-12T17:00:13.665727+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":191,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666182+0000","last_change":"2024-02-12T16:57:47.189653+0000","last_active":"2024-02-12T17:00:13.666182+0000","last_peered":"2024-02-12T17:00:13.666182+0000","last_clean":"2024-02-12T17:00:13.666182+0000","last_became_active":"2024-02-12T16:57:47.188909+0000","last_became_peered":"2024-02-12T16:57:47.188909+0000","last_unstale":"2024-02-12T17:00:13.666182+0000","last_undegraded":"2024-02-12T17:00:13.666182+0000","last_fullsized":"2024-02-12T17:00:13.666182+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":139,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.687535+0000","last_change":"2024-02-12T16:59:09.260492+0000","last_active":"2024-02-12T17:00:13.687535+0000","last_peered":"2024-02-12T17:00:13.687535+0000","last_clean":"2024-02-12T17:00:13.687535+0000","last_became_active":"2024-02-12T16:59:09.260045+0000","last_became_peered":"2024-02-12T16:59:09.260045+0000","last_unstale":"2024-02-12T17:00:13.687535+0000","last_undegraded":"2024-02-12T17:00:13.687535+0000","last_fullsized":"2024-02-12T17:00:13.687535+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:39.248562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.248562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":106,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661171+0000","last_change":"2024-02-12T16:59:09.288225+0000","last_active":"2024-02-12T17:00:13.661171+0000","last_peered":"2024-02-12T17:00:13.661171+0000","last_clean":"2024-02-12T17:00:13.661171+0000","last_became_active":"2024-02-12T16:59:09.253231+0000","last_became_peered":"2024-02-12T16:59:09.253231+0000","last_unstale":"2024-02-12T17:00:13.661171+0000","last_undegraded":"2024-02-12T17:00:13.661171+0000","last_fullsized":"2024-02-12T17:00:13.661171+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":211,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.662236+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T17:00:13.662236+0000","last_peered":"2024-02-12T17:00:13.662236+0000","last_clean":"2024-02-12T17:00:13.662236+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T17:00:13.662236+0000","last_undegraded":"2024-02-12T17:00:13.662236+0000","last_fullsized":"2024-02-12T17:00:13.662236+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":291,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.724231+0000","last_change":"2024-02-12T16:57:47.312130+0000","last_active":"2024-02-12T17:00:15.724231+0000","last_peered":"2024-02-12T17:00:15.724231+0000","last_clean":"2024-02-12T17:00:15.724231+0000","last_became_active":"2024-02-12T16:57:47.185109+0000","last_became_peered":"2024-02-12T16:57:47.185109+0000","last_unstale":"2024-02-12T17:00:15.724231+0000","last_undegraded":"2024-02-12T17:00:15.724231+0000","last_fullsized":"2024-02-12T17:00:15.724231+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":255,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666409+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T17:00:13.666409+0000","last_peered":"2024-02-12T17:00:13.666409+0000","last_clean":"2024-02-12T17:00:13.666409+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T17:00:13.666409+0000","last_undegraded":"2024-02-12T17:00:13.666409+0000","last_fullsized":"2024-02-12T17:00:13.666409+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":309,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.682870+0000","last_change":"2024-02-12T16:59:09.524334+0000","last_active":"2024-02-12T17:00:13.682870+0000","last_peered":"2024-02-12T17:00:13.682870+0000","last_clean":"2024-02-12T17:00:13.682870+0000","last_became_active":"2024-02-12T16:59:09.258515+0000","last_became_peered":"2024-02-12T16:59:09.258515+0000","last_unstale":"2024-02-12T17:00:13.682870+0000","last_undegraded":"2024-02-12T17:00:13.682870+0000","last_fullsized":"2024-02-12T17:00:13.682870+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":254,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.672603+0000","last_change":"2024-02-12T16:57:25.835979+0000","last_active":"2024-02-12T17:00:13.672603+0000","last_peered":"2024-02-12T17:00:13.672603+0000","last_clean":"2024-02-12T17:00:13.672603+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T17:00:13.672603+0000","last_undegraded":"2024-02-12T17:00:13.672603+0000","last_fullsized":"2024-02-12T17:00:13.672603+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.835883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.835883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"208'9","reported_seq":395,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.739957+0000","last_change":"2024-02-12T16:57:47.225402+0000","last_active":"2024-02-12T17:00:15.739957+0000","last_peered":"2024-02-12T17:00:15.739957+0000","last_clean":"2024-02-12T17:00:15.739957+0000","last_became_active":"2024-02-12T16:57:47.178688+0000","last_became_peered":"2024-02-12T16:57:47.178688+0000","last_unstale":"2024-02-12T17:00:15.739957+0000","last_undegraded":"2024-02-12T17:00:15.739957+0000","last_fullsized":"2024-02-12T17:00:15.739957+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":257,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.673850+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T17:00:13.673850+0000","last_peered":"2024-02-12T17:00:13.673850+0000","last_clean":"2024-02-12T17:00:13.673850+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T17:00:13.673850+0000","last_undegraded":"2024-02-12T17:00:13.673850+0000","last_fullsized":"2024-02-12T17:00:13.673850+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":145,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.675759+0000","last_change":"2024-02-12T16:57:48.058146+0000","last_active":"2024-02-12T17:00:13.675759+0000","last_peered":"2024-02-12T17:00:13.675759+0000","last_clean":"2024-02-12T17:00:13.675759+0000","last_became_active":"2024-02-12T16:57:47.199140+0000","last_became_peered":"2024-02-12T16:57:47.199140+0000","last_unstale":"2024-02-12T17:00:13.675759+0000","last_undegraded":"2024-02-12T17:00:13.675759+0000","last_fullsized":"2024-02-12T17:00:13.675759+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":253,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.680553+0000","last_change":"2024-02-12T16:59:09.269750+0000","last_active":"2024-02-12T17:00:13.680553+0000","last_peered":"2024-02-12T17:00:13.680553+0000","last_clean":"2024-02-12T17:00:13.680553+0000","last_became_active":"2024-02-12T16:59:09.269414+0000","last_became_peered":"2024-02-12T16:59:09.269414+0000","last_unstale":"2024-02-12T17:00:13.680553+0000","last_undegraded":"2024-02-12T17:00:13.680553+0000","last_fullsized":"2024-02-12T17:00:13.680553+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":214,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.676218+0000","last_change":"2024-02-12T16:59:09.243989+0000","last_active":"2024-02-12T17:00:13.676218+0000","last_peered":"2024-02-12T17:00:13.676218+0000","last_clean":"2024-02-12T17:00:13.676218+0000","last_became_active":"2024-02-12T16:59:09.243603+0000","last_became_peered":"2024-02-12T16:59:09.243603+0000","last_unstale":"2024-02-12T17:00:13.676218+0000","last_undegraded":"2024-02-12T17:00:13.676218+0000","last_fullsized":"2024-02-12T17:00:13.676218+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.218031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.218031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":233,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.679054+0000","last_change":"2024-02-12T16:59:09.519418+0000","last_active":"2024-02-12T17:00:13.679054+0000","last_peered":"2024-02-12T17:00:13.679054+0000","last_clean":"2024-02-12T17:00:13.679054+0000","last_became_active":"2024-02-12T16:59:09.242413+0000","last_became_peered":"2024-02-12T16:59:09.242413+0000","last_unstale":"2024-02-12T17:00:13.679054+0000","last_undegraded":"2024-02-12T17:00:13.679054+0000","last_fullsized":"2024-02-12T17:00:13.679054+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":247,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.662126+0000","last_change":"2024-02-12T16:56:38.699688+0000","last_active":"2024-02-12T17:00:13.662126+0000","last_peered":"2024-02-12T17:00:13.662126+0000","last_clean":"2024-02-12T17:00:13.662126+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T17:00:13.662126+0000","last_undegraded":"2024-02-12T17:00:13.662126+0000","last_fullsized":"2024-02-12T17:00:13.662126+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.699602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":262,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.660026+0000","last_change":"2024-02-12T16:57:47.171068+0000","last_active":"2024-02-12T17:00:13.660026+0000","last_peered":"2024-02-12T17:00:13.660026+0000","last_clean":"2024-02-12T17:00:13.660026+0000","last_became_active":"2024-02-12T16:57:47.170667+0000","last_became_peered":"2024-02-12T16:57:47.170667+0000","last_unstale":"2024-02-12T17:00:13.660026+0000","last_undegraded":"2024-02-12T17:00:13.660026+0000","last_fullsized":"2024-02-12T17:00:13.660026+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":292,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.672815+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T17:00:13.672815+0000","last_peered":"2024-02-12T17:00:13.672815+0000","last_clean":"2024-02-12T17:00:13.672815+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T17:00:13.672815+0000","last_undegraded":"2024-02-12T17:00:13.672815+0000","last_fullsized":"2024-02-12T17:00:13.672815+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"287'12356","reported_seq":13924,"reported_epoch":287,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:18.430026+0000","last_change":"2024-02-12T16:59:10.444155+0000","last_active":"2024-02-12T17:00:18.430026+0000","last_peered":"2024-02-12T17:00:18.430026+0000","last_clean":"2024-02-12T16:59:08.181528+0000","last_became_active":"2024-02-12T16:59:10.326952+0000","last_became_peered":"2024-02-12T16:59:10.326952+0000","last_unstale":"2024-02-12T17:00:18.430026+0000","last_undegraded":"2024-02-12T17:00:18.430026+0000","last_fullsized":"2024-02-12T17:00:18.430026+0000","mapping_epoch":258,"log_start":"253'8622","ondisk_log_start":"253'8622","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3734,"ondisk_log_size":3734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1199,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12356,"num_write_kb":446080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":47251456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":1584}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":179,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.675003+0000","last_change":"2024-02-12T16:59:09.273017+0000","last_active":"2024-02-12T17:00:13.675003+0000","last_peered":"2024-02-12T17:00:13.675003+0000","last_clean":"2024-02-12T17:00:13.675003+0000","last_became_active":"2024-02-12T16:59:09.272706+0000","last_became_peered":"2024-02-12T16:59:09.272706+0000","last_unstale":"2024-02-12T17:00:13.675003+0000","last_undegraded":"2024-02-12T17:00:13.675003+0000","last_fullsized":"2024-02-12T17:00:13.675003+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":278,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.679950+0000","last_change":"2024-02-12T16:57:22.679795+0000","last_active":"2024-02-12T17:00:13.679950+0000","last_peered":"2024-02-12T17:00:13.679950+0000","last_clean":"2024-02-12T17:00:13.679950+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T17:00:13.679950+0000","last_undegraded":"2024-02-12T17:00:13.679950+0000","last_fullsized":"2024-02-12T17:00:13.679950+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:57:22.679705+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.679705+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":157,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.665850+0000","last_change":"2024-02-12T16:59:09.952265+0000","last_active":"2024-02-12T17:00:13.665850+0000","last_peered":"2024-02-12T17:00:13.665850+0000","last_clean":"2024-02-12T17:00:13.665850+0000","last_became_active":"2024-02-12T16:59:09.264482+0000","last_became_peered":"2024-02-12T16:59:09.264482+0000","last_unstale":"2024-02-12T17:00:13.665850+0000","last_undegraded":"2024-02-12T17:00:13.665850+0000","last_fullsized":"2024-02-12T17:00:13.665850+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:22.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.764984+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":276,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.660169+0000","last_change":"2024-02-12T16:58:10.280441+0000","last_active":"2024-02-12T17:00:13.660169+0000","last_peered":"2024-02-12T17:00:13.660169+0000","last_clean":"2024-02-12T17:00:13.660169+0000","last_became_active":"2024-02-12T16:58:10.280137+0000","last_became_peered":"2024-02-12T16:58:10.280137+0000","last_unstale":"2024-02-12T17:00:13.660169+0000","last_undegraded":"2024-02-12T17:00:13.660169+0000","last_fullsized":"2024-02-12T17:00:13.660169+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.647627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.647627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"287'12461","reported_seq":12580,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.489765+0000","last_change":"2024-02-12T16:56:36.446431+0000","last_active":"2024-02-12T17:00:18.489765+0000","last_peered":"2024-02-12T17:00:18.489765+0000","last_clean":"2024-02-12T17:00:18.489765+0000","last_became_active":"2024-02-12T16:56:36.446107+0000","last_became_peered":"2024-02-12T16:56:36.446107+0000","last_unstale":"2024-02-12T17:00:18.489765+0000","last_undegraded":"2024-02-12T17:00:18.489765+0000","last_fullsized":"2024-02-12T17:00:18.489765+0000","mapping_epoch":176,"log_start":"254'8926","ondisk_log_start":"254'8926","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3535,"ondisk_log_size":3535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":4419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":445888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":254,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.665725+0000","last_change":"2024-02-12T16:57:29.934046+0000","last_active":"2024-02-12T17:00:13.665725+0000","last_peered":"2024-02-12T17:00:13.665725+0000","last_clean":"2024-02-12T17:00:13.665725+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T17:00:13.665725+0000","last_undegraded":"2024-02-12T17:00:13.665725+0000","last_fullsized":"2024-02-12T17:00:13.665725+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":91,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666676+0000","last_change":"2024-02-12T16:57:47.187057+0000","last_active":"2024-02-12T17:00:13.666676+0000","last_peered":"2024-02-12T17:00:13.666676+0000","last_clean":"2024-02-12T17:00:13.666676+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:00:13.666676+0000","last_undegraded":"2024-02-12T17:00:13.666676+0000","last_fullsized":"2024-02-12T17:00:13.666676+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.009094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.009094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":145,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.674165+0000","last_change":"2024-02-12T16:57:48.006795+0000","last_active":"2024-02-12T17:00:13.674165+0000","last_peered":"2024-02-12T17:00:13.674165+0000","last_clean":"2024-02-12T17:00:13.674165+0000","last_became_active":"2024-02-12T16:57:47.202531+0000","last_became_peered":"2024-02-12T16:57:47.202531+0000","last_unstale":"2024-02-12T17:00:13.674165+0000","last_undegraded":"2024-02-12T17:00:13.674165+0000","last_fullsized":"2024-02-12T17:00:13.674165+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"261'9","reported_seq":391,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.716999+0000","last_change":"2024-02-12T16:59:09.377161+0000","last_active":"2024-02-12T17:00:15.716999+0000","last_peered":"2024-02-12T17:00:15.716999+0000","last_clean":"2024-02-12T17:00:15.716999+0000","last_became_active":"2024-02-12T16:59:09.270667+0000","last_became_peered":"2024-02-12T16:59:09.270667+0000","last_unstale":"2024-02-12T17:00:15.716999+0000","last_undegraded":"2024-02-12T17:00:15.716999+0000","last_fullsized":"2024-02-12T17:00:15.716999+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"287'12585","reported_seq":12704,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.448953+0000","last_change":"2024-02-12T16:56:36.788519+0000","last_active":"2024-02-12T17:00:18.448953+0000","last_peered":"2024-02-12T17:00:18.448953+0000","last_clean":"2024-02-12T17:00:18.448953+0000","last_became_active":"2024-02-12T16:56:36.788101+0000","last_became_peered":"2024-02-12T16:56:36.788101+0000","last_unstale":"2024-02-12T17:00:18.448953+0000","last_undegraded":"2024-02-12T17:00:18.448953+0000","last_fullsized":"2024-02-12T17:00:18.448953+0000","mapping_epoch":176,"log_start":"254'9069","ondisk_log_start":"254'9069","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3516,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101646336,"num_objects":1551,"num_object_clones":0,"num_object_copies":4653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12585,"num_write_kb":452352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":179,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.678434+0000","last_change":"2024-02-12T16:59:09.265012+0000","last_active":"2024-02-12T17:00:13.678434+0000","last_peered":"2024-02-12T17:00:13.678434+0000","last_clean":"2024-02-12T17:00:13.678434+0000","last_became_active":"2024-02-12T16:59:09.264617+0000","last_became_peered":"2024-02-12T16:59:09.264617+0000","last_unstale":"2024-02-12T17:00:13.678434+0000","last_undegraded":"2024-02-12T17:00:13.678434+0000","last_fullsized":"2024-02-12T17:00:13.678434+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.653301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.653301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":271,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.673918+0000","last_change":"2024-02-12T16:59:09.263209+0000","last_active":"2024-02-12T17:00:13.673918+0000","last_peered":"2024-02-12T17:00:13.673918+0000","last_clean":"2024-02-12T17:00:13.673918+0000","last_became_active":"2024-02-12T16:59:09.261764+0000","last_became_peered":"2024-02-12T16:59:09.261764+0000","last_unstale":"2024-02-12T17:00:13.673918+0000","last_undegraded":"2024-02-12T17:00:13.673918+0000","last_fullsized":"2024-02-12T17:00:13.673918+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"278'53","reported_seq":498,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.738390+0000","last_change":"2024-02-12T16:59:09.306589+0000","last_active":"2024-02-12T17:00:15.738390+0000","last_peered":"2024-02-12T17:00:15.738390+0000","last_clean":"2024-02-12T17:00:15.738390+0000","last_became_active":"2024-02-12T16:59:09.253257+0000","last_became_peered":"2024-02-12T16:59:09.253257+0000","last_unstale":"2024-02-12T17:00:15.738390+0000","last_undegraded":"2024-02-12T17:00:15.738390+0000","last_fullsized":"2024-02-12T17:00:15.738390+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'39","last_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.083656+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":265,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.667003+0000","last_change":"2024-02-12T16:59:09.264990+0000","last_active":"2024-02-12T17:00:13.667003+0000","last_peered":"2024-02-12T17:00:13.667003+0000","last_clean":"2024-02-12T17:00:13.667003+0000","last_became_active":"2024-02-12T16:59:09.264694+0000","last_became_peered":"2024-02-12T16:59:09.264694+0000","last_unstale":"2024-02-12T17:00:13.667003+0000","last_undegraded":"2024-02-12T17:00:13.667003+0000","last_fullsized":"2024-02-12T17:00:13.667003+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:46.719706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:46.719706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"287'12266","reported_seq":15877,"reported_epoch":287,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:18.453287+0000","last_change":"2024-02-12T16:59:09.891651+0000","last_active":"2024-02-12T17:00:18.453287+0000","last_peered":"2024-02-12T17:00:18.453287+0000","last_clean":"2024-02-12T16:57:46.130664+0000","last_became_active":"2024-02-12T16:57:48.242613+0000","last_became_peered":"2024-02-12T16:57:48.242613+0000","last_unstale":"2024-02-12T17:00:18.453287+0000","last_undegraded":"2024-02-12T17:00:18.453287+0000","last_fullsized":"2024-02-12T17:00:18.453287+0000","mapping_epoch":208,"log_start":"253'8417","ondisk_log_start":"253'8417","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3849,"ondisk_log_size":3849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98172928,"num_objects":1498,"num_object_clones":0,"num_object_copies":4494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":836,"num_objects_unfound":0,"num_objects_dirty":1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12266,"num_write_kb":440448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1738,"num_bytes_recovered":113901568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":1498}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":260,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.680802+0000","last_change":"2024-02-12T16:59:09.269787+0000","last_active":"2024-02-12T17:00:13.680802+0000","last_peered":"2024-02-12T17:00:13.680802+0000","last_clean":"2024-02-12T17:00:13.680802+0000","last_became_active":"2024-02-12T16:59:09.269488+0000","last_became_peered":"2024-02-12T16:59:09.269488+0000","last_unstale":"2024-02-12T17:00:13.680802+0000","last_undegraded":"2024-02-12T17:00:13.680802+0000","last_fullsized":"2024-02-12T17:00:13.680802+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":281,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:16.553486+0000","last_change":"2024-02-12T16:59:10.016594+0000","last_active":"2024-02-12T17:00:16.553486+0000","last_peered":"2024-02-12T17:00:16.553486+0000","last_clean":"2024-02-12T17:00:16.553486+0000","last_became_active":"2024-02-12T16:59:09.273056+0000","last_became_peered":"2024-02-12T16:59:09.273056+0000","last_unstale":"2024-02-12T17:00:16.553486+0000","last_undegraded":"2024-02-12T17:00:16.553486+0000","last_fullsized":"2024-02-12T17:00:16.553486+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":155,"num_read_kb":150,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":280,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661538+0000","last_change":"2024-02-12T16:59:09.248652+0000","last_active":"2024-02-12T17:00:13.661538+0000","last_peered":"2024-02-12T17:00:13.661538+0000","last_clean":"2024-02-12T17:00:13.661538+0000","last_became_active":"2024-02-12T16:59:09.248320+0000","last_became_peered":"2024-02-12T16:59:09.248320+0000","last_unstale":"2024-02-12T17:00:13.661538+0000","last_undegraded":"2024-02-12T17:00:13.661538+0000","last_fullsized":"2024-02-12T17:00:13.661538+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":172,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.683695+0000","last_change":"2024-02-12T16:57:47.220507+0000","last_active":"2024-02-12T17:00:13.683695+0000","last_peered":"2024-02-12T17:00:13.683695+0000","last_clean":"2024-02-12T17:00:13.683695+0000","last_became_active":"2024-02-12T16:57:47.190968+0000","last_became_peered":"2024-02-12T16:57:47.190968+0000","last_unstale":"2024-02-12T17:00:13.683695+0000","last_undegraded":"2024-02-12T17:00:13.683695+0000","last_fullsized":"2024-02-12T17:00:13.683695+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":267,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.662595+0000","last_change":"2024-02-12T16:58:18.266322+0000","last_active":"2024-02-12T17:00:13.662595+0000","last_peered":"2024-02-12T17:00:13.662595+0000","last_clean":"2024-02-12T17:00:13.662595+0000","last_became_active":"2024-02-12T16:58:18.265999+0000","last_became_peered":"2024-02-12T16:58:18.265999+0000","last_unstale":"2024-02-12T17:00:13.662595+0000","last_undegraded":"2024-02-12T17:00:13.662595+0000","last_fullsized":"2024-02-12T17:00:13.662595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"287'12314","reported_seq":12449,"reported_epoch":287,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:00:18.455468+0000","last_change":"2024-02-12T17:00:11.190095+0000","last_active":"2024-02-12T17:00:18.455468+0000","last_peered":"2024-02-12T17:00:18.455468+0000","last_clean":"2024-02-12T16:59:08.180231+0000","last_became_active":"2024-02-12T16:59:10.299589+0000","last_became_peered":"2024-02-12T16:59:10.299589+0000","last_unstale":"2024-02-12T17:00:18.455468+0000","last_undegraded":"2024-02-12T17:00:18.455468+0000","last_fullsized":"2024-02-12T17:00:18.455468+0000","mapping_epoch":258,"log_start":"254'8737","ondisk_log_start":"254'8737","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3577,"ondisk_log_size":3577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100007936,"num_objects":1526,"num_object_clones":0,"num_object_copies":4578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1526,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12314,"num_write_kb":442880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1526}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":260,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.684071+0000","last_change":"2024-02-12T16:57:47.191748+0000","last_active":"2024-02-12T17:00:13.684071+0000","last_peered":"2024-02-12T17:00:13.684071+0000","last_clean":"2024-02-12T17:00:13.684071+0000","last_became_active":"2024-02-12T16:57:47.191375+0000","last_became_peered":"2024-02-12T16:57:47.191375+0000","last_unstale":"2024-02-12T17:00:13.684071+0000","last_undegraded":"2024-02-12T17:00:13.684071+0000","last_fullsized":"2024-02-12T17:00:13.684071+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":175,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.674027+0000","last_change":"2024-02-12T16:59:09.263388+0000","last_active":"2024-02-12T17:00:13.674027+0000","last_peered":"2024-02-12T17:00:13.674027+0000","last_clean":"2024-02-12T17:00:13.674027+0000","last_became_active":"2024-02-12T16:59:09.262266+0000","last_became_peered":"2024-02-12T16:59:09.262266+0000","last_unstale":"2024-02-12T17:00:13.674027+0000","last_undegraded":"2024-02-12T17:00:13.674027+0000","last_fullsized":"2024-02-12T17:00:13.674027+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":253,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.676209+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:00:13.676209+0000","last_peered":"2024-02-12T17:00:13.676209+0000","last_clean":"2024-02-12T17:00:13.676209+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:00:13.676209+0000","last_undegraded":"2024-02-12T17:00:13.676209+0000","last_fullsized":"2024-02-12T17:00:13.676209+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":263,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.677705+0000","last_change":"2024-02-12T16:59:09.245920+0000","last_active":"2024-02-12T17:00:13.677705+0000","last_peered":"2024-02-12T17:00:13.677705+0000","last_clean":"2024-02-12T17:00:13.677705+0000","last_became_active":"2024-02-12T16:59:09.245635+0000","last_became_peered":"2024-02-12T16:59:09.245635+0000","last_unstale":"2024-02-12T17:00:13.677705+0000","last_undegraded":"2024-02-12T17:00:13.677705+0000","last_fullsized":"2024-02-12T17:00:13.677705+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":109,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.676985+0000","last_change":"2024-02-12T16:59:10.440537+0000","last_active":"2024-02-12T17:00:13.676985+0000","last_peered":"2024-02-12T17:00:13.676985+0000","last_clean":"2024-02-12T17:00:13.676985+0000","last_became_active":"2024-02-12T16:59:09.249903+0000","last_became_peered":"2024-02-12T16:59:09.249903+0000","last_unstale":"2024-02-12T17:00:13.676985+0000","last_undegraded":"2024-02-12T17:00:13.676985+0000","last_fullsized":"2024-02-12T17:00:13.676985+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:19.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.280894+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":247,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.687225+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T17:00:13.687225+0000","last_peered":"2024-02-12T17:00:13.687225+0000","last_clean":"2024-02-12T17:00:13.687225+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:00:13.687225+0000","last_undegraded":"2024-02-12T17:00:13.687225+0000","last_fullsized":"2024-02-12T17:00:13.687225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"287'12382","reported_seq":24885,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.471093+0000","last_change":"2024-02-12T16:58:49.227803+0000","last_active":"2024-02-12T17:00:18.471093+0000","last_peered":"2024-02-12T17:00:18.471093+0000","last_clean":"2024-02-12T17:00:18.471093+0000","last_became_active":"2024-02-12T16:58:49.217536+0000","last_became_peered":"2024-02-12T16:58:49.217536+0000","last_unstale":"2024-02-12T17:00:18.471093+0000","last_undegraded":"2024-02-12T17:00:18.471093+0000","last_fullsized":"2024-02-12T17:00:18.471093+0000","mapping_epoch":249,"log_start":"253'8737","ondisk_log_start":"253'8737","created":176,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3645,"ondisk_log_size":3645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104595456,"num_objects":1596,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6182,"num_bytes_recovered":405143552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":193,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.687299+0000","last_change":"2024-02-12T16:59:09.252727+0000","last_active":"2024-02-12T17:00:13.687299+0000","last_peered":"2024-02-12T17:00:13.687299+0000","last_clean":"2024-02-12T17:00:13.687299+0000","last_became_active":"2024-02-12T16:59:09.252442+0000","last_became_peered":"2024-02-12T16:59:09.252442+0000","last_unstale":"2024-02-12T17:00:13.687299+0000","last_undegraded":"2024-02-12T17:00:13.687299+0000","last_fullsized":"2024-02-12T17:00:13.687299+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":255,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666510+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T17:00:13.666510+0000","last_peered":"2024-02-12T17:00:13.666510+0000","last_clean":"2024-02-12T17:00:13.666510+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T17:00:13.666510+0000","last_undegraded":"2024-02-12T17:00:13.666510+0000","last_fullsized":"2024-02-12T17:00:13.666510+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":270,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.677662+0000","last_change":"2024-02-12T16:57:48.350132+0000","last_active":"2024-02-12T17:00:13.677662+0000","last_peered":"2024-02-12T17:00:13.677662+0000","last_clean":"2024-02-12T17:00:13.677662+0000","last_became_active":"2024-02-12T16:57:47.200347+0000","last_became_peered":"2024-02-12T16:57:47.200347+0000","last_unstale":"2024-02-12T17:00:13.677662+0000","last_undegraded":"2024-02-12T17:00:13.677662+0000","last_fullsized":"2024-02-12T17:00:13.677662+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":308,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.672750+0000","last_change":"2024-02-12T16:57:47.204108+0000","last_active":"2024-02-12T17:00:13.672750+0000","last_peered":"2024-02-12T17:00:13.672750+0000","last_clean":"2024-02-12T17:00:13.672750+0000","last_became_active":"2024-02-12T16:57:47.184456+0000","last_became_peered":"2024-02-12T16:57:47.184456+0000","last_unstale":"2024-02-12T17:00:13.672750+0000","last_undegraded":"2024-02-12T17:00:13.672750+0000","last_fullsized":"2024-02-12T17:00:13.672750+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":271,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.683992+0000","last_change":"2024-02-12T16:58:14.391518+0000","last_active":"2024-02-12T17:00:13.683992+0000","last_peered":"2024-02-12T17:00:13.683992+0000","last_clean":"2024-02-12T17:00:13.683992+0000","last_became_active":"2024-02-12T16:58:14.391221+0000","last_became_peered":"2024-02-12T16:58:14.391221+0000","last_unstale":"2024-02-12T17:00:13.683992+0000","last_undegraded":"2024-02-12T17:00:13.683992+0000","last_fullsized":"2024-02-12T17:00:13.683992+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"287'12474","reported_seq":33203,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.408610+0000","last_change":"2024-02-12T16:59:46.062924+0000","last_active":"2024-02-12T17:00:18.408610+0000","last_peered":"2024-02-12T17:00:18.408610+0000","last_clean":"2024-02-12T17:00:18.408610+0000","last_became_active":"2024-02-12T16:59:46.029014+0000","last_became_peered":"2024-02-12T16:59:46.029014+0000","last_unstale":"2024-02-12T17:00:18.408610+0000","last_undegraded":"2024-02-12T17:00:18.408610+0000","last_fullsized":"2024-02-12T17:00:18.408610+0000","mapping_epoch":275,"log_start":"253'8779","ondisk_log_start":"253'8779","created":176,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3695,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100139008,"num_objects":1528,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12474,"num_write_kb":448064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10282,"num_bytes_recovered":673841152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":44,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661616+0000","last_change":"2024-02-12T16:59:09.242340+0000","last_active":"2024-02-12T17:00:13.661616+0000","last_peered":"2024-02-12T17:00:13.661616+0000","last_clean":"2024-02-12T17:00:13.661616+0000","last_became_active":"2024-02-12T16:59:09.242012+0000","last_became_peered":"2024-02-12T16:59:09.242012+0000","last_unstale":"2024-02-12T17:00:13.661616+0000","last_undegraded":"2024-02-12T17:00:13.661616+0000","last_fullsized":"2024-02-12T17:00:13.661616+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.053587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.053587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":251,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.680869+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T17:00:13.680869+0000","last_peered":"2024-02-12T17:00:13.680869+0000","last_clean":"2024-02-12T17:00:13.680869+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T17:00:13.680869+0000","last_undegraded":"2024-02-12T17:00:13.680869+0000","last_fullsized":"2024-02-12T17:00:13.680869+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":300,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666323+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T17:00:13.666323+0000","last_peered":"2024-02-12T17:00:13.666323+0000","last_clean":"2024-02-12T17:00:13.666323+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T17:00:13.666323+0000","last_undegraded":"2024-02-12T17:00:13.666323+0000","last_fullsized":"2024-02-12T17:00:13.666323+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":41,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666176+0000","last_change":"2024-02-12T16:59:09.271420+0000","last_active":"2024-02-12T17:00:13.666176+0000","last_peered":"2024-02-12T17:00:13.666176+0000","last_clean":"2024-02-12T17:00:13.666176+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:00:13.666176+0000","last_undegraded":"2024-02-12T17:00:13.666176+0000","last_fullsized":"2024-02-12T17:00:13.666176+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"287'12501","reported_seq":12620,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.521605+0000","last_change":"2024-02-12T16:56:36.789146+0000","last_active":"2024-02-12T17:00:18.521605+0000","last_peered":"2024-02-12T17:00:18.521605+0000","last_clean":"2024-02-12T17:00:18.521605+0000","last_became_active":"2024-02-12T16:56:36.788871+0000","last_became_peered":"2024-02-12T16:56:36.788871+0000","last_unstale":"2024-02-12T17:00:18.521605+0000","last_undegraded":"2024-02-12T17:00:18.521605+0000","last_fullsized":"2024-02-12T17:00:18.521605+0000","mapping_epoch":176,"log_start":"254'8948","ondisk_log_start":"254'8948","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3553,"ondisk_log_size":3553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":4407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12501,"num_write_kb":447040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":178,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661346+0000","last_change":"2024-02-12T16:59:09.242302+0000","last_active":"2024-02-12T17:00:13.661346+0000","last_peered":"2024-02-12T17:00:13.661346+0000","last_clean":"2024-02-12T17:00:13.661346+0000","last_became_active":"2024-02-12T16:59:09.241987+0000","last_became_peered":"2024-02-12T16:59:09.241987+0000","last_unstale":"2024-02-12T17:00:13.661346+0000","last_undegraded":"2024-02-12T17:00:13.661346+0000","last_fullsized":"2024-02-12T17:00:13.661346+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.678248+0000","last_change":"2024-02-12T16:59:09.242865+0000","last_active":"2024-02-12T17:00:13.678248+0000","last_peered":"2024-02-12T17:00:13.678248+0000","last_clean":"2024-02-12T17:00:13.678248+0000","last_became_active":"2024-02-12T16:59:09.242505+0000","last_became_peered":"2024-02-12T16:59:09.242505+0000","last_unstale":"2024-02-12T17:00:13.678248+0000","last_undegraded":"2024-02-12T17:00:13.678248+0000","last_fullsized":"2024-02-12T17:00:13.678248+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":273,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.677853+0000","last_change":"2024-02-12T16:57:48.521837+0000","last_active":"2024-02-12T17:00:13.677853+0000","last_peered":"2024-02-12T17:00:13.677853+0000","last_clean":"2024-02-12T17:00:13.677853+0000","last_became_active":"2024-02-12T16:57:47.199763+0000","last_became_peered":"2024-02-12T16:57:47.199763+0000","last_unstale":"2024-02-12T17:00:13.677853+0000","last_undegraded":"2024-02-12T17:00:13.677853+0000","last_fullsized":"2024-02-12T17:00:13.677853+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":259,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.676426+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T17:00:13.676426+0000","last_peered":"2024-02-12T17:00:13.676426+0000","last_clean":"2024-02-12T17:00:13.676426+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T17:00:13.676426+0000","last_undegraded":"2024-02-12T17:00:13.676426+0000","last_fullsized":"2024-02-12T17:00:13.676426+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"287'12625","reported_seq":12745,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.412346+0000","last_change":"2024-02-12T16:56:36.788810+0000","last_active":"2024-02-12T17:00:18.412346+0000","last_peered":"2024-02-12T17:00:18.412346+0000","last_clean":"2024-02-12T17:00:18.412346+0000","last_became_active":"2024-02-12T16:56:36.788518+0000","last_became_peered":"2024-02-12T16:56:36.788518+0000","last_unstale":"2024-02-12T17:00:18.412346+0000","last_undegraded":"2024-02-12T17:00:18.412346+0000","last_fullsized":"2024-02-12T17:00:18.412346+0000","mapping_epoch":176,"log_start":"254'9112","ondisk_log_start":"254'9112","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3513,"ondisk_log_size":3513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103153688,"num_objects":1575,"num_object_clones":0,"num_object_copies":4725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12625,"num_write_kb":454337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":259,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.675270+0000","last_change":"2024-02-12T16:59:09.340317+0000","last_active":"2024-02-12T17:00:13.675270+0000","last_peered":"2024-02-12T17:00:13.675270+0000","last_clean":"2024-02-12T17:00:13.675270+0000","last_became_active":"2024-02-12T16:59:09.246616+0000","last_became_peered":"2024-02-12T16:59:09.246616+0000","last_unstale":"2024-02-12T17:00:13.675270+0000","last_undegraded":"2024-02-12T17:00:13.675270+0000","last_fullsized":"2024-02-12T17:00:13.675270+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:57:19.955311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.955311+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.683694+0000","last_change":"2024-02-12T16:59:09.268325+0000","last_active":"2024-02-12T17:00:13.683694+0000","last_peered":"2024-02-12T17:00:13.683694+0000","last_clean":"2024-02-12T17:00:13.683694+0000","last_became_active":"2024-02-12T16:59:09.268028+0000","last_became_peered":"2024-02-12T16:59:09.268028+0000","last_unstale":"2024-02-12T17:00:13.683694+0000","last_undegraded":"2024-02-12T17:00:13.683694+0000","last_fullsized":"2024-02-12T17:00:13.683694+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":290,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.673015+0000","last_change":"2024-02-12T16:59:09.996418+0000","last_active":"2024-02-12T17:00:13.673015+0000","last_peered":"2024-02-12T17:00:13.673015+0000","last_clean":"2024-02-12T17:00:13.673015+0000","last_became_active":"2024-02-12T16:59:09.256058+0000","last_became_peered":"2024-02-12T16:59:09.256058+0000","last_unstale":"2024-02-12T17:00:13.673015+0000","last_undegraded":"2024-02-12T17:00:13.673015+0000","last_fullsized":"2024-02-12T17:00:13.673015+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"208'10","reported_seq":145,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.709884+0000","last_change":"2024-02-12T16:57:47.193205+0000","last_active":"2024-02-12T17:00:15.709884+0000","last_peered":"2024-02-12T17:00:15.709884+0000","last_clean":"2024-02-12T17:00:15.709884+0000","last_became_active":"2024-02-12T16:57:47.167628+0000","last_became_peered":"2024-02-12T16:57:47.167628+0000","last_unstale":"2024-02-12T17:00:15.709884+0000","last_undegraded":"2024-02-12T17:00:15.709884+0000","last_fullsized":"2024-02-12T17:00:15.709884+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"287'12341","reported_seq":12460,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.424534+0000","last_change":"2024-02-12T16:56:36.445962+0000","last_active":"2024-02-12T17:00:18.424534+0000","last_peered":"2024-02-12T17:00:18.424534+0000","last_clean":"2024-02-12T17:00:18.424534+0000","last_became_active":"2024-02-12T16:56:36.445638+0000","last_became_peered":"2024-02-12T16:56:36.445638+0000","last_unstale":"2024-02-12T17:00:18.424534+0000","last_undegraded":"2024-02-12T17:00:18.424534+0000","last_fullsized":"2024-02-12T17:00:18.424534+0000","mapping_epoch":176,"log_start":"253'8523","ondisk_log_start":"253'8523","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3818,"ondisk_log_size":3818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":94699520,"num_objects":1445,"num_object_clones":0,"num_object_copies":4335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12341,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":44,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.687211+0000","last_change":"2024-02-12T16:59:09.260336+0000","last_active":"2024-02-12T17:00:13.687211+0000","last_peered":"2024-02-12T17:00:13.687211+0000","last_clean":"2024-02-12T17:00:13.687211+0000","last_became_active":"2024-02-12T16:59:09.259654+0000","last_became_peered":"2024-02-12T16:59:09.259654+0000","last_unstale":"2024-02-12T17:00:13.687211+0000","last_undegraded":"2024-02-12T17:00:13.687211+0000","last_fullsized":"2024-02-12T17:00:13.687211+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"261'14","reported_seq":327,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.742997+0000","last_change":"2024-02-12T16:59:12.225158+0000","last_active":"2024-02-12T17:00:15.742997+0000","last_peered":"2024-02-12T17:00:15.742997+0000","last_clean":"2024-02-12T17:00:15.742997+0000","last_became_active":"2024-02-12T16:59:12.224830+0000","last_became_peered":"2024-02-12T16:59:12.224830+0000","last_unstale":"2024-02-12T17:00:15.742997+0000","last_undegraded":"2024-02-12T17:00:15.742997+0000","last_fullsized":"2024-02-12T17:00:15.742997+0000","mapping_epoch":260,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":239,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.659889+0000","last_change":"2024-02-12T16:57:47.171714+0000","last_active":"2024-02-12T17:00:13.659889+0000","last_peered":"2024-02-12T17:00:13.659889+0000","last_clean":"2024-02-12T17:00:13.659889+0000","last_became_active":"2024-02-12T16:57:47.171128+0000","last_became_peered":"2024-02-12T16:57:47.171128+0000","last_unstale":"2024-02-12T17:00:13.659889+0000","last_undegraded":"2024-02-12T17:00:13.659889+0000","last_fullsized":"2024-02-12T17:00:13.659889+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":254,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.676566+0000","last_change":"2024-02-12T16:59:09.788881+0000","last_active":"2024-02-12T17:00:13.676566+0000","last_peered":"2024-02-12T17:00:13.676566+0000","last_clean":"2024-02-12T17:00:13.676566+0000","last_became_active":"2024-02-12T16:59:09.277264+0000","last_became_peered":"2024-02-12T16:59:09.277264+0000","last_unstale":"2024-02-12T17:00:13.676566+0000","last_undegraded":"2024-02-12T17:00:13.676566+0000","last_fullsized":"2024-02-12T17:00:13.676566+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"287'12499","reported_seq":16324,"reported_epoch":287,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:18.497525+0000","last_change":"2024-02-12T16:59:09.997308+0000","last_active":"2024-02-12T17:00:18.497525+0000","last_peered":"2024-02-12T17:00:18.497525+0000","last_clean":"2024-02-12T16:57:46.091774+0000","last_became_active":"2024-02-12T16:59:09.335728+0000","last_became_peered":"2024-02-12T16:59:09.335728+0000","last_unstale":"2024-02-12T17:00:18.497525+0000","last_undegraded":"2024-02-12T17:00:18.497525+0000","last_fullsized":"2024-02-12T17:00:18.497525+0000","mapping_epoch":257,"log_start":"254'9001","ondisk_log_start":"254'9001","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3498,"ondisk_log_size":3498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101777408,"num_objects":1553,"num_object_clones":0,"num_object_copies":4659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2258,"num_objects_unfound":0,"num_objects_dirty":1553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1726,"num_bytes_recovered":113115136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5,6","objects":1553}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":256,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.683198+0000","last_change":"2024-02-12T16:57:47.177662+0000","last_active":"2024-02-12T17:00:13.683198+0000","last_peered":"2024-02-12T17:00:13.683198+0000","last_clean":"2024-02-12T17:00:13.683198+0000","last_became_active":"2024-02-12T16:57:47.177329+0000","last_became_peered":"2024-02-12T16:57:47.177329+0000","last_unstale":"2024-02-12T17:00:13.683198+0000","last_undegraded":"2024-02-12T17:00:13.683198+0000","last_fullsized":"2024-02-12T17:00:13.683198+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.152265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"208'12","reported_seq":372,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.742861+0000","last_change":"2024-02-12T16:57:46.177864+0000","last_active":"2024-02-12T17:00:15.742861+0000","last_peered":"2024-02-12T17:00:15.742861+0000","last_clean":"2024-02-12T17:00:15.742861+0000","last_became_active":"2024-02-12T16:57:46.177517+0000","last_became_peered":"2024-02-12T16:57:46.177517+0000","last_unstale":"2024-02-12T17:00:15.742861+0000","last_undegraded":"2024-02-12T17:00:15.742861+0000","last_fullsized":"2024-02-12T17:00:15.742861+0000","mapping_epoch":97,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":39,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.674149+0000","last_change":"2024-02-12T16:59:09.268099+0000","last_active":"2024-02-12T17:00:13.674149+0000","last_peered":"2024-02-12T17:00:13.674149+0000","last_clean":"2024-02-12T17:00:13.674149+0000","last_became_active":"2024-02-12T16:59:09.267755+0000","last_became_peered":"2024-02-12T16:59:09.267755+0000","last_unstale":"2024-02-12T17:00:13.674149+0000","last_undegraded":"2024-02-12T17:00:13.674149+0000","last_fullsized":"2024-02-12T17:00:13.674149+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":259,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.675615+0000","last_change":"2024-02-12T16:57:48.169827+0000","last_active":"2024-02-12T17:00:13.675615+0000","last_peered":"2024-02-12T17:00:13.675615+0000","last_clean":"2024-02-12T17:00:13.675615+0000","last_became_active":"2024-02-12T16:57:47.199008+0000","last_became_peered":"2024-02-12T16:57:47.199008+0000","last_unstale":"2024-02-12T17:00:13.675615+0000","last_undegraded":"2024-02-12T17:00:13.675615+0000","last_fullsized":"2024-02-12T17:00:13.675615+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"287'12489","reported_seq":15899,"reported_epoch":287,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:18.453994+0000","last_change":"2024-02-12T16:59:10.062476+0000","last_active":"2024-02-12T17:00:18.453994+0000","last_peered":"2024-02-12T17:00:18.453994+0000","last_clean":"2024-02-12T16:57:46.133812+0000","last_became_active":"2024-02-12T16:59:09.319664+0000","last_became_peered":"2024-02-12T16:59:09.319664+0000","last_unstale":"2024-02-12T17:00:18.453994+0000","last_undegraded":"2024-02-12T17:00:18.453994+0000","last_fullsized":"2024-02-12T17:00:18.453994+0000","mapping_epoch":257,"log_start":"254'8914","ondisk_log_start":"254'8914","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3575,"ondisk_log_size":3575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98631680,"num_objects":1505,"num_object_clones":0,"num_object_copies":4515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1132,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12489,"num_write_kb":447808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":107216896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":1505}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":94,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.662674+0000","last_change":"2024-02-12T16:57:47.176436+0000","last_active":"2024-02-12T17:00:13.662674+0000","last_peered":"2024-02-12T17:00:13.662674+0000","last_clean":"2024-02-12T17:00:13.662674+0000","last_became_active":"2024-02-12T16:57:47.176041+0000","last_became_peered":"2024-02-12T16:57:47.176041+0000","last_unstale":"2024-02-12T17:00:13.662674+0000","last_undegraded":"2024-02-12T17:00:13.662674+0000","last_fullsized":"2024-02-12T17:00:13.662674+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"211'10","reported_seq":404,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.713009+0000","last_change":"2024-02-12T16:57:50.205871+0000","last_active":"2024-02-12T17:00:15.713009+0000","last_peered":"2024-02-12T17:00:15.713009+0000","last_clean":"2024-02-12T17:00:15.713009+0000","last_became_active":"2024-02-12T16:57:50.205330+0000","last_became_peered":"2024-02-12T16:57:50.205330+0000","last_unstale":"2024-02-12T17:00:15.713009+0000","last_undegraded":"2024-02-12T17:00:15.713009+0000","last_fullsized":"2024-02-12T17:00:15.713009+0000","mapping_epoch":61,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":195,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.679890+0000","last_change":"2024-02-12T16:57:47.191627+0000","last_active":"2024-02-12T17:00:13.679890+0000","last_peered":"2024-02-12T17:00:13.679890+0000","last_clean":"2024-02-12T17:00:13.679890+0000","last_became_active":"2024-02-12T16:57:47.191231+0000","last_became_peered":"2024-02-12T16:57:47.191231+0000","last_unstale":"2024-02-12T17:00:13.679890+0000","last_undegraded":"2024-02-12T17:00:13.679890+0000","last_fullsized":"2024-02-12T17:00:13.679890+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":194,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.675385+0000","last_change":"2024-02-12T16:59:09.782291+0000","last_active":"2024-02-12T17:00:13.675385+0000","last_peered":"2024-02-12T17:00:13.675385+0000","last_clean":"2024-02-12T17:00:13.675385+0000","last_became_active":"2024-02-12T16:59:09.268136+0000","last_became_peered":"2024-02-12T16:59:09.268136+0000","last_unstale":"2024-02-12T17:00:13.675385+0000","last_undegraded":"2024-02-12T17:00:13.675385+0000","last_fullsized":"2024-02-12T17:00:13.675385+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"287'12244","reported_seq":15165,"reported_epoch":287,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:18.562631+0000","last_change":"2024-02-12T16:59:09.891108+0000","last_active":"2024-02-12T17:00:18.562631+0000","last_peered":"2024-02-12T17:00:18.562631+0000","last_clean":"2024-02-12T16:57:46.113428+0000","last_became_active":"2024-02-12T16:57:48.262168+0000","last_became_peered":"2024-02-12T16:57:48.262168+0000","last_unstale":"2024-02-12T17:00:18.562631+0000","last_undegraded":"2024-02-12T17:00:18.562631+0000","last_fullsized":"2024-02-12T17:00:18.562631+0000","mapping_epoch":208,"log_start":"253'8636","ondisk_log_start":"253'8636","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3608,"ondisk_log_size":3608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97910784,"num_objects":1494,"num_object_clones":0,"num_object_copies":4482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":913,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12244,"num_write_kb":439616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":91160576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":1494}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":270,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.662159+0000","last_change":"2024-02-12T16:59:09.246938+0000","last_active":"2024-02-12T17:00:13.662159+0000","last_peered":"2024-02-12T17:00:13.662159+0000","last_clean":"2024-02-12T17:00:13.662159+0000","last_became_active":"2024-02-12T16:59:09.246605+0000","last_became_peered":"2024-02-12T16:59:09.246605+0000","last_unstale":"2024-02-12T17:00:13.662159+0000","last_undegraded":"2024-02-12T17:00:13.662159+0000","last_fullsized":"2024-02-12T17:00:13.662159+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":118,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.674648+0000","last_change":"2024-02-12T16:59:09.247764+0000","last_active":"2024-02-12T17:00:13.674648+0000","last_peered":"2024-02-12T17:00:13.674648+0000","last_clean":"2024-02-12T17:00:13.674648+0000","last_became_active":"2024-02-12T16:59:09.247447+0000","last_became_peered":"2024-02-12T16:59:09.247447+0000","last_unstale":"2024-02-12T17:00:13.674648+0000","last_undegraded":"2024-02-12T17:00:13.674648+0000","last_fullsized":"2024-02-12T17:00:13.674648+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":251,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.674929+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T17:00:13.674929+0000","last_peered":"2024-02-12T17:00:13.674929+0000","last_clean":"2024-02-12T17:00:13.674929+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T17:00:13.674929+0000","last_undegraded":"2024-02-12T17:00:13.674929+0000","last_fullsized":"2024-02-12T17:00:13.674929+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":353,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661425+0000","last_change":"2024-02-12T16:59:10.004273+0000","last_active":"2024-02-12T17:00:13.661425+0000","last_peered":"2024-02-12T17:00:13.661425+0000","last_clean":"2024-02-12T17:00:13.661425+0000","last_became_active":"2024-02-12T16:59:09.262475+0000","last_became_peered":"2024-02-12T16:59:09.262475+0000","last_unstale":"2024-02-12T17:00:13.661425+0000","last_undegraded":"2024-02-12T17:00:13.661425+0000","last_fullsized":"2024-02-12T17:00:13.661425+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"287'12411","reported_seq":21323,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.508969+0000","last_change":"2024-02-12T16:59:27.226608+0000","last_active":"2024-02-12T17:00:18.508969+0000","last_peered":"2024-02-12T17:00:18.508969+0000","last_clean":"2024-02-12T17:00:18.508969+0000","last_became_active":"2024-02-12T16:59:27.206365+0000","last_became_peered":"2024-02-12T16:59:27.206365+0000","last_unstale":"2024-02-12T17:00:18.508969+0000","last_undegraded":"2024-02-12T17:00:18.508969+0000","last_fullsized":"2024-02-12T17:00:18.508969+0000","mapping_epoch":265,"log_start":"254'8830","ondisk_log_start":"254'8830","created":176,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3581,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102432768,"num_objects":1563,"num_object_clones":0,"num_object_copies":4689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12411,"num_write_kb":447168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4385,"num_bytes_recovered":287375360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":263,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.677614+0000","last_change":"2024-02-12T16:59:09.242082+0000","last_active":"2024-02-12T17:00:13.677614+0000","last_peered":"2024-02-12T17:00:13.677614+0000","last_clean":"2024-02-12T17:00:13.677614+0000","last_became_active":"2024-02-12T16:59:09.241648+0000","last_became_peered":"2024-02-12T16:59:09.241648+0000","last_unstale":"2024-02-12T17:00:13.677614+0000","last_undegraded":"2024-02-12T17:00:13.677614+0000","last_fullsized":"2024-02-12T17:00:13.677614+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":272,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661108+0000","last_change":"2024-02-12T16:57:58.203843+0000","last_active":"2024-02-12T17:00:13.661108+0000","last_peered":"2024-02-12T17:00:13.661108+0000","last_clean":"2024-02-12T17:00:13.661108+0000","last_became_active":"2024-02-12T16:57:58.203578+0000","last_became_peered":"2024-02-12T16:57:58.203578+0000","last_unstale":"2024-02-12T17:00:13.661108+0000","last_undegraded":"2024-02-12T17:00:13.661108+0000","last_fullsized":"2024-02-12T17:00:13.661108+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":253,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.662308+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T17:00:13.662308+0000","last_peered":"2024-02-12T17:00:13.662308+0000","last_clean":"2024-02-12T17:00:13.662308+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T17:00:13.662308+0000","last_undegraded":"2024-02-12T17:00:13.662308+0000","last_fullsized":"2024-02-12T17:00:13.662308+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":128,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.680070+0000","last_change":"2024-02-12T16:59:10.836208+0000","last_active":"2024-02-12T17:00:13.680070+0000","last_peered":"2024-02-12T17:00:13.680070+0000","last_clean":"2024-02-12T17:00:13.680070+0000","last_became_active":"2024-02-12T16:59:09.273985+0000","last_became_peered":"2024-02-12T16:59:09.273985+0000","last_unstale":"2024-02-12T17:00:13.680070+0000","last_undegraded":"2024-02-12T17:00:13.680070+0000","last_fullsized":"2024-02-12T17:00:13.680070+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:41.290236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.290236+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"287'12425","reported_seq":33038,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.482272+0000","last_change":"2024-02-12T16:59:45.048235+0000","last_active":"2024-02-12T17:00:18.482272+0000","last_peered":"2024-02-12T17:00:18.482272+0000","last_clean":"2024-02-12T17:00:18.482272+0000","last_became_active":"2024-02-12T16:59:45.029644+0000","last_became_peered":"2024-02-12T16:59:45.029644+0000","last_unstale":"2024-02-12T17:00:18.482272+0000","last_undegraded":"2024-02-12T17:00:18.482272+0000","last_fullsized":"2024-02-12T17:00:18.482272+0000","mapping_epoch":274,"log_start":"253'8772","ondisk_log_start":"253'8772","created":176,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3653,"ondisk_log_size":3653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97320960,"num_objects":1485,"num_object_clones":0,"num_object_copies":4455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":445120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10227,"num_bytes_recovered":670236672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":240,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666969+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T17:00:13.666969+0000","last_peered":"2024-02-12T17:00:13.666969+0000","last_clean":"2024-02-12T17:00:13.666969+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T17:00:13.666969+0000","last_undegraded":"2024-02-12T17:00:13.666969+0000","last_fullsized":"2024-02-12T17:00:13.666969+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":276,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661083+0000","last_change":"2024-02-12T16:58:02.376971+0000","last_active":"2024-02-12T17:00:13.661083+0000","last_peered":"2024-02-12T17:00:13.661083+0000","last_clean":"2024-02-12T17:00:13.661083+0000","last_became_active":"2024-02-12T16:58:02.376628+0000","last_became_peered":"2024-02-12T16:58:02.376628+0000","last_unstale":"2024-02-12T17:00:13.661083+0000","last_undegraded":"2024-02-12T17:00:13.661083+0000","last_fullsized":"2024-02-12T17:00:13.661083+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:19.703739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.703739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":217,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.661329+0000","last_change":"2024-02-12T16:59:09.279916+0000","last_active":"2024-02-12T17:00:13.661329+0000","last_peered":"2024-02-12T17:00:13.661329+0000","last_clean":"2024-02-12T17:00:13.661329+0000","last_became_active":"2024-02-12T16:59:09.262401+0000","last_became_peered":"2024-02-12T16:59:09.262401+0000","last_unstale":"2024-02-12T17:00:13.661329+0000","last_undegraded":"2024-02-12T17:00:13.661329+0000","last_fullsized":"2024-02-12T17:00:13.661329+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":293,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.666478+0000","last_change":"2024-02-12T16:57:47.553993+0000","last_active":"2024-02-12T17:00:13.666478+0000","last_peered":"2024-02-12T17:00:13.666478+0000","last_clean":"2024-02-12T17:00:13.666478+0000","last_became_active":"2024-02-12T16:57:47.189672+0000","last_became_peered":"2024-02-12T16:57:47.189672+0000","last_unstale":"2024-02-12T17:00:13.666478+0000","last_undegraded":"2024-02-12T17:00:13.666478+0000","last_fullsized":"2024-02-12T17:00:13.666478+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"287'12596","reported_seq":12715,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:18.498708+0000","last_change":"2024-02-12T16:56:36.445585+0000","last_active":"2024-02-12T17:00:18.498708+0000","last_peered":"2024-02-12T17:00:18.498708+0000","last_clean":"2024-02-12T17:00:18.498708+0000","last_became_active":"2024-02-12T16:56:36.445291+0000","last_became_peered":"2024-02-12T16:56:36.445291+0000","last_unstale":"2024-02-12T17:00:18.498708+0000","last_undegraded":"2024-02-12T17:00:18.498708+0000","last_fullsized":"2024-02-12T17:00:18.498708+0000","mapping_epoch":176,"log_start":"254'9094","ondisk_log_start":"254'9094","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3502,"ondisk_log_size":3502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98172928,"num_objects":1498,"num_object_clones":0,"num_object_copies":4494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":251,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.673784+0000","last_change":"2024-02-12T16:59:09.262958+0000","last_active":"2024-02-12T17:00:13.673784+0000","last_peered":"2024-02-12T17:00:13.673784+0000","last_clean":"2024-02-12T17:00:13.673784+0000","last_became_active":"2024-02-12T16:59:09.261034+0000","last_became_peered":"2024-02-12T16:59:09.261034+0000","last_unstale":"2024-02-12T17:00:13.673784+0000","last_undegraded":"2024-02-12T17:00:13.673784+0000","last_fullsized":"2024-02-12T17:00:13.673784+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":276,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.662402+0000","last_change":"2024-02-12T16:59:09.250623+0000","last_active":"2024-02-12T17:00:13.662402+0000","last_peered":"2024-02-12T17:00:13.662402+0000","last_clean":"2024-02-12T17:00:13.662402+0000","last_became_active":"2024-02-12T16:59:09.250318+0000","last_became_peered":"2024-02-12T16:59:09.250318+0000","last_unstale":"2024-02-12T17:00:13.662402+0000","last_undegraded":"2024-02-12T17:00:13.662402+0000","last_fullsized":"2024-02-12T17:00:13.662402+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":326,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-02-12T17:00:13.670669+0000","last_change":"2024-02-12T16:59:11.094393+0000","last_active":"2024-02-12T17:00:13.670669+0000","last_peered":"2024-02-12T17:00:13.670669+0000","last_clean":"2024-02-12T17:00:13.670669+0000","last_became_active":"2024-02-12T16:59:09.280858+0000","last_became_peered":"2024-02-12T16:59:09.280858+0000","last_unstale":"2024-02-12T17:00:13.670669+0000","last_undegraded":"2024-02-12T17:00:13.670669+0000","last_fullsized":"2024-02-12T17:00:13.670669+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'22","last_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.138758+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"261'13","reported_seq":424,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-02-12T17:00:15.721039+0000","last_change":"2024-02-12T16:59:12.243084+0000","last_active":"2024-02-12T17:00:15.721039+0000","last_peered":"2024-02-12T17:00:15.721039+0000","last_clean":"2024-02-12T17:00:15.721039+0000","last_became_active":"2024-02-12T16:59:12.242710+0000","last_became_peered":"2024-02-12T16:59:12.242710+0000","last_unstale":"2024-02-12T17:00:15.721039+0000","last_undegraded":"2024-02-12T17:00:15.721039+0000","last_fullsized":"2024-02-12T17:00:15.721039+0000","mapping_epoch":260,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1595277336,"num_objects":24343,"num_object_clones":0,"num_object_copies":73029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7864,"num_objects_unfound":0,"num_objects_dirty":24343,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198969,"num_write_kb":7145921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38288,"num_bytes_recovered":2509242368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6242971648,"data_stored":6242959432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57852,"ondisk_log_size":57852,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":155,"num_read_kb":150,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":5408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234115,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939844,"kb_used_data":642620,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92805308,"statfs":{"total":95995035648,"available":95032635392,"internally_reserved":0,"allocated":658042880,"data_stored":657325891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.407,"15min":1.101},"min":{"1min":0.246,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.5339999999999998,"5min":7.3239999999999998,"15min":36.439},"last":16.759},{"interface":"front","average":{"1min":3.089,"5min":1.851,"15min":1.2529999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.987,"5min":11.231999999999999,"15min":36.490000000000002},"last":17.844000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":1.778,"15min":1.01},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.048999999999999,"5min":12.032,"15min":12.032},"last":2.0600000000000001},{"interface":"front","average":{"1min":2.456,"5min":1.6830000000000001,"15min":0.97699999999999998},"min":{"1min":0.251,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.965999999999999,"5min":11.260999999999999,"15min":11.260999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.2350000000000001,"15min":0.874},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.2450000000000001,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.5920000000000001,"5min":1.8959999999999999,"15min":1.131},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":11.071999999999999,"5min":11.071999999999999,"15min":20.338000000000001},"last":2.0960000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.569,"15min":1.161},"min":{"1min":0.28999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.8600000000000003,"5min":9.1069999999999993,"15min":20.433},"last":0.63},{"interface":"front","average":{"1min":2.7040000000000002,"5min":2.0960000000000001,"15min":1.3340000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.092000000000001,"5min":11.092000000000001,"15min":20.393000000000001},"last":2.121}]},{"osd":4,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":1.9239999999999999,"15min":0.97699999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.219},"max":{"1min":5.556,"5min":50.406999999999996,"15min":50.406999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":4.2519999999999998,"5min":2.8119999999999998,"15min":1.27},"min":{"1min":0.29499999999999998,"5min":0.23999999999999999,"15min":0.23400000000000001},"max":{"1min":14.863,"5min":50.168999999999997,"15min":50.168999999999997},"last":2.1440000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.4790000000000001,"15min":0.83299999999999996},"min":{"1min":0.45800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.5910000000000002,"5min":7.4269999999999996,"15min":7.4269999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":4.4210000000000003,"5min":2.2210000000000001,"15min":1.093},"min":{"1min":0.371,"5min":0.27400000000000002,"15min":0.26200000000000001},"max":{"1min":11.696,"5min":12.352,"15min":12.352},"last":2.1899999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.663,"15min":0.90100000000000002},"min":{"1min":0.30399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":10.929,"5min":11.282999999999999,"15min":11.282999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5669999999999999,"15min":0.91100000000000003},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.30199999999999999},"max":{"1min":4.4930000000000003,"5min":12.407999999999999,"15min":12.407999999999999},"last":0.64300000000000002}]}]},{"osd":6,"up_from":0,"seq":193273528456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738533,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1038008,"kb_used_data":732592,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92707144,"statfs":{"total":95995035648,"available":94932115456,"internally_reserved":0,"allocated":750174208,"data_stored":749468602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.9199999999999999,"15min":1.0840000000000001},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.3839999999999999,"5min":23.510999999999999,"15min":23.510999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.639,"15min":1.02},"min":{"1min":0.21299999999999999,"5min":0.193,"15min":0.193},"max":{"1min":5.0449999999999999,"5min":10.186999999999999,"15min":10.186999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.841,"15min":1.3180000000000001},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.20499999999999999},"max":{"1min":11.122999999999999,"5min":16.521000000000001,"15min":16.521000000000001},"last":0.307},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.6100000000000001,"15min":1.1459999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.984,"5min":10.340999999999999,"15min":10.340999999999999},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":2.3029999999999999,"15min":1.3149999999999999},"min":{"1min":0.26300000000000001,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":2.415,"5min":21.100000000000001,"15min":21.100000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":2.0179999999999998,"15min":1.2809999999999999},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":2.5659999999999998,"5min":16.446000000000002,"15min":16.446000000000002},"last":0.38600000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.7130000000000001,"15min":1.125},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":7.2069999999999999,"5min":16.027000000000001,"15min":16.027000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.716,"15min":1.083},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":8.4659999999999993,"5min":10.141,"15min":10.141},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:00:08 2024","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.5070000000000001,"15min":1.456},"min":{"1min":0.308,"5min":0.308,"15min":0.26400000000000001},"max":{"1min":7.1600000000000001,"5min":16.190999999999999,"15min":16.190999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":3.028,"5min":2.2080000000000002,"15min":1.431},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":11.423,"5min":18.170999999999999,"15min":18.170999999999999},"last":0.498}]},{"osd":6,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":2.0640000000000001,"15min":1.2110000000000001},"min":{"1min":0.41599999999999998,"5min":0.32100000000000001,"15min":0.314},"max":{"1min":5.077,"5min":17.907,"15min":17.907},"last":6.8070000000000004},{"interface":"front","average":{"1min":1.7030000000000001,"5min":2.3599999999999999,"15min":1.3600000000000001},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.27900000000000003},"max":{"1min":5.5880000000000001,"5min":21.062000000000001,"15min":21.062000000000001},"last":1.879}]},{"osd":7,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.472,"5min":1.9390000000000001,"15min":1.2669999999999999},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.32400000000000001},"max":{"1min":12.058,"5min":16.175000000000001,"15min":16.175000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":3.79,"5min":2.1589999999999998,"15min":1.274},"min":{"1min":0.45600000000000002,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":34.722999999999999,"5min":34.722999999999999,"15min":34.722999999999999},"last":0.56799999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411482,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1228696,"kb_used_data":923280,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92516456,"statfs":{"total":95995035648,"available":94736850944,"internally_reserved":0,"allocated":945438720,"data_stored":944765692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.9550000000000001,"15min":1.3440000000000001},"min":{"1min":0.192,"5min":0.186,"15min":0.186},"max":{"1min":3.7400000000000002,"5min":15.808,"15min":15.808},"last":1.587},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.756,"15min":1.1899999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.158},"max":{"1min":6.1070000000000002,"5min":13.497,"15min":19.893000000000001},"last":1.7}]},{"osd":1,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":2.0009999999999999,"15min":1.4099999999999999},"min":{"1min":0.23400000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":6.5819999999999999,"5min":17.739000000000001,"15min":17.739000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.9399999999999999,"15min":1.4059999999999999},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":3.7629999999999999,"5min":15.766999999999999,"15min":15.766999999999999},"last":1.619}]},{"osd":3,"last update":"Mon Feb 12 16:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":2.1949999999999998,"15min":1.5389999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23499999999999999},"max":{"1min":3.702,"5min":15.787000000000001,"15min":51.579999999999998},"last":1.647},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.855,"15min":1.2909999999999999},"min":{"1min":0.249,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.8889999999999998,"5min":13.521000000000001,"15min":48.988999999999997},"last":1.7470000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:59:50 2024","interfaces":[{"interface":"back","average":{"1min":2.415,"5min":2.2930000000000001,"15min":1.361},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.26700000000000002},"max":{"1min":7.2789999999999999,"5min":15.737,"15min":15.737},"last":6.0019999999999998},{"interface":"front","average":{"1min":2.4630000000000001,"5min":2.2080000000000002,"15min":1.252},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.26800000000000002},"max":{"1min":7.5449999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":5.9930000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":2.081,"15min":1.4379999999999999},"min":{"1min":0.41499999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":10.407999999999999,"5min":15.699,"15min":15.699},"last":1.6839999999999999},{"interface":"front","average":{"1min":2.9580000000000002,"5min":2.1560000000000001,"15min":1.518},"min":{"1min":0.33600000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":11.999000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":6.5800000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:59:21 2024","interfaces":[{"interface":"back","average":{"1min":3.0529999999999999,"5min":2.754,"15min":1.5600000000000001},"min":{"1min":0.40999999999999998,"5min":0.36399999999999999,"15min":0.31},"max":{"1min":16.739000000000001,"5min":17.782,"15min":17.782},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.2430000000000001,"5min":2.1960000000000002,"15min":1.292},"min":{"1min":0.36199999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.6449999999999996,"5min":13.446999999999999,"15min":13.446999999999999},"last":3.556}]},{"osd":7,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.8330000000000002,"5min":2.2290000000000001,"15min":1.5149999999999999},"min":{"1min":0.45500000000000002,"5min":0.36299999999999999,"15min":0.32600000000000001},"max":{"1min":8.8520000000000003,"5min":13.473000000000001,"15min":13.473000000000001},"last":3.6339999999999999},{"interface":"front","average":{"1min":3.387,"5min":2.3340000000000001,"15min":1.546},"min":{"1min":0.36199999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":16.768000000000001,"5min":16.768000000000001,"15min":16.768000000000001},"last":0.373}]}]},{"osd":3,"up_from":0,"seq":111669149846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953618,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1220240,"kb_used_data":910664,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92524912,"statfs":{"total":95995035648,"available":94745509888,"internally_reserved":0,"allocated":932519936,"data_stored":931854523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":2.9140000000000001,"5min":1.7889999999999999,"15min":1.2170000000000001},"min":{"1min":0.20899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":38.179000000000002},"last":1.736},{"interface":"front","average":{"1min":3.351,"5min":1.9199999999999999,"15min":1.258},"min":{"1min":0.23599999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.6760000000000002,"5min":20.244,"15min":38.151000000000003},"last":3.8959999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0510000000000002,"5min":2.004,"15min":1.1000000000000001},"min":{"1min":0.251,"5min":0.218,"15min":0.218},"max":{"1min":11.077999999999999,"5min":11.077999999999999,"15min":11.077999999999999},"last":2.7280000000000002},{"interface":"front","average":{"1min":2.8069999999999999,"5min":2.0259999999999998,"15min":1.121},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"last":1.7589999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.1419999999999999,"15min":1.161},"min":{"1min":0.25800000000000001,"5min":0.252,"15min":0.22500000000000001},"max":{"1min":9.5790000000000006,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":3.9180000000000001},{"interface":"front","average":{"1min":3.2759999999999998,"5min":2.129,"15min":1.1830000000000001},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":9.6370000000000005,"5min":10.106,"15min":10.106},"last":3.8769999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":3.177,"5min":2.153,"15min":1.488},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.26700000000000002},"max":{"1min":9.6969999999999992,"5min":20.271000000000001,"15min":38.170999999999999},"last":3.8500000000000001},{"interface":"front","average":{"1min":3.496,"5min":1.9339999999999999,"15min":1.339},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32000000000000001},"max":{"1min":10.865,"5min":10.865,"15min":38.311999999999998},"last":4.758}]},{"osd":5,"last update":"Mon Feb 12 17:00:04 2024","interfaces":[{"interface":"back","average":{"1min":6.6989999999999998,"5min":3.9649999999999999,"15min":1.589},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":42.420000000000002,"5min":79.573999999999998,"15min":79.573999999999998},"last":2.7570000000000001},{"interface":"front","average":{"1min":7.8550000000000004,"5min":3.964,"15min":1.575},"min":{"1min":0.622,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":45.204999999999998,"5min":80.497,"15min":80.497},"last":1.706}]},{"osd":6,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":4.4020000000000001,"5min":3.3130000000000002,"15min":1.3799999999999999},"min":{"1min":0.373,"5min":0.27000000000000002,"15min":0.255},"max":{"1min":10.49,"5min":104.907,"15min":104.907},"last":0.54800000000000004},{"interface":"front","average":{"1min":5.5149999999999997,"5min":3.5910000000000002,"15min":1.4850000000000001},"min":{"1min":0.54100000000000004,"5min":0.28699999999999998,"15min":0.26500000000000001},"max":{"1min":13.739000000000001,"5min":105.395,"15min":105.395},"last":1.7869999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:59:47 2024","interfaces":[{"interface":"back","average":{"1min":3.6819999999999999,"5min":2.1219999999999999,"15min":1.004},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":11.786,"5min":12.311999999999999,"15min":12.311999999999999},"last":5.2080000000000002},{"interface":"front","average":{"1min":5.7000000000000002,"5min":2.7170000000000001,"15min":1.2270000000000001},"min":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.28799999999999998},"max":{"1min":21.079000000000001,"5min":27.773,"15min":27.773},"last":0.61299999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757390,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1345852,"kb_used_data":1044212,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92399300,"statfs":{"total":95995035648,"available":94616883200,"internally_reserved":0,"allocated":1069273088,"data_stored":1068563768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.8050000000000002,"5min":1.72,"15min":1.1839999999999999},"min":{"1min":0.29399999999999998,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":8.7530000000000001,"5min":8.7530000000000001,"15min":36.880000000000003},"last":46.365000000000002},{"interface":"front","average":{"1min":3.4359999999999999,"5min":2.411,"15min":1.417},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.9779999999999998,"5min":10.752000000000001,"15min":36.954000000000001},"last":46.134}]},{"osd":1,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.1699999999999999,"5min":2.198,"15min":1.202},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.8380000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.499},{"interface":"front","average":{"1min":3.1659999999999999,"5min":2.3490000000000002,"15min":1.327},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9459999999999997,"5min":16.712,"15min":16.712},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.2629999999999999,"5min":2.5009999999999999,"15min":1.399},"min":{"1min":0.40000000000000002,"5min":0.247,"15min":0.23499999999999999},"max":{"1min":8.8019999999999996,"5min":16.940999999999999,"15min":18.218},"last":6.3230000000000004},{"interface":"front","average":{"1min":2.3879999999999999,"5min":2.0960000000000001,"15min":1.2889999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":6.819,"5min":17.103999999999999,"15min":18.327000000000002},"last":6.3899999999999997}]},{"osd":3,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":1.9319999999999999,"15min":1.321},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.8230000000000004,"5min":8.8230000000000004,"15min":18.251000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":2.3620000000000001,"5min":1.7490000000000001,"15min":1.2290000000000001},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.7720000000000002,"5min":7.048,"15min":20.263000000000002},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":4.2220000000000004,"5min":2.0569999999999999,"15min":0.98299999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":10.021000000000001,"5min":11.880000000000001,"15min":11.880000000000001},"last":0.318},{"interface":"front","average":{"1min":4.7709999999999999,"5min":2.601,"15min":1.161},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":0.40400000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:59:28 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":1.6140000000000001,"15min":0.88900000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.9499999999999993,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.34999999999999998},{"interface":"front","average":{"1min":2.9980000000000002,"5min":1.8360000000000001,"15min":0.93999999999999995},"min":{"1min":0.32400000000000001,"5min":0.30299999999999999,"15min":0.26700000000000002},"max":{"1min":7.5789999999999997,"5min":22.035,"15min":22.035},"last":1.4770000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":1.889,"15min":0.97899999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":6.7999999999999998,"5min":9.484,"15min":9.484},"last":0.52400000000000002},{"interface":"front","average":{"1min":3.7490000000000001,"5min":2.04,"15min":1.044},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":12.253,"5min":12.253,"15min":12.253},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":738525184,"data_stored":738525184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1161822208,"data_stored":1161822208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":926289920,"data_stored":926285848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":920518656,"data_stored":920518656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1037504512,"data_stored":1037500440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":809041920,"data_stored":809041920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":649269248,"data_stored":649265176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:00:25.505 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T17:00:26.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:24.330450+0000 osd.1 (osd.1) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:26.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:24.330503+0000 osd.1 (osd.1) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:26.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:24.556927+0000 mgr.y (mgr.24481) 411 : cluster [DBG] pgmap v532: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.2 GiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 785 op/s; 6287/61638 objects misplaced (10.200%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:00:26.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:24.754946+0000 mon.a (mon.0) 1201 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:26.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:24.780573+0000 mon.a (mon.0) 1202 : audit [INF] from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:26.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:24.780633+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e288: 8 total, 8 up, 5 in 2024-02-12T17:00:26.135 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:24.330450+0000 osd.1 (osd.1) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:26.136 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:24.330503+0000 osd.1 (osd.1) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:26.136 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:24.556927+0000 mgr.y (mgr.24481) 411 : cluster [DBG] pgmap v532: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.2 GiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 785 op/s; 6287/61638 objects misplaced (10.200%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:00:26.136 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:24.754946+0000 mon.a (mon.0) 1201 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:26.136 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:24.780573+0000 mon.a (mon.0) 1202 : audit [INF] from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:26.136 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:24.780633+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e288: 8 total, 8 up, 5 in 2024-02-12T17:00:26.136 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:26.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:24.330450+0000 osd.1 (osd.1) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:24.330503+0000 osd.1 (osd.1) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:24.556927+0000 mgr.y (mgr.24481) 411 : cluster [DBG] pgmap v532: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.2 GiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 785 op/s; 6287/61638 objects misplaced (10.200%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:00:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:24.754946+0000 mon.a (mon.0) 1201 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:24.780573+0000 mon.a (mon.0) 1202 : audit [INF] from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:24.780633+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e288: 8 total, 8 up, 5 in 2024-02-12T17:00:26.255 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:26.321 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:27.177 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:00:27.201 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:27.226 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:00:27.258 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:00:27.283 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:00:27.307 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:00:27.318 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:27.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:27.341 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:00:27.370 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:00:27.416 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:00:27.417 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:00:27.417 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:00:27.418 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:00:27.418 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:00:27.418 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:00:27.418 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:00:27.418 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:00:27.418 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:00:27.427 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:27.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:26.197367+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e289: 8 total, 8 up, 5 in 2024-02-12T17:00:27.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:26.557258+0000 mgr.y (mgr.24481) 412 : cluster [DBG] pgmap v535: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.2 GiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 619 op/s; 6287/61638 objects misplaced (10.200%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:27.465 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:27.152313+0000 mon.b (mon.1) 70 : audit [DBG] from='client.? v1:172.21.15.33:0/581916171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:27.465 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:26.197367+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e289: 8 total, 8 up, 5 in 2024-02-12T17:00:27.465 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:26.557258+0000 mgr.y (mgr.24481) 412 : cluster [DBG] pgmap v535: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.2 GiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 619 op/s; 6287/61638 objects misplaced (10.200%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:27.465 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:27.152313+0000 mon.b (mon.1) 70 : audit [DBG] from='client.? v1:172.21.15.33:0/581916171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:27.480 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:27.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:26.197367+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e289: 8 total, 8 up, 5 in 2024-02-12T17:00:27.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:26.557258+0000 mgr.y (mgr.24481) 412 : cluster [DBG] pgmap v535: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.2 GiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 619 op/s; 6287/61638 objects misplaced (10.200%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:27.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:27.152313+0000 mon.b (mon.1) 70 : audit [DBG] from='client.? v1:172.21.15.33:0/581916171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:27.694 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:27.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:27.797 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:27.874 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:28.095 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:28.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:28.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:28.613 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:00:28.636 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:28.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:28.758 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:29.361 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:00:29.361 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:00:29.643 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:29.684 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:29.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:29.694 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:28.558012+0000 mgr.y (mgr.24481) 413 : cluster [DBG] pgmap v536: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.1 GiB data, 4.3 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s; 4952/52563 objects misplaced (9.421%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T17:00:29.695 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:28.558012+0000 mgr.y (mgr.24481) 413 : cluster [DBG] pgmap v536: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.1 GiB data, 4.3 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s; 4952/52563 objects misplaced (9.421%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T17:00:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:28.558012+0000 mgr.y (mgr.24481) 413 : cluster [DBG] pgmap v536: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1.1 GiB data, 4.3 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s; 4952/52563 objects misplaced (9.421%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T17:00:29.795 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:29.875 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:30.083 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:00:30.083 INFO:teuthology.orchestra.run.smithi033.stdout:{"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":862,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":289,"num_osds":8,"num_up_osds":8,"osd_up_since":1707756570,"num_in_osds":5,"osd_in_since":1707757148,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":132},{"state_name":"active+remapped+backfilling","count":5},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":138,"num_pools":7,"num_objects":17521,"data_bytes":1135613628,"bytes_used":4657315840,"bytes_avail":475317862400,"bytes_total":479975178240,"misplaced_objects":4952,"misplaced_total":52563,"misplaced_ratio":0.094210756615870472,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":4103203,"recovering_keys_per_sec":0,"num_objects_recovered":501,"num_bytes_recovered":32833536,"num_keys_recovered":0,"read_bytes_sec":1279,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":876},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.33:8443/","prometheus":"http://172.21.15.33:9283/"}},"servicemap":{"epoch":13,"modified":"2024-02-12T16:51:04.286395+0000","services":{"rgw":{"daemons":{"summary":"","24397":{"start_epoch":12,"start_stamp":"2024-02-12T16:49:46.308902+0000","gid":24397,"addr":"172.21.15.33:0/1589842856","metadata":{"arch":"x86_64","ceph_release":"pacific","ceph_version":"ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)","ceph_version_short":"16.2.14-521-g0e714d9a","container_hostname":"smithi033","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi033","id":"foo.a","kernel_description":"#1 SMP Thu Jan 18 15:15:29 UTC 2024","kernel_version":"4.18.0-536.el8.x86_64","mem_swap_kb":"0","mem_total_kb":"32543856","num_handles":"1","os":"Linux","pid":"8","realm_id":"","realm_name":"","zone_id":"4799771a-d28e-4dc8-a5a0-179e1408b6e6","zone_name":"default","zonegroup_id":"0384a922-9dfb-46b6-aed7-e430c5071cb8","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{"71dbd923-2c37-43e9-9369-c7b278dc9c1b":{"message":"Global Recovery Event (2m)\n [==========================..] (remaining: 7s)","progress":0.95652174949645996,"add_to_ceph_s":true}}} 2024-02-12T17:00:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:30.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:30.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T17:00:30.236 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:30.237 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:00:30.325 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:30.357 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:30.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:30.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:30.587 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:30.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:30.669 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:30.695 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:30.748 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:30.852 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:30.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:30.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:30.082156+0000 mon.a (mon.0) 1205 : audit [DBG] from='client.? v1:172.21.15.33:0/3146598918' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:00:30.881 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:30.082156+0000 mon.a (mon.0) 1205 : audit [DBG] from='client.? v1:172.21.15.33:0/3146598918' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:00:30.881 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:00:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:30] "GET /metrics HTTP/1.1" 200 175966 "" "Prometheus/2.33.4" 2024-02-12T17:00:30.960 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:30.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:30.082156+0000 mon.a (mon.0) 1205 : audit [DBG] from='client.? v1:172.21.15.33:0/3146598918' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:00:31.001 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:31.299 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-02-12T17:00:31.300 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:00:31.352 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:31.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:31.372 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:31.457 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:31.504 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:31.637 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:30.558449+0000 mgr.y (mgr.24481) 414 : cluster [DBG] pgmap v537: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1022 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 4747/49632 objects misplaced (9.564%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:31.981 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:30.558449+0000 mgr.y (mgr.24481) 414 : cluster [DBG] pgmap v537: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1022 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 4747/49632 objects misplaced (9.564%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:31.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:30.558449+0000 mgr.y (mgr.24481) 414 : cluster [DBG] pgmap v537: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1022 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 4747/49632 objects misplaced (9.564%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:32.571 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:32.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:32.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:32.718 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:32.743 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:00:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:00:32.850 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:00:32.850 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:00:33.578 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:33.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:33.625 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T17:00:33.683 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:33.739 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:33.800 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:33.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:33.867 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:33.912 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:33.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:32.560466+0000 mgr.y (mgr.24481) 415 : cluster [DBG] pgmap v538: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1022 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 500 op/s; 4747/49632 objects misplaced (9.564%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:34.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:32.560466+0000 mgr.y (mgr.24481) 415 : cluster [DBG] pgmap v538: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1022 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 500 op/s; 4747/49632 objects misplaced (9.564%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:34.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:32.560466+0000 mgr.y (mgr.24481) 415 : cluster [DBG] pgmap v538: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 1022 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 500 op/s; 4747/49632 objects misplaced (9.564%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:34.031 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:34.059 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:34.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:34.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:34.223 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:34.334 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:34.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:34.344 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:00:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:00:34.445 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:34.491 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:34.670 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:34.021489+0000 mgr.y (mgr.24481) 416 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:34.671 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:34.166750+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? v1:172.21.15.33:0/1655092954' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:00:34.671 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:34.241869+0000 osd.2 (osd.2) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:34.671 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:34.241918+0000 osd.2 (osd.2) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:34.680 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:00:34.703 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:34.021489+0000 mgr.y (mgr.24481) 416 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:34.703 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:34.166750+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? v1:172.21.15.33:0/1655092954' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:00:34.703 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:34.241869+0000 osd.2 (osd.2) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:34.703 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:34.241918+0000 osd.2 (osd.2) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:34.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:34.021489+0000 mgr.y (mgr.24481) 416 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:34.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:34.166750+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? v1:172.21.15.33:0/1655092954' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:00:34.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:34.241869+0000 osd.2 (osd.2) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:34.704 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:34.241918+0000 osd.2 (osd.2) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:34.746 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:34.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:34.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:34.933 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:35.057 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:35.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:35.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:35.217 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:35.314 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:35.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:35.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:35.468 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:35.636 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:35.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:34.561285+0000 mgr.y (mgr.24481) 417 : cluster [DBG] pgmap v539: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 766 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 828 op/s; 3233/37356 objects misplaced (8.655%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:00:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:34.653583+0000 mon.a (mon.0) 1207 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:34.667381+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? v1:172.21.15.33:0/1655092954' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:00:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:34.667444+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e290: 8 total, 8 up, 5 in 2024-02-12T17:00:35.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:35.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:34.561285+0000 mgr.y (mgr.24481) 417 : cluster [DBG] pgmap v539: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 766 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 828 op/s; 3233/37356 objects misplaced (8.655%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:00:35.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:34.653583+0000 mon.a (mon.0) 1207 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:35.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:34.667381+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? v1:172.21.15.33:0/1655092954' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:00:35.775 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:34.667444+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e290: 8 total, 8 up, 5 in 2024-02-12T17:00:35.775 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:34.561285+0000 mgr.y (mgr.24481) 417 : cluster [DBG] pgmap v539: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 766 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 828 op/s; 3233/37356 objects misplaced (8.655%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:00:35.775 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:34.653583+0000 mon.a (mon.0) 1207 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:00:35.775 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:34.667381+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? v1:172.21.15.33:0/1655092954' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:00:35.775 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:34.667444+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e290: 8 total, 8 up, 5 in 2024-02-12T17:00:35.781 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:35.989 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:35.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:36.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:36.196 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:36.280 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:36.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:36.335 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:00:36.335 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:00:36.384 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:36.608 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:36.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:36.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:36.714 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:36.773 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:36.938 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:00:36.940 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:00:37.047 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:37.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:37.064 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T17:00:37.160 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:37.197 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:37.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:36.203197+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e291: 8 total, 8 up, 5 in 2024-02-12T17:00:37.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:36.561696+0000 mgr.y (mgr.24481) 418 : cluster [DBG] pgmap v542: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 766 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s; 3233/37356 objects misplaced (8.655%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:37.219 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:36.916441+0000 mgr.y (mgr.24481) 419 : audit [DBG] from='client.25252 v1:172.21.15.33:0/2083224189' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:37.219 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:36.203197+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e291: 8 total, 8 up, 5 in 2024-02-12T17:00:37.219 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:36.561696+0000 mgr.y (mgr.24481) 418 : cluster [DBG] pgmap v542: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 766 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s; 3233/37356 objects misplaced (8.655%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:37.219 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:36.916441+0000 mgr.y (mgr.24481) 419 : audit [DBG] from='client.25252 v1:172.21.15.33:0/2083224189' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:37.352 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:37.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:37.456 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:37.488 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:37.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:36.203197+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e291: 8 total, 8 up, 5 in 2024-02-12T17:00:37.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:36.561696+0000 mgr.y (mgr.24481) 418 : cluster [DBG] pgmap v542: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 766 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s; 3233/37356 objects misplaced (8.655%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:37.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:36.916441+0000 mgr.y (mgr.24481) 419 : audit [DBG] from='client.25252 v1:172.21.15.33:0/2083224189' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:37.589 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:37.751 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:37.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:37.785 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":542,"stamp":"2024-02-12T17:00:36.561416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":803411644,"num_objects":12452,"num_object_clones":0,"num_object_copies":37356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3233,"num_objects_unfound":0,"num_objects_dirty":12452,"num_whiteouts":0,"num_read":1302,"num_read_kb":1054,"num_write":211766,"num_write_kb":7145962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39405,"num_bytes_recovered":2564235375,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58819,"ondisk_log_size":58819,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":3746328,"kb_used_data":2243568,"kb_used_omap":0,"kb_used_meta":1502464,"kb_avail":464979432,"statfs":{"total":479975178240,"available":476138938368,"internally_reserved":0,"allocated":2297413632,"data_stored":2293889656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1538523136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-332201984,"num_objects":-5069,"num_object_clones":0,"num_object_copies":-15207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1719,"num_objects_unfound":0,"num_objects_dirty":-5069,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":22151168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003995"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":186,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802605+0000","last_change":"2024-02-12T16:59:09.246682+0000","last_active":"2024-02-12T17:00:24.802605+0000","last_peered":"2024-02-12T17:00:24.802605+0000","last_clean":"2024-02-12T17:00:24.802605+0000","last_became_active":"2024-02-12T16:59:09.246207+0000","last_became_peered":"2024-02-12T16:59:09.246207+0000","last_unstale":"2024-02-12T17:00:24.802605+0000","last_undegraded":"2024-02-12T17:00:24.802605+0000","last_fullsized":"2024-02-12T17:00:24.802605+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":253,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787578+0000","last_change":"2024-02-12T16:56:35.853405+0000","last_active":"2024-02-12T17:00:24.787578+0000","last_peered":"2024-02-12T17:00:24.787578+0000","last_clean":"2024-02-12T17:00:24.787578+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T17:00:24.787578+0000","last_undegraded":"2024-02-12T17:00:24.787578+0000","last_fullsized":"2024-02-12T17:00:24.787578+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.853330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.853330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":239,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783163+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T17:00:24.783163+0000","last_peered":"2024-02-12T17:00:24.783163+0000","last_clean":"2024-02-12T17:00:24.783163+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T17:00:24.783163+0000","last_undegraded":"2024-02-12T17:00:24.783163+0000","last_fullsized":"2024-02-12T17:00:24.783163+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":217,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.792810+0000","last_change":"2024-02-12T16:57:47.192069+0000","last_active":"2024-02-12T17:00:24.792810+0000","last_peered":"2024-02-12T17:00:24.792810+0000","last_clean":"2024-02-12T17:00:24.792810+0000","last_became_active":"2024-02-12T16:57:47.188886+0000","last_became_peered":"2024-02-12T16:57:47.188886+0000","last_unstale":"2024-02-12T17:00:24.792810+0000","last_undegraded":"2024-02-12T17:00:24.792810+0000","last_fullsized":"2024-02-12T17:00:24.792810+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":168,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.788497+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T17:00:24.788497+0000","last_peered":"2024-02-12T17:00:24.788497+0000","last_clean":"2024-02-12T17:00:24.788497+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T17:00:24.788497+0000","last_undegraded":"2024-02-12T17:00:24.788497+0000","last_fullsized":"2024-02-12T17:00:24.788497+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"277'15","reported_seq":322,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.803136+0000","last_change":"2024-02-12T16:59:09.351570+0000","last_active":"2024-02-12T17:00:24.803136+0000","last_peered":"2024-02-12T17:00:24.803136+0000","last_clean":"2024-02-12T17:00:24.803136+0000","last_became_active":"2024-02-12T16:59:09.243960+0000","last_became_peered":"2024-02-12T16:59:09.243960+0000","last_unstale":"2024-02-12T17:00:24.803136+0000","last_undegraded":"2024-02-12T17:00:24.803136+0000","last_fullsized":"2024-02-12T17:00:24.803136+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:31.812644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.812644+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":267,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.799165+0000","last_change":"2024-02-12T16:57:47.176812+0000","last_active":"2024-02-12T17:00:24.799165+0000","last_peered":"2024-02-12T17:00:24.799165+0000","last_clean":"2024-02-12T17:00:24.799165+0000","last_became_active":"2024-02-12T16:57:47.176448+0000","last_became_peered":"2024-02-12T16:57:47.176448+0000","last_unstale":"2024-02-12T17:00:24.799165+0000","last_undegraded":"2024-02-12T17:00:24.799165+0000","last_fullsized":"2024-02-12T17:00:24.799165+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:42.174992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:42.174992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":270,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801301+0000","last_change":"2024-02-12T16:59:09.248489+0000","last_active":"2024-02-12T17:00:24.801301+0000","last_peered":"2024-02-12T17:00:24.801301+0000","last_clean":"2024-02-12T17:00:24.801301+0000","last_became_active":"2024-02-12T16:59:09.248262+0000","last_became_peered":"2024-02-12T16:59:09.248262+0000","last_unstale":"2024-02-12T17:00:24.801301+0000","last_undegraded":"2024-02-12T17:00:24.801301+0000","last_fullsized":"2024-02-12T17:00:24.801301+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":127,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.784624+0000","last_change":"2024-02-12T16:57:47.319602+0000","last_active":"2024-02-12T17:00:24.784624+0000","last_peered":"2024-02-12T17:00:24.784624+0000","last_clean":"2024-02-12T17:00:24.784624+0000","last_became_active":"2024-02-12T16:57:47.186061+0000","last_became_peered":"2024-02-12T16:57:47.186061+0000","last_unstale":"2024-02-12T17:00:24.784624+0000","last_undegraded":"2024-02-12T17:00:24.784624+0000","last_fullsized":"2024-02-12T17:00:24.784624+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":269,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.788880+0000","last_change":"2024-02-12T16:58:16.945996+0000","last_active":"2024-02-12T17:00:24.788880+0000","last_peered":"2024-02-12T17:00:24.788880+0000","last_clean":"2024-02-12T17:00:24.788880+0000","last_became_active":"2024-02-12T16:57:47.185903+0000","last_became_peered":"2024-02-12T16:57:47.185903+0000","last_unstale":"2024-02-12T17:00:24.788880+0000","last_undegraded":"2024-02-12T17:00:24.788880+0000","last_fullsized":"2024-02-12T17:00:24.788880+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.204277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.204277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":253,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.805461+0000","last_change":"2024-02-12T16:56:35.897211+0000","last_active":"2024-02-12T17:00:24.805461+0000","last_peered":"2024-02-12T17:00:24.805461+0000","last_clean":"2024-02-12T17:00:24.805461+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:00:24.805461+0000","last_undegraded":"2024-02-12T17:00:24.805461+0000","last_fullsized":"2024-02-12T17:00:24.805461+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.897088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.897088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":215,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787228+0000","last_change":"2024-02-12T16:57:18.865914+0000","last_active":"2024-02-12T17:00:24.787228+0000","last_peered":"2024-02-12T17:00:24.787228+0000","last_clean":"2024-02-12T17:00:24.787228+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T17:00:24.787228+0000","last_undegraded":"2024-02-12T17:00:24.787228+0000","last_fullsized":"2024-02-12T17:00:24.787228+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:18.865807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.865807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":132,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801575+0000","last_change":"2024-02-12T16:59:09.260070+0000","last_active":"2024-02-12T17:00:24.801575+0000","last_peered":"2024-02-12T17:00:24.801575+0000","last_clean":"2024-02-12T17:00:24.801575+0000","last_became_active":"2024-02-12T16:59:09.259293+0000","last_became_peered":"2024-02-12T16:59:09.259293+0000","last_unstale":"2024-02-12T17:00:24.801575+0000","last_undegraded":"2024-02-12T17:00:24.801575+0000","last_fullsized":"2024-02-12T17:00:24.801575+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":177,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802822+0000","last_change":"2024-02-12T16:59:09.276561+0000","last_active":"2024-02-12T17:00:24.802822+0000","last_peered":"2024-02-12T17:00:24.802822+0000","last_clean":"2024-02-12T17:00:24.802822+0000","last_became_active":"2024-02-12T16:59:09.272423+0000","last_became_peered":"2024-02-12T16:59:09.272423+0000","last_unstale":"2024-02-12T17:00:24.802822+0000","last_undegraded":"2024-02-12T17:00:24.802822+0000","last_fullsized":"2024-02-12T17:00:24.802822+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":170,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.812873+0000","last_change":"2024-02-12T16:59:09.890505+0000","last_active":"2024-02-12T17:00:24.812873+0000","last_peered":"2024-02-12T17:00:24.812873+0000","last_clean":"2024-02-12T17:00:24.812873+0000","last_became_active":"2024-02-12T16:59:09.249268+0000","last_became_peered":"2024-02-12T16:59:09.249268+0000","last_unstale":"2024-02-12T17:00:24.812873+0000","last_undegraded":"2024-02-12T17:00:24.812873+0000","last_fullsized":"2024-02-12T17:00:24.812873+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:56:45.033441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.033441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":263,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.804068+0000","last_change":"2024-02-12T16:59:09.270880+0000","last_active":"2024-02-12T17:00:24.804068+0000","last_peered":"2024-02-12T17:00:24.804068+0000","last_clean":"2024-02-12T17:00:24.804068+0000","last_became_active":"2024-02-12T16:59:09.270574+0000","last_became_peered":"2024-02-12T16:59:09.270574+0000","last_unstale":"2024-02-12T17:00:24.804068+0000","last_undegraded":"2024-02-12T17:00:24.804068+0000","last_fullsized":"2024-02-12T17:00:24.804068+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":43,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.817639+0000","last_change":"2024-02-12T16:59:09.248658+0000","last_active":"2024-02-12T17:00:24.817639+0000","last_peered":"2024-02-12T17:00:24.817639+0000","last_clean":"2024-02-12T17:00:24.817639+0000","last_became_active":"2024-02-12T16:59:09.248342+0000","last_became_peered":"2024-02-12T16:59:09.248342+0000","last_unstale":"2024-02-12T17:00:24.817639+0000","last_undegraded":"2024-02-12T17:00:24.817639+0000","last_fullsized":"2024-02-12T17:00:24.817639+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":355,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783770+0000","last_change":"2024-02-12T16:57:47.210170+0000","last_active":"2024-02-12T17:00:24.783770+0000","last_peered":"2024-02-12T17:00:24.783770+0000","last_clean":"2024-02-12T17:00:24.783770+0000","last_became_active":"2024-02-12T16:57:47.178386+0000","last_became_peered":"2024-02-12T16:57:47.178386+0000","last_unstale":"2024-02-12T17:00:24.783770+0000","last_undegraded":"2024-02-12T17:00:24.783770+0000","last_fullsized":"2024-02-12T17:00:24.783770+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:57:24.750245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.750245+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":93,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.793390+0000","last_change":"2024-02-12T16:57:47.179029+0000","last_active":"2024-02-12T17:00:24.793390+0000","last_peered":"2024-02-12T17:00:24.793390+0000","last_clean":"2024-02-12T17:00:24.793390+0000","last_became_active":"2024-02-12T16:57:47.178240+0000","last_became_peered":"2024-02-12T16:57:47.178240+0000","last_unstale":"2024-02-12T17:00:24.793390+0000","last_undegraded":"2024-02-12T17:00:24.793390+0000","last_fullsized":"2024-02-12T17:00:24.793390+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":46,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.789686+0000","last_change":"2024-02-12T16:59:09.242061+0000","last_active":"2024-02-12T17:00:24.789686+0000","last_peered":"2024-02-12T17:00:24.789686+0000","last_clean":"2024-02-12T17:00:24.789686+0000","last_became_active":"2024-02-12T16:59:09.241719+0000","last_became_peered":"2024-02-12T16:59:09.241719+0000","last_unstale":"2024-02-12T17:00:24.789686+0000","last_undegraded":"2024-02-12T17:00:24.789686+0000","last_fullsized":"2024-02-12T17:00:24.789686+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":98,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801932+0000","last_change":"2024-02-12T16:59:09.533676+0000","last_active":"2024-02-12T17:00:24.801932+0000","last_peered":"2024-02-12T17:00:24.801932+0000","last_clean":"2024-02-12T17:00:24.801932+0000","last_became_active":"2024-02-12T16:59:09.259731+0000","last_became_peered":"2024-02-12T16:59:09.259731+0000","last_unstale":"2024-02-12T17:00:24.801932+0000","last_undegraded":"2024-02-12T17:00:24.801932+0000","last_fullsized":"2024-02-12T17:00:24.801932+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":249,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.805735+0000","last_change":"2024-02-12T16:57:37.670804+0000","last_active":"2024-02-12T17:00:24.805735+0000","last_peered":"2024-02-12T17:00:24.805735+0000","last_clean":"2024-02-12T17:00:24.805735+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T17:00:24.805735+0000","last_undegraded":"2024-02-12T17:00:24.805735+0000","last_fullsized":"2024-02-12T17:00:24.805735+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:57:37.670713+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.670713+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":255,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.792344+0000","last_change":"2024-02-12T16:57:26.790389+0000","last_active":"2024-02-12T17:00:24.792344+0000","last_peered":"2024-02-12T17:00:24.792344+0000","last_clean":"2024-02-12T17:00:24.792344+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T17:00:24.792344+0000","last_undegraded":"2024-02-12T17:00:24.792344+0000","last_fullsized":"2024-02-12T17:00:24.792344+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":323,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.788639+0000","last_change":"2024-02-12T16:57:47.209446+0000","last_active":"2024-02-12T17:00:24.788639+0000","last_peered":"2024-02-12T17:00:24.788639+0000","last_clean":"2024-02-12T17:00:24.788639+0000","last_became_active":"2024-02-12T16:57:47.189652+0000","last_became_peered":"2024-02-12T16:57:47.189652+0000","last_unstale":"2024-02-12T17:00:24.788639+0000","last_undegraded":"2024-02-12T17:00:24.788639+0000","last_fullsized":"2024-02-12T17:00:24.788639+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":177,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787290+0000","last_change":"2024-02-12T16:59:09.264851+0000","last_active":"2024-02-12T17:00:24.787290+0000","last_peered":"2024-02-12T17:00:24.787290+0000","last_clean":"2024-02-12T17:00:24.787290+0000","last_became_active":"2024-02-12T16:59:09.264513+0000","last_became_peered":"2024-02-12T16:59:09.264513+0000","last_unstale":"2024-02-12T17:00:24.787290+0000","last_undegraded":"2024-02-12T17:00:24.787290+0000","last_fullsized":"2024-02-12T17:00:24.787290+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":253,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.800961+0000","last_change":"2024-02-12T16:59:09.270847+0000","last_active":"2024-02-12T17:00:24.800961+0000","last_peered":"2024-02-12T17:00:24.800961+0000","last_clean":"2024-02-12T17:00:24.800961+0000","last_became_active":"2024-02-12T16:59:09.270516+0000","last_became_peered":"2024-02-12T16:59:09.270516+0000","last_unstale":"2024-02-12T17:00:24.800961+0000","last_undegraded":"2024-02-12T17:00:24.800961+0000","last_fullsized":"2024-02-12T17:00:24.800961+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:05.141249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:05.141249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":43,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801695+0000","last_change":"2024-02-12T16:59:09.260879+0000","last_active":"2024-02-12T17:00:24.801695+0000","last_peered":"2024-02-12T17:00:24.801695+0000","last_clean":"2024-02-12T17:00:24.801695+0000","last_became_active":"2024-02-12T16:59:09.259058+0000","last_became_peered":"2024-02-12T16:59:09.259058+0000","last_unstale":"2024-02-12T17:00:24.801695+0000","last_undegraded":"2024-02-12T17:00:24.801695+0000","last_fullsized":"2024-02-12T17:00:24.801695+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":244,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801479+0000","last_change":"2024-02-12T16:59:10.061885+0000","last_active":"2024-02-12T17:00:24.801479+0000","last_peered":"2024-02-12T17:00:24.801479+0000","last_clean":"2024-02-12T17:00:24.801479+0000","last_became_active":"2024-02-12T16:59:09.267628+0000","last_became_peered":"2024-02-12T16:59:09.267628+0000","last_unstale":"2024-02-12T17:00:24.801479+0000","last_undegraded":"2024-02-12T17:00:24.801479+0000","last_fullsized":"2024-02-12T17:00:24.801479+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":247,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802306+0000","last_change":"2024-02-12T16:57:28.846844+0000","last_active":"2024-02-12T17:00:24.802306+0000","last_peered":"2024-02-12T17:00:24.802306+0000","last_clean":"2024-02-12T17:00:24.802306+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T17:00:24.802306+0000","last_undegraded":"2024-02-12T17:00:24.802306+0000","last_fullsized":"2024-02-12T17:00:24.802306+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":43,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.803723+0000","last_change":"2024-02-12T16:59:09.247624+0000","last_active":"2024-02-12T17:00:24.803723+0000","last_peered":"2024-02-12T17:00:24.803723+0000","last_clean":"2024-02-12T17:00:24.803723+0000","last_became_active":"2024-02-12T16:59:09.247346+0000","last_became_peered":"2024-02-12T16:59:09.247346+0000","last_unstale":"2024-02-12T17:00:24.803723+0000","last_undegraded":"2024-02-12T17:00:24.803723+0000","last_fullsized":"2024-02-12T17:00:24.803723+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.225996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":264,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.795653+0000","last_change":"2024-02-12T16:59:09.265096+0000","last_active":"2024-02-12T17:00:24.795653+0000","last_peered":"2024-02-12T17:00:24.795653+0000","last_clean":"2024-02-12T17:00:24.795653+0000","last_became_active":"2024-02-12T16:59:09.264558+0000","last_became_peered":"2024-02-12T16:59:09.264558+0000","last_unstale":"2024-02-12T17:00:24.795653+0000","last_undegraded":"2024-02-12T17:00:24.795653+0000","last_fullsized":"2024-02-12T17:00:24.795653+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"277'8","reported_seq":76,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.791978+0000","last_change":"2024-02-12T16:59:09.961214+0000","last_active":"2024-02-12T17:00:24.791978+0000","last_peered":"2024-02-12T17:00:24.791978+0000","last_clean":"2024-02-12T17:00:24.791978+0000","last_became_active":"2024-02-12T16:59:09.280196+0000","last_became_peered":"2024-02-12T16:59:09.280196+0000","last_unstale":"2024-02-12T17:00:24.791978+0000","last_undegraded":"2024-02-12T17:00:24.791978+0000","last_fullsized":"2024-02-12T17:00:24.791978+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":168,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.789669+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T17:00:24.789669+0000","last_peered":"2024-02-12T17:00:24.789669+0000","last_clean":"2024-02-12T17:00:24.789669+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T17:00:24.789669+0000","last_undegraded":"2024-02-12T17:00:24.789669+0000","last_fullsized":"2024-02-12T17:00:24.789669+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":202,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.793232+0000","last_change":"2024-02-12T16:59:09.263555+0000","last_active":"2024-02-12T17:00:24.793232+0000","last_peered":"2024-02-12T17:00:24.793232+0000","last_clean":"2024-02-12T17:00:24.793232+0000","last_became_active":"2024-02-12T16:59:09.262674+0000","last_became_peered":"2024-02-12T16:59:09.262674+0000","last_unstale":"2024-02-12T17:00:24.793232+0000","last_undegraded":"2024-02-12T17:00:24.793232+0000","last_fullsized":"2024-02-12T17:00:24.793232+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":102,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.793045+0000","last_change":"2024-02-12T16:59:09.258970+0000","last_active":"2024-02-12T17:00:24.793045+0000","last_peered":"2024-02-12T17:00:24.793045+0000","last_clean":"2024-02-12T17:00:24.793045+0000","last_became_active":"2024-02-12T16:59:09.257167+0000","last_became_peered":"2024-02-12T16:59:09.257167+0000","last_unstale":"2024-02-12T17:00:24.793045+0000","last_undegraded":"2024-02-12T17:00:24.793045+0000","last_fullsized":"2024-02-12T17:00:24.793045+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":334,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.817107+0000","last_change":"2024-02-12T16:59:09.950284+0000","last_active":"2024-02-12T17:00:24.817107+0000","last_peered":"2024-02-12T17:00:24.817107+0000","last_clean":"2024-02-12T17:00:24.817107+0000","last_became_active":"2024-02-12T16:59:09.274991+0000","last_became_peered":"2024-02-12T16:59:09.274991+0000","last_unstale":"2024-02-12T17:00:24.817107+0000","last_undegraded":"2024-02-12T17:00:24.817107+0000","last_fullsized":"2024-02-12T17:00:24.817107+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:37.895287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.895287+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":260,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787650+0000","last_change":"2024-02-12T16:59:09.269463+0000","last_active":"2024-02-12T17:00:24.787650+0000","last_peered":"2024-02-12T17:00:24.787650+0000","last_clean":"2024-02-12T17:00:24.787650+0000","last_became_active":"2024-02-12T16:59:09.269260+0000","last_became_peered":"2024-02-12T16:59:09.269260+0000","last_unstale":"2024-02-12T17:00:24.787650+0000","last_undegraded":"2024-02-12T17:00:24.787650+0000","last_fullsized":"2024-02-12T17:00:24.787650+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":193,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787858+0000","last_change":"2024-02-12T16:57:47.189653+0000","last_active":"2024-02-12T17:00:24.787858+0000","last_peered":"2024-02-12T17:00:24.787858+0000","last_clean":"2024-02-12T17:00:24.787858+0000","last_became_active":"2024-02-12T16:57:47.188909+0000","last_became_peered":"2024-02-12T16:57:47.188909+0000","last_unstale":"2024-02-12T17:00:24.787858+0000","last_undegraded":"2024-02-12T17:00:24.787858+0000","last_fullsized":"2024-02-12T17:00:24.787858+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":141,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.800515+0000","last_change":"2024-02-12T16:59:09.260492+0000","last_active":"2024-02-12T17:00:24.800515+0000","last_peered":"2024-02-12T17:00:24.800515+0000","last_clean":"2024-02-12T17:00:24.800515+0000","last_became_active":"2024-02-12T16:59:09.260045+0000","last_became_peered":"2024-02-12T16:59:09.260045+0000","last_unstale":"2024-02-12T17:00:24.800515+0000","last_undegraded":"2024-02-12T17:00:24.800515+0000","last_fullsized":"2024-02-12T17:00:24.800515+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:39.248562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.248562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":108,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783982+0000","last_change":"2024-02-12T16:59:09.288225+0000","last_active":"2024-02-12T17:00:24.783982+0000","last_peered":"2024-02-12T17:00:24.783982+0000","last_clean":"2024-02-12T17:00:24.783982+0000","last_became_active":"2024-02-12T16:59:09.253231+0000","last_became_peered":"2024-02-12T16:59:09.253231+0000","last_unstale":"2024-02-12T17:00:24.783982+0000","last_undegraded":"2024-02-12T17:00:24.783982+0000","last_fullsized":"2024-02-12T17:00:24.783982+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":213,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.784508+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T17:00:24.784508+0000","last_peered":"2024-02-12T17:00:24.784508+0000","last_clean":"2024-02-12T17:00:24.784508+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T17:00:24.784508+0000","last_undegraded":"2024-02-12T17:00:24.784508+0000","last_fullsized":"2024-02-12T17:00:24.784508+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":296,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.378556+0000","last_change":"2024-02-12T16:57:47.312130+0000","last_active":"2024-02-12T17:00:27.378556+0000","last_peered":"2024-02-12T17:00:27.378556+0000","last_clean":"2024-02-12T17:00:27.378556+0000","last_became_active":"2024-02-12T16:57:47.185109+0000","last_became_peered":"2024-02-12T16:57:47.185109+0000","last_unstale":"2024-02-12T17:00:27.378556+0000","last_undegraded":"2024-02-12T17:00:27.378556+0000","last_fullsized":"2024-02-12T17:00:27.378556+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":257,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787631+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T17:00:24.787631+0000","last_peered":"2024-02-12T17:00:24.787631+0000","last_clean":"2024-02-12T17:00:24.787631+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T17:00:24.787631+0000","last_undegraded":"2024-02-12T17:00:24.787631+0000","last_fullsized":"2024-02-12T17:00:24.787631+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":311,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.792962+0000","last_change":"2024-02-12T16:59:09.524334+0000","last_active":"2024-02-12T17:00:24.792962+0000","last_peered":"2024-02-12T17:00:24.792962+0000","last_clean":"2024-02-12T17:00:24.792962+0000","last_became_active":"2024-02-12T16:59:09.258515+0000","last_became_peered":"2024-02-12T16:59:09.258515+0000","last_unstale":"2024-02-12T17:00:24.792962+0000","last_undegraded":"2024-02-12T17:00:24.792962+0000","last_fullsized":"2024-02-12T17:00:24.792962+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":256,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.788278+0000","last_change":"2024-02-12T16:57:25.835979+0000","last_active":"2024-02-12T17:00:24.788278+0000","last_peered":"2024-02-12T17:00:24.788278+0000","last_clean":"2024-02-12T17:00:24.788278+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T17:00:24.788278+0000","last_undegraded":"2024-02-12T17:00:24.788278+0000","last_fullsized":"2024-02-12T17:00:24.788278+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.835883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.835883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"208'9","reported_seq":400,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.380183+0000","last_change":"2024-02-12T16:57:47.225402+0000","last_active":"2024-02-12T17:00:27.380183+0000","last_peered":"2024-02-12T17:00:27.380183+0000","last_clean":"2024-02-12T17:00:27.380183+0000","last_became_active":"2024-02-12T16:57:47.178688+0000","last_became_peered":"2024-02-12T16:57:47.178688+0000","last_unstale":"2024-02-12T17:00:27.380183+0000","last_undegraded":"2024-02-12T17:00:27.380183+0000","last_fullsized":"2024-02-12T17:00:27.380183+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:54:00.112856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.112856+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":259,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.792955+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T17:00:24.792955+0000","last_peered":"2024-02-12T17:00:24.792955+0000","last_clean":"2024-02-12T17:00:24.792955+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T17:00:24.792955+0000","last_undegraded":"2024-02-12T17:00:24.792955+0000","last_fullsized":"2024-02-12T17:00:24.792955+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":147,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.805513+0000","last_change":"2024-02-12T16:57:48.058146+0000","last_active":"2024-02-12T17:00:24.805513+0000","last_peered":"2024-02-12T17:00:24.805513+0000","last_clean":"2024-02-12T17:00:24.805513+0000","last_became_active":"2024-02-12T16:57:47.199140+0000","last_became_peered":"2024-02-12T16:57:47.199140+0000","last_unstale":"2024-02-12T17:00:24.805513+0000","last_undegraded":"2024-02-12T17:00:24.805513+0000","last_fullsized":"2024-02-12T17:00:24.805513+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":255,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801252+0000","last_change":"2024-02-12T16:59:09.269750+0000","last_active":"2024-02-12T17:00:24.801252+0000","last_peered":"2024-02-12T17:00:24.801252+0000","last_clean":"2024-02-12T17:00:24.801252+0000","last_became_active":"2024-02-12T16:59:09.269414+0000","last_became_peered":"2024-02-12T16:59:09.269414+0000","last_unstale":"2024-02-12T17:00:24.801252+0000","last_undegraded":"2024-02-12T17:00:24.801252+0000","last_fullsized":"2024-02-12T17:00:24.801252+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":216,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.803856+0000","last_change":"2024-02-12T16:59:09.243989+0000","last_active":"2024-02-12T17:00:24.803856+0000","last_peered":"2024-02-12T17:00:24.803856+0000","last_clean":"2024-02-12T17:00:24.803856+0000","last_became_active":"2024-02-12T16:59:09.243603+0000","last_became_peered":"2024-02-12T16:59:09.243603+0000","last_unstale":"2024-02-12T17:00:24.803856+0000","last_undegraded":"2024-02-12T17:00:24.803856+0000","last_fullsized":"2024-02-12T17:00:24.803856+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.218031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.218031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":235,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.804155+0000","last_change":"2024-02-12T16:59:09.519418+0000","last_active":"2024-02-12T17:00:24.804155+0000","last_peered":"2024-02-12T17:00:24.804155+0000","last_clean":"2024-02-12T17:00:24.804155+0000","last_became_active":"2024-02-12T16:59:09.242413+0000","last_became_peered":"2024-02-12T16:59:09.242413+0000","last_unstale":"2024-02-12T17:00:24.804155+0000","last_undegraded":"2024-02-12T17:00:24.804155+0000","last_fullsized":"2024-02-12T17:00:24.804155+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-02-12T16:53:57.641392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:53:57.641392+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":249,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783609+0000","last_change":"2024-02-12T16:56:38.699688+0000","last_active":"2024-02-12T17:00:24.783609+0000","last_peered":"2024-02-12T17:00:24.783609+0000","last_clean":"2024-02-12T17:00:24.783609+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T17:00:24.783609+0000","last_undegraded":"2024-02-12T17:00:24.783609+0000","last_fullsized":"2024-02-12T17:00:24.783609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.699602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":264,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783158+0000","last_change":"2024-02-12T16:57:47.171068+0000","last_active":"2024-02-12T17:00:24.783158+0000","last_peered":"2024-02-12T17:00:24.783158+0000","last_clean":"2024-02-12T17:00:24.783158+0000","last_became_active":"2024-02-12T16:57:47.170667+0000","last_became_peered":"2024-02-12T16:57:47.170667+0000","last_unstale":"2024-02-12T17:00:24.783158+0000","last_undegraded":"2024-02-12T17:00:24.783158+0000","last_fullsized":"2024-02-12T17:00:24.783158+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":294,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.788427+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T17:00:24.788427+0000","last_peered":"2024-02-12T17:00:24.788427+0000","last_clean":"2024-02-12T17:00:24.788427+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T17:00:24.788427+0000","last_undegraded":"2024-02-12T17:00:24.788427+0000","last_fullsized":"2024-02-12T17:00:24.788427+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"289'13155","reported_seq":15043,"reported_epoch":289,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:33.419844+0000","last_change":"2024-02-12T16:59:10.444155+0000","last_active":"2024-02-12T17:00:33.419844+0000","last_peered":"2024-02-12T17:00:33.419844+0000","last_clean":"2024-02-12T16:59:08.181528+0000","last_became_active":"2024-02-12T16:59:10.326952+0000","last_became_peered":"2024-02-12T16:59:10.326952+0000","last_unstale":"2024-02-12T17:00:33.419844+0000","last_undegraded":"2024-02-12T17:00:33.419844+0000","last_fullsized":"2024-02-12T17:00:33.419844+0000","mapping_epoch":258,"log_start":"262'9422","ondisk_log_start":"262'9422","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3733,"ondisk_log_size":3733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51445760,"num_objects":785,"num_object_clones":0,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":489,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13155,"num_write_kb":446080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":879,"num_bytes_recovered":57606144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":785}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":181,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802747+0000","last_change":"2024-02-12T16:59:09.273017+0000","last_active":"2024-02-12T17:00:24.802747+0000","last_peered":"2024-02-12T17:00:24.802747+0000","last_clean":"2024-02-12T17:00:24.802747+0000","last_became_active":"2024-02-12T16:59:09.272706+0000","last_became_peered":"2024-02-12T16:59:09.272706+0000","last_unstale":"2024-02-12T17:00:24.802747+0000","last_undegraded":"2024-02-12T17:00:24.802747+0000","last_fullsized":"2024-02-12T17:00:24.802747+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":280,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.803586+0000","last_change":"2024-02-12T16:57:22.679795+0000","last_active":"2024-02-12T17:00:24.803586+0000","last_peered":"2024-02-12T17:00:24.803586+0000","last_clean":"2024-02-12T17:00:24.803586+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T17:00:24.803586+0000","last_undegraded":"2024-02-12T17:00:24.803586+0000","last_fullsized":"2024-02-12T17:00:24.803586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:57:22.679705+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.679705+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":159,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.789329+0000","last_change":"2024-02-12T16:59:09.952265+0000","last_active":"2024-02-12T17:00:24.789329+0000","last_peered":"2024-02-12T17:00:24.789329+0000","last_clean":"2024-02-12T17:00:24.789329+0000","last_became_active":"2024-02-12T16:59:09.264482+0000","last_became_peered":"2024-02-12T16:59:09.264482+0000","last_unstale":"2024-02-12T17:00:24.789329+0000","last_undegraded":"2024-02-12T17:00:24.789329+0000","last_fullsized":"2024-02-12T17:00:24.789329+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:22.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.764984+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":278,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783308+0000","last_change":"2024-02-12T16:58:10.280441+0000","last_active":"2024-02-12T17:00:24.783308+0000","last_peered":"2024-02-12T17:00:24.783308+0000","last_clean":"2024-02-12T17:00:24.783308+0000","last_became_active":"2024-02-12T16:58:10.280137+0000","last_became_peered":"2024-02-12T16:58:10.280137+0000","last_unstale":"2024-02-12T17:00:24.783308+0000","last_undegraded":"2024-02-12T17:00:24.783308+0000","last_fullsized":"2024-02-12T17:00:24.783308+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.647627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.647627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"289'13205","reported_seq":13326,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.478696+0000","last_change":"2024-02-12T16:56:36.446431+0000","last_active":"2024-02-12T17:00:33.478696+0000","last_peered":"2024-02-12T17:00:33.478696+0000","last_clean":"2024-02-12T17:00:33.478696+0000","last_became_active":"2024-02-12T16:56:36.446107+0000","last_became_peered":"2024-02-12T16:56:36.446107+0000","last_unstale":"2024-02-12T17:00:33.478696+0000","last_undegraded":"2024-02-12T17:00:33.478696+0000","last_fullsized":"2024-02-12T17:00:33.478696+0000","mapping_epoch":176,"log_start":"262'9626","ondisk_log_start":"262'9626","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3579,"ondisk_log_size":3579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47775744,"num_objects":729,"num_object_clones":0,"num_object_copies":2187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":445888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":256,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787728+0000","last_change":"2024-02-12T16:57:29.934046+0000","last_active":"2024-02-12T17:00:24.787728+0000","last_peered":"2024-02-12T17:00:24.787728+0000","last_clean":"2024-02-12T17:00:24.787728+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T17:00:24.787728+0000","last_undegraded":"2024-02-12T17:00:24.787728+0000","last_fullsized":"2024-02-12T17:00:24.787728+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":93,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.789008+0000","last_change":"2024-02-12T16:57:47.187057+0000","last_active":"2024-02-12T17:00:24.789008+0000","last_peered":"2024-02-12T17:00:24.789008+0000","last_clean":"2024-02-12T17:00:24.789008+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:00:24.789008+0000","last_undegraded":"2024-02-12T17:00:24.789008+0000","last_fullsized":"2024-02-12T17:00:24.789008+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.009094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.009094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":147,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.793313+0000","last_change":"2024-02-12T16:57:48.006795+0000","last_active":"2024-02-12T17:00:24.793313+0000","last_peered":"2024-02-12T17:00:24.793313+0000","last_clean":"2024-02-12T17:00:24.793313+0000","last_became_active":"2024-02-12T16:57:47.202531+0000","last_became_peered":"2024-02-12T16:57:47.202531+0000","last_unstale":"2024-02-12T17:00:24.793313+0000","last_undegraded":"2024-02-12T17:00:24.793313+0000","last_fullsized":"2024-02-12T17:00:24.793313+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"261'9","reported_seq":396,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.378576+0000","last_change":"2024-02-12T16:59:09.377161+0000","last_active":"2024-02-12T17:00:27.378576+0000","last_peered":"2024-02-12T17:00:27.378576+0000","last_clean":"2024-02-12T17:00:27.378576+0000","last_became_active":"2024-02-12T16:59:09.270667+0000","last_became_peered":"2024-02-12T16:59:09.270667+0000","last_unstale":"2024-02-12T17:00:27.378576+0000","last_undegraded":"2024-02-12T17:00:27.378576+0000","last_fullsized":"2024-02-12T17:00:27.378576+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:53:59.173005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:53:59.173005+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"289'13352","reported_seq":13473,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.504337+0000","last_change":"2024-02-12T16:56:36.788519+0000","last_active":"2024-02-12T17:00:33.504337+0000","last_peered":"2024-02-12T17:00:33.504337+0000","last_clean":"2024-02-12T17:00:33.504337+0000","last_became_active":"2024-02-12T16:56:36.788101+0000","last_became_peered":"2024-02-12T16:56:36.788101+0000","last_unstale":"2024-02-12T17:00:33.504337+0000","last_undegraded":"2024-02-12T17:00:33.504337+0000","last_fullsized":"2024-02-12T17:00:33.504337+0000","mapping_epoch":176,"log_start":"262'9769","ondisk_log_start":"262'9769","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3583,"ondisk_log_size":3583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51380224,"num_objects":784,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13352,"num_write_kb":452352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":181,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.810190+0000","last_change":"2024-02-12T16:59:09.265012+0000","last_active":"2024-02-12T17:00:24.810190+0000","last_peered":"2024-02-12T17:00:24.810190+0000","last_clean":"2024-02-12T17:00:24.810190+0000","last_became_active":"2024-02-12T16:59:09.264617+0000","last_became_peered":"2024-02-12T16:59:09.264617+0000","last_unstale":"2024-02-12T17:00:24.810190+0000","last_undegraded":"2024-02-12T17:00:24.810190+0000","last_fullsized":"2024-02-12T17:00:24.810190+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.653301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.653301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":273,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.793030+0000","last_change":"2024-02-12T16:59:09.263209+0000","last_active":"2024-02-12T17:00:24.793030+0000","last_peered":"2024-02-12T17:00:24.793030+0000","last_clean":"2024-02-12T17:00:24.793030+0000","last_became_active":"2024-02-12T16:59:09.261764+0000","last_became_peered":"2024-02-12T16:59:09.261764+0000","last_unstale":"2024-02-12T17:00:24.793030+0000","last_undegraded":"2024-02-12T17:00:24.793030+0000","last_fullsized":"2024-02-12T17:00:24.793030+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"278'53","reported_seq":503,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.377813+0000","last_change":"2024-02-12T16:59:09.306589+0000","last_active":"2024-02-12T17:00:27.377813+0000","last_peered":"2024-02-12T17:00:27.377813+0000","last_clean":"2024-02-12T17:00:27.377813+0000","last_became_active":"2024-02-12T16:59:09.253257+0000","last_became_peered":"2024-02-12T16:59:09.253257+0000","last_unstale":"2024-02-12T17:00:27.377813+0000","last_undegraded":"2024-02-12T17:00:27.377813+0000","last_fullsized":"2024-02-12T17:00:27.377813+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'39","last_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.083656+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":267,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.788418+0000","last_change":"2024-02-12T16:59:09.264990+0000","last_active":"2024-02-12T17:00:24.788418+0000","last_peered":"2024-02-12T17:00:24.788418+0000","last_clean":"2024-02-12T17:00:24.788418+0000","last_became_active":"2024-02-12T16:59:09.264694+0000","last_became_peered":"2024-02-12T16:59:09.264694+0000","last_unstale":"2024-02-12T17:00:24.788418+0000","last_undegraded":"2024-02-12T17:00:24.788418+0000","last_fullsized":"2024-02-12T17:00:24.788418+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:46.719706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:46.719706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"289'13012","reported_seq":16970,"reported_epoch":289,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:33.454888+0000","last_change":"2024-02-12T16:59:09.891651+0000","last_active":"2024-02-12T17:00:33.454888+0000","last_peered":"2024-02-12T17:00:33.454888+0000","last_clean":"2024-02-12T16:57:46.130664+0000","last_became_active":"2024-02-12T16:57:48.242613+0000","last_became_peered":"2024-02-12T16:57:48.242613+0000","last_unstale":"2024-02-12T17:00:33.454888+0000","last_undegraded":"2024-02-12T17:00:33.454888+0000","last_fullsized":"2024-02-12T17:00:33.454888+0000","mapping_epoch":208,"log_start":"261'9117","ondisk_log_start":"261'9117","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3895,"ondisk_log_size":3895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49283072,"num_objects":752,"num_object_clones":0,"num_object_copies":2256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":276,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":440448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":125304832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":752}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":262,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801418+0000","last_change":"2024-02-12T16:59:09.269787+0000","last_active":"2024-02-12T17:00:24.801418+0000","last_peered":"2024-02-12T17:00:24.801418+0000","last_clean":"2024-02-12T17:00:24.801418+0000","last_became_active":"2024-02-12T16:59:09.269488+0000","last_became_peered":"2024-02-12T16:59:09.269488+0000","last_unstale":"2024-02-12T17:00:24.801418+0000","last_undegraded":"2024-02-12T17:00:24.801418+0000","last_fullsized":"2024-02-12T17:00:24.801418+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":298,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:31.612399+0000","last_change":"2024-02-12T16:59:10.016594+0000","last_active":"2024-02-12T17:00:31.612399+0000","last_peered":"2024-02-12T17:00:31.612399+0000","last_clean":"2024-02-12T17:00:31.612399+0000","last_became_active":"2024-02-12T16:59:09.273056+0000","last_became_peered":"2024-02-12T16:59:09.273056+0000","last_unstale":"2024-02-12T17:00:31.612399+0000","last_undegraded":"2024-02-12T17:00:31.612399+0000","last_fullsized":"2024-02-12T17:00:31.612399+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:54:30.972451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.972451+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":170,"num_read_kb":165,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":282,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783309+0000","last_change":"2024-02-12T16:59:09.248652+0000","last_active":"2024-02-12T17:00:24.783309+0000","last_peered":"2024-02-12T17:00:24.783309+0000","last_clean":"2024-02-12T17:00:24.783309+0000","last_became_active":"2024-02-12T16:59:09.248320+0000","last_became_peered":"2024-02-12T16:59:09.248320+0000","last_unstale":"2024-02-12T17:00:24.783309+0000","last_undegraded":"2024-02-12T17:00:24.783309+0000","last_fullsized":"2024-02-12T17:00:24.783309+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":174,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801772+0000","last_change":"2024-02-12T16:57:47.220507+0000","last_active":"2024-02-12T17:00:24.801772+0000","last_peered":"2024-02-12T17:00:24.801772+0000","last_clean":"2024-02-12T17:00:24.801772+0000","last_became_active":"2024-02-12T16:57:47.190968+0000","last_became_peered":"2024-02-12T16:57:47.190968+0000","last_unstale":"2024-02-12T17:00:24.801772+0000","last_undegraded":"2024-02-12T17:00:24.801772+0000","last_fullsized":"2024-02-12T17:00:24.801772+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":269,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.784486+0000","last_change":"2024-02-12T16:58:18.266322+0000","last_active":"2024-02-12T17:00:24.784486+0000","last_peered":"2024-02-12T17:00:24.784486+0000","last_clean":"2024-02-12T17:00:24.784486+0000","last_became_active":"2024-02-12T16:58:18.265999+0000","last_became_peered":"2024-02-12T16:58:18.265999+0000","last_unstale":"2024-02-12T17:00:24.784486+0000","last_undegraded":"2024-02-12T17:00:24.784486+0000","last_fullsized":"2024-02-12T17:00:24.784486+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"289'13071","reported_seq":13211,"reported_epoch":289,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:00:33.445454+0000","last_change":"2024-02-12T17:00:11.190095+0000","last_active":"2024-02-12T17:00:33.445454+0000","last_peered":"2024-02-12T17:00:33.445454+0000","last_clean":"2024-02-12T16:59:08.180231+0000","last_became_active":"2024-02-12T16:59:10.299589+0000","last_became_peered":"2024-02-12T16:59:10.299589+0000","last_unstale":"2024-02-12T17:00:33.445454+0000","last_undegraded":"2024-02-12T17:00:33.445454+0000","last_fullsized":"2024-02-12T17:00:33.445454+0000","mapping_epoch":258,"log_start":"262'9537","ondisk_log_start":"262'9537","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3534,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50397184,"num_objects":769,"num_object_clones":0,"num_object_copies":2307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":769,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13071,"num_write_kb":442880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":769}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":262,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802076+0000","last_change":"2024-02-12T16:57:47.191748+0000","last_active":"2024-02-12T17:00:24.802076+0000","last_peered":"2024-02-12T17:00:24.802076+0000","last_clean":"2024-02-12T17:00:24.802076+0000","last_became_active":"2024-02-12T16:57:47.191375+0000","last_became_peered":"2024-02-12T16:57:47.191375+0000","last_unstale":"2024-02-12T17:00:24.802076+0000","last_undegraded":"2024-02-12T17:00:24.802076+0000","last_fullsized":"2024-02-12T17:00:24.802076+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":177,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.793153+0000","last_change":"2024-02-12T16:59:09.263388+0000","last_active":"2024-02-12T17:00:24.793153+0000","last_peered":"2024-02-12T17:00:24.793153+0000","last_clean":"2024-02-12T17:00:24.793153+0000","last_became_active":"2024-02-12T16:59:09.262266+0000","last_became_peered":"2024-02-12T16:59:09.262266+0000","last_unstale":"2024-02-12T17:00:24.793153+0000","last_undegraded":"2024-02-12T17:00:24.793153+0000","last_fullsized":"2024-02-12T17:00:24.793153+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":255,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.792596+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:00:24.792596+0000","last_peered":"2024-02-12T17:00:24.792596+0000","last_clean":"2024-02-12T17:00:24.792596+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:00:24.792596+0000","last_undegraded":"2024-02-12T17:00:24.792596+0000","last_fullsized":"2024-02-12T17:00:24.792596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":265,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.800823+0000","last_change":"2024-02-12T16:59:09.245920+0000","last_active":"2024-02-12T17:00:24.800823+0000","last_peered":"2024-02-12T17:00:24.800823+0000","last_clean":"2024-02-12T17:00:24.800823+0000","last_became_active":"2024-02-12T16:59:09.245635+0000","last_became_peered":"2024-02-12T16:59:09.245635+0000","last_unstale":"2024-02-12T17:00:24.800823+0000","last_undegraded":"2024-02-12T17:00:24.800823+0000","last_fullsized":"2024-02-12T17:00:24.800823+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":111,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.812688+0000","last_change":"2024-02-12T16:59:10.440537+0000","last_active":"2024-02-12T17:00:24.812688+0000","last_peered":"2024-02-12T17:00:24.812688+0000","last_clean":"2024-02-12T17:00:24.812688+0000","last_became_active":"2024-02-12T16:59:09.249903+0000","last_became_peered":"2024-02-12T16:59:09.249903+0000","last_unstale":"2024-02-12T17:00:24.812688+0000","last_undegraded":"2024-02-12T17:00:24.812688+0000","last_fullsized":"2024-02-12T17:00:24.812688+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:19.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.280894+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":249,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.805582+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T17:00:24.805582+0000","last_peered":"2024-02-12T17:00:24.805582+0000","last_clean":"2024-02-12T17:00:24.805582+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:00:24.805582+0000","last_undegraded":"2024-02-12T17:00:24.805582+0000","last_fullsized":"2024-02-12T17:00:24.805582+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"289'13164","reported_seq":25669,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.505128+0000","last_change":"2024-02-12T16:58:49.227803+0000","last_active":"2024-02-12T17:00:33.505128+0000","last_peered":"2024-02-12T17:00:33.505128+0000","last_clean":"2024-02-12T17:00:33.505128+0000","last_became_active":"2024-02-12T16:58:49.217536+0000","last_became_peered":"2024-02-12T16:58:49.217536+0000","last_unstale":"2024-02-12T17:00:33.505128+0000","last_undegraded":"2024-02-12T17:00:33.505128+0000","last_fullsized":"2024-02-12T17:00:33.505128+0000","mapping_epoch":249,"log_start":"262'9537","ondisk_log_start":"262'9537","created":176,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3627,"ondisk_log_size":3627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53346304,"num_objects":814,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6182,"num_bytes_recovered":405143552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":195,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.800292+0000","last_change":"2024-02-12T16:59:09.252727+0000","last_active":"2024-02-12T17:00:24.800292+0000","last_peered":"2024-02-12T17:00:24.800292+0000","last_clean":"2024-02-12T17:00:24.800292+0000","last_became_active":"2024-02-12T16:59:09.252442+0000","last_became_peered":"2024-02-12T16:59:09.252442+0000","last_unstale":"2024-02-12T17:00:24.800292+0000","last_undegraded":"2024-02-12T17:00:24.800292+0000","last_fullsized":"2024-02-12T17:00:24.800292+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":257,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.789634+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T17:00:24.789634+0000","last_peered":"2024-02-12T17:00:24.789634+0000","last_clean":"2024-02-12T17:00:24.789634+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T17:00:24.789634+0000","last_undegraded":"2024-02-12T17:00:24.789634+0000","last_fullsized":"2024-02-12T17:00:24.789634+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":272,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.800805+0000","last_change":"2024-02-12T16:57:48.350132+0000","last_active":"2024-02-12T17:00:24.800805+0000","last_peered":"2024-02-12T17:00:24.800805+0000","last_clean":"2024-02-12T17:00:24.800805+0000","last_became_active":"2024-02-12T16:57:47.200347+0000","last_became_peered":"2024-02-12T16:57:47.200347+0000","last_unstale":"2024-02-12T17:00:24.800805+0000","last_undegraded":"2024-02-12T17:00:24.800805+0000","last_fullsized":"2024-02-12T17:00:24.800805+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":310,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.788346+0000","last_change":"2024-02-12T16:57:47.204108+0000","last_active":"2024-02-12T17:00:24.788346+0000","last_peered":"2024-02-12T17:00:24.788346+0000","last_clean":"2024-02-12T17:00:24.788346+0000","last_became_active":"2024-02-12T16:57:47.184456+0000","last_became_peered":"2024-02-12T16:57:47.184456+0000","last_unstale":"2024-02-12T17:00:24.788346+0000","last_undegraded":"2024-02-12T17:00:24.788346+0000","last_fullsized":"2024-02-12T17:00:24.788346+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":273,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802003+0000","last_change":"2024-02-12T16:58:14.391518+0000","last_active":"2024-02-12T17:00:24.802003+0000","last_peered":"2024-02-12T17:00:24.802003+0000","last_clean":"2024-02-12T17:00:24.802003+0000","last_became_active":"2024-02-12T16:58:14.391221+0000","last_became_peered":"2024-02-12T16:58:14.391221+0000","last_unstale":"2024-02-12T17:00:24.802003+0000","last_undegraded":"2024-02-12T17:00:24.802003+0000","last_fullsized":"2024-02-12T17:00:24.802003+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"289'13281","reported_seq":34012,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.504915+0000","last_change":"2024-02-12T16:59:46.062924+0000","last_active":"2024-02-12T17:00:33.504915+0000","last_peered":"2024-02-12T17:00:33.504915+0000","last_clean":"2024-02-12T17:00:33.504915+0000","last_became_active":"2024-02-12T16:59:46.029014+0000","last_became_peered":"2024-02-12T16:59:46.029014+0000","last_unstale":"2024-02-12T17:00:33.504915+0000","last_undegraded":"2024-02-12T17:00:33.504915+0000","last_fullsized":"2024-02-12T17:00:33.504915+0000","mapping_epoch":275,"log_start":"262'9579","ondisk_log_start":"262'9579","created":176,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3702,"ondisk_log_size":3702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47251456,"num_objects":721,"num_object_clones":0,"num_object_copies":2163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13281,"num_write_kb":448064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10282,"num_bytes_recovered":673841152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":46,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783412+0000","last_change":"2024-02-12T16:59:09.242340+0000","last_active":"2024-02-12T17:00:24.783412+0000","last_peered":"2024-02-12T17:00:24.783412+0000","last_clean":"2024-02-12T17:00:24.783412+0000","last_became_active":"2024-02-12T16:59:09.242012+0000","last_became_peered":"2024-02-12T16:59:09.242012+0000","last_unstale":"2024-02-12T17:00:24.783412+0000","last_undegraded":"2024-02-12T17:00:24.783412+0000","last_fullsized":"2024-02-12T17:00:24.783412+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.053587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.053587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":253,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801432+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T17:00:24.801432+0000","last_peered":"2024-02-12T17:00:24.801432+0000","last_clean":"2024-02-12T17:00:24.801432+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T17:00:24.801432+0000","last_undegraded":"2024-02-12T17:00:24.801432+0000","last_fullsized":"2024-02-12T17:00:24.801432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":302,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787556+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T17:00:24.787556+0000","last_peered":"2024-02-12T17:00:24.787556+0000","last_clean":"2024-02-12T17:00:24.787556+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T17:00:24.787556+0000","last_undegraded":"2024-02-12T17:00:24.787556+0000","last_fullsized":"2024-02-12T17:00:24.787556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":43,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787847+0000","last_change":"2024-02-12T16:59:09.271420+0000","last_active":"2024-02-12T17:00:24.787847+0000","last_peered":"2024-02-12T17:00:24.787847+0000","last_clean":"2024-02-12T17:00:24.787847+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:00:24.787847+0000","last_undegraded":"2024-02-12T17:00:24.787847+0000","last_fullsized":"2024-02-12T17:00:24.787847+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"289'13238","reported_seq":13359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.537636+0000","last_change":"2024-02-12T16:56:36.789146+0000","last_active":"2024-02-12T17:00:33.537636+0000","last_peered":"2024-02-12T17:00:33.537636+0000","last_clean":"2024-02-12T17:00:33.537636+0000","last_became_active":"2024-02-12T16:56:36.788871+0000","last_became_peered":"2024-02-12T16:56:36.788871+0000","last_unstale":"2024-02-12T17:00:33.537636+0000","last_undegraded":"2024-02-12T17:00:33.537636+0000","last_fullsized":"2024-02-12T17:00:33.537636+0000","mapping_epoch":176,"log_start":"262'9648","ondisk_log_start":"262'9648","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3590,"ondisk_log_size":3590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47972352,"num_objects":732,"num_object_clones":0,"num_object_copies":2196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13238,"num_write_kb":447040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":180,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.784071+0000","last_change":"2024-02-12T16:59:09.242302+0000","last_active":"2024-02-12T17:00:24.784071+0000","last_peered":"2024-02-12T17:00:24.784071+0000","last_clean":"2024-02-12T17:00:24.784071+0000","last_became_active":"2024-02-12T16:59:09.241987+0000","last_became_peered":"2024-02-12T16:59:09.241987+0000","last_unstale":"2024-02-12T17:00:24.784071+0000","last_undegraded":"2024-02-12T17:00:24.784071+0000","last_fullsized":"2024-02-12T17:00:24.784071+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":46,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.804139+0000","last_change":"2024-02-12T16:59:09.242865+0000","last_active":"2024-02-12T17:00:24.804139+0000","last_peered":"2024-02-12T17:00:24.804139+0000","last_clean":"2024-02-12T17:00:24.804139+0000","last_became_active":"2024-02-12T16:59:09.242505+0000","last_became_peered":"2024-02-12T16:59:09.242505+0000","last_unstale":"2024-02-12T17:00:24.804139+0000","last_undegraded":"2024-02-12T17:00:24.804139+0000","last_fullsized":"2024-02-12T17:00:24.804139+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:48.828632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:53:48.828632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":275,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.800972+0000","last_change":"2024-02-12T16:57:48.521837+0000","last_active":"2024-02-12T17:00:24.800972+0000","last_peered":"2024-02-12T17:00:24.800972+0000","last_clean":"2024-02-12T17:00:24.800972+0000","last_became_active":"2024-02-12T16:57:47.199763+0000","last_became_peered":"2024-02-12T16:57:47.199763+0000","last_unstale":"2024-02-12T17:00:24.800972+0000","last_undegraded":"2024-02-12T17:00:24.800972+0000","last_fullsized":"2024-02-12T17:00:24.800972+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":261,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.804061+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T17:00:24.804061+0000","last_peered":"2024-02-12T17:00:24.804061+0000","last_clean":"2024-02-12T17:00:24.804061+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T17:00:24.804061+0000","last_undegraded":"2024-02-12T17:00:24.804061+0000","last_fullsized":"2024-02-12T17:00:24.804061+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"289'13386","reported_seq":13508,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.496088+0000","last_change":"2024-02-12T16:56:36.788810+0000","last_active":"2024-02-12T17:00:33.496088+0000","last_peered":"2024-02-12T17:00:33.496088+0000","last_clean":"2024-02-12T17:00:33.496088+0000","last_became_active":"2024-02-12T16:56:36.788518+0000","last_became_peered":"2024-02-12T16:56:36.788518+0000","last_unstale":"2024-02-12T17:00:33.496088+0000","last_undegraded":"2024-02-12T17:00:33.496088+0000","last_fullsized":"2024-02-12T17:00:33.496088+0000","mapping_epoch":176,"log_start":"262'9812","ondisk_log_start":"262'9812","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3574,"ondisk_log_size":3574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53280792,"num_objects":814,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13386,"num_write_kb":454337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":261,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.803007+0000","last_change":"2024-02-12T16:59:09.340317+0000","last_active":"2024-02-12T17:00:24.803007+0000","last_peered":"2024-02-12T17:00:24.803007+0000","last_clean":"2024-02-12T17:00:24.803007+0000","last_became_active":"2024-02-12T16:59:09.246616+0000","last_became_peered":"2024-02-12T16:59:09.246616+0000","last_unstale":"2024-02-12T17:00:24.803007+0000","last_undegraded":"2024-02-12T17:00:24.803007+0000","last_fullsized":"2024-02-12T17:00:24.803007+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:57:19.955311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.955311+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802253+0000","last_change":"2024-02-12T16:59:09.268325+0000","last_active":"2024-02-12T17:00:24.802253+0000","last_peered":"2024-02-12T17:00:24.802253+0000","last_clean":"2024-02-12T17:00:24.802253+0000","last_became_active":"2024-02-12T16:59:09.268028+0000","last_became_peered":"2024-02-12T16:59:09.268028+0000","last_unstale":"2024-02-12T17:00:24.802253+0000","last_undegraded":"2024-02-12T17:00:24.802253+0000","last_fullsized":"2024-02-12T17:00:24.802253+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":292,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.799089+0000","last_change":"2024-02-12T16:59:09.996418+0000","last_active":"2024-02-12T17:00:24.799089+0000","last_peered":"2024-02-12T17:00:24.799089+0000","last_clean":"2024-02-12T17:00:24.799089+0000","last_became_active":"2024-02-12T16:59:09.256058+0000","last_became_peered":"2024-02-12T16:59:09.256058+0000","last_unstale":"2024-02-12T17:00:24.799089+0000","last_undegraded":"2024-02-12T17:00:24.799089+0000","last_fullsized":"2024-02-12T17:00:24.799089+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"208'10","reported_seq":150,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.377717+0000","last_change":"2024-02-12T16:57:47.193205+0000","last_active":"2024-02-12T17:00:27.377717+0000","last_peered":"2024-02-12T17:00:27.377717+0000","last_clean":"2024-02-12T17:00:27.377717+0000","last_became_active":"2024-02-12T16:57:47.167628+0000","last_became_peered":"2024-02-12T16:57:47.167628+0000","last_unstale":"2024-02-12T17:00:27.377717+0000","last_undegraded":"2024-02-12T17:00:27.377717+0000","last_fullsized":"2024-02-12T17:00:27.377717+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:55:28.449050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:55:28.449050+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"289'13044","reported_seq":13165,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.486623+0000","last_change":"2024-02-12T16:56:36.445962+0000","last_active":"2024-02-12T17:00:33.486623+0000","last_peered":"2024-02-12T17:00:33.486623+0000","last_clean":"2024-02-12T17:00:33.486623+0000","last_became_active":"2024-02-12T16:56:36.445638+0000","last_became_peered":"2024-02-12T16:56:36.445638+0000","last_unstale":"2024-02-12T17:00:33.486623+0000","last_undegraded":"2024-02-12T17:00:33.486623+0000","last_fullsized":"2024-02-12T17:00:33.486623+0000","mapping_epoch":176,"log_start":"261'9223","ondisk_log_start":"261'9223","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3821,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48627712,"num_objects":742,"num_object_clones":0,"num_object_copies":2226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13044,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":46,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.800212+0000","last_change":"2024-02-12T16:59:09.260336+0000","last_active":"2024-02-12T17:00:24.800212+0000","last_peered":"2024-02-12T17:00:24.800212+0000","last_clean":"2024-02-12T17:00:24.800212+0000","last_became_active":"2024-02-12T16:59:09.259654+0000","last_became_peered":"2024-02-12T16:59:09.259654+0000","last_unstale":"2024-02-12T17:00:24.800212+0000","last_undegraded":"2024-02-12T17:00:24.800212+0000","last_fullsized":"2024-02-12T17:00:24.800212+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"261'14","reported_seq":332,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.379825+0000","last_change":"2024-02-12T16:59:12.225158+0000","last_active":"2024-02-12T17:00:27.379825+0000","last_peered":"2024-02-12T17:00:27.379825+0000","last_clean":"2024-02-12T17:00:27.379825+0000","last_became_active":"2024-02-12T16:59:12.224830+0000","last_became_peered":"2024-02-12T16:59:12.224830+0000","last_unstale":"2024-02-12T17:00:27.379825+0000","last_undegraded":"2024-02-12T17:00:27.379825+0000","last_fullsized":"2024-02-12T17:00:27.379825+0000","mapping_epoch":260,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":241,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783025+0000","last_change":"2024-02-12T16:57:47.171714+0000","last_active":"2024-02-12T17:00:24.783025+0000","last_peered":"2024-02-12T17:00:24.783025+0000","last_clean":"2024-02-12T17:00:24.783025+0000","last_became_active":"2024-02-12T16:57:47.171128+0000","last_became_peered":"2024-02-12T16:57:47.171128+0000","last_unstale":"2024-02-12T17:00:24.783025+0000","last_undegraded":"2024-02-12T17:00:24.783025+0000","last_fullsized":"2024-02-12T17:00:24.783025+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":256,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.795358+0000","last_change":"2024-02-12T16:59:09.788881+0000","last_active":"2024-02-12T17:00:24.795358+0000","last_peered":"2024-02-12T17:00:24.795358+0000","last_clean":"2024-02-12T17:00:24.795358+0000","last_became_active":"2024-02-12T16:59:09.277264+0000","last_became_peered":"2024-02-12T16:59:09.277264+0000","last_unstale":"2024-02-12T17:00:24.795358+0000","last_undegraded":"2024-02-12T17:00:24.795358+0000","last_fullsized":"2024-02-12T17:00:24.795358+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"289'13245","reported_seq":17595,"reported_epoch":289,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:33.489518+0000","last_change":"2024-02-12T16:59:09.997308+0000","last_active":"2024-02-12T17:00:33.489518+0000","last_peered":"2024-02-12T17:00:33.489518+0000","last_clean":"2024-02-12T16:57:46.091774+0000","last_became_active":"2024-02-12T16:59:09.335728+0000","last_became_peered":"2024-02-12T16:59:09.335728+0000","last_unstale":"2024-02-12T17:00:33.489518+0000","last_undegraded":"2024-02-12T17:00:33.489518+0000","last_fullsized":"2024-02-12T17:00:33.489518+0000","mapping_epoch":257,"log_start":"263'9701","ondisk_log_start":"263'9701","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3544,"ondisk_log_size":3544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52887552,"num_objects":807,"num_object_clones":0,"num_object_copies":2421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":934,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13245,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1900,"num_bytes_recovered":124518400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5,6","objects":807}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":258,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.793321+0000","last_change":"2024-02-12T16:57:47.177662+0000","last_active":"2024-02-12T17:00:24.793321+0000","last_peered":"2024-02-12T17:00:24.793321+0000","last_clean":"2024-02-12T17:00:24.793321+0000","last_became_active":"2024-02-12T16:57:47.177329+0000","last_became_peered":"2024-02-12T16:57:47.177329+0000","last_unstale":"2024-02-12T17:00:24.793321+0000","last_undegraded":"2024-02-12T17:00:24.793321+0000","last_fullsized":"2024-02-12T17:00:24.793321+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.152265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"208'12","reported_seq":377,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.379633+0000","last_change":"2024-02-12T16:57:46.177864+0000","last_active":"2024-02-12T17:00:27.379633+0000","last_peered":"2024-02-12T17:00:27.379633+0000","last_clean":"2024-02-12T17:00:27.379633+0000","last_became_active":"2024-02-12T16:57:46.177517+0000","last_became_peered":"2024-02-12T16:57:46.177517+0000","last_unstale":"2024-02-12T17:00:27.379633+0000","last_undegraded":"2024-02-12T17:00:27.379633+0000","last_fullsized":"2024-02-12T17:00:27.379633+0000","mapping_epoch":97,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":41,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.801277+0000","last_change":"2024-02-12T16:59:09.268099+0000","last_active":"2024-02-12T17:00:24.801277+0000","last_peered":"2024-02-12T17:00:24.801277+0000","last_clean":"2024-02-12T17:00:24.801277+0000","last_became_active":"2024-02-12T16:59:09.267755+0000","last_became_peered":"2024-02-12T16:59:09.267755+0000","last_unstale":"2024-02-12T17:00:24.801277+0000","last_undegraded":"2024-02-12T17:00:24.801277+0000","last_fullsized":"2024-02-12T17:00:24.801277+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":261,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802067+0000","last_change":"2024-02-12T16:57:48.169827+0000","last_active":"2024-02-12T17:00:24.802067+0000","last_peered":"2024-02-12T17:00:24.802067+0000","last_clean":"2024-02-12T17:00:24.802067+0000","last_became_active":"2024-02-12T16:57:47.199008+0000","last_became_peered":"2024-02-12T16:57:47.199008+0000","last_unstale":"2024-02-12T17:00:24.802067+0000","last_undegraded":"2024-02-12T17:00:24.802067+0000","last_fullsized":"2024-02-12T17:00:24.802067+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"289'13254","reported_seq":16983,"reported_epoch":289,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:33.484523+0000","last_change":"2024-02-12T16:59:10.062476+0000","last_active":"2024-02-12T17:00:33.484523+0000","last_peered":"2024-02-12T17:00:33.484523+0000","last_clean":"2024-02-12T16:57:46.133812+0000","last_became_active":"2024-02-12T16:59:09.319664+0000","last_became_peered":"2024-02-12T16:59:09.319664+0000","last_unstale":"2024-02-12T17:00:33.484523+0000","last_undegraded":"2024-02-12T17:00:33.484523+0000","last_fullsized":"2024-02-12T17:00:33.484523+0000","mapping_epoch":257,"log_start":"263'9714","ondisk_log_start":"263'9714","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3540,"ondisk_log_size":3540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":449,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13254,"num_write_kb":447808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1794,"num_bytes_recovered":117571584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,1],"avail_no_missing":["5","1","4","6"],"object_location_counts":[{"shards":"1,4,5","objects":740}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":96,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.784567+0000","last_change":"2024-02-12T16:57:47.176436+0000","last_active":"2024-02-12T17:00:24.784567+0000","last_peered":"2024-02-12T17:00:24.784567+0000","last_clean":"2024-02-12T17:00:24.784567+0000","last_became_active":"2024-02-12T16:57:47.176041+0000","last_became_peered":"2024-02-12T16:57:47.176041+0000","last_unstale":"2024-02-12T17:00:24.784567+0000","last_undegraded":"2024-02-12T17:00:24.784567+0000","last_fullsized":"2024-02-12T17:00:24.784567+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"211'10","reported_seq":409,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.377825+0000","last_change":"2024-02-12T16:57:50.205871+0000","last_active":"2024-02-12T17:00:27.377825+0000","last_peered":"2024-02-12T17:00:27.377825+0000","last_clean":"2024-02-12T17:00:27.377825+0000","last_became_active":"2024-02-12T16:57:50.205330+0000","last_became_peered":"2024-02-12T16:57:50.205330+0000","last_unstale":"2024-02-12T17:00:27.377825+0000","last_undegraded":"2024-02-12T17:00:27.377825+0000","last_fullsized":"2024-02-12T17:00:27.377825+0000","mapping_epoch":61,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":197,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802019+0000","last_change":"2024-02-12T16:57:47.191627+0000","last_active":"2024-02-12T17:00:24.802019+0000","last_peered":"2024-02-12T17:00:24.802019+0000","last_clean":"2024-02-12T17:00:24.802019+0000","last_became_active":"2024-02-12T16:57:47.191231+0000","last_became_peered":"2024-02-12T16:57:47.191231+0000","last_unstale":"2024-02-12T17:00:24.802019+0000","last_undegraded":"2024-02-12T17:00:24.802019+0000","last_fullsized":"2024-02-12T17:00:24.802019+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":196,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.805238+0000","last_change":"2024-02-12T16:59:09.782291+0000","last_active":"2024-02-12T17:00:24.805238+0000","last_peered":"2024-02-12T17:00:24.805238+0000","last_clean":"2024-02-12T17:00:24.805238+0000","last_became_active":"2024-02-12T16:59:09.268136+0000","last_became_peered":"2024-02-12T16:59:09.268136+0000","last_unstale":"2024-02-12T17:00:24.805238+0000","last_undegraded":"2024-02-12T17:00:24.805238+0000","last_fullsized":"2024-02-12T17:00:24.805238+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"289'13015","reported_seq":16288,"reported_epoch":289,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:00:33.527885+0000","last_change":"2024-02-12T16:59:09.891108+0000","last_active":"2024-02-12T17:00:33.527885+0000","last_peered":"2024-02-12T17:00:33.527885+0000","last_clean":"2024-02-12T16:57:46.113428+0000","last_became_active":"2024-02-12T16:57:48.262168+0000","last_became_peered":"2024-02-12T16:57:48.262168+0000","last_unstale":"2024-02-12T17:00:33.527885+0000","last_undegraded":"2024-02-12T17:00:33.527885+0000","last_fullsized":"2024-02-12T17:00:33.527885+0000","mapping_epoch":208,"log_start":"262'9436","ondisk_log_start":"262'9436","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3579,"ondisk_log_size":3579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47382528,"num_objects":723,"num_object_clones":0,"num_object_copies":2169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":316,"num_objects_unfound":0,"num_objects_dirty":723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13015,"num_write_kb":439616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1566,"num_bytes_recovered":102629376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":723}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":272,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.784432+0000","last_change":"2024-02-12T16:59:09.246938+0000","last_active":"2024-02-12T17:00:24.784432+0000","last_peered":"2024-02-12T17:00:24.784432+0000","last_clean":"2024-02-12T17:00:24.784432+0000","last_became_active":"2024-02-12T16:59:09.246605+0000","last_became_peered":"2024-02-12T16:59:09.246605+0000","last_unstale":"2024-02-12T17:00:24.784432+0000","last_undegraded":"2024-02-12T17:00:24.784432+0000","last_fullsized":"2024-02-12T17:00:24.784432+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":120,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.816237+0000","last_change":"2024-02-12T16:59:09.247764+0000","last_active":"2024-02-12T17:00:24.816237+0000","last_peered":"2024-02-12T17:00:24.816237+0000","last_clean":"2024-02-12T17:00:24.816237+0000","last_became_active":"2024-02-12T16:59:09.247447+0000","last_became_peered":"2024-02-12T16:59:09.247447+0000","last_unstale":"2024-02-12T17:00:24.816237+0000","last_undegraded":"2024-02-12T17:00:24.816237+0000","last_fullsized":"2024-02-12T17:00:24.816237+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":253,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802674+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T17:00:24.802674+0000","last_peered":"2024-02-12T17:00:24.802674+0000","last_clean":"2024-02-12T17:00:24.802674+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T17:00:24.802674+0000","last_undegraded":"2024-02-12T17:00:24.802674+0000","last_fullsized":"2024-02-12T17:00:24.802674+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":355,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787422+0000","last_change":"2024-02-12T16:59:10.004273+0000","last_active":"2024-02-12T17:00:24.787422+0000","last_peered":"2024-02-12T17:00:24.787422+0000","last_clean":"2024-02-12T17:00:24.787422+0000","last_became_active":"2024-02-12T16:59:09.262475+0000","last_became_peered":"2024-02-12T16:59:09.262475+0000","last_unstale":"2024-02-12T17:00:24.787422+0000","last_undegraded":"2024-02-12T17:00:24.787422+0000","last_fullsized":"2024-02-12T17:00:24.787422+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"289'13172","reported_seq":22086,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.556670+0000","last_change":"2024-02-12T16:59:27.226608+0000","last_active":"2024-02-12T17:00:33.556670+0000","last_peered":"2024-02-12T17:00:33.556670+0000","last_clean":"2024-02-12T17:00:33.556670+0000","last_became_active":"2024-02-12T16:59:27.206365+0000","last_became_peered":"2024-02-12T16:59:27.206365+0000","last_unstale":"2024-02-12T17:00:33.556670+0000","last_undegraded":"2024-02-12T17:00:33.556670+0000","last_fullsized":"2024-02-12T17:00:33.556670+0000","mapping_epoch":265,"log_start":"262'9630","ondisk_log_start":"262'9630","created":176,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3542,"ondisk_log_size":3542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52559872,"num_objects":802,"num_object_clones":0,"num_object_copies":2406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13172,"num_write_kb":447168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4385,"num_bytes_recovered":287375360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":265,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.800743+0000","last_change":"2024-02-12T16:59:09.242082+0000","last_active":"2024-02-12T17:00:24.800743+0000","last_peered":"2024-02-12T17:00:24.800743+0000","last_clean":"2024-02-12T17:00:24.800743+0000","last_became_active":"2024-02-12T16:59:09.241648+0000","last_became_peered":"2024-02-12T16:59:09.241648+0000","last_unstale":"2024-02-12T17:00:24.800743+0000","last_undegraded":"2024-02-12T17:00:24.800743+0000","last_fullsized":"2024-02-12T17:00:24.800743+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":274,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.784537+0000","last_change":"2024-02-12T16:57:58.203843+0000","last_active":"2024-02-12T17:00:24.784537+0000","last_peered":"2024-02-12T17:00:24.784537+0000","last_clean":"2024-02-12T17:00:24.784537+0000","last_became_active":"2024-02-12T16:57:58.203578+0000","last_became_peered":"2024-02-12T16:57:58.203578+0000","last_unstale":"2024-02-12T17:00:24.784537+0000","last_undegraded":"2024-02-12T17:00:24.784537+0000","last_fullsized":"2024-02-12T17:00:24.784537+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":255,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783712+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T17:00:24.783712+0000","last_peered":"2024-02-12T17:00:24.783712+0000","last_clean":"2024-02-12T17:00:24.783712+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T17:00:24.783712+0000","last_undegraded":"2024-02-12T17:00:24.783712+0000","last_fullsized":"2024-02-12T17:00:24.783712+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":130,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.802214+0000","last_change":"2024-02-12T16:59:10.836208+0000","last_active":"2024-02-12T17:00:24.802214+0000","last_peered":"2024-02-12T17:00:24.802214+0000","last_clean":"2024-02-12T17:00:24.802214+0000","last_became_active":"2024-02-12T16:59:09.273985+0000","last_became_peered":"2024-02-12T16:59:09.273985+0000","last_unstale":"2024-02-12T17:00:24.802214+0000","last_undegraded":"2024-02-12T17:00:24.802214+0000","last_fullsized":"2024-02-12T17:00:24.802214+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:41.290236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.290236+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"289'13137","reported_seq":33752,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.511069+0000","last_change":"2024-02-12T16:59:45.048235+0000","last_active":"2024-02-12T17:00:33.511069+0000","last_peered":"2024-02-12T17:00:33.511069+0000","last_clean":"2024-02-12T17:00:33.511069+0000","last_became_active":"2024-02-12T16:59:45.029644+0000","last_became_peered":"2024-02-12T16:59:45.029644+0000","last_unstale":"2024-02-12T17:00:33.511069+0000","last_undegraded":"2024-02-12T17:00:33.511069+0000","last_fullsized":"2024-02-12T17:00:33.511069+0000","mapping_epoch":274,"log_start":"262'9472","ondisk_log_start":"262'9472","created":176,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3665,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50659328,"num_objects":773,"num_object_clones":0,"num_object_copies":2319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13137,"num_write_kb":445120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10227,"num_bytes_recovered":670236672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":242,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.789858+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T17:00:24.789858+0000","last_peered":"2024-02-12T17:00:24.789858+0000","last_clean":"2024-02-12T17:00:24.789858+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T17:00:24.789858+0000","last_undegraded":"2024-02-12T17:00:24.789858+0000","last_fullsized":"2024-02-12T17:00:24.789858+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":278,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.783964+0000","last_change":"2024-02-12T16:58:02.376971+0000","last_active":"2024-02-12T17:00:24.783964+0000","last_peered":"2024-02-12T17:00:24.783964+0000","last_clean":"2024-02-12T17:00:24.783964+0000","last_became_active":"2024-02-12T16:58:02.376628+0000","last_became_peered":"2024-02-12T16:58:02.376628+0000","last_unstale":"2024-02-12T17:00:24.783964+0000","last_undegraded":"2024-02-12T17:00:24.783964+0000","last_fullsized":"2024-02-12T17:00:24.783964+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:19.703739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.703739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":219,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787328+0000","last_change":"2024-02-12T16:59:09.279916+0000","last_active":"2024-02-12T17:00:24.787328+0000","last_peered":"2024-02-12T17:00:24.787328+0000","last_clean":"2024-02-12T17:00:24.787328+0000","last_became_active":"2024-02-12T16:59:09.262401+0000","last_became_peered":"2024-02-12T16:59:09.262401+0000","last_unstale":"2024-02-12T17:00:24.787328+0000","last_undegraded":"2024-02-12T17:00:24.787328+0000","last_fullsized":"2024-02-12T17:00:24.787328+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T16:53:56.761445+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":295,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.787712+0000","last_change":"2024-02-12T16:57:47.553993+0000","last_active":"2024-02-12T17:00:24.787712+0000","last_peered":"2024-02-12T17:00:24.787712+0000","last_clean":"2024-02-12T17:00:24.787712+0000","last_became_active":"2024-02-12T16:57:47.189672+0000","last_became_peered":"2024-02-12T16:57:47.189672+0000","last_unstale":"2024-02-12T17:00:24.787712+0000","last_undegraded":"2024-02-12T17:00:24.787712+0000","last_fullsized":"2024-02-12T17:00:24.787712+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"289'13321","reported_seq":13442,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:33.503220+0000","last_change":"2024-02-12T16:56:36.445585+0000","last_active":"2024-02-12T17:00:33.503220+0000","last_peered":"2024-02-12T17:00:33.503220+0000","last_clean":"2024-02-12T17:00:33.503220+0000","last_became_active":"2024-02-12T16:56:36.445291+0000","last_became_peered":"2024-02-12T16:56:36.445291+0000","last_unstale":"2024-02-12T17:00:33.503220+0000","last_undegraded":"2024-02-12T17:00:33.503220+0000","last_fullsized":"2024-02-12T17:00:33.503220+0000","mapping_epoch":176,"log_start":"262'9794","ondisk_log_start":"262'9794","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3527,"ondisk_log_size":3527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50659328,"num_objects":773,"num_object_clones":0,"num_object_copies":2319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13321,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":253,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.792884+0000","last_change":"2024-02-12T16:59:09.262958+0000","last_active":"2024-02-12T17:00:24.792884+0000","last_peered":"2024-02-12T17:00:24.792884+0000","last_clean":"2024-02-12T17:00:24.792884+0000","last_became_active":"2024-02-12T16:59:09.261034+0000","last_became_peered":"2024-02-12T16:59:09.261034+0000","last_unstale":"2024-02-12T17:00:24.792884+0000","last_undegraded":"2024-02-12T17:00:24.792884+0000","last_fullsized":"2024-02-12T17:00:24.792884+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":278,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.784293+0000","last_change":"2024-02-12T16:59:09.250623+0000","last_active":"2024-02-12T17:00:24.784293+0000","last_peered":"2024-02-12T17:00:24.784293+0000","last_clean":"2024-02-12T17:00:24.784293+0000","last_became_active":"2024-02-12T16:59:09.250318+0000","last_became_peered":"2024-02-12T16:59:09.250318+0000","last_unstale":"2024-02-12T17:00:24.784293+0000","last_undegraded":"2024-02-12T17:00:24.784293+0000","last_fullsized":"2024-02-12T17:00:24.784293+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":328,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-02-12T17:00:24.792270+0000","last_change":"2024-02-12T16:59:11.094393+0000","last_active":"2024-02-12T17:00:24.792270+0000","last_peered":"2024-02-12T17:00:24.792270+0000","last_clean":"2024-02-12T17:00:24.792270+0000","last_became_active":"2024-02-12T16:59:09.280858+0000","last_became_peered":"2024-02-12T16:59:09.280858+0000","last_unstale":"2024-02-12T17:00:24.792270+0000","last_undegraded":"2024-02-12T17:00:24.792270+0000","last_fullsized":"2024-02-12T17:00:24.792270+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'22","last_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.138758+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"261'13","reported_seq":429,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-02-12T17:00:27.378336+0000","last_change":"2024-02-12T16:59:12.243084+0000","last_active":"2024-02-12T17:00:27.378336+0000","last_peered":"2024-02-12T17:00:27.378336+0000","last_clean":"2024-02-12T17:00:27.378336+0000","last_became_active":"2024-02-12T16:59:12.242710+0000","last_became_peered":"2024-02-12T16:59:12.242710+0000","last_unstale":"2024-02-12T17:00:27.378336+0000","last_undegraded":"2024-02-12T17:00:27.378336+0000","last_fullsized":"2024-02-12T17:00:27.378336+0000","mapping_epoch":260,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:52:59.185010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:52:59.185010+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":803405848,"num_objects":12260,"num_object_clones":0,"num_object_copies":36780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3233,"num_objects_unfound":0,"num_objects_dirty":12260,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211052,"num_write_kb":7145921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39127,"num_bytes_recovered":2564227072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3643682816,"data_stored":3643670600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58035,"ondisk_log_size":58035,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":170,"num_read_kb":165,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":5408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234118,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638940,"kb_used_data":341716,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93106212,"statfs":{"total":95995035648,"available":95340761088,"internally_reserved":0,"allocated":349917184,"data_stored":349189471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.407,"15min":1.101},"min":{"1min":0.246,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.5339999999999998,"5min":7.3239999999999998,"15min":36.439},"last":0.36399999999999999},{"interface":"front","average":{"1min":3.089,"5min":1.851,"15min":1.2529999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.987,"5min":11.231999999999999,"15min":36.490000000000002},"last":0.27700000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":1.778,"15min":1.01},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.048999999999999,"5min":12.032,"15min":12.032},"last":2.3239999999999998},{"interface":"front","average":{"1min":2.456,"5min":1.6830000000000001,"15min":0.97699999999999998},"min":{"1min":0.251,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.965999999999999,"5min":11.260999999999999,"15min":11.260999999999999},"last":0.311}]},{"osd":2,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.2350000000000001,"15min":0.874},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.2450000000000001,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":2.5920000000000001,"5min":1.8959999999999999,"15min":1.131},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":11.071999999999999,"5min":11.071999999999999,"15min":20.338000000000001},"last":2.355}]},{"osd":3,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.569,"15min":1.161},"min":{"1min":0.28999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.8600000000000003,"5min":9.1069999999999993,"15min":20.433},"last":0.44},{"interface":"front","average":{"1min":2.7040000000000002,"5min":2.0960000000000001,"15min":1.3340000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.092000000000001,"5min":11.092000000000001,"15min":20.393000000000001},"last":2.3799999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":1.9239999999999999,"15min":0.97699999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.219},"max":{"1min":5.556,"5min":50.406999999999996,"15min":50.406999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":4.2519999999999998,"5min":2.8119999999999998,"15min":1.27},"min":{"1min":0.29499999999999998,"5min":0.23999999999999999,"15min":0.23400000000000001},"max":{"1min":14.863,"5min":50.168999999999997,"15min":50.168999999999997},"last":2.403}]},{"osd":5,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.4790000000000001,"15min":0.83299999999999996},"min":{"1min":0.45800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.5910000000000002,"5min":7.4269999999999996,"15min":7.4269999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":4.4210000000000003,"5min":2.2210000000000001,"15min":1.093},"min":{"1min":0.371,"5min":0.27400000000000002,"15min":0.26200000000000001},"max":{"1min":11.696,"5min":12.352,"15min":12.352},"last":2.4359999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:59:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.663,"15min":0.90100000000000002},"min":{"1min":0.30399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":10.929,"5min":11.282999999999999,"15min":11.282999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5669999999999999,"15min":0.91100000000000003},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.30199999999999999},"max":{"1min":4.4930000000000003,"5min":12.407999999999999,"15min":12.407999999999999},"last":0.72199999999999998}]}]},{"osd":6,"up_from":0,"seq":193273528459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738536,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":690064,"kb_used_data":388744,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93055088,"statfs":{"total":95995035648,"available":95288410112,"internally_reserved":0,"allocated":398073856,"data_stored":397357526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.9199999999999999,"15min":1.0840000000000001},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.3839999999999999,"5min":23.510999999999999,"15min":23.510999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.639,"15min":1.02},"min":{"1min":0.21299999999999999,"5min":0.193,"15min":0.193},"max":{"1min":5.0449999999999999,"5min":10.186999999999999,"15min":10.186999999999999},"last":3.427}]},{"osd":2,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.841,"15min":1.3180000000000001},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.20499999999999999},"max":{"1min":11.122999999999999,"5min":16.521000000000001,"15min":16.521000000000001},"last":3.2290000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.6100000000000001,"15min":1.1459999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.984,"5min":10.340999999999999,"15min":10.340999999999999},"last":4.3079999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.617,"15min":1.3799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":7.0339999999999998,"5min":16.27,"15min":21.100000000000001},"last":0.314},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.45,"15min":1.381},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27600000000000002},"max":{"1min":11.976000000000001,"5min":11.976000000000001,"15min":16.446000000000002},"last":0.93400000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.7130000000000001,"15min":1.125},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":7.2069999999999999,"5min":16.027000000000001,"15min":16.027000000000001},"last":4.6340000000000003},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.716,"15min":1.083},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":8.4659999999999993,"5min":10.141,"15min":10.141},"last":4.6429999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:00:08 2024","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.5070000000000001,"15min":1.456},"min":{"1min":0.308,"5min":0.308,"15min":0.26400000000000001},"max":{"1min":7.1600000000000001,"5min":16.190999999999999,"15min":16.190999999999999},"last":3.1320000000000001},{"interface":"front","average":{"1min":3.028,"5min":2.2080000000000002,"15min":1.431},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":11.423,"5min":18.170999999999999,"15min":18.170999999999999},"last":3.2570000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":2.0640000000000001,"15min":1.2110000000000001},"min":{"1min":0.41599999999999998,"5min":0.32100000000000001,"15min":0.314},"max":{"1min":5.077,"5min":17.907,"15min":17.907},"last":4.0309999999999997},{"interface":"front","average":{"1min":1.7030000000000001,"5min":2.3599999999999999,"15min":1.3600000000000001},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.27900000000000003},"max":{"1min":5.5880000000000001,"5min":21.062000000000001,"15min":21.062000000000001},"last":3.1629999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.472,"5min":1.9390000000000001,"15min":1.2669999999999999},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.32400000000000001},"max":{"1min":12.058,"5min":16.175000000000001,"15min":16.175000000000001},"last":4.1799999999999997},{"interface":"front","average":{"1min":3.79,"5min":2.1589999999999998,"15min":1.274},"min":{"1min":0.45600000000000002,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":34.722999999999999,"5min":34.722999999999999,"15min":34.722999999999999},"last":3.1829999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411485,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":788144,"kb_used_data":486760,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92957008,"statfs":{"total":95995035648,"available":95187976192,"internally_reserved":0,"allocated":498442240,"data_stored":497758488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.9550000000000001,"15min":1.3440000000000001},"min":{"1min":0.192,"5min":0.186,"15min":0.186},"max":{"1min":3.7400000000000002,"5min":15.808,"15min":15.808},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.756,"15min":1.1899999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.158},"max":{"1min":6.1070000000000002,"5min":13.497,"15min":19.893000000000001},"last":0.32100000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":2.0009999999999999,"15min":1.4099999999999999},"min":{"1min":0.23400000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":6.5819999999999999,"5min":17.739000000000001,"15min":17.739000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.9399999999999999,"15min":1.4059999999999999},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":3.7629999999999999,"5min":15.766999999999999,"15min":15.766999999999999},"last":4.3099999999999996}]},{"osd":3,"last update":"Mon Feb 12 17:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.3939999999999999,"15min":1.5700000000000001},"min":{"1min":0.29899999999999999,"5min":0.28000000000000003,"15min":0.23499999999999999},"max":{"1min":2.976,"5min":13.596,"15min":51.579999999999998},"last":4.2759999999999998},{"interface":"front","average":{"1min":1.159,"5min":1.3720000000000001,"15min":1.337},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1310000000000002,"5min":13.521000000000001,"15min":48.988999999999997},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:59:50 2024","interfaces":[{"interface":"back","average":{"1min":2.415,"5min":2.2930000000000001,"15min":1.361},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.26700000000000002},"max":{"1min":7.2789999999999999,"5min":15.737,"15min":15.737},"last":4.3559999999999999},{"interface":"front","average":{"1min":2.4630000000000001,"5min":2.2080000000000002,"15min":1.252},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.26800000000000002},"max":{"1min":7.5449999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":2.081,"15min":1.4379999999999999},"min":{"1min":0.41499999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":10.407999999999999,"5min":15.699,"15min":15.699},"last":4.335},{"interface":"front","average":{"1min":2.9580000000000002,"5min":2.1560000000000001,"15min":1.518},"min":{"1min":0.33600000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":11.999000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":13.207000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:00:25 2024","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.4430000000000001,"15min":1.7050000000000001},"min":{"1min":0.377,"5min":0.36399999999999999,"15min":0.31},"max":{"1min":6.6550000000000002,"5min":16.739000000000001,"15min":17.782},"last":0.496},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.2879999999999998,"15min":1.476},"min":{"1min":0.39600000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":8.7850000000000001,"5min":13.446999999999999,"15min":13.446999999999999},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.8330000000000002,"5min":2.2290000000000001,"15min":1.5149999999999999},"min":{"1min":0.45500000000000002,"5min":0.36299999999999999,"15min":0.32600000000000001},"max":{"1min":8.8520000000000003,"5min":13.473000000000001,"15min":13.473000000000001},"last":2.5699999999999998},{"interface":"front","average":{"1min":3.387,"5min":2.3340000000000001,"15min":1.546},"min":{"1min":0.36199999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":16.768000000000001,"5min":16.768000000000001,"15min":16.768000000000001},"last":0.46999999999999997}]}]},{"osd":3,"up_from":0,"seq":111669149849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953621,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773672,"kb_used_data":476448,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92971480,"statfs":{"total":95995035648,"available":95202795520,"internally_reserved":0,"allocated":487882752,"data_stored":487206615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":2.9140000000000001,"5min":1.7889999999999999,"15min":1.2170000000000001},"min":{"1min":0.20899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":38.179000000000002},"last":0.214},{"interface":"front","average":{"1min":3.351,"5min":1.9199999999999999,"15min":1.258},"min":{"1min":0.23599999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.6760000000000002,"5min":20.244,"15min":38.151000000000003},"last":5.5659999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0510000000000002,"5min":2.004,"15min":1.1000000000000001},"min":{"1min":0.251,"5min":0.218,"15min":0.218},"max":{"1min":11.077999999999999,"5min":11.077999999999999,"15min":11.077999999999999},"last":10.266999999999999},{"interface":"front","average":{"1min":2.8069999999999999,"5min":2.0259999999999998,"15min":1.121},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"last":0.245}]},{"osd":2,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.1419999999999999,"15min":1.161},"min":{"1min":0.25800000000000001,"5min":0.252,"15min":0.22500000000000001},"max":{"1min":9.5790000000000006,"5min":9.6349999999999998,"15min":9.6349999999999998},"last":0.26600000000000001},{"interface":"front","average":{"1min":3.2759999999999998,"5min":2.129,"15min":1.1830000000000001},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":9.6370000000000005,"5min":10.106,"15min":10.106},"last":5.5119999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:59:37 2024","interfaces":[{"interface":"back","average":{"1min":3.177,"5min":2.153,"15min":1.488},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.26700000000000002},"max":{"1min":9.6969999999999992,"5min":20.271000000000001,"15min":38.170999999999999},"last":5.5860000000000003},{"interface":"front","average":{"1min":3.496,"5min":1.9339999999999999,"15min":1.339},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32000000000000001},"max":{"1min":10.865,"5min":10.865,"15min":38.311999999999998},"last":10.303000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:00:04 2024","interfaces":[{"interface":"back","average":{"1min":6.6989999999999998,"5min":3.9649999999999999,"15min":1.589},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":42.420000000000002,"5min":79.573999999999998,"15min":79.573999999999998},"last":10.678000000000001},{"interface":"front","average":{"1min":7.8550000000000004,"5min":3.964,"15min":1.575},"min":{"1min":0.622,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":45.204999999999998,"5min":80.497,"15min":80.497},"last":3.1299999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:59:33 2024","interfaces":[{"interface":"back","average":{"1min":4.4020000000000001,"5min":3.3130000000000002,"15min":1.3799999999999999},"min":{"1min":0.373,"5min":0.27000000000000002,"15min":0.255},"max":{"1min":10.49,"5min":104.907,"15min":104.907},"last":5.5449999999999999},{"interface":"front","average":{"1min":5.5149999999999997,"5min":3.5910000000000002,"15min":1.4850000000000001},"min":{"1min":0.54100000000000004,"5min":0.28699999999999998,"15min":0.26500000000000001},"max":{"1min":13.739000000000001,"5min":105.395,"15min":105.395},"last":10.336}]},{"osd":7,"last update":"Mon Feb 12 16:59:47 2024","interfaces":[{"interface":"back","average":{"1min":3.6819999999999999,"5min":2.1219999999999999,"15min":1.004},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":11.786,"5min":12.311999999999999,"15min":12.311999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":5.7000000000000002,"5min":2.7170000000000001,"15min":1.2270000000000001},"min":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.28799999999999998},"max":{"1min":21.079000000000001,"5min":27.773,"15min":27.773},"last":5.6040000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757393,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855508,"kb_used_data":549900,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92889644,"statfs":{"total":95995035648,"available":95118995456,"internally_reserved":0,"allocated":563097600,"data_stored":562377556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.8050000000000002,"5min":1.72,"15min":1.1839999999999999},"min":{"1min":0.29399999999999998,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":8.7530000000000001,"5min":8.7530000000000001,"15min":36.880000000000003},"last":10.976000000000001},{"interface":"front","average":{"1min":3.4359999999999999,"5min":2.411,"15min":1.417},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.9779999999999998,"5min":10.752000000000001,"15min":36.954000000000001},"last":9.7579999999999991}]},{"osd":1,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.1699999999999999,"5min":2.198,"15min":1.202},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.8380000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":7.327},{"interface":"front","average":{"1min":3.1659999999999999,"5min":2.3490000000000002,"15min":1.327},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9459999999999997,"5min":16.712,"15min":16.712},"last":9.7729999999999997}]},{"osd":2,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.2629999999999999,"5min":2.5009999999999999,"15min":1.399},"min":{"1min":0.40000000000000002,"5min":0.247,"15min":0.23499999999999999},"max":{"1min":8.8019999999999996,"5min":16.940999999999999,"15min":18.218},"last":11.029},{"interface":"front","average":{"1min":2.3879999999999999,"5min":2.0960000000000001,"15min":1.2889999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":6.819,"5min":17.103999999999999,"15min":18.327000000000002},"last":7.3559999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":1.9319999999999999,"15min":1.321},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.8230000000000004,"5min":8.8230000000000004,"15min":18.251000000000001},"last":11.122},{"interface":"front","average":{"1min":2.3620000000000001,"5min":1.7490000000000001,"15min":1.2290000000000001},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.7720000000000002,"5min":7.048,"15min":20.263000000000002},"last":7.3799999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":4.2220000000000004,"5min":2.0569999999999999,"15min":0.98299999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":10.021000000000001,"5min":11.880000000000001,"15min":11.880000000000001},"last":9.7959999999999994},{"interface":"front","average":{"1min":4.7709999999999999,"5min":2.601,"15min":1.161},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":11.004}]},{"osd":6,"last update":"Mon Feb 12 17:00:29 2024","interfaces":[{"interface":"back","average":{"1min":4.742,"5min":2.415,"15min":1.1759999999999999},"min":{"1min":0.34999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.281000000000001,"5min":12.281000000000001,"15min":12.281000000000001},"last":9.7319999999999993},{"interface":"front","average":{"1min":4.8710000000000004,"5min":2.6789999999999998,"15min":1.2350000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26700000000000002},"max":{"1min":14.323,"5min":22.035,"15min":22.035},"last":14.323}]},{"osd":7,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":1.889,"15min":0.97899999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":6.7999999999999998,"5min":9.484,"15min":9.484},"last":7.5069999999999997},{"interface":"front","average":{"1min":3.7490000000000001,"5min":2.04,"15min":1.044},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":12.253,"5min":12.253,"15min":12.253},"last":9.8079999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383254528,"data_stored":383254528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":874643456,"data_stored":874643456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":474550272,"data_stored":474546200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":470417408,"data_stored":470417408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":531304448,"data_stored":531300376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":569311232,"data_stored":569311232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":340201472,"data_stored":340197400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:00:37.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:37.930 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:37.964 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:37.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:38.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:38.540 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:38.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:38.651 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:38.684 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:38.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:38.822 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:38.859 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:38.905 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:38.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:39.013 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:39.061 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:39.082 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:00:39.087 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:00:39.088 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:39.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:39.111 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:00:39.145 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:00:39.177 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:00:39.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:39.201 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:00:39.248 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:00:39.257 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:39.277 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:00:39.311 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:39.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:39.326 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:00:39.332 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:00:39.332 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:00:39.332 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:00:39.332 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:00:39.332 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:00:39.332 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:00:39.333 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:00:39.333 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:00:39.417 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:39.531 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:39.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:39.624 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:38.562306+0000 mgr.y (mgr.24481) 420 : cluster [DBG] pgmap v543: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 584 MiB data, 3.1 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s; 2093/28590 objects misplaced (7.321%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T17:00:39.624 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:39.041460+0000 mon.c (mon.2) 161 : audit [DBG] from='client.? v1:172.21.15.33:0/198246890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:39.633 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:38.562306+0000 mgr.y (mgr.24481) 420 : cluster [DBG] pgmap v543: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 584 MiB data, 3.1 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s; 2093/28590 objects misplaced (7.321%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T17:00:39.633 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:39.041460+0000 mon.c (mon.2) 161 : audit [DBG] from='client.? v1:172.21.15.33:0/198246890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:39.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:39.704 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:39.709 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:00:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:00:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:215869 8.5 8:a4c85369:::benchmark_data_smithi033_105085_object104211:head [delete] snapc 0=[] ondisk+write+known_if_redirected e291)", 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:39.670195+0000", 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.033687188, 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033715440999999999, 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 215869 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670195+0000", 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670195+0000", 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670193+0000", 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5649999999999998e-06 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670199+0000", 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3430000000000002e-06 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670209+0000", 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5199999999999997e-06 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670214+0000", 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021604630000000001 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.672375+0000", 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013129099999999999 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.672506+0000", 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015892099999999999 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.672665+0000", 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036361330000000002 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.676301+0000", 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018568817000000001 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.694870+0000", 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0251999999999998e-05 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.694930+0000", 2024-02-12T17:00:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0089219280000000008 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.703852+0000", 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7472999999999997e-05 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.703909+0000", 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033714468999999997 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:215871 8.5 8:ae42db45:::benchmark_data_smithi033_105085_object104213:head [delete] snapc 0=[] ondisk+write+known_if_redirected e291)", 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:39.670228+0000", 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0336546, 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033799454999999999, 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 215871 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670228+0000", 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670228+0000", 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:00:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670227+0000", 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.039e-06 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670230+0000", 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0719999999999999e-06 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670235+0000", 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4079999999999998e-06 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.670238+0000", 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042181880000000003 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.674457+0000", 2024-02-12T17:00:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125609 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.674582+0000", 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017578299999999999 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.674758+0000", 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019819970000000001 2024-02-12T17:00:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.676740+0000", 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018167004 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.694907+0000", 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061739639999999997 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701081+0000", 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030853304000000002 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:215882 8.5 8:a8435488:::benchmark_data_smithi033_105085_object104224:head [delete] snapc 0=[] ondisk+write+known_if_redirected e291)", 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:39.678072+0000", 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.025810224, 2024-02-12T17:00:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.026056293000000001, 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 215882 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678072+0000", 2024-02-12T17:00:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678072+0000", 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678069+0000", 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.661e-06 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678077+0000", 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0648e-05 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678088+0000", 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678094+0000", 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113446 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678207+0000", 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4938999999999994e-05 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678292+0000", 2024-02-12T17:00:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124294 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.678417+0000", 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013856021 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.692273+0000", 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062760960000000001 2024-02-12T17:00:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.698549+0000", 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020476815999999998 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:215884 8.d e291/176 8:baf34329:::benchmark_data_smithi033_105085_object104226:head v 291'13709, mlcod=291'13705)", 2024-02-12T17:00:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:39.695047+0000", 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0088356849999999994, 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0091835819999999992, 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.695047+0000", 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.695047+0000", 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.695046+0000", 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9600000000000006e-06 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.695055+0000", 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2460000000000003e-06 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.695060+0000", 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1139999999999999e-06 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.695064+0000", 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0068326080000000004 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701897+0000", 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0145000000000001e-05 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701917+0000", 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0068705470000000003 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.35042.0:215871 8.5 e291/258 ondisk, result = 0)", 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:39.701761+0000", 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0021212729999999999, 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002528757, 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701761+0000", 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701761+0000", 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701761+0000", 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.419e-06 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701763+0000", 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4460000000000003e-06 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701767+0000", 2024-02-12T17:00:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9110000000000003e-06 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701771+0000", 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022950539999999999 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704066+0000", 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2219999999999996e-06 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704074+0000", 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023126240000000001 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.35042.0:215882 8.5 e291/258)", 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:39.701774+0000", 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0021078239999999999, 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025891209999999998, 2024-02-12T17:00:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "queued for pg", 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701774+0000", 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701774+0000", 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701774+0000", 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9719999999999999e-06 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701776+0000", 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.16e-06 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701780+0000", 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.388e-06 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.701784+0000", 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2070000000000001e-06 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:215896 8.8 e291/176 8:167b566c:::benchmark_data_smithi033_105085_object104238:head v 291'13683, mlcod=291'13681)", 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:39.704213+0000", 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4294967295.9996696, 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020260100000000001, 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704213+0000", 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704213+0000", 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704212+0000", 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704219+0000", 2024-02-12T17:00:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7489999999999993e-06 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704229+0000", 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6670000000000001e-06 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704235+0000", 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4570000000000001e-05 2024-02-12T17:00:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704289+0000", 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7751999999999999e-05 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:39.704307+0000", 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4166999999999995e-05 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T17:00:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 7 2024-02-12T17:00:39.746 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:00:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:39.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:39.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:39.888 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:38.562306+0000 mgr.y (mgr.24481) 420 : cluster [DBG] pgmap v543: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 584 MiB data, 3.1 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s; 2093/28590 objects misplaced (7.321%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T17:00:39.888 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:39.041460+0000 mon.c (mon.2) 161 : audit [DBG] from='client.? v1:172.21.15.33:0/198246890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:39.891 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:40.255 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:40.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:40.272 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:40.364 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:40.412 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T17:00:40.454 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:40.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:40.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:00:40.653 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:40.684 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:00:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:40.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:00:40.790 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:40.791 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:00:40.868 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:40.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:41.017 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:41.058 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:41.108 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:41.110 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:00:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:40] "GET /metrics HTTP/1.1" 200 175962 "" "Prometheus/2.33.4" 2024-02-12T17:00:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:41.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:41.212 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:41.246 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:41.802 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:41.810 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:41.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:41.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:41.955 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:40.562860+0000 mgr.y (mgr.24481) 421 : cluster [DBG] pgmap v544: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 523 MiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 998 op/s; 1942/25686 objects misplaced (7.561%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:41.988 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:42.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:40.562860+0000 mgr.y (mgr.24481) 421 : cluster [DBG] pgmap v544: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 523 MiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 998 op/s; 1942/25686 objects misplaced (7.561%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:42.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:40.562860+0000 mgr.y (mgr.24481) 421 : cluster [DBG] pgmap v544: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 523 MiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 998 op/s; 1942/25686 objects misplaced (7.561%); 4.4 MiB/s, 69 objects/s recovering 2024-02-12T17:00:42.265 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:00:42.339 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:42.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:42.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:42.514 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:42.563 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:42.569 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:00:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:00:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:42.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:42.670 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:42.725 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:42.976 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:00:42.976 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:00:42.977 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:00:42.977 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:00:42.977 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:00:43.546 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:43.666 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:00:43.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:42.563408+0000 mgr.y (mgr.24481) 422 : cluster [DBG] pgmap v545: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 523 MiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 1942/25686 objects misplaced (7.561%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:43.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:42.794788+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? v1:172.21.15.33:0/3954345788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:43.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:42.563408+0000 mgr.y (mgr.24481) 422 : cluster [DBG] pgmap v545: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 523 MiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 1942/25686 objects misplaced (7.561%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:43.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:42.794788+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? v1:172.21.15.33:0/3954345788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:43.969 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:42.563408+0000 mgr.y (mgr.24481) 422 : cluster [DBG] pgmap v545: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 523 MiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 1942/25686 objects misplaced (7.561%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:00:43.970 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:42.794788+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? v1:172.21.15.33:0/3954345788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:00:43.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T17:00:44.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:00:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:00:44.435 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:44.715 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:44.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:44.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:44.950 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:43.633236+0000 mon.a (mon.0) 1212 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:00:44.950 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:43.660256+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.33:0/3954345788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:44.950 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:43.660309+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e292: 8 total, 8 up, 5 in 2024-02-12T17:00:44.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:44.035882+0000 mgr.y (mgr.24481) 423 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:44.954 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:43.633236+0000 mon.a (mon.0) 1212 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:00:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:43.660256+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.33:0/3954345788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:43.660309+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e292: 8 total, 8 up, 5 in 2024-02-12T17:00:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:44.035882+0000 mgr.y (mgr.24481) 423 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:43.633236+0000 mon.a (mon.0) 1212 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:00:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:43.660256+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.33:0/3954345788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:00:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:43.660309+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e292: 8 total, 8 up, 5 in 2024-02-12T17:00:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:44.035882+0000 mgr.y (mgr.24481) 423 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:45.066 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:45.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:00:45.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:00:45.232 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:45.382 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:00:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:45.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:00:45.489 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:45.536 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:45.910 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:44.563746+0000 mgr.y (mgr.24481) 424 : cluster [DBG] pgmap v547: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 269 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 773/13509 objects misplaced (5.722%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:00:46.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:44.563746+0000 mgr.y (mgr.24481) 424 : cluster [DBG] pgmap v547: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 269 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 773/13509 objects misplaced (5.722%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:00:46.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:44.563746+0000 mgr.y (mgr.24481) 424 : cluster [DBG] pgmap v547: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 269 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 773/13509 objects misplaced (5.722%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:00:46.509 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:00:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:00:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:00:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:00:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:80708 7.5 7:a793f08c:::benchmark_data_smithi033_81060_object80707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:29.050600+0000", 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 437.45434864499998, 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057351045000000003, 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 80708 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050600+0000", 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050600+0000", 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050598+0000", 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3399000000000002e-05 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050651+0000", 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2693e-05 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050664+0000", 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8950000000000001e-06 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050671+0000", 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6180999999999997e-05 2024-02-12T17:00:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050717+0000", 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0001003 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050817+0000", 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112668 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.050930+0000", 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021345280000000001 2024-02-12T17:00:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053065+0000", 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024767410000000002 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.055541+0000", 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052323124999999998 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107864+0000", 2024-02-12T17:00:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3448000000000001e-05 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107918+0000", 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3254999999999999e-05 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.107951+0000", 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.057351045000000003 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:80710 7.5 7:a5b76bb0:::benchmark_data_smithi033_81060_object80709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:53:29.053581+0000", 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 437.45136742, 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062171000999999997, 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 80710 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053581+0000", 2024-02-12T17:00:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053581+0000", 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053580+0000", 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9789000000000001e-05 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053629+0000", 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6770000000000001e-06 2024-02-12T17:00:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053638+0000", 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5739999999999997e-06 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053643+0000", 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3254999999999998e-05 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053686+0000", 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013029299999999999 2024-02-12T17:00:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053816+0000", 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018151500000000001 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.053998+0000", 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016970679999999999 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.055695+0000", 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053556299999999997 2024-02-12T17:00:46.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.056230+0000", 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059463378999999997 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115694+0000", 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5593999999999998e-05 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115729+0000", 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2915999999999999e-05 2024-02-12T17:00:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:53:29.115752+0000", 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062171000999999997 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.14838.0:157562 7.5 7:abd29c58:::benchmark_data_smithi033_81060_object47110:head [delete] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:55:00.723417+0000", 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 345.78153129899999, 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048460371000000002, 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.14838", 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/384518940", 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 157562 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723417+0000", 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723417+0000", 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723415+0000", 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8360000000000004e-06 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723425+0000", 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8109999999999997e-06 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723435+0000", 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4220000000000002e-06 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723439+0000", 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5435000000000001e-05 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723485+0000", 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013159099999999999 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723616+0000", 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012698599999999999 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.723743+0000", 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017941299999999999 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.725537+0000", 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0089994559999999994 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.734537+0000", 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.03725999 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771797+0000", 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7404000000000002e-05 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771844+0000", 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3670000000000001e-05 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:55:00.771878+0000", 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048460371000000002 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:82097 8.c e203/176 8:3316d78f:::benchmark_data_smithi033_105085_object82096:head v 203'5192, mlcod=203'5187)", 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:43.240223+0000", 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.26472533200001, 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048025215000000003, 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240223+0000", 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240223+0000", 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240223+0000", 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4352e-05 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240247+0000", 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7869999999999999e-06 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240251+0000", 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6129999999999998e-06 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240254+0000", 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001769891 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242023+0000", 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8575000000000001e-05 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.242042+0000", 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046170577999999997 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288213+0000", 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5944e-05 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288249+0000", 2024-02-12T17:00:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048025215000000003 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:82098 8.c e203/176 8:38294761:::benchmark_data_smithi033_105085_object82097:head v 203'5193, mlcod=203'5187)", 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:43.240257+0000", 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.264691465, 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048015358000000001, 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240257+0000", 2024-02-12T17:00:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240257+0000", 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240257+0000", 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1943e-05 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240279+0000", 2024-02-12T17:00:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.847e-06 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240282+0000", 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1069999999999999e-06 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.240284+0000", 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035635609999999998 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.243847+0000", 2024-02-12T17:00:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7524999999999999e-05 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.243865+0000", 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.044389989999999997 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288255+0000", 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7643999999999999e-05 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:43.288273+0000", 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048015358000000001 2024-02-12T17:00:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.35042.0:100923 8.5 e230/176 8:a8c11442:::benchmark_data_smithi033_105085_object100922:head v 230'6276, mlcod=230'6273)", 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:58:09.421674+0000", 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 157.083274569, 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048355071999999999, 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.421674+0000", 2024-02-12T17:00:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.421674+0000", 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.421673+0000", 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2827999999999999e-05 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.421696+0000", 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9420000000000007e-06 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.421706+0000", 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6350000000000003e-06 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.421711+0000", 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011795244999999999 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.433507+0000", 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6489999999999999e-05 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.433533+0000", 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.036458882999999997 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.469992+0000", 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7184000000000002e-05 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:58:09.470029+0000", 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048355071999999999 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(5.3 258/257 [PushOp(5:c609908c:::notify.5:head, version: 65'8, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c609908c:::notify.5:head@65'8, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:09.291291+0000", 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 97.213657834000003, 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.049285852999999998, 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.291291+0000", 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.291291+0000", 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.291290+0000", 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6078000000000001e-05 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.291306+0000", 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6375000000000002e-05 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.291322+0000", 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0363000000000001e-05 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.291332+0000", 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.049102355 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340435+0000", 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7519999999999996e-06 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340443+0000", 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013417799999999999 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340577+0000", 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.049285852999999998 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:161692 8.5 8:a82e2194:::benchmark_data_smithi033_105085_object50034:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:31.373185+0000", 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.131764051000005, 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.073308156999999999, 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 161692 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.373185+0000", 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.373185+0000", 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.373183+0000", 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1749999999999999e-06 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.373190+0000", 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0095000000000001e-05 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.373200+0000", 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6049999999999997e-06 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.373206+0000", 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0060737530000000003 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379279+0000", 2024-02-12T17:00:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014797700000000001 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379427+0000", 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014448099999999999 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379572+0000", 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026724990000000001 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.382244+0000", 2024-02-12T17:00:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079583529999999996 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.390203+0000", 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0094314710000000003 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.399634+0000", 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046625099000000003 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446259+0000", 2024-02-12T17:00:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2578999999999997e-05 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446332+0000", 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016090899999999999 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446493+0000", 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.073308156999999999 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:161693 8.5 8:a71c8a1d:::benchmark_data_smithi033_105085_object50035:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:31.379238+0000", 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.125711117999998, 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067506902999999993, 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 161693 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379238+0000", 2024-02-12T17:00:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379238+0000", 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379236+0000", 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1869999999999997e-06 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379244+0000", 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0469e-05 2024-02-12T17:00:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379264+0000", 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1369999999999999e-06 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379270+0000", 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029963680000000001 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.382267+0000", 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011827100000000001 2024-02-12T17:00:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.382385+0000", 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015503899999999999 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.382540+0000", 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0048907120000000002 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387431+0000", 2024-02-12T17:00:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027789820000000002 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.390210+0000", 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0094584509999999997 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.399668+0000", 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046875905000000002 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446544+0000", 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4128000000000007e-05 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446608+0000", 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000136253 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446744+0000", 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067506902999999993 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:161694 8.5 8:a87abceb:::benchmark_data_smithi033_105085_object50036:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:31.379275+0000", 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.125673542000001, 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067656343999999993, 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 161694 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379275+0000", 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379275+0000", 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379274+0000", 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1209999999999998e-06 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379277+0000", 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2890000000000002e-06 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379281+0000", 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.261e-06 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.379283+0000", 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051585040000000004 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.384442+0000", 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000128467 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.384570+0000", 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016145999999999999 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.384731+0000", 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027340390000000002 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387465+0000", 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002750911 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.390216+0000", 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0094846459999999994 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.399701+0000", 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047091178999999997 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446792+0000", 2024-02-12T17:00:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.448e-05 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446847+0000", 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4800000000000001e-05 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446932+0000", 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067656343999999993 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:161715 8.5 8:a1f10e4d:::benchmark_data_smithi033_105085_object50057:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:31.387223+0000", 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.117725320999995, 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059920349999999997, 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 161715 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387223+0000", 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387223+0000", 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387221+0000", 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7810000000000005e-06 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387229+0000", 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3456e-05 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387242+0000", 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.319e-06 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387248+0000", 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000238323 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387487+0000", 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121977 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387609+0000", 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000153533 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.387762+0000", 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00552186 2024-02-12T17:00:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.393284+0000", 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064571790000000004 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.399741+0000", 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044785090000000003 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.404220+0000", 2024-02-12T17:00:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042758431 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.446978+0000", 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3825999999999997e-05 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.447022+0000", 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121624 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:31.447144+0000", 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059920349999999997 2024-02-12T17:00:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:165537 8.5 8:a9dffffb:::benchmark_data_smithi033_105085_object53879:head [delete] snapc 0=[] ondisk+write+known_if_redirected e271)", 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:36.276334+0000", 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.228615113000004, 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058421941999999998, 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 165537 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.276334+0000", 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.276334+0000", 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.276332+0000", 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0239999999999997e-06 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.276337+0000", 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.52e-06 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.276344+0000", 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6659999999999999e-06 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.276349+0000", 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023989751 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.300339+0000", 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013704200000000001 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.300476+0000", 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016357200000000001 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.300639+0000", 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0039036510000000002 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.304543+0000", 2024-02-12T17:00:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0041530789999999996 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.308696+0000", 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015692517 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.324389+0000", 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010219707 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.334608+0000", 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3112000000000003e-05 2024-02-12T17:00:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.334652+0000", 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000104022 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:36.334756+0000", 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058421941999999998 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(8.5 272/258 [PushReplyOp(8:a1294c77:::benchmark_data_smithi033_105085_object111213:head)])", 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:41.217937+0000", 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.287011598999996, 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.090223213999999996, 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217937+0000", 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217937+0000", 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217937+0000", 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9809999999999997e-06 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217941+0000", 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2790000000000004e-06 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217947+0000", 2024-02-12T17:00:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6910000000000002e-06 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217953+0000", 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089966955000000001 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.307920+0000", 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024068900000000001 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.308160+0000", 2024-02-12T17:00:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.090223213999999996 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:182112 8.5 8:ac00dd64:::benchmark_data_smithi033_105085_object70454:head [delete] snapc 0=[] ondisk+write+known_if_redirected e279)", 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:57.463352+0000", 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.041596472999998, 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.065682243000000001, 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 182112 2024-02-12T17:00:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463352+0000", 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463352+0000", 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463350+0000", 2024-02-12T17:00:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.259e-06 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463358+0000", 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2826000000000001e-05 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463371+0000", 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8789999999999997e-06 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463378+0000", 2024-02-12T17:00:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8349000000000001e-05 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463426+0000", 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010438300000000001 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463530+0000", 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1497999999999997e-05 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.463622+0000", 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026132429999999999 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.466235+0000", 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018736810000000001 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.468109+0000", 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015737279999999999 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.469682+0000", 2024-02-12T17:00:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059178586999999998 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.528861+0000", 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1341000000000002e-05 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.528912+0000", 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012209699999999999 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:57.529034+0000", 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.065682243000000001 2024-02-12T17:00:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:199721 8.5 8:aaf1b513:::benchmark_data_smithi033_105085_object88063:head [delete] snapc 0=[] ondisk+write+known_if_redirected e287)", 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:19.403207+0000", 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.101742045000002, 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048726408999999998, 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 199721 2024-02-12T17:00:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403207+0000", 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403207+0000", 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403204+0000", 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8140000000000001e-06 2024-02-12T17:00:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403212+0000", 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1336e-05 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403223+0000", 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3409999999999999e-06 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403230+0000", 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1972999999999999e-05 2024-02-12T17:00:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403282+0000", 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000132516 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403414+0000", 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156329 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.403570+0000", 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045509829999999998 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.408121+0000", 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044274409999999998 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.412549+0000", 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023417499999999999 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.412783+0000", 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038944355 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.451727+0000", 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8508000000000003e-05 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.451786+0000", 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014710999999999999 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:19.451933+0000", 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048726408999999998 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:201440 8.5 8:a9e0ba37:::benchmark_data_smithi033_105085_object89782:head [delete] snapc 0=[] ondisk+write+known_if_redirected e287)", 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:21.631583+0000", 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.873366013999998, 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051764366999999999, 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 201440 2024-02-12T17:00:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.631583+0000", 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.631583+0000", 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.631580+0000", 2024-02-12T17:00:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7639999999999994e-06 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.631589+0000", 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2557e-05 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.631602+0000", 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7970000000000003e-06 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.631608+0000", 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0041423909999999996 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.635751+0000", 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013522999999999999 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.635886+0000", 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000168586 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.636055+0000", 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038314690000000001 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.639886+0000", 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011358823000000001 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.651245+0000", 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030719639999999999 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.654317+0000", 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028843793 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.683161+0000", 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9735000000000003e-05 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.683220+0000", 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012670199999999999 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:21.683347+0000", 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051764366999999999 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(8.5 287/258 [PushReplyOp(8:a4c25a62:::benchmark_data_smithi033_105085_object101867:head)])", 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:24.801704+0000", 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.70324518, 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086370809000000007, 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801704+0000", 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801704+0000", 2024-02-12T17:00:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801703+0000", 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8499999999999996e-06 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801710+0000", 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9939999999999997e-06 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801716+0000", 2024-02-12T17:00:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.75e-06 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801722+0000", 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086105397 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.887827+0000", 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024725700000000002 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.888074+0000", 2024-02-12T17:00:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086370809000000007 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:219385 8.5 8:a9179a9f:::benchmark_data_smithi033_105085_object107727:head [delete] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:43.985699+0000", 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 2.5192499339999999, 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047828127999999998, 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 219385 2024-02-12T17:00:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.985699+0000", 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.985699+0000", 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.985696+0000", 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3760000000000002e-06 2024-02-12T17:00:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.985704+0000", 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5440000000000004e-06 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.985713+0000", 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5480000000000001e-06 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.985719+0000", 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006493899 2024-02-12T17:00:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.992213+0000", 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000131459 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.992344+0000", 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016835099999999999 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.992513+0000", 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038174099999999998 2024-02-12T17:00:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.996330+0000", 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085233900000000003 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:43.997182+0000", 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034708650000000001 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.000653+0000", 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032670802999999998 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.033324+0000", 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8242999999999998e-05 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.033382+0000", 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014471899999999999 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.033527+0000", 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047828127999999998 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:219621 8.5 8:a1959ce9:::benchmark_data_smithi033_105085_object107963:head [delete] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:44.329212+0000", 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 2.1757369620000002, 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052978697999999998, 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 219621 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329212+0000", 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329212+0000", 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329210+0000", 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1770000000000001e-06 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329217+0000", 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0744000000000001e-05 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329228+0000", 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8780000000000003e-06 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329234+0000", 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0991000000000001e-05 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329274+0000", 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015210700000000001 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329427+0000", 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016951299999999999 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.329596+0000", 2024-02-12T17:00:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088761810000000003 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.338472+0000", 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019016790000000001 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.340374+0000", 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010705189 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.351079+0000", 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030929154 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.382008+0000", 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3706999999999999e-05 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.382062+0000", 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000128417 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:44.382190+0000", 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052978697999999998 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220717 8.5 8:a741cf55:::benchmark_data_smithi033_105085_object109059:head [delete] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.669913+0000", 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.83503595100000005, 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.081121243999999995, 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220717 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:00:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.669913+0000", 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.669913+0000", 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.669911+0000", 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5180000000000001e-06 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.669917+0000", 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0866999999999999e-05 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.669928+0000", 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2750000000000001e-06 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.669934+0000", 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156918 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:00:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.670091+0000", 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014254299999999999 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.670234+0000", 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016736299999999999 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,6", 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.670401+0000", 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002231551 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:00:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.672633+0000", 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036521879999999998 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.676285+0000", 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012995382 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.689280+0000", 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.061570784000000003 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.750851+0000", 2024-02-12T17:00:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2429000000000002e-05 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.750903+0000", 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013058500000000001 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.751034+0000", 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.081121243999999995 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:00:46.614 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:00:46.673 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:45.667617+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e293: 8 total, 8 up, 5 in 2024-02-12T17:00:46.801 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:45.667617+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e293: 8 total, 8 up, 5 in 2024-02-12T17:00:47.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:45.667617+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e293: 8 total, 8 up, 5 in 2024-02-12T17:00:47.146 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T17:00:47.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:46.564065+0000 mgr.y (mgr.24481) 425 : cluster [DBG] pgmap v549: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 269 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 773/13509 objects misplaced (5.722%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:47.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:46.668094+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e294: 8 total, 8 up, 5 in 2024-02-12T17:00:47.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:47.484541+0000 mon.a (mon.0) 1217 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:00:48.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:46.564065+0000 mgr.y (mgr.24481) 425 : cluster [DBG] pgmap v549: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 269 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 773/13509 objects misplaced (5.722%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:48.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:46.668094+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e294: 8 total, 8 up, 5 in 2024-02-12T17:00:48.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:47.484541+0000 mon.a (mon.0) 1217 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:00:48.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:46.564065+0000 mgr.y (mgr.24481) 425 : cluster [DBG] pgmap v549: 138 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 132 active+clean; 269 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 773/13509 objects misplaced (5.722%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:00:48.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:46.668094+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e294: 8 total, 8 up, 5 in 2024-02-12T17:00:48.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:47.484541+0000 mon.a (mon.0) 1217 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:00:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:47.772988+0000 mon.a (mon.0) 1218 : cluster [DBG] osdmap e295: 8 total, 8 up, 5 in 2024-02-12T17:00:49.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:47.772988+0000 mon.a (mon.0) 1218 : cluster [DBG] osdmap e295: 8 total, 8 up, 5 in 2024-02-12T17:00:49.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:47.772988+0000 mon.a (mon.0) 1218 : cluster [DBG] osdmap e295: 8 total, 8 up, 5 in 2024-02-12T17:00:50.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:48.564813+0000 mgr.y (mgr.24481) 426 : cluster [DBG] pgmap v552: 138 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 133 active+clean; 123 MiB data, 1.9 GiB used, 445 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 479 op/s; 152/6465 objects misplaced (2.351%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T17:00:50.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:48.701226+0000 osd.5 (osd.5) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:50.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:48.701279+0000 osd.5 (osd.5) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:50.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:48.782827+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e296: 8 total, 8 up, 5 in 2024-02-12T17:00:50.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:49.301263+0000 osd.7 (osd.7) 121 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:00:50.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:49.304780+0000 osd.7 (osd.7) 122 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:48.564813+0000 mgr.y (mgr.24481) 426 : cluster [DBG] pgmap v552: 138 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 133 active+clean; 123 MiB data, 1.9 GiB used, 445 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 479 op/s; 152/6465 objects misplaced (2.351%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:48.701226+0000 osd.5 (osd.5) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:48.701279+0000 osd.5 (osd.5) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:48.782827+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e296: 8 total, 8 up, 5 in 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:49.301263+0000 osd.7 (osd.7) 121 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:49.304780+0000 osd.7 (osd.7) 122 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:48.564813+0000 mgr.y (mgr.24481) 426 : cluster [DBG] pgmap v552: 138 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 133 active+clean; 123 MiB data, 1.9 GiB used, 445 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 479 op/s; 152/6465 objects misplaced (2.351%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:48.701226+0000 osd.5 (osd.5) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:50.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:48.701279+0000 osd.5 (osd.5) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:50.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:48.782827+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e296: 8 total, 8 up, 5 in 2024-02-12T17:00:50.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:49.301263+0000 osd.7 (osd.7) 121 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:00:50.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:49.304780+0000 osd.7 (osd.7) 122 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:00:50.880 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:49.785477+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e297: 8 total, 8 up, 5 in 2024-02-12T17:00:50.880 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:49.843543+0000 osd.5 (osd.5) 129 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:00:50.880 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:49.846425+0000 osd.5 (osd.5) 130 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:00:50.881 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:49.785477+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e297: 8 total, 8 up, 5 in 2024-02-12T17:00:50.881 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:49.843543+0000 osd.5 (osd.5) 129 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:00:50.881 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:49.846425+0000 osd.5 (osd.5) 130 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:00:50.881 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:00:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:50] "GET /metrics HTTP/1.1" 200 175954 "" "Prometheus/2.33.4" 2024-02-12T17:00:51.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:49.785477+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e297: 8 total, 8 up, 5 in 2024-02-12T17:00:51.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:49.843543+0000 osd.5 (osd.5) 129 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:00:51.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:49.846425+0000 osd.5 (osd.5) 130 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:00:51.296 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Removed 111655 objects 2024-02-12T17:00:51.296 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Clean up completed and total clean up time :146.812 2024-02-12T17:00:51.382 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-02-12T17:00:51.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-02-12T17:00:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:50.565322+0000 mgr.y (mgr.24481) 427 : cluster [DBG] pgmap v555: 138 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 134 active+clean; 77 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 103/4260 objects misplaced (2.418%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:00:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:50.754385+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.d scrub starts 2024-02-12T17:00:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:50.756566+0000 osd.2 (osd.2) 144 : cluster [DBG] 2.d scrub ok 2024-02-12T17:00:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:50.806101+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e298: 8 total, 8 up, 5 in 2024-02-12T17:00:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:50.918442+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.18 scrub starts 2024-02-12T17:00:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:50.923455+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.18 scrub ok 2024-02-12T17:00:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:51.205197+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e299: 8 total, 8 up, 5 in 2024-02-12T17:00:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:51.266122+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:00:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:51.269644+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:00:52.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:50.565322+0000 mgr.y (mgr.24481) 427 : cluster [DBG] pgmap v555: 138 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 134 active+clean; 77 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 103/4260 objects misplaced (2.418%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:50.754385+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.d scrub starts 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:50.756566+0000 osd.2 (osd.2) 144 : cluster [DBG] 2.d scrub ok 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:50.806101+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e298: 8 total, 8 up, 5 in 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:50.918442+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.18 scrub starts 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:50.923455+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.18 scrub ok 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:51.205197+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e299: 8 total, 8 up, 5 in 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:51.266122+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:51.269644+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:50.565322+0000 mgr.y (mgr.24481) 427 : cluster [DBG] pgmap v555: 138 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 134 active+clean; 77 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 103/4260 objects misplaced (2.418%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:50.754385+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.d scrub starts 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:50.756566+0000 osd.2 (osd.2) 144 : cluster [DBG] 2.d scrub ok 2024-02-12T17:00:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:50.806101+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e298: 8 total, 8 up, 5 in 2024-02-12T17:00:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:50.918442+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.18 scrub starts 2024-02-12T17:00:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:50.923455+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.18 scrub ok 2024-02-12T17:00:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:51.205197+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e299: 8 total, 8 up, 5 in 2024-02-12T17:00:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:51.266122+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:00:52.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:51.269644+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:00:52.810 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:00:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:00:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:00:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:51.951396+0000 osd.0 (osd.0) 123 : cluster [DBG] 5.3 deep-scrub starts 2024-02-12T17:00:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:51.955716+0000 osd.0 (osd.0) 124 : cluster [DBG] 5.3 deep-scrub ok 2024-02-12T17:00:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:52.140703+0000 osd.4 (osd.4) 102 : cluster [DBG] 2.6 deep-scrub starts 2024-02-12T17:00:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:52.143792+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.6 deep-scrub ok 2024-02-12T17:00:53.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:51.951396+0000 osd.0 (osd.0) 123 : cluster [DBG] 5.3 deep-scrub starts 2024-02-12T17:00:53.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:51.955716+0000 osd.0 (osd.0) 124 : cluster [DBG] 5.3 deep-scrub ok 2024-02-12T17:00:53.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:52.140703+0000 osd.4 (osd.4) 102 : cluster [DBG] 2.6 deep-scrub starts 2024-02-12T17:00:53.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:52.143792+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.6 deep-scrub ok 2024-02-12T17:00:53.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:51.951396+0000 osd.0 (osd.0) 123 : cluster [DBG] 5.3 deep-scrub starts 2024-02-12T17:00:53.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:51.955716+0000 osd.0 (osd.0) 124 : cluster [DBG] 5.3 deep-scrub ok 2024-02-12T17:00:53.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:52.140703+0000 osd.4 (osd.4) 102 : cluster [DBG] 2.6 deep-scrub starts 2024-02-12T17:00:53.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:52.143792+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.6 deep-scrub ok 2024-02-12T17:00:54.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:52.565838+0000 mgr.y (mgr.24481) 428 : cluster [DBG] pgmap v558: 138 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 134 active+clean; 77 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 183 op/s; 103/4260 objects misplaced (2.418%); 544 KiB/s, 8 objects/s recovering 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:52.836886+0000 osd.2 (osd.2) 145 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:52.838845+0000 osd.2 (osd.2) 146 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.035780+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.035873+0000 osd.7 (osd.7) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.137985+0000 osd.4 (osd.4) 104 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.139970+0000 osd.4 (osd.4) 105 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.265635+0000 osd.7 (osd.7) 127 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:00:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.269102+0000 osd.7 (osd.7) 128 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:00:54.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:52.565838+0000 mgr.y (mgr.24481) 428 : cluster [DBG] pgmap v558: 138 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 134 active+clean; 77 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 183 op/s; 103/4260 objects misplaced (2.418%); 544 KiB/s, 8 objects/s recovering 2024-02-12T17:00:54.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:52.836886+0000 osd.2 (osd.2) 145 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:52.838845+0000 osd.2 (osd.2) 146 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.035780+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.035873+0000 osd.7 (osd.7) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.137985+0000 osd.4 (osd.4) 104 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.139970+0000 osd.4 (osd.4) 105 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.265635+0000 osd.7 (osd.7) 127 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.269102+0000 osd.7 (osd.7) 128 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:52.565838+0000 mgr.y (mgr.24481) 428 : cluster [DBG] pgmap v558: 138 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 134 active+clean; 77 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 183 op/s; 103/4260 objects misplaced (2.418%); 544 KiB/s, 8 objects/s recovering 2024-02-12T17:00:54.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:52.836886+0000 osd.2 (osd.2) 145 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:00:54.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:52.838845+0000 osd.2 (osd.2) 146 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:00:54.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.035780+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:00:54.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.035873+0000 osd.7 (osd.7) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:00:54.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.137985+0000 osd.4 (osd.4) 104 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:00:54.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.139970+0000 osd.4 (osd.4) 105 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:00:54.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.265635+0000 osd.7 (osd.7) 127 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:00:54.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.269102+0000 osd.7 (osd.7) 128 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:00:54.501 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:54.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:54.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:54.644 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:54.665 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:54.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:54.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:54.800 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:54.817 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:00:54.821 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:00:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.790826+0000 osd.2 (osd.2) 147 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:00:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.792974+0000 osd.2 (osd.2) 148 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:00:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.812724+0000 mon.a (mon.0) 1223 : cluster [DBG] osdmap e300: 8 total, 8 up, 5 in 2024-02-12T17:00:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.934437+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:00:54.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:53.936656+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:00:54.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:54.046666+0000 mgr.y (mgr.24481) 429 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:54.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:54.185515+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.1e scrub starts 2024-02-12T17:00:54.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:54.200640+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.1e scrub ok 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.790826+0000 osd.2 (osd.2) 147 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.792974+0000 osd.2 (osd.2) 148 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.812724+0000 mon.a (mon.0) 1223 : cluster [DBG] osdmap e300: 8 total, 8 up, 5 in 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.934437+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:53.936656+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:54.046666+0000 mgr.y (mgr.24481) 429 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:54.185515+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.1e scrub starts 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:54.200640+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.1e scrub ok 2024-02-12T17:00:55.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.790826+0000 osd.2 (osd.2) 147 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:00:55.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.792974+0000 osd.2 (osd.2) 148 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:00:55.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.812724+0000 mon.a (mon.0) 1223 : cluster [DBG] osdmap e300: 8 total, 8 up, 5 in 2024-02-12T17:00:55.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.934437+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:00:55.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:53.936656+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:00:55.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:54.046666+0000 mgr.y (mgr.24481) 429 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:00:55.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:54.185515+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.1e scrub starts 2024-02-12T17:00:55.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:54.200640+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.1e scrub ok 2024-02-12T17:00:55.252 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:55.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:55.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:55.385 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:55.509 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:55.512 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:55.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:55.514 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:55.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:55.642 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:56.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:54.566199+0000 mgr.y (mgr.24481) 430 : cluster [DBG] pgmap v560: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 257 op/s; 147 KiB/s, 2 objects/s recovering 2024-02-12T17:00:56.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:54.792787+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.c deep-scrub starts 2024-02-12T17:00:56.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:54.794220+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.c deep-scrub ok 2024-02-12T17:00:56.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:54.797133+0000 mgr.y (mgr.24481) 431 : audit [DBG] from='client.15384 v1:172.21.15.33:0/1027076691' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:56.078 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:54.938746+0000 osd.0 (osd.0) 127 : cluster [DBG] 3.1 deep-scrub starts 2024-02-12T17:00:56.079 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:54.941768+0000 osd.0 (osd.0) 128 : cluster [DBG] 3.1 deep-scrub ok 2024-02-12T17:00:56.079 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:54.566199+0000 mgr.y (mgr.24481) 430 : cluster [DBG] pgmap v560: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 257 op/s; 147 KiB/s, 2 objects/s recovering 2024-02-12T17:00:56.079 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:54.792787+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.c deep-scrub starts 2024-02-12T17:00:56.079 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:54.794220+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.c deep-scrub ok 2024-02-12T17:00:56.079 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:54.797133+0000 mgr.y (mgr.24481) 431 : audit [DBG] from='client.15384 v1:172.21.15.33:0/1027076691' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:56.079 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:54.938746+0000 osd.0 (osd.0) 127 : cluster [DBG] 3.1 deep-scrub starts 2024-02-12T17:00:56.079 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:54.941768+0000 osd.0 (osd.0) 128 : cluster [DBG] 3.1 deep-scrub ok 2024-02-12T17:00:56.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:54.566199+0000 mgr.y (mgr.24481) 430 : cluster [DBG] pgmap v560: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 257 op/s; 147 KiB/s, 2 objects/s recovering 2024-02-12T17:00:56.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:54.792787+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.c deep-scrub starts 2024-02-12T17:00:56.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:54.794220+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.c deep-scrub ok 2024-02-12T17:00:56.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:54.797133+0000 mgr.y (mgr.24481) 431 : audit [DBG] from='client.15384 v1:172.21.15.33:0/1027076691' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:00:56.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:54.938746+0000 osd.0 (osd.0) 127 : cluster [DBG] 3.1 deep-scrub starts 2024-02-12T17:00:56.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:54.941768+0000 osd.0 (osd.0) 128 : cluster [DBG] 3.1 deep-scrub ok 2024-02-12T17:00:56.393 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:56.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:56.409 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:00:56.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:56.530 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:56.551 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:56.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:56.655 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:00:56.686 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:57.295 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":560,"stamp":"2024-02-12T17:00:54.566003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":1322,"num_read_kb":1074,"num_write":224026,"num_write_kb":7145962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40196,"num_bytes_recovered":2616074351,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58780,"ondisk_log_size":58780,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":1535012,"kb_used_data":7932,"kb_used_omap":0,"kb_used_meta":1527040,"kb_avail":467190748,"statfs":{"total":479975178240,"available":478403325952,"internally_reserved":0,"allocated":8122368,"data_stored":4338848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1563688960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,16],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-80412696,"num_objects":-1228,"num_object_clones":0,"num_object_copies":-3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-103,"num_objects_unfound":0,"num_objects_dirty":-1228,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1228,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.783218"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":196,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808020+0000","last_change":"2024-02-12T16:59:09.246682+0000","last_active":"2024-02-12T17:00:50.808020+0000","last_peered":"2024-02-12T17:00:50.808020+0000","last_clean":"2024-02-12T17:00:50.808020+0000","last_became_active":"2024-02-12T16:59:09.246207+0000","last_became_peered":"2024-02-12T16:59:09.246207+0000","last_unstale":"2024-02-12T17:00:50.808020+0000","last_undegraded":"2024-02-12T17:00:50.808020+0000","last_fullsized":"2024-02-12T17:00:50.808020+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.965924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.965924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":263,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808943+0000","last_change":"2024-02-12T16:56:35.853405+0000","last_active":"2024-02-12T17:00:50.808943+0000","last_peered":"2024-02-12T17:00:50.808943+0000","last_clean":"2024-02-12T17:00:50.808943+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T17:00:50.808943+0000","last_undegraded":"2024-02-12T17:00:50.808943+0000","last_fullsized":"2024-02-12T17:00:50.808943+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.853330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.853330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":249,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805261+0000","last_change":"2024-02-12T16:56:21.699700+0000","last_active":"2024-02-12T17:00:50.805261+0000","last_peered":"2024-02-12T17:00:50.805261+0000","last_clean":"2024-02-12T17:00:50.805261+0000","last_became_active":"2024-02-12T16:54:40.633180+0000","last_became_peered":"2024-02-12T16:54:40.633180+0000","last_unstale":"2024-02-12T17:00:50.805261+0000","last_undegraded":"2024-02-12T17:00:50.805261+0000","last_fullsized":"2024-02-12T17:00:50.805261+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:56:21.699625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:21.699625+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":227,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805916+0000","last_change":"2024-02-12T16:57:47.192069+0000","last_active":"2024-02-12T17:00:50.805916+0000","last_peered":"2024-02-12T17:00:50.805916+0000","last_clean":"2024-02-12T17:00:50.805916+0000","last_became_active":"2024-02-12T16:57:47.188886+0000","last_became_peered":"2024-02-12T16:57:47.188886+0000","last_unstale":"2024-02-12T17:00:50.805916+0000","last_undegraded":"2024-02-12T17:00:50.805916+0000","last_fullsized":"2024-02-12T17:00:50.805916+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:44.946088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:44.946088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":178,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809617+0000","last_change":"2024-02-12T16:56:27.839848+0000","last_active":"2024-02-12T17:00:50.809617+0000","last_peered":"2024-02-12T17:00:50.809617+0000","last_clean":"2024-02-12T17:00:50.809617+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T17:00:50.809617+0000","last_undegraded":"2024-02-12T17:00:50.809617+0000","last_fullsized":"2024-02-12T17:00:50.809617+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.839765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.839765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"277'15","reported_seq":332,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.802380+0000","last_change":"2024-02-12T16:59:09.351570+0000","last_active":"2024-02-12T17:00:50.802380+0000","last_peered":"2024-02-12T17:00:50.802380+0000","last_clean":"2024-02-12T17:00:50.802380+0000","last_became_active":"2024-02-12T16:59:09.243960+0000","last_became_peered":"2024-02-12T16:59:09.243960+0000","last_unstale":"2024-02-12T17:00:50.802380+0000","last_undegraded":"2024-02-12T17:00:50.802380+0000","last_fullsized":"2024-02-12T17:00:50.802380+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:31.812644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.812644+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":277,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805292+0000","last_change":"2024-02-12T16:57:47.176812+0000","last_active":"2024-02-12T17:00:50.805292+0000","last_peered":"2024-02-12T17:00:50.805292+0000","last_clean":"2024-02-12T17:00:50.805292+0000","last_became_active":"2024-02-12T16:57:47.176448+0000","last_became_peered":"2024-02-12T16:57:47.176448+0000","last_unstale":"2024-02-12T17:00:50.805292+0000","last_undegraded":"2024-02-12T17:00:50.805292+0000","last_fullsized":"2024-02-12T17:00:50.805292+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:42.174992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:42.174992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":280,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806150+0000","last_change":"2024-02-12T16:59:09.248489+0000","last_active":"2024-02-12T17:00:50.806150+0000","last_peered":"2024-02-12T17:00:50.806150+0000","last_clean":"2024-02-12T17:00:50.806150+0000","last_became_active":"2024-02-12T16:59:09.248262+0000","last_became_peered":"2024-02-12T16:59:09.248262+0000","last_unstale":"2024-02-12T17:00:50.806150+0000","last_undegraded":"2024-02-12T17:00:50.806150+0000","last_fullsized":"2024-02-12T17:00:50.806150+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.784157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.784157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":137,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806635+0000","last_change":"2024-02-12T16:57:47.319602+0000","last_active":"2024-02-12T17:00:50.806635+0000","last_peered":"2024-02-12T17:00:50.806635+0000","last_clean":"2024-02-12T17:00:50.806635+0000","last_became_active":"2024-02-12T16:57:47.186061+0000","last_became_peered":"2024-02-12T16:57:47.186061+0000","last_unstale":"2024-02-12T17:00:50.806635+0000","last_undegraded":"2024-02-12T17:00:50.806635+0000","last_fullsized":"2024-02-12T17:00:50.806635+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":284,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.923599+0000","last_change":"2024-02-12T17:00:50.923564+0000","last_active":"2024-02-12T17:00:50.923599+0000","last_peered":"2024-02-12T17:00:50.923599+0000","last_clean":"2024-02-12T17:00:50.923599+0000","last_became_active":"2024-02-12T16:57:47.185903+0000","last_became_peered":"2024-02-12T16:57:47.185903+0000","last_unstale":"2024-02-12T17:00:50.923599+0000","last_undegraded":"2024-02-12T17:00:50.923599+0000","last_fullsized":"2024-02-12T17:00:50.923599+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:50.923462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.923462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":263,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806779+0000","last_change":"2024-02-12T16:56:35.897211+0000","last_active":"2024-02-12T17:00:50.806779+0000","last_peered":"2024-02-12T17:00:50.806779+0000","last_clean":"2024-02-12T17:00:50.806779+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:00:50.806779+0000","last_undegraded":"2024-02-12T17:00:50.806779+0000","last_fullsized":"2024-02-12T17:00:50.806779+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.897088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.897088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":225,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808286+0000","last_change":"2024-02-12T16:57:18.865914+0000","last_active":"2024-02-12T17:00:50.808286+0000","last_peered":"2024-02-12T17:00:50.808286+0000","last_clean":"2024-02-12T17:00:50.808286+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T17:00:50.808286+0000","last_undegraded":"2024-02-12T17:00:50.808286+0000","last_fullsized":"2024-02-12T17:00:50.808286+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:18.865807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.865807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":142,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806167+0000","last_change":"2024-02-12T16:59:09.260070+0000","last_active":"2024-02-12T17:00:50.806167+0000","last_peered":"2024-02-12T17:00:50.806167+0000","last_clean":"2024-02-12T17:00:50.806167+0000","last_became_active":"2024-02-12T16:59:09.259293+0000","last_became_peered":"2024-02-12T16:59:09.259293+0000","last_unstale":"2024-02-12T17:00:50.806167+0000","last_undegraded":"2024-02-12T17:00:50.806167+0000","last_fullsized":"2024-02-12T17:00:50.806167+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.345717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.345717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":187,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808166+0000","last_change":"2024-02-12T16:59:09.276561+0000","last_active":"2024-02-12T17:00:50.808166+0000","last_peered":"2024-02-12T17:00:50.808166+0000","last_clean":"2024-02-12T17:00:50.808166+0000","last_became_active":"2024-02-12T16:59:09.272423+0000","last_became_peered":"2024-02-12T16:59:09.272423+0000","last_unstale":"2024-02-12T17:00:50.808166+0000","last_undegraded":"2024-02-12T17:00:50.808166+0000","last_fullsized":"2024-02-12T17:00:50.808166+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":180,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.803393+0000","last_change":"2024-02-12T16:59:09.890505+0000","last_active":"2024-02-12T17:00:50.803393+0000","last_peered":"2024-02-12T17:00:50.803393+0000","last_clean":"2024-02-12T17:00:50.803393+0000","last_became_active":"2024-02-12T16:59:09.249268+0000","last_became_peered":"2024-02-12T16:59:09.249268+0000","last_unstale":"2024-02-12T17:00:50.803393+0000","last_undegraded":"2024-02-12T17:00:50.803393+0000","last_fullsized":"2024-02-12T17:00:50.803393+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:56:45.033441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.033441+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":273,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.803361+0000","last_change":"2024-02-12T16:59:09.270880+0000","last_active":"2024-02-12T17:00:50.803361+0000","last_peered":"2024-02-12T17:00:50.803361+0000","last_clean":"2024-02-12T17:00:50.803361+0000","last_became_active":"2024-02-12T16:59:09.270574+0000","last_became_peered":"2024-02-12T16:59:09.270574+0000","last_unstale":"2024-02-12T17:00:50.803361+0000","last_undegraded":"2024-02-12T17:00:50.803361+0000","last_fullsized":"2024-02-12T17:00:50.803361+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.464226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.464226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":53,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.802863+0000","last_change":"2024-02-12T16:59:09.248658+0000","last_active":"2024-02-12T17:00:50.802863+0000","last_peered":"2024-02-12T17:00:50.802863+0000","last_clean":"2024-02-12T17:00:50.802863+0000","last_became_active":"2024-02-12T16:59:09.248342+0000","last_became_peered":"2024-02-12T16:59:09.248342+0000","last_unstale":"2024-02-12T17:00:50.802863+0000","last_undegraded":"2024-02-12T17:00:50.802863+0000","last_fullsized":"2024-02-12T17:00:50.802863+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.339748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.339748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":365,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807273+0000","last_change":"2024-02-12T16:57:47.210170+0000","last_active":"2024-02-12T17:00:50.807273+0000","last_peered":"2024-02-12T17:00:50.807273+0000","last_clean":"2024-02-12T17:00:50.807273+0000","last_became_active":"2024-02-12T16:57:47.178386+0000","last_became_peered":"2024-02-12T16:57:47.178386+0000","last_unstale":"2024-02-12T17:00:50.807273+0000","last_undegraded":"2024-02-12T17:00:50.807273+0000","last_fullsized":"2024-02-12T17:00:50.807273+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:57:24.750245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.750245+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":103,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806348+0000","last_change":"2024-02-12T16:57:47.179029+0000","last_active":"2024-02-12T17:00:50.806348+0000","last_peered":"2024-02-12T17:00:50.806348+0000","last_clean":"2024-02-12T17:00:50.806348+0000","last_became_active":"2024-02-12T16:57:47.178240+0000","last_became_peered":"2024-02-12T16:57:47.178240+0000","last_unstale":"2024-02-12T17:00:50.806348+0000","last_undegraded":"2024-02-12T17:00:50.806348+0000","last_fullsized":"2024-02-12T17:00:50.806348+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:31.122557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.122557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":56,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809115+0000","last_change":"2024-02-12T16:59:09.242061+0000","last_active":"2024-02-12T17:00:50.809115+0000","last_peered":"2024-02-12T17:00:50.809115+0000","last_clean":"2024-02-12T17:00:50.809115+0000","last_became_active":"2024-02-12T16:59:09.241719+0000","last_became_peered":"2024-02-12T16:59:09.241719+0000","last_unstale":"2024-02-12T17:00:50.809115+0000","last_undegraded":"2024-02-12T17:00:50.809115+0000","last_fullsized":"2024-02-12T17:00:50.809115+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.291990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.291990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":108,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806804+0000","last_change":"2024-02-12T16:59:09.533676+0000","last_active":"2024-02-12T17:00:50.806804+0000","last_peered":"2024-02-12T17:00:50.806804+0000","last_clean":"2024-02-12T17:00:50.806804+0000","last_became_active":"2024-02-12T16:59:09.259731+0000","last_became_peered":"2024-02-12T16:59:09.259731+0000","last_unstale":"2024-02-12T17:00:50.806804+0000","last_undegraded":"2024-02-12T17:00:50.806804+0000","last_fullsized":"2024-02-12T17:00:50.806804+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:27.293681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.293681+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":259,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808152+0000","last_change":"2024-02-12T16:57:37.670804+0000","last_active":"2024-02-12T17:00:50.808152+0000","last_peered":"2024-02-12T17:00:50.808152+0000","last_clean":"2024-02-12T17:00:50.808152+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T17:00:50.808152+0000","last_undegraded":"2024-02-12T17:00:50.808152+0000","last_fullsized":"2024-02-12T17:00:50.808152+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:57:37.670713+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.670713+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":265,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.804638+0000","last_change":"2024-02-12T16:57:26.790389+0000","last_active":"2024-02-12T17:00:50.804638+0000","last_peered":"2024-02-12T17:00:50.804638+0000","last_clean":"2024-02-12T17:00:50.804638+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T17:00:50.804638+0000","last_undegraded":"2024-02-12T17:00:50.804638+0000","last_fullsized":"2024-02-12T17:00:50.804638+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":333,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809705+0000","last_change":"2024-02-12T16:57:47.209446+0000","last_active":"2024-02-12T17:00:50.809705+0000","last_peered":"2024-02-12T17:00:50.809705+0000","last_clean":"2024-02-12T17:00:50.809705+0000","last_became_active":"2024-02-12T16:57:47.189652+0000","last_became_peered":"2024-02-12T16:57:47.189652+0000","last_unstale":"2024-02-12T17:00:50.809705+0000","last_undegraded":"2024-02-12T17:00:50.809705+0000","last_fullsized":"2024-02-12T17:00:50.809705+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:56:32.825183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.825183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":187,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808411+0000","last_change":"2024-02-12T16:59:09.264851+0000","last_active":"2024-02-12T17:00:50.808411+0000","last_peered":"2024-02-12T17:00:50.808411+0000","last_clean":"2024-02-12T17:00:50.808411+0000","last_became_active":"2024-02-12T16:59:09.264513+0000","last_became_peered":"2024-02-12T16:59:09.264513+0000","last_unstale":"2024-02-12T17:00:50.808411+0000","last_undegraded":"2024-02-12T17:00:50.808411+0000","last_fullsized":"2024-02-12T17:00:50.808411+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":268,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.269702+0000","last_change":"2024-02-12T17:00:51.269702+0000","last_active":"2024-02-12T17:00:51.269702+0000","last_peered":"2024-02-12T17:00:51.269702+0000","last_clean":"2024-02-12T17:00:51.269702+0000","last_became_active":"2024-02-12T16:59:09.270516+0000","last_became_peered":"2024-02-12T16:59:09.270516+0000","last_unstale":"2024-02-12T17:00:51.269702+0000","last_undegraded":"2024-02-12T17:00:51.269702+0000","last_fullsized":"2024-02-12T17:00:51.269702+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:51.269648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.269648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":53,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806346+0000","last_change":"2024-02-12T16:59:09.260879+0000","last_active":"2024-02-12T17:00:50.806346+0000","last_peered":"2024-02-12T17:00:50.806346+0000","last_clean":"2024-02-12T17:00:50.806346+0000","last_became_active":"2024-02-12T16:59:09.259058+0000","last_became_peered":"2024-02-12T16:59:09.259058+0000","last_unstale":"2024-02-12T17:00:50.806346+0000","last_undegraded":"2024-02-12T17:00:50.806346+0000","last_fullsized":"2024-02-12T17:00:50.806346+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":254,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806631+0000","last_change":"2024-02-12T16:59:10.061885+0000","last_active":"2024-02-12T17:00:50.806631+0000","last_peered":"2024-02-12T17:00:50.806631+0000","last_clean":"2024-02-12T17:00:50.806631+0000","last_became_active":"2024-02-12T16:59:09.267628+0000","last_became_peered":"2024-02-12T16:59:09.267628+0000","last_unstale":"2024-02-12T17:00:50.806631+0000","last_undegraded":"2024-02-12T17:00:50.806631+0000","last_fullsized":"2024-02-12T17:00:50.806631+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:56:24.854536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:24.854536+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":257,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807926+0000","last_change":"2024-02-12T16:57:28.846844+0000","last_active":"2024-02-12T17:00:50.807926+0000","last_peered":"2024-02-12T17:00:50.807926+0000","last_clean":"2024-02-12T17:00:50.807926+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T17:00:50.807926+0000","last_undegraded":"2024-02-12T17:00:50.807926+0000","last_fullsized":"2024-02-12T17:00:50.807926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":53,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.802474+0000","last_change":"2024-02-12T16:59:09.247624+0000","last_active":"2024-02-12T17:00:50.802474+0000","last_peered":"2024-02-12T17:00:50.802474+0000","last_clean":"2024-02-12T17:00:50.802474+0000","last_became_active":"2024-02-12T16:59:09.247346+0000","last_became_peered":"2024-02-12T16:59:09.247346+0000","last_unstale":"2024-02-12T17:00:50.802474+0000","last_undegraded":"2024-02-12T17:00:50.802474+0000","last_fullsized":"2024-02-12T17:00:50.802474+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.225996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":274,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805690+0000","last_change":"2024-02-12T16:59:09.265096+0000","last_active":"2024-02-12T17:00:50.805690+0000","last_peered":"2024-02-12T17:00:50.805690+0000","last_clean":"2024-02-12T17:00:50.805690+0000","last_became_active":"2024-02-12T16:59:09.264558+0000","last_became_peered":"2024-02-12T16:59:09.264558+0000","last_unstale":"2024-02-12T17:00:50.805690+0000","last_undegraded":"2024-02-12T17:00:50.805690+0000","last_fullsized":"2024-02-12T17:00:50.805690+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"277'8","reported_seq":86,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.804361+0000","last_change":"2024-02-12T16:59:09.961214+0000","last_active":"2024-02-12T17:00:50.804361+0000","last_peered":"2024-02-12T17:00:50.804361+0000","last_clean":"2024-02-12T17:00:50.804361+0000","last_became_active":"2024-02-12T16:59:09.280196+0000","last_became_peered":"2024-02-12T16:59:09.280196+0000","last_unstale":"2024-02-12T17:00:50.804361+0000","last_undegraded":"2024-02-12T17:00:50.804361+0000","last_fullsized":"2024-02-12T17:00:50.804361+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-12T16:54:01.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.670107+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":178,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809023+0000","last_change":"2024-02-12T16:56:29.870171+0000","last_active":"2024-02-12T17:00:50.809023+0000","last_peered":"2024-02-12T17:00:50.809023+0000","last_clean":"2024-02-12T17:00:50.809023+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T17:00:50.809023+0000","last_undegraded":"2024-02-12T17:00:50.809023+0000","last_fullsized":"2024-02-12T17:00:50.809023+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:29.870075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T16:56:29.870075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":212,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806450+0000","last_change":"2024-02-12T16:59:09.263555+0000","last_active":"2024-02-12T17:00:50.806450+0000","last_peered":"2024-02-12T17:00:50.806450+0000","last_clean":"2024-02-12T17:00:50.806450+0000","last_became_active":"2024-02-12T16:59:09.262674+0000","last_became_peered":"2024-02-12T16:59:09.262674+0000","last_unstale":"2024-02-12T17:00:50.806450+0000","last_undegraded":"2024-02-12T17:00:50.806450+0000","last_fullsized":"2024-02-12T17:00:50.806450+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":112,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805911+0000","last_change":"2024-02-12T16:59:09.258970+0000","last_active":"2024-02-12T17:00:50.805911+0000","last_peered":"2024-02-12T17:00:50.805911+0000","last_clean":"2024-02-12T17:00:50.805911+0000","last_became_active":"2024-02-12T16:59:09.257167+0000","last_became_peered":"2024-02-12T16:59:09.257167+0000","last_unstale":"2024-02-12T17:00:50.805911+0000","last_undegraded":"2024-02-12T17:00:50.805911+0000","last_fullsized":"2024-02-12T17:00:50.805911+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.773038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.773038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":344,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.803838+0000","last_change":"2024-02-12T16:59:09.950284+0000","last_active":"2024-02-12T17:00:50.803838+0000","last_peered":"2024-02-12T17:00:50.803838+0000","last_clean":"2024-02-12T17:00:50.803838+0000","last_became_active":"2024-02-12T16:59:09.274991+0000","last_became_peered":"2024-02-12T16:59:09.274991+0000","last_unstale":"2024-02-12T17:00:50.803838+0000","last_undegraded":"2024-02-12T17:00:50.803838+0000","last_fullsized":"2024-02-12T17:00:50.803838+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:37.895287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.895287+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":270,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807647+0000","last_change":"2024-02-12T16:59:09.269463+0000","last_active":"2024-02-12T17:00:50.807647+0000","last_peered":"2024-02-12T17:00:50.807647+0000","last_clean":"2024-02-12T17:00:50.807647+0000","last_became_active":"2024-02-12T16:59:09.269260+0000","last_became_peered":"2024-02-12T16:59:09.269260+0000","last_unstale":"2024-02-12T17:00:50.807647+0000","last_undegraded":"2024-02-12T17:00:50.807647+0000","last_fullsized":"2024-02-12T17:00:50.807647+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.193962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.193962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":203,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808333+0000","last_change":"2024-02-12T16:57:47.189653+0000","last_active":"2024-02-12T17:00:50.808333+0000","last_peered":"2024-02-12T17:00:50.808333+0000","last_clean":"2024-02-12T17:00:50.808333+0000","last_became_active":"2024-02-12T16:57:47.188909+0000","last_became_peered":"2024-02-12T16:57:47.188909+0000","last_unstale":"2024-02-12T17:00:50.808333+0000","last_undegraded":"2024-02-12T17:00:50.808333+0000","last_fullsized":"2024-02-12T17:00:50.808333+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:23.899135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:23.899135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":151,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806571+0000","last_change":"2024-02-12T16:59:09.260492+0000","last_active":"2024-02-12T17:00:50.806571+0000","last_peered":"2024-02-12T17:00:50.806571+0000","last_clean":"2024-02-12T17:00:50.806571+0000","last_became_active":"2024-02-12T16:59:09.260045+0000","last_became_peered":"2024-02-12T16:59:09.260045+0000","last_unstale":"2024-02-12T17:00:50.806571+0000","last_undegraded":"2024-02-12T17:00:50.806571+0000","last_fullsized":"2024-02-12T17:00:50.806571+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:39.248562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.248562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":118,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805697+0000","last_change":"2024-02-12T16:59:09.288225+0000","last_active":"2024-02-12T17:00:50.805697+0000","last_peered":"2024-02-12T17:00:50.805697+0000","last_clean":"2024-02-12T17:00:50.805697+0000","last_became_active":"2024-02-12T16:59:09.253231+0000","last_became_peered":"2024-02-12T16:59:09.253231+0000","last_unstale":"2024-02-12T17:00:50.805697+0000","last_undegraded":"2024-02-12T17:00:50.805697+0000","last_fullsized":"2024-02-12T17:00:50.805697+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'13","last_scrub_stamp":"2024-02-12T16:54:31.953995+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.953995+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":223,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808063+0000","last_change":"2024-02-12T16:56:27.696314+0000","last_active":"2024-02-12T17:00:50.808063+0000","last_peered":"2024-02-12T17:00:50.808063+0000","last_clean":"2024-02-12T17:00:50.808063+0000","last_became_active":"2024-02-12T16:52:20.565127+0000","last_became_peered":"2024-02-12T16:52:20.565127+0000","last_unstale":"2024-02-12T17:00:50.808063+0000","last_undegraded":"2024-02-12T17:00:50.808063+0000","last_fullsized":"2024-02-12T17:00:50.808063+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:27.696175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:27.696175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":310,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.224718+0000","last_change":"2024-02-12T16:57:47.312130+0000","last_active":"2024-02-12T17:00:51.224718+0000","last_peered":"2024-02-12T17:00:51.224718+0000","last_clean":"2024-02-12T17:00:51.224718+0000","last_became_active":"2024-02-12T16:57:47.185109+0000","last_became_peered":"2024-02-12T16:57:47.185109+0000","last_unstale":"2024-02-12T17:00:51.224718+0000","last_undegraded":"2024-02-12T17:00:51.224718+0000","last_fullsized":"2024-02-12T17:00:51.224718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9","last_scrub_stamp":"2024-02-12T16:56:20.829602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.829602+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":267,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808625+0000","last_change":"2024-02-12T16:54:40.629278+0000","last_active":"2024-02-12T17:00:50.808625+0000","last_peered":"2024-02-12T17:00:50.808625+0000","last_clean":"2024-02-12T17:00:50.808625+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T17:00:50.808625+0000","last_undegraded":"2024-02-12T17:00:50.808625+0000","last_fullsized":"2024-02-12T17:00:50.808625+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:35.191041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:35.191041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":321,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805816+0000","last_change":"2024-02-12T16:59:09.524334+0000","last_active":"2024-02-12T17:00:50.805816+0000","last_peered":"2024-02-12T17:00:50.805816+0000","last_clean":"2024-02-12T17:00:50.805816+0000","last_became_active":"2024-02-12T16:59:09.258515+0000","last_became_peered":"2024-02-12T16:59:09.258515+0000","last_unstale":"2024-02-12T17:00:50.805816+0000","last_undegraded":"2024-02-12T17:00:50.805816+0000","last_fullsized":"2024-02-12T17:00:50.805816+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:54:32.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.895402+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":266,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809487+0000","last_change":"2024-02-12T16:57:25.835979+0000","last_active":"2024-02-12T17:00:50.809487+0000","last_peered":"2024-02-12T17:00:50.809487+0000","last_clean":"2024-02-12T17:00:50.809487+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T17:00:50.809487+0000","last_undegraded":"2024-02-12T17:00:50.809487+0000","last_fullsized":"2024-02-12T17:00:50.809487+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.835883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.835883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"208'9","reported_seq":419,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.228376+0000","last_change":"2024-02-12T17:00:49.846525+0000","last_active":"2024-02-12T17:00:51.228376+0000","last_peered":"2024-02-12T17:00:51.228376+0000","last_clean":"2024-02-12T17:00:51.228376+0000","last_became_active":"2024-02-12T16:57:47.178688+0000","last_became_peered":"2024-02-12T16:57:47.178688+0000","last_unstale":"2024-02-12T17:00:51.228376+0000","last_undegraded":"2024-02-12T17:00:51.228376+0000","last_fullsized":"2024-02-12T17:00:51.228376+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'9","last_scrub_stamp":"2024-02-12T17:00:49.846432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.846432+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":269,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806101+0000","last_change":"2024-02-12T16:54:40.626943+0000","last_active":"2024-02-12T17:00:50.806101+0000","last_peered":"2024-02-12T17:00:50.806101+0000","last_clean":"2024-02-12T17:00:50.806101+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T17:00:50.806101+0000","last_undegraded":"2024-02-12T17:00:50.806101+0000","last_fullsized":"2024-02-12T17:00:50.806101+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.875602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.875602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":157,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806897+0000","last_change":"2024-02-12T16:57:48.058146+0000","last_active":"2024-02-12T17:00:50.806897+0000","last_peered":"2024-02-12T17:00:50.806897+0000","last_clean":"2024-02-12T17:00:50.806897+0000","last_became_active":"2024-02-12T16:57:47.199140+0000","last_became_peered":"2024-02-12T16:57:47.199140+0000","last_unstale":"2024-02-12T17:00:50.806897+0000","last_undegraded":"2024-02-12T17:00:50.806897+0000","last_fullsized":"2024-02-12T17:00:50.806897+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":265,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806167+0000","last_change":"2024-02-12T16:59:09.269750+0000","last_active":"2024-02-12T17:00:50.806167+0000","last_peered":"2024-02-12T17:00:50.806167+0000","last_clean":"2024-02-12T17:00:50.806167+0000","last_became_active":"2024-02-12T16:59:09.269414+0000","last_became_peered":"2024-02-12T16:59:09.269414+0000","last_unstale":"2024-02-12T17:00:50.806167+0000","last_undegraded":"2024-02-12T17:00:50.806167+0000","last_fullsized":"2024-02-12T17:00:50.806167+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:51.532562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:53:51.532562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":226,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.802545+0000","last_change":"2024-02-12T16:59:09.243989+0000","last_active":"2024-02-12T17:00:50.802545+0000","last_peered":"2024-02-12T17:00:50.802545+0000","last_clean":"2024-02-12T17:00:50.802545+0000","last_became_active":"2024-02-12T16:59:09.243603+0000","last_became_peered":"2024-02-12T16:59:09.243603+0000","last_unstale":"2024-02-12T17:00:50.802545+0000","last_undegraded":"2024-02-12T17:00:50.802545+0000","last_fullsized":"2024-02-12T17:00:50.802545+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.218031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.218031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":250,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:53.269198+0000","last_change":"2024-02-12T17:00:53.269198+0000","last_active":"2024-02-12T17:00:53.269198+0000","last_peered":"2024-02-12T17:00:53.269198+0000","last_clean":"2024-02-12T17:00:53.269198+0000","last_became_active":"2024-02-12T16:59:09.242413+0000","last_became_peered":"2024-02-12T16:59:09.242413+0000","last_unstale":"2024-02-12T17:00:53.269198+0000","last_undegraded":"2024-02-12T17:00:53.269198+0000","last_fullsized":"2024-02-12T17:00:53.269198+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-02-12T17:00:53.269107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.269107+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":259,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807063+0000","last_change":"2024-02-12T16:56:38.699688+0000","last_active":"2024-02-12T17:00:50.807063+0000","last_peered":"2024-02-12T17:00:50.807063+0000","last_clean":"2024-02-12T17:00:50.807063+0000","last_became_active":"2024-02-12T16:49:45.222331+0000","last_became_peered":"2024-02-12T16:49:45.222331+0000","last_unstale":"2024-02-12T17:00:50.807063+0000","last_undegraded":"2024-02-12T17:00:50.807063+0000","last_fullsized":"2024-02-12T17:00:50.807063+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T16:56:38.699602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":274,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.804988+0000","last_change":"2024-02-12T16:57:47.171068+0000","last_active":"2024-02-12T17:00:50.804988+0000","last_peered":"2024-02-12T17:00:50.804988+0000","last_clean":"2024-02-12T17:00:50.804988+0000","last_became_active":"2024-02-12T16:57:47.170667+0000","last_became_peered":"2024-02-12T16:57:47.170667+0000","last_unstale":"2024-02-12T17:00:50.804988+0000","last_undegraded":"2024-02-12T17:00:50.804988+0000","last_fullsized":"2024-02-12T17:00:50.804988+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:22.699332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:22.699332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":304,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809606+0000","last_change":"2024-02-12T16:54:40.684514+0000","last_active":"2024-02-12T17:00:50.809606+0000","last_peered":"2024-02-12T17:00:50.809606+0000","last_clean":"2024-02-12T17:00:50.809606+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T17:00:50.809606+0000","last_undegraded":"2024-02-12T17:00:50.809606+0000","last_fullsized":"2024-02-12T17:00:50.809606+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-02-12T16:54:02.181108+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T16:54:02.181108+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"299'13940","reported_seq":16186,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.276821+0000","last_change":"2024-02-12T17:00:50.849925+0000","last_active":"2024-02-12T17:00:51.276821+0000","last_peered":"2024-02-12T17:00:51.276821+0000","last_clean":"2024-02-12T17:00:51.276821+0000","last_became_active":"2024-02-12T17:00:50.809910+0000","last_became_peered":"2024-02-12T17:00:50.809910+0000","last_unstale":"2024-02-12T17:00:51.276821+0000","last_undegraded":"2024-02-12T17:00:51.276821+0000","last_fullsized":"2024-02-12T17:00:51.276821+0000","mapping_epoch":297,"log_start":"269'10222","ondisk_log_start":"269'10222","created":176,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3718,"ondisk_log_size":3718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13940,"num_write_kb":446080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1047,"num_bytes_recovered":68616192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":191,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808119+0000","last_change":"2024-02-12T16:59:09.273017+0000","last_active":"2024-02-12T17:00:50.808119+0000","last_peered":"2024-02-12T17:00:50.808119+0000","last_clean":"2024-02-12T17:00:50.808119+0000","last_became_active":"2024-02-12T16:59:09.272706+0000","last_became_peered":"2024-02-12T16:59:09.272706+0000","last_unstale":"2024-02-12T17:00:50.808119+0000","last_undegraded":"2024-02-12T17:00:50.808119+0000","last_fullsized":"2024-02-12T17:00:50.808119+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.715652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.715652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":290,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.803436+0000","last_change":"2024-02-12T16:57:22.679795+0000","last_active":"2024-02-12T17:00:50.803436+0000","last_peered":"2024-02-12T17:00:50.803436+0000","last_clean":"2024-02-12T17:00:50.803436+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T17:00:50.803436+0000","last_undegraded":"2024-02-12T17:00:50.803436+0000","last_fullsized":"2024-02-12T17:00:50.803436+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T16:57:22.679705+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.679705+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":169,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808010+0000","last_change":"2024-02-12T16:59:09.952265+0000","last_active":"2024-02-12T17:00:50.808010+0000","last_peered":"2024-02-12T17:00:50.808010+0000","last_clean":"2024-02-12T17:00:50.808010+0000","last_became_active":"2024-02-12T16:59:09.264482+0000","last_became_peered":"2024-02-12T16:59:09.264482+0000","last_unstale":"2024-02-12T17:00:50.808010+0000","last_undegraded":"2024-02-12T17:00:50.808010+0000","last_fullsized":"2024-02-12T17:00:50.808010+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:22.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.764984+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":288,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805065+0000","last_change":"2024-02-12T16:58:10.280441+0000","last_active":"2024-02-12T17:00:50.805065+0000","last_peered":"2024-02-12T17:00:50.805065+0000","last_clean":"2024-02-12T17:00:50.805065+0000","last_became_active":"2024-02-12T16:58:10.280137+0000","last_became_peered":"2024-02-12T16:58:10.280137+0000","last_unstale":"2024-02-12T17:00:50.805065+0000","last_undegraded":"2024-02-12T17:00:50.805065+0000","last_fullsized":"2024-02-12T17:00:50.805065+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.647627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.647627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"299'13934","reported_seq":14065,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.262761+0000","last_change":"2024-02-12T16:56:36.446431+0000","last_active":"2024-02-12T17:00:51.262761+0000","last_peered":"2024-02-12T17:00:51.262761+0000","last_clean":"2024-02-12T17:00:51.262761+0000","last_became_active":"2024-02-12T16:56:36.446107+0000","last_became_peered":"2024-02-12T16:56:36.446107+0000","last_unstale":"2024-02-12T17:00:51.262761+0000","last_undegraded":"2024-02-12T17:00:51.262761+0000","last_fullsized":"2024-02-12T17:00:51.262761+0000","mapping_epoch":176,"log_start":"269'10326","ondisk_log_start":"269'10326","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3608,"ondisk_log_size":3608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13934,"num_write_kb":445888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":266,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807677+0000","last_change":"2024-02-12T16:57:29.934046+0000","last_active":"2024-02-12T17:00:50.807677+0000","last_peered":"2024-02-12T17:00:50.807677+0000","last_clean":"2024-02-12T17:00:50.807677+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T17:00:50.807677+0000","last_undegraded":"2024-02-12T17:00:50.807677+0000","last_fullsized":"2024-02-12T17:00:50.807677+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":103,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808896+0000","last_change":"2024-02-12T16:57:47.187057+0000","last_active":"2024-02-12T17:00:50.808896+0000","last_peered":"2024-02-12T17:00:50.808896+0000","last_clean":"2024-02-12T17:00:50.808896+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:00:50.808896+0000","last_undegraded":"2024-02-12T17:00:50.808896+0000","last_fullsized":"2024-02-12T17:00:50.808896+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:37.009094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:37.009094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":157,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806554+0000","last_change":"2024-02-12T16:57:48.006795+0000","last_active":"2024-02-12T17:00:50.806554+0000","last_peered":"2024-02-12T17:00:50.806554+0000","last_clean":"2024-02-12T17:00:50.806554+0000","last_became_active":"2024-02-12T16:57:47.202531+0000","last_became_peered":"2024-02-12T16:57:47.202531+0000","last_unstale":"2024-02-12T17:00:50.806554+0000","last_undegraded":"2024-02-12T17:00:50.806554+0000","last_fullsized":"2024-02-12T17:00:50.806554+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"299'10","reported_seq":416,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.955813+0000","last_change":"2024-02-12T17:00:51.955785+0000","last_active":"2024-02-12T17:00:51.955813+0000","last_peered":"2024-02-12T17:00:51.955813+0000","last_clean":"2024-02-12T17:00:51.955813+0000","last_became_active":"2024-02-12T16:59:09.270667+0000","last_became_peered":"2024-02-12T16:59:09.270667+0000","last_unstale":"2024-02-12T17:00:51.955813+0000","last_undegraded":"2024-02-12T17:00:51.955813+0000","last_fullsized":"2024-02-12T17:00:51.955813+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"299'14136","reported_seq":14267,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.295615+0000","last_change":"2024-02-12T16:56:36.788519+0000","last_active":"2024-02-12T17:00:51.295615+0000","last_peered":"2024-02-12T17:00:51.295615+0000","last_clean":"2024-02-12T17:00:51.295615+0000","last_became_active":"2024-02-12T16:56:36.788101+0000","last_became_peered":"2024-02-12T16:56:36.788101+0000","last_unstale":"2024-02-12T17:00:51.295615+0000","last_undegraded":"2024-02-12T17:00:51.295615+0000","last_fullsized":"2024-02-12T17:00:51.295615+0000","mapping_epoch":176,"log_start":"271'10569","ondisk_log_start":"271'10569","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3567,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14136,"num_write_kb":452352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":191,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.803716+0000","last_change":"2024-02-12T16:59:09.265012+0000","last_active":"2024-02-12T17:00:50.803716+0000","last_peered":"2024-02-12T17:00:50.803716+0000","last_clean":"2024-02-12T17:00:50.803716+0000","last_became_active":"2024-02-12T16:59:09.264617+0000","last_became_peered":"2024-02-12T16:59:09.264617+0000","last_unstale":"2024-02-12T17:00:50.803716+0000","last_undegraded":"2024-02-12T17:00:50.803716+0000","last_fullsized":"2024-02-12T17:00:50.803716+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.653301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.653301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":283,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806218+0000","last_change":"2024-02-12T16:59:09.263209+0000","last_active":"2024-02-12T17:00:50.806218+0000","last_peered":"2024-02-12T17:00:50.806218+0000","last_clean":"2024-02-12T17:00:50.806218+0000","last_became_active":"2024-02-12T16:59:09.261764+0000","last_became_peered":"2024-02-12T16:59:09.261764+0000","last_unstale":"2024-02-12T17:00:50.806218+0000","last_undegraded":"2024-02-12T17:00:50.806218+0000","last_fullsized":"2024-02-12T17:00:50.806218+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:25.227586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:25.227586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"278'53","reported_seq":517,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.219774+0000","last_change":"2024-02-12T16:59:09.306589+0000","last_active":"2024-02-12T17:00:51.219774+0000","last_peered":"2024-02-12T17:00:51.219774+0000","last_clean":"2024-02-12T17:00:51.219774+0000","last_became_active":"2024-02-12T16:59:09.253257+0000","last_became_peered":"2024-02-12T16:59:09.253257+0000","last_unstale":"2024-02-12T17:00:51.219774+0000","last_undegraded":"2024-02-12T17:00:51.219774+0000","last_fullsized":"2024-02-12T17:00:51.219774+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'39","last_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T16:56:43.083656+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":277,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809395+0000","last_change":"2024-02-12T16:59:09.264990+0000","last_active":"2024-02-12T17:00:50.809395+0000","last_peered":"2024-02-12T17:00:50.809395+0000","last_clean":"2024-02-12T17:00:50.809395+0000","last_became_active":"2024-02-12T16:59:09.264694+0000","last_became_peered":"2024-02-12T16:59:09.264694+0000","last_unstale":"2024-02-12T17:00:50.809395+0000","last_undegraded":"2024-02-12T17:00:50.809395+0000","last_fullsized":"2024-02-12T17:00:50.809395+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:46.719706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:46.719706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"299'13764","reported_seq":18017,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.255010+0000","last_change":"2024-02-12T17:00:46.745404+0000","last_active":"2024-02-12T17:00:51.255010+0000","last_peered":"2024-02-12T17:00:51.255010+0000","last_clean":"2024-02-12T17:00:51.255010+0000","last_became_active":"2024-02-12T17:00:46.689965+0000","last_became_peered":"2024-02-12T17:00:46.689965+0000","last_unstale":"2024-02-12T17:00:51.255010+0000","last_undegraded":"2024-02-12T17:00:51.255010+0000","last_fullsized":"2024-02-12T17:00:51.255010+0000","mapping_epoch":293,"log_start":"267'9917","ondisk_log_start":"267'9917","created":176,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3847,"ondisk_log_size":3847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13764,"num_write_kb":440448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":134152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":272,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806403+0000","last_change":"2024-02-12T16:59:09.269787+0000","last_active":"2024-02-12T17:00:50.806403+0000","last_peered":"2024-02-12T17:00:50.806403+0000","last_clean":"2024-02-12T17:00:50.806403+0000","last_became_active":"2024-02-12T16:59:09.269488+0000","last_became_peered":"2024-02-12T16:59:09.269488+0000","last_unstale":"2024-02-12T17:00:50.806403+0000","last_undegraded":"2024-02-12T17:00:50.806403+0000","last_fullsized":"2024-02-12T17:00:50.806403+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.551037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.551037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":333,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:52.838944+0000","last_change":"2024-02-12T17:00:52.838944+0000","last_active":"2024-02-12T17:00:52.838944+0000","last_peered":"2024-02-12T17:00:52.838944+0000","last_clean":"2024-02-12T17:00:52.838944+0000","last_became_active":"2024-02-12T16:59:09.273056+0000","last_became_peered":"2024-02-12T16:59:09.273056+0000","last_unstale":"2024-02-12T17:00:52.838944+0000","last_undegraded":"2024-02-12T17:00:52.838944+0000","last_fullsized":"2024-02-12T17:00:52.838944+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:00:52.838852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:52.838852+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":190,"num_read_kb":185,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":292,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805438+0000","last_change":"2024-02-12T16:59:09.248652+0000","last_active":"2024-02-12T17:00:50.805438+0000","last_peered":"2024-02-12T17:00:50.805438+0000","last_clean":"2024-02-12T17:00:50.805438+0000","last_became_active":"2024-02-12T16:59:09.248320+0000","last_became_peered":"2024-02-12T16:59:09.248320+0000","last_unstale":"2024-02-12T17:00:50.805438+0000","last_undegraded":"2024-02-12T17:00:50.805438+0000","last_fullsized":"2024-02-12T17:00:50.805438+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:20.676629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:20.676629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":184,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806443+0000","last_change":"2024-02-12T16:57:47.220507+0000","last_active":"2024-02-12T17:00:50.806443+0000","last_peered":"2024-02-12T17:00:50.806443+0000","last_clean":"2024-02-12T17:00:50.806443+0000","last_became_active":"2024-02-12T16:57:47.190968+0000","last_became_peered":"2024-02-12T16:57:47.190968+0000","last_unstale":"2024-02-12T17:00:50.806443+0000","last_undegraded":"2024-02-12T17:00:50.806443+0000","last_fullsized":"2024-02-12T17:00:50.806443+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-12T16:56:28.195990+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T16:56:28.195990+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":279,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808540+0000","last_change":"2024-02-12T16:58:18.266322+0000","last_active":"2024-02-12T17:00:50.808540+0000","last_peered":"2024-02-12T17:00:50.808540+0000","last_clean":"2024-02-12T17:00:50.808540+0000","last_became_active":"2024-02-12T16:58:18.265999+0000","last_became_peered":"2024-02-12T16:58:18.265999+0000","last_unstale":"2024-02-12T17:00:50.808540+0000","last_undegraded":"2024-02-12T17:00:50.808540+0000","last_fullsized":"2024-02-12T17:00:50.808540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"299'13840","reported_seq":13994,"reported_epoch":299,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:00:51.259074+0000","last_change":"2024-02-12T17:00:41.208813+0000","last_active":"2024-02-12T17:00:51.259074+0000","last_peered":"2024-02-12T17:00:51.259074+0000","last_clean":"2024-02-12T16:59:08.180231+0000","last_became_active":"2024-02-12T16:59:10.299589+0000","last_became_peered":"2024-02-12T16:59:10.299589+0000","last_unstale":"2024-02-12T17:00:51.259074+0000","last_undegraded":"2024-02-12T17:00:51.259074+0000","last_fullsized":"2024-02-12T17:00:51.259074+0000","mapping_epoch":258,"log_start":"271'10337","ondisk_log_start":"271'10337","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3503,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13840,"num_write_kb":442880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,6],"avail_no_missing":["5","0","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":272,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806977+0000","last_change":"2024-02-12T16:57:47.191748+0000","last_active":"2024-02-12T17:00:50.806977+0000","last_peered":"2024-02-12T17:00:50.806977+0000","last_clean":"2024-02-12T17:00:50.806977+0000","last_became_active":"2024-02-12T16:57:47.191375+0000","last_became_peered":"2024-02-12T16:57:47.191375+0000","last_unstale":"2024-02-12T17:00:50.806977+0000","last_undegraded":"2024-02-12T17:00:50.806977+0000","last_fullsized":"2024-02-12T17:00:50.806977+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:53:42.946795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T16:53:42.946795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":187,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806367+0000","last_change":"2024-02-12T16:59:09.263388+0000","last_active":"2024-02-12T17:00:50.806367+0000","last_peered":"2024-02-12T17:00:50.806367+0000","last_clean":"2024-02-12T17:00:50.806367+0000","last_became_active":"2024-02-12T16:59:09.262266+0000","last_became_peered":"2024-02-12T16:59:09.262266+0000","last_unstale":"2024-02-12T17:00:50.806367+0000","last_undegraded":"2024-02-12T17:00:50.806367+0000","last_fullsized":"2024-02-12T17:00:50.806367+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":265,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805814+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:00:50.805814+0000","last_peered":"2024-02-12T17:00:50.805814+0000","last_clean":"2024-02-12T17:00:50.805814+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:00:50.805814+0000","last_undegraded":"2024-02-12T17:00:50.805814+0000","last_fullsized":"2024-02-12T17:00:50.805814+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":275,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807531+0000","last_change":"2024-02-12T16:59:09.245920+0000","last_active":"2024-02-12T17:00:50.807531+0000","last_peered":"2024-02-12T17:00:50.807531+0000","last_clean":"2024-02-12T17:00:50.807531+0000","last_became_active":"2024-02-12T16:59:09.245635+0000","last_became_peered":"2024-02-12T16:59:09.245635+0000","last_unstale":"2024-02-12T17:00:50.807531+0000","last_undegraded":"2024-02-12T17:00:50.807531+0000","last_fullsized":"2024-02-12T17:00:50.807531+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:30.769959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T16:54:30.769959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":121,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.803143+0000","last_change":"2024-02-12T16:59:10.440537+0000","last_active":"2024-02-12T17:00:50.803143+0000","last_peered":"2024-02-12T17:00:50.803143+0000","last_clean":"2024-02-12T17:00:50.803143+0000","last_became_active":"2024-02-12T16:59:09.249903+0000","last_became_peered":"2024-02-12T16:59:09.249903+0000","last_unstale":"2024-02-12T17:00:50.803143+0000","last_undegraded":"2024-02-12T17:00:50.803143+0000","last_fullsized":"2024-02-12T17:00:50.803143+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:19.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.280894+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":259,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807963+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T17:00:50.807963+0000","last_peered":"2024-02-12T17:00:50.807963+0000","last_clean":"2024-02-12T17:00:50.807963+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:00:50.807963+0000","last_undegraded":"2024-02-12T17:00:50.807963+0000","last_fullsized":"2024-02-12T17:00:50.807963+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"299'13978","reported_seq":26493,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.278607+0000","last_change":"2024-02-12T16:58:49.227803+0000","last_active":"2024-02-12T17:00:51.278607+0000","last_peered":"2024-02-12T17:00:51.278607+0000","last_clean":"2024-02-12T17:00:51.278607+0000","last_became_active":"2024-02-12T16:58:49.217536+0000","last_became_peered":"2024-02-12T16:58:49.217536+0000","last_unstale":"2024-02-12T17:00:51.278607+0000","last_undegraded":"2024-02-12T17:00:51.278607+0000","last_fullsized":"2024-02-12T17:00:51.278607+0000","mapping_epoch":249,"log_start":"271'10337","ondisk_log_start":"271'10337","created":176,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3641,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13978,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6182,"num_bytes_recovered":405143552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":205,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805298+0000","last_change":"2024-02-12T16:59:09.252727+0000","last_active":"2024-02-12T17:00:50.805298+0000","last_peered":"2024-02-12T17:00:50.805298+0000","last_clean":"2024-02-12T17:00:50.805298+0000","last_became_active":"2024-02-12T16:59:09.252442+0000","last_became_peered":"2024-02-12T16:59:09.252442+0000","last_unstale":"2024-02-12T17:00:50.805298+0000","last_undegraded":"2024-02-12T17:00:50.805298+0000","last_fullsized":"2024-02-12T17:00:50.805298+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":267,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809381+0000","last_change":"2024-02-12T16:54:40.634249+0000","last_active":"2024-02-12T17:00:50.809381+0000","last_peered":"2024-02-12T17:00:50.809381+0000","last_clean":"2024-02-12T17:00:50.809381+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T17:00:50.809381+0000","last_undegraded":"2024-02-12T17:00:50.809381+0000","last_fullsized":"2024-02-12T17:00:50.809381+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:04.254680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:04.254680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":282,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807440+0000","last_change":"2024-02-12T16:57:48.350132+0000","last_active":"2024-02-12T17:00:50.807440+0000","last_peered":"2024-02-12T17:00:50.807440+0000","last_clean":"2024-02-12T17:00:50.807440+0000","last_became_active":"2024-02-12T16:57:47.200347+0000","last_became_peered":"2024-02-12T16:57:47.200347+0000","last_unstale":"2024-02-12T17:00:50.807440+0000","last_undegraded":"2024-02-12T17:00:50.807440+0000","last_fullsized":"2024-02-12T17:00:50.807440+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":320,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809499+0000","last_change":"2024-02-12T16:57:47.204108+0000","last_active":"2024-02-12T17:00:50.809499+0000","last_peered":"2024-02-12T17:00:50.809499+0000","last_clean":"2024-02-12T17:00:50.809499+0000","last_became_active":"2024-02-12T16:57:47.184456+0000","last_became_peered":"2024-02-12T16:57:47.184456+0000","last_unstale":"2024-02-12T17:00:50.809499+0000","last_undegraded":"2024-02-12T17:00:50.809499+0000","last_fullsized":"2024-02-12T17:00:50.809499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":283,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806888+0000","last_change":"2024-02-12T16:58:14.391518+0000","last_active":"2024-02-12T17:00:50.806888+0000","last_peered":"2024-02-12T17:00:50.806888+0000","last_clean":"2024-02-12T17:00:50.806888+0000","last_became_active":"2024-02-12T16:58:14.391221+0000","last_became_peered":"2024-02-12T16:58:14.391221+0000","last_unstale":"2024-02-12T17:00:50.806888+0000","last_undegraded":"2024-02-12T17:00:50.806888+0000","last_fullsized":"2024-02-12T17:00:50.806888+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:18.162190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:18.162190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"299'14002","reported_seq":34743,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.286832+0000","last_change":"2024-02-12T16:59:46.062924+0000","last_active":"2024-02-12T17:00:51.286832+0000","last_peered":"2024-02-12T17:00:51.286832+0000","last_clean":"2024-02-12T17:00:51.286832+0000","last_became_active":"2024-02-12T16:59:46.029014+0000","last_became_peered":"2024-02-12T16:59:46.029014+0000","last_unstale":"2024-02-12T17:00:51.286832+0000","last_undegraded":"2024-02-12T17:00:51.286832+0000","last_fullsized":"2024-02-12T17:00:51.286832+0000","mapping_epoch":275,"log_start":"271'10379","ondisk_log_start":"271'10379","created":176,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3623,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14002,"num_write_kb":448064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10282,"num_bytes_recovered":673841152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":56,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805512+0000","last_change":"2024-02-12T16:59:09.242340+0000","last_active":"2024-02-12T17:00:50.805512+0000","last_peered":"2024-02-12T17:00:50.805512+0000","last_clean":"2024-02-12T17:00:50.805512+0000","last_became_active":"2024-02-12T16:59:09.242012+0000","last_became_peered":"2024-02-12T16:59:09.242012+0000","last_unstale":"2024-02-12T17:00:50.805512+0000","last_undegraded":"2024-02-12T17:00:50.805512+0000","last_fullsized":"2024-02-12T17:00:50.805512+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:41.053587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.053587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806410+0000","last_change":"2024-02-12T16:54:01.572769+0000","last_active":"2024-02-12T17:00:50.806410+0000","last_peered":"2024-02-12T17:00:50.806410+0000","last_clean":"2024-02-12T17:00:50.806410+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T17:00:50.806410+0000","last_undegraded":"2024-02-12T17:00:50.806410+0000","last_fullsized":"2024-02-12T17:00:50.806410+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:01.572664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:01.572664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":312,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808646+0000","last_change":"2024-02-12T16:56:30.812742+0000","last_active":"2024-02-12T17:00:50.808646+0000","last_peered":"2024-02-12T17:00:50.808646+0000","last_clean":"2024-02-12T17:00:50.808646+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T17:00:50.808646+0000","last_undegraded":"2024-02-12T17:00:50.808646+0000","last_fullsized":"2024-02-12T17:00:50.808646+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:30.812679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.812679+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808215+0000","last_change":"2024-02-12T16:59:09.271420+0000","last_active":"2024-02-12T17:00:50.808215+0000","last_peered":"2024-02-12T17:00:50.808215+0000","last_clean":"2024-02-12T17:00:50.808215+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:00:50.808215+0000","last_undegraded":"2024-02-12T17:00:50.808215+0000","last_fullsized":"2024-02-12T17:00:50.808215+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"299'13970","reported_seq":14101,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.289630+0000","last_change":"2024-02-12T16:56:36.789146+0000","last_active":"2024-02-12T17:00:51.289630+0000","last_peered":"2024-02-12T17:00:51.289630+0000","last_clean":"2024-02-12T17:00:51.289630+0000","last_became_active":"2024-02-12T16:56:36.788871+0000","last_became_peered":"2024-02-12T16:56:36.788871+0000","last_unstale":"2024-02-12T17:00:51.289630+0000","last_undegraded":"2024-02-12T17:00:51.289630+0000","last_fullsized":"2024-02-12T17:00:51.289630+0000","mapping_epoch":176,"log_start":"269'10348","ondisk_log_start":"269'10348","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3622,"ondisk_log_size":3622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13970,"num_write_kb":447040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":190,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805807+0000","last_change":"2024-02-12T16:59:09.242302+0000","last_active":"2024-02-12T17:00:50.805807+0000","last_peered":"2024-02-12T17:00:50.805807+0000","last_clean":"2024-02-12T17:00:50.805807+0000","last_became_active":"2024-02-12T16:59:09.241987+0000","last_became_peered":"2024-02-12T16:59:09.241987+0000","last_unstale":"2024-02-12T17:00:50.805807+0000","last_undegraded":"2024-02-12T17:00:50.805807+0000","last_fullsized":"2024-02-12T17:00:50.805807+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:52.143874+0000","last_change":"2024-02-12T17:00:52.143874+0000","last_active":"2024-02-12T17:00:52.143874+0000","last_peered":"2024-02-12T17:00:52.143874+0000","last_clean":"2024-02-12T17:00:52.143874+0000","last_became_active":"2024-02-12T16:59:09.242505+0000","last_became_peered":"2024-02-12T16:59:09.242505+0000","last_unstale":"2024-02-12T17:00:52.143874+0000","last_undegraded":"2024-02-12T17:00:52.143874+0000","last_fullsized":"2024-02-12T17:00:52.143874+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:00:52.143796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":285,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807874+0000","last_change":"2024-02-12T16:57:48.521837+0000","last_active":"2024-02-12T17:00:50.807874+0000","last_peered":"2024-02-12T17:00:50.807874+0000","last_clean":"2024-02-12T17:00:50.807874+0000","last_became_active":"2024-02-12T16:57:47.199763+0000","last_became_peered":"2024-02-12T16:57:47.199763+0000","last_unstale":"2024-02-12T17:00:50.807874+0000","last_undegraded":"2024-02-12T17:00:50.807874+0000","last_fullsized":"2024-02-12T17:00:50.807874+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-12T16:56:26.889441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.889441+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":271,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.802660+0000","last_change":"2024-02-12T16:56:19.484845+0000","last_active":"2024-02-12T17:00:50.802660+0000","last_peered":"2024-02-12T17:00:50.802660+0000","last_clean":"2024-02-12T17:00:50.802660+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T17:00:50.802660+0000","last_undegraded":"2024-02-12T17:00:50.802660+0000","last_fullsized":"2024-02-12T17:00:50.802660+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:19.484711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:19.484711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"299'14200","reported_seq":14332,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.295852+0000","last_change":"2024-02-12T16:56:36.788810+0000","last_active":"2024-02-12T17:00:51.295852+0000","last_peered":"2024-02-12T17:00:51.295852+0000","last_clean":"2024-02-12T17:00:51.295852+0000","last_became_active":"2024-02-12T16:56:36.788518+0000","last_became_peered":"2024-02-12T16:56:36.788518+0000","last_unstale":"2024-02-12T17:00:51.295852+0000","last_undegraded":"2024-02-12T17:00:51.295852+0000","last_fullsized":"2024-02-12T17:00:51.295852+0000","mapping_epoch":176,"log_start":"271'10612","ondisk_log_start":"271'10612","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3588,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14200,"num_write_kb":454337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":271,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808322+0000","last_change":"2024-02-12T16:59:09.340317+0000","last_active":"2024-02-12T17:00:50.808322+0000","last_peered":"2024-02-12T17:00:50.808322+0000","last_clean":"2024-02-12T17:00:50.808322+0000","last_became_active":"2024-02-12T16:59:09.246616+0000","last_became_peered":"2024-02-12T16:59:09.246616+0000","last_unstale":"2024-02-12T17:00:50.808322+0000","last_undegraded":"2024-02-12T17:00:50.808322+0000","last_fullsized":"2024-02-12T17:00:50.808322+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:57:19.955311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.955311+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":53,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807698+0000","last_change":"2024-02-12T16:59:09.268325+0000","last_active":"2024-02-12T17:00:50.807698+0000","last_peered":"2024-02-12T17:00:50.807698+0000","last_clean":"2024-02-12T17:00:50.807698+0000","last_became_active":"2024-02-12T16:59:09.268028+0000","last_became_peered":"2024-02-12T16:59:09.268028+0000","last_unstale":"2024-02-12T17:00:50.807698+0000","last_undegraded":"2024-02-12T17:00:50.807698+0000","last_fullsized":"2024-02-12T17:00:50.807698+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:26.337455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.337455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":302,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805170+0000","last_change":"2024-02-12T16:59:09.996418+0000","last_active":"2024-02-12T17:00:50.805170+0000","last_peered":"2024-02-12T17:00:50.805170+0000","last_clean":"2024-02-12T17:00:50.805170+0000","last_became_active":"2024-02-12T16:59:09.256058+0000","last_became_peered":"2024-02-12T16:59:09.256058+0000","last_unstale":"2024-02-12T17:00:50.805170+0000","last_undegraded":"2024-02-12T17:00:50.805170+0000","last_fullsized":"2024-02-12T17:00:50.805170+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"208'10","reported_seq":169,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:53.140099+0000","last_change":"2024-02-12T17:00:53.140099+0000","last_active":"2024-02-12T17:00:53.140099+0000","last_peered":"2024-02-12T17:00:53.140099+0000","last_clean":"2024-02-12T17:00:53.140099+0000","last_became_active":"2024-02-12T16:57:47.167628+0000","last_became_peered":"2024-02-12T16:57:47.167628+0000","last_unstale":"2024-02-12T17:00:53.140099+0000","last_undegraded":"2024-02-12T17:00:53.140099+0000","last_fullsized":"2024-02-12T17:00:53.140099+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-12T17:00:53.139979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.139979+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"299'13786","reported_seq":13917,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.274364+0000","last_change":"2024-02-12T16:56:36.445962+0000","last_active":"2024-02-12T17:00:51.274364+0000","last_peered":"2024-02-12T17:00:51.274364+0000","last_clean":"2024-02-12T17:00:51.274364+0000","last_became_active":"2024-02-12T16:56:36.445638+0000","last_became_peered":"2024-02-12T16:56:36.445638+0000","last_unstale":"2024-02-12T17:00:51.274364+0000","last_undegraded":"2024-02-12T17:00:51.274364+0000","last_fullsized":"2024-02-12T17:00:51.274364+0000","mapping_epoch":176,"log_start":"267'9923","ondisk_log_start":"267'9923","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3863,"ondisk_log_size":3863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13786,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":56,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805167+0000","last_change":"2024-02-12T16:59:09.260336+0000","last_active":"2024-02-12T17:00:50.805167+0000","last_peered":"2024-02-12T17:00:50.805167+0000","last_clean":"2024-02-12T17:00:50.805167+0000","last_became_active":"2024-02-12T16:59:09.259654+0000","last_became_peered":"2024-02-12T16:59:09.259654+0000","last_unstale":"2024-02-12T17:00:50.805167+0000","last_undegraded":"2024-02-12T17:00:50.805167+0000","last_fullsized":"2024-02-12T17:00:50.805167+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:32.693800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:32.693800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"261'14","reported_seq":346,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.228166+0000","last_change":"2024-02-12T16:59:12.225158+0000","last_active":"2024-02-12T17:00:51.228166+0000","last_peered":"2024-02-12T17:00:51.228166+0000","last_clean":"2024-02-12T17:00:51.228166+0000","last_became_active":"2024-02-12T16:59:12.224830+0000","last_became_peered":"2024-02-12T16:59:12.224830+0000","last_unstale":"2024-02-12T17:00:51.228166+0000","last_undegraded":"2024-02-12T17:00:51.228166+0000","last_fullsized":"2024-02-12T17:00:51.228166+0000","mapping_epoch":260,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":251,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.804832+0000","last_change":"2024-02-12T16:57:47.171714+0000","last_active":"2024-02-12T17:00:50.804832+0000","last_peered":"2024-02-12T17:00:50.804832+0000","last_clean":"2024-02-12T17:00:50.804832+0000","last_became_active":"2024-02-12T16:57:47.171128+0000","last_became_peered":"2024-02-12T16:57:47.171128+0000","last_unstale":"2024-02-12T17:00:50.804832+0000","last_undegraded":"2024-02-12T17:00:50.804832+0000","last_fullsized":"2024-02-12T17:00:50.804832+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":266,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805471+0000","last_change":"2024-02-12T16:59:09.788881+0000","last_active":"2024-02-12T17:00:50.805471+0000","last_peered":"2024-02-12T17:00:50.805471+0000","last_clean":"2024-02-12T17:00:50.805471+0000","last_became_active":"2024-02-12T16:59:09.277264+0000","last_became_peered":"2024-02-12T16:59:09.277264+0000","last_unstale":"2024-02-12T17:00:50.805471+0000","last_undegraded":"2024-02-12T17:00:50.805471+0000","last_fullsized":"2024-02-12T17:00:50.805471+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"299'14052","reported_seq":18961,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.286779+0000","last_change":"2024-02-12T17:00:50.846964+0000","last_active":"2024-02-12T17:00:51.286779+0000","last_peered":"2024-02-12T17:00:51.286779+0000","last_clean":"2024-02-12T17:00:51.286779+0000","last_became_active":"2024-02-12T17:00:50.809500+0000","last_became_peered":"2024-02-12T17:00:50.809500+0000","last_unstale":"2024-02-12T17:00:51.286779+0000","last_undegraded":"2024-02-12T17:00:51.286779+0000","last_fullsized":"2024-02-12T17:00:51.286779+0000","mapping_epoch":297,"log_start":"271'10501","ondisk_log_start":"271'10501","created":176,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3551,"ondisk_log_size":3551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14052,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2079,"num_bytes_recovered":136249344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":268,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806258+0000","last_change":"2024-02-12T16:57:47.177662+0000","last_active":"2024-02-12T17:00:50.806258+0000","last_peered":"2024-02-12T17:00:50.806258+0000","last_clean":"2024-02-12T17:00:50.806258+0000","last_became_active":"2024-02-12T16:57:47.177329+0000","last_became_peered":"2024-02-12T16:57:47.177329+0000","last_unstale":"2024-02-12T17:00:50.806258+0000","last_undegraded":"2024-02-12T17:00:50.806258+0000","last_fullsized":"2024-02-12T17:00:50.806258+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:40.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:56:40.152265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"208'12","reported_seq":391,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.225398+0000","last_change":"2024-02-12T16:57:46.177864+0000","last_active":"2024-02-12T17:00:51.225398+0000","last_peered":"2024-02-12T17:00:51.225398+0000","last_clean":"2024-02-12T17:00:51.225398+0000","last_became_active":"2024-02-12T16:57:46.177517+0000","last_became_peered":"2024-02-12T16:57:46.177517+0000","last_unstale":"2024-02-12T17:00:51.225398+0000","last_undegraded":"2024-02-12T17:00:51.225398+0000","last_fullsized":"2024-02-12T17:00:51.225398+0000","mapping_epoch":97,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-02-12T16:56:32.292727+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.292727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":51,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805873+0000","last_change":"2024-02-12T16:59:09.268099+0000","last_active":"2024-02-12T17:00:50.805873+0000","last_peered":"2024-02-12T17:00:50.805873+0000","last_clean":"2024-02-12T17:00:50.805873+0000","last_became_active":"2024-02-12T16:59:09.267755+0000","last_became_peered":"2024-02-12T16:59:09.267755+0000","last_unstale":"2024-02-12T17:00:50.805873+0000","last_undegraded":"2024-02-12T17:00:50.805873+0000","last_fullsized":"2024-02-12T17:00:50.805873+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:37.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:37.041320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":271,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807485+0000","last_change":"2024-02-12T16:57:48.169827+0000","last_active":"2024-02-12T17:00:50.807485+0000","last_peered":"2024-02-12T17:00:50.807485+0000","last_clean":"2024-02-12T17:00:50.807485+0000","last_became_active":"2024-02-12T16:57:47.199008+0000","last_became_peered":"2024-02-12T16:57:47.199008+0000","last_unstale":"2024-02-12T17:00:50.807485+0000","last_undegraded":"2024-02-12T17:00:50.807485+0000","last_fullsized":"2024-02-12T17:00:50.807485+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2024-02-12T16:54:38.016467+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.016467+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"299'13994","reported_seq":18061,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.270478+0000","last_change":"2024-02-12T17:00:49.832054+0000","last_active":"2024-02-12T17:00:51.270478+0000","last_peered":"2024-02-12T17:00:51.270478+0000","last_clean":"2024-02-12T17:00:51.270478+0000","last_became_active":"2024-02-12T17:00:49.792541+0000","last_became_peered":"2024-02-12T17:00:49.792541+0000","last_unstale":"2024-02-12T17:00:51.270478+0000","last_undegraded":"2024-02-12T17:00:51.270478+0000","last_fullsized":"2024-02-12T17:00:51.270478+0000","mapping_epoch":296,"log_start":"271'10414","ondisk_log_start":"271'10414","created":176,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3580,"ondisk_log_size":3580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13994,"num_write_kb":447808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1952,"num_bytes_recovered":127926272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":106,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808627+0000","last_change":"2024-02-12T16:57:47.176436+0000","last_active":"2024-02-12T17:00:50.808627+0000","last_peered":"2024-02-12T17:00:50.808627+0000","last_clean":"2024-02-12T17:00:50.808627+0000","last_became_active":"2024-02-12T16:57:47.176041+0000","last_became_peered":"2024-02-12T16:57:47.176041+0000","last_unstale":"2024-02-12T17:00:50.808627+0000","last_undegraded":"2024-02-12T17:00:50.808627+0000","last_fullsized":"2024-02-12T17:00:50.808627+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"211'10","reported_seq":423,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.218398+0000","last_change":"2024-02-12T16:57:50.205871+0000","last_active":"2024-02-12T17:00:51.218398+0000","last_peered":"2024-02-12T17:00:51.218398+0000","last_clean":"2024-02-12T17:00:51.218398+0000","last_became_active":"2024-02-12T16:57:50.205330+0000","last_became_peered":"2024-02-12T16:57:50.205330+0000","last_unstale":"2024-02-12T17:00:51.218398+0000","last_undegraded":"2024-02-12T17:00:51.218398+0000","last_fullsized":"2024-02-12T17:00:51.218398+0000","mapping_epoch":61,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-12T16:56:26.673483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:26.673483+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":207,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806440+0000","last_change":"2024-02-12T16:57:47.191627+0000","last_active":"2024-02-12T17:00:50.806440+0000","last_peered":"2024-02-12T17:00:50.806440+0000","last_clean":"2024-02-12T17:00:50.806440+0000","last_became_active":"2024-02-12T16:57:47.191231+0000","last_became_peered":"2024-02-12T16:57:47.191231+0000","last_unstale":"2024-02-12T17:00:50.806440+0000","last_undegraded":"2024-02-12T17:00:50.806440+0000","last_fullsized":"2024-02-12T17:00:50.806440+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:39.006483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:39.006483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":206,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806025+0000","last_change":"2024-02-12T16:59:09.782291+0000","last_active":"2024-02-12T17:00:50.806025+0000","last_peered":"2024-02-12T17:00:50.806025+0000","last_clean":"2024-02-12T17:00:50.806025+0000","last_became_active":"2024-02-12T16:59:09.268136+0000","last_became_peered":"2024-02-12T16:59:09.268136+0000","last_unstale":"2024-02-12T17:00:50.806025+0000","last_undegraded":"2024-02-12T17:00:50.806025+0000","last_fullsized":"2024-02-12T17:00:50.806025+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:55:46.162840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:55:46.162840+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"299'13738","reported_seq":17335,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.278851+0000","last_change":"2024-02-12T17:00:47.813948+0000","last_active":"2024-02-12T17:00:51.278851+0000","last_peered":"2024-02-12T17:00:51.278851+0000","last_clean":"2024-02-12T17:00:51.278851+0000","last_became_active":"2024-02-12T17:00:47.776923+0000","last_became_peered":"2024-02-12T17:00:47.776923+0000","last_unstale":"2024-02-12T17:00:51.278851+0000","last_undegraded":"2024-02-12T17:00:51.278851+0000","last_fullsized":"2024-02-12T17:00:51.278851+0000","mapping_epoch":294,"log_start":"269'10136","ondisk_log_start":"269'10136","created":176,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3602,"ondisk_log_size":3602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13738,"num_write_kb":439616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":112525312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":282,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807978+0000","last_change":"2024-02-12T16:59:09.246938+0000","last_active":"2024-02-12T17:00:50.807978+0000","last_peered":"2024-02-12T17:00:50.807978+0000","last_clean":"2024-02-12T17:00:50.807978+0000","last_became_active":"2024-02-12T16:59:09.246605+0000","last_became_peered":"2024-02-12T16:59:09.246605+0000","last_unstale":"2024-02-12T17:00:50.807978+0000","last_undegraded":"2024-02-12T17:00:50.807978+0000","last_fullsized":"2024-02-12T17:00:50.807978+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":130,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.803222+0000","last_change":"2024-02-12T16:59:09.247764+0000","last_active":"2024-02-12T17:00:50.803222+0000","last_peered":"2024-02-12T17:00:50.803222+0000","last_clean":"2024-02-12T17:00:50.803222+0000","last_became_active":"2024-02-12T16:59:09.247447+0000","last_became_peered":"2024-02-12T16:59:09.247447+0000","last_unstale":"2024-02-12T17:00:50.803222+0000","last_undegraded":"2024-02-12T17:00:50.803222+0000","last_fullsized":"2024-02-12T17:00:50.803222+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T16:56:33.052447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":263,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808081+0000","last_change":"2024-02-12T16:54:31.784714+0000","last_active":"2024-02-12T17:00:50.808081+0000","last_peered":"2024-02-12T17:00:50.808081+0000","last_clean":"2024-02-12T17:00:50.808081+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T17:00:50.808081+0000","last_undegraded":"2024-02-12T17:00:50.808081+0000","last_fullsized":"2024-02-12T17:00:50.808081+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:31.784622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:31.784622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":365,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807222+0000","last_change":"2024-02-12T16:59:10.004273+0000","last_active":"2024-02-12T17:00:50.807222+0000","last_peered":"2024-02-12T17:00:50.807222+0000","last_clean":"2024-02-12T17:00:50.807222+0000","last_became_active":"2024-02-12T16:59:09.262475+0000","last_became_peered":"2024-02-12T16:59:09.262475+0000","last_unstale":"2024-02-12T17:00:50.807222+0000","last_undegraded":"2024-02-12T17:00:50.807222+0000","last_fullsized":"2024-02-12T17:00:50.807222+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-12T16:56:30.714432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:30.714432+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"299'13974","reported_seq":22898,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.284744+0000","last_change":"2024-02-12T16:59:27.226608+0000","last_active":"2024-02-12T17:00:51.284744+0000","last_peered":"2024-02-12T17:00:51.284744+0000","last_clean":"2024-02-12T17:00:51.284744+0000","last_became_active":"2024-02-12T16:59:27.206365+0000","last_became_peered":"2024-02-12T16:59:27.206365+0000","last_unstale":"2024-02-12T17:00:51.284744+0000","last_undegraded":"2024-02-12T17:00:51.284744+0000","last_fullsized":"2024-02-12T17:00:51.284744+0000","mapping_epoch":265,"log_start":"271'10430","ondisk_log_start":"271'10430","created":176,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3544,"ondisk_log_size":3544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13974,"num_write_kb":447168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4385,"num_bytes_recovered":287375360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":275,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807358+0000","last_change":"2024-02-12T16:59:09.242082+0000","last_active":"2024-02-12T17:00:50.807358+0000","last_peered":"2024-02-12T17:00:50.807358+0000","last_clean":"2024-02-12T17:00:50.807358+0000","last_became_active":"2024-02-12T16:59:09.241648+0000","last_became_peered":"2024-02-12T16:59:09.241648+0000","last_unstale":"2024-02-12T17:00:50.807358+0000","last_undegraded":"2024-02-12T17:00:50.807358+0000","last_fullsized":"2024-02-12T17:00:50.807358+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":284,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806552+0000","last_change":"2024-02-12T16:57:58.203843+0000","last_active":"2024-02-12T17:00:50.806552+0000","last_peered":"2024-02-12T17:00:50.806552+0000","last_clean":"2024-02-12T17:00:50.806552+0000","last_became_active":"2024-02-12T16:57:58.203578+0000","last_became_peered":"2024-02-12T16:57:58.203578+0000","last_unstale":"2024-02-12T17:00:50.806552+0000","last_undegraded":"2024-02-12T17:00:50.806552+0000","last_fullsized":"2024-02-12T17:00:50.806552+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":265,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807228+0000","last_change":"2024-02-12T16:54:38.398912+0000","last_active":"2024-02-12T17:00:50.807228+0000","last_peered":"2024-02-12T17:00:50.807228+0000","last_clean":"2024-02-12T17:00:50.807228+0000","last_became_active":"2024-02-12T16:49:39.106312+0000","last_became_peered":"2024-02-12T16:49:39.106312+0000","last_unstale":"2024-02-12T17:00:50.807228+0000","last_undegraded":"2024-02-12T17:00:50.807228+0000","last_fullsized":"2024-02-12T17:00:50.807228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:38.398801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:54:38.398801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":140,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806941+0000","last_change":"2024-02-12T16:59:10.836208+0000","last_active":"2024-02-12T17:00:50.806941+0000","last_peered":"2024-02-12T17:00:50.806941+0000","last_clean":"2024-02-12T17:00:50.806941+0000","last_became_active":"2024-02-12T16:59:09.273985+0000","last_became_peered":"2024-02-12T16:59:09.273985+0000","last_unstale":"2024-02-12T17:00:50.806941+0000","last_undegraded":"2024-02-12T17:00:50.806941+0000","last_fullsized":"2024-02-12T17:00:50.806941+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-12T16:56:41.290236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:41.290236+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"299'13910","reported_seq":34535,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.286826+0000","last_change":"2024-02-12T16:59:45.048235+0000","last_active":"2024-02-12T17:00:51.286826+0000","last_peered":"2024-02-12T17:00:51.286826+0000","last_clean":"2024-02-12T17:00:51.286826+0000","last_became_active":"2024-02-12T16:59:45.029644+0000","last_became_peered":"2024-02-12T16:59:45.029644+0000","last_unstale":"2024-02-12T17:00:51.286826+0000","last_undegraded":"2024-02-12T17:00:51.286826+0000","last_fullsized":"2024-02-12T17:00:51.286826+0000","mapping_epoch":274,"log_start":"269'10272","ondisk_log_start":"269'10272","created":176,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3638,"ondisk_log_size":3638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13910,"num_write_kb":445120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10227,"num_bytes_recovered":670236672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":252,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.809518+0000","last_change":"2024-02-12T16:54:00.186737+0000","last_active":"2024-02-12T17:00:50.809518+0000","last_peered":"2024-02-12T17:00:50.809518+0000","last_clean":"2024-02-12T17:00:50.809518+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T17:00:50.809518+0000","last_undegraded":"2024-02-12T17:00:50.809518+0000","last_fullsized":"2024-02-12T17:00:50.809518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:00.186645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:00.186645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":288,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.805667+0000","last_change":"2024-02-12T16:58:02.376971+0000","last_active":"2024-02-12T17:00:50.805667+0000","last_peered":"2024-02-12T17:00:50.805667+0000","last_clean":"2024-02-12T17:00:50.805667+0000","last_became_active":"2024-02-12T16:58:02.376628+0000","last_became_peered":"2024-02-12T16:58:02.376628+0000","last_unstale":"2024-02-12T17:00:50.805667+0000","last_undegraded":"2024-02-12T17:00:50.805667+0000","last_fullsized":"2024-02-12T17:00:50.805667+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:19.703739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.703739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":234,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.807407+0000","last_change":"2024-02-12T17:00:50.756710+0000","last_active":"2024-02-12T17:00:50.807407+0000","last_peered":"2024-02-12T17:00:50.807407+0000","last_clean":"2024-02-12T17:00:50.807407+0000","last_became_active":"2024-02-12T16:59:09.262401+0000","last_became_peered":"2024-02-12T16:59:09.262401+0000","last_unstale":"2024-02-12T17:00:50.807407+0000","last_undegraded":"2024-02-12T17:00:50.807407+0000","last_fullsized":"2024-02-12T17:00:50.807407+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:00:50.756578+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.756578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":305,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808733+0000","last_change":"2024-02-12T16:57:47.553993+0000","last_active":"2024-02-12T17:00:50.808733+0000","last_peered":"2024-02-12T17:00:50.808733+0000","last_clean":"2024-02-12T17:00:50.808733+0000","last_became_active":"2024-02-12T16:57:47.189672+0000","last_became_peered":"2024-02-12T16:57:47.189672+0000","last_unstale":"2024-02-12T17:00:50.808733+0000","last_undegraded":"2024-02-12T17:00:50.808733+0000","last_fullsized":"2024-02-12T17:00:50.808733+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:56:31.836773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:56:31.836773+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"299'14094","reported_seq":14225,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.292687+0000","last_change":"2024-02-12T16:56:36.445585+0000","last_active":"2024-02-12T17:00:51.292687+0000","last_peered":"2024-02-12T17:00:51.292687+0000","last_clean":"2024-02-12T17:00:51.292687+0000","last_became_active":"2024-02-12T16:56:36.445291+0000","last_became_peered":"2024-02-12T16:56:36.445291+0000","last_unstale":"2024-02-12T17:00:51.292687+0000","last_undegraded":"2024-02-12T17:00:51.292687+0000","last_fullsized":"2024-02-12T17:00:51.292687+0000","mapping_epoch":176,"log_start":"271'10594","ondisk_log_start":"271'10594","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:35.434299+0000","last_clean_scrub_stamp":"2024-02-12T16:56:35.434299+0000","log_size":3500,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14094,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":263,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.806011+0000","last_change":"2024-02-12T16:59:09.262958+0000","last_active":"2024-02-12T17:00:50.806011+0000","last_peered":"2024-02-12T17:00:50.806011+0000","last_clean":"2024-02-12T17:00:50.806011+0000","last_became_active":"2024-02-12T16:59:09.261034+0000","last_became_peered":"2024-02-12T16:59:09.261034+0000","last_unstale":"2024-02-12T17:00:50.806011+0000","last_undegraded":"2024-02-12T17:00:50.806011+0000","last_fullsized":"2024-02-12T17:00:50.806011+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:03.996127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:54:03.996127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":288,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.808298+0000","last_change":"2024-02-12T16:59:09.250623+0000","last_active":"2024-02-12T17:00:50.808298+0000","last_peered":"2024-02-12T17:00:50.808298+0000","last_clean":"2024-02-12T17:00:50.808298+0000","last_became_active":"2024-02-12T16:59:09.250318+0000","last_became_peered":"2024-02-12T16:59:09.250318+0000","last_unstale":"2024-02-12T17:00:50.808298+0000","last_undegraded":"2024-02-12T17:00:50.808298+0000","last_fullsized":"2024-02-12T17:00:50.808298+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:32.717104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:56:32.717104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":338,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-12T17:00:50.804549+0000","last_change":"2024-02-12T16:59:11.094393+0000","last_active":"2024-02-12T17:00:50.804549+0000","last_peered":"2024-02-12T17:00:50.804549+0000","last_clean":"2024-02-12T17:00:50.804549+0000","last_became_active":"2024-02-12T16:59:09.280858+0000","last_became_peered":"2024-02-12T16:59:09.280858+0000","last_unstale":"2024-02-12T17:00:50.804549+0000","last_undegraded":"2024-02-12T17:00:50.804549+0000","last_fullsized":"2024-02-12T17:00:50.804549+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'22","last_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T16:56:39.138758+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"261'13","reported_seq":448,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-02-12T17:00:51.222039+0000","last_change":"2024-02-12T17:00:49.304845+0000","last_active":"2024-02-12T17:00:51.222039+0000","last_peered":"2024-02-12T17:00:51.222039+0000","last_clean":"2024-02-12T17:00:51.222039+0000","last_became_active":"2024-02-12T16:59:12.242710+0000","last_became_peered":"2024-02-12T16:59:12.242710+0000","last_unstale":"2024-02-12T17:00:51.222039+0000","last_undegraded":"2024-02-12T17:00:51.222039+0000","last_fullsized":"2024-02-12T17:00:51.222039+0000","mapping_epoch":260,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-02-12T17:00:49.304784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.304784+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":223312,"num_write_kb":7145921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39918,"num_bytes_recovered":2616066048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":846462976,"data_stored":846462976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57995,"ondisk_log_size":57995,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":190,"num_read_kb":185,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":5408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53,"ondisk_log_size":53,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234122,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298832,"kb_used_data":1608,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446320,"statfs":{"total":95995035648,"available":95689031680,"internally_reserved":0,"allocated":1646592,"data_stored":868599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:00:51 2024","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":1.6679999999999999,"15min":1.2909999999999999},"min":{"1min":0.31900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":16.759,"5min":16.759,"15min":36.439},"last":0.60099999999999998},{"interface":"front","average":{"1min":4.5439999999999996,"5min":2.4820000000000002,"15min":1.5209999999999999},"min":{"1min":0.20399999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":17.844000000000001,"5min":17.844000000000001,"15min":36.490000000000002},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Feb 12 17:00:51 2024","interfaces":[{"interface":"back","average":{"1min":2.0499999999999998,"5min":1.754,"15min":1.1100000000000001},"min":{"1min":0.40600000000000003,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.1820000000000004,"5min":11.048999999999999,"15min":12.032},"last":0.54200000000000004},{"interface":"front","average":{"1min":3.4359999999999999,"5min":1.9399999999999999,"15min":1.169},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":17.324999999999999,"5min":17.324999999999999,"15min":17.324999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Mon Feb 12 17:00:51 2024","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":1.371,"15min":0.97899999999999998},"min":{"1min":0.316,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.7319999999999993,"5min":9.7319999999999993,"15min":9.7319999999999993},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.9490000000000001,"15min":1.2350000000000001},"min":{"1min":0.38200000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":6.1580000000000004,"5min":11.071999999999999,"15min":20.338000000000001},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Feb 12 17:00:51 2024","interfaces":[{"interface":"back","average":{"1min":2.2709999999999999,"5min":1.575,"15min":1.274},"min":{"1min":0.29699999999999999,"5min":0.28999999999999998,"15min":0.26400000000000001},"max":{"1min":9.7490000000000006,"5min":9.7490000000000006,"15min":20.433},"last":0.54800000000000004},{"interface":"front","average":{"1min":2.194,"5min":2.0339999999999998,"15min":1.4419999999999999},"min":{"1min":0.441,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.1200000000000001,"5min":11.092000000000001,"15min":20.393000000000001},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:00:51 2024","interfaces":[{"interface":"back","average":{"1min":3.3530000000000002,"5min":2.4199999999999999,"15min":1.1679999999999999},"min":{"1min":0.33600000000000002,"5min":0.26300000000000001,"15min":0.219},"max":{"1min":9.7089999999999996,"5min":50.406999999999996,"15min":50.406999999999996},"last":0.40100000000000002},{"interface":"front","average":{"1min":3.4580000000000002,"5min":3.3050000000000002,"15min":1.4650000000000001},"min":{"1min":0.32900000000000001,"5min":0.28100000000000003,"15min":0.23400000000000001},"max":{"1min":11.512,"5min":50.168999999999997,"15min":50.168999999999997},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:00:51 2024","interfaces":[{"interface":"back","average":{"1min":4.3449999999999998,"5min":2.202,"15min":1.087},"min":{"1min":0.41999999999999998,"5min":0.29299999999999998,"15min":0.24199999999999999},"max":{"1min":10.91,"5min":10.91,"15min":10.91},"last":0.44500000000000001},{"interface":"front","average":{"1min":3.714,"5min":2.7759999999999998,"15min":1.304},"min":{"1min":0.495,"5min":0.33000000000000002,"15min":0.26200000000000001},"max":{"1min":18.651,"5min":18.651,"15min":18.651},"last":0.496}]},{"osd":6,"last update":"Mon Feb 12 17:00:51 2024","interfaces":[{"interface":"back","average":{"1min":3.6720000000000002,"5min":2.2189999999999999,"15min":1.1120000000000001},"min":{"1min":0.35799999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":17.234999999999999,"5min":17.234999999999999,"15min":17.234999999999999},"last":0.628},{"interface":"front","average":{"1min":2.9279999999999999,"5min":1.9450000000000001,"15min":1.0680000000000001},"min":{"1min":0.39200000000000002,"5min":0.33100000000000002,"15min":0.30199999999999999},"max":{"1min":9.7739999999999991,"5min":12.407999999999999,"15min":12.407999999999999},"last":0.57199999999999995}]}]},{"osd":6,"up_from":0,"seq":193273528463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738540,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315208,"kb_used_data":1600,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93429944,"statfs":{"total":95995035648,"available":95672262656,"internally_reserved":0,"allocated":1638400,"data_stored":867718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.393,"15min":1.1479999999999999},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":13.013,"5min":16.146999999999998,"15min":23.510999999999999},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.3740000000000001,"15min":1.0740000000000001},"min":{"1min":0.20100000000000001,"5min":0.193,"15min":0.193},"max":{"1min":3.5,"5min":8.6500000000000004,"15min":10.186999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.841,"15min":1.3180000000000001},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.20499999999999999},"max":{"1min":11.122999999999999,"5min":16.521000000000001,"15min":16.521000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.6100000000000001,"15min":1.1459999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.984,"5min":10.340999999999999,"15min":10.340999999999999},"last":0.378}]},{"osd":3,"last update":"Mon Feb 12 17:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.617,"15min":1.3799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":7.0339999999999998,"5min":16.27,"15min":21.100000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.45,"15min":1.381},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27600000000000002},"max":{"1min":11.976000000000001,"5min":11.976000000000001,"15min":16.446000000000002},"last":0.311}]},{"osd":4,"last update":"Mon Feb 12 17:00:34 2024","interfaces":[{"interface":"back","average":{"1min":3.4089999999999998,"5min":1.8169999999999999,"15min":1.3140000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":16.888999999999999,"5min":16.888999999999999,"15min":16.888999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":3.2440000000000002,"5min":1.9790000000000001,"15min":1.2609999999999999},"min":{"1min":0.376,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":16.899999999999999,"5min":16.899999999999999,"15min":16.899999999999999},"last":0.44500000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:00:08 2024","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.5070000000000001,"15min":1.456},"min":{"1min":0.308,"5min":0.308,"15min":0.26400000000000001},"max":{"1min":7.1600000000000001,"5min":16.190999999999999,"15min":16.190999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":3.028,"5min":2.2080000000000002,"15min":1.431},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":11.423,"5min":18.170999999999999,"15min":18.170999999999999},"last":0.435}]},{"osd":6,"last update":"Mon Feb 12 17:00:34 2024","interfaces":[{"interface":"back","average":{"1min":2.4769999999999999,"5min":2.1110000000000002,"15min":1.3380000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":9.8460000000000001,"5min":17.907,"15min":17.907},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.9220000000000002,"5min":2.4129999999999998,"15min":1.516},"min":{"1min":0.372,"5min":0.33600000000000002,"15min":0.27900000000000003},"max":{"1min":9.6739999999999995,"5min":16.294,"15min":21.062000000000001},"last":0.39000000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.472,"5min":1.9390000000000001,"15min":1.2669999999999999},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.32400000000000001},"max":{"1min":12.058,"5min":16.175000000000001,"15min":16.175000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":3.79,"5min":2.1589999999999998,"15min":1.274},"min":{"1min":0.45600000000000002,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":34.722999999999999,"5min":34.722999999999999,"15min":34.722999999999999},"last":0.46400000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302884,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442268,"statfs":{"total":95995035648,"available":95684882432,"internally_reserved":0,"allocated":1601536,"data_stored":867504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.9550000000000001,"15min":1.3440000000000001},"min":{"1min":0.192,"5min":0.186,"15min":0.186},"max":{"1min":3.7400000000000002,"5min":15.808,"15min":15.808},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.756,"15min":1.1899999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.158},"max":{"1min":6.1070000000000002,"5min":13.497,"15min":19.893000000000001},"last":0.371}]},{"osd":1,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":2.0009999999999999,"15min":1.4099999999999999},"min":{"1min":0.23400000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":6.5819999999999999,"5min":17.739000000000001,"15min":17.739000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.9399999999999999,"15min":1.4059999999999999},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":3.7629999999999999,"5min":15.766999999999999,"15min":15.766999999999999},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon Feb 12 17:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.3939999999999999,"15min":1.5700000000000001},"min":{"1min":0.29899999999999999,"5min":0.28000000000000003,"15min":0.23499999999999999},"max":{"1min":2.976,"5min":13.596,"15min":51.579999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.3720000000000001,"15min":1.337},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1310000000000002,"5min":13.521000000000001,"15min":48.988999999999997},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:00:53 2024","interfaces":[{"interface":"back","average":{"1min":2.9180000000000001,"5min":2.1779999999999999,"15min":1.5129999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":18.658999999999999,"5min":18.658999999999999,"15min":18.658999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":3.4729999999999999,"5min":2.3919999999999999,"15min":1.4430000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.26800000000000002},"max":{"1min":17.771000000000001,"5min":17.771000000000001,"15min":17.771000000000001},"last":0.749}]},{"osd":5,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":2.081,"15min":1.4379999999999999},"min":{"1min":0.41499999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":10.407999999999999,"5min":15.699,"15min":15.699},"last":0.78300000000000003},{"interface":"front","average":{"1min":2.9580000000000002,"5min":2.1560000000000001,"15min":1.518},"min":{"1min":0.33600000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":11.999000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.81399999999999995}]},{"osd":6,"last update":"Mon Feb 12 17:00:25 2024","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.4430000000000001,"15min":1.7050000000000001},"min":{"1min":0.377,"5min":0.36399999999999999,"15min":0.31},"max":{"1min":6.6550000000000002,"5min":16.739000000000001,"15min":17.782},"last":0.73099999999999998},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.2879999999999998,"15min":1.476},"min":{"1min":0.39600000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":8.7850000000000001,"5min":13.446999999999999,"15min":13.446999999999999},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:59:59 2024","interfaces":[{"interface":"back","average":{"1min":2.8330000000000002,"5min":2.2290000000000001,"15min":1.5149999999999999},"min":{"1min":0.45500000000000002,"5min":0.36299999999999999,"15min":0.32600000000000001},"max":{"1min":8.8520000000000003,"5min":13.473000000000001,"15min":13.473000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":3.387,"5min":2.3340000000000001,"15min":1.546},"min":{"1min":0.36199999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":16.768000000000001,"5min":16.768000000000001,"15min":16.768000000000001},"last":0.68799999999999994}]}]},{"osd":3,"up_from":0,"seq":111669149853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953625,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306976,"kb_used_data":1560,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438176,"statfs":{"total":95995035648,"available":95680692224,"internally_reserved":0,"allocated":1597440,"data_stored":866951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:00:39 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":1.9019999999999999,"15min":1.3029999999999999},"min":{"1min":0.214,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":6.4169999999999998,"5min":9.5090000000000003,"15min":38.179000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":3.3639999999999999,"5min":2.319,"15min":1.4470000000000001},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":6.7939999999999996,"5min":20.244,"15min":38.151000000000003},"last":0.66400000000000003}]},{"osd":1,"last update":"Mon Feb 12 17:00:39 2024","interfaces":[{"interface":"back","average":{"1min":3.0830000000000002,"5min":2.2389999999999999,"15min":1.27},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":10.266999999999999,"5min":11.077999999999999,"15min":11.077999999999999},"last":0.503},{"interface":"front","average":{"1min":2.202,"5min":2.0840000000000001,"15min":1.23},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.0069999999999997,"5min":9.5579999999999998,"15min":9.5579999999999998},"last":0.68500000000000005}]},{"osd":2,"last update":"Mon Feb 12 17:00:39 2024","interfaces":[{"interface":"back","average":{"1min":1.972,"5min":2.1110000000000002,"15min":1.254},"min":{"1min":0.26400000000000001,"5min":0.252,"15min":0.22500000000000001},"max":{"1min":6.4699999999999998,"5min":9.5790000000000006,"15min":9.6349999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":3.1150000000000002,"5min":2.2869999999999999,"15min":1.353},"min":{"1min":0.245,"5min":0.224,"15min":0.224},"max":{"1min":6.8239999999999998,"5min":10.106,"15min":10.106},"last":0.73499999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:00:39 2024","interfaces":[{"interface":"back","average":{"1min":3.5649999999999999,"5min":2.4660000000000002,"15min":1.6870000000000001},"min":{"1min":0.317,"5min":0.28899999999999998,"15min":0.26700000000000002},"max":{"1min":6.8479999999999999,"5min":20.271000000000001,"15min":38.170999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":3.6179999999999999,"5min":2.3359999999999999,"15min":1.5429999999999999},"min":{"1min":0.34799999999999998,"5min":0.32100000000000001,"15min":0.32000000000000001},"max":{"1min":10.303000000000001,"5min":10.865,"15min":38.311999999999998},"last":0.64000000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:00:04 2024","interfaces":[{"interface":"back","average":{"1min":6.6989999999999998,"5min":3.9649999999999999,"15min":1.589},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":42.420000000000002,"5min":79.573999999999998,"15min":79.573999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":7.8550000000000004,"5min":3.964,"15min":1.575},"min":{"1min":0.622,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":45.204999999999998,"5min":80.497,"15min":80.497},"last":0.35399999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:00:35 2024","interfaces":[{"interface":"back","average":{"1min":3.2330000000000001,"5min":2.653,"15min":1.571},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":9.1950000000000003,"5min":27.678000000000001,"15min":104.907},"last":0.40300000000000002},{"interface":"front","average":{"1min":4.1619999999999999,"5min":3.113,"15min":1.7370000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.26500000000000001},"max":{"1min":10.336,"5min":15.593,"15min":105.395},"last":0.376}]},{"osd":7,"last update":"Mon Feb 12 17:00:48 2024","interfaces":[{"interface":"back","average":{"1min":3.8260000000000001,"5min":2.7029999999999998,"15min":1.2310000000000001},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":10.786,"5min":12.311999999999999,"15min":12.311999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":3.6459999999999999,"5min":3.1899999999999999,"15min":1.4410000000000001},"min":{"1min":0.34200000000000003,"5min":0.33500000000000002,"15min":0.28799999999999998},"max":{"1min":12.303000000000001,"5min":27.773,"15min":27.773},"last":0.44400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757397,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311112,"kb_used_data":1600,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434040,"statfs":{"total":95995035648,"available":95676456960,"internally_reserved":0,"allocated":1638400,"data_stored":868076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,16],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.8050000000000002,"5min":1.72,"15min":1.1839999999999999},"min":{"1min":0.29399999999999998,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":8.7530000000000001,"5min":8.7530000000000001,"15min":36.880000000000003},"last":0.315},{"interface":"front","average":{"1min":3.4359999999999999,"5min":2.411,"15min":1.417},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.9779999999999998,"5min":10.752000000000001,"15min":36.954000000000001},"last":0.438}]},{"osd":1,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":2.1699999999999999,"5min":2.198,"15min":1.202},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.8380000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":3.1659999999999999,"5min":2.3490000000000002,"15min":1.327},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9459999999999997,"5min":16.712,"15min":16.712},"last":0.39400000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.2629999999999999,"5min":2.5009999999999999,"15min":1.399},"min":{"1min":0.40000000000000002,"5min":0.247,"15min":0.23499999999999999},"max":{"1min":8.8019999999999996,"5min":16.940999999999999,"15min":18.218},"last":0.34899999999999998},{"interface":"front","average":{"1min":2.3879999999999999,"5min":2.0960000000000001,"15min":1.2889999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":6.819,"5min":17.103999999999999,"15min":18.327000000000002},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":1.9319999999999999,"15min":1.321},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.8230000000000004,"5min":8.8230000000000004,"15min":18.251000000000001},"last":0.502},{"interface":"front","average":{"1min":2.3620000000000001,"5min":1.7490000000000001,"15min":1.2290000000000001},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.7720000000000002,"5min":7.048,"15min":20.263000000000002},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":4.2220000000000004,"5min":2.0569999999999999,"15min":0.98299999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":10.021000000000001,"5min":11.880000000000001,"15min":11.880000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":4.7709999999999999,"5min":2.601,"15min":1.161},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":0.44900000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:00:29 2024","interfaces":[{"interface":"back","average":{"1min":4.742,"5min":2.415,"15min":1.1759999999999999},"min":{"1min":0.34999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":12.281000000000001,"5min":12.281000000000001,"15min":12.281000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":4.8710000000000004,"5min":2.6789999999999998,"15min":1.2350000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26700000000000002},"max":{"1min":14.323,"5min":22.035,"15min":22.035},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:00:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":1.889,"15min":0.97899999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":6.7999999999999998,"5min":9.484,"15min":9.484},"last":0.57599999999999996},{"interface":"front","average":{"1min":3.7490000000000001,"5min":2.04,"15min":1.044},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":12.253,"5min":12.253,"15min":12.253},"last":0.48699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":559480832,"data_stored":559480832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":286982144,"data_stored":286982144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:00:57.301 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T17:00:57.301 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:00:57.301 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-12T17:00:57.301 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T17:00:57.301 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T17:00:57.301 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-12T17:00:57.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.204197+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e301: 8 total, 8 up, 5 in 2024-02-12T17:00:57.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.208215+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:00:57.415 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.212945+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:00:57.415 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.566562+0000 mgr.y (mgr.24481) 432 : cluster [DBG] pgmap v562: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 888 B/s rd, 0 B/s wr, 213 op/s; 122 KiB/s, 1 objects/s recovering 2024-02-12T17:00:57.415 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:56.933455+0000 mon.a (mon.0) 1225 : audit [DBG] from='client.? v1:172.21.15.33:0/3369269599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:57.415 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.204197+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e301: 8 total, 8 up, 5 in 2024-02-12T17:00:57.415 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.208215+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:00:57.415 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.212945+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:00:57.415 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.566562+0000 mgr.y (mgr.24481) 432 : cluster [DBG] pgmap v562: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 888 B/s rd, 0 B/s wr, 213 op/s; 122 KiB/s, 1 objects/s recovering 2024-02-12T17:00:57.415 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:56.933455+0000 mon.a (mon.0) 1225 : audit [DBG] from='client.? v1:172.21.15.33:0/3369269599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.204197+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e301: 8 total, 8 up, 5 in 2024-02-12T17:00:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.208215+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:00:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.212945+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:00:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.566562+0000 mgr.y (mgr.24481) 432 : cluster [DBG] pgmap v562: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 888 B/s rd, 0 B/s wr, 213 op/s; 122 KiB/s, 1 objects/s recovering 2024-02-12T17:00:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:56.933455+0000 mon.a (mon.0) 1225 : audit [DBG] from='client.? v1:172.21.15.33:0/3369269599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:00:57.577 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:00:58.266 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:00:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:58.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:00:58.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:58.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:00:58.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:00:58.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:00:58.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:00:58.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:00:58.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:00:58.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:00:58.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:00:58.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:00:58.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:00:58.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:00:58.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:00:58.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:00:58.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:00:58.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:00:58.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:00:58.400 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:58.453 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:58.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:58.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.823333+0000 osd.2 (osd.2) 151 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:00:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.824190+0000 osd.2 (osd.2) 152 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:00:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.889828+0000 osd.5 (osd.5) 131 : cluster [DBG] 6.1a deep-scrub starts 2024-02-12T17:00:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.892752+0000 osd.5 (osd.5) 132 : cluster [DBG] 6.1a deep-scrub ok 2024-02-12T17:00:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.977869+0000 osd.0 (osd.0) 129 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:00:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:56.986129+0000 osd.0 (osd.0) 130 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:00:58.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.823333+0000 osd.2 (osd.2) 151 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:00:58.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.824190+0000 osd.2 (osd.2) 152 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:00:58.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.889828+0000 osd.5 (osd.5) 131 : cluster [DBG] 6.1a deep-scrub starts 2024-02-12T17:00:58.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.892752+0000 osd.5 (osd.5) 132 : cluster [DBG] 6.1a deep-scrub ok 2024-02-12T17:00:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.977869+0000 osd.0 (osd.0) 129 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:00:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:56.986129+0000 osd.0 (osd.0) 130 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:00:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.823333+0000 osd.2 (osd.2) 151 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:00:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.824190+0000 osd.2 (osd.2) 152 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:00:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.889828+0000 osd.5 (osd.5) 131 : cluster [DBG] 6.1a deep-scrub starts 2024-02-12T17:00:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.892752+0000 osd.5 (osd.5) 132 : cluster [DBG] 6.1a deep-scrub ok 2024-02-12T17:00:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.977869+0000 osd.0 (osd.0) 129 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:00:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:56.986129+0000 osd.0 (osd.0) 130 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:00:58.564 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:00:58.591 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:58.789 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:00:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:58.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:00:58.892 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:00:58.917 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:59.041 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:00:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:59.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:00:59.145 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:00:59.175 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:59.314 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:00:59.315 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T17:00:59.315 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:00:59.378 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:00:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:59.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:00:59.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:00:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:57.926665+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:00:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:57.950280+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:00:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:58.016917+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:00:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:58.019809+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:00:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:58.175035+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.15 deep-scrub starts 2024-02-12T17:00:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:58.187733+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.15 deep-scrub ok 2024-02-12T17:00:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:58.567342+0000 mgr.y (mgr.24481) 433 : cluster [DBG] pgmap v563: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 168 op/s; 96 KiB/s, 1 objects/s recovering 2024-02-12T17:00:59.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:00:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.068213+0000 mon.c (mon.2) 162 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:00:59.511 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:57.926665+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:00:59.511 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:57.950280+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:00:59.511 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:58.016917+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:00:59.511 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:58.019809+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:00:59.511 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:58.175035+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.15 deep-scrub starts 2024-02-12T17:00:59.511 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:58.187733+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.15 deep-scrub ok 2024-02-12T17:00:59.511 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:58.567342+0000 mgr.y (mgr.24481) 433 : cluster [DBG] pgmap v563: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 168 op/s; 96 KiB/s, 1 objects/s recovering 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.068213+0000 mon.c (mon.2) 162 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:57.926665+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:57.950280+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:58.016917+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:58.019809+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:58.175035+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.15 deep-scrub starts 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:58.187733+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.15 deep-scrub ok 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:58.567342+0000 mgr.y (mgr.24481) 433 : cluster [DBG] pgmap v563: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 168 op/s; 96 KiB/s, 1 objects/s recovering 2024-02-12T17:00:59.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:00:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.068213+0000 mon.c (mon.2) 162 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:00:59.513 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:59.715 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:00:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:59.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:00:59.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:00:59.847 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:00:59.912 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:00:59.913 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:00:59.947 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:00:59.970 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:00:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:00:59.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:00.074 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:58.934677+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.10 deep-scrub starts 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:00:58.938800+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.10 deep-scrub ok 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.400146+0000 mon.c (mon.2) 163 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.400469+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.401254+0000 mon.c (mon.2) 164 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.401477+0000 mon.a (mon.0) 1227 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.402159+0000 mon.c (mon.2) 165 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.402390+0000 mon.a (mon.0) 1228 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.403117+0000 mon.c (mon.2) 166 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.403330+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T17:00:59.403927+0000 mgr.y (mgr.24481) 434 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 3256M 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.409472+0000 mon.a (mon.0) 1230 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.510271+0000 mon.c (mon.2) 167 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.510612+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.511403+0000 mon.c (mon.2) 168 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.511713+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.512535+0000 mon.c (mon.2) 169 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.512786+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.513511+0000 mon.c (mon.2) 170 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.513786+0000 mon.a (mon.0) 1234 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cephadm 2024-02-12T17:00:59.514524+0000 mgr.y (mgr.24481) 435 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 3257M 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.519421+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.521059+0000 mon.c (mon.2) 171 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.521789+0000 mon.c (mon.2) 172 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:01:00.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:00:59.527047+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:58.934677+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.10 deep-scrub starts 2024-02-12T17:01:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:00:58.938800+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.10 deep-scrub ok 2024-02-12T17:01:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.400146+0000 mon.c (mon.2) 163 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.400469+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.401254+0000 mon.c (mon.2) 164 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.401477+0000 mon.a (mon.0) 1227 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.402159+0000 mon.c (mon.2) 165 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.402390+0000 mon.a (mon.0) 1228 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.403117+0000 mon.c (mon.2) 166 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.403330+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T17:00:59.403927+0000 mgr.y (mgr.24481) 434 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 3256M 2024-02-12T17:01:00.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.409472+0000 mon.a (mon.0) 1230 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.510271+0000 mon.c (mon.2) 167 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.510612+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.511403+0000 mon.c (mon.2) 168 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:58.934677+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.10 deep-scrub starts 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:00:58.938800+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.10 deep-scrub ok 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.400146+0000 mon.c (mon.2) 163 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.400469+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.401254+0000 mon.c (mon.2) 164 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.401477+0000 mon.a (mon.0) 1227 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.402159+0000 mon.c (mon.2) 165 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.402390+0000 mon.a (mon.0) 1228 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.403117+0000 mon.c (mon.2) 166 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.403330+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T17:00:59.403927+0000 mgr.y (mgr.24481) 434 : cephadm [INF] Adjusting osd_memory_target on smithi094 to 3256M 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.409472+0000 mon.a (mon.0) 1230 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.510271+0000 mon.c (mon.2) 167 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.510612+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.511403+0000 mon.c (mon.2) 168 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.511713+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.512535+0000 mon.c (mon.2) 169 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.512786+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.513511+0000 mon.c (mon.2) 170 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.513786+0000 mon.a (mon.0) 1234 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cephadm 2024-02-12T17:00:59.514524+0000 mgr.y (mgr.24481) 435 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 3257M 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.519421+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.521059+0000 mon.c (mon.2) 171 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:01:00.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.521789+0000 mon.c (mon.2) 172 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:00:59.527047+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.511713+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.512535+0000 mon.c (mon.2) 169 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.512786+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.513511+0000 mon.c (mon.2) 170 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.513786+0000 mon.a (mon.0) 1234 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cephadm 2024-02-12T17:00:59.514524+0000 mgr.y (mgr.24481) 435 : cephadm [INF] Adjusting osd_memory_target on smithi033 to 3257M 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.519421+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.521059+0000 mon.c (mon.2) 171 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.521789+0000 mon.c (mon.2) 172 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:01:00.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:00:59.527047+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:00.553 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:00.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:00.657 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:00.687 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:00.707 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:00.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:00.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:00.845 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:00.866 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:00.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:00.970 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:00.971 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:01:01.000 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:01.179 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:01.181 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:01:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:00] "GET /metrics HTTP/1.1" 200 175954 "" "Prometheus/2.33.4" 2024-02-12T17:01:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:01.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:01.218 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_1' removed 2024-02-12T17:01:01.283 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:01.317 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:01.336 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:01.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:01.439 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:01.472 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:00.178206+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:01:01.472 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:00.181285+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:01:01.472 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:00.567762+0000 mgr.y (mgr.24481) 436 : cluster [DBG] pgmap v564: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 154 op/s; 88 KiB/s, 1 objects/s recovering 2024-02-12T17:01:01.472 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:00.852325+0000 mon.c (mon.2) 173 : audit [INF] from='client.? v1:172.21.15.33:0/3373443450' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:01:01.472 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:00.852663+0000 mon.a (mon.0) 1237 : audit [INF] from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:01:01.472 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:01.206907+0000 mon.a (mon.0) 1238 : audit [INF] from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:01:01.472 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:01.206982+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e302: 8 total, 8 up, 5 in 2024-02-12T17:01:01.473 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:01.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:00.178206+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:01:01.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:00.181285+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:01:01.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:00.567762+0000 mgr.y (mgr.24481) 436 : cluster [DBG] pgmap v564: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 154 op/s; 88 KiB/s, 1 objects/s recovering 2024-02-12T17:01:01.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:00.852325+0000 mon.c (mon.2) 173 : audit [INF] from='client.? v1:172.21.15.33:0/3373443450' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:01:01.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:00.852663+0000 mon.a (mon.0) 1237 : audit [INF] from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:01:01.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:01.206907+0000 mon.a (mon.0) 1238 : audit [INF] from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:01:01.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:01.206982+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e302: 8 total, 8 up, 5 in 2024-02-12T17:01:01.540 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:00.178206+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:01:01.541 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:00.181285+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:01:01.541 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:00.567762+0000 mgr.y (mgr.24481) 436 : cluster [DBG] pgmap v564: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 5.7 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 154 op/s; 88 KiB/s, 1 objects/s recovering 2024-02-12T17:01:01.541 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:00.852325+0000 mon.c (mon.2) 173 : audit [INF] from='client.? v1:172.21.15.33:0/3373443450' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:01:01.541 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:00.852663+0000 mon.a (mon.0) 1237 : audit [INF] from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:01:01.541 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:01.206907+0000 mon.a (mon.0) 1238 : audit [INF] from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:01:01.541 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:01.206982+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e302: 8 total, 8 up, 5 in 2024-02-12T17:01:01.599 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:01.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:01.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T17:01:01.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:01.731 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:01.750 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:01.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:01.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:01.854 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:01.882 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:01.910 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:01.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:02.014 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:02.042 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:02.060 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:02.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:02.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:02.193 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:02.211 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:02.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:02.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:02.360 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:02.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:02.464 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:02.491 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:02.510 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:02.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:02.549 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T17:01:02.550 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T17:01:02.550 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-02-12T17:01:02.550 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-02-12T17:01:02.614 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:02.641 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:02.133880+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:01:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:02.137339+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:01:02.741 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:01:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:01:02.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:02.133880+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:01:02.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:02.137339+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:01:02.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:02.133880+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:01:02.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:02.137339+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:01:03.291 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:03.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:03.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:03.450 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:03.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:03.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:01.788034+0000 osd.2 (osd.2) 153 : cluster [DBG] 2.e scrub starts 2024-02-12T17:01:03.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:01.789450+0000 osd.2 (osd.2) 154 : cluster [DBG] 2.e scrub ok 2024-02-12T17:01:03.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:02.568117+0000 mgr.y (mgr.24481) 437 : cluster [DBG] pgmap v566: 122 pgs: 122 active+clean; 5.7 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T17:01:03.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:02.950870+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:01:03.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:02.953885+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:01:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:01.788034+0000 osd.2 (osd.2) 153 : cluster [DBG] 2.e scrub starts 2024-02-12T17:01:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:01.789450+0000 osd.2 (osd.2) 154 : cluster [DBG] 2.e scrub ok 2024-02-12T17:01:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:02.568117+0000 mgr.y (mgr.24481) 437 : cluster [DBG] pgmap v566: 122 pgs: 122 active+clean; 5.7 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T17:01:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:02.950870+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:01:03.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:02.953885+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:01:03.554 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:03.589 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:03.609 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:03.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:03.713 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:01.788034+0000 osd.2 (osd.2) 153 : cluster [DBG] 2.e scrub starts 2024-02-12T17:01:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:01.789450+0000 osd.2 (osd.2) 154 : cluster [DBG] 2.e scrub ok 2024-02-12T17:01:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:02.568117+0000 mgr.y (mgr.24481) 437 : cluster [DBG] pgmap v566: 122 pgs: 122 active+clean; 5.7 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T17:01:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:02.950870+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:01:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:02.953885+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:01:03.746 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:03.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:03.865 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:03.865 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:03.871 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:03.907 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:03.923 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:04.162 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:04.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:04.267 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:04.297 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:04.424 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:04.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:04.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:01:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:01:04.511 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:01:04.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:04.579 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:04.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:04.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:02.857608+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:02.858407+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:03.490188+0000 mon.a (mon.0) 1240 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:03.490238+0000 mon.a (mon.0) 1241 : cluster [INF] Cluster is now healthy 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:04.024063+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.33:0/2919593306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:04.024372+0000 mon.a (mon.0) 1242 : audit [INF] from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:04.057557+0000 mgr.y (mgr.24481) 438 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:02.857608+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:01:04.710 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:02.858407+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:01:04.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:03.490188+0000 mon.a (mon.0) 1240 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:01:04.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:03.490238+0000 mon.a (mon.0) 1241 : cluster [INF] Cluster is now healthy 2024-02-12T17:01:04.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:04.024063+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.33:0/2919593306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:04.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:04.024372+0000 mon.a (mon.0) 1242 : audit [INF] from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:04.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:04.057557+0000 mgr.y (mgr.24481) 438 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:04.713 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:04.966 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7] out_osds: [3, 1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:01:04.967 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:01:04.967 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-12T17:01:04.967 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2024-02-12T17:01:04.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 2 0 2024-02-12T17:01:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:02.857608+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:01:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:02.858407+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:01:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:03.490188+0000 mon.a (mon.0) 1240 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:01:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:03.490238+0000 mon.a (mon.0) 1241 : cluster [INF] Cluster is now healthy 2024-02-12T17:01:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:04.024063+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.33:0/2919593306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:04.024372+0000 mon.a (mon.0) 1242 : audit [INF] from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:04.057557+0000 mgr.y (mgr.24481) 438 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:05.467 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:05.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:05.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:05.604 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:05.626 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:05.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:05.730 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:05.759 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:04.066284+0000 osd.7 (osd.7) 133 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:01:05.759 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:04.085822+0000 osd.7 (osd.7) 134 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:01:05.759 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:04.495888+0000 mon.a (mon.0) 1243 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:05.759 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:04.503584+0000 mon.a (mon.0) 1244 : audit [INF] from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:01:05.759 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:04.503652+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e303: 8 total, 8 up, 5 in 2024-02-12T17:01:05.759 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:04.568461+0000 mgr.y (mgr.24481) 439 : cluster [DBG] pgmap v568: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:05.759 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:04.066284+0000 osd.7 (osd.7) 133 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:01:05.759 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:04.085822+0000 osd.7 (osd.7) 134 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:01:05.760 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:04.495888+0000 mon.a (mon.0) 1243 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:05.760 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:04.503584+0000 mon.a (mon.0) 1244 : audit [INF] from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:01:05.760 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:04.503652+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e303: 8 total, 8 up, 5 in 2024-02-12T17:01:05.760 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:04.568461+0000 mgr.y (mgr.24481) 439 : cluster [DBG] pgmap v568: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:05.761 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:04.066284+0000 osd.7 (osd.7) 133 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:01:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:04.085822+0000 osd.7 (osd.7) 134 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:01:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:04.495888+0000 mon.a (mon.0) 1243 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:04.503584+0000 mon.a (mon.0) 1244 : audit [INF] from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:01:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:04.503652+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e303: 8 total, 8 up, 5 in 2024-02-12T17:01:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:04.568461+0000 mgr.y (mgr.24481) 439 : cluster [DBG] pgmap v568: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:06.139 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:06.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:06.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:06.287 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:06.309 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:06.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:06.412 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:06.447 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:07.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:06.569016+0000 mgr.y (mgr.24481) 440 : cluster [DBG] pgmap v569: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T17:01:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:06.569016+0000 mgr.y (mgr.24481) 440 : cluster [DBG] pgmap v569: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T17:01:07.318 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:01:07.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:06.569016+0000 mgr.y (mgr.24481) 440 : cluster [DBG] pgmap v569: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-12T17:01:07.700 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:07.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:07.808 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:07.842 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:07.865 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:07.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:07.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:07.969 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:08.000 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:08.149 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:08.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:08.185 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:01:08.202 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:01:08.219 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:01:08.236 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:01:08.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:08.254 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:01:08.272 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:01:08.287 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:08.291 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:01:08.308 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:08.309 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:01:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:01:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:01:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:01:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:01:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:01:08.310 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:01:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:01:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:01:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:08.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:08.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:08.443 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:08.467 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:08.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:08.478 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:01:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:01:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:01:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:01:08.479 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:01:08.571 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:08.600 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:08.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:07.857632+0000 mon.a (mon.0) 1246 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:08.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:08.160633+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? v1:172.21.15.33:0/2125596439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:07.857632+0000 mon.a (mon.0) 1246 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:08.160633+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? v1:172.21.15.33:0/2125596439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:09.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:07.857632+0000 mon.a (mon.0) 1246 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:09.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:08.160633+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? v1:172.21.15.33:0/2125596439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:09.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T17:01:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:09.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:09.553 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:09.580 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:10.165 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:08.569656+0000 mgr.y (mgr.24481) 441 : cluster [DBG] pgmap v570: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:10.165 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:08.569656+0000 mgr.y (mgr.24481) 441 : cluster [DBG] pgmap v570: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:08.569656+0000 mgr.y (mgr.24481) 441 : cluster [DBG] pgmap v570: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:10.586 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:10.586 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:11.173 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:01:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:10] "GET /metrics HTTP/1.1" 200 173585 "" "Prometheus/2.33.4" 2024-02-12T17:01:12.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:10.570177+0000 mgr.y (mgr.24481) 442 : cluster [DBG] pgmap v571: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:12.113 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:10.570177+0000 mgr.y (mgr.24481) 442 : cluster [DBG] pgmap v571: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:10.570177+0000 mgr.y (mgr.24481) 442 : cluster [DBG] pgmap v571: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:01:12.616 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:01:12.869 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:01:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:01:13.031 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_2' created 2024-02-12T17:01:13.233 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:13.235 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:12.159986+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? v1:172.21.15.33:0/2376949278' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-12T17:01:13.236 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:12.159986+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? v1:172.21.15.33:0/2376949278' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-12T17:01:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:13.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:13.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:12.159986+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? v1:172.21.15.33:0/2376949278' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-12T17:01:13.337 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:13.371 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:13.393 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:13.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:13.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:13.535 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:13.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:13.754 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:13.787 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:13.808 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:13.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:13.912 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:13.940 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:13.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:14.063 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:12.193088+0000 osd.6 (osd.6) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:14.063 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:12.193148+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:14.063 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:12.570519+0000 mgr.y (mgr.24481) 443 : cluster [DBG] pgmap v572: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:01:14.063 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:12.874909+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.33:0/2376949278' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-12T17:01:14.063 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:12.875004+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e304: 8 total, 8 up, 5 in 2024-02-12T17:01:14.064 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:14.092 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:14.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:12.193088+0000 osd.6 (osd.6) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:14.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:12.193148+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:14.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:12.570519+0000 mgr.y (mgr.24481) 443 : cluster [DBG] pgmap v572: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:01:14.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:12.874909+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.33:0/2376949278' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-12T17:01:14.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:12.875004+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e304: 8 total, 8 up, 5 in 2024-02-12T17:01:14.137 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:12.193088+0000 osd.6 (osd.6) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:14.137 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:12.193148+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:14.137 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:12.570519+0000 mgr.y (mgr.24481) 443 : cluster [DBG] pgmap v572: 122 pgs: 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:01:14.137 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:12.874909+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.33:0/2376949278' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-12T17:01:14.137 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:12.875004+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e304: 8 total, 8 up, 5 in 2024-02-12T17:01:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:14.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:14.247 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:14.250 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:14.296 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:14.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:14.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:14.427 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:01:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:01:14.429 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:14.449 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:14.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:14.553 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:14.584 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:14.733 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:14.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:14.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:14.867 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:14.899 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:01:15.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:13.876067+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e305: 8 total, 8 up, 5 in 2024-02-12T17:01:15.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:14.030147+0000 mon.c (mon.2) 176 : audit [INF] from='client.? v1:172.21.15.33:0/3562231707' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:15.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:14.030516+0000 mon.a (mon.0) 1251 : audit [INF] from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:15.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:14.061686+0000 mgr.y (mgr.24481) 444 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:15.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:13.876067+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e305: 8 total, 8 up, 5 in 2024-02-12T17:01:15.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:14.030147+0000 mon.c (mon.2) 176 : audit [INF] from='client.? v1:172.21.15.33:0/3562231707' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:15.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:14.030516+0000 mon.a (mon.0) 1251 : audit [INF] from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:15.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:14.061686+0000 mgr.y (mgr.24481) 444 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:15.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:13.876067+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e305: 8 total, 8 up, 5 in 2024-02-12T17:01:15.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:14.030147+0000 mon.c (mon.2) 176 : audit [INF] from='client.? v1:172.21.15.33:0/3562231707' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:15.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:14.030516+0000 mon.a (mon.0) 1251 : audit [INF] from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:15.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:14.061686+0000 mgr.y (mgr.24481) 444 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:15.395 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:15.747 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:15.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:15.851 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:15.883 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:15.908 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.2 primary-affinity to 0 (802) 2024-02-12T17:01:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:14.570881+0000 mgr.y (mgr.24481) 445 : cluster [DBG] pgmap v575: 138 pgs: 16 creating+peering, 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:14.879658+0000 mon.a (mon.0) 1252 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:14.893939+0000 mon.a (mon.0) 1253 : audit [INF] from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:14.894005+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e306: 8 total, 8 up, 5 in 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:15.178489+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.33:0/1930140509' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:15.178913+0000 mon.a (mon.0) 1255 : audit [INF] from='client.25303 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:14.570881+0000 mgr.y (mgr.24481) 445 : cluster [DBG] pgmap v575: 138 pgs: 16 creating+peering, 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:14.879658+0000 mon.a (mon.0) 1252 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:14.893939+0000 mon.a (mon.0) 1253 : audit [INF] from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:14.894005+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e306: 8 total, 8 up, 5 in 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:15.178489+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.33:0/1930140509' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-02-12T17:01:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:15.178913+0000 mon.a (mon.0) 1255 : audit [INF] from='client.25303 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-02-12T17:01:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:14.570881+0000 mgr.y (mgr.24481) 445 : cluster [DBG] pgmap v575: 138 pgs: 16 creating+peering, 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:14.879658+0000 mon.a (mon.0) 1252 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:14.893939+0000 mon.a (mon.0) 1253 : audit [INF] from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:01:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:14.894005+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e306: 8 total, 8 up, 5 in 2024-02-12T17:01:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:15.178489+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.33:0/1930140509' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-02-12T17:01:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:15.178913+0000 mon.a (mon.0) 1255 : audit [INF] from='client.25303 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-02-12T17:01:16.345 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:16.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:16.451 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:16.481 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:16.500 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:16.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:16.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:16.634 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:16.673 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:16.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:16.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-02-12T17:01:16.778 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:16.813 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:16.836 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:16.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:16.894 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:15.893207+0000 mon.a (mon.0) 1256 : audit [INF] from='client.25303 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-02-12T17:01:16.894 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:15.893296+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e307: 8 total, 8 up, 5 in 2024-02-12T17:01:16.939 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:16.975 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:17.187 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:15.893207+0000 mon.a (mon.0) 1256 : audit [INF] from='client.25303 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-02-12T17:01:17.188 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:15.893296+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e307: 8 total, 8 up, 5 in 2024-02-12T17:01:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:15.893207+0000 mon.a (mon.0) 1256 : audit [INF] from='client.25303 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-02-12T17:01:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:15.893296+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e307: 8 total, 8 up, 5 in 2024-02-12T17:01:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:16.571565+0000 mgr.y (mgr.24481) 446 : cluster [DBG] pgmap v578: 138 pgs: 16 creating+peering, 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:16.899776+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e308: 8 total, 8 up, 5 in 2024-02-12T17:01:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:17.484041+0000 mon.a (mon.0) 1259 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:16.571565+0000 mgr.y (mgr.24481) 446 : cluster [DBG] pgmap v578: 138 pgs: 16 creating+peering, 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:16.899776+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e308: 8 total, 8 up, 5 in 2024-02-12T17:01:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:17.484041+0000 mon.a (mon.0) 1259 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:16.571565+0000 mgr.y (mgr.24481) 446 : cluster [DBG] pgmap v578: 138 pgs: 16 creating+peering, 122 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:16.899776+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e308: 8 total, 8 up, 5 in 2024-02-12T17:01:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:17.484041+0000 mon.a (mon.0) 1259 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:01:18.371 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:01:19.066 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:01:19.083 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:01:19.099 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:01:19.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:01:19.134 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:01:19.151 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:01:19.169 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:01:19.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:01:19.189 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:01:19.189 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:01:19.189 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:01:19.189 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:01:19.189 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:01:19.189 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:01:19.189 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:01:19.189 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:01:19.289 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:01:19.289 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:01:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:18.572122+0000 mgr.y (mgr.24481) 447 : cluster [DBG] pgmap v580: 138 pgs: 6 creating+peering, 132 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:19.047580+0000 mon.a (mon.0) 1260 : audit [DBG] from='client.? v1:172.21.15.33:0/3015201335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:19.430971+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:19.431052+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:20.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:18.572122+0000 mgr.y (mgr.24481) 447 : cluster [DBG] pgmap v580: 138 pgs: 6 creating+peering, 132 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:20.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:19.047580+0000 mon.a (mon.0) 1260 : audit [DBG] from='client.? v1:172.21.15.33:0/3015201335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:20.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:19.430971+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:20.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:19.431052+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:18.572122+0000 mgr.y (mgr.24481) 447 : cluster [DBG] pgmap v580: 138 pgs: 6 creating+peering, 132 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:19.047580+0000 mon.a (mon.0) 1260 : audit [DBG] from='client.? v1:172.21.15.33:0/3015201335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:19.430971+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:19.431052+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:20.530 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:20.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:20.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:20.669 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:20.816 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:20.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:20.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T17:01:20.923 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:20.953 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:20.973 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:20.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:21.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:21.105 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:21.124 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:21.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:21.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:21.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:19.903442+0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e309: 8 total, 8 up, 5 in 2024-02-12T17:01:21.255 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:21.273 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:21.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:21.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:01:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:20] "GET /metrics HTTP/1.1" 200 175960 "" "Prometheus/2.33.4" 2024-02-12T17:01:21.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:19.903442+0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e309: 8 total, 8 up, 5 in 2024-02-12T17:01:21.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:19.903442+0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e309: 8 total, 8 up, 5 in 2024-02-12T17:01:21.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:21.385 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:21.386 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:21.402 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:21.421 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:21.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:21.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:21.551 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:21.569 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:21.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:21.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:21.699 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:21.717 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:21.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:21.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:21.847 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:21.864 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:21.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:21.969 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:21.972 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:20.572547+0000 mgr.y (mgr.24481) 448 : cluster [DBG] pgmap v582: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:21.972 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:20.572547+0000 mgr.y (mgr.24481) 448 : cluster [DBG] pgmap v582: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:21.999 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:22.052 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:22.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:22.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:22.181 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:20.572547+0000 mgr.y (mgr.24481) 448 : cluster [DBG] pgmap v582: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:22.183 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:22.202 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:22.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:22.305 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:22.332 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:22.552 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:01:22.581 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:22.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:22.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:22.716 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:22.965 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:22.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:01:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:01:23.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:23.102 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:23.123 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:23.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:23.227 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:23.257 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:23.456 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:23.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:23.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:23.593 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:23.615 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:23.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:23.718 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:23.751 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:22.572916+0000 mgr.y (mgr.24481) 449 : cluster [DBG] pgmap v583: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:22.572916+0000 mgr.y (mgr.24481) 449 : cluster [DBG] pgmap v583: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:22.572916+0000 mgr.y (mgr.24481) 449 : cluster [DBG] pgmap v583: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:24.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:01:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:01:24.634 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:24.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:24.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:24.774 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:24.788 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:24.793 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:24.795 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:24.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:24.899 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:24.935 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:24.936 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-02-12T17:01:24.936 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:01:25.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:24.065820+0000 mgr.y (mgr.24481) 450 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:25.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:24.716718+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.33:0/1723498737' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:01:25.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:24.820110+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.33:0/229645218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:01:25.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:24.065820+0000 mgr.y (mgr.24481) 450 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:25.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:24.716718+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.33:0/1723498737' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:01:25.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:24.820110+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.33:0/229645218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:01:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:24.065820+0000 mgr.y (mgr.24481) 450 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:24.716718+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.33:0/1723498737' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:01:25.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:24.820110+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.33:0/229645218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:01:25.787 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:25.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:25.891 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:25.921 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:24.573721+0000 mgr.y (mgr.24481) 451 : cluster [DBG] pgmap v584: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:24.915229+0000 mon.a (mon.0) 1264 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:24.930520+0000 mon.a (mon.0) 1265 : audit [INF] from='client.? v1:172.21.15.33:0/1723498737' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:01:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:24.930596+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? v1:172.21.15.33:0/229645218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:01:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:24.930637+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e310: 8 total, 8 up, 5 in 2024-02-12T17:01:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:25.156860+0000 osd.4 (osd.4) 112 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:01:26.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:25.160211+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:01:26.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:24.573721+0000 mgr.y (mgr.24481) 451 : cluster [DBG] pgmap v584: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:26.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:24.915229+0000 mon.a (mon.0) 1264 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:26.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:24.930520+0000 mon.a (mon.0) 1265 : audit [INF] from='client.? v1:172.21.15.33:0/1723498737' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:01:26.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:24.930596+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? v1:172.21.15.33:0/229645218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:01:26.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:24.930637+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e310: 8 total, 8 up, 5 in 2024-02-12T17:01:26.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:25.156860+0000 osd.4 (osd.4) 112 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:01:26.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:25.160211+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:01:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:24.573721+0000 mgr.y (mgr.24481) 451 : cluster [DBG] pgmap v584: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-12T17:01:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:24.915229+0000 mon.a (mon.0) 1264 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:24.930520+0000 mon.a (mon.0) 1265 : audit [INF] from='client.? v1:172.21.15.33:0/1723498737' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:01:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:24.930596+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? v1:172.21.15.33:0/229645218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:01:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:24.930637+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e310: 8 total, 8 up, 5 in 2024-02-12T17:01:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:25.156860+0000 osd.4 (osd.4) 112 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:01:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:25.160211+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:01:26.368 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7] out_osds: [3, 1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:01:26.368 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:01:26.368 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-12T17:01:26.368 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T17:01:27.178 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:25.444361+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:27.178 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:25.444460+0000 osd.4 (osd.4) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:27.178 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:25.867981+0000 osd.7 (osd.7) 135 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:01:27.178 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:25.870911+0000 osd.7 (osd.7) 136 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:01:27.178 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:25.934193+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e311: 8 total, 8 up, 5 in 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:26.042748+0000 osd.0 (osd.0) 135 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:26.047522+0000 osd.0 (osd.0) 136 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:25.444361+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:25.444460+0000 osd.4 (osd.4) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:25.867981+0000 osd.7 (osd.7) 135 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:25.870911+0000 osd.7 (osd.7) 136 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:25.934193+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e311: 8 total, 8 up, 5 in 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:26.042748+0000 osd.0 (osd.0) 135 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:01:27.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:26.047522+0000 osd.0 (osd.0) 136 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:01:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:25.444361+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:25.444460+0000 osd.4 (osd.4) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:25.867981+0000 osd.7 (osd.7) 135 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:01:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:25.870911+0000 osd.7 (osd.7) 136 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:01:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:25.934193+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e311: 8 total, 8 up, 5 in 2024-02-12T17:01:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:26.042748+0000 osd.0 (osd.0) 135 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:01:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:26.047522+0000 osd.0 (osd.0) 136 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:01:27.473 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:27.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:27.577 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:27.610 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:27.632 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:27.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:27.736 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:27.771 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:28.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:26.574143+0000 mgr.y (mgr.24481) 452 : cluster [DBG] pgmap v587: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 767 B/s rd, 0 op/s 2024-02-12T17:01:28.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:26.801007+0000 osd.5 (osd.5) 139 : cluster [DBG] 6.a scrub starts 2024-02-12T17:01:28.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:26.803899+0000 osd.5 (osd.5) 140 : cluster [DBG] 6.a scrub ok 2024-02-12T17:01:28.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:27.154258+0000 osd.4 (osd.4) 116 : cluster [DBG] 2.1a scrub starts 2024-02-12T17:01:28.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:27.157352+0000 osd.4 (osd.4) 117 : cluster [DBG] 2.1a scrub ok 2024-02-12T17:01:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:26.574143+0000 mgr.y (mgr.24481) 452 : cluster [DBG] pgmap v587: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 767 B/s rd, 0 op/s 2024-02-12T17:01:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:26.801007+0000 osd.5 (osd.5) 139 : cluster [DBG] 6.a scrub starts 2024-02-12T17:01:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:26.803899+0000 osd.5 (osd.5) 140 : cluster [DBG] 6.a scrub ok 2024-02-12T17:01:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:27.154258+0000 osd.4 (osd.4) 116 : cluster [DBG] 2.1a scrub starts 2024-02-12T17:01:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:27.157352+0000 osd.4 (osd.4) 117 : cluster [DBG] 2.1a scrub ok 2024-02-12T17:01:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:26.574143+0000 mgr.y (mgr.24481) 452 : cluster [DBG] pgmap v587: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 767 B/s rd, 0 op/s 2024-02-12T17:01:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:26.801007+0000 osd.5 (osd.5) 139 : cluster [DBG] 6.a scrub starts 2024-02-12T17:01:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:26.803899+0000 osd.5 (osd.5) 140 : cluster [DBG] 6.a scrub ok 2024-02-12T17:01:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:27.154258+0000 osd.4 (osd.4) 116 : cluster [DBG] 2.1a scrub starts 2024-02-12T17:01:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:27.157352+0000 osd.4 (osd.4) 117 : cluster [DBG] 2.1a scrub ok 2024-02-12T17:01:28.429 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:01:28.722 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:28.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:28.826 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:28.858 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:28.879 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:28.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:28.983 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:29.000 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:27.855997+0000 osd.7 (osd.7) 137 : cluster [DBG] 2.b scrub starts 2024-02-12T17:01:29.000 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:27.858951+0000 osd.7 (osd.7) 138 : cluster [DBG] 2.b scrub ok 2024-02-12T17:01:29.000 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:28.092922+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-02-12T17:01:29.001 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:28.096704+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-02-12T17:01:29.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:27.855997+0000 osd.7 (osd.7) 137 : cluster [DBG] 2.b scrub starts 2024-02-12T17:01:29.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:27.858951+0000 osd.7 (osd.7) 138 : cluster [DBG] 2.b scrub ok 2024-02-12T17:01:29.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:28.092922+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-02-12T17:01:29.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:28.096704+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-02-12T17:01:29.016 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:29.215 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:29.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:29.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:27.855997+0000 osd.7 (osd.7) 137 : cluster [DBG] 2.b scrub starts 2024-02-12T17:01:29.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:27.858951+0000 osd.7 (osd.7) 138 : cluster [DBG] 2.b scrub ok 2024-02-12T17:01:29.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:28.092922+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-02-12T17:01:29.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:28.096704+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-02-12T17:01:29.244 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:01:29.244 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:01:29.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:29.351 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:29.439 DEBUG:teuthology.orchestra.run.smithi033:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-02-12T17:01:29.442 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T17:01:29.442 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T17:01:29.491 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:hints = 1 2024-02-12T17:01:29.491 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T17:01:29.491 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object prefix: benchmark_data_smithi033_131576 2024-02-12T17:01:29.491 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:01:29.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 0 0 0 0 0 0 - 0 2024-02-12T17:01:29.660 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:01:29.660 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:01:29.661 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:01:29.661 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:01:29.661 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:01:29.674 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:01:29.691 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:29.691 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:29.702 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:29.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:29.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:29.722 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:01:29.750 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:01:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:01:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:01:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:01:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:01:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:01:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:01:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:01:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:01:29.815 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:29.857 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:29.883 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:29.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:29.987 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:28.574667+0000 mgr.y (mgr.24481) 453 : cluster [DBG] pgmap v588: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:01:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:28.877256+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.b scrub starts 2024-02-12T17:01:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:28.912868+0000 osd.7 (osd.7) 140 : cluster [DBG] 6.b scrub ok 2024-02-12T17:01:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:29.150302+0000 osd.4 (osd.4) 118 : cluster [DBG] 5.c scrub starts 2024-02-12T17:01:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:29.153214+0000 osd.4 (osd.4) 119 : cluster [DBG] 5.c scrub ok 2024-02-12T17:01:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:29.475802+0000 mon.c (mon.2) 177 : audit [DBG] from='client.? v1:172.21.15.33:0/2507013493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:28.574667+0000 mgr.y (mgr.24481) 453 : cluster [DBG] pgmap v588: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:01:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:28.877256+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.b scrub starts 2024-02-12T17:01:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:28.912868+0000 osd.7 (osd.7) 140 : cluster [DBG] 6.b scrub ok 2024-02-12T17:01:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:29.150302+0000 osd.4 (osd.4) 118 : cluster [DBG] 5.c scrub starts 2024-02-12T17:01:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:29.153214+0000 osd.4 (osd.4) 119 : cluster [DBG] 5.c scrub ok 2024-02-12T17:01:30.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:29.475802+0000 mon.c (mon.2) 177 : audit [DBG] from='client.? v1:172.21.15.33:0/2507013493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:30.035 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:28.574667+0000 mgr.y (mgr.24481) 453 : cluster [DBG] pgmap v588: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:01:30.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:28.877256+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.b scrub starts 2024-02-12T17:01:30.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:28.912868+0000 osd.7 (osd.7) 140 : cluster [DBG] 6.b scrub ok 2024-02-12T17:01:30.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:29.150302+0000 osd.4 (osd.4) 118 : cluster [DBG] 5.c scrub starts 2024-02-12T17:01:30.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:29.153214+0000 osd.4 (osd.4) 119 : cluster [DBG] 5.c scrub ok 2024-02-12T17:01:30.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:29.475802+0000 mon.c (mon.2) 177 : audit [DBG] from='client.? v1:172.21.15.33:0/2507013493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:30.491 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 1 15 3710 3695 230.925 230.938 0.00820001 0.00431541 2024-02-12T17:01:30.495 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T17:01:31.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:29.883323+0000 osd.7 (osd.7) 141 : cluster [DBG] 2.a scrub starts 2024-02-12T17:01:31.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:29.888938+0000 osd.7 (osd.7) 142 : cluster [DBG] 2.a scrub ok 2024-02-12T17:01:31.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.047438+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:01:31.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.051520+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:01:31.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.189743+0000 osd.4 (osd.4) 120 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:01:31.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.193724+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:01:31.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:29.883323+0000 osd.7 (osd.7) 141 : cluster [DBG] 2.a scrub starts 2024-02-12T17:01:31.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:29.888938+0000 osd.7 (osd.7) 142 : cluster [DBG] 2.a scrub ok 2024-02-12T17:01:31.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.047438+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:01:31.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.051520+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:01:31.245 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.189743+0000 osd.4 (osd.4) 120 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:01:31.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.193724+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:01:31.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:29.883323+0000 osd.7 (osd.7) 141 : cluster [DBG] 2.a scrub starts 2024-02-12T17:01:31.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:29.888938+0000 osd.7 (osd.7) 142 : cluster [DBG] 2.a scrub ok 2024-02-12T17:01:31.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.047438+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:01:31.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.051520+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:01:31.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.189743+0000 osd.4 (osd.4) 120 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:01:31.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.193724+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:01:31.246 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:01:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:30] "GET /metrics HTTP/1.1" 200 175960 "" "Prometheus/2.33.4" 2024-02-12T17:01:31.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 2 15 7102 7087 221.454 212 0.00447269 0.00450511 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.575215+0000 mgr.y (mgr.24481) 454 : cluster [DBG] pgmap v589: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.809881+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.c scrub starts 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.811144+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.c scrub ok 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.860375+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:30.863526+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.575215+0000 mgr.y (mgr.24481) 454 : cluster [DBG] pgmap v589: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.809881+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.c scrub starts 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.811144+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.c scrub ok 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.860375+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:01:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:30.863526+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:01:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.575215+0000 mgr.y (mgr.24481) 454 : cluster [DBG] pgmap v589: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:01:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.809881+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.c scrub starts 2024-02-12T17:01:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.811144+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.c scrub ok 2024-02-12T17:01:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.860375+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:01:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:30.863526+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:01:32.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 3 16 10438 10422 217.109 208.438 0.00451746 0.00459914 2024-02-12T17:01:32.983 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:31.822269+0000 osd.5 (osd.5) 143 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:01:32.983 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:31.827625+0000 osd.5 (osd.5) 144 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:01:32.984 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:32.123069+0000 osd.4 (osd.4) 122 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:01:32.984 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:32.127586+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:01:32.984 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:01:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:01:33.002 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:33.002 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:33.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:33.107 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:33.152 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:31.822269+0000 osd.5 (osd.5) 143 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:01:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:31.827625+0000 osd.5 (osd.5) 144 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:01:33.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:32.123069+0000 osd.4 (osd.4) 122 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:01:33.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:32.127586+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:01:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:31.822269+0000 osd.5 (osd.5) 143 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:01:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:31.827625+0000 osd.5 (osd.5) 144 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:01:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:32.123069+0000 osd.4 (osd.4) 122 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:01:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:32.127586+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:01:33.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 4 16 13447 13431 209.843 188.062 0.00595119 0.00475846 2024-02-12T17:01:33.796 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:01:34.120 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:32.575644+0000 mgr.y (mgr.24481) 455 : cluster [DBG] pgmap v590: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:01:34.120 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:33.145073+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:01:34.120 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:33.149123+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:01:34.121 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:32.575644+0000 mgr.y (mgr.24481) 455 : cluster [DBG] pgmap v590: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:01:34.121 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:33.145073+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:01:34.121 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:33.149123+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:01:34.240 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:32.575644+0000 mgr.y (mgr.24481) 455 : cluster [DBG] pgmap v590: 138 pgs: 138 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:01:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:33.145073+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:01:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:33.149123+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:01:34.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:01:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:01:34.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:34.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:34.344 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:34.404 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:34.489 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:34.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 5 16 16041 16025 200.297 162.125 0.00390801 0.0049782 2024-02-12T17:01:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:34.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:34.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:34.671 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 311, 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2024-02-12T16:43:50.282468+0000", 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2024-02-12T17:01:25.925466+0000", 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2024-02-12T16:49:30.933575+0000", 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2024-02-12T16:59:08.202987+0000", 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5804032, 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "nodeep-scrub", 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2024-02-12T17:01:34.755 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 18, 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 9, 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 8, 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "luminous", 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "pacific", 2024-02-12T17:01:34.756 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "device_health_metrics", 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:47:33.056945+0000", 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:01:34.757 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "71", 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:01:34.758 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:01:34.759 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:01:34.760 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_devicehealth": {} 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.761 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".rgw.root", 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:37.647343+0000", 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:01:34.762 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "59", 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:01:34.763 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:01:34.764 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:01:34.765 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "iscsi", 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:38.076200+0000", 2024-02-12T17:01:34.766 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 3, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 3, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 3, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 3, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 3, 2024-02-12T17:01:34.767 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "60", 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:01:34.768 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:01:34.769 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:01:34.770 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 4, 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.log", 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:39.589565+0000", 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:01:34.771 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:01:34.772 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "61", 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:01:34.773 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:01:34.774 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:01:34.775 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 5, 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.control", 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:41.623856+0000", 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:01:34.776 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "5.16", 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 238, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 237, 2024-02-12T17:01:34.777 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 237, 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "238", 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "238", 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "236", 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:01:34.778 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.779 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.780 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 6, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.meta", 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:43.693806+0000", 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:01:34.781 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "65", 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:01:34.782 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:01:34.783 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:01:34.784 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_bias": 4 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 9, 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_2", 2024-02-12T17:01:34.785 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T17:01:12.591563+0000", 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2024-02-12T17:01:34.786 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "310", 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:01:34.787 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:01:34.788 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:01:34.789 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "rados": {} 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7", 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:01:34.790 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 8, 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 307, 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:01:34.791 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.792 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/2545411201", 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6802/2545411201", 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6804/2545411201", 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6803/2545411201", 2024-02-12T17:01:34.793 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24", 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 13, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 198, 2024-02-12T17:01:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2024-02-12T17:01:34.795 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6805/1723231708", 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6806/1723231708", 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6808/1723231708", 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6807/1723231708", 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c", 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 18, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 304, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.798 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.799 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:01:34.800 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3126580659", 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6810/3126580659", 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6812/3126580659", 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6811/3126580659", 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9", 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:01:34.801 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 26, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 97, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:01:34.802 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.803 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.804 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6813/3941314957", 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6814/3941314957", 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6816/3941314957", 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6815/3941314957", 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0498129c-861c-4f7e-b8ba-219949da3130", 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.54083251953125, 2024-02-12T17:01:34.805 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 32, 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 307, 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6800", 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:01:34.806 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6801", 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6803", 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6802", 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6800/544126779", 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6801/544126779", 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6803/544126779", 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6802/544126779", 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:01:34.808 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50", 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 38, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 307, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:01:34.809 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6804", 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6805", 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:01:34.810 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6807", 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.811 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6806", 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6804/3229884005", 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6805/3229884005", 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6807/3229884005", 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6806/3229884005", 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821", 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 45, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 248, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6808", 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6809", 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6811", 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6810", 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.815 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6808/507305558", 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6809/507305558", 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6811/507305558", 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6810/507305558", 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66", 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:01:34.816 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 54, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 307, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.817 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6812", 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6813", 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6815", 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6814", 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.819 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6812/3401977873", 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6813/3401977873", 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6815/3401977873", 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6814/3401977873", 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:01:34.820 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:00:05.711159+0000", 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:00:24.330505+0000", 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.821 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:00:34.241920+0000", 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:01:34.822 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:01:19.431056+0000", 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:01:25.444463+0000", 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.823 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:00:48.701281+0000", 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:01:34.824 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:01:12.193152+0000", 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:00:53.035879+0000", 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.825 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [ 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.8", 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: "mappings": [ 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: "from": 7, 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: "to": 2 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.9", 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: "mappings": [ 2024-02-12T17:01:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "from": 7, 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "to": 4 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [], 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": { 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/2829464018": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/1835128904": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2277321750": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2116602769": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3967118778": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:01:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2298361247": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/218227980": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2070582473": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/4291575956": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3157562110": "2024-02-13T16:44:22.727241+0000" 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "ruleset-failure-domain": "osd", 2024-02-12T17:01:34.828 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [], 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2024-02-12T17:01:34.831 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2024-02-12T17:01:34.832 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2024-02-12T17:01:34.832 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2024-02-12T17:01:34.832 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:01:34.832 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:01:35.006 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:01:35.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:34.075853+0000 mgr.y (mgr.24481) 456 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:34.742215+0000 mon.a (mon.0) 1269 : audit [DBG] from='client.? v1:172.21.15.33:0/3782255178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:01:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:34.969369+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.33:0/154062647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:34.970429+0000 mon.a (mon.0) 1270 : audit [INF] from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:34.075853+0000 mgr.y (mgr.24481) 456 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:34.742215+0000 mon.a (mon.0) 1269 : audit [DBG] from='client.? v1:172.21.15.33:0/3782255178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:01:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:34.969369+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.33:0/154062647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:34.970429+0000 mon.a (mon.0) 1270 : audit [INF] from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:35.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:34.075853+0000 mgr.y (mgr.24481) 456 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:35.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:34.742215+0000 mon.a (mon.0) 1269 : audit [DBG] from='client.? v1:172.21.15.33:0/3782255178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:01:35.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:34.969369+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.33:0/154062647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:35.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:34.970429+0000 mon.a (mon.0) 1270 : audit [INF] from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:35.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 6 16 18497 18481 192.495 153.5 0.0120211 0.00518645 2024-02-12T17:01:35.934 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:35.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:36.046 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:36.098 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:36.152 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:34.576662+0000 mgr.y (mgr.24481) 457 : cluster [DBG] pgmap v591: 138 pgs: 138 active+clean; 847 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-12T17:01:36.152 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:34.844187+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:34.844238+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:34.870238+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:34.874357+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:34.984877+0000 mon.a (mon.0) 1271 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:34.984922+0000 mon.a (mon.0) 1272 : cluster [INF] Cluster is now healthy 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:34.997170+0000 mon.a (mon.0) 1273 : audit [INF] from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:34.997976+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e312: 8 total, 8 up, 5 in 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:35.156792+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.d scrub starts 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:35.160138+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.d scrub ok 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:34.576662+0000 mgr.y (mgr.24481) 457 : cluster [DBG] pgmap v591: 138 pgs: 138 active+clean; 847 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:34.844187+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:36.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:34.844238+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:36.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:34.870238+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:01:36.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:34.874357+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:01:36.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:34.984877+0000 mon.a (mon.0) 1271 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:01:36.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:34.984922+0000 mon.a (mon.0) 1272 : cluster [INF] Cluster is now healthy 2024-02-12T17:01:36.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:34.997170+0000 mon.a (mon.0) 1273 : audit [INF] from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:01:36.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:34.997976+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e312: 8 total, 8 up, 5 in 2024-02-12T17:01:36.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:35.156792+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.d scrub starts 2024-02-12T17:01:36.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:35.160138+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.d scrub ok 2024-02-12T17:01:36.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:34.576662+0000 mgr.y (mgr.24481) 457 : cluster [DBG] pgmap v591: 138 pgs: 138 active+clean; 847 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:34.844187+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:34.844238+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:34.870238+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:34.874357+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:34.984877+0000 mon.a (mon.0) 1271 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:34.984922+0000 mon.a (mon.0) 1272 : cluster [INF] Cluster is now healthy 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:34.997170+0000 mon.a (mon.0) 1273 : audit [INF] from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:34.997976+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e312: 8 total, 8 up, 5 in 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:35.156792+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.d scrub starts 2024-02-12T17:01:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:35.160138+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.d scrub ok 2024-02-12T17:01:36.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 7 16 20915 20899 186.583 151.125 0.00652449 0.00535369 2024-02-12T17:01:36.913 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:36.958 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:36.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:37.066 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:37.109 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:37.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:36.221926+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e313: 8 total, 8 up, 5 in 2024-02-12T17:01:37.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:36.577110+0000 mgr.y (mgr.24481) 458 : cluster [DBG] pgmap v594: 138 pgs: 138 active+clean; 847 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-02-12T17:01:37.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 8 16 23350 23334 182.282 152.188 0.00722607 0.0054801 2024-02-12T17:01:37.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:36.221926+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e313: 8 total, 8 up, 5 in 2024-02-12T17:01:37.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:36.577110+0000 mgr.y (mgr.24481) 458 : cluster [DBG] pgmap v594: 138 pgs: 138 active+clean; 847 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-02-12T17:01:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:36.221926+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e313: 8 total, 8 up, 5 in 2024-02-12T17:01:37.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:36.577110+0000 mgr.y (mgr.24481) 458 : cluster [DBG] pgmap v594: 138 pgs: 138 active+clean; 847 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-02-12T17:01:37.658 INFO:tasks.thrashosds.thrasher:j is {'epoch': 311, 'fsid': 'cf6ad642-c9c5-11ee-95b9-87774f69a715', 'created': '2024-02-12T16:43:50.282468+0000', 'modified': '2024-02-12T17:01:25.925466+0000', 'last_up_change': '2024-02-12T16:49:30.933575+0000', 'last_in_change': '2024-02-12T16:59:08.202987+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T16:47:33.056945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-12T16:49:37.647343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-12T16:49:38.076200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-12T16:49:39.589565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-12T16:49:41.623856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 238, 'last_epoch_started': 237, 'last_epoch_clean': 237, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '238', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '238', 'last_force_op_resend_preluminous': '236', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-12T16:49:43.693806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-12T17:01:12.591563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '310', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '5dd172eb-ee9d-4700-9b80-60f91c5f2cb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 2545411201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6802', 'nonce': 2545411201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6804', 'nonce': 2545411201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 2545411201}]}, 'public_addr': '172.21.15.33:6801/2545411201', 'cluster_addr': '172.21.15.33:6802/2545411201', 'heartbeat_back_addr': '172.21.15.33:6804/2545411201', 'heartbeat_front_addr': '172.21.15.33:6803/2545411201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71231bc4-1716-431d-bd88-f6a3cad0bd24', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1723231708}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6806', 'nonce': 1723231708}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6808', 'nonce': 1723231708}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1723231708}]}, 'public_addr': '172.21.15.33:6805/1723231708', 'cluster_addr': '172.21.15.33:6806/1723231708', 'heartbeat_back_addr': '172.21.15.33:6808/1723231708', 'heartbeat_front_addr': '172.21.15.33:6807/1723231708', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8a9c7127-d17c-4c44-b666-4211006de47c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 304, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3126580659}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6810', 'nonce': 3126580659}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6812', 'nonce': 3126580659}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3126580659}]}, 'public_addr': '172.21.15.33:6809/3126580659', 'cluster_addr': '172.21.15.33:6810/3126580659', 'heartbeat_back_addr': '172.21.15.33:6812/3126580659', 'heartbeat_front_addr': '172.21.15.33:6811/3126580659', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3cc40671-c8f7-412c-a0f1-987f48a016b9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3941314957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6814', 'nonce': 3941314957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6816', 'nonce': 3941314957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3941314957}]}, 'public_addr': '172.21.15.33:6813/3941314957', 'cluster_addr': '172.21.15.33:6814/3941314957', 'heartbeat_back_addr': '172.21.15.33:6816/3941314957', 'heartbeat_front_addr': '172.21.15.33:6815/3941314957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0498129c-861c-4f7e-b8ba-219949da3130', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.54083251953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6800', 'nonce': 544126779}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 544126779}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 544126779}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 544126779}]}, 'public_addr': '172.21.15.94:6800/544126779', 'cluster_addr': '172.21.15.94:6801/544126779', 'heartbeat_back_addr': '172.21.15.94:6803/544126779', 'heartbeat_front_addr': '172.21.15.94:6802/544126779', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '05718a1e-1164-4b54-a493-23c1cd3adb50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 3229884005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 3229884005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 3229884005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 3229884005}]}, 'public_addr': '172.21.15.94:6804/3229884005', 'cluster_addr': '172.21.15.94:6805/3229884005', 'heartbeat_back_addr': '172.21.15.94:6807/3229884005', 'heartbeat_front_addr': '172.21.15.94:6806/3229884005', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dc48640f-8f27-405c-a9a4-cf1e3e6c8821', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 507305558}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 507305558}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 507305558}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 507305558}]}, 'public_addr': '172.21.15.94:6808/507305558', 'cluster_addr': '172.21.15.94:6809/507305558', 'heartbeat_back_addr': '172.21.15.94:6811/507305558', 'heartbeat_front_addr': '172.21.15.94:6810/507305558', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7d55aa3f-06b6-4fc3-97fd-62b535f78b66', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3401977873}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 3401977873}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 3401977873}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 3401977873}]}, 'public_addr': '172.21.15.94:6812/3401977873', 'cluster_addr': '172.21.15.94:6813/3401977873', 'heartbeat_back_addr': '172.21.15.94:6815/3401977873', 'heartbeat_front_addr': '172.21.15.94:6814/3401977873', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:00:05.711159+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:00:24.330505+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:00:34.241920+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:01:19.431056+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:01:25.444463+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:00:48.701281+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:01:12.193152+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:00:53.035879+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.8', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.9', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.33:6800/2829464018': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/1835128904': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2277321750': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2116602769': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:6800/3967118778': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/2298361247': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/218227980': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/2070582473': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/4291575956': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:6800/3157562110': '2024-02-13T16:44:22.727241+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T17:01:37.659 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-02-12T17:01:37.668 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:37.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:37.773 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:37.820 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:37.894 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:01:38.084 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:38.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:38.189 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:38.231 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:38.262 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:38.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:38.368 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:38.420 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:38.455 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:38.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:37.096832+0000 osd.4 (osd.4) 124 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:01:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:37.127483+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:01:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:37.405949+0000 osd.1 (osd.1) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:38.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:37.405997+0000 osd.1 (osd.1) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:38.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 9 16 25569 25553 177.437 138.688 0.00727505 0.00563073 2024-02-12T17:01:38.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:37.096832+0000 osd.4 (osd.4) 124 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:01:38.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:37.127483+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:01:38.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:37.405949+0000 osd.1 (osd.1) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:38.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:37.405997+0000 osd.1 (osd.1) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:37.096832+0000 osd.4 (osd.4) 124 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:01:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:37.127483+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:01:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:37.405949+0000 osd.1 (osd.1) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:37.405997+0000 osd.1 (osd.1) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:38.561 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:38.645 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:38.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:38.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:38.794 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:39.153 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:39.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:39.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:39.302 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:39.329 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:39.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:39.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:39.479 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:37.806627+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1d scrub starts 2024-02-12T17:01:39.479 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:37.813004+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1d scrub ok 2024-02-12T17:01:39.479 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:38.136479+0000 osd.0 (osd.0) 147 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:01:39.479 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:38.142596+0000 osd.0 (osd.0) 148 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:01:39.479 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:38.249926+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e314: 8 total, 8 up, 5 in 2024-02-12T17:01:39.479 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:38.577990+0000 mgr.y (mgr.24481) 459 : cluster [DBG] pgmap v596: 138 pgs: 138 active+clean; 1.3 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 220 MiB/s wr, 3.51k op/s 2024-02-12T17:01:39.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:37.806627+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1d scrub starts 2024-02-12T17:01:39.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:37.813004+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1d scrub ok 2024-02-12T17:01:39.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:38.136479+0000 osd.0 (osd.0) 147 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:01:39.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:38.142596+0000 osd.0 (osd.0) 148 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:01:39.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:38.249926+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e314: 8 total, 8 up, 5 in 2024-02-12T17:01:39.480 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:38.577990+0000 mgr.y (mgr.24481) 459 : cluster [DBG] pgmap v596: 138 pgs: 138 active+clean; 1.3 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 220 MiB/s wr, 3.51k op/s 2024-02-12T17:01:39.484 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:39.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 10 16 27685 27669 172.917 132.25 0.0047799 0.00577768 2024-02-12T17:01:39.549 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:39.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:39.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:39.703 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:37.806627+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1d scrub starts 2024-02-12T17:01:39.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:37.813004+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1d scrub ok 2024-02-12T17:01:39.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:38.136479+0000 osd.0 (osd.0) 147 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:01:39.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:38.142596+0000 osd.0 (osd.0) 148 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:01:39.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:38.249926+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e314: 8 total, 8 up, 5 in 2024-02-12T17:01:39.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:38.577990+0000 mgr.y (mgr.24481) 459 : cluster [DBG] pgmap v596: 138 pgs: 138 active+clean; 1.3 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 220 MiB/s wr, 3.51k op/s 2024-02-12T17:01:39.708 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:39.737 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:39.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:39.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:39.886 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:01:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:01:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:01:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:01:39.886 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:01:39.900 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:39.994 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:01:40.004 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:01:40.034 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:01:40.067 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:01:40.096 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:01:40.125 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:01:40.157 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:01:40.189 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:01:40.190 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:01:40.190 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:01:40.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:01:40.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:01:40.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:01:40.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:01:40.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:01:40.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:01:40.245 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:40.266 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:40.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:40.371 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:40.416 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:40.478 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:40.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:40.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:39.071611+0000 osd.4 (osd.4) 126 : cluster [DBG] 5.a scrub starts 2024-02-12T17:01:40.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:39.076433+0000 osd.4 (osd.4) 127 : cluster [DBG] 5.a scrub ok 2024-02-12T17:01:40.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:39.175101+0000 osd.0 (osd.0) 149 : cluster [DBG] 2.1e deep-scrub starts 2024-02-12T17:01:40.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:39.176424+0000 osd.0 (osd.0) 150 : cluster [DBG] 2.1e deep-scrub ok 2024-02-12T17:01:40.487 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:39.935929+0000 mon.b (mon.1) 73 : audit [DBG] from='client.? v1:172.21.15.33:0/596811850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:40.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:39.071611+0000 osd.4 (osd.4) 126 : cluster [DBG] 5.a scrub starts 2024-02-12T17:01:40.487 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:39.076433+0000 osd.4 (osd.4) 127 : cluster [DBG] 5.a scrub ok 2024-02-12T17:01:40.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:39.175101+0000 osd.0 (osd.0) 149 : cluster [DBG] 2.1e deep-scrub starts 2024-02-12T17:01:40.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:39.176424+0000 osd.0 (osd.0) 150 : cluster [DBG] 2.1e deep-scrub ok 2024-02-12T17:01:40.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:39.935929+0000 mon.b (mon.1) 73 : audit [DBG] from='client.? v1:172.21.15.33:0/596811850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:40.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 11 16 29675 29659 168.503 124.375 0.00266875 0.00592746 2024-02-12T17:01:40.585 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:40.631 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:40.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:39.071611+0000 osd.4 (osd.4) 126 : cluster [DBG] 5.a scrub starts 2024-02-12T17:01:40.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:39.076433+0000 osd.4 (osd.4) 127 : cluster [DBG] 5.a scrub ok 2024-02-12T17:01:40.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:39.175101+0000 osd.0 (osd.0) 149 : cluster [DBG] 2.1e deep-scrub starts 2024-02-12T17:01:40.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:39.176424+0000 osd.0 (osd.0) 150 : cluster [DBG] 2.1e deep-scrub ok 2024-02-12T17:01:40.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:39.935929+0000 mon.b (mon.1) 73 : audit [DBG] from='client.? v1:172.21.15.33:0/596811850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:41.272 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:01:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:40] "GET /metrics HTTP/1.1" 200 175963 "" "Prometheus/2.33.4" 2024-02-12T17:01:41.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:40.578548+0000 mgr.y (mgr.24481) 460 : cluster [DBG] pgmap v597: 138 pgs: 138 active+clean; 1.6 GiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-02-12T17:01:41.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:41.223827+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e315: 8 total, 8 up, 5 in 2024-02-12T17:01:41.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 12 16 31746 31730 165.247 129.438 0.00666645 0.0060456 2024-02-12T17:01:41.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:40.578548+0000 mgr.y (mgr.24481) 460 : cluster [DBG] pgmap v597: 138 pgs: 138 active+clean; 1.6 GiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-02-12T17:01:41.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:41.223827+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e315: 8 total, 8 up, 5 in 2024-02-12T17:01:41.641 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T17:01:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:40.578548+0000 mgr.y (mgr.24481) 460 : cluster [DBG] pgmap v597: 138 pgs: 138 active+clean; 1.6 GiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-02-12T17:01:41.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:41.223827+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e315: 8 total, 8 up, 5 in 2024-02-12T17:01:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:41.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:42.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:41.688657+0000 osd.7 (osd.7) 147 : cluster [DBG] 4.f scrub starts 2024-02-12T17:01:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:41.694866+0000 osd.7 (osd.7) 148 : cluster [DBG] 4.f scrub ok 2024-02-12T17:01:42.492 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 13 16 33662 33646 161.746 119.75 0.00832742 0.00617744 2024-02-12T17:01:42.611 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:42.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:42.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:41.688657+0000 osd.7 (osd.7) 147 : cluster [DBG] 4.f scrub starts 2024-02-12T17:01:42.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:41.694866+0000 osd.7 (osd.7) 148 : cluster [DBG] 4.f scrub ok 2024-02-12T17:01:42.660 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7] out_osds: [3, 1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:01:42.660 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T17:01:42.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:42.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:01:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:01:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:41.688657+0000 osd.7 (osd.7) 147 : cluster [DBG] 4.f scrub starts 2024-02-12T17:01:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:41.694866+0000 osd.7 (osd.7) 148 : cluster [DBG] 4.f scrub ok 2024-02-12T17:01:42.764 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:42.798 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:42.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:42.903 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:42.945 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:43.189 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:43.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:43.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:43.488 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:01:43.494 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 14 16 35461 35445 158.205 112.438 0.021886 0.00631699 2024-02-12T17:01:43.515 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:42.072674+0000 osd.0 (osd.0) 151 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:01:43.515 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:42.076579+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:01:43.515 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:42.578927+0000 mgr.y (mgr.24481) 461 : cluster [DBG] pgmap v599: 138 pgs: 138 active+clean; 1.6 GiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 805 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-02-12T17:01:43.515 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:43.516 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:42.072674+0000 osd.0 (osd.0) 151 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:01:43.516 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:42.076579+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:01:43.516 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:42.578927+0000 mgr.y (mgr.24481) 461 : cluster [DBG] pgmap v599: 138 pgs: 138 active+clean; 1.6 GiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 805 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-02-12T17:01:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:43.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:43.623 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:43.672 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:42.072674+0000 osd.0 (osd.0) 151 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:01:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:42.076579+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:01:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:42.578927+0000 mgr.y (mgr.24481) 461 : cluster [DBG] pgmap v599: 138 pgs: 138 active+clean; 1.6 GiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 805 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-02-12T17:01:43.779 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:43.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:43.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:43.927 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:43.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:44.059 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:44.105 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:44.240 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:44.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:01:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:01:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:44.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:44.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:44.401 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:44.428 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:44.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:44.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:44.494 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 15 16 37185 37169 154.841 107.75 0.00782679 0.00645236 2024-02-12T17:01:44.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:42.884580+0000 osd.5 (osd.5) 147 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:01:44.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:42.892555+0000 osd.5 (osd.5) 148 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:01:44.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:43.159697+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:01:44.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:43.163717+0000 osd.4 (osd.4) 129 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:01:44.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:42.884580+0000 osd.5 (osd.5) 147 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:01:44.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:42.892555+0000 osd.5 (osd.5) 148 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:01:44.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:43.159697+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:01:44.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:43.163717+0000 osd.4 (osd.4) 129 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:01:44.532 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:44.584 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:44.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:42.884580+0000 osd.5 (osd.5) 147 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:01:44.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:42.892555+0000 osd.5 (osd.5) 148 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:01:44.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:43.159697+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:01:44.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:43.163717+0000 osd.4 (osd.4) 129 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:01:45.158 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:45.226 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:45.307 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:01:45.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:44.088396+0000 mgr.y (mgr.24481) 462 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:45.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:44.581938+0000 mgr.y (mgr.24481) 463 : cluster [DBG] pgmap v600: 138 pgs: 138 active+clean; 2.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 171 MiB/s wr, 2.74k op/s 2024-02-12T17:01:45.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:44.860329+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? v1:172.21.15.33:0/1895938377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:45.488 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:44.088396+0000 mgr.y (mgr.24481) 462 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:45.489 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:44.581938+0000 mgr.y (mgr.24481) 463 : cluster [DBG] pgmap v600: 138 pgs: 138 active+clean; 2.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 171 MiB/s wr, 2.74k op/s 2024-02-12T17:01:45.489 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:44.860329+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? v1:172.21.15.33:0/1895938377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:45.494 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 16 16 38899 38883 151.858 107.125 0.0113251 0.00658003 2024-02-12T17:01:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:44.088396+0000 mgr.y (mgr.24481) 462 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:44.581938+0000 mgr.y (mgr.24481) 463 : cluster [DBG] pgmap v600: 138 pgs: 138 active+clean; 2.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 171 MiB/s wr, 2.74k op/s 2024-02-12T17:01:45.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:44.860329+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? v1:172.21.15.33:0/1895938377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:01:46.148 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:46.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:46.255 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:46.301 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:46.330 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:46.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:46.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:46.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:45.140304+0000 osd.4 (osd.4) 130 : cluster [DBG] 4.18 deep-scrub starts 2024-02-12T17:01:46.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:45.154156+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.18 deep-scrub ok 2024-02-12T17:01:46.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:45.288820+0000 mon.a (mon.0) 1279 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:46.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:45.302393+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? v1:172.21.15.33:0/1895938377' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:01:46.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:45.302472+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e316: 8 total, 8 up, 5 in 2024-02-12T17:01:46.479 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:46.223837+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e317: 8 total, 8 up, 5 in 2024-02-12T17:01:46.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:45.140304+0000 osd.4 (osd.4) 130 : cluster [DBG] 4.18 deep-scrub starts 2024-02-12T17:01:46.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:45.154156+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.18 deep-scrub ok 2024-02-12T17:01:46.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:45.288820+0000 mon.a (mon.0) 1279 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:46.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:45.302393+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? v1:172.21.15.33:0/1895938377' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:01:46.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:45.302472+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e316: 8 total, 8 up, 5 in 2024-02-12T17:01:46.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:46.223837+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e317: 8 total, 8 up, 5 in 2024-02-12T17:01:46.484 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:46.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 17 16 40639 40623 149.322 108.75 0.0118696 0.0066909 2024-02-12T17:01:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:45.140304+0000 osd.4 (osd.4) 130 : cluster [DBG] 4.18 deep-scrub starts 2024-02-12T17:01:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:45.154156+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.18 deep-scrub ok 2024-02-12T17:01:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:45.288820+0000 mon.a (mon.0) 1279 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:01:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:45.302393+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? v1:172.21.15.33:0/1895938377' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:01:46.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:45.302472+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e316: 8 total, 8 up, 5 in 2024-02-12T17:01:46.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:46.223837+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e317: 8 total, 8 up, 5 in 2024-02-12T17:01:46.944 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:46.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:47.051 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:47.089 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:47.117 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:47.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:47.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:47.266 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:47.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:47.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:47.432 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:01:47.432 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:01:47.434 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:46.582334+0000 mgr.y (mgr.24481) 464 : cluster [DBG] pgmap v603: 138 pgs: 138 active+clean; 2.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 852 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-02-12T17:01:47.437 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:47.473 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:47.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:47.494 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 18 16 42195 42179 146.429 97.25 0.00998722 0.0068227 2024-02-12T17:01:47.578 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:47.663 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:47.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:47.699 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:46.582334+0000 mgr.y (mgr.24481) 464 : cluster [DBG] pgmap v603: 138 pgs: 138 active+clean; 2.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 852 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-02-12T17:01:47.699 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:46.582334+0000 mgr.y (mgr.24481) 464 : cluster [DBG] pgmap v603: 138 pgs: 138 active+clean; 2.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 852 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-02-12T17:01:47.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:47.827 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:48.015 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:48.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:48.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:48.172 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:48.471 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T17:01:48.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 19 16 43851 43835 144.17 103.5 0.010454 0.00693166 2024-02-12T17:01:48.930 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:48.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:49.038 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:49.081 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:49.441 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:49.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:49.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:01:49.494453+0000 min lat: 0.00147002 max lat: 0.0620815 avg lat: 0.00703356 2024-02-12T17:01:49.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:01:49.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 20 16 45485 45469 142.067 102.125 0.00857354 0.00703356 2024-02-12T17:01:49.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:49.576 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:49.701 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:49.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:49.705 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:48.583196+0000 mgr.y (mgr.24481) 465 : cluster [DBG] pgmap v604: 138 pgs: 138 active+clean; 2.5 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 695 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-02-12T17:01:49.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:48.583196+0000 mgr.y (mgr.24481) 465 : cluster [DBG] pgmap v604: 138 pgs: 138 active+clean; 2.5 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 695 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-02-12T17:01:49.806 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:49.887 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:49.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:48.583196+0000 mgr.y (mgr.24481) 465 : cluster [DBG] pgmap v604: 138 pgs: 138 active+clean; 2.5 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 695 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-02-12T17:01:49.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:50.083 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:50.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:50.187 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:50.232 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:50.260 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:50.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:50.332 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:01:50.360 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:01:50.366 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:50.393 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:01:50.412 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:50.436 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:01:50.474 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:01:50.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 21 16 46935 46919 139.617 90.625 0.00931611 0.00715788 2024-02-12T17:01:50.511 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:01:50.544 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:01:50.584 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:01:50.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:01:50.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:01:50.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:01:50.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:01:50.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:01:50.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:01:50.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:01:50.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:01:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:50.300677+0000 mon.a (mon.0) 1283 : audit [DBG] from='client.? v1:172.21.15.33:0/2005702687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:50.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:50.300677+0000 mon.a (mon.0) 1283 : audit [DBG] from='client.? v1:172.21.15.33:0/2005702687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:50.981 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:50.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:50.300677+0000 mon.a (mon.0) 1283 : audit [DBG] from='client.? v1:172.21.15.33:0/2005702687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:51.087 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:01:51.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:51.264 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:01:51.266 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:01:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:50] "GET /metrics HTTP/1.1" 200 176036 "" "Prometheus/2.33.4" 2024-02-12T17:01:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:51.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:01:51.368 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:51.414 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:51.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 22 16 48438 48422 137.541 93.9375 0.00565978 0.00726638 2024-02-12T17:01:51.670 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T17:01:51.726 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:51.974 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:51.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:51.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:50.583796+0000 mgr.y (mgr.24481) 466 : cluster [DBG] pgmap v605: 138 pgs: 138 active+clean; 2.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.28k op/s 2024-02-12T17:01:51.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:50.826146+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.33:0/810089116' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:01:51.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:50.827656+0000 mon.a (mon.0) 1284 : audit [INF] from='client.35429 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:01:51.981 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:50.583796+0000 mgr.y (mgr.24481) 466 : cluster [DBG] pgmap v605: 138 pgs: 138 active+clean; 2.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.28k op/s 2024-02-12T17:01:51.981 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:50.826146+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.33:0/810089116' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:01:51.981 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:50.827656+0000 mon.a (mon.0) 1284 : audit [INF] from='client.35429 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:01:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:50.583796+0000 mgr.y (mgr.24481) 466 : cluster [DBG] pgmap v605: 138 pgs: 138 active+clean; 2.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.28k op/s 2024-02-12T17:01:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:50.826146+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.33:0/810089116' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:01:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:50.827656+0000 mon.a (mon.0) 1284 : audit [INF] from='client.35429 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:01:52.081 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:52.143 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:52.175 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:52.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:52.280 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:52.337 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:52.362 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:52.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:52.466 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:52.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 23 16 49921 49905 135.59 92.6875 0.0103154 0.00736989 2024-02-12T17:01:52.504 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:52.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:51.657929+0000 mon.a (mon.0) 1285 : audit [INF] from='client.35429 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:01:52.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:51.658016+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e318: 8 total, 8 up, 5 in 2024-02-12T17:01:52.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:51.657929+0000 mon.a (mon.0) 1285 : audit [INF] from='client.35429 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:01:52.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:51.658016+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e318: 8 total, 8 up, 5 in 2024-02-12T17:01:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:51.657929+0000 mon.a (mon.0) 1285 : audit [INF] from='client.35429 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:01:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:51.658016+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e318: 8 total, 8 up, 5 in 2024-02-12T17:01:52.992 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:01:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:01:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:01:53.010 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:01:53.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 24 16 51355 51339 133.675 89.625 0.0101886 0.00747574 2024-02-12T17:01:53.757 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:01:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:52.584191+0000 mgr.y (mgr.24481) 467 : cluster [DBG] pgmap v607: 138 pgs: 138 active+clean; 2.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 703 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:01:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:53.543091+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:53.543142+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:52.584191+0000 mgr.y (mgr.24481) 467 : cluster [DBG] pgmap v607: 138 pgs: 138 active+clean; 2.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 703 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:01:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:53.543091+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:53.543142+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:52.584191+0000 mgr.y (mgr.24481) 467 : cluster [DBG] pgmap v607: 138 pgs: 138 active+clean; 2.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 703 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:01:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:53.543091+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:01:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:53.543142+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:01:54.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T17:01:54.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:01:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:01:54.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 25 16 52827 52811 132.008 92 0.0113766 0.00757045 2024-02-12T17:01:54.572 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:54.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:54.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:54.731 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:54.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:54.874 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:54.910 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:53.682921+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e319: 8 total, 8 up, 5 in 2024-02-12T17:01:54.910 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:53.682921+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e319: 8 total, 8 up, 5 in 2024-02-12T17:01:54.913 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:53.682921+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e319: 8 total, 8 up, 5 in 2024-02-12T17:01:55.044 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:55.045 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:55.047 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:01:55.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:55.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:55.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:55.200 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:55.338 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:55.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:55.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:55.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 26 16 54189 54173 130.205 85.125 0.0124204 0.00767546 2024-02-12T17:01:55.499 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:55.530 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:55.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:55.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:55.742 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:55.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:55.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:01:55.861 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:54.099665+0000 mgr.y (mgr.24481) 468 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:55.861 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:54.584575+0000 mgr.y (mgr.24481) 469 : cluster [DBG] pgmap v609: 138 pgs: 138 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-02-12T17:01:55.861 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:54.099665+0000 mgr.y (mgr.24481) 468 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:55.861 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:54.584575+0000 mgr.y (mgr.24481) 469 : cluster [DBG] pgmap v609: 138 pgs: 138 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-02-12T17:01:55.901 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:55.990 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:01:55.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:54.099665+0000 mgr.y (mgr.24481) 468 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:01:55.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:54.584575+0000 mgr.y (mgr.24481) 469 : cluster [DBG] pgmap v609: 138 pgs: 138 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-02-12T17:01:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:55.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:01:56.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:56.154 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:56.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:56.286 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:56.375 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:56.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:56.479 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:56.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 27 16 55516 55500 128.454 82.9375 0.0119632 0.00778046 2024-02-12T17:01:56.522 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:56.679 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:56.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:56.784 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:56.824 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:56.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:57.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:57.174 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:57.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:57.280 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:57.349 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:57.378 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:57.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:56.584938+0000 mgr.y (mgr.24481) 470 : cluster [DBG] pgmap v610: 138 pgs: 138 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-12T17:01:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:57.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:57.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:01:57.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 28 16 56886 56870 126.924 85.625 0.012923 0.00787334 2024-02-12T17:01:57.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:56.584938+0000 mgr.y (mgr.24481) 470 : cluster [DBG] pgmap v610: 138 pgs: 138 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-12T17:01:57.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:56.584938+0000 mgr.y (mgr.24481) 470 : cluster [DBG] pgmap v610: 138 pgs: 138 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-12T17:01:57.540 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:57.568 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:01:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:57.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:01:57.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:57.758 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:57.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:57.867 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:57.916 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:58.052 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:01:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:58.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:01:58.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:01:58.201 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:58.235 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:01:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:58.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:01:58.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:58.392 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:58.495 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 29 15 58217 58202 125.418 83.25 0.0142577 0.00796774 2024-02-12T17:01:58.534 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:58.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:58.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:01:58.682 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:58.706 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:01:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:58.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:01:58.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:01:58.916 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:58.552006+0000 mon.a (mon.0) 1288 : audit [DBG] from='client.? v1:172.21.15.33:0/1017072933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:58.916 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:58.552006+0000 mon.a (mon.0) 1288 : audit [DBG] from='client.? v1:172.21.15.33:0/1017072933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:58.936 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:58.965 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:01:58.965 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:01:58.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:58.552006+0000 mon.a (mon.0) 1288 : audit [DBG] from='client.? v1:172.21.15.33:0/1017072933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:01:59.061 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:01:59.061 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:01:59.061 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:01:59.061 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:01:59.061 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:01:59.061 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:01:59.062 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:01:59.294 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:01:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:59.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:01:59.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:01:59.444 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:59.476 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:01:59.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:01:59.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:01:59.496 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 30 16 59506 59490 123.92 80.5 0.0108546 0.00806507 2024-02-12T17:01:59.584 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:01:59.632 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:01:59.660 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:01:59.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:58.586804+0000 mgr.y (mgr.24481) 471 : cluster [DBG] pgmap v611: 138 pgs: 138 active+clean; 3.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-12T17:01:59.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:59.044026+0000 mon.b (mon.1) 74 : audit [INF] from='client.? v1:172.21.15.33:0/6866010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:59.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:59.045516+0000 mon.a (mon.0) 1289 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:59.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:01:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:59.532436+0000 mon.c (mon.2) 179 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:01:59.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:58.586804+0000 mgr.y (mgr.24481) 471 : cluster [DBG] pgmap v611: 138 pgs: 138 active+clean; 3.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-12T17:01:59.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:59.044026+0000 mon.b (mon.1) 74 : audit [INF] from='client.? v1:172.21.15.33:0/6866010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:59.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:59.045516+0000 mon.a (mon.0) 1289 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:59.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:01:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:59.532436+0000 mon.c (mon.2) 179 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:01:59.888 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:58.586804+0000 mgr.y (mgr.24481) 471 : cluster [DBG] pgmap v611: 138 pgs: 138 active+clean; 3.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-12T17:01:59.888 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:59.044026+0000 mon.b (mon.1) 74 : audit [INF] from='client.? v1:172.21.15.33:0/6866010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:59.888 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:59.045516+0000 mon.a (mon.0) 1289 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:01:59.888 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:01:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:59.532436+0000 mon.c (mon.2) 179 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:02:00.073 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:00.496 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 31 16 60742 60726 122.414 77.25 0.0114262 0.00816375 2024-02-12T17:02:00.650 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:00.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:00.660 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T17:02:00.757 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:00.808 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:58.717835+0000 osd.5 (osd.5) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:00.808 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:58.717886+0000 osd.5 (osd.5) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:00.808 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:59.640963+0000 mon.a (mon.0) 1290 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:00.808 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:01:59.646935+0000 mon.a (mon.0) 1291 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:02:00.808 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:01:59.648801+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e320: 8 total, 8 up, 5 in 2024-02-12T17:02:00.812 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:00.848 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:00.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:00.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:58.717835+0000 osd.5 (osd.5) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:00.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:58.717886+0000 osd.5 (osd.5) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:00.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:59.640963+0000 mon.a (mon.0) 1290 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:00.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:01:59.646935+0000 mon.a (mon.0) 1291 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:02:00.885 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:01:59.648801+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e320: 8 total, 8 up, 5 in 2024-02-12T17:02:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:58.717835+0000 osd.5 (osd.5) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:58.717886+0000 osd.5 (osd.5) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:59.640963+0000 mon.a (mon.0) 1290 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:01:59.646935+0000 mon.a (mon.0) 1291 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:02:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:01:59.648801+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e320: 8 total, 8 up, 5 in 2024-02-12T17:02:00.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:01.002 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:01.239 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:02:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:00] "GET /metrics HTTP/1.1" 200 176036 "" "Prometheus/2.33.4" 2024-02-12T17:02:01.496 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 32 16 61989 61973 121.024 77.9375 0.00722256 0.00825812 2024-02-12T17:02:01.913 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:02:01.949 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:00.587215+0000 mgr.y (mgr.24481) 472 : cluster [DBG] pgmap v613: 138 pgs: 138 active+clean; 3.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-02-12T17:02:01.949 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:01.230991+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e321: 8 total, 8 up, 5 in 2024-02-12T17:02:01.949 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:00.587215+0000 mgr.y (mgr.24481) 472 : cluster [DBG] pgmap v613: 138 pgs: 138 active+clean; 3.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-02-12T17:02:01.949 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:01.230991+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e321: 8 total, 8 up, 5 in 2024-02-12T17:02:01.989 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:01.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:00.587215+0000 mgr.y (mgr.24481) 472 : cluster [DBG] pgmap v613: 138 pgs: 138 active+clean; 3.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-02-12T17:02:01.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:01.230991+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e321: 8 total, 8 up, 5 in 2024-02-12T17:02:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:02.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:02.102 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:02.168 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:02.382 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:02.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:02.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:02.496 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 33 16 63236 63220 119.719 77.9375 0.00968913 0.00834857 2024-02-12T17:02:02.557 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:02.595 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:02.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:02.700 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:02.740 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:02.903 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:02.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:02.992 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:02:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:02:03.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:03.076 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:03.192 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:03.192 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:03.287 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:03.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:03.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:03.443 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:03.468 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:03.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:03.498 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 34 16 64422 64406 118.371 74.125 0.0117581 0.0084435 2024-02-12T17:02:03.575 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:03.635 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:03.701 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:02.587554+0000 mgr.y (mgr.24481) 473 : cluster [DBG] pgmap v615: 138 pgs: 138 active+clean; 3.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-02-12T17:02:03.701 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:03.554919+0000 mon.c (mon.2) 180 : audit [INF] from='client.? v1:172.21.15.33:0/1531102761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:02:03.701 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:03.555583+0000 mon.a (mon.0) 1294 : audit [INF] from='client.35441 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:02:03.709 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T17:02:03.993 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:02:04.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:02.587554+0000 mgr.y (mgr.24481) 473 : cluster [DBG] pgmap v615: 138 pgs: 138 active+clean; 3.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-02-12T17:02:04.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:03.554919+0000 mon.c (mon.2) 180 : audit [INF] from='client.? v1:172.21.15.33:0/1531102761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:02:04.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:03.555583+0000 mon.a (mon.0) 1294 : audit [INF] from='client.35441 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:02:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:02.587554+0000 mgr.y (mgr.24481) 473 : cluster [DBG] pgmap v615: 138 pgs: 138 active+clean; 3.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-02-12T17:02:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:03.554919+0000 mon.c (mon.2) 180 : audit [INF] from='client.? v1:172.21.15.33:0/1531102761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:02:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:03.555583+0000 mon.a (mon.0) 1294 : audit [INF] from='client.35441 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:02:04.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:02:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:02:04.498 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 35 16 65691 65675 117.254 79.3125 0.0113173 0.008523 2024-02-12T17:02:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:03.690684+0000 mon.a (mon.0) 1295 : audit [INF] from='client.35441 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:02:04.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:03.690742+0000 mon.a (mon.0) 1296 : cluster [DBG] osdmap e322: 8 total, 8 up, 5 in 2024-02-12T17:02:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:03.690684+0000 mon.a (mon.0) 1295 : audit [INF] from='client.35441 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:02:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:03.690742+0000 mon.a (mon.0) 1296 : cluster [DBG] osdmap e322: 8 total, 8 up, 5 in 2024-02-12T17:02:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:03.690684+0000 mon.a (mon.0) 1295 : audit [INF] from='client.35441 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:02:04.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:03.690742+0000 mon.a (mon.0) 1296 : cluster [DBG] osdmap e322: 8 total, 8 up, 5 in 2024-02-12T17:02:05.498 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 36 16 67016 67000 116.298 82.8125 0.00904939 0.00859354 2024-02-12T17:02:06.026 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:04.111370+0000 mgr.y (mgr.24481) 474 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:06.026 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:04.588083+0000 mgr.y (mgr.24481) 475 : cluster [DBG] pgmap v617: 138 pgs: 138 active+clean; 3.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-12T17:02:06.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:04.783399+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:04.111370+0000 mgr.y (mgr.24481) 474 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:04.588083+0000 mgr.y (mgr.24481) 475 : cluster [DBG] pgmap v617: 138 pgs: 138 active+clean; 3.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-12T17:02:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:04.783399+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:04.111370+0000 mgr.y (mgr.24481) 474 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:04.588083+0000 mgr.y (mgr.24481) 475 : cluster [DBG] pgmap v617: 138 pgs: 138 active+clean; 3.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-12T17:02:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:04.783399+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:06.498 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 37 16 68334 68318 115.381 82.375 0.0106063 0.00866213 2024-02-12T17:02:06.801 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:06.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:06.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:06.979 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:07.017 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:07.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:07.123 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:07.167 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:07.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:07.238 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:06.234784+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e323: 8 total, 8 up, 5 in 2024-02-12T17:02:07.238 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:06.588462+0000 mgr.y (mgr.24481) 476 : cluster [DBG] pgmap v619: 138 pgs: 138 active+clean; 3.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-12T17:02:07.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:07.358 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:06.234784+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e323: 8 total, 8 up, 5 in 2024-02-12T17:02:07.358 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:06.588462+0000 mgr.y (mgr.24481) 476 : cluster [DBG] pgmap v619: 138 pgs: 138 active+clean; 3.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-12T17:02:07.359 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:06.234784+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e323: 8 total, 8 up, 5 in 2024-02-12T17:02:07.359 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:06.588462+0000 mgr.y (mgr.24481) 476 : cluster [DBG] pgmap v619: 138 pgs: 138 active+clean; 3.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-12T17:02:07.374 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:07.410 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:07.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:07.498 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 38 16 69657 69641 114.52 82.6875 0.0117094 0.00872757 2024-02-12T17:02:07.516 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:07.568 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:07.600 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:07.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:07.628 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:02:07.628 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:02:07.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:07.788 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:07.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:07.896 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:07.942 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:07.974 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:07.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:08.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:02:08.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:02:08.117 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:02:08.117 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:02:08.117 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:02:08.117 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:02:08.117 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:02:08.117 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:02:08.123 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:08.153 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:02:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:08.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:02:08.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:08.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:08.367 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:08.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:08.464 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:07.869673+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.33:0/689510932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:08.464 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:07.869673+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.33:0/689510932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:08.471 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:08.498 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 39 16 70948 70932 113.653 80.6875 0.011852 0.00879357 2024-02-12T17:02:08.536 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:08.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:07.869673+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.33:0/689510932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:08.827 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:08.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:08.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T17:02:08.894 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:08.932 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:08.972 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:09.041 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:09.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:09.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:09.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:09.213 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:09.351 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:09.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:09.384 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T17:02:09.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:02:09.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:08.589038+0000 mgr.y (mgr.24481) 477 : cluster [DBG] pgmap v620: 138 pgs: 138 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 695 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-12T17:02:09.467 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:08.589038+0000 mgr.y (mgr.24481) 477 : cluster [DBG] pgmap v620: 138 pgs: 138 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 695 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-12T17:02:09.498 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:02:09.498199+0000 min lat: 0.00147002 max lat: 0.0722358 avg lat: 0.00885292 2024-02-12T17:02:09.498 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:02:09.499 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 40 16 72277 72261 112.888 83.0625 0.0212134 0.00885292 2024-02-12T17:02:09.504 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:09.536 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:02:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:09.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:02:09.541 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:08.589038+0000 mgr.y (mgr.24481) 477 : cluster [DBG] pgmap v620: 138 pgs: 138 active+clean; 4.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 695 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-12T17:02:09.642 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:09.698 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:10.499 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 41 16 73577 73561 112.116 81.25 0.010797 0.00891504 2024-02-12T17:02:11.142 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:02:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:10] "GET /metrics HTTP/1.1" 200 176035 "" "Prometheus/2.33.4" 2024-02-12T17:02:11.162 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:11.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:11.267 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:11.308 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:11.440 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:11.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:11.501 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 42 16 74966 74950 111.508 86.8125 0.0167748 0.00896349 2024-02-12T17:02:11.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:11.597 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:11.759 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:11.762 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:10.589547+0000 mgr.y (mgr.24481) 478 : cluster [DBG] pgmap v621: 138 pgs: 138 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-12T17:02:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:11.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:11.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:10.589547+0000 mgr.y (mgr.24481) 478 : cluster [DBG] pgmap v621: 138 pgs: 138 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-12T17:02:11.864 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:11.909 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:11.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:10.589547+0000 mgr.y (mgr.24481) 478 : cluster [DBG] pgmap v621: 138 pgs: 138 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-12T17:02:12.115 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:12.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:12.221 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:12.421 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:12.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:12.501 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 43 16 76342 76326 110.915 86 0.00463779 0.00901114 2024-02-12T17:02:12.528 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:02:12.573 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:02:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:02:13.032 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:13.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:13.044 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:13.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:13.144 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:02:13.202 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:13.233 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:13.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:13.340 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:13.387 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:13.447 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:13.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:13.501 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 44 16 77668 77652 110.276 82.875 0.0171159 0.00906368 2024-02-12T17:02:13.552 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:13.628 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:13.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:13.665 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:12.590943+0000 mgr.y (mgr.24481) 479 : cluster [DBG] pgmap v622: 138 pgs: 138 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 575 B/s rd, 46 MiB/s wr, 735 op/s 2024-02-12T17:02:13.732 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:13.786 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:13.818 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:13.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:13.922 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:02:13.963 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:12.590943+0000 mgr.y (mgr.24481) 479 : cluster [DBG] pgmap v622: 138 pgs: 138 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 575 B/s rd, 46 MiB/s wr, 735 op/s 2024-02-12T17:02:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:14.003 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:02:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:14.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:02:14.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:12.590943+0000 mgr.y (mgr.24481) 479 : cluster [DBG] pgmap v622: 138 pgs: 138 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 575 B/s rd, 46 MiB/s wr, 735 op/s 2024-02-12T17:02:14.108 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:14.153 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:14.234 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:14.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:14.243 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:02:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:02:14.339 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:14.501 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 45 16 78987 78971 109.657 82.4375 0.0172522 0.00911508 2024-02-12T17:02:14.692 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:02:14.742 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:14.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:14.848 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:14.893 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:14.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:14.105919+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.33:0/1937158566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:02:14.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:14.106371+0000 mon.a (mon.0) 1299 : audit [INF] from='client.25384 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:02:15.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:14.105919+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.33:0/1937158566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:02:15.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:14.106371+0000 mon.a (mon.0) 1299 : audit [INF] from='client.25384 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:02:15.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:14.105919+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.33:0/1937158566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:02:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:14.106371+0000 mon.a (mon.0) 1299 : audit [INF] from='client.25384 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:02:15.502 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 46 16 80395 80379 109.186 88 0.0145586 0.00915447 2024-02-12T17:02:15.983 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:15.987 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:14.123541+0000 mgr.y (mgr.24481) 480 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:15.987 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:14.591759+0000 mgr.y (mgr.24481) 481 : cluster [DBG] pgmap v623: 138 pgs: 138 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:02:15.987 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:14.661012+0000 mon.a (mon.0) 1300 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:15.987 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:14.678112+0000 mon.a (mon.0) 1301 : audit [INF] from='client.25384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:02:15.987 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:14.678172+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e324: 8 total, 8 up, 5 in 2024-02-12T17:02:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:15.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:15.988 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:14.123541+0000 mgr.y (mgr.24481) 480 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:15.989 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:14.591759+0000 mgr.y (mgr.24481) 481 : cluster [DBG] pgmap v623: 138 pgs: 138 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:02:15.989 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:14.661012+0000 mon.a (mon.0) 1300 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:15.989 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:14.678112+0000 mon.a (mon.0) 1301 : audit [INF] from='client.25384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:02:15.989 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:14.678172+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e324: 8 total, 8 up, 5 in 2024-02-12T17:02:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:14.123541+0000 mgr.y (mgr.24481) 480 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:14.591759+0000 mgr.y (mgr.24481) 481 : cluster [DBG] pgmap v623: 138 pgs: 138 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:02:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:14.661012+0000 mon.a (mon.0) 1300 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:14.678112+0000 mon.a (mon.0) 1301 : audit [INF] from='client.25384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:02:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:14.678172+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e324: 8 total, 8 up, 5 in 2024-02-12T17:02:16.089 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:16.133 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:16.243 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:16.362 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:16.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:16.466 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:16.502 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 47 16 81685 81669 108.579 80.625 0.00750884 0.00920533 2024-02-12T17:02:16.517 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:16.552 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:16.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:16.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:02:16.709 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:02:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:16.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:02:16.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:16.912 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:16.973 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:16.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:17.070 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:17.081 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:17.133 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:17.166 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:17.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:16.024220+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:16.024271+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:16.060376+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:16.064959+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:16.243000+0000 mon.a (mon.0) 1303 : cluster [DBG] osdmap e325: 8 total, 8 up, 5 in 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:16.577316+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.33:0/2985605331' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:16.579414+0000 mon.a (mon.0) 1304 : audit [INF] from='client.35459 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:16.592112+0000 mgr.y (mgr.24481) 482 : cluster [DBG] pgmap v626: 138 pgs: 138 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:16.722419+0000 mon.a (mon.0) 1305 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:16.736338+0000 osd.7 (osd.7) 151 : cluster [DBG] 4.c scrub starts 2024-02-12T17:02:17.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:16.750395+0000 osd.7 (osd.7) 152 : cluster [DBG] 4.c scrub ok 2024-02-12T17:02:17.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:17.187147+0000 mon.c (mon.2) 183 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:02:17.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:17.190546+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:02:17.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:17.199021+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:17.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:17.273 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T17:02:17.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:17.363 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:17.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:17.468 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:16.024220+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:16.024271+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:16.060376+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:16.064959+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:16.243000+0000 mon.a (mon.0) 1303 : cluster [DBG] osdmap e325: 8 total, 8 up, 5 in 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:16.577316+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.33:0/2985605331' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:16.579414+0000 mon.a (mon.0) 1304 : audit [INF] from='client.35459 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:16.592112+0000 mgr.y (mgr.24481) 482 : cluster [DBG] pgmap v626: 138 pgs: 138 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:16.722419+0000 mon.a (mon.0) 1305 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:16.736338+0000 osd.7 (osd.7) 151 : cluster [DBG] 4.c scrub starts 2024-02-12T17:02:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:16.750395+0000 osd.7 (osd.7) 152 : cluster [DBG] 4.c scrub ok 2024-02-12T17:02:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:17.187147+0000 mon.c (mon.2) 183 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:02:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:17.190546+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:02:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:17.199021+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:17.503 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 48 15 82930 82915 107.937 77.875 0.0127269 0.0092601 2024-02-12T17:02:17.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:16.024220+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:17.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:16.024271+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:17.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:16.060376+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:02:17.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:16.064959+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:02:17.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:16.243000+0000 mon.a (mon.0) 1303 : cluster [DBG] osdmap e325: 8 total, 8 up, 5 in 2024-02-12T17:02:17.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:16.577316+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.33:0/2985605331' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:02:17.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:16.579414+0000 mon.a (mon.0) 1304 : audit [INF] from='client.35459 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:02:17.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:16.592112+0000 mgr.y (mgr.24481) 482 : cluster [DBG] pgmap v626: 138 pgs: 138 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-12T17:02:17.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:16.722419+0000 mon.a (mon.0) 1305 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:17.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:16.736338+0000 osd.7 (osd.7) 151 : cluster [DBG] 4.c scrub starts 2024-02-12T17:02:17.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:16.750395+0000 osd.7 (osd.7) 152 : cluster [DBG] 4.c scrub ok 2024-02-12T17:02:17.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:17.187147+0000 mon.c (mon.2) 183 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:02:17.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:17.190546+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:02:17.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:17.199021+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:17.521 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:17.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:17.059265+0000 osd.4 (osd.4) 134 : cluster [DBG] 2.f scrub starts 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:17.066322+0000 osd.4 (osd.4) 135 : cluster [DBG] 2.f scrub ok 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:17.258789+0000 mon.a (mon.0) 1307 : audit [INF] from='client.35459 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:17.258852+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e326: 8 total, 8 up, 5 in 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:17.678635+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:17.689212+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:17.059265+0000 osd.4 (osd.4) 134 : cluster [DBG] 2.f scrub starts 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:17.066322+0000 osd.4 (osd.4) 135 : cluster [DBG] 2.f scrub ok 2024-02-12T17:02:18.267 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:17.258789+0000 mon.a (mon.0) 1307 : audit [INF] from='client.35459 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:02:18.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:17.258852+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e326: 8 total, 8 up, 5 in 2024-02-12T17:02:18.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:17.678635+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:02:18.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:17.689212+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:02:18.503 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 49 16 84283 84267 107.458 84.5 0.0113438 0.00930174 2024-02-12T17:02:18.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:17.059265+0000 osd.4 (osd.4) 134 : cluster [DBG] 2.f scrub starts 2024-02-12T17:02:18.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:17.066322+0000 osd.4 (osd.4) 135 : cluster [DBG] 2.f scrub ok 2024-02-12T17:02:18.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:17.258789+0000 mon.a (mon.0) 1307 : audit [INF] from='client.35459 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:02:18.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:17.258852+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e326: 8 total, 8 up, 5 in 2024-02-12T17:02:18.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:17.678635+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:02:18.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:17.689212+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:02:18.950 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:18.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:18.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:18.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T17:02:19.058 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:19.145 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:19.237 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:19.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:19.338 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:19.390 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:19.503 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 50 16 85574 85558 106.923 80.6875 0.0154771 0.00934755 2024-02-12T17:02:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:18.080168+0000 osd.4 (osd.4) 136 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:02:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:18.084469+0000 osd.4 (osd.4) 137 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:02:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:18.592722+0000 mgr.y (mgr.24481) 483 : cluster [DBG] pgmap v628: 138 pgs: 138 active+clean; 5.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-02-12T17:02:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:18.690849+0000 osd.7 (osd.7) 155 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:02:19.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:18.703121+0000 osd.7 (osd.7) 156 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:02:19.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:18.080168+0000 osd.4 (osd.4) 136 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:02:19.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:18.084469+0000 osd.4 (osd.4) 137 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:02:19.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:18.592722+0000 mgr.y (mgr.24481) 483 : cluster [DBG] pgmap v628: 138 pgs: 138 active+clean; 5.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-02-12T17:02:19.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:18.690849+0000 osd.7 (osd.7) 155 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:02:19.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:18.703121+0000 osd.7 (osd.7) 156 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:02:19.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:18.080168+0000 osd.4 (osd.4) 136 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:02:19.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:18.084469+0000 osd.4 (osd.4) 137 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:02:19.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:18.592722+0000 mgr.y (mgr.24481) 483 : cluster [DBG] pgmap v628: 138 pgs: 138 active+clean; 5.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-02-12T17:02:19.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:18.690849+0000 osd.7 (osd.7) 155 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:02:19.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:18.703121+0000 osd.7 (osd.7) 156 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:02:20.147 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:20.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:20.255 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:02:20.327 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:02:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:20.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:02:20.467 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:20.503 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 51 16 86937 86921 106.497 85.1875 0.0130526 0.00938565 2024-02-12T17:02:20.519 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:18.884306+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.1f scrub starts 2024-02-12T17:02:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:18.973473+0000 osd.5 (osd.5) 152 : cluster [DBG] 6.1f scrub ok 2024-02-12T17:02:20.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:18.884306+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.1f scrub starts 2024-02-12T17:02:20.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:18.973473+0000 osd.5 (osd.5) 152 : cluster [DBG] 6.1f scrub ok 2024-02-12T17:02:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:20.551 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:18.884306+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.1f scrub starts 2024-02-12T17:02:20.551 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:18.973473+0000 osd.5 (osd.5) 152 : cluster [DBG] 6.1f scrub ok 2024-02-12T17:02:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:20.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:20.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:20.695 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:20.833 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:20.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:20.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:02:20.915 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:02:20.938 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:20.945 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:02:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:20.985 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:02:21.014 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:02:21.016 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:21.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:21.043 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:02:21.074 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:02:21.089 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:02:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:20] "GET /metrics HTTP/1.1" 200 176049 "" "Prometheus/2.33.4" 2024-02-12T17:02:21.112 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:02:21.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:02:21.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:02:21.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:02:21.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:02:21.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:02:21.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:02:21.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:02:21.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:02:21.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:21.164 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:21.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:20.593175+0000 mgr.y (mgr.24481) 484 : cluster [DBG] pgmap v629: 138 pgs: 138 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-12T17:02:21.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:20.850808+0000 mon.c (mon.2) 185 : audit [DBG] from='client.? v1:172.21.15.33:0/1518564113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:21.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:21.243353+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e327: 8 total, 8 up, 5 in 2024-02-12T17:02:21.414 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:20.593175+0000 mgr.y (mgr.24481) 484 : cluster [DBG] pgmap v629: 138 pgs: 138 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-12T17:02:21.414 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:20.850808+0000 mon.c (mon.2) 185 : audit [DBG] from='client.? v1:172.21.15.33:0/1518564113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:21.414 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:21.243353+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e327: 8 total, 8 up, 5 in 2024-02-12T17:02:21.503 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 52 15 88157 88142 105.916 76.3125 0.0104992 0.00943747 2024-02-12T17:02:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:20.593175+0000 mgr.y (mgr.24481) 484 : cluster [DBG] pgmap v629: 138 pgs: 138 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-12T17:02:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:20.850808+0000 mon.c (mon.2) 185 : audit [DBG] from='client.? v1:172.21.15.33:0/1518564113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:21.243353+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e327: 8 total, 8 up, 5 in 2024-02-12T17:02:22.503 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 53 16 89469 89453 105.464 81.9375 0.00641664 0.0094778 2024-02-12T17:02:22.672 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:20.859774+0000 osd.5 (osd.5) 153 : cluster [DBG] 5.e scrub starts 2024-02-12T17:02:22.673 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:20.866583+0000 osd.5 (osd.5) 154 : cluster [DBG] 5.e scrub ok 2024-02-12T17:02:22.673 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:21.121021+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub starts 2024-02-12T17:02:22.673 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:21.149797+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.1e scrub ok 2024-02-12T17:02:22.673 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:20.859774+0000 osd.5 (osd.5) 153 : cluster [DBG] 5.e scrub starts 2024-02-12T17:02:22.673 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:20.866583+0000 osd.5 (osd.5) 154 : cluster [DBG] 5.e scrub ok 2024-02-12T17:02:22.674 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:21.121021+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub starts 2024-02-12T17:02:22.674 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:21.149797+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.1e scrub ok 2024-02-12T17:02:22.741 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:02:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:02:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:20.859774+0000 osd.5 (osd.5) 153 : cluster [DBG] 5.e scrub starts 2024-02-12T17:02:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:20.866583+0000 osd.5 (osd.5) 154 : cluster [DBG] 5.e scrub ok 2024-02-12T17:02:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:21.121021+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub starts 2024-02-12T17:02:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:21.149797+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.1e scrub ok 2024-02-12T17:02:22.811 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:02:22.811 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:02:23.415 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:23.436 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:23.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:23.503 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 54 16 90757 90741 105.001 80.5 0.0101846 0.00951955 2024-02-12T17:02:23.544 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:23.598 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:23.642 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:23.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:23.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:22.139713+0000 osd.4 (osd.4) 140 : cluster [DBG] 9.c scrub starts 2024-02-12T17:02:23.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:22.593621+0000 mgr.y (mgr.24481) 485 : cluster [DBG] pgmap v631: 138 pgs: 138 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 806 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-12T17:02:23.714 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:22.139713+0000 osd.4 (osd.4) 140 : cluster [DBG] 9.c scrub starts 2024-02-12T17:02:23.714 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:22.593621+0000 mgr.y (mgr.24481) 485 : cluster [DBG] pgmap v631: 138 pgs: 138 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 806 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-12T17:02:23.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:22.139713+0000 osd.4 (osd.4) 140 : cluster [DBG] 9.c scrub starts 2024-02-12T17:02:23.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:22.593621+0000 mgr.y (mgr.24481) 485 : cluster [DBG] pgmap v631: 138 pgs: 138 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 806 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-12T17:02:23.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:23.811 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:23.844 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:23.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:23.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:24.013 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:24.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:24.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:24.174 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T17:02:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:24.241 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:24.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:24.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:24.407 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:24.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:02:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:02:24.503 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 55 16 91907 91891 104.399 71.875 0.0120887 0.00957458 2024-02-12T17:02:25.156 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:25.176 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:25.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:25.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:25.349 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:25.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:25.445 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:02:25.486 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:25.504 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 56 16 93228 93212 104.007 82.5625 0.00926041 0.00961058 2024-02-12T17:02:25.547 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:25.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:24.134380+0000 mgr.y (mgr.24481) 486 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:25.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:24.595844+0000 mgr.y (mgr.24481) 487 : cluster [DBG] pgmap v632: 138 pgs: 1 active+clean+scrubbing, 137 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-12T17:02:25.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:24.719407+0000 osd.4 (osd.4) 141 : cluster [DBG] 9.c scrub ok 2024-02-12T17:02:25.745 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:24.815701+0000 osd.0 (osd.0) 153 : cluster [DBG] 2.19 scrub starts 2024-02-12T17:02:25.746 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:24.819659+0000 osd.0 (osd.0) 154 : cluster [DBG] 2.19 scrub ok 2024-02-12T17:02:25.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:24.134380+0000 mgr.y (mgr.24481) 486 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:25.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:24.595844+0000 mgr.y (mgr.24481) 487 : cluster [DBG] pgmap v632: 138 pgs: 1 active+clean+scrubbing, 137 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-12T17:02:25.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:24.719407+0000 osd.4 (osd.4) 141 : cluster [DBG] 9.c scrub ok 2024-02-12T17:02:25.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:24.815701+0000 osd.0 (osd.0) 153 : cluster [DBG] 2.19 scrub starts 2024-02-12T17:02:25.746 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:24.819659+0000 osd.0 (osd.0) 154 : cluster [DBG] 2.19 scrub ok 2024-02-12T17:02:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:24.134380+0000 mgr.y (mgr.24481) 486 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:24.595844+0000 mgr.y (mgr.24481) 487 : cluster [DBG] pgmap v632: 138 pgs: 1 active+clean+scrubbing, 137 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-12T17:02:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:24.719407+0000 osd.4 (osd.4) 141 : cluster [DBG] 9.c scrub ok 2024-02-12T17:02:25.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:24.815701+0000 osd.0 (osd.0) 153 : cluster [DBG] 2.19 scrub starts 2024-02-12T17:02:25.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:24.819659+0000 osd.0 (osd.0) 154 : cluster [DBG] 2.19 scrub ok 2024-02-12T17:02:26.505 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 57 16 94506 94490 103.584 79.875 0.0119898 0.00965013 2024-02-12T17:02:27.281 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:02:27.303 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:25.760163+0000 osd.5 (osd.5) 155 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:02:27.303 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:25.762164+0000 osd.5 (osd.5) 156 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:02:27.304 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:25.811978+0000 osd.0 (osd.0) 155 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:02:27.304 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:25.815809+0000 osd.0 (osd.0) 156 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:02:27.304 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:26.359891+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? v1:172.21.15.33:0/416002654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:02:27.304 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:26.596187+0000 mgr.y (mgr.24481) 488 : cluster [DBG] pgmap v633: 138 pgs: 1 active+clean+scrubbing, 137 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-12T17:02:27.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:25.760163+0000 osd.5 (osd.5) 155 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:02:27.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:25.762164+0000 osd.5 (osd.5) 156 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:02:27.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:25.811978+0000 osd.0 (osd.0) 155 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:02:27.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:25.815809+0000 osd.0 (osd.0) 156 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:02:27.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:26.359891+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? v1:172.21.15.33:0/416002654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:02:27.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:26.596187+0000 mgr.y (mgr.24481) 488 : cluster [DBG] pgmap v633: 138 pgs: 1 active+clean+scrubbing, 137 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-12T17:02:27.322 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:27.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:27.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:27.487 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:27.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 58 16 95884 95868 103.277 86.125 0.00784243 0.00967847 2024-02-12T17:02:27.568 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:27.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:27.674 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:27.720 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:27.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:25.760163+0000 osd.5 (osd.5) 155 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:02:27.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:25.762164+0000 osd.5 (osd.5) 156 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:02:27.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:25.811978+0000 osd.0 (osd.0) 155 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:02:27.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:25.815809+0000 osd.0 (osd.0) 156 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:02:27.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:26.359891+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? v1:172.21.15.33:0/416002654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:02:27.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:26.596187+0000 mgr.y (mgr.24481) 488 : cluster [DBG] pgmap v633: 138 pgs: 1 active+clean+scrubbing, 137 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-12T17:02:28.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 59 16 97229 97213 102.951 84.0625 0.0263817 0.00970938 2024-02-12T17:02:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:27.252716+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:02:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:27.252750+0000 mon.a (mon.0) 1312 : cluster [INF] Cluster is now healthy 2024-02-12T17:02:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:27.271076+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? v1:172.21.15.33:0/416002654' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:02:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:27.271113+0000 mon.a (mon.0) 1314 : cluster [DBG] osdmap e328: 8 total, 8 up, 5 in 2024-02-12T17:02:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:27.252716+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:02:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:27.252750+0000 mon.a (mon.0) 1312 : cluster [INF] Cluster is now healthy 2024-02-12T17:02:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:27.271076+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? v1:172.21.15.33:0/416002654' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:02:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:27.271113+0000 mon.a (mon.0) 1314 : cluster [DBG] osdmap e328: 8 total, 8 up, 5 in 2024-02-12T17:02:28.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:27.252716+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:02:28.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:27.252750+0000 mon.a (mon.0) 1312 : cluster [INF] Cluster is now healthy 2024-02-12T17:02:28.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:27.271076+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? v1:172.21.15.33:0/416002654' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:02:28.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:27.271113+0000 mon.a (mon.0) 1314 : cluster [DBG] osdmap e328: 8 total, 8 up, 5 in 2024-02-12T17:02:29.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:02:29.507544+0000 min lat: 0.00147002 max lat: 0.0761484 avg lat: 0.00975309 2024-02-12T17:02:29.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:02:29.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 60 16 98431 98415 102.488 75.125 0.00883095 0.00975309 2024-02-12T17:02:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:28.596717+0000 mgr.y (mgr.24481) 489 : cluster [DBG] pgmap v635: 138 pgs: 138 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:02:29.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:28.596717+0000 mgr.y (mgr.24481) 489 : cluster [DBG] pgmap v635: 138 pgs: 138 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:02:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:28.596717+0000 mgr.y (mgr.24481) 489 : cluster [DBG] pgmap v635: 138 pgs: 138 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:02:30.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 61 16 99772 99756 102.181 83.8125 0.0146517 0.00978253 2024-02-12T17:02:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:29.056055+0000 osd.4 (osd.4) 142 : cluster [DBG] 5.0 deep-scrub starts 2024-02-12T17:02:30.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:29.064198+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.0 deep-scrub ok 2024-02-12T17:02:30.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:29.056055+0000 osd.4 (osd.4) 142 : cluster [DBG] 5.0 deep-scrub starts 2024-02-12T17:02:30.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:29.064198+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.0 deep-scrub ok 2024-02-12T17:02:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:29.056055+0000 osd.4 (osd.4) 142 : cluster [DBG] 5.0 deep-scrub starts 2024-02-12T17:02:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:29.064198+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.0 deep-scrub ok 2024-02-12T17:02:31.110 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:31.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:02:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:30] "GET /metrics HTTP/1.1" 200 176049 "" "Prometheus/2.33.4" 2024-02-12T17:02:31.313 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T17:02:31.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 62 16 100935 100919 101.706 72.6875 0.00879937 0.00982835 2024-02-12T17:02:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:30.018609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:02:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:30.025319+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:02:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:30.483860+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? v1:172.21.15.33:0/1120926054' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:02:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:30.556862+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.1b scrub starts 2024-02-12T17:02:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:30.597247+0000 mgr.y (mgr.24481) 490 : cluster [DBG] pgmap v636: 138 pgs: 138 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-12T17:02:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:30.689638+0000 osd.7 (osd.7) 158 : cluster [DBG] 6.1b scrub ok 2024-02-12T17:02:31.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:30.018609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:02:31.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:30.025319+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:02:31.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:30.483860+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? v1:172.21.15.33:0/1120926054' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:02:31.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:30.556862+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.1b scrub starts 2024-02-12T17:02:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:30.597247+0000 mgr.y (mgr.24481) 490 : cluster [DBG] pgmap v636: 138 pgs: 138 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-12T17:02:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:30.689638+0000 osd.7 (osd.7) 158 : cluster [DBG] 6.1b scrub ok 2024-02-12T17:02:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:30.018609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:02:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:30.025319+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:02:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:30.483860+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? v1:172.21.15.33:0/1120926054' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:02:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:30.556862+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.1b scrub starts 2024-02-12T17:02:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:30.597247+0000 mgr.y (mgr.24481) 490 : cluster [DBG] pgmap v636: 138 pgs: 138 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-12T17:02:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:30.689638+0000 osd.7 (osd.7) 158 : cluster [DBG] 6.1b scrub ok 2024-02-12T17:02:32.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 63 15 102246 102231 101.393 82 0.00990204 0.00985833 2024-02-12T17:02:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:30.752263+0000 osd.0 (osd.0) 157 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:02:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:30.758516+0000 osd.0 (osd.0) 158 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:02:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:31.303142+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? v1:172.21.15.33:0/1120926054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:02:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:31.303190+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e329: 8 total, 8 up, 5 in 2024-02-12T17:02:32.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:31.539373+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.d scrub starts 2024-02-12T17:02:32.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:31.546958+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.d scrub ok 2024-02-12T17:02:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:30.752263+0000 osd.0 (osd.0) 157 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:02:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:30.758516+0000 osd.0 (osd.0) 158 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:02:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:31.303142+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? v1:172.21.15.33:0/1120926054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:02:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:31.303190+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e329: 8 total, 8 up, 5 in 2024-02-12T17:02:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:31.539373+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.d scrub starts 2024-02-12T17:02:32.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:31.546958+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.d scrub ok 2024-02-12T17:02:32.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:30.752263+0000 osd.0 (osd.0) 157 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:02:32.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:30.758516+0000 osd.0 (osd.0) 158 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:02:32.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:31.303142+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? v1:172.21.15.33:0/1120926054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:02:32.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:31.303190+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e329: 8 total, 8 up, 5 in 2024-02-12T17:02:32.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:31.539373+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.d scrub starts 2024-02-12T17:02:32.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:31.546958+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.d scrub ok 2024-02-12T17:02:32.747 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:02:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:02:33.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 64 16 103598 103582 101.128 84.4375 0.0151303 0.00988414 2024-02-12T17:02:33.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:32.495668+0000 mon.a (mon.0) 1318 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:33.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:32.597726+0000 mgr.y (mgr.24481) 491 : cluster [DBG] pgmap v638: 138 pgs: 138 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 808 op/s 2024-02-12T17:02:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:32.495668+0000 mon.a (mon.0) 1318 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:32.597726+0000 mgr.y (mgr.24481) 491 : cluster [DBG] pgmap v638: 138 pgs: 138 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 808 op/s 2024-02-12T17:02:33.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:32.495668+0000 mon.a (mon.0) 1318 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:02:33.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:32.597726+0000 mgr.y (mgr.24481) 491 : cluster [DBG] pgmap v638: 138 pgs: 138 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 808 op/s 2024-02-12T17:02:34.244 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:02:34.465 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:34.472 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:34.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:34.491 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:02:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:02:34.508 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 65 16 104984 104968 100.904 86.625 0.00414942 0.00990541 2024-02-12T17:02:34.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:34.636 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:34.670 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:34.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:34.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:33.537127+0000 osd.6 (osd.6) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:34.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:33.537193+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:34.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:33.537127+0000 osd.6 (osd.6) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:33.537193+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:34.780 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:02:34.807 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:34.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:33.537127+0000 osd.6 (osd.6) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:34.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:33.537193+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:35.509 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 66 16 106312 106296 100.632 83 0.00927289 0.00993319 2024-02-12T17:02:35.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:33.732900+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:33.737552+0000 osd.5 (osd.5) 158 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:34.147051+0000 mgr.y (mgr.24481) 492 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:34.498736+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.1c deep-scrub starts 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:34.502325+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.1c deep-scrub ok 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:34.512586+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e330: 8 total, 8 up, 5 in 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:34.598033+0000 mgr.y (mgr.24481) 493 : cluster [DBG] pgmap v640: 138 pgs: 138 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:33.732900+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:33.737552+0000 osd.5 (osd.5) 158 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:34.147051+0000 mgr.y (mgr.24481) 492 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:34.498736+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.1c deep-scrub starts 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:34.502325+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.1c deep-scrub ok 2024-02-12T17:02:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:34.512586+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e330: 8 total, 8 up, 5 in 2024-02-12T17:02:35.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:34.598033+0000 mgr.y (mgr.24481) 493 : cluster [DBG] pgmap v640: 138 pgs: 138 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-02-12T17:02:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:33.732900+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:02:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:33.737552+0000 osd.5 (osd.5) 158 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:02:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:34.147051+0000 mgr.y (mgr.24481) 492 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:34.498736+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.1c deep-scrub starts 2024-02-12T17:02:35.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:34.502325+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.1c deep-scrub ok 2024-02-12T17:02:35.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:34.512586+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e330: 8 total, 8 up, 5 in 2024-02-12T17:02:35.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:34.598033+0000 mgr.y (mgr.24481) 493 : cluster [DBG] pgmap v640: 138 pgs: 138 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-02-12T17:02:36.509 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 67 16 107496 107480 100.235 74 0.0109272 0.00997265 2024-02-12T17:02:36.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:34.734650+0000 osd.5 (osd.5) 159 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:02:36.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:34.738382+0000 osd.5 (osd.5) 160 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:02:36.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:35.670687+0000 osd.4 (osd.4) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:36.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:35.670741+0000 osd.4 (osd.4) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:36.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:35.704406+0000 osd.0 (osd.0) 159 : cluster [DBG] 5.5 deep-scrub starts 2024-02-12T17:02:36.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:35.760740+0000 osd.0 (osd.0) 160 : cluster [DBG] 5.5 deep-scrub ok 2024-02-12T17:02:37.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:34.734650+0000 osd.5 (osd.5) 159 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:02:37.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:34.738382+0000 osd.5 (osd.5) 160 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:02:37.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:35.670687+0000 osd.4 (osd.4) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:37.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:35.670741+0000 osd.4 (osd.4) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:37.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:35.704406+0000 osd.0 (osd.0) 159 : cluster [DBG] 5.5 deep-scrub starts 2024-02-12T17:02:37.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:35.760740+0000 osd.0 (osd.0) 160 : cluster [DBG] 5.5 deep-scrub ok 2024-02-12T17:02:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:34.734650+0000 osd.5 (osd.5) 159 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:02:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:34.738382+0000 osd.5 (osd.5) 160 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:02:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:35.670687+0000 osd.4 (osd.4) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:35.670741+0000 osd.4 (osd.4) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:35.704406+0000 osd.0 (osd.0) 159 : cluster [DBG] 5.5 deep-scrub starts 2024-02-12T17:02:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:35.760740+0000 osd.0 (osd.0) 160 : cluster [DBG] 5.5 deep-scrub ok 2024-02-12T17:02:37.509 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 68 16 108822 108806 99.9794 82.875 0.0145424 0.00999764 2024-02-12T17:02:37.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:36.546530+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e331: 8 total, 8 up, 5 in 2024-02-12T17:02:37.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:36.598416+0000 mgr.y (mgr.24481) 494 : cluster [DBG] pgmap v642: 138 pgs: 138 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-12T17:02:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:36.708638+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.17 scrub starts 2024-02-12T17:02:38.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:36.546530+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e331: 8 total, 8 up, 5 in 2024-02-12T17:02:38.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:36.598416+0000 mgr.y (mgr.24481) 494 : cluster [DBG] pgmap v642: 138 pgs: 138 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-12T17:02:38.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:36.708638+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.17 scrub starts 2024-02-12T17:02:38.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:36.546530+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e331: 8 total, 8 up, 5 in 2024-02-12T17:02:38.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:36.598416+0000 mgr.y (mgr.24481) 494 : cluster [DBG] pgmap v642: 138 pgs: 138 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-12T17:02:38.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:36.708638+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.17 scrub starts 2024-02-12T17:02:38.509 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 69 16 110004 109988 99.601 73.875 0.00926481 0.0100366 2024-02-12T17:02:38.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:36.771441+0000 osd.5 (osd.5) 162 : cluster [DBG] 6.17 scrub ok 2024-02-12T17:02:38.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:37.533510+0000 osd.7 (osd.7) 163 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:02:38.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:37.537682+0000 osd.7 (osd.7) 164 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:02:38.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:37.707591+0000 osd.5 (osd.5) 163 : cluster [DBG] 6.1d scrub starts 2024-02-12T17:02:39.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:36.771441+0000 osd.5 (osd.5) 162 : cluster [DBG] 6.17 scrub ok 2024-02-12T17:02:39.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:37.533510+0000 osd.7 (osd.7) 163 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:02:39.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:37.537682+0000 osd.7 (osd.7) 164 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:02:39.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:37.707591+0000 osd.5 (osd.5) 163 : cluster [DBG] 6.1d scrub starts 2024-02-12T17:02:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:36.771441+0000 osd.5 (osd.5) 162 : cluster [DBG] 6.17 scrub ok 2024-02-12T17:02:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:37.533510+0000 osd.7 (osd.7) 163 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:02:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:37.537682+0000 osd.7 (osd.7) 164 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:02:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:37.707591+0000 osd.5 (osd.5) 163 : cluster [DBG] 6.1d scrub starts 2024-02-12T17:02:39.510 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 70 16 111270 111254 99.3085 79.125 0.0142159 0.0100655 2024-02-12T17:02:39.655 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T17:02:39.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:37.745832+0000 osd.5 (osd.5) 164 : cluster [DBG] 6.1d scrub ok 2024-02-12T17:02:39.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:38.598979+0000 mgr.y (mgr.24481) 495 : cluster [DBG] pgmap v643: 138 pgs: 138 active+clean; 6.6 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 701 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-12T17:02:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:37.745832+0000 osd.5 (osd.5) 164 : cluster [DBG] 6.1d scrub ok 2024-02-12T17:02:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:38.598979+0000 mgr.y (mgr.24481) 495 : cluster [DBG] pgmap v643: 138 pgs: 138 active+clean; 6.6 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 701 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-12T17:02:39.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:37.745832+0000 osd.5 (osd.5) 164 : cluster [DBG] 6.1d scrub ok 2024-02-12T17:02:39.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:38.598979+0000 mgr.y (mgr.24481) 495 : cluster [DBG] pgmap v643: 138 pgs: 138 active+clean; 6.6 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 701 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-12T17:02:40.510 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 71 16 112554 112538 99.04 80.25 0.00881086 0.010093 2024-02-12T17:02:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:39.559207+0000 osd.7 (osd.7) 165 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:02:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:39.569595+0000 osd.7 (osd.7) 166 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:02:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:39.559207+0000 osd.7 (osd.7) 165 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:02:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:39.569595+0000 osd.7 (osd.7) 166 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:02:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:39.559207+0000 osd.7 (osd.7) 165 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:02:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:39.569595+0000 osd.7 (osd.7) 166 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:02:41.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:02:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:40] "GET /metrics HTTP/1.1" 200 176046 "" "Prometheus/2.33.4" 2024-02-12T17:02:41.465 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:41.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:41.509 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 72 16 113943 113927 98.8701 86.8125 0.00916992 0.0101104 2024-02-12T17:02:41.571 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:02:41.619 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:41.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:40.553739+0000 osd.7 (osd.7) 167 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:02:41.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:40.562417+0000 osd.7 (osd.7) 168 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:02:41.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:40.599880+0000 mgr.y (mgr.24481) 496 : cluster [DBG] pgmap v644: 138 pgs: 138 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-02-12T17:02:41.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:40.654292+0000 osd.5 (osd.5) 165 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:02:41.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:40.656508+0000 osd.5 (osd.5) 166 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:02:41.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:40.942594+0000 osd.4 (osd.4) 148 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:02:41.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:40.949846+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:02:41.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:40.553739+0000 osd.7 (osd.7) 167 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:02:41.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:40.562417+0000 osd.7 (osd.7) 168 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:02:41.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:40.599880+0000 mgr.y (mgr.24481) 496 : cluster [DBG] pgmap v644: 138 pgs: 138 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-02-12T17:02:41.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:40.654292+0000 osd.5 (osd.5) 165 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:02:41.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:40.656508+0000 osd.5 (osd.5) 166 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:02:41.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:40.942594+0000 osd.4 (osd.4) 148 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:02:41.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:40.949846+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:02:41.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:40.553739+0000 osd.7 (osd.7) 167 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:02:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:40.562417+0000 osd.7 (osd.7) 168 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:02:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:40.599880+0000 mgr.y (mgr.24481) 496 : cluster [DBG] pgmap v644: 138 pgs: 138 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-02-12T17:02:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:40.654292+0000 osd.5 (osd.5) 165 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:02:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:40.656508+0000 osd.5 (osd.5) 166 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:02:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:40.942594+0000 osd.4 (osd.4) 148 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:02:41.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:40.949846+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:02:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:42.510 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 73 15 115300 115285 98.6784 84.875 0.0137236 0.0101302 2024-02-12T17:02:42.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:41.640445+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.d scrub starts 2024-02-12T17:02:42.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:41.672519+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.d scrub ok 2024-02-12T17:02:42.779 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:42.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:41.640445+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.d scrub starts 2024-02-12T17:02:42.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:41.672519+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.d scrub ok 2024-02-12T17:02:42.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:42.792 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:02:42.792 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:02:42.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:42.951 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:42.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:41.640445+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.d scrub starts 2024-02-12T17:02:42.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:41.672519+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.d scrub ok 2024-02-12T17:02:42.994 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:02:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:02:43.510 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 74 15 116623 116608 98.4622 82.6875 0.0127511 0.0101522 2024-02-12T17:02:43.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:42.600267+0000 mgr.y (mgr.24481) 497 : cluster [DBG] pgmap v645: 138 pgs: 138 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 633 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T17:02:43.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:42.666262+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? v1:172.21.15.33:0/2247496525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:43.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:42.930573+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:02:43.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:42.958465+0000 osd.4 (osd.4) 151 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:02:43.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:42.600267+0000 mgr.y (mgr.24481) 497 : cluster [DBG] pgmap v645: 138 pgs: 138 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 633 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T17:02:43.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:42.666262+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? v1:172.21.15.33:0/2247496525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:43.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:42.930573+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:02:43.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:42.958465+0000 osd.4 (osd.4) 151 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:02:43.851 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:02:43.851 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:02:43.851 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:02:43.851 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:02:43.851 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:02:43.852 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:02:43.852 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:02:43.852 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:02:43.852 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:02:43.852 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:02:43.852 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:02:43.852 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:02:43.852 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:02:43.853 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:02:43.853 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:02:43.853 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:02:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:42.600267+0000 mgr.y (mgr.24481) 497 : cluster [DBG] pgmap v645: 138 pgs: 138 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 633 B/s rd, 50 MiB/s wr, 794 op/s 2024-02-12T17:02:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:42.666262+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? v1:172.21.15.33:0/2247496525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:42.930573+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:02:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:42.958465+0000 osd.4 (osd.4) 151 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:02:44.493 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:02:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:02:44.510 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 75 16 118028 118012 98.3191 87.75 0.0115623 0.0101669 2024-02-12T17:02:44.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:43.656629+0000 osd.5 (osd.5) 169 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:02:44.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:43.665241+0000 osd.5 (osd.5) 170 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:02:44.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:43.974874+0000 osd.4 (osd.4) 152 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:02:44.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:43.982617+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:02:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:43.656629+0000 osd.5 (osd.5) 169 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:02:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:43.665241+0000 osd.5 (osd.5) 170 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:02:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:43.974874+0000 osd.4 (osd.4) 152 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:02:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:43.982617+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:02:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:43.656629+0000 osd.5 (osd.5) 169 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:02:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:43.665241+0000 osd.5 (osd.5) 170 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:02:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:43.974874+0000 osd.4 (osd.4) 152 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:02:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:43.982617+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:02:45.153 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:45.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T17:02:45.515 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 76 16 119377 119361 98.1344 84.3125 0.0086178 0.0101863 2024-02-12T17:02:45.523 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:45.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:45.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:45.626 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:45.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:45.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:44.158521+0000 mgr.y (mgr.24481) 498 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:44.600926+0000 mgr.y (mgr.24481) 499 : cluster [DBG] pgmap v646: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-12T17:02:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:44.681340+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:02:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:44.688395+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:02:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:44.158521+0000 mgr.y (mgr.24481) 498 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:44.600926+0000 mgr.y (mgr.24481) 499 : cluster [DBG] pgmap v646: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-12T17:02:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:44.681340+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:02:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:44.688395+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:02:45.805 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:45.854 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:45.887 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:45.889 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:44.158521+0000 mgr.y (mgr.24481) 498 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:45.889 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:44.600926+0000 mgr.y (mgr.24481) 499 : cluster [DBG] pgmap v646: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-12T17:02:45.889 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:44.681340+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:02:45.889 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:44.688395+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:02:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:45.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:45.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:02:46.047 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:46.077 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:02:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:46.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:02:46.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:46.233 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:46.269 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:46.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:46.374 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:46.417 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:46.510 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 77 16 120628 120612 97.8753 78.1875 0.00794771 0.0102132 2024-02-12T17:02:46.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:02:46.709 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:45.586473+0000 osd.7 (osd.7) 169 : cluster [DBG] 4.a scrub starts 2024-02-12T17:02:46.709 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:45.594707+0000 osd.7 (osd.7) 170 : cluster [DBG] 4.a scrub ok 2024-02-12T17:02:46.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:45.586473+0000 osd.7 (osd.7) 169 : cluster [DBG] 4.a scrub starts 2024-02-12T17:02:46.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:45.594707+0000 osd.7 (osd.7) 170 : cluster [DBG] 4.a scrub ok 2024-02-12T17:02:46.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:45.586473+0000 osd.7 (osd.7) 169 : cluster [DBG] 4.a scrub starts 2024-02-12T17:02:46.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:45.594707+0000 osd.7 (osd.7) 170 : cluster [DBG] 4.a scrub ok 2024-02-12T17:02:47.181 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:47.182 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:47.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:47.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:47.365 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:47.403 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:47.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:47.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:47.512 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 78 16 121960 121944 97.6865 83.25 0.0132821 0.0102331 2024-02-12T17:02:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:47.586 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:47.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:47.684 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:46.556522+0000 osd.5 (osd.5) 171 : cluster [DBG] 6.19 scrub starts 2024-02-12T17:02:47.684 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:46.577040+0000 osd.5 (osd.5) 172 : cluster [DBG] 6.19 scrub ok 2024-02-12T17:02:47.684 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:46.606657+0000 mgr.y (mgr.24481) 500 : cluster [DBG] pgmap v647: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1017 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-12T17:02:47.684 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:47.017610+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:02:47.684 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:47.020839+0000 osd.4 (osd.4) 155 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:02:47.685 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:46.556522+0000 osd.5 (osd.5) 171 : cluster [DBG] 6.19 scrub starts 2024-02-12T17:02:47.685 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:46.577040+0000 osd.5 (osd.5) 172 : cluster [DBG] 6.19 scrub ok 2024-02-12T17:02:47.685 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:46.606657+0000 mgr.y (mgr.24481) 500 : cluster [DBG] pgmap v647: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1017 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-12T17:02:47.685 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:47.017610+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:02:47.685 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:47.020839+0000 osd.4 (osd.4) 155 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:02:47.691 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:47.716 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:47.747 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:47.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:47.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:02:47.915 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:47.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:46.556522+0000 osd.5 (osd.5) 171 : cluster [DBG] 6.19 scrub starts 2024-02-12T17:02:47.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:46.577040+0000 osd.5 (osd.5) 172 : cluster [DBG] 6.19 scrub ok 2024-02-12T17:02:47.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:46.606657+0000 mgr.y (mgr.24481) 500 : cluster [DBG] pgmap v647: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1017 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-12T17:02:47.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:47.017610+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:02:47.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:47.020839+0000 osd.4 (osd.4) 155 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:02:48.145 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:48.172 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:48.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:48.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:02:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:48.375 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:02:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:48.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:02:48.481 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:48.511 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 79 16 123091 123075 97.3447 70.6875 0.0240173 0.0102685 2024-02-12T17:02:48.524 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:48.555 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:48.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:48.651 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T17:02:48.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:47.729707+0000 osd.0 (osd.0) 163 : cluster [DBG] 2.1f scrub starts 2024-02-12T17:02:48.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:47.732286+0000 osd.0 (osd.0) 164 : cluster [DBG] 2.1f scrub ok 2024-02-12T17:02:48.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:48.240516+0000 mon.c (mon.2) 186 : audit [INF] from='client.? v1:172.21.15.33:0/1791519548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:02:48.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:48.240948+0000 mon.a (mon.0) 1321 : audit [INF] from='client.35486 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:02:48.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:48.499540+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? v1:172.21.15.33:0/1618478237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:02:48.658 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:48.661 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:02:48.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:47.729707+0000 osd.0 (osd.0) 163 : cluster [DBG] 2.1f scrub starts 2024-02-12T17:02:48.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:47.732286+0000 osd.0 (osd.0) 164 : cluster [DBG] 2.1f scrub ok 2024-02-12T17:02:48.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:48.240516+0000 mon.c (mon.2) 186 : audit [INF] from='client.? v1:172.21.15.33:0/1791519548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:02:48.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:48.240948+0000 mon.a (mon.0) 1321 : audit [INF] from='client.35486 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:02:48.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:48.499540+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? v1:172.21.15.33:0/1618478237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:02:48.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:47.729707+0000 osd.0 (osd.0) 163 : cluster [DBG] 2.1f scrub starts 2024-02-12T17:02:48.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:47.732286+0000 osd.0 (osd.0) 164 : cluster [DBG] 2.1f scrub ok 2024-02-12T17:02:48.716 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:48.240516+0000 mon.c (mon.2) 186 : audit [INF] from='client.? v1:172.21.15.33:0/1791519548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:02:48.716 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:48.240948+0000 mon.a (mon.0) 1321 : audit [INF] from='client.35486 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:02:48.716 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:48.499540+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? v1:172.21.15.33:0/1618478237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:02:48.718 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:49.512 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:02:49.511325+0000 min lat: 0.00147002 max lat: 0.119037 avg lat: 0.0102863 2024-02-12T17:02:49.512 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:02:49.512 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 80 15 124436 124421 97.1794 84.125 0.0102285 0.0102863 2024-02-12T17:02:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:48.607197+0000 mgr.y (mgr.24481) 501 : cluster [DBG] pgmap v648: 138 pgs: 138 active+clean; 7.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T17:02:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:48.614105+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:02:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:48.617909+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:02:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:48.633647+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:48.641860+0000 mon.a (mon.0) 1324 : audit [INF] from='client.35486 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:02:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:48.641933+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? v1:172.21.15.33:0/1618478237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:02:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:48.641973+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e332: 8 total, 8 up, 5 in 2024-02-12T17:02:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:48.607197+0000 mgr.y (mgr.24481) 501 : cluster [DBG] pgmap v648: 138 pgs: 138 active+clean; 7.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T17:02:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:48.614105+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:02:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:48.617909+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:02:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:48.633647+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:48.641860+0000 mon.a (mon.0) 1324 : audit [INF] from='client.35486 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:02:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:48.641933+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? v1:172.21.15.33:0/1618478237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:02:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:48.641973+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e332: 8 total, 8 up, 5 in 2024-02-12T17:02:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:48.607197+0000 mgr.y (mgr.24481) 501 : cluster [DBG] pgmap v648: 138 pgs: 138 active+clean; 7.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T17:02:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:48.614105+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:02:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:48.617909+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:02:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:48.633647+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:02:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:48.641860+0000 mon.a (mon.0) 1324 : audit [INF] from='client.35486 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:02:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:48.641933+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? v1:172.21.15.33:0/1618478237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:02:49.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:48.641973+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e332: 8 total, 8 up, 5 in 2024-02-12T17:02:50.182 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:50.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:50.288 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:50.342 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:50.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:50.484 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:02:50.513 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 81 16 125801 125785 97.0306 85.25 0.0216719 0.0103023 2024-02-12T17:02:50.535 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:50.673 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:49.798306+0000 osd.3 (osd.3) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:50.674 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:49.798382+0000 osd.3 (osd.3) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:50.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:49.798306+0000 osd.3 (osd.3) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:50.880 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:49.798382+0000 osd.3 (osd.3) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:50.880 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:49.798306+0000 osd.3 (osd.3) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:50.881 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:49.798382+0000 osd.3 (osd.3) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:51.024 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T17:02:51.025 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:02:51.100 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:51.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:51.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:51.258 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:51.279 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:02:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:50] "GET /metrics HTTP/1.1" 200 176059 "" "Prometheus/2.33.4" 2024-02-12T17:02:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:51.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:51.397 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:02:51.449 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:51.513 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 82 16 127028 127012 96.7824 76.6875 0.0114907 0.0103282 2024-02-12T17:02:51.614 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:02:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:51.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:02:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:50.607602+0000 mgr.y (mgr.24481) 502 : cluster [DBG] pgmap v650: 138 pgs: 138 active+clean; 7.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-12T17:02:51.676 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:50.669723+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e333: 8 total, 8 up, 5 in 2024-02-12T17:02:51.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:51.773 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:51.829 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:51.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:51.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:51.979 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:50.607602+0000 mgr.y (mgr.24481) 502 : cluster [DBG] pgmap v650: 138 pgs: 138 active+clean; 7.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-12T17:02:51.979 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:50.669723+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e333: 8 total, 8 up, 5 in 2024-02-12T17:02:51.982 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:50.607602+0000 mgr.y (mgr.24481) 502 : cluster [DBG] pgmap v650: 138 pgs: 138 active+clean; 7.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-12T17:02:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:50.669723+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e333: 8 total, 8 up, 5 in 2024-02-12T17:02:52.067 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:52.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:52.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:52.219 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:52.324 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:52.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:52.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:52.476 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:52.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:52.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:52.513 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 83 16 128369 128353 96.6261 83.8125 0.0152005 0.0103454 2024-02-12T17:02:52.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:52.665 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:52.694 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:52.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:52.741 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:02:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:02:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:02:52.799 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:02:52.846 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:52.928 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:52.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:53.035 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:53.075 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:53.159 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:02:53.159 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:02:53.159 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:02:53.159 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:02:53.159 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:02:53.452 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:02:53.513 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 84 16 129695 129679 96.4624 82.875 0.0183036 0.0103629 2024-02-12T17:02:53.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:52.608025+0000 mgr.y (mgr.24481) 503 : cluster [DBG] pgmap v652: 138 pgs: 138 active+clean; 7.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-02-12T17:02:53.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:53.412440+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.33:0/3218791291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:53.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:52.608025+0000 mgr.y (mgr.24481) 503 : cluster [DBG] pgmap v652: 138 pgs: 138 active+clean; 7.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-02-12T17:02:53.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:53.412440+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.33:0/3218791291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:53.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:52.608025+0000 mgr.y (mgr.24481) 503 : cluster [DBG] pgmap v652: 138 pgs: 138 active+clean; 7.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-02-12T17:02:53.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:53.412440+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.33:0/3218791291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:02:54.019 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T17:02:54.098 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:02:54.099 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:02:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:02:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:02:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:02:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:02:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:02:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:02:54.289 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:54.342 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:54.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:54.454 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:54.491 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:02:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:02:54.502 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:54.513 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 85 16 131003 130987 96.2892 81.75 0.0106011 0.0103818 2024-02-12T17:02:54.586 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:54.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:54.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:02:54.736 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:54.896 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:02:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:54.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:02:55.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:55.063 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:55.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:55.194 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:55.240 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:55.322 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:55.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:55.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:55.429 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:55.513 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 86 16 132350 132334 96.1485 84.1875 0.00939462 0.0103964 2024-02-12T17:02:55.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:54.168702+0000 mgr.y (mgr.24481) 504 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:55.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:54.608733+0000 mgr.y (mgr.24481) 505 : cluster [DBG] pgmap v653: 138 pgs: 138 active+clean; 7.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-12T17:02:55.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:54.945384+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:55.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:54.945444+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:54.168702+0000 mgr.y (mgr.24481) 504 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:54.608733+0000 mgr.y (mgr.24481) 505 : cluster [DBG] pgmap v653: 138 pgs: 138 active+clean; 7.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-12T17:02:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:54.945384+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:54.945444+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:54.168702+0000 mgr.y (mgr.24481) 504 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:02:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:54.608733+0000 mgr.y (mgr.24481) 505 : cluster [DBG] pgmap v653: 138 pgs: 138 active+clean; 7.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-12T17:02:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:54.945384+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:55.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:54.945444+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:56.435 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:56.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:56.446 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:56.513 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 87 16 133676 133660 95.9958 82.875 0.00748516 0.0104133 2024-02-12T17:02:56.547 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:02:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:56.739 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:56.742 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:02:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:56.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:56.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:56.898 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:56.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:55.686170+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e334: 8 total, 8 up, 5 in 2024-02-12T17:02:57.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:55.686170+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e334: 8 total, 8 up, 5 in 2024-02-12T17:02:57.008 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:55.686170+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e334: 8 total, 8 up, 5 in 2024-02-12T17:02:57.350 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:02:57.513 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 88 16 135004 134988 95.8481 83 0.01169 0.0104293 2024-02-12T17:02:57.701 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:56.609167+0000 mgr.y (mgr.24481) 506 : cluster [DBG] pgmap v655: 138 pgs: 138 active+clean; 7.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 642 B/s rd, 52 MiB/s wr, 828 op/s 2024-02-12T17:02:57.701 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:56.609167+0000 mgr.y (mgr.24481) 506 : cluster [DBG] pgmap v655: 138 pgs: 138 active+clean; 7.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 642 B/s rd, 52 MiB/s wr, 828 op/s 2024-02-12T17:02:57.770 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:02:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:57.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:02:57.882 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:02:57.934 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:57.969 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:02:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:57.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:02:57.975 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:56.609167+0000 mgr.y (mgr.24481) 506 : cluster [DBG] pgmap v655: 138 pgs: 138 active+clean; 7.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 642 B/s rd, 52 MiB/s wr, 828 op/s 2024-02-12T17:02:58.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:58.119 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:58.152 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:58.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:58.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:58.312 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:58.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:58.453 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:58.504 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:58.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 89 16 136309 136293 95.6875 81.5625 0.0100007 0.0104469 2024-02-12T17:02:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:58.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:58.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:58.689 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:02:58.692 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:02:58.702 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:58.735 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:58.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:58.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:57.671613+0000 mgr.y (mgr.24481) 507 : audit [DBG] from='client.25408 v1:172.21.15.33:0/4195017652' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:02:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:57.671613+0000 mgr.y (mgr.24481) 507 : audit [DBG] from='client.25408 v1:172.21.15.33:0/4195017652' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:02:58.842 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:02:58.885 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:58.931 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:02:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:58.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:02:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:57.671613+0000 mgr.y (mgr.24481) 507 : audit [DBG] from='client.25408 v1:172.21.15.33:0/4195017652' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:02:59.035 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:59.092 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:59.127 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:59.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:59.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:02:59.298 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:59.339 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:02:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:59.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:02:59.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:02:59.497 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total time run: 90.0033 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total writes made: 137587 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Write size: 65536 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object size: 65536 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Bandwidth (MB/sec): 95.543 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Bandwidth: 31.545 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max bandwidth (MB/sec): 230.938 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min bandwidth (MB/sec): 70.6875 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average IOPS: 1528 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev IOPS: 504.719 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max IOPS: 3695 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min IOPS: 1131 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average Latency(s): 0.0104633 2024-02-12T17:02:59.514 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Latency(s): 0.00555763 2024-02-12T17:02:59.515 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max latency(s): 0.119037 2024-02-12T17:02:59.515 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min latency(s): 0.00147002 2024-02-12T17:02:59.516 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:02:59.517 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T17:02:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:02:59.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:02:59.621 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:02:59.683 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:02:59.720 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:58.609881+0000 mgr.y (mgr.24481) 508 : cluster [DBG] pgmap v656: 138 pgs: 138 active+clean; 8.2 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-12T17:02:59.720 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:59.318546+0000 osd.2 (osd.2) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:59.720 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:02:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:59.318606+0000 osd.2 (osd.2) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:59.914 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2024-02-12T17:02:56.608983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8503695012,"num_objects":129948,"num_object_clones":0,"num_object_copies":389844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129948,"num_whiteouts":0,"num_read":1228,"num_read_kb":980,"num_write":130470,"num_write_kb":8304425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":6652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57750,"ondisk_log_size":57750,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":27391680,"kb_used_data":24727448,"kb_used_omap":0,"kb_used_meta":2664192,"kb_avail":441334080,"statfs":{"total":479975178240,"available":451926097920,"internally_reserved":0,"allocated":25320906752,"data_stored":25318781397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2728132608},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,7,7,8,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":432144384,"num_objects":6594,"num_object_clones":0,"num_object_copies":19782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6594,"num_write_kb":422016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.969190"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":235,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701643+0000","last_change":"2024-02-12T17:02:30.689748+0000","last_active":"2024-02-12T17:02:48.701643+0000","last_peered":"2024-02-12T17:02:48.701643+0000","last_clean":"2024-02-12T17:02:48.701643+0000","last_became_active":"2024-02-12T16:59:09.246207+0000","last_became_peered":"2024-02-12T16:59:09.246207+0000","last_unstale":"2024-02-12T17:02:48.701643+0000","last_undegraded":"2024-02-12T17:02:48.701643+0000","last_fullsized":"2024-02-12T17:02:48.701643+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.689645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.689645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":302,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700469+0000","last_change":"2024-02-12T17:02:47.732393+0000","last_active":"2024-02-12T17:02:48.700469+0000","last_peered":"2024-02-12T17:02:48.700469+0000","last_clean":"2024-02-12T17:02:48.700469+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T17:02:48.700469+0000","last_undegraded":"2024-02-12T17:02:48.700469+0000","last_fullsized":"2024-02-12T17:02:48.700469+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.732293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.732293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":268,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704416+0000","last_change":"2024-02-12T17:01:26.047598+0000","last_active":"2024-02-12T17:02:48.704416+0000","last_peered":"2024-02-12T17:02:48.704416+0000","last_clean":"2024-02-12T17:02:48.704416+0000","last_became_active":"2024-02-12T17:01:16.912117+0000","last_became_peered":"2024-02-12T17:01:16.912117+0000","last_unstale":"2024-02-12T17:02:48.704416+0000","last_undegraded":"2024-02-12T17:02:48.704416+0000","last_fullsized":"2024-02-12T17:02:48.704416+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:01:26.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.047527+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":266,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.707873+0000","last_change":"2024-02-12T17:00:56.892829+0000","last_active":"2024-02-12T17:02:48.707873+0000","last_peered":"2024-02-12T17:02:48.707873+0000","last_clean":"2024-02-12T17:02:48.707873+0000","last_became_active":"2024-02-12T16:57:47.188886+0000","last_became_peered":"2024-02-12T16:57:47.188886+0000","last_unstale":"2024-02-12T17:02:48.707873+0000","last_undegraded":"2024-02-12T17:02:48.707873+0000","last_fullsized":"2024-02-12T17:02:48.707873+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.892756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:56.892756+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.892756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":217,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.706660+0000","last_change":"2024-02-12T17:01:39.176533+0000","last_active":"2024-02-12T17:02:48.706660+0000","last_peered":"2024-02-12T17:02:48.706660+0000","last_clean":"2024-02-12T17:02:48.706660+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T17:02:48.706660+0000","last_undegraded":"2024-02-12T17:02:48.706660+0000","last_fullsized":"2024-02-12T17:02:48.706660+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.176431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":375,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.703640+0000","last_change":"2024-02-12T17:01:45.154273+0000","last_active":"2024-02-12T17:02:48.703640+0000","last_peered":"2024-02-12T17:02:48.703640+0000","last_clean":"2024-02-12T17:02:48.703640+0000","last_became_active":"2024-02-12T16:59:09.243960+0000","last_became_peered":"2024-02-12T16:59:09.243960+0000","last_unstale":"2024-02-12T17:02:48.703640+0000","last_undegraded":"2024-02-12T17:02:48.703640+0000","last_fullsized":"2024-02-12T17:02:48.703640+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:01:45.154164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":316,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.707682+0000","last_change":"2024-02-12T17:02:46.577138+0000","last_active":"2024-02-12T17:02:48.707682+0000","last_peered":"2024-02-12T17:02:48.707682+0000","last_clean":"2024-02-12T17:02:48.707682+0000","last_became_active":"2024-02-12T16:57:47.176448+0000","last_became_peered":"2024-02-12T16:57:47.176448+0000","last_unstale":"2024-02-12T17:02:48.707682+0000","last_undegraded":"2024-02-12T17:02:48.707682+0000","last_fullsized":"2024-02-12T17:02:48.707682+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:46.577048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:46.577048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":319,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701824+0000","last_change":"2024-02-12T17:01:37.813077+0000","last_active":"2024-02-12T17:02:48.701824+0000","last_peered":"2024-02-12T17:02:48.701824+0000","last_clean":"2024-02-12T17:02:48.701824+0000","last_became_active":"2024-02-12T16:59:09.248262+0000","last_became_peered":"2024-02-12T16:59:09.248262+0000","last_unstale":"2024-02-12T17:02:48.701824+0000","last_undegraded":"2024-02-12T17:02:48.701824+0000","last_fullsized":"2024-02-12T17:02:48.701824+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:37.813008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:01:37.813008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":151,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701494+0000","last_change":"2024-02-12T17:01:16.912824+0000","last_active":"2024-02-12T17:02:48.701494+0000","last_peered":"2024-02-12T17:02:48.701494+0000","last_clean":"2024-02-12T17:02:48.701494+0000","last_became_active":"2024-02-12T17:01:16.912468+0000","last_became_peered":"2024-02-12T17:01:16.912468+0000","last_unstale":"2024-02-12T17:02:48.701494+0000","last_undegraded":"2024-02-12T17:02:48.701494+0000","last_fullsized":"2024-02-12T17:02:48.701494+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":318,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.703931+0000","last_change":"2024-02-12T17:00:50.923564+0000","last_active":"2024-02-12T17:02:48.703931+0000","last_peered":"2024-02-12T17:02:48.703931+0000","last_clean":"2024-02-12T17:02:48.703931+0000","last_became_active":"2024-02-12T16:57:47.185903+0000","last_became_peered":"2024-02-12T16:57:47.185903+0000","last_unstale":"2024-02-12T17:02:48.703931+0000","last_undegraded":"2024-02-12T17:02:48.703931+0000","last_fullsized":"2024-02-12T17:02:48.703931+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:50.923462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.923462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":302,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701825+0000","last_change":"2024-02-12T17:02:34.502434+0000","last_active":"2024-02-12T17:02:48.701825+0000","last_peered":"2024-02-12T17:02:48.701825+0000","last_clean":"2024-02-12T17:02:48.701825+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:02:48.701825+0000","last_undegraded":"2024-02-12T17:02:48.701825+0000","last_fullsized":"2024-02-12T17:02:48.701825+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.502332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":259,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.703835+0000","last_change":"2024-02-12T16:57:18.865914+0000","last_active":"2024-02-12T17:02:48.703835+0000","last_peered":"2024-02-12T17:02:48.703835+0000","last_clean":"2024-02-12T17:02:48.703835+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T17:02:48.703835+0000","last_undegraded":"2024-02-12T17:02:48.703835+0000","last_fullsized":"2024-02-12T17:02:48.703835+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:18.865807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.865807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":181,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710914+0000","last_change":"2024-02-12T17:02:18.973581+0000","last_active":"2024-02-12T17:02:48.710914+0000","last_peered":"2024-02-12T17:02:48.710914+0000","last_clean":"2024-02-12T17:02:48.710914+0000","last_became_active":"2024-02-12T16:59:09.259293+0000","last_became_peered":"2024-02-12T16:59:09.259293+0000","last_unstale":"2024-02-12T17:02:48.710914+0000","last_undegraded":"2024-02-12T17:02:48.710914+0000","last_fullsized":"2024-02-12T17:02:48.710914+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.973480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.973480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":221,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701790+0000","last_change":"2024-02-12T16:59:09.276561+0000","last_active":"2024-02-12T17:02:48.701790+0000","last_peered":"2024-02-12T17:02:48.701790+0000","last_clean":"2024-02-12T17:02:48.701790+0000","last_became_active":"2024-02-12T16:59:09.272423+0000","last_became_peered":"2024-02-12T16:59:09.272423+0000","last_unstale":"2024-02-12T17:02:48.701790+0000","last_undegraded":"2024-02-12T17:02:48.701790+0000","last_fullsized":"2024-02-12T17:02:48.701790+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":224,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.705829+0000","last_change":"2024-02-12T17:02:40.949925+0000","last_active":"2024-02-12T17:02:48.705829+0000","last_peered":"2024-02-12T17:02:48.705829+0000","last_clean":"2024-02-12T17:02:48.705829+0000","last_became_active":"2024-02-12T16:59:09.249268+0000","last_became_peered":"2024-02-12T16:59:09.249268+0000","last_unstale":"2024-02-12T17:02:48.705829+0000","last_undegraded":"2024-02-12T17:02:48.705829+0000","last_fullsized":"2024-02-12T17:02:48.705829+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":317,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.706068+0000","last_change":"2024-02-12T17:02:21.149890+0000","last_active":"2024-02-12T17:02:48.706068+0000","last_peered":"2024-02-12T17:02:48.706068+0000","last_clean":"2024-02-12T17:02:48.706068+0000","last_became_active":"2024-02-12T16:59:09.270574+0000","last_became_peered":"2024-02-12T16:59:09.270574+0000","last_unstale":"2024-02-12T17:02:48.706068+0000","last_undegraded":"2024-02-12T17:02:48.706068+0000","last_fullsized":"2024-02-12T17:02:48.706068+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:21.149801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T17:02:21.149801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":92,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704269+0000","last_change":"2024-02-12T17:01:27.157444+0000","last_active":"2024-02-12T17:02:48.704269+0000","last_peered":"2024-02-12T17:02:48.704269+0000","last_clean":"2024-02-12T17:02:48.704269+0000","last_became_active":"2024-02-12T16:59:09.248342+0000","last_became_peered":"2024-02-12T16:59:09.248342+0000","last_unstale":"2024-02-12T17:02:48.704269+0000","last_undegraded":"2024-02-12T17:02:48.704269+0000","last_fullsized":"2024-02-12T17:02:48.704269+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.157357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.157357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":379,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700119+0000","last_change":"2024-02-12T17:01:16.911758+0000","last_active":"2024-02-12T17:02:48.700119+0000","last_peered":"2024-02-12T17:02:48.700119+0000","last_clean":"2024-02-12T17:02:48.700119+0000","last_became_active":"2024-02-12T17:01:16.911392+0000","last_became_peered":"2024-02-12T17:01:16.911392+0000","last_unstale":"2024-02-12T17:02:48.700119+0000","last_undegraded":"2024-02-12T17:02:48.700119+0000","last_fullsized":"2024-02-12T17:02:48.700119+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:57:24.750245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.750245+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":142,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710423+0000","last_change":"2024-02-12T17:02:37.745936+0000","last_active":"2024-02-12T17:02:48.710423+0000","last_peered":"2024-02-12T17:02:48.710423+0000","last_clean":"2024-02-12T17:02:48.710423+0000","last_became_active":"2024-02-12T16:57:47.178240+0000","last_became_peered":"2024-02-12T16:57:47.178240+0000","last_unstale":"2024-02-12T17:02:48.710423+0000","last_undegraded":"2024-02-12T17:02:48.710423+0000","last_fullsized":"2024-02-12T17:02:48.710423+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:37.745838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.745838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":95,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704293+0000","last_change":"2024-02-12T17:02:24.819770+0000","last_active":"2024-02-12T17:02:48.704293+0000","last_peered":"2024-02-12T17:02:48.704293+0000","last_clean":"2024-02-12T17:02:48.704293+0000","last_became_active":"2024-02-12T16:59:09.241719+0000","last_became_peered":"2024-02-12T16:59:09.241719+0000","last_unstale":"2024-02-12T17:02:48.704293+0000","last_undegraded":"2024-02-12T17:02:48.704293+0000","last_fullsized":"2024-02-12T17:02:48.704293+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":147,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.711160+0000","last_change":"2024-02-12T17:02:34.738452+0000","last_active":"2024-02-12T17:02:48.711160+0000","last_peered":"2024-02-12T17:02:48.711160+0000","last_clean":"2024-02-12T17:02:48.711160+0000","last_became_active":"2024-02-12T16:59:09.259731+0000","last_became_peered":"2024-02-12T16:59:09.259731+0000","last_unstale":"2024-02-12T17:02:48.711160+0000","last_undegraded":"2024-02-12T17:02:48.711160+0000","last_fullsized":"2024-02-12T17:02:48.711160+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:34.738385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.738385+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":293,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.702319+0000","last_change":"2024-02-12T16:57:37.670804+0000","last_active":"2024-02-12T17:02:48.702319+0000","last_peered":"2024-02-12T17:02:48.702319+0000","last_clean":"2024-02-12T17:02:48.702319+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T17:02:48.702319+0000","last_undegraded":"2024-02-12T17:02:48.702319+0000","last_fullsized":"2024-02-12T17:02:48.702319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:57:37.670713+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.670713+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":299,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.707977+0000","last_change":"2024-02-12T16:57:26.790389+0000","last_active":"2024-02-12T17:02:48.707977+0000","last_peered":"2024-02-12T17:02:48.707977+0000","last_clean":"2024-02-12T17:02:48.707977+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T17:02:48.707977+0000","last_undegraded":"2024-02-12T17:02:48.707977+0000","last_fullsized":"2024-02-12T17:02:48.707977+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":372,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.706752+0000","last_change":"2024-02-12T17:01:42.076658+0000","last_active":"2024-02-12T17:02:48.706752+0000","last_peered":"2024-02-12T17:02:48.706752+0000","last_clean":"2024-02-12T17:02:48.706752+0000","last_became_active":"2024-02-12T16:57:47.189652+0000","last_became_peered":"2024-02-12T16:57:47.189652+0000","last_unstale":"2024-02-12T17:02:48.706752+0000","last_undegraded":"2024-02-12T17:02:48.706752+0000","last_fullsized":"2024-02-12T17:02:48.706752+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:01:42.076584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.076584+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":221,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704102+0000","last_change":"2024-02-12T16:59:09.264851+0000","last_active":"2024-02-12T17:02:48.704102+0000","last_peered":"2024-02-12T17:02:48.704102+0000","last_clean":"2024-02-12T17:02:48.704102+0000","last_became_active":"2024-02-12T16:59:09.264513+0000","last_became_peered":"2024-02-12T16:59:09.264513+0000","last_unstale":"2024-02-12T17:02:48.704102+0000","last_undegraded":"2024-02-12T17:02:48.704102+0000","last_fullsized":"2024-02-12T17:02:48.704102+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":302,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.698612+0000","last_change":"2024-02-12T17:00:51.269702+0000","last_active":"2024-02-12T17:02:48.698612+0000","last_peered":"2024-02-12T17:02:48.698612+0000","last_clean":"2024-02-12T17:02:48.698612+0000","last_became_active":"2024-02-12T16:59:09.270516+0000","last_became_peered":"2024-02-12T16:59:09.270516+0000","last_unstale":"2024-02-12T17:02:48.698612+0000","last_undegraded":"2024-02-12T17:02:48.698612+0000","last_fullsized":"2024-02-12T17:02:48.698612+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:51.269648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.269648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":87,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.711016+0000","last_change":"2024-02-12T16:59:09.260879+0000","last_active":"2024-02-12T17:02:48.711016+0000","last_peered":"2024-02-12T17:02:48.711016+0000","last_clean":"2024-02-12T17:02:48.711016+0000","last_became_active":"2024-02-12T16:59:09.259058+0000","last_became_peered":"2024-02-12T16:59:09.259058+0000","last_unstale":"2024-02-12T17:02:48.711016+0000","last_undegraded":"2024-02-12T17:02:48.711016+0000","last_fullsized":"2024-02-12T17:02:48.711016+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":293,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.699160+0000","last_change":"2024-02-12T17:02:39.569673+0000","last_active":"2024-02-12T17:02:48.699160+0000","last_peered":"2024-02-12T17:02:48.699160+0000","last_clean":"2024-02-12T17:02:48.699160+0000","last_became_active":"2024-02-12T16:59:09.267628+0000","last_became_peered":"2024-02-12T16:59:09.267628+0000","last_unstale":"2024-02-12T17:02:48.699160+0000","last_undegraded":"2024-02-12T17:02:48.699160+0000","last_fullsized":"2024-02-12T17:02:48.699160+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":291,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700967+0000","last_change":"2024-02-12T16:57:28.846844+0000","last_active":"2024-02-12T17:02:48.700967+0000","last_peered":"2024-02-12T17:02:48.700967+0000","last_clean":"2024-02-12T17:02:48.700967+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T17:02:48.700967+0000","last_undegraded":"2024-02-12T17:02:48.700967+0000","last_fullsized":"2024-02-12T17:02:48.700967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":97,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.705470+0000","last_change":"2024-02-12T17:02:47.020937+0000","last_active":"2024-02-12T17:02:48.705470+0000","last_peered":"2024-02-12T17:02:48.705470+0000","last_clean":"2024-02-12T17:02:48.705470+0000","last_became_active":"2024-02-12T16:59:09.247346+0000","last_became_peered":"2024-02-12T16:59:09.247346+0000","last_unstale":"2024-02-12T17:02:48.705470+0000","last_undegraded":"2024-02-12T17:02:48.705470+0000","last_fullsized":"2024-02-12T17:02:48.705470+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.020845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.020845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":308,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710816+0000","last_change":"2024-02-12T16:59:09.265096+0000","last_active":"2024-02-12T17:02:48.710816+0000","last_peered":"2024-02-12T17:02:48.710816+0000","last_clean":"2024-02-12T17:02:48.710816+0000","last_became_active":"2024-02-12T16:59:09.264558+0000","last_became_peered":"2024-02-12T16:59:09.264558+0000","last_unstale":"2024-02-12T17:02:48.710816+0000","last_undegraded":"2024-02-12T17:02:48.710816+0000","last_fullsized":"2024-02-12T17:02:48.710816+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":127,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.702626+0000","last_change":"2024-02-12T17:00:58.938867+0000","last_active":"2024-02-12T17:02:48.702626+0000","last_peered":"2024-02-12T17:02:48.702626+0000","last_clean":"2024-02-12T17:02:48.702626+0000","last_became_active":"2024-02-12T16:59:09.280196+0000","last_became_peered":"2024-02-12T16:59:09.280196+0000","last_unstale":"2024-02-12T17:02:48.702626+0000","last_undegraded":"2024-02-12T17:02:48.702626+0000","last_fullsized":"2024-02-12T17:02:48.702626+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.938805+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":217,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704266+0000","last_change":"2024-02-12T17:01:30.051615+0000","last_active":"2024-02-12T17:02:48.704266+0000","last_peered":"2024-02-12T17:02:48.704266+0000","last_clean":"2024-02-12T17:02:48.704266+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T17:02:48.704266+0000","last_undegraded":"2024-02-12T17:02:48.704266+0000","last_fullsized":"2024-02-12T17:02:48.704266+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.051528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.051528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":246,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.708336+0000","last_change":"2024-02-12T16:59:09.263555+0000","last_active":"2024-02-12T17:02:48.708336+0000","last_peered":"2024-02-12T17:02:48.708336+0000","last_clean":"2024-02-12T17:02:48.708336+0000","last_became_active":"2024-02-12T16:59:09.262674+0000","last_became_peered":"2024-02-12T16:59:09.262674+0000","last_unstale":"2024-02-12T17:02:48.708336+0000","last_undegraded":"2024-02-12T17:02:48.708336+0000","last_fullsized":"2024-02-12T17:02:48.708336+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":151,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710208+0000","last_change":"2024-02-12T17:01:02.858477+0000","last_active":"2024-02-12T17:02:48.710208+0000","last_peered":"2024-02-12T17:02:48.710208+0000","last_clean":"2024-02-12T17:02:48.710208+0000","last_became_active":"2024-02-12T16:59:09.257167+0000","last_became_peered":"2024-02-12T16:59:09.257167+0000","last_unstale":"2024-02-12T17:02:48.710208+0000","last_undegraded":"2024-02-12T17:02:48.710208+0000","last_fullsized":"2024-02-12T17:02:48.710208+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.858411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.858411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":383,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704135+0000","last_change":"2024-02-12T17:02:16.065061+0000","last_active":"2024-02-12T17:02:48.704135+0000","last_peered":"2024-02-12T17:02:48.704135+0000","last_clean":"2024-02-12T17:02:48.704135+0000","last_became_active":"2024-02-12T16:59:09.274991+0000","last_became_peered":"2024-02-12T16:59:09.274991+0000","last_unstale":"2024-02-12T17:02:48.704135+0000","last_undegraded":"2024-02-12T17:02:48.704135+0000","last_fullsized":"2024-02-12T17:02:48.704135+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:16.064966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.064966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":309,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700034+0000","last_change":"2024-02-12T17:00:58.019883+0000","last_active":"2024-02-12T17:02:48.700034+0000","last_peered":"2024-02-12T17:02:48.700034+0000","last_clean":"2024-02-12T17:02:48.700034+0000","last_became_active":"2024-02-12T16:59:09.269260+0000","last_became_peered":"2024-02-12T16:59:09.269260+0000","last_unstale":"2024-02-12T17:02:48.700034+0000","last_undegraded":"2024-02-12T17:02:48.700034+0000","last_fullsized":"2024-02-12T17:02:48.700034+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.019813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.019813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":242,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704187+0000","last_change":"2024-02-12T17:01:33.149239+0000","last_active":"2024-02-12T17:02:48.704187+0000","last_peered":"2024-02-12T17:02:48.704187+0000","last_clean":"2024-02-12T17:02:48.704187+0000","last_became_active":"2024-02-12T16:57:47.188909+0000","last_became_peered":"2024-02-12T16:57:47.188909+0000","last_unstale":"2024-02-12T17:02:48.704187+0000","last_undegraded":"2024-02-12T17:02:48.704187+0000","last_fullsized":"2024-02-12T17:02:48.704187+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:33.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:01:33.149132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":190,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.709990+0000","last_change":"2024-02-12T17:02:40.656607+0000","last_active":"2024-02-12T17:02:48.709990+0000","last_peered":"2024-02-12T17:02:48.709990+0000","last_clean":"2024-02-12T17:02:48.709990+0000","last_became_active":"2024-02-12T16:59:09.260045+0000","last_became_peered":"2024-02-12T16:59:09.260045+0000","last_unstale":"2024-02-12T17:02:48.709990+0000","last_undegraded":"2024-02-12T17:02:48.709990+0000","last_fullsized":"2024-02-12T17:02:48.709990+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":132,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.703589+0000","last_change":"2024-02-12T17:01:16.911465+0000","last_active":"2024-02-12T17:02:48.703589+0000","last_peered":"2024-02-12T17:02:48.703589+0000","last_clean":"2024-02-12T17:02:48.703589+0000","last_became_active":"2024-02-12T17:01:16.911137+0000","last_became_peered":"2024-02-12T17:01:16.911137+0000","last_unstale":"2024-02-12T17:02:48.703589+0000","last_undegraded":"2024-02-12T17:02:48.703589+0000","last_fullsized":"2024-02-12T17:02:48.703589+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:00:53.792980+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.792980+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":45,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710973+0000","last_change":"2024-02-12T17:02:36.771544+0000","last_active":"2024-02-12T17:02:48.710973+0000","last_peered":"2024-02-12T17:02:48.710973+0000","last_clean":"2024-02-12T17:02:48.710973+0000","last_became_active":"2024-02-12T17:01:16.909120+0000","last_became_peered":"2024-02-12T17:01:16.909120+0000","last_unstale":"2024-02-12T17:02:48.710973+0000","last_undegraded":"2024-02-12T17:02:48.710973+0000","last_fullsized":"2024-02-12T17:02:48.710973+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:36.771446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:36.771446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":373,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.677919+0000","last_change":"2024-02-12T17:02:17.689314+0000","last_active":"2024-02-12T17:02:50.677919+0000","last_peered":"2024-02-12T17:02:50.677919+0000","last_clean":"2024-02-12T17:02:50.677919+0000","last_became_active":"2024-02-12T16:57:47.185109+0000","last_became_peered":"2024-02-12T16:57:47.185109+0000","last_unstale":"2024-02-12T17:02:50.677919+0000","last_undegraded":"2024-02-12T17:02:50.677919+0000","last_fullsized":"2024-02-12T17:02:50.677919+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-12T17:02:17.689218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.689218+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":306,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704370+0000","last_change":"2024-02-12T17:01:02.953989+0000","last_active":"2024-02-12T17:02:48.704370+0000","last_peered":"2024-02-12T17:02:48.704370+0000","last_clean":"2024-02-12T17:02:48.704370+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T17:02:48.704370+0000","last_undegraded":"2024-02-12T17:02:48.704370+0000","last_fullsized":"2024-02-12T17:02:48.704370+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.953891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.953891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":360,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710164+0000","last_change":"2024-02-12T17:01:31.827739+0000","last_active":"2024-02-12T17:02:48.710164+0000","last_peered":"2024-02-12T17:02:48.710164+0000","last_clean":"2024-02-12T17:02:48.710164+0000","last_became_active":"2024-02-12T16:59:09.258515+0000","last_became_peered":"2024-02-12T16:59:09.258515+0000","last_unstale":"2024-02-12T17:02:48.710164+0000","last_undegraded":"2024-02-12T17:02:48.710164+0000","last_fullsized":"2024-02-12T17:02:48.710164+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:31.827632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:31.827632+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":300,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.706574+0000","last_change":"2024-02-12T16:57:25.835979+0000","last_active":"2024-02-12T17:02:48.706574+0000","last_peered":"2024-02-12T17:02:48.706574+0000","last_clean":"2024-02-12T17:02:48.706574+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T17:02:48.706574+0000","last_undegraded":"2024-02-12T17:02:48.706574+0000","last_fullsized":"2024-02-12T17:02:48.706574+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.835883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.835883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"208'9","reported_seq":477,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.683267+0000","last_change":"2024-02-12T17:00:49.846525+0000","last_active":"2024-02-12T17:02:50.683267+0000","last_peered":"2024-02-12T17:02:50.683267+0000","last_clean":"2024-02-12T17:02:50.683267+0000","last_became_active":"2024-02-12T16:57:47.178688+0000","last_became_peered":"2024-02-12T16:57:47.178688+0000","last_unstale":"2024-02-12T17:02:50.683267+0000","last_undegraded":"2024-02-12T17:02:50.683267+0000","last_fullsized":"2024-02-12T17:02:50.683267+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'9","last_scrub_stamp":"2024-02-12T17:00:49.846432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.846432+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":308,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.708026+0000","last_change":"2024-02-12T17:01:34.874427+0000","last_active":"2024-02-12T17:02:48.708026+0000","last_peered":"2024-02-12T17:02:48.708026+0000","last_clean":"2024-02-12T17:02:48.708026+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T17:02:48.708026+0000","last_undegraded":"2024-02-12T17:02:48.708026+0000","last_fullsized":"2024-02-12T17:02:48.708026+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:34.874363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:34.874363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":191,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701906+0000","last_change":"2024-02-12T16:57:48.058146+0000","last_active":"2024-02-12T17:02:48.701906+0000","last_peered":"2024-02-12T17:02:48.701906+0000","last_clean":"2024-02-12T17:02:48.701906+0000","last_became_active":"2024-02-12T16:57:47.199140+0000","last_became_peered":"2024-02-12T16:57:47.199140+0000","last_unstale":"2024-02-12T17:02:48.701906+0000","last_undegraded":"2024-02-12T17:02:48.701906+0000","last_fullsized":"2024-02-12T17:02:48.701906+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":304,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701816+0000","last_change":"2024-02-12T17:00:58.187842+0000","last_active":"2024-02-12T17:02:48.701816+0000","last_peered":"2024-02-12T17:02:48.701816+0000","last_clean":"2024-02-12T17:02:48.701816+0000","last_became_active":"2024-02-12T16:59:09.269414+0000","last_became_peered":"2024-02-12T16:59:09.269414+0000","last_unstale":"2024-02-12T17:02:48.701816+0000","last_undegraded":"2024-02-12T17:02:48.701816+0000","last_fullsized":"2024-02-12T17:02:48.701816+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.187738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":265,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.705689+0000","last_change":"2024-02-12T17:01:32.127692+0000","last_active":"2024-02-12T17:02:48.705689+0000","last_peered":"2024-02-12T17:02:48.705689+0000","last_clean":"2024-02-12T17:02:48.705689+0000","last_became_active":"2024-02-12T16:59:09.243603+0000","last_became_peered":"2024-02-12T16:59:09.243603+0000","last_unstale":"2024-02-12T17:02:48.705689+0000","last_undegraded":"2024-02-12T17:02:48.705689+0000","last_fullsized":"2024-02-12T17:02:48.705689+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:32.127593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:32.127593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":284,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.703021+0000","last_change":"2024-02-12T17:00:53.269198+0000","last_active":"2024-02-12T17:02:48.703021+0000","last_peered":"2024-02-12T17:02:48.703021+0000","last_clean":"2024-02-12T17:02:48.703021+0000","last_became_active":"2024-02-12T16:59:09.242413+0000","last_became_peered":"2024-02-12T16:59:09.242413+0000","last_unstale":"2024-02-12T17:02:48.703021+0000","last_undegraded":"2024-02-12T17:02:48.703021+0000","last_fullsized":"2024-02-12T17:02:48.703021+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-02-12T17:00:53.269107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.269107+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":47,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704720+0000","last_change":"2024-02-12T17:02:43.982708+0000","last_active":"2024-02-12T17:02:48.704720+0000","last_peered":"2024-02-12T17:02:48.704720+0000","last_clean":"2024-02-12T17:02:48.704720+0000","last_became_active":"2024-02-12T17:01:16.910812+0000","last_became_peered":"2024-02-12T17:01:16.910812+0000","last_unstale":"2024-02-12T17:02:48.704720+0000","last_undegraded":"2024-02-12T17:02:48.704720+0000","last_fullsized":"2024-02-12T17:02:48.704720+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:43.982623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.982623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":211,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.703710+0000","last_change":"2024-02-12T17:01:16.912921+0000","last_active":"2024-02-12T17:02:48.703710+0000","last_peered":"2024-02-12T17:02:48.703710+0000","last_clean":"2024-02-12T17:02:48.703710+0000","last_became_active":"2024-02-12T17:01:16.911289+0000","last_became_peered":"2024-02-12T17:01:16.911289+0000","last_unstale":"2024-02-12T17:02:48.703710+0000","last_undegraded":"2024-02-12T17:02:48.703710+0000","last_fullsized":"2024-02-12T17:02:48.703710+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.824194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.824194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":343,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.706658+0000","last_change":"2024-02-12T17:00:53.936787+0000","last_active":"2024-02-12T17:02:48.706658+0000","last_peered":"2024-02-12T17:02:48.706658+0000","last_clean":"2024-02-12T17:02:48.706658+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T17:02:48.706658+0000","last_undegraded":"2024-02-12T17:02:48.706658+0000","last_fullsized":"2024-02-12T17:02:48.706658+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:00:53.936664+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.936664+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"333'8082","reported_seq":8119,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.568596+0000","last_change":"2024-02-12T17:01:13.881464+0000","last_active":"2024-02-12T17:02:53.568596+0000","last_peered":"2024-02-12T17:02:53.568596+0000","last_clean":"2024-02-12T17:02:53.568596+0000","last_became_active":"2024-02-12T17:01:13.881178+0000","last_became_peered":"2024-02-12T17:01:13.881178+0000","last_unstale":"2024-02-12T17:02:53.568596+0000","last_undegraded":"2024-02-12T17:02:53.568596+0000","last_fullsized":"2024-02-12T17:02:53.568596+0000","mapping_epoch":304,"log_start":"323'4600","ondisk_log_start":"323'4600","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3482,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":529661952,"num_objects":8082,"num_object_clones":0,"num_object_copies":24246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":517248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":230,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701740+0000","last_change":"2024-02-12T17:01:28.912974+0000","last_active":"2024-02-12T17:02:48.701740+0000","last_peered":"2024-02-12T17:02:48.701740+0000","last_clean":"2024-02-12T17:02:48.701740+0000","last_became_active":"2024-02-12T16:59:09.272706+0000","last_became_peered":"2024-02-12T16:59:09.272706+0000","last_unstale":"2024-02-12T17:02:48.701740+0000","last_undegraded":"2024-02-12T17:02:48.701740+0000","last_fullsized":"2024-02-12T17:02:48.701740+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:28.912872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.912872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":329,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.705786+0000","last_change":"2024-02-12T17:02:17.066439+0000","last_active":"2024-02-12T17:02:48.705786+0000","last_peered":"2024-02-12T17:02:48.705786+0000","last_clean":"2024-02-12T17:02:48.705786+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T17:02:48.705786+0000","last_undegraded":"2024-02-12T17:02:48.705786+0000","last_fullsized":"2024-02-12T17:02:48.705786+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:02:17.066331+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.066331+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":203,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700041+0000","last_change":"2024-02-12T16:59:09.952265+0000","last_active":"2024-02-12T17:02:48.700041+0000","last_peered":"2024-02-12T17:02:48.700041+0000","last_clean":"2024-02-12T17:02:48.700041+0000","last_became_active":"2024-02-12T16:59:09.264482+0000","last_became_peered":"2024-02-12T16:59:09.264482+0000","last_unstale":"2024-02-12T17:02:48.700041+0000","last_undegraded":"2024-02-12T17:02:48.700041+0000","last_fullsized":"2024-02-12T17:02:48.700041+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:22.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.764984+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":253,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704007+0000","last_change":"2024-02-12T17:02:25.815917+0000","last_active":"2024-02-12T17:02:48.704007+0000","last_peered":"2024-02-12T17:02:48.704007+0000","last_clean":"2024-02-12T17:02:48.704007+0000","last_became_active":"2024-02-12T17:01:16.912888+0000","last_became_peered":"2024-02-12T17:01:16.912888+0000","last_unstale":"2024-02-12T17:02:48.704007+0000","last_undegraded":"2024-02-12T17:02:48.704007+0000","last_fullsized":"2024-02-12T17:02:48.704007+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:25.815816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.815816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"333'8218","reported_seq":8255,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.544447+0000","last_change":"2024-02-12T17:01:13.886736+0000","last_active":"2024-02-12T17:02:53.544447+0000","last_peered":"2024-02-12T17:02:53.544447+0000","last_clean":"2024-02-12T17:02:53.544447+0000","last_became_active":"2024-02-12T17:01:13.885880+0000","last_became_peered":"2024-02-12T17:01:13.885880+0000","last_unstale":"2024-02-12T17:02:53.544447+0000","last_undegraded":"2024-02-12T17:02:53.544447+0000","last_fullsized":"2024-02-12T17:02:53.544447+0000","mapping_epoch":304,"log_start":"323'4700","ondisk_log_start":"323'4700","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3518,"ondisk_log_size":3518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":538574848,"num_objects":8218,"num_object_clones":0,"num_object_copies":24654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8218,"num_write_kb":525952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":300,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700120+0000","last_change":"2024-02-12T16:57:29.934046+0000","last_active":"2024-02-12T17:02:48.700120+0000","last_peered":"2024-02-12T17:02:48.700120+0000","last_clean":"2024-02-12T17:02:48.700120+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T17:02:48.700120+0000","last_undegraded":"2024-02-12T17:02:48.700120+0000","last_fullsized":"2024-02-12T17:02:48.700120+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":142,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700378+0000","last_change":"2024-02-12T17:02:30.758581+0000","last_active":"2024-02-12T17:02:48.700378+0000","last_peered":"2024-02-12T17:02:48.700378+0000","last_clean":"2024-02-12T17:02:48.700378+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:02:48.700378+0000","last_undegraded":"2024-02-12T17:02:48.700378+0000","last_fullsized":"2024-02-12T17:02:48.700378+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.758520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.758520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":191,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.708414+0000","last_change":"2024-02-12T16:57:48.006795+0000","last_active":"2024-02-12T17:02:48.708414+0000","last_peered":"2024-02-12T17:02:48.708414+0000","last_clean":"2024-02-12T17:02:48.708414+0000","last_became_active":"2024-02-12T16:57:47.202531+0000","last_became_peered":"2024-02-12T16:57:47.202531+0000","last_unstale":"2024-02-12T17:02:48.708414+0000","last_undegraded":"2024-02-12T17:02:48.708414+0000","last_fullsized":"2024-02-12T17:02:48.708414+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"299'10","reported_seq":474,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.679327+0000","last_change":"2024-02-12T17:00:51.955785+0000","last_active":"2024-02-12T17:02:50.679327+0000","last_peered":"2024-02-12T17:02:50.679327+0000","last_clean":"2024-02-12T17:02:50.679327+0000","last_became_active":"2024-02-12T16:59:09.270667+0000","last_became_peered":"2024-02-12T16:59:09.270667+0000","last_unstale":"2024-02-12T17:02:50.679327+0000","last_undegraded":"2024-02-12T17:02:50.679327+0000","last_fullsized":"2024-02-12T17:02:50.679327+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"333'8091","reported_seq":8128,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.604749+0000","last_change":"2024-02-12T17:01:13.884406+0000","last_active":"2024-02-12T17:02:53.604749+0000","last_peered":"2024-02-12T17:02:53.604749+0000","last_clean":"2024-02-12T17:02:53.604749+0000","last_became_active":"2024-02-12T17:01:13.884065+0000","last_became_peered":"2024-02-12T17:01:13.884065+0000","last_unstale":"2024-02-12T17:02:53.604749+0000","last_undegraded":"2024-02-12T17:02:53.604749+0000","last_fullsized":"2024-02-12T17:02:53.604749+0000","mapping_epoch":304,"log_start":"323'4600","ondisk_log_start":"323'4600","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3491,"ondisk_log_size":3491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":530251776,"num_objects":8091,"num_object_clones":0,"num_object_copies":24273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8091,"num_write_kb":517824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":235,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.706318+0000","last_change":"2024-02-12T17:02:42.958589+0000","last_active":"2024-02-12T17:02:48.706318+0000","last_peered":"2024-02-12T17:02:48.706318+0000","last_clean":"2024-02-12T17:02:48.706318+0000","last_became_active":"2024-02-12T16:59:09.264617+0000","last_became_peered":"2024-02-12T16:59:09.264617+0000","last_unstale":"2024-02-12T17:02:48.706318+0000","last_undegraded":"2024-02-12T17:02:48.706318+0000","last_fullsized":"2024-02-12T17:02:48.706318+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":322,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.708107+0000","last_change":"2024-02-12T17:01:42.892672+0000","last_active":"2024-02-12T17:02:48.708107+0000","last_peered":"2024-02-12T17:02:48.708107+0000","last_clean":"2024-02-12T17:02:48.708107+0000","last_became_active":"2024-02-12T16:59:09.261764+0000","last_became_peered":"2024-02-12T16:59:09.261764+0000","last_unstale":"2024-02-12T17:02:48.708107+0000","last_undegraded":"2024-02-12T17:02:48.708107+0000","last_fullsized":"2024-02-12T17:02:48.708107+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:42.892563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.892563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"308'54","reported_seq":543,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.677838+0000","last_change":"2024-02-12T17:02:37.537794+0000","last_active":"2024-02-12T17:02:50.677838+0000","last_peered":"2024-02-12T17:02:50.677838+0000","last_clean":"2024-02-12T17:02:50.677838+0000","last_became_active":"2024-02-12T17:01:16.911487+0000","last_became_peered":"2024-02-12T17:01:16.911487+0000","last_unstale":"2024-02-12T17:02:50.677838+0000","last_undegraded":"2024-02-12T17:02:50.677838+0000","last_fullsized":"2024-02-12T17:02:50.677838+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":316,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.705149+0000","last_change":"2024-02-12T17:02:35.760851+0000","last_active":"2024-02-12T17:02:48.705149+0000","last_peered":"2024-02-12T17:02:48.705149+0000","last_clean":"2024-02-12T17:02:48.705149+0000","last_became_active":"2024-02-12T16:59:09.264694+0000","last_became_peered":"2024-02-12T16:59:09.264694+0000","last_unstale":"2024-02-12T17:02:48.705149+0000","last_undegraded":"2024-02-12T17:02:48.705149+0000","last_fullsized":"2024-02-12T17:02:48.705149+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"333'8209","reported_seq":8246,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.499265+0000","last_change":"2024-02-12T17:01:13.884788+0000","last_active":"2024-02-12T17:02:53.499265+0000","last_peered":"2024-02-12T17:02:53.499265+0000","last_clean":"2024-02-12T17:02:53.499265+0000","last_became_active":"2024-02-12T17:01:13.884280+0000","last_became_peered":"2024-02-12T17:01:13.884280+0000","last_unstale":"2024-02-12T17:02:53.499265+0000","last_undegraded":"2024-02-12T17:02:53.499265+0000","last_fullsized":"2024-02-12T17:02:53.499265+0000","mapping_epoch":304,"log_start":"323'4700","ondisk_log_start":"323'4700","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3509,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":537985024,"num_objects":8209,"num_object_clones":0,"num_object_copies":24627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8209,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":311,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.702004+0000","last_change":"2024-02-12T17:01:04.085939+0000","last_active":"2024-02-12T17:02:48.702004+0000","last_peered":"2024-02-12T17:02:48.702004+0000","last_clean":"2024-02-12T17:02:48.702004+0000","last_became_active":"2024-02-12T16:59:09.269488+0000","last_became_peered":"2024-02-12T16:59:09.269488+0000","last_unstale":"2024-02-12T17:02:48.702004+0000","last_undegraded":"2024-02-12T17:02:48.702004+0000","last_fullsized":"2024-02-12T17:02:48.702004+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:04.085827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:04.085827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":160,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:52.214555+0000","last_change":"2024-02-12T17:01:16.913133+0000","last_active":"2024-02-12T17:02:52.214555+0000","last_peered":"2024-02-12T17:02:52.214555+0000","last_clean":"2024-02-12T17:02:52.214555+0000","last_became_active":"2024-02-12T17:01:16.912041+0000","last_became_peered":"2024-02-12T17:01:16.912041+0000","last_unstale":"2024-02-12T17:02:52.214555+0000","last_undegraded":"2024-02-12T17:02:52.214555+0000","last_fullsized":"2024-02-12T17:02:52.214555+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:00:52.838852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:52.838852+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":103,"num_read_kb":98,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":286,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704507+0000","last_change":"2024-02-12T17:01:38.142697+0000","last_active":"2024-02-12T17:02:48.704507+0000","last_peered":"2024-02-12T17:02:48.704507+0000","last_clean":"2024-02-12T17:02:48.704507+0000","last_became_active":"2024-02-12T17:01:16.911675+0000","last_became_peered":"2024-02-12T17:01:16.911675+0000","last_unstale":"2024-02-12T17:02:48.704507+0000","last_undegraded":"2024-02-12T17:02:48.704507+0000","last_fullsized":"2024-02-12T17:02:48.704507+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:38.142602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:38.142602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":223,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.711061+0000","last_change":"2024-02-12T17:02:25.762303+0000","last_active":"2024-02-12T17:02:48.711061+0000","last_peered":"2024-02-12T17:02:48.711061+0000","last_clean":"2024-02-12T17:02:48.711061+0000","last_became_active":"2024-02-12T16:57:47.190968+0000","last_became_peered":"2024-02-12T16:57:47.190968+0000","last_unstale":"2024-02-12T17:02:48.711061+0000","last_undegraded":"2024-02-12T17:02:48.711061+0000","last_fullsized":"2024-02-12T17:02:48.711061+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":247,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710888+0000","last_change":"2024-02-12T17:01:16.909653+0000","last_active":"2024-02-12T17:02:48.710888+0000","last_peered":"2024-02-12T17:02:48.710888+0000","last_clean":"2024-02-12T17:02:48.710888+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:02:48.710888+0000","last_undegraded":"2024-02-12T17:02:48.710888+0000","last_fullsized":"2024-02-12T17:02:48.710888+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"333'8102","reported_seq":8136,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.621305+0000","last_change":"2024-02-12T17:01:13.886964+0000","last_active":"2024-02-12T17:02:53.621305+0000","last_peered":"2024-02-12T17:02:53.621305+0000","last_clean":"2024-02-12T17:02:53.621305+0000","last_became_active":"2024-02-12T17:01:13.886738+0000","last_became_peered":"2024-02-12T17:01:13.886738+0000","last_unstale":"2024-02-12T17:02:53.621305+0000","last_undegraded":"2024-02-12T17:02:53.621305+0000","last_fullsized":"2024-02-12T17:02:53.621305+0000","mapping_epoch":304,"log_start":"323'4600","ondisk_log_start":"323'4600","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3502,"ondisk_log_size":3502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":530972672,"num_objects":8102,"num_object_clones":0,"num_object_copies":24306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8102,"num_write_kb":518528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":311,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.711269+0000","last_change":"2024-02-12T17:00:57.950368+0000","last_active":"2024-02-12T17:02:48.711269+0000","last_peered":"2024-02-12T17:02:48.711269+0000","last_clean":"2024-02-12T17:02:48.711269+0000","last_became_active":"2024-02-12T16:57:47.191375+0000","last_became_peered":"2024-02-12T16:57:47.191375+0000","last_unstale":"2024-02-12T17:02:48.711269+0000","last_undegraded":"2024-02-12T17:02:48.711269+0000","last_fullsized":"2024-02-12T17:02:48.711269+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:57.950284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:00:57.950284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":221,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.708258+0000","last_change":"2024-02-12T16:59:09.263388+0000","last_active":"2024-02-12T17:02:48.708258+0000","last_peered":"2024-02-12T17:02:48.708258+0000","last_clean":"2024-02-12T17:02:48.708258+0000","last_became_active":"2024-02-12T16:59:09.262266+0000","last_became_peered":"2024-02-12T16:59:09.262266+0000","last_unstale":"2024-02-12T17:02:48.708258+0000","last_undegraded":"2024-02-12T17:02:48.708258+0000","last_fullsized":"2024-02-12T17:02:48.708258+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":299,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.709608+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:02:48.709608+0000","last_peered":"2024-02-12T17:02:48.709608+0000","last_clean":"2024-02-12T17:02:48.709608+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:02:48.709608+0000","last_undegraded":"2024-02-12T17:02:48.709608+0000","last_fullsized":"2024-02-12T17:02:48.709608+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":314,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.698509+0000","last_change":"2024-02-12T17:01:25.871008+0000","last_active":"2024-02-12T17:02:48.698509+0000","last_peered":"2024-02-12T17:02:48.698509+0000","last_clean":"2024-02-12T17:02:48.698509+0000","last_became_active":"2024-02-12T16:59:09.245635+0000","last_became_peered":"2024-02-12T16:59:09.245635+0000","last_unstale":"2024-02-12T17:02:48.698509+0000","last_undegraded":"2024-02-12T17:02:48.698509+0000","last_fullsized":"2024-02-12T17:02:48.698509+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:25.870916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:01:25.870916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":160,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.705644+0000","last_change":"2024-02-12T17:01:43.163816+0000","last_active":"2024-02-12T17:02:48.705644+0000","last_peered":"2024-02-12T17:02:48.705644+0000","last_clean":"2024-02-12T17:02:48.705644+0000","last_became_active":"2024-02-12T16:59:09.249903+0000","last_became_peered":"2024-02-12T16:59:09.249903+0000","last_unstale":"2024-02-12T17:02:48.705644+0000","last_undegraded":"2024-02-12T17:02:48.705644+0000","last_fullsized":"2024-02-12T17:02:48.705644+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'26","last_scrub_stamp":"2024-02-12T17:01:43.163724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:43.163724+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":293,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.702026+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T17:02:48.702026+0000","last_peered":"2024-02-12T17:02:48.702026+0000","last_clean":"2024-02-12T17:02:48.702026+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:02:48.702026+0000","last_undegraded":"2024-02-12T17:02:48.702026+0000","last_fullsized":"2024-02-12T17:02:48.702026+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"333'8119","reported_seq":8156,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.591472+0000","last_change":"2024-02-12T17:01:13.885922+0000","last_active":"2024-02-12T17:02:53.591472+0000","last_peered":"2024-02-12T17:02:53.591472+0000","last_clean":"2024-02-12T17:02:53.591472+0000","last_became_active":"2024-02-12T17:01:13.885633+0000","last_became_peered":"2024-02-12T17:01:13.885633+0000","last_unstale":"2024-02-12T17:02:53.591472+0000","last_undegraded":"2024-02-12T17:02:53.591472+0000","last_fullsized":"2024-02-12T17:02:53.591472+0000","mapping_epoch":304,"log_start":"323'4600","ondisk_log_start":"323'4600","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3519,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":532086784,"num_objects":8119,"num_object_clones":0,"num_object_copies":24357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8119,"num_write_kb":519616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":239,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.709661+0000","last_change":"2024-02-12T16:59:09.252727+0000","last_active":"2024-02-12T17:02:48.709661+0000","last_peered":"2024-02-12T17:02:48.709661+0000","last_clean":"2024-02-12T17:02:48.709661+0000","last_became_active":"2024-02-12T16:59:09.252442+0000","last_became_peered":"2024-02-12T16:59:09.252442+0000","last_unstale":"2024-02-12T17:02:48.709661+0000","last_undegraded":"2024-02-12T17:02:48.709661+0000","last_fullsized":"2024-02-12T17:02:48.709661+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":306,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704178+0000","last_change":"2024-02-12T17:00:54.941856+0000","last_active":"2024-02-12T17:02:48.704178+0000","last_peered":"2024-02-12T17:02:48.704178+0000","last_clean":"2024-02-12T17:02:48.704178+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T17:02:48.704178+0000","last_undegraded":"2024-02-12T17:02:48.704178+0000","last_fullsized":"2024-02-12T17:02:48.704178+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.941776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":316,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.698454+0000","last_change":"2024-02-12T16:57:48.350132+0000","last_active":"2024-02-12T17:02:48.698454+0000","last_peered":"2024-02-12T17:02:48.698454+0000","last_clean":"2024-02-12T17:02:48.698454+0000","last_became_active":"2024-02-12T16:57:47.200347+0000","last_became_peered":"2024-02-12T16:57:47.200347+0000","last_unstale":"2024-02-12T17:02:48.698454+0000","last_undegraded":"2024-02-12T17:02:48.698454+0000","last_fullsized":"2024-02-12T17:02:48.698454+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":354,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.706585+0000","last_change":"2024-02-12T16:57:47.204108+0000","last_active":"2024-02-12T17:02:48.706585+0000","last_peered":"2024-02-12T17:02:48.706585+0000","last_clean":"2024-02-12T17:02:48.706585+0000","last_became_active":"2024-02-12T16:57:47.184456+0000","last_became_peered":"2024-02-12T16:57:47.184456+0000","last_unstale":"2024-02-12T17:02:48.706585+0000","last_undegraded":"2024-02-12T17:02:48.706585+0000","last_fullsized":"2024-02-12T17:02:48.706585+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":322,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.711212+0000","last_change":"2024-02-12T17:02:33.737620+0000","last_active":"2024-02-12T17:02:48.711212+0000","last_peered":"2024-02-12T17:02:48.711212+0000","last_clean":"2024-02-12T17:02:48.711212+0000","last_became_active":"2024-02-12T16:58:14.391221+0000","last_became_peered":"2024-02-12T16:58:14.391221+0000","last_unstale":"2024-02-12T17:02:48.711212+0000","last_undegraded":"2024-02-12T17:02:48.711212+0000","last_fullsized":"2024-02-12T17:02:48.711212+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"333'8214","reported_seq":8251,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.530439+0000","last_change":"2024-02-12T17:01:13.890189+0000","last_active":"2024-02-12T17:02:53.530439+0000","last_peered":"2024-02-12T17:02:53.530439+0000","last_clean":"2024-02-12T17:02:53.530439+0000","last_became_active":"2024-02-12T17:01:13.889786+0000","last_became_peered":"2024-02-12T17:01:13.889786+0000","last_unstale":"2024-02-12T17:02:53.530439+0000","last_undegraded":"2024-02-12T17:02:53.530439+0000","last_fullsized":"2024-02-12T17:02:53.530439+0000","mapping_epoch":304,"log_start":"323'4600","ondisk_log_start":"323'4600","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3614,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":538312704,"num_objects":8214,"num_object_clones":0,"num_object_copies":24642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":50,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.699037+0000","last_change":"2024-02-12T17:02:48.617976+0000","last_active":"2024-02-12T17:02:48.699037+0000","last_peered":"2024-02-12T17:02:48.699037+0000","last_clean":"2024-02-12T17:02:48.699037+0000","last_became_active":"2024-02-12T17:01:16.910938+0000","last_became_peered":"2024-02-12T17:01:16.910938+0000","last_unstale":"2024-02-12T17:02:48.699037+0000","last_undegraded":"2024-02-12T17:02:48.699037+0000","last_fullsized":"2024-02-12T17:02:48.699037+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:48.617913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:02:48.617913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":302,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.702022+0000","last_change":"2024-02-12T17:00:56.213043+0000","last_active":"2024-02-12T17:02:48.702022+0000","last_peered":"2024-02-12T17:02:48.702022+0000","last_clean":"2024-02-12T17:02:48.702022+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T17:02:48.702022+0000","last_undegraded":"2024-02-12T17:02:48.702022+0000","last_fullsized":"2024-02-12T17:02:48.702022+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.212949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.212949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":353,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704285+0000","last_change":"2024-02-12T17:02:44.688499+0000","last_active":"2024-02-12T17:02:48.704285+0000","last_peered":"2024-02-12T17:02:48.704285+0000","last_clean":"2024-02-12T17:02:48.704285+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T17:02:48.704285+0000","last_undegraded":"2024-02-12T17:02:48.704285+0000","last_fullsized":"2024-02-12T17:02:48.704285+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:02:44.688402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:44.688402+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":87,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.703686+0000","last_change":"2024-02-12T16:59:09.271420+0000","last_active":"2024-02-12T17:02:48.703686+0000","last_peered":"2024-02-12T17:02:48.703686+0000","last_clean":"2024-02-12T17:02:48.703686+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:02:48.703686+0000","last_undegraded":"2024-02-12T17:02:48.703686+0000","last_fullsized":"2024-02-12T17:02:48.703686+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"333'7947","reported_seq":7984,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.486194+0000","last_change":"2024-02-12T17:01:13.886816+0000","last_active":"2024-02-12T17:02:53.486194+0000","last_peered":"2024-02-12T17:02:53.486194+0000","last_clean":"2024-02-12T17:02:53.486194+0000","last_became_active":"2024-02-12T17:01:13.886430+0000","last_became_peered":"2024-02-12T17:01:13.886430+0000","last_unstale":"2024-02-12T17:02:53.486194+0000","last_undegraded":"2024-02-12T17:02:53.486194+0000","last_fullsized":"2024-02-12T17:02:53.486194+0000","mapping_epoch":304,"log_start":"323'4400","ondisk_log_start":"323'4400","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3547,"ondisk_log_size":3547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":520814592,"num_objects":7947,"num_object_clones":0,"num_object_copies":23841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7947,"num_write_kb":508608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":177,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700347+0000","last_change":"2024-02-12T17:01:16.911456+0000","last_active":"2024-02-12T17:02:48.700347+0000","last_peered":"2024-02-12T17:02:48.700347+0000","last_clean":"2024-02-12T17:02:48.700347+0000","last_became_active":"2024-02-12T17:01:16.911041+0000","last_became_peered":"2024-02-12T17:01:16.911041+0000","last_unstale":"2024-02-12T17:02:48.700347+0000","last_undegraded":"2024-02-12T17:02:48.700347+0000","last_fullsized":"2024-02-12T17:02:48.700347+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":100,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.706137+0000","last_change":"2024-02-12T17:02:18.084576+0000","last_active":"2024-02-12T17:02:48.706137+0000","last_peered":"2024-02-12T17:02:48.706137+0000","last_clean":"2024-02-12T17:02:48.706137+0000","last_became_active":"2024-02-12T16:59:09.242505+0000","last_became_peered":"2024-02-12T16:59:09.242505+0000","last_unstale":"2024-02-12T17:02:48.706137+0000","last_undegraded":"2024-02-12T17:02:48.706137+0000","last_fullsized":"2024-02-12T17:02:48.706137+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":324,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.698650+0000","last_change":"2024-02-12T17:02:18.703238+0000","last_active":"2024-02-12T17:02:48.698650+0000","last_peered":"2024-02-12T17:02:48.698650+0000","last_clean":"2024-02-12T17:02:48.698650+0000","last_became_active":"2024-02-12T16:57:47.199763+0000","last_became_peered":"2024-02-12T16:57:47.199763+0000","last_unstale":"2024-02-12T17:02:48.698650+0000","last_undegraded":"2024-02-12T17:02:48.698650+0000","last_fullsized":"2024-02-12T17:02:48.698650+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":315,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.705896+0000","last_change":"2024-02-12T17:02:30.025410+0000","last_active":"2024-02-12T17:02:48.705896+0000","last_peered":"2024-02-12T17:02:48.705896+0000","last_clean":"2024-02-12T17:02:48.705896+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T17:02:48.705896+0000","last_undegraded":"2024-02-12T17:02:48.705896+0000","last_fullsized":"2024-02-12T17:02:48.705896+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"333'8217","reported_seq":8259,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.513630+0000","last_change":"2024-02-12T17:02:24.719511+0000","last_active":"2024-02-12T17:02:53.513630+0000","last_peered":"2024-02-12T17:02:53.513630+0000","last_clean":"2024-02-12T17:02:53.513630+0000","last_became_active":"2024-02-12T17:01:13.881372+0000","last_became_peered":"2024-02-12T17:01:13.881372+0000","last_unstale":"2024-02-12T17:02:53.513630+0000","last_undegraded":"2024-02-12T17:02:53.513630+0000","last_fullsized":"2024-02-12T17:02:53.513630+0000","mapping_epoch":304,"log_start":"322'4100","ondisk_log_start":"322'4100","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'5831","last_scrub_stamp":"2024-02-12T17:02:24.719414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.719414+0000","log_size":4117,"ondisk_log_size":4117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":538509312,"num_objects":8217,"num_object_clones":0,"num_object_copies":24651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8217,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":305,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701915+0000","last_change":"2024-02-12T16:59:09.340317+0000","last_active":"2024-02-12T17:02:48.701915+0000","last_peered":"2024-02-12T17:02:48.701915+0000","last_clean":"2024-02-12T17:02:48.701915+0000","last_became_active":"2024-02-12T16:59:09.246616+0000","last_became_peered":"2024-02-12T16:59:09.246616+0000","last_unstale":"2024-02-12T17:02:48.701915+0000","last_undegraded":"2024-02-12T17:02:48.701915+0000","last_fullsized":"2024-02-12T17:02:48.701915+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:57:19.955311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.955311+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":92,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.702951+0000","last_change":"2024-02-12T17:02:40.562525+0000","last_active":"2024-02-12T17:02:48.702951+0000","last_peered":"2024-02-12T17:02:48.702951+0000","last_clean":"2024-02-12T17:02:48.702951+0000","last_became_active":"2024-02-12T16:59:09.268028+0000","last_became_peered":"2024-02-12T16:59:09.268028+0000","last_unstale":"2024-02-12T17:02:48.702951+0000","last_undegraded":"2024-02-12T17:02:48.702951+0000","last_fullsized":"2024-02-12T17:02:48.702951+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.562425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.562425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":336,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.707572+0000","last_change":"2024-02-12T16:59:09.996418+0000","last_active":"2024-02-12T17:02:48.707572+0000","last_peered":"2024-02-12T17:02:48.707572+0000","last_clean":"2024-02-12T17:02:48.707572+0000","last_became_active":"2024-02-12T16:59:09.256058+0000","last_became_peered":"2024-02-12T16:59:09.256058+0000","last_unstale":"2024-02-12T17:02:48.707572+0000","last_undegraded":"2024-02-12T17:02:48.707572+0000","last_fullsized":"2024-02-12T17:02:48.707572+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"328'11","reported_seq":233,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.673301+0000","last_change":"2024-02-12T17:02:29.064318+0000","last_active":"2024-02-12T17:02:50.673301+0000","last_peered":"2024-02-12T17:02:50.673301+0000","last_clean":"2024-02-12T17:02:50.673301+0000","last_became_active":"2024-02-12T16:57:47.167628+0000","last_became_peered":"2024-02-12T16:57:47.167628+0000","last_unstale":"2024-02-12T17:02:50.673301+0000","last_undegraded":"2024-02-12T17:02:50.673301+0000","last_fullsized":"2024-02-12T17:02:50.673301+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"333'7968","reported_seq":8005,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.510278+0000","last_change":"2024-02-12T17:01:13.886956+0000","last_active":"2024-02-12T17:02:53.510278+0000","last_peered":"2024-02-12T17:02:53.510278+0000","last_clean":"2024-02-12T17:02:53.510278+0000","last_became_active":"2024-02-12T17:01:13.886648+0000","last_became_peered":"2024-02-12T17:01:13.886648+0000","last_unstale":"2024-02-12T17:02:53.510278+0000","last_undegraded":"2024-02-12T17:02:53.510278+0000","last_fullsized":"2024-02-12T17:02:53.510278+0000","mapping_epoch":304,"log_start":"323'4400","ondisk_log_start":"323'4400","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3568,"ondisk_log_size":3568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":522190848,"num_objects":7968,"num_object_clones":0,"num_object_copies":23904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":95,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.709569+0000","last_change":"2024-02-12T17:01:30.811252+0000","last_active":"2024-02-12T17:02:48.709569+0000","last_peered":"2024-02-12T17:02:48.709569+0000","last_clean":"2024-02-12T17:02:48.709569+0000","last_became_active":"2024-02-12T16:59:09.259654+0000","last_became_peered":"2024-02-12T16:59:09.259654+0000","last_unstale":"2024-02-12T17:02:48.709569+0000","last_undegraded":"2024-02-12T17:02:48.709569+0000","last_fullsized":"2024-02-12T17:02:48.709569+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.811153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.811153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"261'14","reported_seq":404,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.681921+0000","last_change":"2024-02-12T16:59:12.225158+0000","last_active":"2024-02-12T17:02:50.681921+0000","last_peered":"2024-02-12T17:02:50.681921+0000","last_clean":"2024-02-12T17:02:50.681921+0000","last_became_active":"2024-02-12T16:59:12.224830+0000","last_became_peered":"2024-02-12T16:59:12.224830+0000","last_unstale":"2024-02-12T17:02:50.681921+0000","last_undegraded":"2024-02-12T17:02:50.681921+0000","last_fullsized":"2024-02-12T17:02:50.681921+0000","mapping_epoch":260,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":188,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700329+0000","last_change":"2024-02-12T17:01:16.913111+0000","last_active":"2024-02-12T17:02:48.700329+0000","last_peered":"2024-02-12T17:02:48.700329+0000","last_clean":"2024-02-12T17:02:48.700329+0000","last_became_active":"2024-02-12T17:01:16.911732+0000","last_became_peered":"2024-02-12T17:01:16.911732+0000","last_unstale":"2024-02-12T17:02:48.700329+0000","last_undegraded":"2024-02-12T17:02:48.700329+0000","last_fullsized":"2024-02-12T17:02:48.700329+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":300,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710455+0000","last_change":"2024-02-12T16:59:09.788881+0000","last_active":"2024-02-12T17:02:48.710455+0000","last_peered":"2024-02-12T17:02:48.710455+0000","last_clean":"2024-02-12T17:02:48.710455+0000","last_became_active":"2024-02-12T16:59:09.277264+0000","last_became_peered":"2024-02-12T16:59:09.277264+0000","last_unstale":"2024-02-12T17:02:48.710455+0000","last_undegraded":"2024-02-12T17:02:48.710455+0000","last_fullsized":"2024-02-12T17:02:48.710455+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"333'7991","reported_seq":8028,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.530484+0000","last_change":"2024-02-12T17:01:16.911200+0000","last_active":"2024-02-12T17:02:53.530484+0000","last_peered":"2024-02-12T17:02:53.530484+0000","last_clean":"2024-02-12T17:02:53.530484+0000","last_became_active":"2024-02-12T17:01:16.910877+0000","last_became_peered":"2024-02-12T17:01:16.910877+0000","last_unstale":"2024-02-12T17:02:53.530484+0000","last_undegraded":"2024-02-12T17:02:53.530484+0000","last_fullsized":"2024-02-12T17:02:53.530484+0000","mapping_epoch":307,"log_start":"323'4400","ondisk_log_start":"323'4400","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3591,"ondisk_log_size":3591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":523698176,"num_objects":7991,"num_object_clones":0,"num_object_copies":23973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":511424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":307,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710373+0000","last_change":"2024-02-12T17:02:41.672588+0000","last_active":"2024-02-12T17:02:48.710373+0000","last_peered":"2024-02-12T17:02:48.710373+0000","last_clean":"2024-02-12T17:02:48.710373+0000","last_became_active":"2024-02-12T16:57:47.177329+0000","last_became_peered":"2024-02-12T16:57:47.177329+0000","last_unstale":"2024-02-12T17:02:48.710373+0000","last_undegraded":"2024-02-12T17:02:48.710373+0000","last_fullsized":"2024-02-12T17:02:48.710373+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"208'12","reported_seq":454,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.681452+0000","last_change":"2024-02-12T17:02:20.866677+0000","last_active":"2024-02-12T17:02:50.681452+0000","last_peered":"2024-02-12T17:02:50.681452+0000","last_clean":"2024-02-12T17:02:50.681452+0000","last_became_active":"2024-02-12T16:57:46.177517+0000","last_became_peered":"2024-02-12T16:57:46.177517+0000","last_unstale":"2024-02-12T17:02:50.681452+0000","last_undegraded":"2024-02-12T17:02:50.681452+0000","last_fullsized":"2024-02-12T17:02:50.681452+0000","mapping_epoch":97,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-12T17:02:20.866591+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:02:20.866591+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":90,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.698962+0000","last_change":"2024-02-12T17:01:30.863612+0000","last_active":"2024-02-12T17:02:48.698962+0000","last_peered":"2024-02-12T17:02:48.698962+0000","last_clean":"2024-02-12T17:02:48.698962+0000","last_became_active":"2024-02-12T16:59:09.267755+0000","last_became_peered":"2024-02-12T16:59:09.267755+0000","last_unstale":"2024-02-12T17:02:48.698962+0000","last_undegraded":"2024-02-12T17:02:48.698962+0000","last_fullsized":"2024-02-12T17:02:48.698962+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.863530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.863530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":310,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.705256+0000","last_change":"2024-02-12T17:01:41.694955+0000","last_active":"2024-02-12T17:02:48.705256+0000","last_peered":"2024-02-12T17:02:48.705256+0000","last_clean":"2024-02-12T17:02:48.705256+0000","last_became_active":"2024-02-12T16:57:47.199008+0000","last_became_peered":"2024-02-12T16:57:47.199008+0000","last_unstale":"2024-02-12T17:02:48.705256+0000","last_undegraded":"2024-02-12T17:02:48.705256+0000","last_fullsized":"2024-02-12T17:02:48.705256+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:01:41.694875+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:01:41.694875+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"333'8086","reported_seq":8123,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.529344+0000","last_change":"2024-02-12T17:01:13.884499+0000","last_active":"2024-02-12T17:02:53.529344+0000","last_peered":"2024-02-12T17:02:53.529344+0000","last_clean":"2024-02-12T17:02:53.529344+0000","last_became_active":"2024-02-12T17:01:13.884154+0000","last_became_peered":"2024-02-12T17:01:13.884154+0000","last_unstale":"2024-02-12T17:02:53.529344+0000","last_undegraded":"2024-02-12T17:02:53.529344+0000","last_fullsized":"2024-02-12T17:02:53.529344+0000","mapping_epoch":304,"log_start":"323'4500","ondisk_log_start":"323'4500","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3586,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":529924096,"num_objects":8086,"num_object_clones":0,"num_object_copies":24258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":517504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710602+0000","last_change":"2024-02-12T17:01:16.909471+0000","last_active":"2024-02-12T17:02:48.710602+0000","last_peered":"2024-02-12T17:02:48.710602+0000","last_clean":"2024-02-12T17:02:48.710602+0000","last_became_active":"2024-02-12T17:01:16.909039+0000","last_became_peered":"2024-02-12T17:01:16.909039+0000","last_unstale":"2024-02-12T17:02:48.710602+0000","last_undegraded":"2024-02-12T17:02:48.710602+0000","last_fullsized":"2024-02-12T17:02:48.710602+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"308'11","reported_seq":81,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.678551+0000","last_change":"2024-02-12T17:02:31.547056+0000","last_active":"2024-02-12T17:02:50.678551+0000","last_peered":"2024-02-12T17:02:50.678551+0000","last_clean":"2024-02-12T17:02:50.678551+0000","last_became_active":"2024-02-12T17:01:16.912683+0000","last_became_peered":"2024-02-12T17:01:16.912683+0000","last_unstale":"2024-02-12T17:02:50.678551+0000","last_undegraded":"2024-02-12T17:02:50.678551+0000","last_fullsized":"2024-02-12T17:02:50.678551+0000","mapping_epoch":307,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'11","last_scrub_stamp":"2024-02-12T17:02:31.546965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:31.546965+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":246,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700651+0000","last_change":"2024-02-12T17:01:29.889009+0000","last_active":"2024-02-12T17:02:48.700651+0000","last_peered":"2024-02-12T17:02:48.700651+0000","last_clean":"2024-02-12T17:02:48.700651+0000","last_became_active":"2024-02-12T16:57:47.191231+0000","last_became_peered":"2024-02-12T16:57:47.191231+0000","last_unstale":"2024-02-12T17:02:48.700651+0000","last_undegraded":"2024-02-12T17:02:48.700651+0000","last_fullsized":"2024-02-12T17:02:48.700651+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.888942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.888942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":245,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701528+0000","last_change":"2024-02-12T17:02:16.750469+0000","last_active":"2024-02-12T17:02:48.701528+0000","last_peered":"2024-02-12T17:02:48.701528+0000","last_clean":"2024-02-12T17:02:48.701528+0000","last_became_active":"2024-02-12T16:59:09.268136+0000","last_became_peered":"2024-02-12T16:59:09.268136+0000","last_unstale":"2024-02-12T17:02:48.701528+0000","last_undegraded":"2024-02-12T17:02:48.701528+0000","last_fullsized":"2024-02-12T17:02:48.701528+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"333'8114","reported_seq":8151,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.621454+0000","last_change":"2024-02-12T17:01:16.913292+0000","last_active":"2024-02-12T17:02:53.621454+0000","last_peered":"2024-02-12T17:02:53.621454+0000","last_clean":"2024-02-12T17:02:53.621454+0000","last_became_active":"2024-02-12T17:01:16.912605+0000","last_became_peered":"2024-02-12T17:01:16.912605+0000","last_unstale":"2024-02-12T17:02:53.621454+0000","last_undegraded":"2024-02-12T17:02:53.621454+0000","last_fullsized":"2024-02-12T17:02:53.621454+0000","mapping_epoch":307,"log_start":"323'4600","ondisk_log_start":"323'4600","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3514,"ondisk_log_size":3514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":531759104,"num_objects":8114,"num_object_clones":0,"num_object_copies":24342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8114,"num_write_kb":519296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":269,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.710783+0000","last_change":"2024-02-12T17:01:16.909740+0000","last_active":"2024-02-12T17:02:48.710783+0000","last_peered":"2024-02-12T17:02:48.710783+0000","last_clean":"2024-02-12T17:02:48.710783+0000","last_became_active":"2024-02-12T17:01:16.909123+0000","last_became_peered":"2024-02-12T17:01:16.909123+0000","last_unstale":"2024-02-12T17:02:48.710783+0000","last_undegraded":"2024-02-12T17:02:48.710783+0000","last_fullsized":"2024-02-12T17:02:48.710783+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":169,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.707391+0000","last_change":"2024-02-12T17:01:29.153285+0000","last_active":"2024-02-12T17:02:48.707391+0000","last_peered":"2024-02-12T17:02:48.707391+0000","last_clean":"2024-02-12T17:02:48.707391+0000","last_became_active":"2024-02-12T16:59:09.247447+0000","last_became_peered":"2024-02-12T16:59:09.247447+0000","last_unstale":"2024-02-12T17:02:48.707391+0000","last_undegraded":"2024-02-12T17:02:48.707391+0000","last_fullsized":"2024-02-12T17:02:48.707391+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.153217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.153217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":302,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701691+0000","last_change":"2024-02-12T17:01:27.859045+0000","last_active":"2024-02-12T17:02:48.701691+0000","last_peered":"2024-02-12T17:02:48.701691+0000","last_clean":"2024-02-12T17:02:48.701691+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T17:02:48.701691+0000","last_undegraded":"2024-02-12T17:02:48.701691+0000","last_fullsized":"2024-02-12T17:02:48.701691+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.858955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.858955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":384,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704783+0000","last_change":"2024-02-12T17:01:35.160233+0000","last_active":"2024-02-12T17:02:48.704783+0000","last_peered":"2024-02-12T17:02:48.704783+0000","last_clean":"2024-02-12T17:02:48.704783+0000","last_became_active":"2024-02-12T17:01:16.911545+0000","last_became_peered":"2024-02-12T17:01:16.911545+0000","last_unstale":"2024-02-12T17:02:48.704783+0000","last_undegraded":"2024-02-12T17:02:48.704783+0000","last_fullsized":"2024-02-12T17:02:48.704783+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:01:35.160145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:35.160145+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"333'7963","reported_seq":8000,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.577675+0000","last_change":"2024-02-12T17:01:13.886968+0000","last_active":"2024-02-12T17:02:53.577675+0000","last_peered":"2024-02-12T17:02:53.577675+0000","last_clean":"2024-02-12T17:02:53.577675+0000","last_became_active":"2024-02-12T17:01:13.886370+0000","last_became_peered":"2024-02-12T17:01:13.886370+0000","last_unstale":"2024-02-12T17:02:53.577675+0000","last_undegraded":"2024-02-12T17:02:53.577675+0000","last_fullsized":"2024-02-12T17:02:53.577675+0000","mapping_epoch":304,"log_start":"323'4500","ondisk_log_start":"323'4500","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3463,"ondisk_log_size":3463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":521863168,"num_objects":7963,"num_object_clones":0,"num_object_copies":23889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7963,"num_write_kb":509632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":309,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.698371+0000","last_change":"2024-02-12T16:59:09.242082+0000","last_active":"2024-02-12T17:02:48.698371+0000","last_peered":"2024-02-12T17:02:48.698371+0000","last_clean":"2024-02-12T17:02:48.698371+0000","last_became_active":"2024-02-12T16:59:09.241648+0000","last_became_peered":"2024-02-12T16:59:09.241648+0000","last_unstale":"2024-02-12T17:02:48.698371+0000","last_undegraded":"2024-02-12T17:02:48.698371+0000","last_fullsized":"2024-02-12T17:02:48.698371+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":232,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704190+0000","last_change":"2024-02-12T17:01:16.911775+0000","last_active":"2024-02-12T17:02:48.704190+0000","last_peered":"2024-02-12T17:02:48.704190+0000","last_clean":"2024-02-12T17:02:48.704190+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:02:48.704190+0000","last_undegraded":"2024-02-12T17:02:48.704190+0000","last_fullsized":"2024-02-12T17:02:48.704190+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":37,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.709912+0000","last_change":"2024-02-12T17:01:16.909537+0000","last_active":"2024-02-12T17:02:48.709912+0000","last_peered":"2024-02-12T17:02:48.709912+0000","last_clean":"2024-02-12T17:02:48.709912+0000","last_became_active":"2024-02-12T17:01:16.909207+0000","last_became_peered":"2024-02-12T17:01:16.909207+0000","last_unstale":"2024-02-12T17:02:48.709912+0000","last_undegraded":"2024-02-12T17:02:48.709912+0000","last_fullsized":"2024-02-12T17:02:48.709912+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.794228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":179,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.700831+0000","last_change":"2024-02-12T17:02:45.594799+0000","last_active":"2024-02-12T17:02:48.700831+0000","last_peered":"2024-02-12T17:02:48.700831+0000","last_clean":"2024-02-12T17:02:48.700831+0000","last_became_active":"2024-02-12T16:59:09.273985+0000","last_became_peered":"2024-02-12T16:59:09.273985+0000","last_unstale":"2024-02-12T17:02:48.700831+0000","last_undegraded":"2024-02-12T17:02:48.700831+0000","last_fullsized":"2024-02-12T17:02:48.700831+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:02:45.594719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:45.594719+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"333'8183","reported_seq":8220,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.557760+0000","last_change":"2024-02-12T17:01:13.890011+0000","last_active":"2024-02-12T17:02:53.557760+0000","last_peered":"2024-02-12T17:02:53.557760+0000","last_clean":"2024-02-12T17:02:53.557760+0000","last_became_active":"2024-02-12T17:01:13.889691+0000","last_became_peered":"2024-02-12T17:01:13.889691+0000","last_unstale":"2024-02-12T17:02:53.557760+0000","last_undegraded":"2024-02-12T17:02:53.557760+0000","last_fullsized":"2024-02-12T17:02:53.557760+0000","mapping_epoch":304,"log_start":"323'4700","ondisk_log_start":"323'4700","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3483,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":536281088,"num_objects":8183,"num_object_clones":0,"num_object_copies":24549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8183,"num_write_kb":523712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":291,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704507+0000","last_change":"2024-02-12T17:00:56.986244+0000","last_active":"2024-02-12T17:02:48.704507+0000","last_peered":"2024-02-12T17:02:48.704507+0000","last_clean":"2024-02-12T17:02:48.704507+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T17:02:48.704507+0000","last_undegraded":"2024-02-12T17:02:48.704507+0000","last_fullsized":"2024-02-12T17:02:48.704507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.986137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.986137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":245,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704431+0000","last_change":"2024-02-12T17:01:39.076533+0000","last_active":"2024-02-12T17:02:48.704431+0000","last_peered":"2024-02-12T17:02:48.704431+0000","last_clean":"2024-02-12T17:02:48.704431+0000","last_became_active":"2024-02-12T17:01:16.910805+0000","last_became_peered":"2024-02-12T17:01:16.910805+0000","last_unstale":"2024-02-12T17:02:48.704431+0000","last_undegraded":"2024-02-12T17:02:48.704431+0000","last_fullsized":"2024-02-12T17:02:48.704431+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.076442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.076442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":212,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.701709+0000","last_change":"2024-02-12T17:01:16.912916+0000","last_active":"2024-02-12T17:02:48.701709+0000","last_peered":"2024-02-12T17:02:48.701709+0000","last_clean":"2024-02-12T17:02:48.701709+0000","last_became_active":"2024-02-12T17:01:16.912576+0000","last_became_peered":"2024-02-12T17:01:16.912576+0000","last_unstale":"2024-02-12T17:02:48.701709+0000","last_undegraded":"2024-02-12T17:02:48.701709+0000","last_fullsized":"2024-02-12T17:02:48.701709+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:00:50.756578+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.756578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":344,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.704461+0000","last_change":"2024-02-12T17:01:28.096786+0000","last_active":"2024-02-12T17:02:48.704461+0000","last_peered":"2024-02-12T17:02:48.704461+0000","last_clean":"2024-02-12T17:02:48.704461+0000","last_became_active":"2024-02-12T16:57:47.189672+0000","last_became_peered":"2024-02-12T16:57:47.189672+0000","last_unstale":"2024-02-12T17:02:48.704461+0000","last_undegraded":"2024-02-12T17:02:48.704461+0000","last_fullsized":"2024-02-12T17:02:48.704461+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:28.096708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.096708+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"333'8252","reported_seq":8288,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:53.614375+0000","last_change":"2024-02-12T17:01:13.886545+0000","last_active":"2024-02-12T17:02:53.614375+0000","last_peered":"2024-02-12T17:02:53.614375+0000","last_clean":"2024-02-12T17:02:53.614375+0000","last_became_active":"2024-02-12T17:01:13.886238+0000","last_became_peered":"2024-02-12T17:01:13.886238+0000","last_unstale":"2024-02-12T17:02:53.614375+0000","last_undegraded":"2024-02-12T17:02:53.614375+0000","last_fullsized":"2024-02-12T17:02:53.614375+0000","mapping_epoch":304,"log_start":"323'4800","ondisk_log_start":"323'4800","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3452,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":540803072,"num_objects":8252,"num_object_clones":0,"num_object_copies":24756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8252,"num_write_kb":528128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":302,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.707950+0000","last_change":"2024-02-12T17:01:26.804003+0000","last_active":"2024-02-12T17:02:48.707950+0000","last_peered":"2024-02-12T17:02:48.707950+0000","last_clean":"2024-02-12T17:02:48.707950+0000","last_became_active":"2024-02-12T16:59:09.261034+0000","last_became_peered":"2024-02-12T16:59:09.261034+0000","last_unstale":"2024-02-12T17:02:48.707950+0000","last_undegraded":"2024-02-12T17:02:48.707950+0000","last_fullsized":"2024-02-12T17:02:48.707950+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:26.803903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.803903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":275,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.703029+0000","last_change":"2024-02-12T17:01:16.911268+0000","last_active":"2024-02-12T17:02:48.703029+0000","last_peered":"2024-02-12T17:02:48.703029+0000","last_clean":"2024-02-12T17:02:48.703029+0000","last_became_active":"2024-02-12T17:01:16.910854+0000","last_became_peered":"2024-02-12T17:01:16.910854+0000","last_unstale":"2024-02-12T17:02:48.703029+0000","last_undegraded":"2024-02-12T17:02:48.703029+0000","last_fullsized":"2024-02-12T17:02:48.703029+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:01.789464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:01.789464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":377,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-12T17:02:48.707390+0000","last_change":"2024-02-12T17:02:43.665354+0000","last_active":"2024-02-12T17:02:48.707390+0000","last_peered":"2024-02-12T17:02:48.707390+0000","last_clean":"2024-02-12T17:02:48.707390+0000","last_became_active":"2024-02-12T16:59:09.280858+0000","last_became_peered":"2024-02-12T16:59:09.280858+0000","last_unstale":"2024-02-12T17:02:48.707390+0000","last_undegraded":"2024-02-12T17:02:48.707390+0000","last_fullsized":"2024-02-12T17:02:48.707390+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:02:43.665248+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.665248+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"261'13","reported_seq":506,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-02-12T17:02:50.680029+0000","last_change":"2024-02-12T17:00:49.304845+0000","last_active":"2024-02-12T17:02:50.680029+0000","last_peered":"2024-02-12T17:02:50.680029+0000","last_clean":"2024-02-12T17:02:50.680029+0000","last_became_active":"2024-02-12T16:59:12.242710+0000","last_became_peered":"2024-02-12T16:59:12.242710+0000","last_unstale":"2024-02-12T17:02:50.680029+0000","last_undegraded":"2024-02-12T17:02:50.680029+0000","last_fullsized":"2024-02-12T17:02:50.680029+0000","mapping_epoch":260,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-02-12T17:00:49.304784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.304784+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8503689216,"num_objects":129756,"num_object_clones":0,"num_object_copies":389268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129756,"num_write_kb":8304384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25510084608,"data_stored":25510084608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56956,"ondisk_log_size":56956,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1231,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":103,"num_read_kb":98,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1103,"num_read_kb":860,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":5408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":724,"ondisk_log_size":724,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234146,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5940860,"kb_used_data":5643636,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":87804292,"statfs":{"total":95995035648,"available":89911595008,"internally_reserved":0,"allocated":5779083264,"data_stored":5778636904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.8360000000000001,"15min":1.415},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.9939999999999998,"5min":16.759,"15min":36.439},"last":1.7629999999999999},{"interface":"front","average":{"1min":1.9770000000000001,"5min":2.7090000000000001,"15min":1.659},"min":{"1min":0.28299999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.7430000000000003,"5min":17.844000000000001,"15min":36.490000000000002},"last":0.58899999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.839,"15min":1.194},"min":{"1min":0.23400000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.7200000000000002,"5min":11.048999999999999,"15min":12.032},"last":0.98699999999999999},{"interface":"front","average":{"1min":2.1349999999999998,"5min":2.2269999999999999,"15min":1.3120000000000001},"min":{"1min":0.41399999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.9720000000000004,"5min":17.324999999999999,"15min":17.324999999999999},"last":0.91100000000000003}]},{"osd":2,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.464,"15min":1.0700000000000001},"min":{"1min":0.31900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.8150000000000004,"5min":9.7319999999999993,"15min":9.7319999999999993},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.5589999999999999,"5min":2.0529999999999999,"15min":1.3340000000000001},"min":{"1min":0.29999999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.6899999999999999,"5min":11.071999999999999,"15min":20.338000000000001},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.6200000000000001,"15min":1.3580000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":4.2039999999999997,"5min":9.7490000000000006,"15min":20.433},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.77,"5min":2.149,"15min":1.5529999999999999},"min":{"1min":0.374,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.8620000000000001,"5min":11.092000000000001,"15min":20.393000000000001},"last":1.02}]},{"osd":4,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.782,"5min":2.0169999999999999,"15min":1.2829999999999999},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.219},"max":{"1min":5.1040000000000001,"5min":9.7089999999999996,"15min":50.406999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.6419999999999999,"5min":2.806,"15min":1.577},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23400000000000001},"max":{"1min":5.8029999999999999,"5min":14.863,"15min":50.168999999999997},"last":1.0700000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4969999999999999,"5min":2.5960000000000001,"15min":1.258},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.24199999999999999},"max":{"1min":9.8439999999999994,"5min":10.91,"15min":10.91},"last":0.72999999999999998},{"interface":"front","average":{"1min":2.1259999999999999,"5min":3.0659999999999998,"15min":1.454},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.26200000000000001},"max":{"1min":6.0590000000000002,"5min":18.651,"15min":18.651},"last":0.34300000000000003}]},{"osd":6,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":2.5139999999999998,"5min":2.645,"15min":1.2969999999999999},"min":{"1min":0.37,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":5.9240000000000004,"5min":17.234999999999999,"15min":17.234999999999999},"last":0.37},{"interface":"front","average":{"1min":1.6859999999999999,"5min":2.1309999999999998,"15min":1.1850000000000001},"min":{"1min":0.435,"5min":0.33700000000000002,"15min":0.30199999999999999},"max":{"1min":5.0599999999999996,"5min":12.407999999999999,"15min":12.407999999999999},"last":0.435}]}]},{"osd":6,"up_from":0,"seq":193273528487,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738564,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6255860,"kb_used_data":5670252,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":87489292,"statfs":{"total":95995035648,"available":89589035008,"internally_reserved":0,"allocated":5806338048,"data_stored":5805898999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.008,"15min":1.1970000000000001},"min":{"1min":0.23400000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.2149999999999999,"5min":16.146999999999998,"15min":23.510999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.85899999999999999,"15min":1.1240000000000001},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":3.1200000000000001,"5min":5.0449999999999999,"15min":10.186999999999999},"last":0.82999999999999996}]},{"osd":2,"last update":"Mon Feb 12 17:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1419999999999999,"15min":1.345},"min":{"1min":0.26400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0980000000000001,"5min":11.122999999999999,"15min":16.521000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":1.0920000000000001,"15min":1.1890000000000001},"min":{"1min":0.22700000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.984,"5min":5.149,"15min":10.340999999999999},"last":0.90700000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.296,"15min":1.4410000000000001},"min":{"1min":0.28199999999999997,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":5.7380000000000004,"5min":16.27,"15min":21.100000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.363,"5min":1.2529999999999999,"15min":1.4490000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":8.0690000000000008,"5min":11.976000000000001,"15min":16.446000000000002},"last":1.288}]},{"osd":4,"last update":"Mon Feb 12 17:02:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.792,"15min":1.3740000000000001},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.524,"5min":16.888999999999999,"15min":16.888999999999999},"last":2.6520000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.9239999999999999,"15min":1.341},"min":{"1min":0.318,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":7.2850000000000001,"5min":16.899999999999999,"15min":16.899999999999999},"last":1.913}]},{"osd":5,"last update":"Mon Feb 12 17:02:10 2024","interfaces":[{"interface":"back","average":{"1min":3.9470000000000001,"5min":2.8730000000000002,"15min":1.776},"min":{"1min":0.309,"5min":0.30199999999999999,"15min":0.26400000000000001},"max":{"1min":50.404000000000003,"5min":50.404000000000003,"15min":50.404000000000003},"last":2.6259999999999999},{"interface":"front","average":{"1min":3.6579999999999999,"5min":2.653,"15min":1.6839999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":49.979999999999997,"5min":49.979999999999997,"15min":49.979999999999997},"last":1.675}]},{"osd":6,"last update":"Mon Feb 12 17:02:39 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.9119999999999999,"15min":1.419},"min":{"1min":0.378,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.4489999999999998,"5min":17.907,"15min":17.907},"last":1.458},{"interface":"front","average":{"1min":1.9990000000000001,"5min":2.274,"15min":1.6459999999999999},"min":{"1min":0.35099999999999998,"5min":0.33600000000000002,"15min":0.27900000000000003},"max":{"1min":5.883,"5min":16.294,"15min":21.062000000000001},"last":1.393}]},{"osd":7,"last update":"Mon Feb 12 17:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":2.0089999999999999,"15min":1.4330000000000001},"min":{"1min":0.499,"5min":0.39100000000000001,"15min":0.32400000000000001},"max":{"1min":3.2370000000000001,"5min":16.021000000000001,"15min":16.175000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.4079999999999999,"5min":2.5070000000000001,"15min":1.5109999999999999},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.33600000000000002},"max":{"1min":7.2400000000000002,"5min":34.722999999999999,"15min":34.722999999999999},"last":1.861}]}]},{"osd":2,"up_from":18,"seq":77309411513,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4733136,"kb_used_data":4139336,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":89012016,"statfs":{"total":95995035648,"available":91148304384,"internally_reserved":0,"allocated":4238680064,"data_stored":4238277665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":1.1140000000000001,"15min":1.3720000000000001},"min":{"1min":0.214,"5min":0.186,"15min":0.186},"max":{"1min":4.0970000000000004,"5min":13.459,"15min":15.808},"last":2.5329999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":1.2390000000000001,"15min":1.2529999999999999},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.158},"max":{"1min":4.6749999999999998,"5min":6.1070000000000002,"15min":19.893000000000001},"last":3.738}]},{"osd":1,"last update":"Mon Feb 12 17:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.254,"15min":1.4390000000000001},"min":{"1min":0.26000000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":5.0890000000000004,"5min":12.433999999999999,"15min":17.739000000000001},"last":1.7010000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.208,"15min":1.4650000000000001},"min":{"1min":0.26900000000000002,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":3.2130000000000001,"5min":8.2260000000000009,"15min":15.766999999999999},"last":2.4689999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:02:30 2024","interfaces":[{"interface":"back","average":{"1min":1.7589999999999999,"5min":1.1919999999999999,"15min":1.6659999999999999},"min":{"1min":0.35199999999999998,"5min":0.27000000000000002,"15min":0.23499999999999999},"max":{"1min":5.1239999999999997,"5min":5.1239999999999997,"15min":51.579999999999998},"last":4.1180000000000003},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.1579999999999999,"15min":1.399},"min":{"1min":0.311,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":3.5270000000000001,"5min":5.5800000000000001,"15min":48.988999999999997},"last":3.5779999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":2.0680000000000001,"15min":1.526},"min":{"1min":0.44700000000000001,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":2.6669999999999998,"5min":18.658999999999999,"15min":18.658999999999999},"last":2.5049999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":2.206,"15min":1.4550000000000001},"min":{"1min":0.36099999999999999,"5min":0.308,"15min":0.26800000000000002},"max":{"1min":1.4910000000000001,"5min":17.771000000000001,"15min":17.771000000000001},"last":3.6030000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.7709999999999999,"15min":1.554},"min":{"1min":0.40799999999999997,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":3.6899999999999999,"5min":10.407999999999999,"15min":15.699},"last":2.5569999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":2.3210000000000002,"15min":1.6879999999999999},"min":{"1min":0.36599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":8.9250000000000007,"5min":13.207000000000001,"15min":13.207000000000001},"last":1.774}]},{"osd":6,"last update":"Mon Feb 12 17:02:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":2.3359999999999999,"15min":1.778},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.1909999999999998,"5min":16.739000000000001,"15min":17.782},"last":1.73},{"interface":"front","average":{"1min":1.6739999999999999,"5min":2.1339999999999999,"15min":1.5509999999999999},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.29799999999999999},"max":{"1min":4.6390000000000002,"5min":8.9459999999999997,"15min":13.446999999999999},"last":3.6520000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":2.0710000000000002,"15min":1.585},"min":{"1min":0.39900000000000002,"5min":0.377,"15min":0.32600000000000001},"max":{"1min":2.1299999999999999,"5min":8.8520000000000003,"15min":13.473000000000001},"last":3.629},{"interface":"front","average":{"1min":1.1180000000000001,"5min":2.2400000000000002,"15min":1.6659999999999999},"min":{"1min":0.51800000000000002,"5min":0.36199999999999999,"15min":0.35499999999999998},"max":{"1min":5.1479999999999997,"5min":16.768000000000001,"15min":16.768000000000001},"last":1.6579999999999999}]}]},{"osd":3,"up_from":0,"seq":111669149877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953649,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3698252,"kb_used_data":3104452,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":90046900,"statfs":{"total":95995035648,"available":92208025600,"internally_reserved":0,"allocated":3178958848,"data_stored":3178559992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:02:41 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":2.1579999999999999,"15min":1.4610000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":7.0519999999999996,"5min":9.5090000000000003,"15min":38.179000000000002},"last":2.0819999999999999},{"interface":"front","average":{"1min":1.762,"5min":2.6000000000000001,"15min":1.6100000000000001},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":5.1180000000000003,"5min":20.244,"15min":38.151000000000003},"last":0.34300000000000003}]},{"osd":1,"last update":"Mon Feb 12 17:02:41 2024","interfaces":[{"interface":"back","average":{"1min":2.0590000000000002,"5min":2.3170000000000002,"15min":1.401},"min":{"1min":0.254,"5min":0.251,"15min":0.218},"max":{"1min":5.7839999999999998,"5min":11.077999999999999,"15min":11.077999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.9079999999999999,"5min":2.2200000000000002,"15min":1.3759999999999999},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.1520000000000001,"5min":9.5579999999999998,"15min":9.5579999999999998},"last":2.1349999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:02:41 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":2.1539999999999999,"15min":1.3740000000000001},"min":{"1min":0.25900000000000001,"5min":0.252,"15min":0.22500000000000001},"max":{"1min":6.1799999999999997,"5min":9.5790000000000006,"15min":9.6349999999999998},"last":2.1099999999999999},{"interface":"front","average":{"1min":1.911,"5min":2.4670000000000001,"15min":1.5089999999999999},"min":{"1min":0.29799999999999999,"5min":0.24399999999999999,"15min":0.224},"max":{"1min":5.1529999999999996,"5min":12.246,"15min":12.246},"last":0.92100000000000004}]},{"osd":3,"last update":"Mon Feb 12 17:02:41 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":2.6339999999999999,"15min":1.8360000000000001},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.26700000000000002},"max":{"1min":5.181,"5min":20.271000000000001,"15min":38.170999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.607,"5min":2.4329999999999998,"15min":1.641},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.8209999999999997,"5min":10.865,"15min":38.311999999999998},"last":1.1040000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":4.609,"15min":1.921},"min":{"1min":0.29099999999999998,"5min":0.253,"15min":0.24099999999999999},"max":{"1min":6.5679999999999996,"5min":79.573999999999998,"15min":79.573999999999998},"last":0.31},{"interface":"front","average":{"1min":1.3160000000000001,"5min":4.8579999999999997,"15min":1.968},"min":{"1min":0.33300000000000002,"5min":0.27900000000000003,"15min":0.24099999999999999},"max":{"1min":8.3450000000000006,"5min":80.497,"15min":80.497},"last":0.374}]},{"osd":6,"last update":"Mon Feb 12 17:02:39 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":3.1080000000000001,"15min":1.7849999999999999},"min":{"1min":0.32300000000000001,"5min":0.28699999999999998,"15min":0.255},"max":{"1min":5.399,"5min":27.678000000000001,"15min":104.907},"last":2.0600000000000001},{"interface":"front","average":{"1min":2.2599999999999998,"5min":3.343,"15min":1.925},"min":{"1min":0.373,"5min":0.30499999999999999,"15min":0.26500000000000001},"max":{"1min":6.5069999999999997,"5min":15.593,"15min":105.395},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:02:52 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.8130000000000002,"15min":1.369},"min":{"1min":0.35599999999999998,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":7.633,"5min":12.311999999999999,"15min":12.311999999999999},"last":2.6019999999999999},{"interface":"front","average":{"1min":2.5019999999999998,"5min":3.4359999999999999,"15min":1.6100000000000001},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.28799999999999998},"max":{"1min":6.0019999999999998,"5min":27.773,"15min":27.773},"last":4.0339999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757421,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6763572,"kb_used_data":6169772,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":86981580,"statfs":{"total":95995035648,"available":89069137920,"internally_reserved":0,"allocated":6317846528,"data_stored":6317407837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":2.5569999999999999,"15min":1.528},"min":{"1min":0.22800000000000001,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":5.4379999999999997,"5min":46.365000000000002,"15min":46.365000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.0640000000000001,"5min":3.0329999999999999,"15min":1.77},"min":{"1min":0.23999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.4020000000000001,"5min":46.134,"15min":46.134},"last":2.8260000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.9119999999999999,"15min":1.365},"min":{"1min":0.28999999999999998,"5min":0.26700000000000002,"15min":0.23200000000000001},"max":{"1min":4.0940000000000003,"5min":7.3869999999999996,"15min":16.655000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.3280000000000001,"5min":2.2389999999999999,"15min":1.5349999999999999},"min":{"1min":0.29599999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.3660000000000001,"5min":13.130000000000001,"15min":16.712},"last":4.008}]},{"osd":2,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.4620000000000002,"15min":1.6339999999999999},"min":{"1min":0.26600000000000001,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":5.4630000000000001,"5min":11.15,"15min":18.218},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":2.0699999999999998,"15min":1.502},"min":{"1min":0.34200000000000003,"5min":0.254,"15min":0.254},"max":{"1min":4.2030000000000003,"5min":9.1199999999999992,"15min":18.327000000000002},"last":0.26700000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":2.2810000000000001,"15min":1.544},"min":{"1min":0.314,"5min":0.314,"15min":0.28000000000000003},"max":{"1min":5.5030000000000001,"5min":11.122,"15min":18.251000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.458,"5min":2.028,"15min":1.423},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.30399999999999999},"max":{"1min":4.2279999999999998,"5min":7.4109999999999996,"15min":20.263000000000002},"last":0.32200000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":2.7999999999999998,"15min":1.278},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":9.5879999999999992,"5min":11.880000000000001,"15min":11.880000000000001},"last":2.73},{"interface":"front","average":{"1min":1.2290000000000001,"5min":3.4399999999999999,"15min":1.5029999999999999},"min":{"1min":0.28100000000000003,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":5.4139999999999997,"5min":20.550000000000001,"15min":20.550000000000001},"last":0.67200000000000004}]},{"osd":6,"last update":"Mon Feb 12 17:02:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":2.698,"15min":1.357},"min":{"1min":0.40999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.7629999999999999,"5min":12.281000000000001,"15min":12.281000000000001},"last":3.972},{"interface":"front","average":{"1min":1.8069999999999999,"5min":3.1709999999999998,"15min":1.462},"min":{"1min":0.37,"5min":0.28100000000000003,"15min":0.26700000000000002},"max":{"1min":6.6710000000000003,"5min":22.035,"15min":22.035},"last":0.57199999999999995}]},{"osd":7,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.0289999999999999,"5min":2.8330000000000002,"15min":1.3600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27500000000000002},"max":{"1min":9.9049999999999994,"5min":19.184999999999999,"15min":19.184999999999999},"last":2.2309999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":2.6749999999999998,"15min":1.3049999999999999},"min":{"1min":0.30499999999999999,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":4.8330000000000002,"5min":12.253,"15min":12.253},"last":2.8559999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5856362496,"data_stored":5856362496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4251779072,"data_stored":4251779072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3195469824,"data_stored":3195469824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6372720640,"data_stored":6372720640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5833752576,"data_stored":5833752576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:02:59.920 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:02:59.954 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:58.609881+0000 mgr.y (mgr.24481) 508 : cluster [DBG] pgmap v656: 138 pgs: 138 active+clean; 8.2 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-12T17:02:59.954 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:59.318546+0000 osd.2 (osd.2) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:59.955 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:02:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:59.318606+0000 osd.2 (osd.2) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:02:59.955 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:58.609881+0000 mgr.y (mgr.24481) 508 : cluster [DBG] pgmap v656: 138 pgs: 138 active+clean; 8.2 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-12T17:02:59.955 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:59.318546+0000 osd.2 (osd.2) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:02:59.955 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:02:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:59.318606+0000 osd.2 (osd.2) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:00.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:02:59.715199+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e335: 8 total, 8 up, 5 in 2024-02-12T17:03:00.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:59.827857+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.33:0/3584512471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:00.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:02:59.828217+0000 mon.a (mon.0) 1330 : audit [INF] from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:00.977 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:03:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:02:59.715199+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e335: 8 total, 8 up, 5 in 2024-02-12T17:03:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:59.827857+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.33:0/3584512471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:02:59.828217+0000 mon.a (mon.0) 1330 : audit [INF] from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:01.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:03:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:00] "GET /metrics HTTP/1.1" 200 176059 "" "Prometheus/2.33.4" 2024-02-12T17:03:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:02:59.715199+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e335: 8 total, 8 up, 5 in 2024-02-12T17:03:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:59.827857+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.33:0/3584512471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:02:59.828217+0000 mon.a (mon.0) 1330 : audit [INF] from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:01.172 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:01.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:01.283 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:01.323 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:01.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:00.610265+0000 mgr.y (mgr.24481) 509 : cluster [DBG] pgmap v658: 138 pgs: 138 active+clean; 8.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-12T17:03:01.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:00.716907+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:01.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:00.733283+0000 mon.a (mon.0) 1332 : audit [INF] from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:01.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:00.733342+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e336: 8 total, 8 up, 5 in 2024-02-12T17:03:01.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:00.610265+0000 mgr.y (mgr.24481) 509 : cluster [DBG] pgmap v658: 138 pgs: 138 active+clean; 8.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-12T17:03:01.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:00.716907+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:01.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:00.733283+0000 mon.a (mon.0) 1332 : audit [INF] from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:01.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:00.733342+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e336: 8 total, 8 up, 5 in 2024-02-12T17:03:01.978 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:01.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:01.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:00.610265+0000 mgr.y (mgr.24481) 509 : cluster [DBG] pgmap v658: 138 pgs: 138 active+clean; 8.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-12T17:03:01.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:00.716907+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:01.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:00.733283+0000 mon.a (mon.0) 1332 : audit [INF] from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:01.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:00.733342+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e336: 8 total, 8 up, 5 in 2024-02-12T17:03:02.086 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:02.111 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:02.147 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:02.261 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:02.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:02.284 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:03:02.284 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:03:02.376 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:02.435 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:02.472 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:02.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:02.577 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:02.635 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:02.673 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:02.678 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:03:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:03:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:02.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:02.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:02.834 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:02.862 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:02.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:02.968 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:03.061 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:03.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:03.168 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:03.211 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:03.498 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:02.488294+0000 mon.a (mon.0) 1334 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:03.498 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:02.611594+0000 mgr.y (mgr.24481) 510 : cluster [DBG] pgmap v660: 138 pgs: 138 active+clean; 8.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 739 B/s rd, 60 MiB/s wr, 959 op/s 2024-02-12T17:03:03.499 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:02.488294+0000 mon.a (mon.0) 1334 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:03.499 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:02.611594+0000 mgr.y (mgr.24481) 510 : cluster [DBG] pgmap v660: 138 pgs: 138 active+clean; 8.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 739 B/s rd, 60 MiB/s wr, 959 op/s 2024-02-12T17:03:03.662 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:03:03.666 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T17:03:03.677 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:03.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:03.688 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:03.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:03.835 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:03.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:02.488294+0000 mon.a (mon.0) 1334 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:03.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:02.611594+0000 mgr.y (mgr.24481) 510 : cluster [DBG] pgmap v660: 138 pgs: 138 active+clean; 8.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 739 B/s rd, 60 MiB/s wr, 959 op/s 2024-02-12T17:03:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:04.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:04.304 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:04.353 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:04.383 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:04.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:04.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:03:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:03:04.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:04.994 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:05.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:05.102 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:05.164 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:05.418 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:03:05.439 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:03:05.466 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:03:05.478 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:05.500 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:03:05.536 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:03:05.567 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:03:05.624 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:03:05.624 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:03:05.624 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:03:05.624 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:03:05.624 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:03:05.625 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:03:05.625 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:03:05.625 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:03:05.625 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:03:05.625 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:03:05.630 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:05.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:05.735 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:05.777 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:05.857 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:04.179466+0000 mgr.y (mgr.24481) 511 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:05.857 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:04.612394+0000 mgr.y (mgr.24481) 512 : cluster [DBG] pgmap v661: 138 pgs: 138 active+clean; 8.0 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.71k op/s 2024-02-12T17:03:05.857 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:05.381380+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? v1:172.21.15.33:0/2407811368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:05.858 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:05.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:05.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:05.962 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:06.001 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:04.179466+0000 mgr.y (mgr.24481) 511 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:06.001 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:04.612394+0000 mgr.y (mgr.24481) 512 : cluster [DBG] pgmap v661: 138 pgs: 138 active+clean; 8.0 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.71k op/s 2024-02-12T17:03:06.001 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:05.381380+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? v1:172.21.15.33:0/2407811368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:06.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:04.179466+0000 mgr.y (mgr.24481) 511 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:06.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:04.612394+0000 mgr.y (mgr.24481) 512 : cluster [DBG] pgmap v661: 138 pgs: 138 active+clean; 8.0 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.71k op/s 2024-02-12T17:03:06.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:05.381380+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? v1:172.21.15.33:0/2407811368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:06.004 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.15459.0:146819 9.1 e336/304 9:8e76c44a:::benchmark_data_smithi033_131576_object9229:head v 336'9141, mlcod=336'9138)", 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:06.189074+0000", 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0063056259999999999, 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063333850000000004, 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:03:06.201 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189074+0000", 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189074+0000", 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189074+0000", 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2400000000000007e-06 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189082+0000", 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5120000000000002e-06 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189087+0000", 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5999999999999998e-06 2024-02-12T17:03:06.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189090+0000", 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014316900000000001 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189233+0000", 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9998999999999999e-05 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:06.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189253+0000", 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017922700000000001 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.15459.0:146822 9.1 e336/304 9:8d8cb5d6:::benchmark_data_smithi033_131576_object9232:head v 336'9142, mlcod=336'9139)", 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:06.189106+0000", 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0062734599999999998, 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063697650000000003, 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:03:06.205 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189106+0000", 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189106+0000", 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189106+0000", 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2100000000000002e-06 2024-02-12T17:03:06.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189109+0000", 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4329999999999997e-06 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189113+0000", 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1259999999999998e-06 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.189117+0000", 2024-02-12T17:03:06.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019987939999999999 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.191115+0000", 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6339999999999999e-05 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.191132+0000", 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020254930000000002 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.15459.0:146832 9.1 e336/304 9:8a96ff41:::benchmark_data_smithi033_131576_object9242:head v 336'9143, mlcod=336'9139)", 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:06.194328+0000", 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.001051225, 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001208803, 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.194328+0000", 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.194328+0000", 2024-02-12T17:03:06.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.194328+0000", 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9509999999999997e-06 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.194333+0000", 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8040000000000001e-06 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.194338+0000", 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.597e-06 2024-02-12T17:03:06.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.194341+0000", 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7558e-05 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.194399+0000", 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9660000000000002e-05 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.194419+0000", 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0144999999999998e-05 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:06.211 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:06.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:06.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:06.212 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T17:03:06.212 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 3 2024-02-12T17:03:06.212 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:03:06.565 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:06.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:06.670 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:07.046 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T17:03:07.270 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:06.612803+0000 mgr.y (mgr.24481) 513 : cluster [DBG] pgmap v662: 138 pgs: 138 active+clean; 8.0 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.4 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-02-12T17:03:07.466 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:07.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:06.612803+0000 mgr.y (mgr.24481) 513 : cluster [DBG] pgmap v662: 138 pgs: 138 active+clean; 8.0 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.4 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-02-12T17:03:07.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:06.612803+0000 mgr.y (mgr.24481) 513 : cluster [DBG] pgmap v662: 138 pgs: 138 active+clean; 8.0 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.4 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-02-12T17:03:08.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:07.396301+0000 osd.1 (osd.1) 154 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:08.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:07.396371+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:08.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:07.396301+0000 osd.1 (osd.1) 154 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:08.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:07.396371+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:08.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:07.396301+0000 osd.1 (osd.1) 154 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:08.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:07.396371+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:08.278481+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e337: 8 total, 8 up, 5 in 2024-02-12T17:03:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:08.613419+0000 mgr.y (mgr.24481) 514 : cluster [DBG] pgmap v664: 138 pgs: 138 active+clean; 7.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 767 B/s rd, 9.3 MiB/s wr, 1.37k op/s 2024-02-12T17:03:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:08.709849+0000 osd.5 (osd.5) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:09.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:08.709903+0000 osd.5 (osd.5) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:09.508 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:03:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:03:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:03:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:03:09.508 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:03:09.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:08.278481+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e337: 8 total, 8 up, 5 in 2024-02-12T17:03:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:08.613419+0000 mgr.y (mgr.24481) 514 : cluster [DBG] pgmap v664: 138 pgs: 138 active+clean; 7.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 767 B/s rd, 9.3 MiB/s wr, 1.37k op/s 2024-02-12T17:03:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:08.709849+0000 osd.5 (osd.5) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:08.709903+0000 osd.5 (osd.5) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:08.278481+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e337: 8 total, 8 up, 5 in 2024-02-12T17:03:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:08.613419+0000 mgr.y (mgr.24481) 514 : cluster [DBG] pgmap v664: 138 pgs: 138 active+clean; 7.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 767 B/s rd, 9.3 MiB/s wr, 1.37k op/s 2024-02-12T17:03:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:08.709849+0000 osd.5 (osd.5) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:08.709903+0000 osd.5 (osd.5) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:10.348 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T17:03:10.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:09.288280+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e338: 8 total, 8 up, 5 in 2024-02-12T17:03:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:10.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:09.288280+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e338: 8 total, 8 up, 5 in 2024-02-12T17:03:10.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:09.288280+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e338: 8 total, 8 up, 5 in 2024-02-12T17:03:11.279 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:03:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:10] "GET /metrics HTTP/1.1" 200 176054 "" "Prometheus/2.33.4" 2024-02-12T17:03:11.694 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:10.613743+0000 mgr.y (mgr.24481) 515 : cluster [DBG] pgmap v666: 138 pgs: 138 active+clean; 7.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.4 KiB/s rd, 9.3 MiB/s wr, 1.67k op/s 2024-02-12T17:03:11.694 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:10.613743+0000 mgr.y (mgr.24481) 515 : cluster [DBG] pgmap v666: 138 pgs: 138 active+clean; 7.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.4 KiB/s rd, 9.3 MiB/s wr, 1.67k op/s 2024-02-12T17:03:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:10.613743+0000 mgr.y (mgr.24481) 515 : cluster [DBG] pgmap v666: 138 pgs: 138 active+clean; 7.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.4 KiB/s rd, 9.3 MiB/s wr, 1.67k op/s 2024-02-12T17:03:12.030 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:12.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:12.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:12.177 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:12.226 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:12.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:12.332 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:12.387 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:12.529 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:12.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:12.636 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:12.673 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:12.753 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:12.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:12.825 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:03:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:03:12.860 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:12.897 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:12.977 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:12.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:12.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:13.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85057 8.b 8:df0e10c0:::benchmark_data_smithi033_105085_object85056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.130368+0000", 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 327.03137139099999, 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.117763128, 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:03:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85057 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130368+0000", 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130368+0000", 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:03:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130366+0000", 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.603e-05 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130392+0000", 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7159999999999997e-06 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130399+0000", 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130404+0000", 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00032313900000000001 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130727+0000", 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124036 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130851+0000", 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017825299999999999 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.131029+0000", 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024958350000000001 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133525+0000", 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062191540000000002 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.139744+0000", 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017399532999999998 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157144+0000", 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0638e-05 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157154+0000", 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0026200919999999 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159774+0000", 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0857e-05 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159785+0000", 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.088274985 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248060+0000", 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0550999999999996e-05 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248131+0000", 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.117763128 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85060 8.b 8:d0c504d4:::benchmark_data_smithi033_105085_object85059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.133233+0000", 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 327.02850551400002, 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1475086289999998, 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85060 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133233+0000", 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133233+0000", 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133230+0000", 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6168000000000003e-05 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133266+0000", 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1382999999999999e-05 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133278+0000", 2024-02-12T17:03:13.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7170000000000002e-06 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133284+0000", 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025424699999999998 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133538+0000", 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112871 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133651+0000", 2024-02-12T17:03:13.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015755700000000001 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133809+0000", 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023381417000000002 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157190+0000", 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7670000000000003e-06 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157198+0000", 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.002615364 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159813+0000", 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1559999999999998e-06 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159821+0000", 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0883366249999999 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248157+0000", 2024-02-12T17:03:13.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044692669999999999 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for degraded object", 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.252626+0000", 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028011846999999999 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.280638+0000", 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103669 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.280742+0000", 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1475086289999998 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85062 8.b 8:d12d443b:::benchmark_data_smithi033_105085_object85061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.141630+0000", 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 327.02010867600001, 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.139506715, 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.184 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85062 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141630+0000", 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141630+0000", 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141628+0000", 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4357999999999998e-05 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141672+0000", 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2106000000000001e-05 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141684+0000", 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7390000000000002e-06 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141691+0000", 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3009e-05 2024-02-12T17:03:13.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141744+0000", 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012559 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141869+0000", 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016855700000000001 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.142038+0000", 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015187798000000001 2024-02-12T17:03:13.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157226+0000", 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8449999999999997e-06 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157233+0000", 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.002611798 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159844+0000", 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1400000000000002e-06 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159852+0000", 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0928147509999999 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.252666+0000", 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038766899999999999 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for degraded object", 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253054+0000", 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028008100000000001 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.281062+0000", 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4779999999999999e-05 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.281137+0000", 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.139506715 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85069 8.b 8:d240be51:::benchmark_data_smithi033_105085_object85068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.166508+0000", 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 326.99523105100002, 2024-02-12T17:03:13.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1456601989999999, 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85069 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166508+0000", 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166508+0000", 2024-02-12T17:03:13.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166506+0000", 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7187000000000002e-05 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166543+0000", 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6649999999999998e-06 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166553+0000", 2024-02-12T17:03:13.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3049999999999998e-06 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166559+0000", 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124027 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166683+0000", 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0033e-05 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166693+0000", 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.993179433 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159873+0000", 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.328e-06 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159880+0000", 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0931984909999999 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253079+0000", 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013465999999999999 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253213+0000", 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051193199999999999 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,7", 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253725+0000", 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0084671769999999993 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.262192+0000", 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047363939999999997 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.266929+0000", 2024-02-12T17:03:13.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030863169999999999 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.270015+0000", 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.041951342000000003 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.311966+0000", 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4244000000000004e-05 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.312031+0000", 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013736999999999999 2024-02-12T17:03:13.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.312168+0000", 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1456601989999999 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220735 8.b 8:dc5cc4b6:::benchmark_data_smithi033_105085_object109077:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.748274+0000", 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.41346529200001, 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99742273999999997, 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220735 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748274+0000", 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748274+0000", 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748271+0000", 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1210000000000007e-06 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748280+0000", 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6354e-05 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748296+0000", 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4479999999999999e-06 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748302+0000", 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5645999999999999e-05 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748398+0000", 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2930000000000003e-06 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748405+0000", 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97336656499999996 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.721772+0000", 2024-02-12T17:03:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012956099999999999 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.721901+0000", 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016388700000000001 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.722065+0000", 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0099480499999999999 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732013+0000", 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013501833 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745515+0000", 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5191000000000004e-05 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745610+0000", 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4626000000000003e-05 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745655+0000", 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1374000000000001e-05 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745696+0000", 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99742273999999997 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220736 8.b 8:d7ee9352:::benchmark_data_smithi033_105085_object109078:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.748314+0000", 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.41342485800001, 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99754634200000003, 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220736 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748314+0000", 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748314+0000", 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748313+0000", 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2619999999999999e-06 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748317+0000", 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1290000000000004e-06 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748322+0000", 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9429999999999999e-06 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748327+0000", 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7467999999999996e-05 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748424+0000", 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8879999999999996e-06 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748429+0000", 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97531739399999995 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.723746+0000", 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115288 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.723862+0000", 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013617699999999999 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.723998+0000", 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080225620000000004 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732020+0000", 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013542134000000001 2024-02-12T17:03:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745563+0000", 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000208486 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745771+0000", 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7753999999999997e-05 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745819+0000", 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1597000000000002e-05 2024-02-12T17:03:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745860+0000", 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99754634200000003 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220747 8.b 8:dc732349:::benchmark_data_smithi033_105085_object109089:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.764408+0000", 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.397331337, 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98163293600000001, 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220747 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764408+0000", 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764408+0000", 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764405+0000", 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6539999999999999e-06 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764413+0000", 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7130000000000005e-06 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764421+0000", 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0069999999999997e-06 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764426+0000", 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071895100000000001 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.765145+0000", 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-06 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.765153+0000", 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96046406799999995 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.725617+0000", 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000105274 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.725722+0000", 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014608899999999999 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.725868+0000", 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061591099999999998 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732027+0000", 2024-02-12T17:03:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013887171 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745915+0000", 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3433999999999998e-05 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745958+0000", 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3881999999999998e-05 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746002+0000", 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8677999999999998e-05 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746040+0000", 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98163293600000001 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220785 8.b 8:daf40cf0:::benchmark_data_smithi033_105085_object109127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.797112+0000", 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.36462687400001, 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94916868899999995, 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220785 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797112+0000", 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797112+0000", 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797111+0000", 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3400000000000002e-06 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797115+0000", 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3419999999999998e-06 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797120+0000", 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6969999999999999e-06 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797124+0000", 2024-02-12T17:03:13.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8279999999999999e-05 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797172+0000", 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6473999999999998e-05 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797188+0000", 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.92994460499999998 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.727133+0000", 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013374799999999999 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.727267+0000", 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012754699999999999 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.727394+0000", 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072825829999999996 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734677+0000", 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011405406999999999 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746082+0000", 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114867 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746197+0000", 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3798000000000003e-05 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746241+0000", 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9749999999999997e-05 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746281+0000", 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94916868899999995 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220787 8.b 8:d18a4d72:::benchmark_data_smithi033_105085_object109129:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.797229+0000", 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.36450963199999, 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94926104300000003, 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.221 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220787 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797229+0000", 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797229+0000", 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797229+0000", 2024-02-12T17:03:13.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4499999999999998e-06 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797231+0000", 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5850000000000001e-06 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797236+0000", 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5480000000000002e-06 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797239+0000", 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000140223 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797380+0000", 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6250000000000004e-06 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797385+0000", 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.931185033 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.728570+0000", 2024-02-12T17:03:13.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106504 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.728677+0000", 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014063899999999999 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.728817+0000", 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058666860000000003 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734684+0000", 2024-02-12T17:03:13.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011638766 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746323+0000", 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6386999999999998e-05 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746409+0000", 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8921000000000002e-05 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746448+0000", 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2188000000000003e-05 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746490+0000", 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94926104300000003 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220835 8.b 8:dd40dfe1:::benchmark_data_smithi033_105085_object109177:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.845315+0000", 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.31642340900001, 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.90127469299999996, 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.227 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220835 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845315+0000", 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845315+0000", 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845314+0000", 2024-02-12T17:03:13.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0217999999999999e-05 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845324+0000", 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1071e-05 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845335+0000", 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5480000000000004e-06 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845342+0000", 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051822960000000003 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850525+0000", 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1790000000000008e-06 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850533+0000", 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87922209299999998 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.729755+0000", 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0649000000000002e-05 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.729816+0000", 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7392999999999995e-05 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.729903+0000", 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047881829999999997 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734691+0000", 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011659630000000001 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746351+0000", 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173017 2024-02-12T17:03:13.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746524+0000", 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5839e-05 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746560+0000", 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0505e-05 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746590+0000", 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.90127469299999996 2024-02-12T17:03:13.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220840 8.b 8:de442c69:::benchmark_data_smithi033_105085_object109182:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.850406+0000", 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.31133306000001, 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.89630858400000002, 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220840 2024-02-12T17:03:13.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850406+0000", 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850406+0000", 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850404+0000", 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1919000000000001e-05 2024-02-12T17:03:13.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850416+0000", 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0019999999999995e-06 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850425+0000", 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9190000000000004e-06 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850431+0000", 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121835 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850553+0000", 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2240000000000003e-06 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850558+0000", 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.880290087 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.730848+0000", 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111928 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.730960+0000", 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000146126 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.731106+0000", 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061862439999999996 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737292+0000", 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090883839999999997 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746381+0000", 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026667999999999999 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746648+0000", 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6155999999999997e-05 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746684+0000", 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0743e-05 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746714+0000", 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.89630858400000002 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220880 8.b 8:d0ab2051:::benchmark_data_smithi033_105085_object109222:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.900313+0000", 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.261425825, 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84649108299999998, 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220880 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900313+0000", 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900313+0000", 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900311+0000", 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5899999999999997e-06 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900320+0000", 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1103e-05 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900331+0000", 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.959e-06 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900337+0000", 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7706e-05 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900385+0000", 2024-02-12T17:03:13.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.469e-06 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900392+0000", 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83167645999999995 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732069+0000", 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114713 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732184+0000", 2024-02-12T17:03:13.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013649699999999999 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732320+0000", 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0049788640000000004 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737299+0000", 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0093221889999999998 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746621+0000", 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012349600000000001 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746745+0000", 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0648000000000002e-05 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746775+0000", 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8821000000000001e-05 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746804+0000", 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84649108299999998 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.244 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220883 8.b 8:d17dac6e:::benchmark_data_smithi033_105085_object109225:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.904632+0000", 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.25710682499999, 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84231522400000003, 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220883 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904632+0000", 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904632+0000", 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904630+0000", 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4369999999999996e-06 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904640+0000", 2024-02-12T17:03:13.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0437e-05 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904650+0000", 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3119999999999999e-06 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904657+0000", 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5430999999999999e-05 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904702+0000", 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7610000000000007e-06 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904710+0000", 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82888116999999994 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.733591+0000", 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000110407 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.733701+0000", 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014007599999999999 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.733841+0000", 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064799719999999996 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740321+0000", 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065120630000000002 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746833+0000", 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2596000000000003e-05 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746886+0000", 2024-02-12T17:03:13.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1346000000000001e-05 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746917+0000", 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9869000000000002e-05 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746947+0000", 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84231522400000003 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220885 8.b 8:d01c3d69:::benchmark_data_smithi033_105085_object109227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.250 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.907595+0000", 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.25414423399999, 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83944173600000005, 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220885 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907595+0000", 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907595+0000", 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907593+0000", 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3400000000000003e-06 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907599+0000", 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1269999999999992e-06 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907608+0000", 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6330000000000001e-06 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907614+0000", 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796000000000003e-05 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907654+0000", 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3150000000000003e-06 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907661+0000", 2024-02-12T17:03:13.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.827071259 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734732+0000", 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000104963 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734837+0000", 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000139192 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734977+0000", 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053569009999999999 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740333+0000", 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065265089999999998 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746860+0000", 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116649 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746977+0000", 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0667999999999998e-05 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747007+0000", 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9127000000000001e-05 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747036+0000", 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83944173600000005 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220895 8.b 8:d966b5f5:::benchmark_data_smithi033_105085_object109237:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.919760+0000", 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.24197936100001, 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82739850299999995, 2024-02-12T17:03:13.256 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220895 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919760+0000", 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919760+0000", 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:03:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919757+0000", 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6240000000000008e-06 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919766+0000", 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5795e-05 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919782+0000", 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.64e-06 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919788+0000", 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5522999999999998e-05 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919834+0000", 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.592e-06 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919839+0000", 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.81639381799999999 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.736233+0000", 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5377999999999994e-05 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.736329+0000", 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4993000000000005e-05 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.736404+0000", 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.007085727 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.743489+0000", 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035814969999999999 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747071+0000", 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6551999999999999e-05 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747097+0000", 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1009000000000001e-05 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747128+0000", 2024-02-12T17:03:13.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9694999999999999e-05 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747158+0000", 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82739850299999995 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220897 8.b 8:d66e080d:::benchmark_data_smithi033_105085_object109239:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.926673+0000", 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.23506561600001, 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82061067099999996, 2024-02-12T17:03:13.262 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220897 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926673+0000", 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926673+0000", 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:03:13.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926671+0000", 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6340000000000007e-06 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926679+0000", 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3509999999999999e-05 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926693+0000", 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7900000000000002e-06 2024-02-12T17:03:13.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926700+0000", 2024-02-12T17:03:13.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2910000000000001e-05 2024-02-12T17:03:13.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926743+0000", 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0366999999999999e-05 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926753+0000", 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.810602445 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737355+0000", 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010326 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737459+0000", 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013548499999999999 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737594+0000", 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058989669999999998 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.743493+0000", 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037014119999999998 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747195+0000", 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7673000000000001e-05 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747222+0000", 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1764999999999998e-05 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747254+0000", 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9954000000000002e-05 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747284+0000", 2024-02-12T17:03:13.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82061067099999996 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220902 8.b 8:d534fbf5:::benchmark_data_smithi033_105085_object109244:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.931051+0000", 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.230687531, 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.816350519, 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220902 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.269 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931051+0000", 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931051+0000", 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931048+0000", 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0699999999999999e-05 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931059+0000", 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.925e-05 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931078+0000", 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3809999999999999e-06 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931086+0000", 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4478999999999998e-05 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931141+0000", 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8519999999999999e-06 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931151+0000", 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.80772856699999995 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.738879+0000", 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5386e-05 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.738945+0000", 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6019000000000004e-05 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.739031+0000", 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0060882009999999997 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745119+0000", 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021945689999999999 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747314+0000", 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6559000000000001e-05 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747340+0000", 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1337000000000002e-05 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747371+0000", 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0459000000000001e-05 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747402+0000", 2024-02-12T17:03:13.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.816350519 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220903 8.b 8:d138e6c7:::benchmark_data_smithi033_105085_object109245:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.932215+0000", 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.229524328, 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.81530424899999998, 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220903 2024-02-12T17:03:13.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932215+0000", 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932215+0000", 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932212+0000", 2024-02-12T17:03:13.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3659999999999992e-06 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932221+0000", 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4065e-05 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932235+0000", 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4860000000000002e-06 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932241+0000", 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1485e-05 2024-02-12T17:03:13.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932283+0000", 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8480000000000003e-06 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932290+0000", 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.80808374500000002 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740373+0000", 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010814700000000001 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740482+0000", 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013815 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740620+0000", 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045471419999999997 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745167+0000", 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022638350000000001 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747431+0000", 2024-02-12T17:03:13.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6512999999999998e-05 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747457+0000", 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1942e-05 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747489+0000", 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9643999999999999e-05 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747519+0000", 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.81530424899999998 2024-02-12T17:03:13.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220958 8.b 8:dc590d84:::benchmark_data_smithi033_105085_object109300:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:46.036007+0000", 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.125731844, 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.71162536899999995, 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220958 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036007+0000", 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036007+0000", 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036005+0000", 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9109999999999993e-06 2024-02-12T17:03:13.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036012+0000", 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2116000000000001e-05 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036025+0000", 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0010000000000004e-06 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036032+0000", 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9883999999999999e-05 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036081+0000", 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5270000000000004e-06 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036091+0000", 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.70584844700000005 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.741939+0000", 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7015999999999996e-05 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.742016+0000", 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4719000000000001e-05 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.742091+0000", 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00343203 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745523+0000", 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020237580000000001 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747547+0000", 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5474e-05 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747572+0000", 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0948000000000003e-05 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747603+0000", 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9077e-05 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747632+0000", 2024-02-12T17:03:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.71162536899999995 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220974 8.b 8:d7cbd98b:::benchmark_data_smithi033_105085_object109316:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:46.130419+0000", 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.03132031300001, 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.63342457100000005, 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220974 2024-02-12T17:03:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.130419+0000", 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.130419+0000", 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.130417+0000", 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.058e-06 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.130424+0000", 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1970000000000001e-05 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.130436+0000", 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7050000000000002e-06 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.130442+0000", 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5399999999999999e-05 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.130488+0000", 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5759999999999995e-06 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.130497+0000", 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.61303083700000005 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.743528+0000", 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000108824 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.743637+0000", 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016643800000000001 2024-02-12T17:03:13.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.743803+0000", 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017249399999999999 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745528+0000", 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021329840000000001 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747661+0000", 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016087445999999998 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.763749+0000", 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9684000000000001e-05 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.763798+0000", 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4663000000000003e-05 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.763843+0000", 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.63342457100000005 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:13.293 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:13.293 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:03:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:13.293 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:13.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:13.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:13.444 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:13.728 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:03:13.730 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:03:13.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:12.614067+0000 mgr.y (mgr.24481) 516 : cluster [DBG] pgmap v667: 138 pgs: 138 active+clean; 7.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-02-12T17:03:13.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T17:03:13.906 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:13.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:14.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:14.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:12.614067+0000 mgr.y (mgr.24481) 516 : cluster [DBG] pgmap v667: 138 pgs: 138 active+clean; 7.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-02-12T17:03:14.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:12.614067+0000 mgr.y (mgr.24481) 516 : cluster [DBG] pgmap v667: 138 pgs: 138 active+clean; 7.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-02-12T17:03:14.055 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:14.196 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:03:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:03:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:14.590 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-02-12T17:03:12.613884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8222611132,"num_objects":125660,"num_object_clones":0,"num_object_copies":376980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125660,"num_whiteouts":0,"num_read":1244,"num_read_kb":996,"num_write":150422,"num_write_kb":8805610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":6652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57802,"ondisk_log_size":57802,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":27208524,"kb_used_data":24239268,"kb_used_omap":0,"kb_used_meta":2968960,"kb_avail":441517236,"statfs":{"total":479975178240,"available":452113649664,"internally_reserved":0,"allocated":24821010432,"data_stored":24818784051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3040215040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,5,4,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-409927680,"num_objects":-6255,"num_object_clones":0,"num_object_copies":-18765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6255,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6255,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002139"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":239,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754512+0000","last_change":"2024-02-12T17:02:30.689748+0000","last_active":"2024-02-12T17:03:00.754512+0000","last_peered":"2024-02-12T17:03:00.754512+0000","last_clean":"2024-02-12T17:03:00.754512+0000","last_became_active":"2024-02-12T16:59:09.246207+0000","last_became_peered":"2024-02-12T16:59:09.246207+0000","last_unstale":"2024-02-12T17:03:00.754512+0000","last_undegraded":"2024-02-12T17:03:00.754512+0000","last_fullsized":"2024-02-12T17:03:00.754512+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.689645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.689645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":306,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745989+0000","last_change":"2024-02-12T17:02:47.732393+0000","last_active":"2024-02-12T17:03:00.745989+0000","last_peered":"2024-02-12T17:03:00.745989+0000","last_clean":"2024-02-12T17:03:00.745989+0000","last_became_active":"2024-02-12T16:49:39.102868+0000","last_became_peered":"2024-02-12T16:49:39.102868+0000","last_unstale":"2024-02-12T17:03:00.745989+0000","last_undegraded":"2024-02-12T17:03:00.745989+0000","last_fullsized":"2024-02-12T17:03:00.745989+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.732293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.732293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":272,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746991+0000","last_change":"2024-02-12T17:01:26.047598+0000","last_active":"2024-02-12T17:03:00.746991+0000","last_peered":"2024-02-12T17:03:00.746991+0000","last_clean":"2024-02-12T17:03:00.746991+0000","last_became_active":"2024-02-12T17:01:16.912117+0000","last_became_peered":"2024-02-12T17:01:16.912117+0000","last_unstale":"2024-02-12T17:03:00.746991+0000","last_undegraded":"2024-02-12T17:03:00.746991+0000","last_fullsized":"2024-02-12T17:03:00.746991+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:01:26.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.047527+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":270,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.751038+0000","last_change":"2024-02-12T17:00:56.892829+0000","last_active":"2024-02-12T17:03:00.751038+0000","last_peered":"2024-02-12T17:03:00.751038+0000","last_clean":"2024-02-12T17:03:00.751038+0000","last_became_active":"2024-02-12T16:57:47.188886+0000","last_became_peered":"2024-02-12T16:57:47.188886+0000","last_unstale":"2024-02-12T17:03:00.751038+0000","last_undegraded":"2024-02-12T17:03:00.751038+0000","last_fullsized":"2024-02-12T17:03:00.751038+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.892756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:56.892756+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.892756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":221,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746339+0000","last_change":"2024-02-12T17:01:39.176533+0000","last_active":"2024-02-12T17:03:00.746339+0000","last_peered":"2024-02-12T17:03:00.746339+0000","last_clean":"2024-02-12T17:03:00.746339+0000","last_became_active":"2024-02-12T16:54:40.642035+0000","last_became_peered":"2024-02-12T16:54:40.642035+0000","last_unstale":"2024-02-12T17:03:00.746339+0000","last_undegraded":"2024-02-12T17:03:00.746339+0000","last_fullsized":"2024-02-12T17:03:00.746339+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.176431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":379,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.739954+0000","last_change":"2024-02-12T17:01:45.154273+0000","last_active":"2024-02-12T17:03:00.739954+0000","last_peered":"2024-02-12T17:03:00.739954+0000","last_clean":"2024-02-12T17:03:00.739954+0000","last_became_active":"2024-02-12T16:59:09.243960+0000","last_became_peered":"2024-02-12T16:59:09.243960+0000","last_unstale":"2024-02-12T17:03:00.739954+0000","last_undegraded":"2024-02-12T17:03:00.739954+0000","last_fullsized":"2024-02-12T17:03:00.739954+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:01:45.154164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":320,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.750467+0000","last_change":"2024-02-12T17:02:46.577138+0000","last_active":"2024-02-12T17:03:00.750467+0000","last_peered":"2024-02-12T17:03:00.750467+0000","last_clean":"2024-02-12T17:03:00.750467+0000","last_became_active":"2024-02-12T16:57:47.176448+0000","last_became_peered":"2024-02-12T16:57:47.176448+0000","last_unstale":"2024-02-12T17:03:00.750467+0000","last_undegraded":"2024-02-12T17:03:00.750467+0000","last_fullsized":"2024-02-12T17:03:00.750467+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:46.577048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:46.577048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":323,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753888+0000","last_change":"2024-02-12T17:01:37.813077+0000","last_active":"2024-02-12T17:03:00.753888+0000","last_peered":"2024-02-12T17:03:00.753888+0000","last_clean":"2024-02-12T17:03:00.753888+0000","last_became_active":"2024-02-12T16:59:09.248262+0000","last_became_peered":"2024-02-12T16:59:09.248262+0000","last_unstale":"2024-02-12T17:03:00.753888+0000","last_undegraded":"2024-02-12T17:03:00.753888+0000","last_fullsized":"2024-02-12T17:03:00.753888+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:37.813008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:01:37.813008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":155,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754439+0000","last_change":"2024-02-12T17:01:16.912824+0000","last_active":"2024-02-12T17:03:00.754439+0000","last_peered":"2024-02-12T17:03:00.754439+0000","last_clean":"2024-02-12T17:03:00.754439+0000","last_became_active":"2024-02-12T17:01:16.912468+0000","last_became_peered":"2024-02-12T17:01:16.912468+0000","last_unstale":"2024-02-12T17:03:00.754439+0000","last_undegraded":"2024-02-12T17:03:00.754439+0000","last_fullsized":"2024-02-12T17:03:00.754439+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":322,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745778+0000","last_change":"2024-02-12T17:00:50.923564+0000","last_active":"2024-02-12T17:03:00.745778+0000","last_peered":"2024-02-12T17:03:00.745778+0000","last_clean":"2024-02-12T17:03:00.745778+0000","last_became_active":"2024-02-12T16:57:47.185903+0000","last_became_peered":"2024-02-12T16:57:47.185903+0000","last_unstale":"2024-02-12T17:03:00.745778+0000","last_undegraded":"2024-02-12T17:03:00.745778+0000","last_fullsized":"2024-02-12T17:03:00.745778+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:50.923462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.923462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":306,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753841+0000","last_change":"2024-02-12T17:02:34.502434+0000","last_active":"2024-02-12T17:03:00.753841+0000","last_peered":"2024-02-12T17:03:00.753841+0000","last_clean":"2024-02-12T17:03:00.753841+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:03:00.753841+0000","last_undegraded":"2024-02-12T17:03:00.753841+0000","last_fullsized":"2024-02-12T17:03:00.753841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.502332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":263,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746153+0000","last_change":"2024-02-12T16:57:18.865914+0000","last_active":"2024-02-12T17:03:00.746153+0000","last_peered":"2024-02-12T17:03:00.746153+0000","last_clean":"2024-02-12T17:03:00.746153+0000","last_became_active":"2024-02-12T16:54:40.640606+0000","last_became_peered":"2024-02-12T16:54:40.640606+0000","last_unstale":"2024-02-12T17:03:00.746153+0000","last_undegraded":"2024-02-12T17:03:00.746153+0000","last_fullsized":"2024-02-12T17:03:00.746153+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-12T16:57:18.865807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.865807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":185,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.757888+0000","last_change":"2024-02-12T17:02:18.973581+0000","last_active":"2024-02-12T17:03:00.757888+0000","last_peered":"2024-02-12T17:03:00.757888+0000","last_clean":"2024-02-12T17:03:00.757888+0000","last_became_active":"2024-02-12T16:59:09.259293+0000","last_became_peered":"2024-02-12T16:59:09.259293+0000","last_unstale":"2024-02-12T17:03:00.757888+0000","last_undegraded":"2024-02-12T17:03:00.757888+0000","last_fullsized":"2024-02-12T17:03:00.757888+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.973480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.973480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":225,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754632+0000","last_change":"2024-02-12T16:59:09.276561+0000","last_active":"2024-02-12T17:03:00.754632+0000","last_peered":"2024-02-12T17:03:00.754632+0000","last_clean":"2024-02-12T17:03:00.754632+0000","last_became_active":"2024-02-12T16:59:09.272423+0000","last_became_peered":"2024-02-12T16:59:09.272423+0000","last_unstale":"2024-02-12T17:03:00.754632+0000","last_undegraded":"2024-02-12T17:03:00.754632+0000","last_fullsized":"2024-02-12T17:03:00.754632+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":228,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.742862+0000","last_change":"2024-02-12T17:02:40.949925+0000","last_active":"2024-02-12T17:03:00.742862+0000","last_peered":"2024-02-12T17:03:00.742862+0000","last_clean":"2024-02-12T17:03:00.742862+0000","last_became_active":"2024-02-12T16:59:09.249268+0000","last_became_peered":"2024-02-12T16:59:09.249268+0000","last_unstale":"2024-02-12T17:03:00.742862+0000","last_undegraded":"2024-02-12T17:03:00.742862+0000","last_fullsized":"2024-02-12T17:03:00.742862+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":321,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.742053+0000","last_change":"2024-02-12T17:02:21.149890+0000","last_active":"2024-02-12T17:03:00.742053+0000","last_peered":"2024-02-12T17:03:00.742053+0000","last_clean":"2024-02-12T17:03:00.742053+0000","last_became_active":"2024-02-12T16:59:09.270574+0000","last_became_peered":"2024-02-12T16:59:09.270574+0000","last_unstale":"2024-02-12T17:03:00.742053+0000","last_undegraded":"2024-02-12T17:03:00.742053+0000","last_fullsized":"2024-02-12T17:03:00.742053+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:21.149801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T17:02:21.149801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":96,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.741584+0000","last_change":"2024-02-12T17:01:27.157444+0000","last_active":"2024-02-12T17:03:00.741584+0000","last_peered":"2024-02-12T17:03:00.741584+0000","last_clean":"2024-02-12T17:03:00.741584+0000","last_became_active":"2024-02-12T16:59:09.248342+0000","last_became_peered":"2024-02-12T16:59:09.248342+0000","last_unstale":"2024-02-12T17:03:00.741584+0000","last_undegraded":"2024-02-12T17:03:00.741584+0000","last_fullsized":"2024-02-12T17:03:00.741584+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.157357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.157357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":383,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745617+0000","last_change":"2024-02-12T17:01:16.911758+0000","last_active":"2024-02-12T17:03:00.745617+0000","last_peered":"2024-02-12T17:03:00.745617+0000","last_clean":"2024-02-12T17:03:00.745617+0000","last_became_active":"2024-02-12T17:01:16.911392+0000","last_became_peered":"2024-02-12T17:01:16.911392+0000","last_unstale":"2024-02-12T17:03:00.745617+0000","last_undegraded":"2024-02-12T17:03:00.745617+0000","last_fullsized":"2024-02-12T17:03:00.745617+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-12T16:57:24.750245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.750245+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":146,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.756290+0000","last_change":"2024-02-12T17:02:37.745936+0000","last_active":"2024-02-12T17:03:00.756290+0000","last_peered":"2024-02-12T17:03:00.756290+0000","last_clean":"2024-02-12T17:03:00.756290+0000","last_became_active":"2024-02-12T16:57:47.178240+0000","last_became_peered":"2024-02-12T16:57:47.178240+0000","last_unstale":"2024-02-12T17:03:00.756290+0000","last_undegraded":"2024-02-12T17:03:00.756290+0000","last_fullsized":"2024-02-12T17:03:00.756290+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:37.745838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.745838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":99,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746772+0000","last_change":"2024-02-12T17:02:24.819770+0000","last_active":"2024-02-12T17:03:00.746772+0000","last_peered":"2024-02-12T17:03:00.746772+0000","last_clean":"2024-02-12T17:03:00.746772+0000","last_became_active":"2024-02-12T16:59:09.241719+0000","last_became_peered":"2024-02-12T16:59:09.241719+0000","last_unstale":"2024-02-12T17:03:00.746772+0000","last_undegraded":"2024-02-12T17:03:00.746772+0000","last_fullsized":"2024-02-12T17:03:00.746772+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":151,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.758265+0000","last_change":"2024-02-12T17:02:34.738452+0000","last_active":"2024-02-12T17:03:00.758265+0000","last_peered":"2024-02-12T17:03:00.758265+0000","last_clean":"2024-02-12T17:03:00.758265+0000","last_became_active":"2024-02-12T16:59:09.259731+0000","last_became_peered":"2024-02-12T16:59:09.259731+0000","last_unstale":"2024-02-12T17:03:00.758265+0000","last_undegraded":"2024-02-12T17:03:00.758265+0000","last_fullsized":"2024-02-12T17:03:00.758265+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:34.738385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.738385+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":297,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754124+0000","last_change":"2024-02-12T16:57:37.670804+0000","last_active":"2024-02-12T17:03:00.754124+0000","last_peered":"2024-02-12T17:03:00.754124+0000","last_clean":"2024-02-12T17:03:00.754124+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T17:03:00.754124+0000","last_undegraded":"2024-02-12T17:03:00.754124+0000","last_fullsized":"2024-02-12T17:03:00.754124+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:57:37.670713+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.670713+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":303,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.750277+0000","last_change":"2024-02-12T16:57:26.790389+0000","last_active":"2024-02-12T17:03:00.750277+0000","last_peered":"2024-02-12T17:03:00.750277+0000","last_clean":"2024-02-12T17:03:00.750277+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T17:03:00.750277+0000","last_undegraded":"2024-02-12T17:03:00.750277+0000","last_fullsized":"2024-02-12T17:03:00.750277+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":376,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746404+0000","last_change":"2024-02-12T17:01:42.076658+0000","last_active":"2024-02-12T17:03:00.746404+0000","last_peered":"2024-02-12T17:03:00.746404+0000","last_clean":"2024-02-12T17:03:00.746404+0000","last_became_active":"2024-02-12T16:57:47.189652+0000","last_became_peered":"2024-02-12T16:57:47.189652+0000","last_unstale":"2024-02-12T17:03:00.746404+0000","last_undegraded":"2024-02-12T17:03:00.746404+0000","last_fullsized":"2024-02-12T17:03:00.746404+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:01:42.076584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.076584+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":225,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746236+0000","last_change":"2024-02-12T16:59:09.264851+0000","last_active":"2024-02-12T17:03:00.746236+0000","last_peered":"2024-02-12T17:03:00.746236+0000","last_clean":"2024-02-12T17:03:00.746236+0000","last_became_active":"2024-02-12T16:59:09.264513+0000","last_became_peered":"2024-02-12T16:59:09.264513+0000","last_unstale":"2024-02-12T17:03:00.746236+0000","last_undegraded":"2024-02-12T17:03:00.746236+0000","last_fullsized":"2024-02-12T17:03:00.746236+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":306,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.749410+0000","last_change":"2024-02-12T17:00:51.269702+0000","last_active":"2024-02-12T17:03:00.749410+0000","last_peered":"2024-02-12T17:03:00.749410+0000","last_clean":"2024-02-12T17:03:00.749410+0000","last_became_active":"2024-02-12T16:59:09.270516+0000","last_became_peered":"2024-02-12T16:59:09.270516+0000","last_unstale":"2024-02-12T17:03:00.749410+0000","last_undegraded":"2024-02-12T17:03:00.749410+0000","last_fullsized":"2024-02-12T17:03:00.749410+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:51.269648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.269648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":91,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.758013+0000","last_change":"2024-02-12T16:59:09.260879+0000","last_active":"2024-02-12T17:03:00.758013+0000","last_peered":"2024-02-12T17:03:00.758013+0000","last_clean":"2024-02-12T17:03:00.758013+0000","last_became_active":"2024-02-12T16:59:09.259058+0000","last_became_peered":"2024-02-12T16:59:09.259058+0000","last_unstale":"2024-02-12T17:03:00.758013+0000","last_undegraded":"2024-02-12T17:03:00.758013+0000","last_fullsized":"2024-02-12T17:03:00.758013+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":297,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753223+0000","last_change":"2024-02-12T17:02:39.569673+0000","last_active":"2024-02-12T17:03:00.753223+0000","last_peered":"2024-02-12T17:03:00.753223+0000","last_clean":"2024-02-12T17:03:00.753223+0000","last_became_active":"2024-02-12T16:59:09.267628+0000","last_became_peered":"2024-02-12T16:59:09.267628+0000","last_unstale":"2024-02-12T17:03:00.753223+0000","last_undegraded":"2024-02-12T17:03:00.753223+0000","last_fullsized":"2024-02-12T17:03:00.753223+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":295,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755446+0000","last_change":"2024-02-12T16:57:28.846844+0000","last_active":"2024-02-12T17:03:00.755446+0000","last_peered":"2024-02-12T17:03:00.755446+0000","last_clean":"2024-02-12T17:03:00.755446+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T17:03:00.755446+0000","last_undegraded":"2024-02-12T17:03:00.755446+0000","last_fullsized":"2024-02-12T17:03:00.755446+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":101,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.740550+0000","last_change":"2024-02-12T17:02:47.020937+0000","last_active":"2024-02-12T17:03:00.740550+0000","last_peered":"2024-02-12T17:03:00.740550+0000","last_clean":"2024-02-12T17:03:00.740550+0000","last_became_active":"2024-02-12T16:59:09.247346+0000","last_became_peered":"2024-02-12T16:59:09.247346+0000","last_unstale":"2024-02-12T17:03:00.740550+0000","last_undegraded":"2024-02-12T17:03:00.740550+0000","last_fullsized":"2024-02-12T17:03:00.740550+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.020845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.020845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":312,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.756032+0000","last_change":"2024-02-12T16:59:09.265096+0000","last_active":"2024-02-12T17:03:00.756032+0000","last_peered":"2024-02-12T17:03:00.756032+0000","last_clean":"2024-02-12T17:03:00.756032+0000","last_became_active":"2024-02-12T16:59:09.264558+0000","last_became_peered":"2024-02-12T16:59:09.264558+0000","last_unstale":"2024-02-12T17:03:00.756032+0000","last_undegraded":"2024-02-12T17:03:00.756032+0000","last_fullsized":"2024-02-12T17:03:00.756032+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":131,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.750055+0000","last_change":"2024-02-12T17:00:58.938867+0000","last_active":"2024-02-12T17:03:00.750055+0000","last_peered":"2024-02-12T17:03:00.750055+0000","last_clean":"2024-02-12T17:03:00.750055+0000","last_became_active":"2024-02-12T16:59:09.280196+0000","last_became_peered":"2024-02-12T16:59:09.280196+0000","last_unstale":"2024-02-12T17:03:00.750055+0000","last_undegraded":"2024-02-12T17:03:00.750055+0000","last_fullsized":"2024-02-12T17:03:00.750055+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.938805+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":221,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746704+0000","last_change":"2024-02-12T17:01:30.051615+0000","last_active":"2024-02-12T17:03:00.746704+0000","last_peered":"2024-02-12T17:03:00.746704+0000","last_clean":"2024-02-12T17:03:00.746704+0000","last_became_active":"2024-02-12T16:54:40.641212+0000","last_became_peered":"2024-02-12T16:54:40.641212+0000","last_unstale":"2024-02-12T17:03:00.746704+0000","last_undegraded":"2024-02-12T17:03:00.746704+0000","last_fullsized":"2024-02-12T17:03:00.746704+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.051528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.051528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":250,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.751470+0000","last_change":"2024-02-12T16:59:09.263555+0000","last_active":"2024-02-12T17:03:00.751470+0000","last_peered":"2024-02-12T17:03:00.751470+0000","last_clean":"2024-02-12T17:03:00.751470+0000","last_became_active":"2024-02-12T16:59:09.262674+0000","last_became_peered":"2024-02-12T16:59:09.262674+0000","last_unstale":"2024-02-12T17:03:00.751470+0000","last_undegraded":"2024-02-12T17:03:00.751470+0000","last_fullsized":"2024-02-12T17:03:00.751470+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":155,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755988+0000","last_change":"2024-02-12T17:01:02.858477+0000","last_active":"2024-02-12T17:03:00.755988+0000","last_peered":"2024-02-12T17:03:00.755988+0000","last_clean":"2024-02-12T17:03:00.755988+0000","last_became_active":"2024-02-12T16:59:09.257167+0000","last_became_peered":"2024-02-12T16:59:09.257167+0000","last_unstale":"2024-02-12T17:03:00.755988+0000","last_undegraded":"2024-02-12T17:03:00.755988+0000","last_fullsized":"2024-02-12T17:03:00.755988+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.858411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.858411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":387,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.740328+0000","last_change":"2024-02-12T17:02:16.065061+0000","last_active":"2024-02-12T17:03:00.740328+0000","last_peered":"2024-02-12T17:03:00.740328+0000","last_clean":"2024-02-12T17:03:00.740328+0000","last_became_active":"2024-02-12T16:59:09.274991+0000","last_became_peered":"2024-02-12T16:59:09.274991+0000","last_unstale":"2024-02-12T17:03:00.740328+0000","last_undegraded":"2024-02-12T17:03:00.740328+0000","last_fullsized":"2024-02-12T17:03:00.740328+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:16.064966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.064966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":313,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745282+0000","last_change":"2024-02-12T17:00:58.019883+0000","last_active":"2024-02-12T17:03:00.745282+0000","last_peered":"2024-02-12T17:03:00.745282+0000","last_clean":"2024-02-12T17:03:00.745282+0000","last_became_active":"2024-02-12T16:59:09.269260+0000","last_became_peered":"2024-02-12T16:59:09.269260+0000","last_unstale":"2024-02-12T17:03:00.745282+0000","last_undegraded":"2024-02-12T17:03:00.745282+0000","last_fullsized":"2024-02-12T17:03:00.745282+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.019813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.019813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":246,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746161+0000","last_change":"2024-02-12T17:01:33.149239+0000","last_active":"2024-02-12T17:03:00.746161+0000","last_peered":"2024-02-12T17:03:00.746161+0000","last_clean":"2024-02-12T17:03:00.746161+0000","last_became_active":"2024-02-12T16:57:47.188909+0000","last_became_peered":"2024-02-12T16:57:47.188909+0000","last_unstale":"2024-02-12T17:03:00.746161+0000","last_undegraded":"2024-02-12T17:03:00.746161+0000","last_fullsized":"2024-02-12T17:03:00.746161+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:33.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:01:33.149132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":194,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755471+0000","last_change":"2024-02-12T17:02:40.656607+0000","last_active":"2024-02-12T17:03:00.755471+0000","last_peered":"2024-02-12T17:03:00.755471+0000","last_clean":"2024-02-12T17:03:00.755471+0000","last_became_active":"2024-02-12T16:59:09.260045+0000","last_became_peered":"2024-02-12T16:59:09.260045+0000","last_unstale":"2024-02-12T17:03:00.755471+0000","last_undegraded":"2024-02-12T17:03:00.755471+0000","last_fullsized":"2024-02-12T17:03:00.755471+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":136,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745563+0000","last_change":"2024-02-12T17:01:16.911465+0000","last_active":"2024-02-12T17:03:00.745563+0000","last_peered":"2024-02-12T17:03:00.745563+0000","last_clean":"2024-02-12T17:03:00.745563+0000","last_became_active":"2024-02-12T17:01:16.911137+0000","last_became_peered":"2024-02-12T17:01:16.911137+0000","last_unstale":"2024-02-12T17:03:00.745563+0000","last_undegraded":"2024-02-12T17:03:00.745563+0000","last_fullsized":"2024-02-12T17:03:00.745563+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:00:53.792980+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.792980+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":49,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.757950+0000","last_change":"2024-02-12T17:02:36.771544+0000","last_active":"2024-02-12T17:03:00.757950+0000","last_peered":"2024-02-12T17:03:00.757950+0000","last_clean":"2024-02-12T17:03:00.757950+0000","last_became_active":"2024-02-12T17:01:16.909120+0000","last_became_peered":"2024-02-12T17:01:16.909120+0000","last_unstale":"2024-02-12T17:03:00.757950+0000","last_undegraded":"2024-02-12T17:03:00.757950+0000","last_fullsized":"2024-02-12T17:03:00.757950+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:36.771446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:36.771446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":380,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.410534+0000","last_change":"2024-02-12T17:02:17.689314+0000","last_active":"2024-02-12T17:03:07.410534+0000","last_peered":"2024-02-12T17:03:07.410534+0000","last_clean":"2024-02-12T17:03:07.410534+0000","last_became_active":"2024-02-12T16:57:47.185109+0000","last_became_peered":"2024-02-12T16:57:47.185109+0000","last_unstale":"2024-02-12T17:03:07.410534+0000","last_undegraded":"2024-02-12T17:03:07.410534+0000","last_fullsized":"2024-02-12T17:03:07.410534+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-12T17:02:17.689218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.689218+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":310,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746508+0000","last_change":"2024-02-12T17:01:02.953989+0000","last_active":"2024-02-12T17:03:00.746508+0000","last_peered":"2024-02-12T17:03:00.746508+0000","last_clean":"2024-02-12T17:03:00.746508+0000","last_became_active":"2024-02-12T16:54:40.628945+0000","last_became_peered":"2024-02-12T16:54:40.628945+0000","last_unstale":"2024-02-12T17:03:00.746508+0000","last_undegraded":"2024-02-12T17:03:00.746508+0000","last_fullsized":"2024-02-12T17:03:00.746508+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.953891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.953891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":364,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755905+0000","last_change":"2024-02-12T17:01:31.827739+0000","last_active":"2024-02-12T17:03:00.755905+0000","last_peered":"2024-02-12T17:03:00.755905+0000","last_clean":"2024-02-12T17:03:00.755905+0000","last_became_active":"2024-02-12T16:59:09.258515+0000","last_became_peered":"2024-02-12T16:59:09.258515+0000","last_unstale":"2024-02-12T17:03:00.755905+0000","last_undegraded":"2024-02-12T17:03:00.755905+0000","last_fullsized":"2024-02-12T17:03:00.755905+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:31.827632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:31.827632+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":304,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746033+0000","last_change":"2024-02-12T16:57:25.835979+0000","last_active":"2024-02-12T17:03:00.746033+0000","last_peered":"2024-02-12T17:03:00.746033+0000","last_clean":"2024-02-12T17:03:00.746033+0000","last_became_active":"2024-02-12T16:54:40.632717+0000","last_became_peered":"2024-02-12T16:54:40.632717+0000","last_unstale":"2024-02-12T17:03:00.746033+0000","last_undegraded":"2024-02-12T17:03:00.746033+0000","last_fullsized":"2024-02-12T17:03:00.746033+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.835883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.835883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"208'9","reported_seq":484,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.410628+0000","last_change":"2024-02-12T17:00:49.846525+0000","last_active":"2024-02-12T17:03:07.410628+0000","last_peered":"2024-02-12T17:03:07.410628+0000","last_clean":"2024-02-12T17:03:07.410628+0000","last_became_active":"2024-02-12T16:57:47.178688+0000","last_became_peered":"2024-02-12T16:57:47.178688+0000","last_unstale":"2024-02-12T17:03:07.410628+0000","last_undegraded":"2024-02-12T17:03:07.410628+0000","last_fullsized":"2024-02-12T17:03:07.410628+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'9","last_scrub_stamp":"2024-02-12T17:00:49.846432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.846432+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":312,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.751190+0000","last_change":"2024-02-12T17:01:34.874427+0000","last_active":"2024-02-12T17:03:00.751190+0000","last_peered":"2024-02-12T17:03:00.751190+0000","last_clean":"2024-02-12T17:03:00.751190+0000","last_became_active":"2024-02-12T16:54:40.626637+0000","last_became_peered":"2024-02-12T16:54:40.626637+0000","last_unstale":"2024-02-12T17:03:00.751190+0000","last_undegraded":"2024-02-12T17:03:00.751190+0000","last_fullsized":"2024-02-12T17:03:00.751190+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:34.874363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:34.874363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":195,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753886+0000","last_change":"2024-02-12T16:57:48.058146+0000","last_active":"2024-02-12T17:03:00.753886+0000","last_peered":"2024-02-12T17:03:00.753886+0000","last_clean":"2024-02-12T17:03:00.753886+0000","last_became_active":"2024-02-12T16:57:47.199140+0000","last_became_peered":"2024-02-12T16:57:47.199140+0000","last_unstale":"2024-02-12T17:03:00.753886+0000","last_undegraded":"2024-02-12T17:03:00.753886+0000","last_fullsized":"2024-02-12T17:03:00.753886+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":308,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753819+0000","last_change":"2024-02-12T17:00:58.187842+0000","last_active":"2024-02-12T17:03:00.753819+0000","last_peered":"2024-02-12T17:03:00.753819+0000","last_clean":"2024-02-12T17:03:00.753819+0000","last_became_active":"2024-02-12T16:59:09.269414+0000","last_became_peered":"2024-02-12T16:59:09.269414+0000","last_unstale":"2024-02-12T17:03:00.753819+0000","last_undegraded":"2024-02-12T17:03:00.753819+0000","last_fullsized":"2024-02-12T17:03:00.753819+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.187738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":269,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.740673+0000","last_change":"2024-02-12T17:01:32.127692+0000","last_active":"2024-02-12T17:03:00.740673+0000","last_peered":"2024-02-12T17:03:00.740673+0000","last_clean":"2024-02-12T17:03:00.740673+0000","last_became_active":"2024-02-12T16:59:09.243603+0000","last_became_peered":"2024-02-12T16:59:09.243603+0000","last_unstale":"2024-02-12T17:03:00.740673+0000","last_undegraded":"2024-02-12T17:03:00.740673+0000","last_fullsized":"2024-02-12T17:03:00.740673+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:32.127593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:32.127593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":288,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755406+0000","last_change":"2024-02-12T17:00:53.269198+0000","last_active":"2024-02-12T17:03:00.755406+0000","last_peered":"2024-02-12T17:03:00.755406+0000","last_clean":"2024-02-12T17:03:00.755406+0000","last_became_active":"2024-02-12T16:59:09.242413+0000","last_became_peered":"2024-02-12T16:59:09.242413+0000","last_unstale":"2024-02-12T17:03:00.755406+0000","last_undegraded":"2024-02-12T17:03:00.755406+0000","last_fullsized":"2024-02-12T17:03:00.755406+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-02-12T17:00:53.269107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.269107+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":51,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.742632+0000","last_change":"2024-02-12T17:02:43.982708+0000","last_active":"2024-02-12T17:03:00.742632+0000","last_peered":"2024-02-12T17:03:00.742632+0000","last_clean":"2024-02-12T17:03:00.742632+0000","last_became_active":"2024-02-12T17:01:16.910812+0000","last_became_peered":"2024-02-12T17:01:16.910812+0000","last_unstale":"2024-02-12T17:03:00.742632+0000","last_undegraded":"2024-02-12T17:03:00.742632+0000","last_fullsized":"2024-02-12T17:03:00.742632+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:43.982623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.982623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":215,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745534+0000","last_change":"2024-02-12T17:01:16.912921+0000","last_active":"2024-02-12T17:03:00.745534+0000","last_peered":"2024-02-12T17:03:00.745534+0000","last_clean":"2024-02-12T17:03:00.745534+0000","last_became_active":"2024-02-12T17:01:16.911289+0000","last_became_peered":"2024-02-12T17:01:16.911289+0000","last_unstale":"2024-02-12T17:03:00.745534+0000","last_undegraded":"2024-02-12T17:03:00.745534+0000","last_fullsized":"2024-02-12T17:03:00.745534+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.824194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.824194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":347,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746140+0000","last_change":"2024-02-12T17:00:53.936787+0000","last_active":"2024-02-12T17:03:00.746140+0000","last_peered":"2024-02-12T17:03:00.746140+0000","last_clean":"2024-02-12T17:03:00.746140+0000","last_became_active":"2024-02-12T16:54:40.642507+0000","last_became_peered":"2024-02-12T16:54:40.642507+0000","last_unstale":"2024-02-12T17:03:00.746140+0000","last_undegraded":"2024-02-12T17:03:00.746140+0000","last_fullsized":"2024-02-12T17:03:00.746140+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:00:53.936664+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.936664+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"337'9311","reported_seq":9352,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.605451+0000","last_change":"2024-02-12T17:01:13.881464+0000","last_active":"2024-02-12T17:03:08.605451+0000","last_peered":"2024-02-12T17:03:08.605451+0000","last_clean":"2024-02-12T17:03:08.605451+0000","last_became_active":"2024-02-12T17:01:13.881178+0000","last_became_peered":"2024-02-12T17:01:13.881178+0000","last_unstale":"2024-02-12T17:03:08.605451+0000","last_undegraded":"2024-02-12T17:03:08.605451+0000","last_fullsized":"2024-02-12T17:03:08.605451+0000","mapping_epoch":304,"log_start":"327'5800","ondisk_log_start":"327'5800","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3511,"ondisk_log_size":3511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":508624896,"num_objects":7761,"num_object_clones":0,"num_object_copies":23283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9311,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":234,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754592+0000","last_change":"2024-02-12T17:01:28.912974+0000","last_active":"2024-02-12T17:03:00.754592+0000","last_peered":"2024-02-12T17:03:00.754592+0000","last_clean":"2024-02-12T17:03:00.754592+0000","last_became_active":"2024-02-12T16:59:09.272706+0000","last_became_peered":"2024-02-12T16:59:09.272706+0000","last_unstale":"2024-02-12T17:03:00.754592+0000","last_undegraded":"2024-02-12T17:03:00.754592+0000","last_fullsized":"2024-02-12T17:03:00.754592+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:28.912872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.912872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":333,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.741652+0000","last_change":"2024-02-12T17:02:17.066439+0000","last_active":"2024-02-12T17:03:00.741652+0000","last_peered":"2024-02-12T17:03:00.741652+0000","last_clean":"2024-02-12T17:03:00.741652+0000","last_became_active":"2024-02-12T16:49:39.105096+0000","last_became_peered":"2024-02-12T16:49:39.105096+0000","last_unstale":"2024-02-12T17:03:00.741652+0000","last_undegraded":"2024-02-12T17:03:00.741652+0000","last_fullsized":"2024-02-12T17:03:00.741652+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:02:17.066331+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.066331+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":207,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746068+0000","last_change":"2024-02-12T16:59:09.952265+0000","last_active":"2024-02-12T17:03:00.746068+0000","last_peered":"2024-02-12T17:03:00.746068+0000","last_clean":"2024-02-12T17:03:00.746068+0000","last_became_active":"2024-02-12T16:59:09.264482+0000","last_became_peered":"2024-02-12T16:59:09.264482+0000","last_unstale":"2024-02-12T17:03:00.746068+0000","last_undegraded":"2024-02-12T17:03:00.746068+0000","last_fullsized":"2024-02-12T17:03:00.746068+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:22.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.764984+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":257,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745871+0000","last_change":"2024-02-12T17:02:25.815917+0000","last_active":"2024-02-12T17:03:00.745871+0000","last_peered":"2024-02-12T17:03:00.745871+0000","last_clean":"2024-02-12T17:03:00.745871+0000","last_became_active":"2024-02-12T17:01:16.912888+0000","last_became_peered":"2024-02-12T17:01:16.912888+0000","last_unstale":"2024-02-12T17:03:00.745871+0000","last_undegraded":"2024-02-12T17:03:00.745871+0000","last_fullsized":"2024-02-12T17:03:00.745871+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:25.815816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.815816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"337'9494","reported_seq":9535,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.508319+0000","last_change":"2024-02-12T17:01:13.886736+0000","last_active":"2024-02-12T17:03:08.508319+0000","last_peered":"2024-02-12T17:03:08.508319+0000","last_clean":"2024-02-12T17:03:08.508319+0000","last_became_active":"2024-02-12T17:01:13.885880+0000","last_became_peered":"2024-02-12T17:01:13.885880+0000","last_unstale":"2024-02-12T17:03:08.508319+0000","last_undegraded":"2024-02-12T17:03:08.508319+0000","last_fullsized":"2024-02-12T17:03:08.508319+0000","mapping_epoch":304,"log_start":"327'6000","ondisk_log_start":"327'6000","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3494,"ondisk_log_size":3494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":524550144,"num_objects":8004,"num_object_clones":0,"num_object_copies":24012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":304,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745283+0000","last_change":"2024-02-12T16:57:29.934046+0000","last_active":"2024-02-12T17:03:00.745283+0000","last_peered":"2024-02-12T17:03:00.745283+0000","last_clean":"2024-02-12T17:03:00.745283+0000","last_became_active":"2024-02-12T16:54:40.637737+0000","last_became_peered":"2024-02-12T16:54:40.637737+0000","last_unstale":"2024-02-12T17:03:00.745283+0000","last_undegraded":"2024-02-12T17:03:00.745283+0000","last_fullsized":"2024-02-12T17:03:00.745283+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":146,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746413+0000","last_change":"2024-02-12T17:02:30.758581+0000","last_active":"2024-02-12T17:03:00.746413+0000","last_peered":"2024-02-12T17:03:00.746413+0000","last_clean":"2024-02-12T17:03:00.746413+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:03:00.746413+0000","last_undegraded":"2024-02-12T17:03:00.746413+0000","last_fullsized":"2024-02-12T17:03:00.746413+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.758520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.758520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":195,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.751549+0000","last_change":"2024-02-12T16:57:48.006795+0000","last_active":"2024-02-12T17:03:00.751549+0000","last_peered":"2024-02-12T17:03:00.751549+0000","last_clean":"2024-02-12T17:03:00.751549+0000","last_became_active":"2024-02-12T16:57:47.202531+0000","last_became_peered":"2024-02-12T16:57:47.202531+0000","last_unstale":"2024-02-12T17:03:00.751549+0000","last_undegraded":"2024-02-12T17:03:00.751549+0000","last_fullsized":"2024-02-12T17:03:00.751549+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"299'10","reported_seq":481,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.409770+0000","last_change":"2024-02-12T17:00:51.955785+0000","last_active":"2024-02-12T17:03:07.409770+0000","last_peered":"2024-02-12T17:03:07.409770+0000","last_clean":"2024-02-12T17:03:07.409770+0000","last_became_active":"2024-02-12T16:59:09.270667+0000","last_became_peered":"2024-02-12T16:59:09.270667+0000","last_unstale":"2024-02-12T17:03:07.409770+0000","last_undegraded":"2024-02-12T17:03:07.409770+0000","last_fullsized":"2024-02-12T17:03:07.409770+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"337'9331","reported_seq":9372,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.582286+0000","last_change":"2024-02-12T17:01:13.884406+0000","last_active":"2024-02-12T17:03:08.582286+0000","last_peered":"2024-02-12T17:03:08.582286+0000","last_clean":"2024-02-12T17:03:08.582286+0000","last_became_active":"2024-02-12T17:01:13.884065+0000","last_became_peered":"2024-02-12T17:01:13.884065+0000","last_unstale":"2024-02-12T17:03:08.582286+0000","last_undegraded":"2024-02-12T17:03:08.582286+0000","last_fullsized":"2024-02-12T17:03:08.582286+0000","mapping_epoch":304,"log_start":"327'5800","ondisk_log_start":"327'5800","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3531,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":511639552,"num_objects":7807,"num_object_clones":0,"num_object_copies":23421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9331,"num_write_kb":548416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":239,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.742295+0000","last_change":"2024-02-12T17:02:42.958589+0000","last_active":"2024-02-12T17:03:00.742295+0000","last_peered":"2024-02-12T17:03:00.742295+0000","last_clean":"2024-02-12T17:03:00.742295+0000","last_became_active":"2024-02-12T16:59:09.264617+0000","last_became_peered":"2024-02-12T16:59:09.264617+0000","last_unstale":"2024-02-12T17:03:00.742295+0000","last_undegraded":"2024-02-12T17:03:00.742295+0000","last_fullsized":"2024-02-12T17:03:00.742295+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":326,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.751263+0000","last_change":"2024-02-12T17:01:42.892672+0000","last_active":"2024-02-12T17:03:00.751263+0000","last_peered":"2024-02-12T17:03:00.751263+0000","last_clean":"2024-02-12T17:03:00.751263+0000","last_became_active":"2024-02-12T16:59:09.261764+0000","last_became_peered":"2024-02-12T16:59:09.261764+0000","last_unstale":"2024-02-12T17:03:00.751263+0000","last_undegraded":"2024-02-12T17:03:00.751263+0000","last_fullsized":"2024-02-12T17:03:00.751263+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:42.892563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.892563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"308'54","reported_seq":550,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.410714+0000","last_change":"2024-02-12T17:02:37.537794+0000","last_active":"2024-02-12T17:03:07.410714+0000","last_peered":"2024-02-12T17:03:07.410714+0000","last_clean":"2024-02-12T17:03:07.410714+0000","last_became_active":"2024-02-12T17:01:16.911487+0000","last_became_peered":"2024-02-12T17:01:16.911487+0000","last_unstale":"2024-02-12T17:03:07.410714+0000","last_undegraded":"2024-02-12T17:03:07.410714+0000","last_fullsized":"2024-02-12T17:03:07.410714+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":320,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746155+0000","last_change":"2024-02-12T17:02:35.760851+0000","last_active":"2024-02-12T17:03:00.746155+0000","last_peered":"2024-02-12T17:03:00.746155+0000","last_clean":"2024-02-12T17:03:00.746155+0000","last_became_active":"2024-02-12T16:59:09.264694+0000","last_became_peered":"2024-02-12T16:59:09.264694+0000","last_unstale":"2024-02-12T17:03:00.746155+0000","last_undegraded":"2024-02-12T17:03:00.746155+0000","last_fullsized":"2024-02-12T17:03:00.746155+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"337'9463","reported_seq":9504,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.544119+0000","last_change":"2024-02-12T17:01:13.884788+0000","last_active":"2024-02-12T17:03:08.544119+0000","last_peered":"2024-02-12T17:03:08.544119+0000","last_clean":"2024-02-12T17:03:08.544119+0000","last_became_active":"2024-02-12T17:01:13.884280+0000","last_became_peered":"2024-02-12T17:01:13.884280+0000","last_unstale":"2024-02-12T17:03:08.544119+0000","last_undegraded":"2024-02-12T17:03:08.544119+0000","last_fullsized":"2024-02-12T17:03:08.544119+0000","mapping_epoch":304,"log_start":"327'6000","ondisk_log_start":"327'6000","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3463,"ondisk_log_size":3463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":518979584,"num_objects":7919,"num_object_clones":0,"num_object_copies":23757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9463,"num_write_kb":556224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":315,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754986+0000","last_change":"2024-02-12T17:01:04.085939+0000","last_active":"2024-02-12T17:03:00.754986+0000","last_peered":"2024-02-12T17:03:00.754986+0000","last_clean":"2024-02-12T17:03:00.754986+0000","last_became_active":"2024-02-12T16:59:09.269488+0000","last_became_peered":"2024-02-12T16:59:09.269488+0000","last_unstale":"2024-02-12T17:03:00.754986+0000","last_undegraded":"2024-02-12T17:03:00.754986+0000","last_fullsized":"2024-02-12T17:03:00.754986+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:04.085827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:04.085827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":179,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.262527+0000","last_change":"2024-02-12T17:01:16.913133+0000","last_active":"2024-02-12T17:03:08.262527+0000","last_peered":"2024-02-12T17:03:08.262527+0000","last_clean":"2024-02-12T17:03:08.262527+0000","last_became_active":"2024-02-12T17:01:16.912041+0000","last_became_peered":"2024-02-12T17:01:16.912041+0000","last_unstale":"2024-02-12T17:03:08.262527+0000","last_undegraded":"2024-02-12T17:03:08.262527+0000","last_fullsized":"2024-02-12T17:03:08.262527+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:00:52.838852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:52.838852+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":290,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746905+0000","last_change":"2024-02-12T17:01:38.142697+0000","last_active":"2024-02-12T17:03:00.746905+0000","last_peered":"2024-02-12T17:03:00.746905+0000","last_clean":"2024-02-12T17:03:00.746905+0000","last_became_active":"2024-02-12T17:01:16.911675+0000","last_became_peered":"2024-02-12T17:01:16.911675+0000","last_unstale":"2024-02-12T17:03:00.746905+0000","last_undegraded":"2024-02-12T17:03:00.746905+0000","last_fullsized":"2024-02-12T17:03:00.746905+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:38.142602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:38.142602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":227,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.758082+0000","last_change":"2024-02-12T17:02:25.762303+0000","last_active":"2024-02-12T17:03:00.758082+0000","last_peered":"2024-02-12T17:03:00.758082+0000","last_clean":"2024-02-12T17:03:00.758082+0000","last_became_active":"2024-02-12T16:57:47.190968+0000","last_became_peered":"2024-02-12T16:57:47.190968+0000","last_unstale":"2024-02-12T17:03:00.758082+0000","last_undegraded":"2024-02-12T17:03:00.758082+0000","last_fullsized":"2024-02-12T17:03:00.758082+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":251,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.756102+0000","last_change":"2024-02-12T17:01:16.909653+0000","last_active":"2024-02-12T17:03:00.756102+0000","last_peered":"2024-02-12T17:03:00.756102+0000","last_clean":"2024-02-12T17:03:00.756102+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:03:00.756102+0000","last_undegraded":"2024-02-12T17:03:00.756102+0000","last_fullsized":"2024-02-12T17:03:00.756102+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"337'9359","reported_seq":9400,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.632017+0000","last_change":"2024-02-12T17:01:13.886964+0000","last_active":"2024-02-12T17:03:08.632017+0000","last_peered":"2024-02-12T17:03:08.632017+0000","last_clean":"2024-02-12T17:03:08.632017+0000","last_became_active":"2024-02-12T17:01:13.886738+0000","last_became_peered":"2024-02-12T17:01:13.886738+0000","last_unstale":"2024-02-12T17:03:08.632017+0000","last_undegraded":"2024-02-12T17:03:08.632017+0000","last_fullsized":"2024-02-12T17:03:08.632017+0000","mapping_epoch":304,"log_start":"327'5900","ondisk_log_start":"327'5900","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3459,"ondisk_log_size":3459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512557056,"num_objects":7821,"num_object_clones":0,"num_object_copies":23463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9359,"num_write_kb":549760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":315,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.758394+0000","last_change":"2024-02-12T17:00:57.950368+0000","last_active":"2024-02-12T17:03:00.758394+0000","last_peered":"2024-02-12T17:03:00.758394+0000","last_clean":"2024-02-12T17:03:00.758394+0000","last_became_active":"2024-02-12T16:57:47.191375+0000","last_became_peered":"2024-02-12T16:57:47.191375+0000","last_unstale":"2024-02-12T17:03:00.758394+0000","last_undegraded":"2024-02-12T17:03:00.758394+0000","last_fullsized":"2024-02-12T17:03:00.758394+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:57.950284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:00:57.950284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":225,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.751398+0000","last_change":"2024-02-12T16:59:09.263388+0000","last_active":"2024-02-12T17:03:00.751398+0000","last_peered":"2024-02-12T17:03:00.751398+0000","last_clean":"2024-02-12T17:03:00.751398+0000","last_became_active":"2024-02-12T16:59:09.262266+0000","last_became_peered":"2024-02-12T16:59:09.262266+0000","last_unstale":"2024-02-12T17:03:00.751398+0000","last_undegraded":"2024-02-12T17:03:00.751398+0000","last_fullsized":"2024-02-12T17:03:00.751398+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":303,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.751157+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:03:00.751157+0000","last_peered":"2024-02-12T17:03:00.751157+0000","last_clean":"2024-02-12T17:03:00.751157+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:03:00.751157+0000","last_undegraded":"2024-02-12T17:03:00.751157+0000","last_fullsized":"2024-02-12T17:03:00.751157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":318,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.749311+0000","last_change":"2024-02-12T17:01:25.871008+0000","last_active":"2024-02-12T17:03:00.749311+0000","last_peered":"2024-02-12T17:03:00.749311+0000","last_clean":"2024-02-12T17:03:00.749311+0000","last_became_active":"2024-02-12T16:59:09.245635+0000","last_became_peered":"2024-02-12T16:59:09.245635+0000","last_unstale":"2024-02-12T17:03:00.749311+0000","last_undegraded":"2024-02-12T17:03:00.749311+0000","last_fullsized":"2024-02-12T17:03:00.749311+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:25.870916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:01:25.870916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":164,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.742685+0000","last_change":"2024-02-12T17:01:43.163816+0000","last_active":"2024-02-12T17:03:00.742685+0000","last_peered":"2024-02-12T17:03:00.742685+0000","last_clean":"2024-02-12T17:03:00.742685+0000","last_became_active":"2024-02-12T16:59:09.249903+0000","last_became_peered":"2024-02-12T16:59:09.249903+0000","last_unstale":"2024-02-12T17:03:00.742685+0000","last_undegraded":"2024-02-12T17:03:00.742685+0000","last_fullsized":"2024-02-12T17:03:00.742685+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'26","last_scrub_stamp":"2024-02-12T17:01:43.163724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:43.163724+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":297,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753960+0000","last_change":"2024-02-12T16:57:17.945040+0000","last_active":"2024-02-12T17:03:00.753960+0000","last_peered":"2024-02-12T17:03:00.753960+0000","last_clean":"2024-02-12T17:03:00.753960+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:03:00.753960+0000","last_undegraded":"2024-02-12T17:03:00.753960+0000","last_fullsized":"2024-02-12T17:03:00.753960+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.944951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.944951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"337'9367","reported_seq":9408,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.594525+0000","last_change":"2024-02-12T17:01:13.885922+0000","last_active":"2024-02-12T17:03:08.594525+0000","last_peered":"2024-02-12T17:03:08.594525+0000","last_clean":"2024-02-12T17:03:08.594525+0000","last_became_active":"2024-02-12T17:01:13.885633+0000","last_became_peered":"2024-02-12T17:01:13.885633+0000","last_unstale":"2024-02-12T17:03:08.594525+0000","last_undegraded":"2024-02-12T17:03:08.594525+0000","last_fullsized":"2024-02-12T17:03:08.594525+0000","mapping_epoch":304,"log_start":"327'5900","ondisk_log_start":"327'5900","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3467,"ondisk_log_size":3467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":513867776,"num_objects":7841,"num_object_clones":0,"num_object_copies":23523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":550656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":243,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.750432+0000","last_change":"2024-02-12T16:59:09.252727+0000","last_active":"2024-02-12T17:03:00.750432+0000","last_peered":"2024-02-12T17:03:00.750432+0000","last_clean":"2024-02-12T17:03:00.750432+0000","last_became_active":"2024-02-12T16:59:09.252442+0000","last_became_peered":"2024-02-12T16:59:09.252442+0000","last_unstale":"2024-02-12T17:03:00.750432+0000","last_undegraded":"2024-02-12T17:03:00.750432+0000","last_fullsized":"2024-02-12T17:03:00.750432+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":310,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746623+0000","last_change":"2024-02-12T17:00:54.941856+0000","last_active":"2024-02-12T17:03:00.746623+0000","last_peered":"2024-02-12T17:03:00.746623+0000","last_clean":"2024-02-12T17:03:00.746623+0000","last_became_active":"2024-02-12T16:54:40.633903+0000","last_became_peered":"2024-02-12T16:54:40.633903+0000","last_unstale":"2024-02-12T17:03:00.746623+0000","last_undegraded":"2024-02-12T17:03:00.746623+0000","last_fullsized":"2024-02-12T17:03:00.746623+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.941776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":320,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.749220+0000","last_change":"2024-02-12T16:57:48.350132+0000","last_active":"2024-02-12T17:03:00.749220+0000","last_peered":"2024-02-12T17:03:00.749220+0000","last_clean":"2024-02-12T17:03:00.749220+0000","last_became_active":"2024-02-12T16:57:47.200347+0000","last_became_peered":"2024-02-12T16:57:47.200347+0000","last_unstale":"2024-02-12T17:03:00.749220+0000","last_undegraded":"2024-02-12T17:03:00.749220+0000","last_fullsized":"2024-02-12T17:03:00.749220+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":358,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746030+0000","last_change":"2024-02-12T16:57:47.204108+0000","last_active":"2024-02-12T17:03:00.746030+0000","last_peered":"2024-02-12T17:03:00.746030+0000","last_clean":"2024-02-12T17:03:00.746030+0000","last_became_active":"2024-02-12T16:57:47.184456+0000","last_became_peered":"2024-02-12T16:57:47.184456+0000","last_unstale":"2024-02-12T17:03:00.746030+0000","last_undegraded":"2024-02-12T17:03:00.746030+0000","last_fullsized":"2024-02-12T17:03:00.746030+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":326,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.764475+0000","last_change":"2024-02-12T17:02:33.737620+0000","last_active":"2024-02-12T17:03:00.764475+0000","last_peered":"2024-02-12T17:03:00.764475+0000","last_clean":"2024-02-12T17:03:00.764475+0000","last_became_active":"2024-02-12T16:58:14.391221+0000","last_became_peered":"2024-02-12T16:58:14.391221+0000","last_unstale":"2024-02-12T17:03:00.764475+0000","last_undegraded":"2024-02-12T17:03:00.764475+0000","last_fullsized":"2024-02-12T17:03:00.764475+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"337'9513","reported_seq":9554,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.520299+0000","last_change":"2024-02-12T17:01:13.890189+0000","last_active":"2024-02-12T17:03:08.520299+0000","last_peered":"2024-02-12T17:03:08.520299+0000","last_clean":"2024-02-12T17:03:08.520299+0000","last_became_active":"2024-02-12T17:01:13.889786+0000","last_became_peered":"2024-02-12T17:01:13.889786+0000","last_unstale":"2024-02-12T17:03:08.520299+0000","last_undegraded":"2024-02-12T17:03:08.520299+0000","last_fullsized":"2024-02-12T17:03:08.520299+0000","mapping_epoch":304,"log_start":"327'5900","ondisk_log_start":"327'5900","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3613,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":522387456,"num_objects":7971,"num_object_clones":0,"num_object_copies":23913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9513,"num_write_kb":559488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":54,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753123+0000","last_change":"2024-02-12T17:02:48.617976+0000","last_active":"2024-02-12T17:03:00.753123+0000","last_peered":"2024-02-12T17:03:00.753123+0000","last_clean":"2024-02-12T17:03:00.753123+0000","last_became_active":"2024-02-12T17:01:16.910938+0000","last_became_peered":"2024-02-12T17:01:16.910938+0000","last_unstale":"2024-02-12T17:03:00.753123+0000","last_undegraded":"2024-02-12T17:03:00.753123+0000","last_fullsized":"2024-02-12T17:03:00.753123+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:48.617913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:02:48.617913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":306,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755011+0000","last_change":"2024-02-12T17:00:56.213043+0000","last_active":"2024-02-12T17:03:00.755011+0000","last_peered":"2024-02-12T17:03:00.755011+0000","last_clean":"2024-02-12T17:03:00.755011+0000","last_became_active":"2024-02-12T16:49:39.101391+0000","last_became_peered":"2024-02-12T16:49:39.101391+0000","last_unstale":"2024-02-12T17:03:00.755011+0000","last_undegraded":"2024-02-12T17:03:00.755011+0000","last_fullsized":"2024-02-12T17:03:00.755011+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.212949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.212949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":357,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746426+0000","last_change":"2024-02-12T17:02:44.688499+0000","last_active":"2024-02-12T17:03:00.746426+0000","last_peered":"2024-02-12T17:03:00.746426+0000","last_clean":"2024-02-12T17:03:00.746426+0000","last_became_active":"2024-02-12T16:49:41.669092+0000","last_became_peered":"2024-02-12T16:49:41.669092+0000","last_unstale":"2024-02-12T17:03:00.746426+0000","last_undegraded":"2024-02-12T17:03:00.746426+0000","last_fullsized":"2024-02-12T17:03:00.746426+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:02:44.688402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:44.688402+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":91,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746054+0000","last_change":"2024-02-12T16:59:09.271420+0000","last_active":"2024-02-12T17:03:00.746054+0000","last_peered":"2024-02-12T17:03:00.746054+0000","last_clean":"2024-02-12T17:03:00.746054+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:03:00.746054+0000","last_undegraded":"2024-02-12T17:03:00.746054+0000","last_fullsized":"2024-02-12T17:03:00.746054+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.283580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.283580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"337'9163","reported_seq":9205,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.540909+0000","last_change":"2024-02-12T17:01:13.886816+0000","last_active":"2024-02-12T17:03:08.540909+0000","last_peered":"2024-02-12T17:03:08.540909+0000","last_clean":"2024-02-12T17:03:08.540909+0000","last_became_active":"2024-02-12T17:01:13.886430+0000","last_became_peered":"2024-02-12T17:01:13.886430+0000","last_unstale":"2024-02-12T17:03:08.540909+0000","last_undegraded":"2024-02-12T17:03:08.540909+0000","last_fullsized":"2024-02-12T17:03:08.540909+0000","mapping_epoch":304,"log_start":"327'5600","ondisk_log_start":"327'5600","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3563,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":500301848,"num_objects":7635,"num_object_clones":0,"num_object_copies":22905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9163,"num_write_kb":537473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":181,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754975+0000","last_change":"2024-02-12T17:01:16.911456+0000","last_active":"2024-02-12T17:03:00.754975+0000","last_peered":"2024-02-12T17:03:00.754975+0000","last_clean":"2024-02-12T17:03:00.754975+0000","last_became_active":"2024-02-12T17:01:16.911041+0000","last_became_peered":"2024-02-12T17:01:16.911041+0000","last_unstale":"2024-02-12T17:03:00.754975+0000","last_undegraded":"2024-02-12T17:03:00.754975+0000","last_fullsized":"2024-02-12T17:03:00.754975+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":104,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.742122+0000","last_change":"2024-02-12T17:02:18.084576+0000","last_active":"2024-02-12T17:03:00.742122+0000","last_peered":"2024-02-12T17:03:00.742122+0000","last_clean":"2024-02-12T17:03:00.742122+0000","last_became_active":"2024-02-12T16:59:09.242505+0000","last_became_peered":"2024-02-12T16:59:09.242505+0000","last_unstale":"2024-02-12T17:03:00.742122+0000","last_undegraded":"2024-02-12T17:03:00.742122+0000","last_fullsized":"2024-02-12T17:03:00.742122+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":328,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.749456+0000","last_change":"2024-02-12T17:02:18.703238+0000","last_active":"2024-02-12T17:03:00.749456+0000","last_peered":"2024-02-12T17:03:00.749456+0000","last_clean":"2024-02-12T17:03:00.749456+0000","last_became_active":"2024-02-12T16:57:47.199763+0000","last_became_peered":"2024-02-12T16:57:47.199763+0000","last_unstale":"2024-02-12T17:03:00.749456+0000","last_undegraded":"2024-02-12T17:03:00.749456+0000","last_fullsized":"2024-02-12T17:03:00.749456+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":319,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.740835+0000","last_change":"2024-02-12T17:02:30.025410+0000","last_active":"2024-02-12T17:03:00.740835+0000","last_peered":"2024-02-12T17:03:00.740835+0000","last_clean":"2024-02-12T17:03:00.740835+0000","last_became_active":"2024-02-12T16:54:40.628772+0000","last_became_peered":"2024-02-12T16:54:40.628772+0000","last_unstale":"2024-02-12T17:03:00.740835+0000","last_undegraded":"2024-02-12T17:03:00.740835+0000","last_fullsized":"2024-02-12T17:03:00.740835+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"337'9514","reported_seq":9560,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.526038+0000","last_change":"2024-02-12T17:02:24.719511+0000","last_active":"2024-02-12T17:03:08.526038+0000","last_peered":"2024-02-12T17:03:08.526038+0000","last_clean":"2024-02-12T17:03:08.526038+0000","last_became_active":"2024-02-12T17:01:13.881372+0000","last_became_peered":"2024-02-12T17:01:13.881372+0000","last_unstale":"2024-02-12T17:03:08.526038+0000","last_undegraded":"2024-02-12T17:03:08.526038+0000","last_fullsized":"2024-02-12T17:03:08.526038+0000","mapping_epoch":304,"log_start":"326'5400","ondisk_log_start":"326'5400","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'5831","last_scrub_stamp":"2024-02-12T17:02:24.719414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.719414+0000","log_size":4114,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":521928704,"num_objects":7964,"num_object_clones":0,"num_object_copies":23892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9514,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":309,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754735+0000","last_change":"2024-02-12T16:59:09.340317+0000","last_active":"2024-02-12T17:03:00.754735+0000","last_peered":"2024-02-12T17:03:00.754735+0000","last_clean":"2024-02-12T17:03:00.754735+0000","last_became_active":"2024-02-12T16:59:09.246616+0000","last_became_peered":"2024-02-12T16:59:09.246616+0000","last_unstale":"2024-02-12T17:03:00.754735+0000","last_undegraded":"2024-02-12T17:03:00.754735+0000","last_fullsized":"2024-02-12T17:03:00.754735+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T16:57:19.955311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:19.955311+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":96,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755395+0000","last_change":"2024-02-12T17:02:40.562525+0000","last_active":"2024-02-12T17:03:00.755395+0000","last_peered":"2024-02-12T17:03:00.755395+0000","last_clean":"2024-02-12T17:03:00.755395+0000","last_became_active":"2024-02-12T16:59:09.268028+0000","last_became_peered":"2024-02-12T16:59:09.268028+0000","last_unstale":"2024-02-12T17:03:00.755395+0000","last_undegraded":"2024-02-12T17:03:00.755395+0000","last_fullsized":"2024-02-12T17:03:00.755395+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.562425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.562425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":340,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.750380+0000","last_change":"2024-02-12T16:59:09.996418+0000","last_active":"2024-02-12T17:03:00.750380+0000","last_peered":"2024-02-12T17:03:00.750380+0000","last_clean":"2024-02-12T17:03:00.750380+0000","last_became_active":"2024-02-12T16:59:09.256058+0000","last_became_peered":"2024-02-12T16:59:09.256058+0000","last_unstale":"2024-02-12T17:03:00.750380+0000","last_undegraded":"2024-02-12T17:03:00.750380+0000","last_fullsized":"2024-02-12T17:03:00.750380+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"328'11","reported_seq":240,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.409817+0000","last_change":"2024-02-12T17:02:29.064318+0000","last_active":"2024-02-12T17:03:07.409817+0000","last_peered":"2024-02-12T17:03:07.409817+0000","last_clean":"2024-02-12T17:03:07.409817+0000","last_became_active":"2024-02-12T16:57:47.167628+0000","last_became_peered":"2024-02-12T16:57:47.167628+0000","last_unstale":"2024-02-12T17:03:07.409817+0000","last_undegraded":"2024-02-12T17:03:07.409817+0000","last_fullsized":"2024-02-12T17:03:07.409817+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"337'9168","reported_seq":9209,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.530359+0000","last_change":"2024-02-12T17:01:13.886956+0000","last_active":"2024-02-12T17:03:08.530359+0000","last_peered":"2024-02-12T17:03:08.530359+0000","last_clean":"2024-02-12T17:03:08.530359+0000","last_became_active":"2024-02-12T17:01:13.886648+0000","last_became_peered":"2024-02-12T17:01:13.886648+0000","last_unstale":"2024-02-12T17:03:08.530359+0000","last_undegraded":"2024-02-12T17:03:08.530359+0000","last_fullsized":"2024-02-12T17:03:08.530359+0000","mapping_epoch":304,"log_start":"327'5600","ondisk_log_start":"327'5600","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3568,"ondisk_log_size":3568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":508035072,"num_objects":7752,"num_object_clones":0,"num_object_copies":23256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9168,"num_write_kb":541440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":99,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.750383+0000","last_change":"2024-02-12T17:01:30.811252+0000","last_active":"2024-02-12T17:03:00.750383+0000","last_peered":"2024-02-12T17:03:00.750383+0000","last_clean":"2024-02-12T17:03:00.750383+0000","last_became_active":"2024-02-12T16:59:09.259654+0000","last_became_peered":"2024-02-12T16:59:09.259654+0000","last_unstale":"2024-02-12T17:03:00.750383+0000","last_undegraded":"2024-02-12T17:03:00.750383+0000","last_fullsized":"2024-02-12T17:03:00.750383+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.811153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.811153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"261'14","reported_seq":411,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.410015+0000","last_change":"2024-02-12T16:59:12.225158+0000","last_active":"2024-02-12T17:03:07.410015+0000","last_peered":"2024-02-12T17:03:07.410015+0000","last_clean":"2024-02-12T17:03:07.410015+0000","last_became_active":"2024-02-12T16:59:12.224830+0000","last_became_peered":"2024-02-12T16:59:12.224830+0000","last_unstale":"2024-02-12T17:03:07.410015+0000","last_undegraded":"2024-02-12T17:03:07.410015+0000","last_fullsized":"2024-02-12T17:03:07.410015+0000","mapping_epoch":260,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":192,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745384+0000","last_change":"2024-02-12T17:01:16.913111+0000","last_active":"2024-02-12T17:03:00.745384+0000","last_peered":"2024-02-12T17:03:00.745384+0000","last_clean":"2024-02-12T17:03:00.745384+0000","last_became_active":"2024-02-12T17:01:16.911732+0000","last_became_peered":"2024-02-12T17:01:16.911732+0000","last_unstale":"2024-02-12T17:03:00.745384+0000","last_undegraded":"2024-02-12T17:03:00.745384+0000","last_fullsized":"2024-02-12T17:03:00.745384+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":304,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755642+0000","last_change":"2024-02-12T16:59:09.788881+0000","last_active":"2024-02-12T17:03:00.755642+0000","last_peered":"2024-02-12T17:03:00.755642+0000","last_clean":"2024-02-12T17:03:00.755642+0000","last_became_active":"2024-02-12T16:59:09.277264+0000","last_became_peered":"2024-02-12T16:59:09.277264+0000","last_unstale":"2024-02-12T17:03:00.755642+0000","last_undegraded":"2024-02-12T17:03:00.755642+0000","last_fullsized":"2024-02-12T17:03:00.755642+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"337'9267","reported_seq":9308,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.510786+0000","last_change":"2024-02-12T17:01:16.911200+0000","last_active":"2024-02-12T17:03:08.510786+0000","last_peered":"2024-02-12T17:03:08.510786+0000","last_clean":"2024-02-12T17:03:08.510786+0000","last_became_active":"2024-02-12T17:01:16.910877+0000","last_became_peered":"2024-02-12T17:01:16.910877+0000","last_unstale":"2024-02-12T17:03:08.510786+0000","last_undegraded":"2024-02-12T17:03:08.510786+0000","last_fullsized":"2024-02-12T17:03:08.510786+0000","mapping_epoch":307,"log_start":"327'5700","ondisk_log_start":"327'5700","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3567,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":510066688,"num_objects":7783,"num_object_clones":0,"num_object_copies":23349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":545600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":311,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.756247+0000","last_change":"2024-02-12T17:02:41.672588+0000","last_active":"2024-02-12T17:03:00.756247+0000","last_peered":"2024-02-12T17:03:00.756247+0000","last_clean":"2024-02-12T17:03:00.756247+0000","last_became_active":"2024-02-12T16:57:47.177329+0000","last_became_peered":"2024-02-12T16:57:47.177329+0000","last_unstale":"2024-02-12T17:03:00.756247+0000","last_undegraded":"2024-02-12T17:03:00.756247+0000","last_fullsized":"2024-02-12T17:03:00.756247+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"208'12","reported_seq":461,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.410788+0000","last_change":"2024-02-12T17:02:20.866677+0000","last_active":"2024-02-12T17:03:07.410788+0000","last_peered":"2024-02-12T17:03:07.410788+0000","last_clean":"2024-02-12T17:03:07.410788+0000","last_became_active":"2024-02-12T16:57:46.177517+0000","last_became_peered":"2024-02-12T16:57:46.177517+0000","last_unstale":"2024-02-12T17:03:07.410788+0000","last_undegraded":"2024-02-12T17:03:07.410788+0000","last_fullsized":"2024-02-12T17:03:07.410788+0000","mapping_epoch":97,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-12T17:02:20.866591+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:02:20.866591+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":94,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753075+0000","last_change":"2024-02-12T17:01:30.863612+0000","last_active":"2024-02-12T17:03:00.753075+0000","last_peered":"2024-02-12T17:03:00.753075+0000","last_clean":"2024-02-12T17:03:00.753075+0000","last_became_active":"2024-02-12T16:59:09.267755+0000","last_became_peered":"2024-02-12T16:59:09.267755+0000","last_unstale":"2024-02-12T17:03:00.753075+0000","last_undegraded":"2024-02-12T17:03:00.753075+0000","last_fullsized":"2024-02-12T17:03:00.753075+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.863530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.863530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":314,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755294+0000","last_change":"2024-02-12T17:01:41.694955+0000","last_active":"2024-02-12T17:03:00.755294+0000","last_peered":"2024-02-12T17:03:00.755294+0000","last_clean":"2024-02-12T17:03:00.755294+0000","last_became_active":"2024-02-12T16:57:47.199008+0000","last_became_peered":"2024-02-12T16:57:47.199008+0000","last_unstale":"2024-02-12T17:03:00.755294+0000","last_undegraded":"2024-02-12T17:03:00.755294+0000","last_fullsized":"2024-02-12T17:03:00.755294+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:01:41.694875+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:01:41.694875+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"337'9322","reported_seq":9363,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.530044+0000","last_change":"2024-02-12T17:01:13.884499+0000","last_active":"2024-02-12T17:03:08.530044+0000","last_peered":"2024-02-12T17:03:08.530044+0000","last_clean":"2024-02-12T17:03:08.530044+0000","last_became_active":"2024-02-12T17:01:13.884154+0000","last_became_peered":"2024-02-12T17:01:13.884154+0000","last_unstale":"2024-02-12T17:03:08.530044+0000","last_undegraded":"2024-02-12T17:03:08.530044+0000","last_fullsized":"2024-02-12T17:03:08.530044+0000","mapping_epoch":304,"log_start":"327'5700","ondisk_log_start":"327'5700","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3622,"ondisk_log_size":3622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512753664,"num_objects":7824,"num_object_clones":0,"num_object_copies":23472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9322,"num_write_kb":548672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":47,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755767+0000","last_change":"2024-02-12T17:01:16.909471+0000","last_active":"2024-02-12T17:03:00.755767+0000","last_peered":"2024-02-12T17:03:00.755767+0000","last_clean":"2024-02-12T17:03:00.755767+0000","last_became_active":"2024-02-12T17:01:16.909039+0000","last_became_peered":"2024-02-12T17:01:16.909039+0000","last_unstale":"2024-02-12T17:03:00.755767+0000","last_undegraded":"2024-02-12T17:03:00.755767+0000","last_fullsized":"2024-02-12T17:03:00.755767+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"308'11","reported_seq":88,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.410435+0000","last_change":"2024-02-12T17:02:31.547056+0000","last_active":"2024-02-12T17:03:07.410435+0000","last_peered":"2024-02-12T17:03:07.410435+0000","last_clean":"2024-02-12T17:03:07.410435+0000","last_became_active":"2024-02-12T17:01:16.912683+0000","last_became_peered":"2024-02-12T17:01:16.912683+0000","last_unstale":"2024-02-12T17:03:07.410435+0000","last_undegraded":"2024-02-12T17:03:07.410435+0000","last_fullsized":"2024-02-12T17:03:07.410435+0000","mapping_epoch":307,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'11","last_scrub_stamp":"2024-02-12T17:02:31.546965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:31.546965+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":250,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755149+0000","last_change":"2024-02-12T17:01:29.889009+0000","last_active":"2024-02-12T17:03:00.755149+0000","last_peered":"2024-02-12T17:03:00.755149+0000","last_clean":"2024-02-12T17:03:00.755149+0000","last_became_active":"2024-02-12T16:57:47.191231+0000","last_became_peered":"2024-02-12T16:57:47.191231+0000","last_unstale":"2024-02-12T17:03:00.755149+0000","last_undegraded":"2024-02-12T17:03:00.755149+0000","last_fullsized":"2024-02-12T17:03:00.755149+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.888942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.888942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":249,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753665+0000","last_change":"2024-02-12T17:02:16.750469+0000","last_active":"2024-02-12T17:03:00.753665+0000","last_peered":"2024-02-12T17:03:00.753665+0000","last_clean":"2024-02-12T17:03:00.753665+0000","last_became_active":"2024-02-12T16:59:09.268136+0000","last_became_peered":"2024-02-12T16:59:09.268136+0000","last_unstale":"2024-02-12T17:03:00.753665+0000","last_undegraded":"2024-02-12T17:03:00.753665+0000","last_fullsized":"2024-02-12T17:03:00.753665+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"337'9347","reported_seq":9387,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.617548+0000","last_change":"2024-02-12T17:01:16.913292+0000","last_active":"2024-02-12T17:03:08.617548+0000","last_peered":"2024-02-12T17:03:08.617548+0000","last_clean":"2024-02-12T17:03:08.617548+0000","last_became_active":"2024-02-12T17:01:16.912605+0000","last_became_peered":"2024-02-12T17:01:16.912605+0000","last_unstale":"2024-02-12T17:03:08.617548+0000","last_undegraded":"2024-02-12T17:03:08.617548+0000","last_fullsized":"2024-02-12T17:03:08.617548+0000","mapping_epoch":307,"log_start":"327'5800","ondisk_log_start":"327'5800","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3547,"ondisk_log_size":3547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":510984192,"num_objects":7797,"num_object_clones":0,"num_object_copies":23391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":548608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":273,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.757750+0000","last_change":"2024-02-12T17:01:16.909740+0000","last_active":"2024-02-12T17:03:00.757750+0000","last_peered":"2024-02-12T17:03:00.757750+0000","last_clean":"2024-02-12T17:03:00.757750+0000","last_became_active":"2024-02-12T17:01:16.909123+0000","last_became_peered":"2024-02-12T17:01:16.909123+0000","last_unstale":"2024-02-12T17:03:00.757750+0000","last_undegraded":"2024-02-12T17:03:00.757750+0000","last_fullsized":"2024-02-12T17:03:00.757750+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":173,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.742772+0000","last_change":"2024-02-12T17:01:29.153285+0000","last_active":"2024-02-12T17:03:00.742772+0000","last_peered":"2024-02-12T17:03:00.742772+0000","last_clean":"2024-02-12T17:03:00.742772+0000","last_became_active":"2024-02-12T16:59:09.247447+0000","last_became_peered":"2024-02-12T16:59:09.247447+0000","last_unstale":"2024-02-12T17:03:00.742772+0000","last_undegraded":"2024-02-12T17:03:00.742772+0000","last_fullsized":"2024-02-12T17:03:00.742772+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.153217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.153217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":306,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.754550+0000","last_change":"2024-02-12T17:01:27.859045+0000","last_active":"2024-02-12T17:03:00.754550+0000","last_peered":"2024-02-12T17:03:00.754550+0000","last_clean":"2024-02-12T17:03:00.754550+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T17:03:00.754550+0000","last_undegraded":"2024-02-12T17:03:00.754550+0000","last_fullsized":"2024-02-12T17:03:00.754550+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.858955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.858955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":388,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.745734+0000","last_change":"2024-02-12T17:01:35.160233+0000","last_active":"2024-02-12T17:03:00.745734+0000","last_peered":"2024-02-12T17:03:00.745734+0000","last_clean":"2024-02-12T17:03:00.745734+0000","last_became_active":"2024-02-12T17:01:16.911545+0000","last_became_peered":"2024-02-12T17:01:16.911545+0000","last_unstale":"2024-02-12T17:03:00.745734+0000","last_undegraded":"2024-02-12T17:03:00.745734+0000","last_fullsized":"2024-02-12T17:03:00.745734+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:01:35.160145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:35.160145+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"337'9183","reported_seq":9224,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.520749+0000","last_change":"2024-02-12T17:01:13.886968+0000","last_active":"2024-02-12T17:03:08.520749+0000","last_peered":"2024-02-12T17:03:08.520749+0000","last_clean":"2024-02-12T17:03:08.520749+0000","last_became_active":"2024-02-12T17:01:13.886370+0000","last_became_peered":"2024-02-12T17:01:13.886370+0000","last_unstale":"2024-02-12T17:03:08.520749+0000","last_undegraded":"2024-02-12T17:03:08.520749+0000","last_fullsized":"2024-02-12T17:03:08.520749+0000","mapping_epoch":304,"log_start":"327'5700","ondisk_log_start":"327'5700","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3483,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":502202368,"num_objects":7663,"num_object_clones":0,"num_object_copies":22989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":313,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.749136+0000","last_change":"2024-02-12T16:59:09.242082+0000","last_active":"2024-02-12T17:03:00.749136+0000","last_peered":"2024-02-12T17:03:00.749136+0000","last_clean":"2024-02-12T17:03:00.749136+0000","last_became_active":"2024-02-12T16:59:09.241648+0000","last_became_peered":"2024-02-12T16:59:09.241648+0000","last_unstale":"2024-02-12T17:03:00.749136+0000","last_undegraded":"2024-02-12T17:03:00.749136+0000","last_fullsized":"2024-02-12T17:03:00.749136+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":236,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746317+0000","last_change":"2024-02-12T17:01:16.911775+0000","last_active":"2024-02-12T17:03:00.746317+0000","last_peered":"2024-02-12T17:03:00.746317+0000","last_clean":"2024-02-12T17:03:00.746317+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:03:00.746317+0000","last_undegraded":"2024-02-12T17:03:00.746317+0000","last_fullsized":"2024-02-12T17:03:00.746317+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":41,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755381+0000","last_change":"2024-02-12T17:01:16.909537+0000","last_active":"2024-02-12T17:03:00.755381+0000","last_peered":"2024-02-12T17:03:00.755381+0000","last_clean":"2024-02-12T17:03:00.755381+0000","last_became_active":"2024-02-12T17:01:16.909207+0000","last_became_peered":"2024-02-12T17:01:16.909207+0000","last_unstale":"2024-02-12T17:03:00.755381+0000","last_undegraded":"2024-02-12T17:03:00.755381+0000","last_fullsized":"2024-02-12T17:03:00.755381+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.794228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":183,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755297+0000","last_change":"2024-02-12T17:02:45.594799+0000","last_active":"2024-02-12T17:03:00.755297+0000","last_peered":"2024-02-12T17:03:00.755297+0000","last_clean":"2024-02-12T17:03:00.755297+0000","last_became_active":"2024-02-12T16:59:09.273985+0000","last_became_peered":"2024-02-12T16:59:09.273985+0000","last_unstale":"2024-02-12T17:03:00.755297+0000","last_undegraded":"2024-02-12T17:03:00.755297+0000","last_fullsized":"2024-02-12T17:03:00.755297+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:02:45.594719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:45.594719+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"337'9395","reported_seq":9436,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.538344+0000","last_change":"2024-02-12T17:01:13.890011+0000","last_active":"2024-02-12T17:03:08.538344+0000","last_peered":"2024-02-12T17:03:08.538344+0000","last_clean":"2024-02-12T17:03:08.538344+0000","last_became_active":"2024-02-12T17:01:13.889691+0000","last_became_peered":"2024-02-12T17:01:13.889691+0000","last_unstale":"2024-02-12T17:03:08.538344+0000","last_undegraded":"2024-02-12T17:03:08.538344+0000","last_fullsized":"2024-02-12T17:03:08.538344+0000","mapping_epoch":304,"log_start":"327'5900","ondisk_log_start":"327'5900","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3495,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":521601024,"num_objects":7959,"num_object_clones":0,"num_object_copies":23877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9395,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":295,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.747074+0000","last_change":"2024-02-12T17:00:56.986244+0000","last_active":"2024-02-12T17:03:00.747074+0000","last_peered":"2024-02-12T17:03:00.747074+0000","last_clean":"2024-02-12T17:03:00.747074+0000","last_became_active":"2024-02-12T16:49:45.222600+0000","last_became_peered":"2024-02-12T16:49:45.222600+0000","last_unstale":"2024-02-12T17:03:00.747074+0000","last_undegraded":"2024-02-12T17:03:00.747074+0000","last_fullsized":"2024-02-12T17:03:00.747074+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.986137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.986137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":249,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.741758+0000","last_change":"2024-02-12T17:01:39.076533+0000","last_active":"2024-02-12T17:03:00.741758+0000","last_peered":"2024-02-12T17:03:00.741758+0000","last_clean":"2024-02-12T17:03:00.741758+0000","last_became_active":"2024-02-12T17:01:16.910805+0000","last_became_peered":"2024-02-12T17:01:16.910805+0000","last_unstale":"2024-02-12T17:03:00.741758+0000","last_undegraded":"2024-02-12T17:03:00.741758+0000","last_fullsized":"2024-02-12T17:03:00.741758+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.076442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.076442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":216,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.753741+0000","last_change":"2024-02-12T17:01:16.912916+0000","last_active":"2024-02-12T17:03:00.753741+0000","last_peered":"2024-02-12T17:03:00.753741+0000","last_clean":"2024-02-12T17:03:00.753741+0000","last_became_active":"2024-02-12T17:01:16.912576+0000","last_became_peered":"2024-02-12T17:01:16.912576+0000","last_unstale":"2024-02-12T17:03:00.753741+0000","last_undegraded":"2024-02-12T17:03:00.753741+0000","last_fullsized":"2024-02-12T17:03:00.753741+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:00:50.756578+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.756578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":348,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.746593+0000","last_change":"2024-02-12T17:01:28.096786+0000","last_active":"2024-02-12T17:03:00.746593+0000","last_peered":"2024-02-12T17:03:00.746593+0000","last_clean":"2024-02-12T17:03:00.746593+0000","last_became_active":"2024-02-12T16:57:47.189672+0000","last_became_peered":"2024-02-12T16:57:47.189672+0000","last_unstale":"2024-02-12T17:03:00.746593+0000","last_undegraded":"2024-02-12T17:03:00.746593+0000","last_fullsized":"2024-02-12T17:03:00.746593+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:28.096708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.096708+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"337'9511","reported_seq":9552,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-12T17:03:08.630591+0000","last_change":"2024-02-12T17:01:13.886545+0000","last_active":"2024-02-12T17:03:08.630591+0000","last_peered":"2024-02-12T17:03:08.630591+0000","last_clean":"2024-02-12T17:03:08.630591+0000","last_became_active":"2024-02-12T17:01:13.886238+0000","last_became_peered":"2024-02-12T17:01:13.886238+0000","last_unstale":"2024-02-12T17:03:08.630591+0000","last_undegraded":"2024-02-12T17:03:08.630591+0000","last_fullsized":"2024-02-12T17:03:08.630591+0000","mapping_epoch":304,"log_start":"327'6000","ondisk_log_start":"327'6000","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3511,"ondisk_log_size":3511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":522125312,"num_objects":7967,"num_object_clones":0,"num_object_copies":23901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9511,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":306,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.751110+0000","last_change":"2024-02-12T17:01:26.804003+0000","last_active":"2024-02-12T17:03:00.751110+0000","last_peered":"2024-02-12T17:03:00.751110+0000","last_clean":"2024-02-12T17:03:00.751110+0000","last_became_active":"2024-02-12T16:59:09.261034+0000","last_became_peered":"2024-02-12T16:59:09.261034+0000","last_unstale":"2024-02-12T17:03:00.751110+0000","last_undegraded":"2024-02-12T17:03:00.751110+0000","last_fullsized":"2024-02-12T17:03:00.751110+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:26.803903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.803903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":279,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.755860+0000","last_change":"2024-02-12T17:01:16.911268+0000","last_active":"2024-02-12T17:03:00.755860+0000","last_peered":"2024-02-12T17:03:00.755860+0000","last_clean":"2024-02-12T17:03:00.755860+0000","last_became_active":"2024-02-12T17:01:16.910854+0000","last_became_peered":"2024-02-12T17:01:16.910854+0000","last_unstale":"2024-02-12T17:03:00.755860+0000","last_undegraded":"2024-02-12T17:03:00.755860+0000","last_fullsized":"2024-02-12T17:03:00.755860+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:01.789464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:01.789464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":381,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:00.750230+0000","last_change":"2024-02-12T17:02:43.665354+0000","last_active":"2024-02-12T17:03:00.750230+0000","last_peered":"2024-02-12T17:03:00.750230+0000","last_clean":"2024-02-12T17:03:00.750230+0000","last_became_active":"2024-02-12T16:59:09.280858+0000","last_became_peered":"2024-02-12T16:59:09.280858+0000","last_unstale":"2024-02-12T17:03:00.750230+0000","last_undegraded":"2024-02-12T17:03:00.750230+0000","last_fullsized":"2024-02-12T17:03:00.750230+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:02:43.665248+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.665248+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"261'13","reported_seq":513,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-12T17:03:07.410909+0000","last_change":"2024-02-12T17:00:49.304845+0000","last_active":"2024-02-12T17:03:07.410909+0000","last_peered":"2024-02-12T17:03:07.410909+0000","last_clean":"2024-02-12T17:03:07.410909+0000","last_became_active":"2024-02-12T16:59:12.242710+0000","last_became_peered":"2024-02-12T16:59:12.242710+0000","last_unstale":"2024-02-12T17:03:07.410909+0000","last_undegraded":"2024-02-12T17:03:07.410909+0000","last_fullsized":"2024-02-12T17:03:07.410909+0000","mapping_epoch":260,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-02-12T17:00:49.304784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.304784+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8222605336,"num_objects":125468,"num_object_clones":0,"num_object_copies":376404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149708,"num_write_kb":8805569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24668090368,"data_stored":24668078152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57008,"ondisk_log_size":57008,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1231,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1103,"num_read_kb":860,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":5408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":724,"ondisk_log_size":724,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234149,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6142308,"kb_used_data":5552604,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":87602844,"statfs":{"total":95995035648,"available":89705312256,"internally_reserved":0,"allocated":5685866496,"data_stored":5685399409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.8360000000000001,"15min":1.415},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.9939999999999998,"5min":16.759,"15min":36.439},"last":2.8639999999999999},{"interface":"front","average":{"1min":1.9770000000000001,"5min":2.7090000000000001,"15min":1.659},"min":{"1min":0.28299999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.7430000000000003,"5min":17.844000000000001,"15min":36.490000000000002},"last":0.23100000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.839,"15min":1.194},"min":{"1min":0.23400000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.7200000000000002,"5min":11.048999999999999,"15min":12.032},"last":6.3479999999999999},{"interface":"front","average":{"1min":2.1349999999999998,"5min":2.2269999999999999,"15min":1.3120000000000001},"min":{"1min":0.41399999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.9720000000000004,"5min":17.324999999999999,"15min":17.324999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.464,"15min":1.0700000000000001},"min":{"1min":0.31900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.8150000000000004,"5min":9.7319999999999993,"15min":9.7319999999999993},"last":2.8959999999999999},{"interface":"front","average":{"1min":1.5589999999999999,"5min":2.0529999999999999,"15min":1.3340000000000001},"min":{"1min":0.29999999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.6899999999999999,"5min":11.071999999999999,"15min":20.338000000000001},"last":6.3120000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.6200000000000001,"15min":1.3580000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":4.2039999999999997,"5min":9.7490000000000006,"15min":20.433},"last":2.915},{"interface":"front","average":{"1min":1.77,"5min":2.149,"15min":1.5529999999999999},"min":{"1min":0.374,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.8620000000000001,"5min":11.092000000000001,"15min":20.393000000000001},"last":6.375}]},{"osd":4,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.782,"5min":2.0169999999999999,"15min":1.2829999999999999},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.219},"max":{"1min":5.1040000000000001,"5min":9.7089999999999996,"15min":50.406999999999996},"last":2.9329999999999998},{"interface":"front","average":{"1min":1.6419999999999999,"5min":2.806,"15min":1.577},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23400000000000001},"max":{"1min":5.8029999999999999,"5min":14.863,"15min":50.168999999999997},"last":6.4729999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4969999999999999,"5min":2.5960000000000001,"15min":1.258},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.24199999999999999},"max":{"1min":9.8439999999999994,"5min":10.91,"15min":10.91},"last":5.8559999999999999},{"interface":"front","average":{"1min":2.1259999999999999,"5min":3.0659999999999998,"15min":1.454},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.26200000000000001},"max":{"1min":6.0590000000000002,"5min":18.651,"15min":18.651},"last":6.8869999999999996}]},{"osd":6,"last update":"Mon Feb 12 17:02:53 2024","interfaces":[{"interface":"back","average":{"1min":2.5139999999999998,"5min":2.645,"15min":1.2969999999999999},"min":{"1min":0.37,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":5.9240000000000004,"5min":17.234999999999999,"15min":17.234999999999999},"last":0.379},{"interface":"front","average":{"1min":1.6859999999999999,"5min":2.1309999999999998,"15min":1.1850000000000001},"min":{"1min":0.435,"5min":0.33700000000000002,"15min":0.30199999999999999},"max":{"1min":5.0599999999999996,"5min":12.407999999999999,"15min":12.407999999999999},"last":2.9529999999999998}]}]},{"osd":6,"up_from":0,"seq":193273528490,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738567,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6168024,"kb_used_data":5569936,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":87577128,"statfs":{"total":95995035648,"available":89678979072,"internally_reserved":0,"allocated":5703614464,"data_stored":5703158760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.008,"15min":1.1970000000000001},"min":{"1min":0.23400000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.2149999999999999,"5min":16.146999999999998,"15min":23.510999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.85899999999999999,"15min":1.1240000000000001},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":3.1200000000000001,"5min":5.0449999999999999,"15min":10.186999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.94599999999999995,"15min":1.3759999999999999},"min":{"1min":0.20999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.496,"5min":11.122999999999999,"15min":16.521000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.361,"5min":1,"15min":1.2529999999999999},"min":{"1min":0.253,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.5999999999999996,"5min":4.984,"15min":10.340999999999999},"last":0.29599999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.296,"15min":1.4410000000000001},"min":{"1min":0.28199999999999997,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":5.7380000000000004,"5min":16.27,"15min":21.100000000000001},"last":2.4689999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.2529999999999999,"15min":1.4490000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":8.0690000000000008,"5min":11.976000000000001,"15min":16.446000000000002},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:02:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.792,"15min":1.3740000000000001},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.524,"5min":16.888999999999999,"15min":16.888999999999999},"last":6.9379999999999997},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.9239999999999999,"15min":1.341},"min":{"1min":0.318,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":7.2850000000000001,"5min":16.899999999999999,"15min":16.899999999999999},"last":9.1020000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:02:10 2024","interfaces":[{"interface":"back","average":{"1min":3.9470000000000001,"5min":2.8730000000000002,"15min":1.776},"min":{"1min":0.309,"5min":0.30199999999999999,"15min":0.26400000000000001},"max":{"1min":50.404000000000003,"5min":50.404000000000003,"15min":50.404000000000003},"last":5.4539999999999997},{"interface":"front","average":{"1min":3.6579999999999999,"5min":2.653,"15min":1.6839999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":49.979999999999997,"5min":49.979999999999997,"15min":49.979999999999997},"last":2.6339999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:02:39 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.9119999999999999,"15min":1.419},"min":{"1min":0.378,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.4489999999999998,"5min":17.907,"15min":17.907},"last":0.624},{"interface":"front","average":{"1min":1.9990000000000001,"5min":2.274,"15min":1.6459999999999999},"min":{"1min":0.35099999999999998,"5min":0.33600000000000002,"15min":0.27900000000000003},"max":{"1min":5.883,"5min":16.294,"15min":21.062000000000001},"last":1.7230000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.881,"15min":1.496},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.32400000000000001},"max":{"1min":3.4409999999999998,"5min":12.058,"15min":16.175000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":2.1280000000000001,"5min":2.5339999999999998,"15min":1.613},"min":{"1min":0.45100000000000001,"5min":0.39900000000000002,"15min":0.33600000000000002},"max":{"1min":7.0960000000000001,"5min":34.722999999999999,"15min":34.722999999999999},"last":1.956}]}]},{"osd":2,"up_from":18,"seq":77309411516,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4620416,"kb_used_data":4022520,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":89124736,"statfs":{"total":95995035648,"available":91263729664,"internally_reserved":0,"allocated":4119060480,"data_stored":4118635891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:03:06 2024","interfaces":[{"interface":"back","average":{"1min":2.7949999999999999,"5min":1.2529999999999999,"15min":1.5329999999999999},"min":{"1min":0.19400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":36.100999999999999,"5min":36.100999999999999,"15min":36.100999999999999},"last":2.1619999999999999},{"interface":"front","average":{"1min":3.5059999999999998,"5min":1.6359999999999999,"15min":1.46},"min":{"1min":0.23599999999999999,"5min":0.193,"15min":0.158},"max":{"1min":36.299999999999997,"5min":36.299999999999997,"15min":36.299999999999997},"last":0.88600000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:03:06 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.98999999999999999,"15min":1.4730000000000001},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":2.556,"5min":6.5819999999999999,"15min":17.739000000000001},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.0700000000000001,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":4.1699999999999999,"5min":4.3099999999999996,"15min":15.766999999999999},"last":2.0350000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:02:30 2024","interfaces":[{"interface":"back","average":{"1min":1.7589999999999999,"5min":1.1919999999999999,"15min":1.6659999999999999},"min":{"1min":0.35199999999999998,"5min":0.27000000000000002,"15min":0.23499999999999999},"max":{"1min":5.1239999999999997,"5min":5.1239999999999997,"15min":51.579999999999998},"last":2.101},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.1579999999999999,"15min":1.399},"min":{"1min":0.311,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":3.5270000000000001,"5min":5.5800000000000001,"15min":48.988999999999997},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Feb 12 17:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":2.0270000000000001,"15min":1.569},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":4.1280000000000001,"5min":18.658999999999999,"15min":18.658999999999999},"last":2.1299999999999999},{"interface":"front","average":{"1min":2.2280000000000002,"5min":2.3380000000000001,"15min":1.5629999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":6.5860000000000003,"5min":17.771000000000001,"15min":17.771000000000001},"last":1.8120000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:03:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.6679999999999999,"15min":1.589},"min":{"1min":0.33000000000000002,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":3.359,"5min":10.407999999999999,"15min":15.699},"last":2.069},{"interface":"front","average":{"1min":1.0760000000000001,"5min":2.0489999999999999,"15min":1.72},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.27700000000000002},"max":{"1min":2.9350000000000001,"5min":13.207000000000001,"15min":13.207000000000001},"last":1.419}]},{"osd":6,"last update":"Mon Feb 12 17:02:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":2.3359999999999999,"15min":1.778},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.1909999999999998,"5min":16.739000000000001,"15min":17.782},"last":1.444},{"interface":"front","average":{"1min":1.6739999999999999,"5min":2.1339999999999999,"15min":1.5509999999999999},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.29799999999999999},"max":{"1min":4.6390000000000002,"5min":8.9459999999999997,"15min":13.446999999999999},"last":1.47}]},{"osd":7,"last update":"Mon Feb 12 17:03:06 2024","interfaces":[{"interface":"back","average":{"1min":2.1360000000000001,"5min":2.0529999999999999,"15min":1.6619999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.32600000000000001},"max":{"1min":6.1299999999999999,"5min":8.8520000000000003,"15min":13.473000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.863,"5min":2.1509999999999998,"15min":1.748},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":6.4340000000000002,"5min":16.768000000000001,"15min":16.768000000000001},"last":2.4409999999999998}]}]},{"osd":3,"up_from":0,"seq":111669149880,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953652,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3619700,"kb_used_data":3034028,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":90125452,"statfs":{"total":95995035648,"available":92288462848,"internally_reserved":0,"allocated":3106844672,"data_stored":3106425089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:02:41 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":2.1579999999999999,"15min":1.4610000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":7.0519999999999996,"5min":9.5090000000000003,"15min":38.179000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.762,"5min":2.6000000000000001,"15min":1.6100000000000001},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":5.1180000000000003,"5min":20.244,"15min":38.151000000000003},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:02:41 2024","interfaces":[{"interface":"back","average":{"1min":2.0590000000000002,"5min":2.3170000000000002,"15min":1.401},"min":{"1min":0.254,"5min":0.251,"15min":0.218},"max":{"1min":5.7839999999999998,"5min":11.077999999999999,"15min":11.077999999999999},"last":2.226},{"interface":"front","average":{"1min":1.9079999999999999,"5min":2.2200000000000002,"15min":1.3759999999999999},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.1520000000000001,"5min":9.5579999999999998,"15min":9.5579999999999998},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:02:41 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":2.1539999999999999,"15min":1.3740000000000001},"min":{"1min":0.25900000000000001,"5min":0.252,"15min":0.22500000000000001},"max":{"1min":6.1799999999999997,"5min":9.5790000000000006,"15min":9.6349999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.911,"5min":2.4670000000000001,"15min":1.5089999999999999},"min":{"1min":0.29799999999999999,"5min":0.24399999999999999,"15min":0.224},"max":{"1min":5.1529999999999996,"5min":12.246,"15min":12.246},"last":0.56499999999999995}]},{"osd":3,"last update":"Mon Feb 12 17:02:41 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":2.6339999999999999,"15min":1.8360000000000001},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.26700000000000002},"max":{"1min":5.181,"5min":20.271000000000001,"15min":38.170999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.607,"5min":2.4329999999999998,"15min":1.641},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.8209999999999997,"5min":10.865,"15min":38.311999999999998},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Feb 12 17:02:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":4.609,"15min":1.921},"min":{"1min":0.29099999999999998,"5min":0.253,"15min":0.24099999999999999},"max":{"1min":6.5679999999999996,"5min":79.573999999999998,"15min":79.573999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":4.8579999999999997,"15min":1.968},"min":{"1min":0.33300000000000002,"5min":0.27900000000000003,"15min":0.24099999999999999},"max":{"1min":8.3450000000000006,"5min":80.497,"15min":80.497},"last":0.434}]},{"osd":6,"last update":"Mon Feb 12 17:02:39 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":3.1080000000000001,"15min":1.7849999999999999},"min":{"1min":0.32300000000000001,"5min":0.28699999999999998,"15min":0.255},"max":{"1min":5.399,"5min":27.678000000000001,"15min":104.907},"last":0.44800000000000001},{"interface":"front","average":{"1min":2.2599999999999998,"5min":3.343,"15min":1.925},"min":{"1min":0.373,"5min":0.30499999999999999,"15min":0.26500000000000001},"max":{"1min":6.5069999999999997,"5min":15.593,"15min":105.395},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:02:52 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.8130000000000002,"15min":1.369},"min":{"1min":0.35599999999999998,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":7.633,"5min":12.311999999999999,"15min":12.311999999999999},"last":2.3460000000000001},{"interface":"front","average":{"1min":2.5019999999999998,"5min":3.4359999999999999,"15min":1.6100000000000001},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.28799999999999998},"max":{"1min":6.0019999999999998,"5min":27.773,"15min":27.773},"last":1.3859999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757424,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6658076,"kb_used_data":6060180,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":87087076,"statfs":{"total":95995035648,"available":89177165824,"internally_reserved":0,"allocated":6205624320,"data_stored":6205164902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":2.5569999999999999,"15min":1.528},"min":{"1min":0.22800000000000001,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":5.4379999999999997,"5min":46.365000000000002,"15min":46.365000000000002},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":3.0329999999999999,"15min":1.77},"min":{"1min":0.23999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.4020000000000001,"5min":46.134,"15min":46.134},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.9119999999999999,"15min":1.365},"min":{"1min":0.28999999999999998,"5min":0.26700000000000002,"15min":0.23200000000000001},"max":{"1min":4.0940000000000003,"5min":7.3869999999999996,"15min":16.655000000000001},"last":2.5310000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":2.2389999999999999,"15min":1.5349999999999999},"min":{"1min":0.29599999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.3660000000000001,"5min":13.130000000000001,"15min":16.712},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.4620000000000002,"15min":1.6339999999999999},"min":{"1min":0.26600000000000001,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":5.4630000000000001,"5min":11.15,"15min":18.218},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.4319999999999999,"5min":2.0699999999999998,"15min":1.502},"min":{"1min":0.34200000000000003,"5min":0.254,"15min":0.254},"max":{"1min":4.2030000000000003,"5min":9.1199999999999992,"15min":18.327000000000002},"last":2.5590000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":2.2810000000000001,"15min":1.544},"min":{"1min":0.314,"5min":0.314,"15min":0.28000000000000003},"max":{"1min":5.5030000000000001,"5min":11.122,"15min":18.251000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.458,"5min":2.028,"15min":1.423},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.30399999999999999},"max":{"1min":4.2279999999999998,"5min":7.4109999999999996,"15min":20.263000000000002},"last":2.5910000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":2.7999999999999998,"15min":1.278},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":9.5879999999999992,"5min":11.880000000000001,"15min":11.880000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":3.4399999999999999,"15min":1.5029999999999999},"min":{"1min":0.28100000000000003,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":5.4139999999999997,"5min":20.550000000000001,"15min":20.550000000000001},"last":0.96599999999999997}]},{"osd":6,"last update":"Mon Feb 12 17:02:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":2.698,"15min":1.357},"min":{"1min":0.40999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.7629999999999999,"5min":12.281000000000001,"15min":12.281000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.8069999999999999,"5min":3.1709999999999998,"15min":1.462},"min":{"1min":0.37,"5min":0.28100000000000003,"15min":0.26700000000000002},"max":{"1min":6.6710000000000003,"5min":22.035,"15min":22.035},"last":0.99099999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:02:05 2024","interfaces":[{"interface":"back","average":{"1min":2.0289999999999999,"5min":2.8330000000000002,"15min":1.3600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27500000000000002},"max":{"1min":9.9049999999999994,"5min":19.184999999999999,"15min":19.184999999999999},"last":2.6240000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":2.6749999999999998,"15min":1.3049999999999999},"min":{"1min":0.30499999999999999,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":4.8330000000000002,"5min":12.253,"15min":12.253},"last":2.742}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5661917184,"data_stored":5661917184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4115529728,"data_stored":4115529728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3090747392,"data_stored":3090743320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6156193792,"data_stored":6156189720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5643702272,"data_stored":5643698200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:03:14.595 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T17:03:14.595 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:03:14.595 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.2548466439590671 2024-02-12T17:03:14.595 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 0 0.2548466439590671 2024-02-12T17:03:14.781 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:03:14.823 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:13.661786+0000 mgr.y (mgr.24481) 517 : audit [DBG] from='client.25420 v1:172.21.15.33:0/701782526' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:03:14.823 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:13.661786+0000 mgr.y (mgr.24481) 517 : audit [DBG] from='client.25420 v1:172.21.15.33:0/701782526' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:03:14.848 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:14.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:14.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:13.661786+0000 mgr.y (mgr.24481) 517 : audit [DBG] from='client.25420 v1:172.21.15.33:0/701782526' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:03:14.959 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:15.011 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:15.253 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:15.274 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:15.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:15.294 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:15.379 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:15.571 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:15.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:15.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:15.709 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:03:15.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:14.190456+0000 mgr.y (mgr.24481) 518 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:15.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:14.614857+0000 mgr.y (mgr.24481) 519 : cluster [DBG] pgmap v668: 138 pgs: 138 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-02-12T17:03:15.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:15.504076+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.33:0/1288266490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:15.728 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:15.767 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:15.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:15.882 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:15.923 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:15.981 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:14.190456+0000 mgr.y (mgr.24481) 518 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:15.981 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:14.614857+0000 mgr.y (mgr.24481) 519 : cluster [DBG] pgmap v668: 138 pgs: 138 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-02-12T17:03:15.981 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:15.504076+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.33:0/1288266490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:14.190456+0000 mgr.y (mgr.24481) 518 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:14.614857+0000 mgr.y (mgr.24481) 519 : cluster [DBG] pgmap v668: 138 pgs: 138 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-02-12T17:03:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:15.504076+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.33:0/1288266490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:16.053 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:16.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:16.157 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:16.193 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:16.307 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:03:16.307 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:03:16.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.307 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:159830 9.6 9:68a713b1:::benchmark_data_smithi033_131576_object22240:head [delete] snapc 0=[] ondisk+write+known_if_redirected e339)", 2024-02-12T17:03:16.307 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:16.280587+0000", 2024-02-12T17:03:16.307 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.024559773, 2024-02-12T17:03:16.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024601562, 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 159830 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280587+0000", 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:16.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280587+0000", 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280586+0000", 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4599999999999999e-06 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280590+0000", 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9790000000000001e-06 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280595+0000", 2024-02-12T17:03:16.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3889999999999998e-06 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280599+0000", 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016381300000000001 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280763+0000", 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000118862 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280882+0000", 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000127303 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.281009+0000", 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010894951999999999 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.291904+0000", 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011317097999999999 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:159835 9.f 9:f5ce3a54:::benchmark_data_smithi033_131576_object22245:head [delete] snapc 0=[] ondisk+write+known_if_redirected e339)", 2024-02-12T17:03:16.311 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:16.282980+0000", 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.022167026999999999, 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022274318000000001, 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 159835 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.282980+0000", 2024-02-12T17:03:16.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.282980+0000", 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.282978+0000", 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8359999999999997e-06 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.282984+0000", 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2110000000000001e-06 2024-02-12T17:03:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.282991+0000", 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2920000000000003e-06 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.282996+0000", 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6108999999999998e-05 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283083+0000", 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111338 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283194+0000", 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011598 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283310+0000", 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0085558790000000006 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.291866+0000", 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002858055 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.294724+0000", 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011743951000000001 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:159836 9.7 9:e3bbb173:::benchmark_data_smithi033_131576_object22246:head [delete] snapc 0=[] ondisk+write+known_if_redirected e339)", 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:16.283007+0000", 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.022140244999999999, 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022300614, 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:03:16.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 159836 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283007+0000", 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283007+0000", 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:16.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283006+0000", 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0860000000000002e-06 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283009+0000", 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3029999999999999e-06 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283014+0000", 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9550000000000002e-06 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283018+0000", 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020829220000000001 2024-02-12T17:03:16.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.285101+0000", 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000102771 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.285204+0000", 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011896300000000001 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.285323+0000", 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0089402019999999995 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.294263+0000", 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000312489 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.294576+0000", 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01156891 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:159840 9.6 9:66573884:::benchmark_data_smithi033_131576_object22250:head [delete] snapc 0=[] ondisk+write+known_if_redirected e339)", 2024-02-12T17:03:16.320 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:16.283022+0000", 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.022124751000000002, 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022337586999999999, 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 159840 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283022+0000", 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283022+0000", 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283022+0000", 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.407e-06 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283024+0000", 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4530000000000004e-06 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:16.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283029+0000", 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8010000000000001e-06 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283032+0000", 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013767200000000001 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283170+0000", 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010037100000000001 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283270+0000", 2024-02-12T17:03:16.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113198 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.283384+0000", 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0085277430000000008 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.291911+0000", 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088891890000000005 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.324 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:159829 9.a 9:57ea2cb6:::benchmark_data_smithi033_131576_object22239:head [delete] snapc 0=[] ondisk+write+known_if_redirected e339)", 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:16.280558+0000", 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.024589303, 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.02485004, 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 159829 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280558+0000", 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:03:16.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280558+0000", 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280555+0000", 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7810000000000005e-06 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280563+0000", 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0010000000000001e-05 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280573+0000", 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5969999999999999e-06 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280578+0000", 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00029135499999999998 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280870+0000", 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6193000000000005e-05 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.280966+0000", 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012091600000000001 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.281087+0000", 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010760903 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.291848+0000", 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025652549999999998 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:03:16.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:16.294413+0000", 2024-02-12T17:03:16.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013855513 2024-02-12T17:03:16.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.329 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:03:16.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:03:16.329 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T17:03:16.329 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 5 2024-02-12T17:03:16.329 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:03:16.551 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:16.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:16.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:16.724 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:16.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:15.691215+0000 mon.a (mon.0) 1338 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:16.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:15.702059+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? v1:172.21.15.33:0/1288266490' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:03:16.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:15.702120+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e339: 8 total, 8 up, 5 in 2024-02-12T17:03:16.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:15.786547+0000 osd.7 (osd.7) 173 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:03:16.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:15.807410+0000 osd.0 (osd.0) 167 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:03:16.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:15.810394+0000 osd.0 (osd.0) 168 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:03:17.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:15.691215+0000 mon.a (mon.0) 1338 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:17.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:15.702059+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? v1:172.21.15.33:0/1288266490' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:03:17.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:15.702120+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e339: 8 total, 8 up, 5 in 2024-02-12T17:03:17.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:15.786547+0000 osd.7 (osd.7) 173 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:03:17.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:15.807410+0000 osd.0 (osd.0) 167 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:03:17.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:15.810394+0000 osd.0 (osd.0) 168 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:03:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:15.691215+0000 mon.a (mon.0) 1338 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:15.702059+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? v1:172.21.15.33:0/1288266490' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:03:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:15.702120+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e339: 8 total, 8 up, 5 in 2024-02-12T17:03:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:15.786547+0000 osd.7 (osd.7) 173 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:03:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:15.807410+0000 osd.0 (osd.0) 167 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:03:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:15.810394+0000 osd.0 (osd.0) 168 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:03:17.048 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T17:03:17.055 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:17.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:17.160 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:17.199 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:17.513 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:17.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:15.796122+0000 osd.7 (osd.7) 174 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:16.615248+0000 mgr.y (mgr.24481) 520 : cluster [DBG] pgmap v670: 138 pgs: 138 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:16.778438+0000 osd.0 (osd.0) 169 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:16.782767+0000 osd.0 (osd.0) 170 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:17.118183+0000 osd.4 (osd.4) 156 : cluster [DBG] 5.a scrub starts 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:17.122180+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.a scrub ok 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:17.209114+0000 mon.c (mon.2) 190 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:17.680424+0000 mon.c (mon.2) 191 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:17.681201+0000 mon.c (mon.2) 192 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:03:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:17.693636+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:18.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:15.796122+0000 osd.7 (osd.7) 174 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:03:18.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:16.615248+0000 mgr.y (mgr.24481) 520 : cluster [DBG] pgmap v670: 138 pgs: 138 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-02-12T17:03:18.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:16.778438+0000 osd.0 (osd.0) 169 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:03:18.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:16.782767+0000 osd.0 (osd.0) 170 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:03:18.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:17.118183+0000 osd.4 (osd.4) 156 : cluster [DBG] 5.a scrub starts 2024-02-12T17:03:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:17.122180+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.a scrub ok 2024-02-12T17:03:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:17.209114+0000 mon.c (mon.2) 190 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:03:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:17.680424+0000 mon.c (mon.2) 191 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:03:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:17.681201+0000 mon.c (mon.2) 192 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:03:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:17.693636+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:18.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:15.796122+0000 osd.7 (osd.7) 174 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:03:18.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:16.615248+0000 mgr.y (mgr.24481) 520 : cluster [DBG] pgmap v670: 138 pgs: 138 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-02-12T17:03:18.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:16.778438+0000 osd.0 (osd.0) 169 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:03:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:16.782767+0000 osd.0 (osd.0) 170 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:03:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:17.118183+0000 osd.4 (osd.4) 156 : cluster [DBG] 5.a scrub starts 2024-02-12T17:03:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:17.122180+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.a scrub ok 2024-02-12T17:03:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:17.209114+0000 mon.c (mon.2) 190 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:03:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:17.680424+0000 mon.c (mon.2) 191 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:03:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:17.681201+0000 mon.c (mon.2) 192 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:03:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:17.693636+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:18.068 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:18.096 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:18.947 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:18.951 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:17.741896+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:03:18.951 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:17.746914+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:03:18.951 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:17.779226+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:03:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:18.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:18.953 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:17.741896+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:03:18.953 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:17.746914+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:03:18.953 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:17.779226+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:03:18.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:17.741896+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:03:18.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:17.746914+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:03:18.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:17.779226+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:03:19.053 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:19.264 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:03:19.264 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:03:19.290 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:03:19.329 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:03:19.358 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:03:19.452 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:03:19.478 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:03:19.509 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:03:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:03:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:03:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:03:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:03:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:03:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:03:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:03:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:03:19.763 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 0.254847 (413d) 2024-02-12T17:03:19.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:17.811648+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:03:19.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:18.615845+0000 mgr.y (mgr.24481) 521 : cluster [DBG] pgmap v671: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 548 B/s rd, 0 B/s wr, 964 op/s 2024-02-12T17:03:19.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:19.077294+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.c scrub starts 2024-02-12T17:03:19.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:19.084542+0000 osd.4 (osd.4) 159 : cluster [DBG] 5.c scrub ok 2024-02-12T17:03:19.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:19.190257+0000 mon.a (mon.0) 1342 : audit [DBG] from='client.? v1:172.21.15.33:0/1329573288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:19.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:19.552953+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? v1:172.21.15.33:0/1461401586' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 0.2548466439590671}]: dispatch 2024-02-12T17:03:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:17.811648+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:03:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:18.615845+0000 mgr.y (mgr.24481) 521 : cluster [DBG] pgmap v671: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 548 B/s rd, 0 B/s wr, 964 op/s 2024-02-12T17:03:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:19.077294+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.c scrub starts 2024-02-12T17:03:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:19.084542+0000 osd.4 (osd.4) 159 : cluster [DBG] 5.c scrub ok 2024-02-12T17:03:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:19.190257+0000 mon.a (mon.0) 1342 : audit [DBG] from='client.? v1:172.21.15.33:0/1329573288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:19.789 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:19.552953+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? v1:172.21.15.33:0/1461401586' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 0.2548466439590671}]: dispatch 2024-02-12T17:03:19.860 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:03:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:17.811648+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:03:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:18.615845+0000 mgr.y (mgr.24481) 521 : cluster [DBG] pgmap v671: 138 pgs: 138 active+clean; 7.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 548 B/s rd, 0 B/s wr, 964 op/s 2024-02-12T17:03:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:19.077294+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.c scrub starts 2024-02-12T17:03:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:19.084542+0000 osd.4 (osd.4) 159 : cluster [DBG] 5.c scrub ok 2024-02-12T17:03:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:19.190257+0000 mon.a (mon.0) 1342 : audit [DBG] from='client.? v1:172.21.15.33:0/1329573288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:20.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:19.552953+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? v1:172.21.15.33:0/1461401586' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 0.2548466439590671}]: dispatch 2024-02-12T17:03:20.269 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:20.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:20.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:20.373 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:20.402 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:20.424 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:20.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:20.529 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:20.562 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:20.763 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:19.680218+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:03:20.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:19.689426+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:03:20.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:19.740264+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.33:0/1461401586' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.2548466439590671}]': finished 2024-02-12T17:03:20.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:19.740330+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e340: 8 total, 8 up, 5 in 2024-02-12T17:03:20.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:20.742311+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e341: 8 total, 8 up, 5 in 2024-02-12T17:03:20.979 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:19.680218+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:03:20.979 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:19.689426+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:03:20.979 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:19.740264+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.33:0/1461401586' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.2548466439590671}]': finished 2024-02-12T17:03:20.979 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:19.740330+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e340: 8 total, 8 up, 5 in 2024-02-12T17:03:20.979 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:20.742311+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e341: 8 total, 8 up, 5 in 2024-02-12T17:03:21.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:20] "GET /metrics HTTP/1.1" 200 176061 "" "Prometheus/2.33.4" 2024-02-12T17:03:21.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:19.680218+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:03:21.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:19.689426+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:03:21.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:19.740264+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.33:0/1461401586' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.2548466439590671}]': finished 2024-02-12T17:03:21.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:19.740330+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e340: 8 total, 8 up, 5 in 2024-02-12T17:03:21.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:20.742311+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e341: 8 total, 8 up, 5 in 2024-02-12T17:03:21.068 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:21.315 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:21.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:21.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:21.574 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:21.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:21.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:21.713 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:21.912 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:21.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:20.616199+0000 mgr.y (mgr.24481) 522 : cluster [DBG] pgmap v673: 138 pgs: 138 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-12T17:03:21.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:21.742828+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e342: 8 total, 8 up, 5 in 2024-02-12T17:03:21.914 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:20.616199+0000 mgr.y (mgr.24481) 522 : cluster [DBG] pgmap v673: 138 pgs: 138 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-12T17:03:21.914 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:21.742828+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e342: 8 total, 8 up, 5 in 2024-02-12T17:03:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:21.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:21.940 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:03:21.941 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:03:21.972 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:20.616199+0000 mgr.y (mgr.24481) 522 : cluster [DBG] pgmap v673: 138 pgs: 138 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-12T17:03:21.972 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:21.742828+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e342: 8 total, 8 up, 5 in 2024-02-12T17:03:22.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:22.047 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:22.433 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:03:22.611 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:03:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:03:22.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T17:03:22.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:21.745399+0000 osd.7 (osd.7) 177 : cluster [DBG] 9.9 starting backfill to osd.4 from (0'0,0'0] MAX to 339'10192 2024-02-12T17:03:22.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:21.746307+0000 osd.4 (osd.4) 160 : cluster [DBG] 9.c starting backfill to osd.5 from (0'0,0'0] MAX to 339'10413 2024-02-12T17:03:22.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:21.747605+0000 osd.5 (osd.5) 175 : cluster [DBG] 9.b starting backfill to osd.4 from (0'0,0'0] MAX to 339'10269 2024-02-12T17:03:22.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:21.749315+0000 osd.5 (osd.5) 176 : cluster [DBG] 9.6 starting backfill to osd.7 from (0'0,0'0] MAX to 339'10320 2024-02-12T17:03:22.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:21.767096+0000 osd.7 (osd.7) 178 : cluster [DBG] 9.e starting backfill to osd.5 from (0'0,0'0] MAX to 339'10393 2024-02-12T17:03:22.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:21.794515+0000 osd.7 (osd.7) 179 : cluster [DBG] 9.3 starting backfill to osd.2 from (0'0,0'0] MAX to 339'10053 2024-02-12T17:03:23.108 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:23.110 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:21.745399+0000 osd.7 (osd.7) 177 : cluster [DBG] 9.9 starting backfill to osd.4 from (0'0,0'0] MAX to 339'10192 2024-02-12T17:03:23.110 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:21.746307+0000 osd.4 (osd.4) 160 : cluster [DBG] 9.c starting backfill to osd.5 from (0'0,0'0] MAX to 339'10413 2024-02-12T17:03:23.110 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:21.747605+0000 osd.5 (osd.5) 175 : cluster [DBG] 9.b starting backfill to osd.4 from (0'0,0'0] MAX to 339'10269 2024-02-12T17:03:23.110 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:21.749315+0000 osd.5 (osd.5) 176 : cluster [DBG] 9.6 starting backfill to osd.7 from (0'0,0'0] MAX to 339'10320 2024-02-12T17:03:23.110 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:21.767096+0000 osd.7 (osd.7) 178 : cluster [DBG] 9.e starting backfill to osd.5 from (0'0,0'0] MAX to 339'10393 2024-02-12T17:03:23.110 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:21.794515+0000 osd.7 (osd.7) 179 : cluster [DBG] 9.3 starting backfill to osd.2 from (0'0,0'0] MAX to 339'10053 2024-02-12T17:03:23.110 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:21.745399+0000 osd.7 (osd.7) 177 : cluster [DBG] 9.9 starting backfill to osd.4 from (0'0,0'0] MAX to 339'10192 2024-02-12T17:03:23.110 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:21.746307+0000 osd.4 (osd.4) 160 : cluster [DBG] 9.c starting backfill to osd.5 from (0'0,0'0] MAX to 339'10413 2024-02-12T17:03:23.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:21.747605+0000 osd.5 (osd.5) 175 : cluster [DBG] 9.b starting backfill to osd.4 from (0'0,0'0] MAX to 339'10269 2024-02-12T17:03:23.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:21.749315+0000 osd.5 (osd.5) 176 : cluster [DBG] 9.6 starting backfill to osd.7 from (0'0,0'0] MAX to 339'10320 2024-02-12T17:03:23.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:21.767096+0000 osd.7 (osd.7) 178 : cluster [DBG] 9.e starting backfill to osd.5 from (0'0,0'0] MAX to 339'10393 2024-02-12T17:03:23.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:21.794515+0000 osd.7 (osd.7) 179 : cluster [DBG] 9.3 starting backfill to osd.2 from (0'0,0'0] MAX to 339'10053 2024-02-12T17:03:23.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:23.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:23.123 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:23.128 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:23.213 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:23.254 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:23.361 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:23.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:23.466 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:23.523 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:23.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:23.681 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:23.716 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:03:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:21.808786+0000 osd.7 (osd.7) 180 : cluster [DBG] 9.2 starting backfill to osd.5 from (0'0,0'0] MAX to 339'10183 2024-02-12T17:03:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:22.616603+0000 mgr.y (mgr.24481) 523 : cluster [DBG] pgmap v676: 138 pgs: 138 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-12T17:03:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:23.327425+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? v1:172.21.15.33:0/442381237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:21.808786+0000 osd.7 (osd.7) 180 : cluster [DBG] 9.2 starting backfill to osd.5 from (0'0,0'0] MAX to 339'10183 2024-02-12T17:03:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:22.616603+0000 mgr.y (mgr.24481) 523 : cluster [DBG] pgmap v676: 138 pgs: 138 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-12T17:03:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:23.327425+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? v1:172.21.15.33:0/442381237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:24.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:21.808786+0000 osd.7 (osd.7) 180 : cluster [DBG] 9.2 starting backfill to osd.5 from (0'0,0'0] MAX to 339'10183 2024-02-12T17:03:24.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:22.616603+0000 mgr.y (mgr.24481) 523 : cluster [DBG] pgmap v676: 138 pgs: 138 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-12T17:03:24.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:23.327425+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? v1:172.21.15.33:0/442381237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:24.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:03:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:03:24.938 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:23.786273+0000 mon.a (mon.0) 1349 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:03:24.938 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:23.786312+0000 mon.a (mon.0) 1350 : cluster [INF] Cluster is now healthy 2024-02-12T17:03:24.938 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:23.795054+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? v1:172.21.15.33:0/442381237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:24.938 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:23.795124+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e343: 8 total, 8 up, 5 in 2024-02-12T17:03:25.213 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:23.786273+0000 mon.a (mon.0) 1349 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:03:25.213 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:23.786312+0000 mon.a (mon.0) 1350 : cluster [INF] Cluster is now healthy 2024-02-12T17:03:25.213 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:23.795054+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? v1:172.21.15.33:0/442381237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:25.213 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:23.795124+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e343: 8 total, 8 up, 5 in 2024-02-12T17:03:25.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:23.786273+0000 mon.a (mon.0) 1349 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:03:25.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:23.786312+0000 mon.a (mon.0) 1350 : cluster [INF] Cluster is now healthy 2024-02-12T17:03:25.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:23.795054+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? v1:172.21.15.33:0/442381237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:25.214 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:23.795124+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e343: 8 total, 8 up, 5 in 2024-02-12T17:03:25.267 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:25.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:25.343 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:03:25.344 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:03:25.370 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:25.407 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:25.508 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:25.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:25.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:25.652 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:25.777 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:25.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:25.788 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:03:25.813 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:03:25.836 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:03:25.860 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:03:25.881 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:25.897 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:03:25.913 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:25.936 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:25.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:25.939 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7] out_osds: [3, 1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:03:25.939 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:03:25.939 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-12T17:03:25.939 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T17:03:25.951 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:03:25.983 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:03:26.025 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:03:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:03:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:03:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:03:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:03:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:03:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:03:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:03:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:03:26.039 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:26.051 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:24.200507+0000 mgr.y (mgr.24481) 524 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:26.051 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:24.616955+0000 mgr.y (mgr.24481) 525 : cluster [DBG] pgmap v678: 138 pgs: 2 active+remapped+backfilling, 126 active+clean, 3 active+recovering, 4 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+remapped; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 590 op/s; 3/329031 objects degraded (0.001%); 48166/329031 objects misplaced (14.639%); 302 KiB/s, 28 objects/s recovering 2024-02-12T17:03:26.051 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:24.798469+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check failed: Degraded data redundancy: 3/329031 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:03:26.051 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:25.764913+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? v1:172.21.15.33:0/2205909179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:26.051 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:24.200507+0000 mgr.y (mgr.24481) 524 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:26.051 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:24.616955+0000 mgr.y (mgr.24481) 525 : cluster [DBG] pgmap v678: 138 pgs: 2 active+remapped+backfilling, 126 active+clean, 3 active+recovering, 4 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+remapped; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 590 op/s; 3/329031 objects degraded (0.001%); 48166/329031 objects misplaced (14.639%); 302 KiB/s, 28 objects/s recovering 2024-02-12T17:03:26.051 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:24.798469+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check failed: Degraded data redundancy: 3/329031 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:03:26.051 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:25.764913+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? v1:172.21.15.33:0/2205909179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:26.068 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:24.200507+0000 mgr.y (mgr.24481) 524 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:26.068 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:24.616955+0000 mgr.y (mgr.24481) 525 : cluster [DBG] pgmap v678: 138 pgs: 2 active+remapped+backfilling, 126 active+clean, 3 active+recovering, 4 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+remapped; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 590 op/s; 3/329031 objects degraded (0.001%); 48166/329031 objects misplaced (14.639%); 302 KiB/s, 28 objects/s recovering 2024-02-12T17:03:26.068 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:24.798469+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check failed: Degraded data redundancy: 3/329031 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:03:26.068 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:25.764913+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? v1:172.21.15.33:0/2205909179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:26.103 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T17:03:26.148 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:26.173 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:26.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:26.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:26.362 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:26.390 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:26.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:26.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:26.551 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:26.580 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:26.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:26.659 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:26.684 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:26.713 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:26.865 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:26.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:26.970 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:26.999 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:27.039 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:27.076 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:27.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:27.190 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:27.220 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:27.237 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:03:27.246 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:27.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:27.353 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:27.520 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:26.617302+0000 mgr.y (mgr.24481) 526 : cluster [DBG] pgmap v679: 138 pgs: 2 active+remapped+backfilling, 126 active+clean, 3 active+recovering, 4 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+remapped; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 479 op/s; 3/329031 objects degraded (0.001%); 48166/329031 objects misplaced (14.639%); 246 KiB/s, 22 objects/s recovering 2024-02-12T17:03:27.521 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:26.617302+0000 mgr.y (mgr.24481) 526 : cluster [DBG] pgmap v679: 138 pgs: 2 active+remapped+backfilling, 126 active+clean, 3 active+recovering, 4 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+remapped; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 479 op/s; 3/329031 objects degraded (0.001%); 48166/329031 objects misplaced (14.639%); 246 KiB/s, 22 objects/s recovering 2024-02-12T17:03:27.540 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:27.546 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:26.617302+0000 mgr.y (mgr.24481) 526 : cluster [DBG] pgmap v679: 138 pgs: 2 active+remapped+backfilling, 126 active+clean, 3 active+recovering, 4 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+remapped; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 479 op/s; 3/329031 objects degraded (0.001%); 48166/329031 objects misplaced (14.639%); 246 KiB/s, 22 objects/s recovering 2024-02-12T17:03:27.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:27.647 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:27.678 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:27.806 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:27.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:27.911 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:28.302 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:28.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:28.408 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:28.442 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:28.544 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:28.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:28.563 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:28.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:28.757 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:03:28.757 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:03:28.758 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-02-12T17:03:28.758 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:03:28.772 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:28.888 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:03:28.922 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:28.668897+0000 mon.a (mon.0) 1354 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/329031 objects degraded (0.001%), 2 pgs degraded) 2024-02-12T17:03:28.922 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:28.668933+0000 mon.a (mon.0) 1355 : cluster [INF] Cluster is now healthy 2024-02-12T17:03:28.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:28.668897+0000 mon.a (mon.0) 1354 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/329031 objects degraded (0.001%), 2 pgs degraded) 2024-02-12T17:03:28.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:28.668933+0000 mon.a (mon.0) 1355 : cluster [INF] Cluster is now healthy 2024-02-12T17:03:28.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:28.668897+0000 mon.a (mon.0) 1354 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/329031 objects degraded (0.001%), 2 pgs degraded) 2024-02-12T17:03:28.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:28.668933+0000 mon.a (mon.0) 1355 : cluster [INF] Cluster is now healthy 2024-02-12T17:03:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:28.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:29.096 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:03:29.096 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:03:29.096 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 343, 2024-02-12T17:03:29.096 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T17:03:29.096 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2024-02-12T16:43:50.282468+0000", 2024-02-12T17:03:29.096 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2024-02-12T17:03:23.786003+0000", 2024-02-12T17:03:29.096 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2024-02-12T16:49:30.933575+0000", 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2024-02-12T16:59:08.202987+0000", 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5799936, 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 18, 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 9, 2024-02-12T17:03:29.097 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 8, 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "luminous", 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "pacific", 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "device_health_metrics", 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:47:33.056945+0000", 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:03:29.098 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:03:29.099 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "71", 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:03:29.100 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:03:29.101 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:03:29.102 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_devicehealth": {} 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".rgw.root", 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:37.647343+0000", 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:03:29.103 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:03:29.104 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "59", 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:03:29.105 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:03:29.106 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:03:29.107 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "iscsi", 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:38.076200+0000", 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:03:29.108 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 3, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 3, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 3, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 3, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 3, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:03:29.109 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "60", 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:03:29.110 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:03:29.111 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 4, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.log", 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:39.589565+0000", 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:03:29.113 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "61", 2024-02-12T17:03:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:03:29.115 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 5, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.control", 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:41.623856+0000", 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:03:29.117 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:03:29.118 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "5.16", 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 238, 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 237, 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 237, 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "238", 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "238", 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "236", 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:03:29.119 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:03:29.120 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:03:29.121 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 6, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.meta", 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:43.693806+0000", 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:03:29.122 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.123 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "65", 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:03:29.124 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:03:29.125 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_bias": 4 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.126 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 9, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_2", 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T17:01:12.591563+0000", 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2024-02-12T17:03:29.127 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "310", 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:03:29.128 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:03:29.129 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:03:29.130 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "rados": {} 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7", 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0.2548370361328125, 2024-02-12T17:03:29.131 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 8, 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 307, 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2024-02-12T17:03:29.133 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/2545411201", 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6802/2545411201", 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6804/2545411201", 2024-02-12T17:03:29.134 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6803/2545411201", 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24", 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 13, 2024-02-12T17:03:29.135 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 198, 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.136 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6805/1723231708", 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6806/1723231708", 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6808/1723231708", 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6807/1723231708", 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.138 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c", 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 18, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 304, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.139 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:03:29.140 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3126580659", 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6810/3126580659", 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6812/3126580659", 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6811/3126580659", 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9", 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:03:29.142 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 26, 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 97, 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:03:29.143 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.144 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.145 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6813/3941314957", 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6814/3941314957", 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6816/3941314957", 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6815/3941314957", 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0498129c-861c-4f7e-b8ba-219949da3130", 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:03:29.146 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.54083251953125, 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 32, 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 341, 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6800", 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.147 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6801", 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.148 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6803", 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6802", 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6800/544126779", 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6801/544126779", 2024-02-12T17:03:29.149 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6803/544126779", 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6802/544126779", 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50", 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:03:29.150 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 38, 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 341, 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6804", 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6805", 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6807", 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6806", 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6804/3229884005", 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6805/3229884005", 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6807/3229884005", 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6806/3229884005", 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:03:29.153 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821", 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 45, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 248, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:03:29.154 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6808", 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6809", 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:03:29.155 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6811", 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.156 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6810", 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6808/507305558", 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6809/507305558", 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6811/507305558", 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6810/507305558", 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:03:29.157 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66", 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 54, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 341, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6812", 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6813", 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6815", 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6814", 2024-02-12T17:03:29.160 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6812/3401977873", 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6813/3401977873", 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6815/3401977873", 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6814/3401977873", 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2024-02-12T17:03:29.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:02:54.945447+0000", 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:03:29.162 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:03:07.396374+0000", 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:02:59.318609+0000", 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.163 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:02:49.798384+0000", 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:03:29.164 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:02:35.670744+0000", 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:03:08.709906+0000", 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:03:29.165 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:02:33.537195+0000", 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:02:16.024274+0000", 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [ 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.8", 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: "mappings": [ 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: "from": 7, 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: "to": 2 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.9", 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: "mappings": [ 2024-02-12T17:03:29.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: "from": 7, 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: "to": 4 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [ 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "9.2", 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "9.3", 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "9.6", 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "9.9", 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "9.b", 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.170 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "9.c", 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "9.e", 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:03:29.171 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": { 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/2829464018": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/1835128904": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2277321750": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2116602769": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3967118778": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2298361247": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/218227980": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2070582473": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/4291575956": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3157562110": "2024-02-13T16:44:22.727241+0000" 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2024-02-12T17:03:29.172 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "ruleset-failure-domain": "osd", 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [], 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2024-02-12T17:03:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2024-02-12T17:03:29.174 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2024-02-12T17:03:29.174 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2024-02-12T17:03:29.174 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2024-02-12T17:03:29.174 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:03:29.174 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:03:29.174 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:29.213 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:29.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T17:03:29.736 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:28.617764+0000 mgr.y (mgr.24481) 527 : cluster [DBG] pgmap v680: 138 pgs: 6 active+remapped+backfilling, 130 active+clean, 1 active+recovering, 1 active+remapped+backfill_wait; 6.6 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 572 op/s; 46845/324141 objects misplaced (14.452%); 1.7 MiB/s, 43 objects/s recovering 2024-02-12T17:03:29.736 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:29.092850+0000 mon.a (mon.0) 1356 : audit [DBG] from='client.? v1:172.21.15.33:0/1874280076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:03:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:28.617764+0000 mgr.y (mgr.24481) 527 : cluster [DBG] pgmap v680: 138 pgs: 6 active+remapped+backfilling, 130 active+clean, 1 active+recovering, 1 active+remapped+backfill_wait; 6.6 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 572 op/s; 46845/324141 objects misplaced (14.452%); 1.7 MiB/s, 43 objects/s recovering 2024-02-12T17:03:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:29.092850+0000 mon.a (mon.0) 1356 : audit [DBG] from='client.? v1:172.21.15.33:0/1874280076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:03:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:28.617764+0000 mgr.y (mgr.24481) 527 : cluster [DBG] pgmap v680: 138 pgs: 6 active+remapped+backfilling, 130 active+clean, 1 active+recovering, 1 active+remapped+backfill_wait; 6.6 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 572 op/s; 46845/324141 objects misplaced (14.452%); 1.7 MiB/s, 43 objects/s recovering 2024-02-12T17:03:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:29.092850+0000 mon.a (mon.0) 1356 : audit [DBG] from='client.? v1:172.21.15.33:0/1874280076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:03:29.865 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:29.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:29.969 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:29.980 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:30.001 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:30.452 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:30.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:30.482 INFO:tasks.thrashosds.thrasher:j is {'epoch': 343, 'fsid': 'cf6ad642-c9c5-11ee-95b9-87774f69a715', 'created': '2024-02-12T16:43:50.282468+0000', 'modified': '2024-02-12T17:03:23.786003+0000', 'last_up_change': '2024-02-12T16:49:30.933575+0000', 'last_in_change': '2024-02-12T16:59:08.202987+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T16:47:33.056945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-12T16:49:37.647343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-12T16:49:38.076200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-12T16:49:39.589565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-12T16:49:41.623856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 238, 'last_epoch_started': 237, 'last_epoch_clean': 237, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '238', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '238', 'last_force_op_resend_preluminous': '236', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-12T16:49:43.693806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-12T17:01:12.591563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '310', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '5dd172eb-ee9d-4700-9b80-60f91c5f2cb7', 'up': 1, 'in': 1, 'weight': 0.2548370361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 2545411201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6802', 'nonce': 2545411201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6804', 'nonce': 2545411201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 2545411201}]}, 'public_addr': '172.21.15.33:6801/2545411201', 'cluster_addr': '172.21.15.33:6802/2545411201', 'heartbeat_back_addr': '172.21.15.33:6804/2545411201', 'heartbeat_front_addr': '172.21.15.33:6803/2545411201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71231bc4-1716-431d-bd88-f6a3cad0bd24', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1723231708}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6806', 'nonce': 1723231708}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6808', 'nonce': 1723231708}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1723231708}]}, 'public_addr': '172.21.15.33:6805/1723231708', 'cluster_addr': '172.21.15.33:6806/1723231708', 'heartbeat_back_addr': '172.21.15.33:6808/1723231708', 'heartbeat_front_addr': '172.21.15.33:6807/1723231708', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8a9c7127-d17c-4c44-b666-4211006de47c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 304, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3126580659}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6810', 'nonce': 3126580659}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6812', 'nonce': 3126580659}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3126580659}]}, 'public_addr': '172.21.15.33:6809/3126580659', 'cluster_addr': '172.21.15.33:6810/3126580659', 'heartbeat_back_addr': '172.21.15.33:6812/3126580659', 'heartbeat_front_addr': '172.21.15.33:6811/3126580659', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3cc40671-c8f7-412c-a0f1-987f48a016b9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3941314957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6814', 'nonce': 3941314957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6816', 'nonce': 3941314957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3941314957}]}, 'public_addr': '172.21.15.33:6813/3941314957', 'cluster_addr': '172.21.15.33:6814/3941314957', 'heartbeat_back_addr': '172.21.15.33:6816/3941314957', 'heartbeat_front_addr': '172.21.15.33:6815/3941314957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0498129c-861c-4f7e-b8ba-219949da3130', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.54083251953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6800', 'nonce': 544126779}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 544126779}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 544126779}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 544126779}]}, 'public_addr': '172.21.15.94:6800/544126779', 'cluster_addr': '172.21.15.94:6801/544126779', 'heartbeat_back_addr': '172.21.15.94:6803/544126779', 'heartbeat_front_addr': '172.21.15.94:6802/544126779', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '05718a1e-1164-4b54-a493-23c1cd3adb50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 3229884005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 3229884005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 3229884005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 3229884005}]}, 'public_addr': '172.21.15.94:6804/3229884005', 'cluster_addr': '172.21.15.94:6805/3229884005', 'heartbeat_back_addr': '172.21.15.94:6807/3229884005', 'heartbeat_front_addr': '172.21.15.94:6806/3229884005', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dc48640f-8f27-405c-a9a4-cf1e3e6c8821', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 507305558}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 507305558}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 507305558}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 507305558}]}, 'public_addr': '172.21.15.94:6808/507305558', 'cluster_addr': '172.21.15.94:6809/507305558', 'heartbeat_back_addr': '172.21.15.94:6811/507305558', 'heartbeat_front_addr': '172.21.15.94:6810/507305558', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7d55aa3f-06b6-4fc3-97fd-62b535f78b66', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3401977873}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 3401977873}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 3401977873}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 3401977873}]}, 'public_addr': '172.21.15.94:6812/3401977873', 'cluster_addr': '172.21.15.94:6813/3401977873', 'heartbeat_back_addr': '172.21.15.94:6815/3401977873', 'heartbeat_front_addr': '172.21.15.94:6814/3401977873', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:02:54.945447+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:03:07.396374+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:02:59.318609+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:02:49.798384+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:02:35.670744+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:03:08.709906+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:02:33.537195+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:02:16.024274+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.8', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.9', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [{'pgid': '9.2', 'osds': [7, 2, 0]}, {'pgid': '9.3', 'osds': [7, 5, 0]}, {'pgid': '9.6', 'osds': [5, 2, 0]}, {'pgid': '9.9', 'osds': [7, 5, 0]}, {'pgid': '9.b', 'osds': [5, 2, 0]}, {'pgid': '9.c', 'osds': [4, 7, 0]}, {'pgid': '9.e', 'osds': [7, 4, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.33:6800/2829464018': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/1835128904': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2277321750': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2116602769': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:6800/3967118778': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/2298361247': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/218227980': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/2070582473': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/4291575956': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:6800/3157562110': '2024-02-13T16:44:22.727241+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T17:03:30.482 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-02-12T17:03:30.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:30.680 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:31.038 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:31.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:31.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:31.225 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:03:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:30] "GET /metrics HTTP/1.1" 200 176061 "" "Prometheus/2.33.4" 2024-02-12T17:03:31.226 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:31.511 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:31.591 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:31.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:31.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:31.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:31.745 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:31.789 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:31.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:31.870 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:30.618284+0000 mgr.y (mgr.24481) 528 : cluster [DBG] pgmap v681: 138 pgs: 6 active+remapped+backfilling, 131 active+clean, 1 active+remapped+backfill_wait; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 576 B/s rd, 0 B/s wr, 735 op/s; 46072/318084 objects misplaced (14.484%); 3.8 MiB/s, 75 objects/s recovering 2024-02-12T17:03:31.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:31.985 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:32.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:30.618284+0000 mgr.y (mgr.24481) 528 : cluster [DBG] pgmap v681: 138 pgs: 6 active+remapped+backfilling, 131 active+clean, 1 active+remapped+backfill_wait; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 576 B/s rd, 0 B/s wr, 735 op/s; 46072/318084 objects misplaced (14.484%); 3.8 MiB/s, 75 objects/s recovering 2024-02-12T17:03:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:30.618284+0000 mgr.y (mgr.24481) 528 : cluster [DBG] pgmap v681: 138 pgs: 6 active+remapped+backfilling, 131 active+clean, 1 active+remapped+backfill_wait; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 576 B/s rd, 0 B/s wr, 735 op/s; 46072/318084 objects misplaced (14.484%); 3.8 MiB/s, 75 objects/s recovering 2024-02-12T17:03:32.097 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:32.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:32.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:32.237 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:32.297 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:03:32.297 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:03:32.297 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:03:32.297 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:03:32.297 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:03:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:32.688 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:03:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:03:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:32.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:32.762 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:03:32.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:32.828 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:32.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:31.779699+0000 mon.c (mon.2) 194 : audit [INF] from='client.? v1:172.21.15.33:0/294188758' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:03:32.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:31.780026+0000 mon.a (mon.0) 1357 : audit [INF] from='client.35552 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:03:32.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:32.498619+0000 mon.a (mon.0) 1358 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:33.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:31.779699+0000 mon.c (mon.2) 194 : audit [INF] from='client.? v1:172.21.15.33:0/294188758' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:03:33.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:31.780026+0000 mon.a (mon.0) 1357 : audit [INF] from='client.35552 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:03:33.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:32.498619+0000 mon.a (mon.0) 1358 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:31.779699+0000 mon.c (mon.2) 194 : audit [INF] from='client.? v1:172.21.15.33:0/294188758' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:03:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:31.780026+0000 mon.a (mon.0) 1357 : audit [INF] from='client.35552 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:03:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:32.498619+0000 mon.a (mon.0) 1358 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:33.064 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T17:03:33.514 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:33.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:33.564 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:33.620 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:33.655 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:33.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:32.618643+0000 mgr.y (mgr.24481) 529 : cluster [DBG] pgmap v682: 138 pgs: 6 active+remapped+backfilling, 131 active+clean, 1 active+remapped+backfill_wait; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 511 B/s rd, 0 B/s wr, 653 op/s; 46072/318084 objects misplaced (14.484%); 3.4 MiB/s, 67 objects/s recovering 2024-02-12T17:03:33.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:32.739252+0000 mon.a (mon.0) 1359 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:33.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:32.756597+0000 mon.a (mon.0) 1360 : audit [INF] from='client.35552 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:03:33.747 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:32.756669+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e344: 8 total, 8 up, 5 in 2024-02-12T17:03:34.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:32.618643+0000 mgr.y (mgr.24481) 529 : cluster [DBG] pgmap v682: 138 pgs: 6 active+remapped+backfilling, 131 active+clean, 1 active+remapped+backfill_wait; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 511 B/s rd, 0 B/s wr, 653 op/s; 46072/318084 objects misplaced (14.484%); 3.4 MiB/s, 67 objects/s recovering 2024-02-12T17:03:34.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:32.739252+0000 mon.a (mon.0) 1359 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:34.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:32.756597+0000 mon.a (mon.0) 1360 : audit [INF] from='client.35552 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:03:34.022 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:32.756669+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e344: 8 total, 8 up, 5 in 2024-02-12T17:03:34.108 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:34.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:34.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:32.618643+0000 mgr.y (mgr.24481) 529 : cluster [DBG] pgmap v682: 138 pgs: 6 active+remapped+backfilling, 131 active+clean, 1 active+remapped+backfill_wait; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 511 B/s rd, 0 B/s wr, 653 op/s; 46072/318084 objects misplaced (14.484%); 3.4 MiB/s, 67 objects/s recovering 2024-02-12T17:03:34.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:32.739252+0000 mon.a (mon.0) 1359 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:34.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:32.756597+0000 mon.a (mon.0) 1360 : audit [INF] from='client.35552 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:03:34.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:32.756669+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e344: 8 total, 8 up, 5 in 2024-02-12T17:03:34.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:34.305 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:34.390 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:34.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:34.397 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:03:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:03:34.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:34.558 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:34.631 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:03:34.651 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:03:34.674 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:03:34.695 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:03:34.742 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:03:34.756 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:34.610764+0000 mon.a (mon.0) 1362 : audit [DBG] from='client.? v1:172.21.15.33:0/141658699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:34.757 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:34.610764+0000 mon.a (mon.0) 1362 : audit [DBG] from='client.? v1:172.21.15.33:0/141658699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:34.769 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:03:34.812 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:03:34.843 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:34.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:34.865 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:03:34.866 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:03:34.866 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:03:34.866 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:03:34.867 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:03:34.867 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:03:34.867 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:03:34.867 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:03:34.867 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:03:34.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:35.016 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:35.163 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:35.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:35.177 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:34.610764+0000 mon.a (mon.0) 1362 : audit [DBG] from='client.? v1:172.21.15.33:0/141658699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:35.278 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:35.348 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:35.483 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7] out_osds: [3, 1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:03:35.483 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:03:35.483 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-02-12T17:03:35.483 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T17:03:35.483 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T17:03:35.483 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-02-12T17:03:35.487 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:35.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:03:35.598 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:35.631 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:35.709 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:35.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:35.767 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:34.212486+0000 mgr.y (mgr.24481) 530 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:35.767 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:34.619477+0000 mgr.y (mgr.24481) 531 : cluster [DBG] pgmap v684: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 44038/307785 objects misplaced (14.308%); 6.5 MiB/s, 105 objects/s recovering 2024-02-12T17:03:35.767 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:34.212486+0000 mgr.y (mgr.24481) 530 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:35.768 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:34.619477+0000 mgr.y (mgr.24481) 531 : cluster [DBG] pgmap v684: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 44038/307785 objects misplaced (14.308%); 6.5 MiB/s, 105 objects/s recovering 2024-02-12T17:03:35.818 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:35.827 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:35.853 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:36.045 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:03:36.111 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:36.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:36.219 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:34.212486+0000 mgr.y (mgr.24481) 530 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:34.619477+0000 mgr.y (mgr.24481) 531 : cluster [DBG] pgmap v684: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 44038/307785 objects misplaced (14.308%); 6.5 MiB/s, 105 objects/s recovering 2024-02-12T17:03:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:36.337 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:36.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:36.444 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:36.477 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:36.729 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:36.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:36.751 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:36.836 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:36.868 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:36.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T17:03:37.437 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:37.737 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:03:37.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:36.619858+0000 mgr.y (mgr.24481) 532 : cluster [DBG] pgmap v685: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 44038/307785 objects misplaced (14.308%); 6.5 MiB/s, 105 objects/s recovering 2024-02-12T17:03:37.767 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:36.619858+0000 mgr.y (mgr.24481) 532 : cluster [DBG] pgmap v685: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 44038/307785 objects misplaced (14.308%); 6.5 MiB/s, 105 objects/s recovering 2024-02-12T17:03:37.768 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:36.619858+0000 mgr.y (mgr.24481) 532 : cluster [DBG] pgmap v685: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 709 op/s; 44038/307785 objects misplaced (14.308%); 6.5 MiB/s, 105 objects/s recovering 2024-02-12T17:03:37.952 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:37.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:38.058 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:38.123 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:38.394 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:38.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:38.468 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:03:38.468 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T17:03:38.468 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:03:38.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:38.582 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:38.939 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:38.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:39.047 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:39.084 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:39.545 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:03:39.545 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:03:39.746 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:39.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:39.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:39.908 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:38.620478+0000 mgr.y (mgr.24481) 533 : cluster [DBG] pgmap v686: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 693 op/s; 42742/303372 objects misplaced (14.089%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T17:03:39.945 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:40.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:38.620478+0000 mgr.y (mgr.24481) 533 : cluster [DBG] pgmap v686: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 693 op/s; 42742/303372 objects misplaced (14.089%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T17:03:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:38.620478+0000 mgr.y (mgr.24481) 533 : cluster [DBG] pgmap v686: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 693 op/s; 42742/303372 objects misplaced (14.089%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T17:03:40.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T17:03:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:40.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:40.346 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:40.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:40.454 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:40.509 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:40.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:40.616 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:40.733 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:40.780 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:40.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:40.805 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:40.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:40.948 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:41.098 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:41.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:41.202 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:41.231 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:03:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:40] "GET /metrics HTTP/1.1" 200 176120 "" "Prometheus/2.33.4" 2024-02-12T17:03:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:41.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:41.362 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:41.396 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:41.594 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:41.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:41.699 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:41.725 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:40.621237+0000 mgr.y (mgr.24481) 534 : cluster [DBG] pgmap v687: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 680 op/s; 42043/297681 objects misplaced (14.124%); 6.7 MiB/s, 107 objects/s recovering 2024-02-12T17:03:41.725 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:41.013168+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.33:0/1393360019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:41.725 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:41.013464+0000 mon.a (mon.0) 1363 : audit [INF] from='client.35564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:41.734 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:03:41.753 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:42.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:40.621237+0000 mgr.y (mgr.24481) 534 : cluster [DBG] pgmap v687: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 680 op/s; 42043/297681 objects misplaced (14.124%); 6.7 MiB/s, 107 objects/s recovering 2024-02-12T17:03:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:41.013168+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.33:0/1393360019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:42.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:41.013464+0000 mon.a (mon.0) 1363 : audit [INF] from='client.35564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:40.621237+0000 mgr.y (mgr.24481) 534 : cluster [DBG] pgmap v687: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 680 op/s; 42043/297681 objects misplaced (14.124%); 6.7 MiB/s, 107 objects/s recovering 2024-02-12T17:03:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:41.013168+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.33:0/1393360019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:41.013464+0000 mon.a (mon.0) 1363 : audit [INF] from='client.35564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:03:42.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:03:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:03:42.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:41.714805+0000 mon.a (mon.0) 1364 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:41.728794+0000 mon.a (mon.0) 1365 : audit [INF] from='client.35564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:41.728862+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 5 in 2024-02-12T17:03:43.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:41.714805+0000 mon.a (mon.0) 1364 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:43.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:41.728794+0000 mon.a (mon.0) 1365 : audit [INF] from='client.35564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:43.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:41.728862+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 5 in 2024-02-12T17:03:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:41.714805+0000 mon.a (mon.0) 1364 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:41.728794+0000 mon.a (mon.0) 1365 : audit [INF] from='client.35564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:03:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:41.728862+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 5 in 2024-02-12T17:03:43.080 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:03:43.088 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:03:43.111 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:03:43.136 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:03:43.174 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:03:43.205 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:03:43.234 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:03:43.269 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:03:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:03:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:03:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:03:43.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:03:43.271 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:03:43.271 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:03:43.271 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:03:43.271 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:03:44.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:42.621616+0000 mgr.y (mgr.24481) 535 : cluster [DBG] pgmap v689: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 690 op/s; 42043/297681 objects misplaced (14.124%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T17:03:44.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:43.044128+0000 mon.a (mon.0) 1367 : audit [DBG] from='client.? v1:172.21.15.33:0/4289851357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:44.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:42.621616+0000 mgr.y (mgr.24481) 535 : cluster [DBG] pgmap v689: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 690 op/s; 42043/297681 objects misplaced (14.124%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T17:03:44.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:43.044128+0000 mon.a (mon.0) 1367 : audit [DBG] from='client.? v1:172.21.15.33:0/4289851357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:44.229 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:42.621616+0000 mgr.y (mgr.24481) 535 : cluster [DBG] pgmap v689: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 690 op/s; 42043/297681 objects misplaced (14.124%); 6.8 MiB/s, 108 objects/s recovering 2024-02-12T17:03:44.229 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:43.044128+0000 mon.a (mon.0) 1367 : audit [DBG] from='client.? v1:172.21.15.33:0/4289851357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:44.229 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:03:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:03:45.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:42.963379+0000 osd.7 (osd.7) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:45.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:42.963434+0000 osd.7 (osd.7) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:45.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:43.754195+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e346: 8 total, 8 up, 5 in 2024-02-12T17:03:45.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:42.963379+0000 osd.7 (osd.7) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:45.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:42.963434+0000 osd.7 (osd.7) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:45.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:43.754195+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e346: 8 total, 8 up, 5 in 2024-02-12T17:03:45.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:42.963379+0000 osd.7 (osd.7) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:45.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:42.963434+0000 osd.7 (osd.7) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:45.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:43.754195+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e346: 8 total, 8 up, 5 in 2024-02-12T17:03:45.339 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:45.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:45.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:45.375 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7] out_osds: [3, 1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:03:45.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:03:45.375 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-02-12T17:03:45.375 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd in 3 2024-02-12T17:03:45.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:45.543 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:45.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:03:45.702 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:45.971 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:44.225691+0000 mgr.y (mgr.24481) 536 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:45.971 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:44.621972+0000 mgr.y (mgr.24481) 537 : cluster [DBG] pgmap v691: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 855 op/s; 40045/287271 objects misplaced (13.940%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T17:03:45.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:44.225691+0000 mgr.y (mgr.24481) 536 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:45.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:44.621972+0000 mgr.y (mgr.24481) 537 : cluster [DBG] pgmap v691: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 855 op/s; 40045/287271 objects misplaced (13.940%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T17:03:46.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:44.225691+0000 mgr.y (mgr.24481) 536 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:46.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:44.621972+0000 mgr.y (mgr.24481) 537 : cluster [DBG] pgmap v691: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 855 op/s; 40045/287271 objects misplaced (13.940%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T17:03:46.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:03:46.528 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:46.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:46.582 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:46.648 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:03:46.681 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:46.740 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:03:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:46.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:03:46.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:46.886 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:46.622431+0000 mgr.y (mgr.24481) 538 : cluster [DBG] pgmap v692: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 671 op/s; 40045/287271 objects misplaced (13.940%); 6.8 MiB/s, 109 objects/s recovering 2024-02-12T17:03:47.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:46.622431+0000 mgr.y (mgr.24481) 538 : cluster [DBG] pgmap v692: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 671 op/s; 40045/287271 objects misplaced (13.940%); 6.8 MiB/s, 109 objects/s recovering 2024-02-12T17:03:47.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:46.622431+0000 mgr.y (mgr.24481) 538 : cluster [DBG] pgmap v692: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 671 op/s; 40045/287271 objects misplaced (13.940%); 6.8 MiB/s, 109 objects/s recovering 2024-02-12T17:03:47.839 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:47.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:47.944 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:47.972 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:48.124 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:48.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:48.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:48.308 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:48.495 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:47.488557+0000 mon.a (mon.0) 1369 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:48.727 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:48.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:47.488557+0000 mon.a (mon.0) 1369 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:48.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:47.488557+0000 mon.a (mon.0) 1369 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:03:49.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:03:49.436+0000 7fd96a9de700 -1 mon.a@0(leader).osd e346 definitely_dead 0 2024-02-12T17:03:49.536 INFO:teuthology.orchestra.run.smithi033.stderr:marked in osd.3. 2024-02-12T17:03:49.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:48.622970+0000 mgr.y (mgr.24481) 539 : cluster [DBG] pgmap v693: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 38713/282597 objects misplaced (13.699%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T17:03:49.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:49.437630+0000 mon.a (mon.0) 1370 : audit [DBG] from='client.? v1:172.21.15.33:0/3701309988' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-12T17:03:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:48.622970+0000 mgr.y (mgr.24481) 539 : cluster [DBG] pgmap v693: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 38713/282597 objects misplaced (13.699%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T17:03:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:49.437630+0000 mon.a (mon.0) 1370 : audit [DBG] from='client.? v1:172.21.15.33:0/3701309988' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-12T17:03:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:48.622970+0000 mgr.y (mgr.24481) 539 : cluster [DBG] pgmap v693: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 38713/282597 objects misplaced (13.699%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T17:03:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:49.437630+0000 mon.a (mon.0) 1370 : audit [DBG] from='client.? v1:172.21.15.33:0/3701309988' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-12T17:03:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:50.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:50.368 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:50.398 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:50.720 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-02-12T17:03:50.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:49.514121+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.33:0/3701309988' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-12T17:03:50.828 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:49.514188+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e347: 8 total, 8 up, 6 in 2024-02-12T17:03:50.829 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:49.514121+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.33:0/3701309988' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-12T17:03:50.829 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:49.514188+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e347: 8 total, 8 up, 6 in 2024-02-12T17:03:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:49.514121+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.33:0/3701309988' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-12T17:03:50.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:49.514188+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e347: 8 total, 8 up, 6 in 2024-02-12T17:03:51.125 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:03:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:50] "GET /metrics HTTP/1.1" 200 176117 "" "Prometheus/2.33.4" 2024-02-12T17:03:51.570 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:03:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:50.518053+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e348: 8 total, 8 up, 6 in 2024-02-12T17:03:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:50.530766+0000 osd.7 (osd.7) 183 : cluster [DBG] 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11350 2024-02-12T17:03:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:50.535982+0000 osd.5 (osd.5) 177 : cluster [DBG] 9.b starting backfill to osd.3 from (0'0,0'0] MAX to 346'11474 2024-02-12T17:03:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:50.542038+0000 osd.4 (osd.4) 161 : cluster [DBG] 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 346'11636 2024-02-12T17:03:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:50.555504+0000 osd.7 (osd.7) 184 : cluster [DBG] 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11326 2024-02-12T17:03:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:50.559434+0000 osd.7 (osd.7) 185 : cluster [DBG] 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11448 2024-02-12T17:03:51.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:50.623383+0000 mgr.y (mgr.24481) 540 : cluster [DBG] pgmap v696: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 745 B/s rd, 0 B/s wr, 515 op/s; 38010/276657 objects misplaced (13.739%); 5.0 MiB/s, 79 objects/s recovering 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:50.688314+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.33:0/3520421185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:50.688832+0000 mon.a (mon.0) 1374 : audit [INF] from='client.35582 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:50.518053+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e348: 8 total, 8 up, 6 in 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:50.530766+0000 osd.7 (osd.7) 183 : cluster [DBG] 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11350 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:50.535982+0000 osd.5 (osd.5) 177 : cluster [DBG] 9.b starting backfill to osd.3 from (0'0,0'0] MAX to 346'11474 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:50.542038+0000 osd.4 (osd.4) 161 : cluster [DBG] 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 346'11636 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:50.555504+0000 osd.7 (osd.7) 184 : cluster [DBG] 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11326 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:50.559434+0000 osd.7 (osd.7) 185 : cluster [DBG] 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11448 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:50.623383+0000 mgr.y (mgr.24481) 540 : cluster [DBG] pgmap v696: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 745 B/s rd, 0 B/s wr, 515 op/s; 38010/276657 objects misplaced (13.739%); 5.0 MiB/s, 79 objects/s recovering 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:50.688314+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.33:0/3520421185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:51.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:50.688832+0000 mon.a (mon.0) 1374 : audit [INF] from='client.35582 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:51.911 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:03:51.953 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:03:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:50.518053+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e348: 8 total, 8 up, 6 in 2024-02-12T17:03:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:50.530766+0000 osd.7 (osd.7) 183 : cluster [DBG] 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11350 2024-02-12T17:03:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:50.535982+0000 osd.5 (osd.5) 177 : cluster [DBG] 9.b starting backfill to osd.3 from (0'0,0'0] MAX to 346'11474 2024-02-12T17:03:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:50.542038+0000 osd.4 (osd.4) 161 : cluster [DBG] 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 346'11636 2024-02-12T17:03:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:50.555504+0000 osd.7 (osd.7) 184 : cluster [DBG] 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11326 2024-02-12T17:03:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:50.559434+0000 osd.7 (osd.7) 185 : cluster [DBG] 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11448 2024-02-12T17:03:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:50.623383+0000 mgr.y (mgr.24481) 540 : cluster [DBG] pgmap v696: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 745 B/s rd, 0 B/s wr, 515 op/s; 38010/276657 objects misplaced (13.739%); 5.0 MiB/s, 79 objects/s recovering 2024-02-12T17:03:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:50.688314+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.33:0/3520421185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:51.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:50.688832+0000 mon.a (mon.0) 1374 : audit [INF] from='client.35582 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:03:51.993 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:03:52.034 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:03:52.083 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:03:52.154 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:03:52.183 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:03:52.202 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:03:52.204 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:03:52.204 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:03:52.204 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:03:52.205 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:03:52.205 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:03:52.205 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:03:52.205 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:03:52.205 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:03:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:52.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:52.763 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:52.809 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.524979+0000 mon.a (mon.0) 1375 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:51.538874+0000 mon.a (mon.0) 1376 : audit [INF] from='client.35582 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.538915+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e349: 8 total, 8 up, 6 in 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.541078+0000 osd.2 (osd.2) 159 : cluster [DBG] 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11446 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.541412+0000 osd.5 (osd.5) 178 : cluster [DBG] 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 346'11677 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.541547+0000 osd.7 (osd.7) 186 : cluster [DBG] 5.9 starting backfill to osd.3 from (0'0,0'0] MAX to 261'13 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.543087+0000 osd.0 (osd.0) 175 : cluster [DBG] 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11527 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.547396+0000 osd.5 (osd.5) 179 : cluster [DBG] 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 346'11658 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.549739+0000 osd.0 (osd.0) 176 : cluster [DBG] 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11382 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:51.550469+0000 osd.5 (osd.5) 180 : cluster [DBG] 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11267 2024-02-12T17:03:52.810 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:51.869444+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? v1:172.21.15.33:0/2877536660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:52.811 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:03:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:03:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:03:52.817 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:52.858 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:52.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:52.967 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:53.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.524979+0000 mon.a (mon.0) 1375 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:53.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:51.538874+0000 mon.a (mon.0) 1376 : audit [INF] from='client.35582 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:03:53.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.538915+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e349: 8 total, 8 up, 6 in 2024-02-12T17:03:53.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.541078+0000 osd.2 (osd.2) 159 : cluster [DBG] 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11446 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.541412+0000 osd.5 (osd.5) 178 : cluster [DBG] 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 346'11677 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.541547+0000 osd.7 (osd.7) 186 : cluster [DBG] 5.9 starting backfill to osd.3 from (0'0,0'0] MAX to 261'13 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.543087+0000 osd.0 (osd.0) 175 : cluster [DBG] 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11527 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.547396+0000 osd.5 (osd.5) 179 : cluster [DBG] 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 346'11658 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.549739+0000 osd.0 (osd.0) 176 : cluster [DBG] 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11382 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:51.550469+0000 osd.5 (osd.5) 180 : cluster [DBG] 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11267 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:51.869444+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? v1:172.21.15.33:0/2877536660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.524979+0000 mon.a (mon.0) 1375 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:51.538874+0000 mon.a (mon.0) 1376 : audit [INF] from='client.35582 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.538915+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e349: 8 total, 8 up, 6 in 2024-02-12T17:03:53.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.541078+0000 osd.2 (osd.2) 159 : cluster [DBG] 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11446 2024-02-12T17:03:53.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.541412+0000 osd.5 (osd.5) 178 : cluster [DBG] 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 346'11677 2024-02-12T17:03:53.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.541547+0000 osd.7 (osd.7) 186 : cluster [DBG] 5.9 starting backfill to osd.3 from (0'0,0'0] MAX to 261'13 2024-02-12T17:03:53.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.543087+0000 osd.0 (osd.0) 175 : cluster [DBG] 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11527 2024-02-12T17:03:53.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.547396+0000 osd.5 (osd.5) 179 : cluster [DBG] 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 346'11658 2024-02-12T17:03:53.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.549739+0000 osd.0 (osd.0) 176 : cluster [DBG] 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11382 2024-02-12T17:03:53.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:51.550469+0000 osd.5 (osd.5) 180 : cluster [DBG] 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 346'11267 2024-02-12T17:03:53.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:51.869444+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? v1:172.21.15.33:0/2877536660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:03:53.029 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:53.145 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:53.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:53.231 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:03:53.231 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:03:53.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:53.328 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:53.490 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:53.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:53.596 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:53.663 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:53.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:52.623752+0000 mgr.y (mgr.24481) 541 : cluster [DBG] pgmap v698: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 38010/276657 objects misplaced (13.739%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T17:03:53.906 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:52.623752+0000 mgr.y (mgr.24481) 541 : cluster [DBG] pgmap v698: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 38010/276657 objects misplaced (13.739%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T17:03:53.982 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:52.623752+0000 mgr.y (mgr.24481) 541 : cluster [DBG] pgmap v698: 138 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 131 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 590 op/s; 38010/276657 objects misplaced (13.739%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T17:03:54.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T17:03:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:54.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:54.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:54.225 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:54.260 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:54.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:54.366 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:54.428 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:54.446 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:54.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:03:54.554 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:54.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:03:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:03:54.597 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:54.736 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:54.676195+0000 mon.a (mon.0) 1378 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:03:54.736 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:54.676231+0000 mon.a (mon.0) 1379 : cluster [WRN] Health check failed: Degraded data redundancy: 35/270186 objects degraded (0.013%), 6 pgs degraded (PG_DEGRADED) 2024-02-12T17:03:54.736 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:54.676195+0000 mon.a (mon.0) 1378 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:03:54.736 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:54.676231+0000 mon.a (mon.0) 1379 : cluster [WRN] Health check failed: Degraded data redundancy: 35/270186 objects degraded (0.013%), 6 pgs degraded (PG_DEGRADED) 2024-02-12T17:03:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:54.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:54.873 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:54.933 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:54.967 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:54.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:54.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:54.676195+0000 mon.a (mon.0) 1378 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:03:54.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:54.676231+0000 mon.a (mon.0) 1379 : cluster [WRN] Health check failed: Degraded data redundancy: 35/270186 objects degraded (0.013%), 6 pgs degraded (PG_DEGRADED) 2024-02-12T17:03:55.070 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:55.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:55.163 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:03:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:55.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:03:55.269 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:03:55.331 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:55.361 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:03:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:55.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:03:55.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:55.540 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:55.691 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:03:55.721 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7, 3] out_osds: [1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:03:55.721 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:03:55.722 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.meta 2024-02-12T17:03:55.722 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.meta by 10 min 8 2024-02-12T17:03:55.722 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:03:55.906 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:55.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:55.923 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:03:56.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:56.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:03:54.240052+0000 mgr.y (mgr.24481) 542 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:56.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:54.624575+0000 mgr.y (mgr.24481) 543 : cluster [DBG] pgmap v699: 138 pgs: 7 active+remapped+backfill_toofull, 6 active+recovery_wait+degraded, 1 active+recovery_wait, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 690 op/s; 35/270186 objects degraded (0.013%); 70697/270186 objects misplaced (26.166%); 15 MiB/s, 249 objects/s recovering 2024-02-12T17:03:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:03:54.240052+0000 mgr.y (mgr.24481) 542 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:56.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:54.624575+0000 mgr.y (mgr.24481) 543 : cluster [DBG] pgmap v699: 138 pgs: 7 active+remapped+backfill_toofull, 6 active+recovery_wait+degraded, 1 active+recovery_wait, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 690 op/s; 35/270186 objects degraded (0.013%); 70697/270186 objects misplaced (26.166%); 15 MiB/s, 249 objects/s recovering 2024-02-12T17:03:56.072 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:03:54.240052+0000 mgr.y (mgr.24481) 542 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:03:56.072 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:54.624575+0000 mgr.y (mgr.24481) 543 : cluster [DBG] pgmap v699: 138 pgs: 7 active+remapped+backfill_toofull, 6 active+recovery_wait+degraded, 1 active+recovery_wait, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 690 op/s; 35/270186 objects degraded (0.013%); 70697/270186 objects misplaced (26.166%); 15 MiB/s, 249 objects/s recovering 2024-02-12T17:03:56.084 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:56.176 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:56.180 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:56.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:03:56.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:03:56.336 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:56.397 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:03:56.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:56.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:03:56.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:56.584 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:56.588 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:03:56.588 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:03:56.623 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:56.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:56.728 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:57.306 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:55.955468+0000 osd.3 (osd.3) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:57.306 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:55.955525+0000 osd.3 (osd.3) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:57.306 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:56.624969+0000 mgr.y (mgr.24481) 544 : cluster [DBG] pgmap v700: 138 pgs: 7 active+remapped+backfill_toofull, 6 active+recovery_wait+degraded, 1 active+recovery_wait, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 287 B/s rd, 0 B/s wr, 303 op/s; 35/270186 objects degraded (0.013%); 70697/270186 objects misplaced (26.166%); 10 MiB/s, 166 objects/s recovering 2024-02-12T17:03:57.306 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:56.777073+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e350: 8 total, 8 up, 6 in 2024-02-12T17:03:57.344 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:57.346 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T17:03:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:57.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:57.448 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:03:57.532 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:57.750 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:03:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:57.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:03:57.756 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:55.955468+0000 osd.3 (osd.3) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:57.756 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:55.955525+0000 osd.3 (osd.3) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:57.757 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:56.624969+0000 mgr.y (mgr.24481) 544 : cluster [DBG] pgmap v700: 138 pgs: 7 active+remapped+backfill_toofull, 6 active+recovery_wait+degraded, 1 active+recovery_wait, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 287 B/s rd, 0 B/s wr, 303 op/s; 35/270186 objects degraded (0.013%); 70697/270186 objects misplaced (26.166%); 10 MiB/s, 166 objects/s recovering 2024-02-12T17:03:57.757 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:56.777073+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e350: 8 total, 8 up, 6 in 2024-02-12T17:03:57.757 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:55.955468+0000 osd.3 (osd.3) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:57.757 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:55.955525+0000 osd.3 (osd.3) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:57.757 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:56.624969+0000 mgr.y (mgr.24481) 544 : cluster [DBG] pgmap v700: 138 pgs: 7 active+remapped+backfill_toofull, 6 active+recovery_wait+degraded, 1 active+recovery_wait, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 287 B/s rd, 0 B/s wr, 303 op/s; 35/270186 objects degraded (0.013%); 70697/270186 objects misplaced (26.166%); 10 MiB/s, 166 objects/s recovering 2024-02-12T17:03:57.757 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:56.777073+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e350: 8 total, 8 up, 6 in 2024-02-12T17:03:57.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:03:57.926 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:57.974 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:03:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:57.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:03:58.080 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:03:58.142 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:58.183 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:03:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:58.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:03:58.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:03:58.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:03:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:03:58.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:03:58.469 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:03:58.532 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:03:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:58.608436+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:03:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:58.608476+0000 osd.6 (osd.6) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:03:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:58.625991+0000 mgr.y (mgr.24481) 545 : cluster [DBG] pgmap v702: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 491 op/s; 69061/264870 objects misplaced (26.074%); 9.8 MiB/s, 167 objects/s recovering 2024-02-12T17:03:59.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:58.669119+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e351: 8 total, 8 up, 6 in 2024-02-12T17:03:59.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:03:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:03:58.686483+0000 mon.a (mon.0) 1382 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/270186 objects degraded (0.013%), 6 pgs degraded) 2024-02-12T17:04:00.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:58.608436+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:00.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:58.608476+0000 osd.6 (osd.6) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:00.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:58.625991+0000 mgr.y (mgr.24481) 545 : cluster [DBG] pgmap v702: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 491 op/s; 69061/264870 objects misplaced (26.074%); 9.8 MiB/s, 167 objects/s recovering 2024-02-12T17:04:00.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:58.669119+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e351: 8 total, 8 up, 6 in 2024-02-12T17:04:00.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:03:58.686483+0000 mon.a (mon.0) 1382 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/270186 objects degraded (0.013%), 6 pgs degraded) 2024-02-12T17:04:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:58.608436+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:58.608476+0000 osd.6 (osd.6) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:58.625991+0000 mgr.y (mgr.24481) 545 : cluster [DBG] pgmap v702: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 491 op/s; 69061/264870 objects misplaced (26.074%); 9.8 MiB/s, 167 objects/s recovering 2024-02-12T17:04:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:58.669119+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e351: 8 total, 8 up, 6 in 2024-02-12T17:04:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:03:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:03:58.686483+0000 mon.a (mon.0) 1382 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/270186 objects degraded (0.013%), 6 pgs degraded) 2024-02-12T17:04:01.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:04:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:00] "GET /metrics HTTP/1.1" 200 176117 "" "Prometheus/2.33.4" 2024-02-12T17:04:01.837 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:01.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:01.892 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:01.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:01.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:00.630146+0000 mgr.y (mgr.24481) 546 : cluster [DBG] pgmap v704: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 895 B/s rd, 0 B/s wr, 776 op/s; 66734/258021 objects misplaced (25.864%); 15 MiB/s, 248 objects/s recovering 2024-02-12T17:04:01.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:00.849896+0000 osd.4 (osd.4) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:01.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:00.849949+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:02.025 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:02.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:00.630146+0000 mgr.y (mgr.24481) 546 : cluster [DBG] pgmap v704: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 895 B/s rd, 0 B/s wr, 776 op/s; 66734/258021 objects misplaced (25.864%); 15 MiB/s, 248 objects/s recovering 2024-02-12T17:04:02.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:00.849896+0000 osd.4 (osd.4) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:02.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:00.849949+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:00.630146+0000 mgr.y (mgr.24481) 546 : cluster [DBG] pgmap v704: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 895 B/s rd, 0 B/s wr, 776 op/s; 66734/258021 objects misplaced (25.864%); 15 MiB/s, 248 objects/s recovering 2024-02-12T17:04:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:00.849896+0000 osd.4 (osd.4) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:00.849949+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:02.064 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:02.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:02.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:02.218 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:02.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:01.706599+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e352: 8 total, 8 up, 6 in 2024-02-12T17:04:02.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:02.489562+0000 mon.a (mon.0) 1384 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:02.995 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:04:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:04:03.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:01.706599+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e352: 8 total, 8 up, 6 in 2024-02-12T17:04:03.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:02.489562+0000 mon.a (mon.0) 1384 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:03.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:01.706599+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e352: 8 total, 8 up, 6 in 2024-02-12T17:04:03.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:02.489562+0000 mon.a (mon.0) 1384 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:04.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:02.630617+0000 mgr.y (mgr.24481) 547 : cluster [DBG] pgmap v706: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 676 op/s; 66734/258021 objects misplaced (25.864%); 7.9 MiB/s, 134 objects/s recovering 2024-02-12T17:04:04.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:02.948601+0000 mon.a (mon.0) 1385 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:04.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:02.630617+0000 mgr.y (mgr.24481) 547 : cluster [DBG] pgmap v706: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 676 op/s; 66734/258021 objects misplaced (25.864%); 7.9 MiB/s, 134 objects/s recovering 2024-02-12T17:04:04.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:02.948601+0000 mon.a (mon.0) 1385 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:04.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:02.630617+0000 mgr.y (mgr.24481) 547 : cluster [DBG] pgmap v706: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 121 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 676 op/s; 66734/258021 objects misplaced (25.864%); 7.9 MiB/s, 134 objects/s recovering 2024-02-12T17:04:04.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:02.948601+0000 mon.a (mon.0) 1385 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:04.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:04:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:04:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:04:04.968 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:04:05.037 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:04:05.076 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:04:05.106 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:04:05.135 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:04:05.166 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:04:05.203 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:04:05.236 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:04:05.268 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:04:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:04:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:04:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:04:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:04:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:04:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:04:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:04:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:04:05.271 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:05.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:05.375 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:05.423 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:06.050 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2024-02-12T17:04:04.632986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5333325500,"num_objects":81573,"num_object_clones":0,"num_object_copies":244719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":62413,"num_objects_unfound":0,"num_objects_dirty":81573,"num_whiteouts":0,"num_read":1291,"num_read_kb":1043,"num_write":194509,"num_write_kb":8805610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6036,"num_bytes_recovered":362886209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64945,"ondisk_log_size":64945,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":433,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19646464,"kb_used_data":16060752,"kb_used_omap":0,"kb_used_meta":3585344,"kb_avail":542824448,"statfs":{"total":575970213888,"available":555852234752,"internally_reserved":0,"allocated":16446210048,"data_stored":16443834624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3671392256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-556335104,"num_objects":-8489,"num_object_clones":0,"num_object_copies":-25467,"num_objects_missing_on_primary":-46,"num_objects_missing":-46,"num_objects_degraded":-35,"num_objects_misplaced":-8284,"num_objects_unfound":0,"num_objects_dirty":-8489,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8489,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":110166786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.852740"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686171+0000","last_change":"2024-02-12T17:03:50.610133+0000","last_active":"2024-02-12T17:03:58.686171+0000","last_peered":"2024-02-12T17:03:58.686171+0000","last_clean":"2024-02-12T17:03:58.686171+0000","last_became_active":"2024-02-12T17:03:50.608823+0000","last_became_peered":"2024-02-12T17:03:50.608823+0000","last_unstale":"2024-02-12T17:03:58.686171+0000","last_undegraded":"2024-02-12T17:03:58.686171+0000","last_fullsized":"2024-02-12T17:03:58.686171+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.689645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.689645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":32,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679407+0000","last_change":"2024-02-12T17:03:50.565051+0000","last_active":"2024-02-12T17:03:58.679407+0000","last_peered":"2024-02-12T17:03:58.679407+0000","last_clean":"2024-02-12T17:03:58.679407+0000","last_became_active":"2024-02-12T17:03:50.564751+0000","last_became_peered":"2024-02-12T17:03:50.564751+0000","last_unstale":"2024-02-12T17:03:58.679407+0000","last_undegraded":"2024-02-12T17:03:58.679407+0000","last_fullsized":"2024-02-12T17:03:58.679407+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.732293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.732293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":30,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685348+0000","last_change":"2024-02-12T17:03:52.280899+0000","last_active":"2024-02-12T17:03:58.685348+0000","last_peered":"2024-02-12T17:03:58.685348+0000","last_clean":"2024-02-12T17:03:58.685348+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:03:58.685348+0000","last_undegraded":"2024-02-12T17:03:58.685348+0000","last_fullsized":"2024-02-12T17:03:58.685348+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:01:26.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.047527+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":285,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.690919+0000","last_change":"2024-02-12T17:00:56.892829+0000","last_active":"2024-02-12T17:03:58.690919+0000","last_peered":"2024-02-12T17:03:58.690919+0000","last_clean":"2024-02-12T17:03:58.690919+0000","last_became_active":"2024-02-12T16:57:47.188886+0000","last_became_peered":"2024-02-12T16:57:47.188886+0000","last_unstale":"2024-02-12T17:03:58.690919+0000","last_undegraded":"2024-02-12T17:03:58.690919+0000","last_fullsized":"2024-02-12T17:03:58.690919+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.892756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:56.892756+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.892756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686863+0000","last_change":"2024-02-12T17:03:50.545988+0000","last_active":"2024-02-12T17:03:58.686863+0000","last_peered":"2024-02-12T17:03:58.686863+0000","last_clean":"2024-02-12T17:03:58.686863+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:03:58.686863+0000","last_undegraded":"2024-02-12T17:03:58.686863+0000","last_fullsized":"2024-02-12T17:03:58.686863+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.176431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":432,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.673114+0000","last_change":"2024-02-12T17:03:50.625045+0000","last_active":"2024-02-12T17:03:58.673114+0000","last_peered":"2024-02-12T17:03:58.673114+0000","last_clean":"2024-02-12T17:03:58.673114+0000","last_became_active":"2024-02-12T17:03:50.550838+0000","last_became_peered":"2024-02-12T17:03:50.550838+0000","last_unstale":"2024-02-12T17:03:58.673114+0000","last_undegraded":"2024-02-12T17:03:58.673114+0000","last_fullsized":"2024-02-12T17:03:58.673114+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:01:45.154164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":353,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687050+0000","last_change":"2024-02-12T17:03:50.544387+0000","last_active":"2024-02-12T17:03:58.687050+0000","last_peered":"2024-02-12T17:03:58.687050+0000","last_clean":"2024-02-12T17:03:58.687050+0000","last_became_active":"2024-02-12T17:03:50.544048+0000","last_became_peered":"2024-02-12T17:03:50.544048+0000","last_unstale":"2024-02-12T17:03:58.687050+0000","last_undegraded":"2024-02-12T17:03:58.687050+0000","last_fullsized":"2024-02-12T17:03:58.687050+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:46.577048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:46.577048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":347,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679915+0000","last_change":"2024-02-12T17:03:50.565273+0000","last_active":"2024-02-12T17:03:58.679915+0000","last_peered":"2024-02-12T17:03:58.679915+0000","last_clean":"2024-02-12T17:03:58.679915+0000","last_became_active":"2024-02-12T17:03:50.562938+0000","last_became_peered":"2024-02-12T17:03:50.562938+0000","last_unstale":"2024-02-12T17:03:58.679915+0000","last_undegraded":"2024-02-12T17:03:58.679915+0000","last_fullsized":"2024-02-12T17:03:58.679915+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:37.813008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:01:37.813008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":20,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686263+0000","last_change":"2024-02-12T17:03:52.773691+0000","last_active":"2024-02-12T17:03:58.686263+0000","last_peered":"2024-02-12T17:03:58.686263+0000","last_clean":"2024-02-12T17:03:58.686263+0000","last_became_active":"2024-02-12T17:03:50.550921+0000","last_became_peered":"2024-02-12T17:03:50.550921+0000","last_unstale":"2024-02-12T17:03:58.686263+0000","last_undegraded":"2024-02-12T17:03:58.686263+0000","last_fullsized":"2024-02-12T17:03:58.686263+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":188,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.677914+0000","last_change":"2024-02-12T17:03:20.775409+0000","last_active":"2024-02-12T17:03:58.677914+0000","last_peered":"2024-02-12T17:03:58.677914+0000","last_clean":"2024-02-12T17:03:58.677914+0000","last_became_active":"2024-02-12T17:03:20.774370+0000","last_became_peered":"2024-02-12T17:03:20.774370+0000","last_unstale":"2024-02-12T17:03:58.677914+0000","last_undegraded":"2024-02-12T17:03:58.677914+0000","last_fullsized":"2024-02-12T17:03:58.677914+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:50.923462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.923462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":321,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678627+0000","last_change":"2024-02-12T17:02:34.502434+0000","last_active":"2024-02-12T17:03:58.678627+0000","last_peered":"2024-02-12T17:03:58.678627+0000","last_clean":"2024-02-12T17:03:58.678627+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:03:58.678627+0000","last_undegraded":"2024-02-12T17:03:58.678627+0000","last_fullsized":"2024-02-12T17:03:58.678627+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.502332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":24,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685589+0000","last_change":"2024-02-12T17:03:52.417482+0000","last_active":"2024-02-12T17:03:58.685589+0000","last_peered":"2024-02-12T17:03:58.685589+0000","last_clean":"2024-02-12T17:03:58.685589+0000","last_became_active":"2024-02-12T17:03:50.549628+0000","last_became_peered":"2024-02-12T17:03:50.549628+0000","last_unstale":"2024-02-12T17:03:58.685589+0000","last_undegraded":"2024-02-12T17:03:58.685589+0000","last_fullsized":"2024-02-12T17:03:58.685589+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686789+0000","last_change":"2024-02-12T17:03:50.558277+0000","last_active":"2024-02-12T17:03:58.686789+0000","last_peered":"2024-02-12T17:03:58.686789+0000","last_clean":"2024-02-12T17:03:58.686789+0000","last_became_active":"2024-02-12T17:03:50.557508+0000","last_became_peered":"2024-02-12T17:03:50.557508+0000","last_unstale":"2024-02-12T17:03:58.686789+0000","last_undegraded":"2024-02-12T17:03:58.686789+0000","last_fullsized":"2024-02-12T17:03:58.686789+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.973480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.973480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685864+0000","last_change":"2024-02-12T17:03:50.610470+0000","last_active":"2024-02-12T17:03:58.685864+0000","last_peered":"2024-02-12T17:03:58.685864+0000","last_clean":"2024-02-12T17:03:58.685864+0000","last_became_active":"2024-02-12T17:03:50.609556+0000","last_became_peered":"2024-02-12T17:03:50.609556+0000","last_unstale":"2024-02-12T17:03:58.685864+0000","last_undegraded":"2024-02-12T17:03:58.685864+0000","last_fullsized":"2024-02-12T17:03:58.685864+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":26,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686112+0000","last_change":"2024-02-12T17:03:51.702322+0000","last_active":"2024-02-12T17:03:58.686112+0000","last_peered":"2024-02-12T17:03:58.686112+0000","last_clean":"2024-02-12T17:03:58.686112+0000","last_became_active":"2024-02-12T17:03:50.548107+0000","last_became_peered":"2024-02-12T17:03:50.548107+0000","last_unstale":"2024-02-12T17:03:58.686112+0000","last_undegraded":"2024-02-12T17:03:58.686112+0000","last_fullsized":"2024-02-12T17:03:58.686112+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":354,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.676215+0000","last_change":"2024-02-12T17:03:50.552465+0000","last_active":"2024-02-12T17:03:58.676215+0000","last_peered":"2024-02-12T17:03:58.676215+0000","last_clean":"2024-02-12T17:03:58.676215+0000","last_became_active":"2024-02-12T17:03:50.551661+0000","last_became_peered":"2024-02-12T17:03:50.551661+0000","last_unstale":"2024-02-12T17:03:58.676215+0000","last_undegraded":"2024-02-12T17:03:58.676215+0000","last_fullsized":"2024-02-12T17:03:58.676215+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:21.149801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:10.898594+0000","last_clean_scrub_stamp":"2024-02-12T17:02:21.149801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":111,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.673363+0000","last_change":"2024-02-12T17:01:27.157444+0000","last_active":"2024-02-12T17:03:58.673363+0000","last_peered":"2024-02-12T17:03:58.673363+0000","last_clean":"2024-02-12T17:03:58.673363+0000","last_became_active":"2024-02-12T16:59:09.248342+0000","last_became_peered":"2024-02-12T16:59:09.248342+0000","last_unstale":"2024-02-12T17:03:58.673363+0000","last_undegraded":"2024-02-12T17:03:58.673363+0000","last_fullsized":"2024-02-12T17:03:58.673363+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.157357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.157357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":30,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.768348+0000","last_change":"2024-02-12T17:03:58.768015+0000","last_active":"2024-02-12T17:03:58.768348+0000","last_peered":"2024-02-12T17:03:58.768348+0000","last_clean":"2024-02-12T17:03:58.768348+0000","last_became_active":"2024-02-12T17:03:50.567155+0000","last_became_peered":"2024-02-12T17:03:50.567155+0000","last_unstale":"2024-02-12T17:03:58.768348+0000","last_undegraded":"2024-02-12T17:03:58.768348+0000","last_fullsized":"2024-02-12T17:03:58.768348+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":161,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.689932+0000","last_change":"2024-02-12T17:02:37.745936+0000","last_active":"2024-02-12T17:03:58.689932+0000","last_peered":"2024-02-12T17:03:58.689932+0000","last_clean":"2024-02-12T17:03:58.689932+0000","last_became_active":"2024-02-12T16:57:47.178240+0000","last_became_peered":"2024-02-12T16:57:47.178240+0000","last_unstale":"2024-02-12T17:03:58.689932+0000","last_undegraded":"2024-02-12T17:03:58.689932+0000","last_fullsized":"2024-02-12T17:03:58.689932+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:37.745838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.745838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686743+0000","last_change":"2024-02-12T17:03:50.533143+0000","last_active":"2024-02-12T17:03:58.686743+0000","last_peered":"2024-02-12T17:03:58.686743+0000","last_clean":"2024-02-12T17:03:58.686743+0000","last_became_active":"2024-02-12T17:03:50.532233+0000","last_became_peered":"2024-02-12T17:03:50.532233+0000","last_unstale":"2024-02-12T17:03:58.686743+0000","last_undegraded":"2024-02-12T17:03:58.686743+0000","last_fullsized":"2024-02-12T17:03:58.686743+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":208,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.692550+0000","last_change":"2024-02-12T17:03:50.670847+0000","last_active":"2024-02-12T17:03:58.692550+0000","last_peered":"2024-02-12T17:03:58.692550+0000","last_clean":"2024-02-12T17:03:58.692550+0000","last_became_active":"2024-02-12T17:03:50.570118+0000","last_became_peered":"2024-02-12T17:03:50.570118+0000","last_unstale":"2024-02-12T17:03:58.692550+0000","last_undegraded":"2024-02-12T17:03:58.692550+0000","last_fullsized":"2024-02-12T17:03:58.692550+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:34.738385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.738385+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":312,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679064+0000","last_change":"2024-02-12T16:57:37.670804+0000","last_active":"2024-02-12T17:03:58.679064+0000","last_peered":"2024-02-12T17:03:58.679064+0000","last_clean":"2024-02-12T17:03:58.679064+0000","last_became_active":"2024-02-12T16:49:45.223151+0000","last_became_peered":"2024-02-12T16:49:45.223151+0000","last_unstale":"2024-02-12T17:03:58.679064+0000","last_undegraded":"2024-02-12T17:03:58.679064+0000","last_fullsized":"2024-02-12T17:03:58.679064+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T16:57:37.670713+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-02-12T16:51:28.103969+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.670713+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":318,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687720+0000","last_change":"2024-02-12T16:57:26.790389+0000","last_active":"2024-02-12T17:03:58.687720+0000","last_peered":"2024-02-12T17:03:58.687720+0000","last_clean":"2024-02-12T17:03:58.687720+0000","last_became_active":"2024-02-12T16:49:39.105604+0000","last_became_peered":"2024-02-12T16:49:39.105604+0000","last_unstale":"2024-02-12T17:03:58.687720+0000","last_undegraded":"2024-02-12T17:03:58.687720+0000","last_fullsized":"2024-02-12T17:03:58.687720+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":366,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.681681+0000","last_change":"2024-02-12T17:03:51.528465+0000","last_active":"2024-02-12T17:03:58.681681+0000","last_peered":"2024-02-12T17:03:58.681681+0000","last_clean":"2024-02-12T17:03:58.681681+0000","last_became_active":"2024-02-12T17:03:50.576714+0000","last_became_peered":"2024-02-12T17:03:50.576714+0000","last_unstale":"2024-02-12T17:03:58.681681+0000","last_undegraded":"2024-02-12T17:03:58.681681+0000","last_fullsized":"2024-02-12T17:03:58.681681+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:01:42.076584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.076584+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686269+0000","last_change":"2024-02-12T17:03:50.551528+0000","last_active":"2024-02-12T17:03:58.686269+0000","last_peered":"2024-02-12T17:03:58.686269+0000","last_clean":"2024-02-12T17:03:58.686269+0000","last_became_active":"2024-02-12T17:03:50.551226+0000","last_became_peered":"2024-02-12T17:03:50.551226+0000","last_unstale":"2024-02-12T17:03:58.686269+0000","last_undegraded":"2024-02-12T17:03:58.686269+0000","last_fullsized":"2024-02-12T17:03:58.686269+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":330,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678840+0000","last_change":"2024-02-12T17:03:20.777172+0000","last_active":"2024-02-12T17:03:58.678840+0000","last_peered":"2024-02-12T17:03:58.678840+0000","last_clean":"2024-02-12T17:03:58.678840+0000","last_became_active":"2024-02-12T17:03:20.776820+0000","last_became_peered":"2024-02-12T17:03:20.776820+0000","last_unstale":"2024-02-12T17:03:58.678840+0000","last_undegraded":"2024-02-12T17:03:58.678840+0000","last_fullsized":"2024-02-12T17:03:58.678840+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:51.269648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.269648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":115,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.692308+0000","last_change":"2024-02-12T17:03:20.771577+0000","last_active":"2024-02-12T17:03:58.692308+0000","last_peered":"2024-02-12T17:03:58.692308+0000","last_clean":"2024-02-12T17:03:58.692308+0000","last_became_active":"2024-02-12T17:03:20.770459+0000","last_became_peered":"2024-02-12T17:03:20.770459+0000","last_unstale":"2024-02-12T17:03:58.692308+0000","last_undegraded":"2024-02-12T17:03:58.692308+0000","last_fullsized":"2024-02-12T17:03:58.692308+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":26,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:59.035633+0000","last_change":"2024-02-12T17:03:59.035349+0000","last_active":"2024-02-12T17:03:59.035633+0000","last_peered":"2024-02-12T17:03:59.035633+0000","last_clean":"2024-02-12T17:03:59.035633+0000","last_became_active":"2024-02-12T17:03:50.639321+0000","last_became_peered":"2024-02-12T17:03:50.639321+0000","last_unstale":"2024-02-12T17:03:59.035633+0000","last_undegraded":"2024-02-12T17:03:59.035633+0000","last_fullsized":"2024-02-12T17:03:59.035633+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":310,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678773+0000","last_change":"2024-02-12T16:57:28.846844+0000","last_active":"2024-02-12T17:03:58.678773+0000","last_peered":"2024-02-12T17:03:58.678773+0000","last_clean":"2024-02-12T17:03:58.678773+0000","last_became_active":"2024-02-12T16:49:45.223150+0000","last_became_peered":"2024-02-12T16:49:45.223150+0000","last_unstale":"2024-02-12T17:03:58.678773+0000","last_undegraded":"2024-02-12T17:03:58.678773+0000","last_fullsized":"2024-02-12T17:03:58.678773+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":134,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.673865+0000","last_change":"2024-02-12T17:03:50.580689+0000","last_active":"2024-02-12T17:03:58.673865+0000","last_peered":"2024-02-12T17:03:58.673865+0000","last_clean":"2024-02-12T17:03:58.673865+0000","last_became_active":"2024-02-12T17:03:50.579907+0000","last_became_peered":"2024-02-12T17:03:50.579907+0000","last_unstale":"2024-02-12T17:03:58.673865+0000","last_undegraded":"2024-02-12T17:03:58.673865+0000","last_fullsized":"2024-02-12T17:03:58.673865+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.020845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.020845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.688248+0000","last_change":"2024-02-12T17:03:50.551444+0000","last_active":"2024-02-12T17:03:58.688248+0000","last_peered":"2024-02-12T17:03:58.688248+0000","last_clean":"2024-02-12T17:03:58.688248+0000","last_became_active":"2024-02-12T17:03:50.550931+0000","last_became_peered":"2024-02-12T17:03:50.550931+0000","last_unstale":"2024-02-12T17:03:58.688248+0000","last_undegraded":"2024-02-12T17:03:58.688248+0000","last_fullsized":"2024-02-12T17:03:58.688248+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":20,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.684870+0000","last_change":"2024-02-12T17:03:51.098140+0000","last_active":"2024-02-12T17:03:58.684870+0000","last_peered":"2024-02-12T17:03:58.684870+0000","last_clean":"2024-02-12T17:03:58.684870+0000","last_became_active":"2024-02-12T17:03:50.548199+0000","last_became_peered":"2024-02-12T17:03:50.548199+0000","last_unstale":"2024-02-12T17:03:58.684870+0000","last_undegraded":"2024-02-12T17:03:58.684870+0000","last_fullsized":"2024-02-12T17:03:58.684870+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.938805+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685532+0000","last_change":"2024-02-12T17:03:50.545703+0000","last_active":"2024-02-12T17:03:58.685532+0000","last_peered":"2024-02-12T17:03:58.685532+0000","last_clean":"2024-02-12T17:03:58.685532+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:03:58.685532+0000","last_undegraded":"2024-02-12T17:03:58.685532+0000","last_fullsized":"2024-02-12T17:03:58.685532+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.051528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.051528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":274,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.691286+0000","last_change":"2024-02-12T17:03:50.577696+0000","last_active":"2024-02-12T17:03:58.691286+0000","last_peered":"2024-02-12T17:03:58.691286+0000","last_clean":"2024-02-12T17:03:58.691286+0000","last_became_active":"2024-02-12T17:03:50.577250+0000","last_became_peered":"2024-02-12T17:03:50.577250+0000","last_unstale":"2024-02-12T17:03:58.691286+0000","last_undegraded":"2024-02-12T17:03:58.691286+0000","last_fullsized":"2024-02-12T17:03:58.691286+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":179,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.689611+0000","last_change":"2024-02-12T17:03:20.768326+0000","last_active":"2024-02-12T17:03:58.689611+0000","last_peered":"2024-02-12T17:03:58.689611+0000","last_clean":"2024-02-12T17:03:58.689611+0000","last_became_active":"2024-02-12T17:03:20.767943+0000","last_became_peered":"2024-02-12T17:03:20.767943+0000","last_unstale":"2024-02-12T17:03:58.689611+0000","last_undegraded":"2024-02-12T17:03:58.689611+0000","last_fullsized":"2024-02-12T17:03:58.689611+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.858411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.858411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":402,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678100+0000","last_change":"2024-02-12T17:02:16.065061+0000","last_active":"2024-02-12T17:03:58.678100+0000","last_peered":"2024-02-12T17:03:58.678100+0000","last_clean":"2024-02-12T17:03:58.678100+0000","last_became_active":"2024-02-12T16:59:09.274991+0000","last_became_peered":"2024-02-12T16:59:09.274991+0000","last_unstale":"2024-02-12T17:03:58.678100+0000","last_undegraded":"2024-02-12T17:03:58.678100+0000","last_fullsized":"2024-02-12T17:03:58.678100+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:16.064966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.064966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":252,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687797+0000","last_change":"2024-02-12T17:03:50.547985+0000","last_active":"2024-02-12T17:03:58.687797+0000","last_peered":"2024-02-12T17:03:58.687797+0000","last_clean":"2024-02-12T17:03:58.687797+0000","last_became_active":"2024-02-12T17:03:50.547658+0000","last_became_peered":"2024-02-12T17:03:50.547658+0000","last_unstale":"2024-02-12T17:03:58.687797+0000","last_undegraded":"2024-02-12T17:03:58.687797+0000","last_fullsized":"2024-02-12T17:03:58.687797+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.019813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.019813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685967+0000","last_change":"2024-02-12T17:03:50.609884+0000","last_active":"2024-02-12T17:03:58.685967+0000","last_peered":"2024-02-12T17:03:58.685967+0000","last_clean":"2024-02-12T17:03:58.685967+0000","last_became_active":"2024-02-12T17:03:50.605533+0000","last_became_peered":"2024-02-12T17:03:50.605533+0000","last_unstale":"2024-02-12T17:03:58.685967+0000","last_undegraded":"2024-02-12T17:03:58.685967+0000","last_fullsized":"2024-02-12T17:03:58.685967+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:33.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:01:33.149132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686842+0000","last_change":"2024-02-12T17:03:50.559509+0000","last_active":"2024-02-12T17:03:58.686842+0000","last_peered":"2024-02-12T17:03:58.686842+0000","last_clean":"2024-02-12T17:03:58.686842+0000","last_became_active":"2024-02-12T17:03:50.558598+0000","last_became_peered":"2024-02-12T17:03:50.558598+0000","last_unstale":"2024-02-12T17:03:58.686842+0000","last_undegraded":"2024-02-12T17:03:58.686842+0000","last_fullsized":"2024-02-12T17:03:58.686842+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":128,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.690631+0000","last_change":"2024-02-12T17:03:22.527733+0000","last_active":"2024-02-12T17:03:58.690631+0000","last_peered":"2024-02-12T17:03:58.690631+0000","last_clean":"2024-02-12T17:03:58.690631+0000","last_became_active":"2024-02-12T17:03:20.774315+0000","last_became_peered":"2024-02-12T17:03:20.774315+0000","last_unstale":"2024-02-12T17:03:58.690631+0000","last_undegraded":"2024-02-12T17:03:58.690631+0000","last_fullsized":"2024-02-12T17:03:58.690631+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:00:53.792980+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.792980+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":64,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.692232+0000","last_change":"2024-02-12T17:02:36.771544+0000","last_active":"2024-02-12T17:03:58.692232+0000","last_peered":"2024-02-12T17:03:58.692232+0000","last_clean":"2024-02-12T17:03:58.692232+0000","last_became_active":"2024-02-12T17:01:16.909120+0000","last_became_peered":"2024-02-12T17:01:16.909120+0000","last_unstale":"2024-02-12T17:03:58.692232+0000","last_undegraded":"2024-02-12T17:03:58.692232+0000","last_fullsized":"2024-02-12T17:03:58.692232+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:36.771446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:36.771446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"348'11","reported_seq":22,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:01.734242+0000","last_change":"2024-02-12T17:03:54.814978+0000","last_active":"2024-02-12T17:04:01.734242+0000","last_peered":"2024-02-12T17:04:01.734242+0000","last_clean":"2024-02-12T17:04:01.734242+0000","last_became_active":"2024-02-12T17:03:50.558517+0000","last_became_peered":"2024-02-12T17:03:50.558517+0000","last_unstale":"2024-02-12T17:04:01.734242+0000","last_undegraded":"2024-02-12T17:04:01.734242+0000","last_fullsized":"2024-02-12T17:04:01.734242+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-12T17:02:17.689218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.689218+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":129,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.677899+0000","last_change":"2024-02-12T17:03:50.580157+0000","last_active":"2024-02-12T17:03:58.677899+0000","last_peered":"2024-02-12T17:03:58.677899+0000","last_clean":"2024-02-12T17:03:58.677899+0000","last_became_active":"2024-02-12T17:03:50.579869+0000","last_became_peered":"2024-02-12T17:03:50.579869+0000","last_unstale":"2024-02-12T17:03:58.677899+0000","last_undegraded":"2024-02-12T17:03:58.677899+0000","last_fullsized":"2024-02-12T17:03:58.677899+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.953891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.953891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":417,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.689540+0000","last_change":"2024-02-12T17:03:50.690222+0000","last_active":"2024-02-12T17:03:58.689540+0000","last_peered":"2024-02-12T17:03:58.689540+0000","last_clean":"2024-02-12T17:03:58.689540+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:03:58.689540+0000","last_undegraded":"2024-02-12T17:03:58.689540+0000","last_fullsized":"2024-02-12T17:03:58.689540+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:31.827632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:31.827632+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":123,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.681576+0000","last_change":"2024-02-12T17:03:50.576365+0000","last_active":"2024-02-12T17:03:58.681576+0000","last_peered":"2024-02-12T17:03:58.681576+0000","last_clean":"2024-02-12T17:03:58.681576+0000","last_became_active":"2024-02-12T17:03:50.569140+0000","last_became_peered":"2024-02-12T17:03:50.569140+0000","last_unstale":"2024-02-12T17:03:58.681576+0000","last_undegraded":"2024-02-12T17:03:58.681576+0000","last_fullsized":"2024-02-12T17:03:58.681576+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.835883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:49.213945+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.835883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"348'11","reported_seq":537,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:01.724565+0000","last_change":"2024-02-12T17:03:50.628719+0000","last_active":"2024-02-12T17:04:01.724565+0000","last_peered":"2024-02-12T17:04:01.724565+0000","last_clean":"2024-02-12T17:04:01.724565+0000","last_became_active":"2024-02-12T17:03:50.552460+0000","last_became_peered":"2024-02-12T17:03:50.552460+0000","last_unstale":"2024-02-12T17:04:01.724565+0000","last_undegraded":"2024-02-12T17:04:01.724565+0000","last_fullsized":"2024-02-12T17:04:01.724565+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'9","last_scrub_stamp":"2024-02-12T17:00:49.846432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.846432+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":336,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.691057+0000","last_change":"2024-02-12T17:03:50.577996+0000","last_active":"2024-02-12T17:03:58.691057+0000","last_peered":"2024-02-12T17:03:58.691057+0000","last_clean":"2024-02-12T17:03:58.691057+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:03:58.691057+0000","last_undegraded":"2024-02-12T17:03:58.691057+0000","last_fullsized":"2024-02-12T17:03:58.691057+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:34.874363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:34.874363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":26,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686735+0000","last_change":"2024-02-12T17:03:55.244602+0000","last_active":"2024-02-12T17:03:58.686735+0000","last_peered":"2024-02-12T17:03:58.686735+0000","last_clean":"2024-02-12T17:03:58.686735+0000","last_became_active":"2024-02-12T17:03:50.556444+0000","last_became_peered":"2024-02-12T17:03:50.556444+0000","last_unstale":"2024-02-12T17:03:58.686735+0000","last_undegraded":"2024-02-12T17:03:58.686735+0000","last_fullsized":"2024-02-12T17:03:58.686735+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":323,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679841+0000","last_change":"2024-02-12T17:00:58.187842+0000","last_active":"2024-02-12T17:03:58.679841+0000","last_peered":"2024-02-12T17:03:58.679841+0000","last_clean":"2024-02-12T17:03:58.679841+0000","last_became_active":"2024-02-12T16:59:09.269414+0000","last_became_peered":"2024-02-12T16:59:09.269414+0000","last_unstale":"2024-02-12T17:03:58.679841+0000","last_undegraded":"2024-02-12T17:03:58.679841+0000","last_fullsized":"2024-02-12T17:03:58.679841+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.187738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":302,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.673958+0000","last_change":"2024-02-12T17:03:50.553332+0000","last_active":"2024-02-12T17:03:58.673958+0000","last_peered":"2024-02-12T17:03:58.673958+0000","last_clean":"2024-02-12T17:03:58.673958+0000","last_became_active":"2024-02-12T17:03:50.552762+0000","last_became_peered":"2024-02-12T17:03:50.552762+0000","last_unstale":"2024-02-12T17:03:58.673958+0000","last_undegraded":"2024-02-12T17:03:58.673958+0000","last_fullsized":"2024-02-12T17:03:58.673958+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:32.127593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:32.127593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":22,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686704+0000","last_change":"2024-02-12T17:03:53.944994+0000","last_active":"2024-02-12T17:03:58.686704+0000","last_peered":"2024-02-12T17:03:58.686704+0000","last_clean":"2024-02-12T17:03:58.686704+0000","last_became_active":"2024-02-12T17:03:50.555238+0000","last_became_peered":"2024-02-12T17:03:50.555238+0000","last_unstale":"2024-02-12T17:03:58.686704+0000","last_undegraded":"2024-02-12T17:03:58.686704+0000","last_fullsized":"2024-02-12T17:03:58.686704+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-02-12T17:00:53.269107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.269107+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":66,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.675680+0000","last_change":"2024-02-12T17:02:43.982708+0000","last_active":"2024-02-12T17:03:58.675680+0000","last_peered":"2024-02-12T17:03:58.675680+0000","last_clean":"2024-02-12T17:03:58.675680+0000","last_became_active":"2024-02-12T17:01:16.910812+0000","last_became_peered":"2024-02-12T17:01:16.910812+0000","last_unstale":"2024-02-12T17:03:58.675680+0000","last_undegraded":"2024-02-12T17:03:58.675680+0000","last_fullsized":"2024-02-12T17:03:58.675680+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:43.982623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.982623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":230,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.675732+0000","last_change":"2024-02-12T17:01:16.912921+0000","last_active":"2024-02-12T17:03:58.675732+0000","last_peered":"2024-02-12T17:03:58.675732+0000","last_clean":"2024-02-12T17:03:58.675732+0000","last_became_active":"2024-02-12T17:01:16.911289+0000","last_became_peered":"2024-02-12T17:01:16.911289+0000","last_unstale":"2024-02-12T17:03:58.675732+0000","last_undegraded":"2024-02-12T17:03:58.675732+0000","last_fullsized":"2024-02-12T17:03:58.675732+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.824194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.824194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":24,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686918+0000","last_change":"2024-02-12T17:03:55.514799+0000","last_active":"2024-02-12T17:03:58.686918+0000","last_peered":"2024-02-12T17:03:58.686918+0000","last_clean":"2024-02-12T17:03:58.686918+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:03:58.686918+0000","last_undegraded":"2024-02-12T17:03:58.686918+0000","last_fullsized":"2024-02-12T17:03:58.686918+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:00:53.936664+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.936664+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"352'12026","reported_seq":12092,"reported_epoch":352,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:03.614938+0000","last_change":"2024-02-12T17:03:51.782951+0000","last_active":"2024-02-12T17:04:03.614938+0000","last_peered":"2024-02-12T17:04:03.614938+0000","last_clean":"2024-02-12T17:03:49.505679+0000","last_became_active":"2024-02-12T17:03:51.720907+0000","last_became_peered":"2024-02-12T17:03:51.720907+0000","last_unstale":"2024-02-12T17:04:03.614938+0000","last_undegraded":"2024-02-12T17:04:03.614938+0000","last_fullsized":"2024-02-12T17:04:03.614938+0000","mapping_epoch":348,"log_start":"331'6700","ondisk_log_start":"331'6700","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":5326,"ondisk_log_size":5326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330694656,"num_objects":5046,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5046,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12026,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":5046}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686063+0000","last_change":"2024-02-12T17:03:50.610304+0000","last_active":"2024-02-12T17:03:58.686063+0000","last_peered":"2024-02-12T17:03:58.686063+0000","last_clean":"2024-02-12T17:03:58.686063+0000","last_became_active":"2024-02-12T17:03:50.609176+0000","last_became_peered":"2024-02-12T17:03:50.609176+0000","last_unstale":"2024-02-12T17:03:58.686063+0000","last_undegraded":"2024-02-12T17:03:58.686063+0000","last_fullsized":"2024-02-12T17:03:58.686063+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:28.912872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.912872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":363,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.675628+0000","last_change":"2024-02-12T17:03:21.569463+0000","last_active":"2024-02-12T17:03:58.675628+0000","last_peered":"2024-02-12T17:03:58.675628+0000","last_clean":"2024-02-12T17:03:58.675628+0000","last_became_active":"2024-02-12T17:03:20.776312+0000","last_became_peered":"2024-02-12T17:03:20.776312+0000","last_unstale":"2024-02-12T17:03:58.675628+0000","last_undegraded":"2024-02-12T17:03:58.675628+0000","last_fullsized":"2024-02-12T17:03:58.675628+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:02:17.066331+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.066331+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":26,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687307+0000","last_change":"2024-02-12T17:03:53.226021+0000","last_active":"2024-02-12T17:03:58.687307+0000","last_peered":"2024-02-12T17:03:58.687307+0000","last_clean":"2024-02-12T17:03:58.687307+0000","last_became_active":"2024-02-12T17:03:50.550146+0000","last_became_peered":"2024-02-12T17:03:50.550146+0000","last_unstale":"2024-02-12T17:03:58.687307+0000","last_undegraded":"2024-02-12T17:03:58.687307+0000","last_fullsized":"2024-02-12T17:03:58.687307+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":237,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687508+0000","last_change":"2024-02-12T17:03:20.774660+0000","last_active":"2024-02-12T17:03:58.687508+0000","last_peered":"2024-02-12T17:03:58.687508+0000","last_clean":"2024-02-12T17:03:58.687508+0000","last_became_active":"2024-02-12T17:03:20.773747+0000","last_became_peered":"2024-02-12T17:03:20.773747+0000","last_unstale":"2024-02-12T17:03:58.687508+0000","last_undegraded":"2024-02-12T17:03:58.687508+0000","last_fullsized":"2024-02-12T17:03:58.687508+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:25.815816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.815816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"352'12279","reported_seq":15455,"reported_epoch":352,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:04:03.696979+0000","last_change":"2024-02-12T17:03:51.708272+0000","last_active":"2024-02-12T17:04:03.696979+0000","last_peered":"2024-02-12T17:04:03.696979+0000","last_clean":"2024-02-12T17:03:49.519642+0000","last_became_active":"2024-02-12T17:03:51.663026+0000","last_became_peered":"2024-02-12T17:03:51.663026+0000","last_unstale":"2024-02-12T17:04:03.696979+0000","last_undegraded":"2024-02-12T17:04:03.696979+0000","last_fullsized":"2024-02-12T17:04:03.696979+0000","mapping_epoch":348,"log_start":"336'8927","ondisk_log_start":"336'8927","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3352,"ondisk_log_size":3352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":342032384,"num_objects":5219,"num_object_clones":0,"num_object_copies":15657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3742,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12279,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":101974016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,2,4],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":5219}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":328,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.675634+0000","last_change":"2024-02-12T17:03:50.533429+0000","last_active":"2024-02-12T17:03:58.675634+0000","last_peered":"2024-02-12T17:03:58.675634+0000","last_clean":"2024-02-12T17:03:58.675634+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:03:58.675634+0000","last_undegraded":"2024-02-12T17:03:58.675634+0000","last_fullsized":"2024-02-12T17:03:58.675634+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":161,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.677397+0000","last_change":"2024-02-12T17:02:30.758581+0000","last_active":"2024-02-12T17:03:58.677397+0000","last_peered":"2024-02-12T17:03:58.677397+0000","last_clean":"2024-02-12T17:03:58.677397+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:03:58.677397+0000","last_undegraded":"2024-02-12T17:03:58.677397+0000","last_fullsized":"2024-02-12T17:03:58.677397+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.758520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.758520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":231,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.691362+0000","last_change":"2024-02-12T17:03:22.958122+0000","last_active":"2024-02-12T17:03:58.691362+0000","last_peered":"2024-02-12T17:03:58.691362+0000","last_clean":"2024-02-12T17:03:58.691362+0000","last_became_active":"2024-02-12T17:03:20.774839+0000","last_became_peered":"2024-02-12T17:03:20.774839+0000","last_unstale":"2024-02-12T17:03:58.691362+0000","last_undegraded":"2024-02-12T17:03:58.691362+0000","last_fullsized":"2024-02-12T17:03:58.691362+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"348'12","reported_seq":460,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:01.724465+0000","last_change":"2024-02-12T17:03:51.176776+0000","last_active":"2024-02-12T17:04:01.724465+0000","last_peered":"2024-02-12T17:04:01.724465+0000","last_clean":"2024-02-12T17:04:01.724465+0000","last_became_active":"2024-02-12T17:03:50.577424+0000","last_became_peered":"2024-02-12T17:03:50.577424+0000","last_unstale":"2024-02-12T17:04:01.724465+0000","last_undegraded":"2024-02-12T17:04:01.724465+0000","last_fullsized":"2024-02-12T17:04:01.724465+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"352'12072","reported_seq":12744,"reported_epoch":352,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:03.508437+0000","last_change":"2024-02-12T17:03:51.704941+0000","last_active":"2024-02-12T17:04:03.508437+0000","last_peered":"2024-02-12T17:04:03.508437+0000","last_clean":"2024-02-12T17:03:19.725881+0000","last_became_active":"2024-02-12T17:03:50.651275+0000","last_became_peered":"2024-02-12T17:03:50.651275+0000","last_unstale":"2024-02-12T17:04:03.508437+0000","last_undegraded":"2024-02-12T17:04:03.508437+0000","last_fullsized":"2024-02-12T17:04:03.508437+0000","mapping_epoch":347,"log_start":"334'8392","ondisk_log_start":"334'8392","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3680,"ondisk_log_size":3680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":332005376,"num_objects":5066,"num_object_clones":0,"num_object_copies":15198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5066,"num_objects_unfound":0,"num_objects_dirty":5066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":548416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":19398656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,0],"avail_no_missing":["7","0","4","5"],"object_location_counts":[{"shards":"0,5,7","objects":5066}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686255+0000","last_change":"2024-02-12T17:03:50.554016+0000","last_active":"2024-02-12T17:03:58.686255+0000","last_peered":"2024-02-12T17:03:58.686255+0000","last_clean":"2024-02-12T17:03:58.686255+0000","last_became_active":"2024-02-12T17:03:50.553570+0000","last_became_peered":"2024-02-12T17:03:50.553570+0000","last_unstale":"2024-02-12T17:03:58.686255+0000","last_undegraded":"2024-02-12T17:03:58.686255+0000","last_fullsized":"2024-02-12T17:03:58.686255+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":350,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.691123+0000","last_change":"2024-02-12T17:03:50.549240+0000","last_active":"2024-02-12T17:03:58.691123+0000","last_peered":"2024-02-12T17:03:58.691123+0000","last_clean":"2024-02-12T17:03:58.691123+0000","last_became_active":"2024-02-12T17:03:50.548926+0000","last_became_peered":"2024-02-12T17:03:50.548926+0000","last_unstale":"2024-02-12T17:03:58.691123+0000","last_undegraded":"2024-02-12T17:03:58.691123+0000","last_fullsized":"2024-02-12T17:03:58.691123+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:42.892563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.892563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"348'55","reported_seq":40,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:01.734153+0000","last_change":"2024-02-12T17:03:58.596193+0000","last_active":"2024-02-12T17:04:01.734153+0000","last_peered":"2024-02-12T17:04:01.734153+0000","last_clean":"2024-02-12T17:04:01.734153+0000","last_became_active":"2024-02-12T17:03:50.557154+0000","last_became_peered":"2024-02-12T17:03:50.557154+0000","last_unstale":"2024-02-12T17:04:01.734153+0000","last_undegraded":"2024-02-12T17:04:01.734153+0000","last_fullsized":"2024-02-12T17:04:01.734153+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":259,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679992+0000","last_change":"2024-02-12T17:03:50.566197+0000","last_active":"2024-02-12T17:03:58.679992+0000","last_peered":"2024-02-12T17:03:58.679992+0000","last_clean":"2024-02-12T17:03:58.679992+0000","last_became_active":"2024-02-12T17:03:50.562747+0000","last_became_peered":"2024-02-12T17:03:50.562747+0000","last_unstale":"2024-02-12T17:03:58.679992+0000","last_undegraded":"2024-02-12T17:03:58.679992+0000","last_fullsized":"2024-02-12T17:03:58.679992+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"352'12303","reported_seq":12630,"reported_epoch":352,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:04:03.677878+0000","last_change":"2024-02-12T17:03:52.392320+0000","last_active":"2024-02-12T17:04:03.677878+0000","last_peered":"2024-02-12T17:04:03.677878+0000","last_clean":"2024-02-12T17:03:49.453764+0000","last_became_active":"2024-02-12T17:03:51.655919+0000","last_became_peered":"2024-02-12T17:03:51.655919+0000","last_unstale":"2024-02-12T17:04:03.677878+0000","last_undegraded":"2024-02-12T17:04:03.677878+0000","last_fullsized":"2024-02-12T17:04:03.677878+0000","mapping_epoch":348,"log_start":"336'8954","ondisk_log_start":"336'8954","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3349,"ondisk_log_size":3349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":332857344,"num_objects":5079,"num_object_clones":0,"num_object_copies":15237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4960,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":556224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8585216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":5079}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":339,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.680122+0000","last_change":"2024-02-12T17:03:50.566072+0000","last_active":"2024-02-12T17:03:58.680122+0000","last_peered":"2024-02-12T17:03:58.680122+0000","last_clean":"2024-02-12T17:03:58.680122+0000","last_became_active":"2024-02-12T17:03:50.564700+0000","last_became_peered":"2024-02-12T17:03:50.564700+0000","last_unstale":"2024-02-12T17:03:58.680122+0000","last_undegraded":"2024-02-12T17:03:58.680122+0000","last_fullsized":"2024-02-12T17:03:58.680122+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:04.085827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:04.085827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":249,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:02.454465+0000","last_change":"2024-02-12T17:01:16.913133+0000","last_active":"2024-02-12T17:04:02.454465+0000","last_peered":"2024-02-12T17:04:02.454465+0000","last_clean":"2024-02-12T17:04:02.454465+0000","last_became_active":"2024-02-12T17:01:16.912041+0000","last_became_peered":"2024-02-12T17:01:16.912041+0000","last_unstale":"2024-02-12T17:04:02.454465+0000","last_undegraded":"2024-02-12T17:04:02.454465+0000","last_fullsized":"2024-02-12T17:04:02.454465+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:00:52.838852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:52.838852+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":173,"num_read_kb":168,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686659+0000","last_change":"2024-02-12T17:03:50.533363+0000","last_active":"2024-02-12T17:03:58.686659+0000","last_peered":"2024-02-12T17:03:58.686659+0000","last_clean":"2024-02-12T17:03:58.686659+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:03:58.686659+0000","last_undegraded":"2024-02-12T17:03:58.686659+0000","last_fullsized":"2024-02-12T17:03:58.686659+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:38.142602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:38.142602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":284,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.692389+0000","last_change":"2024-02-12T17:03:51.175255+0000","last_active":"2024-02-12T17:03:58.692389+0000","last_peered":"2024-02-12T17:03:58.692389+0000","last_clean":"2024-02-12T17:03:58.692389+0000","last_became_active":"2024-02-12T17:03:50.573814+0000","last_became_peered":"2024-02-12T17:03:50.573814+0000","last_unstale":"2024-02-12T17:03:58.692389+0000","last_undegraded":"2024-02-12T17:03:58.692389+0000","last_fullsized":"2024-02-12T17:03:58.692389+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":266,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.688317+0000","last_change":"2024-02-12T17:01:16.909653+0000","last_active":"2024-02-12T17:03:58.688317+0000","last_peered":"2024-02-12T17:03:58.688317+0000","last_clean":"2024-02-12T17:03:58.688317+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:03:58.688317+0000","last_undegraded":"2024-02-12T17:03:58.688317+0000","last_fullsized":"2024-02-12T17:03:58.688317+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"352'12142","reported_seq":12206,"reported_epoch":352,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:03.612859+0000","last_change":"2024-02-12T17:03:51.712372+0000","last_active":"2024-02-12T17:04:03.612859+0000","last_peered":"2024-02-12T17:04:03.612859+0000","last_clean":"2024-02-12T17:03:49.506422+0000","last_became_active":"2024-02-12T17:03:51.676105+0000","last_became_peered":"2024-02-12T17:03:51.676105+0000","last_unstale":"2024-02-12T17:04:03.612859+0000","last_undegraded":"2024-02-12T17:04:03.612859+0000","last_fullsized":"2024-02-12T17:04:03.612859+0000","mapping_epoch":348,"log_start":"331'6800","ondisk_log_start":"331'6800","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":5342,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330170368,"num_objects":5038,"num_object_clones":0,"num_object_copies":15114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5038,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":549760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":5038}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":339,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.692691+0000","last_change":"2024-02-12T17:03:50.575145+0000","last_active":"2024-02-12T17:03:58.692691+0000","last_peered":"2024-02-12T17:03:58.692691+0000","last_clean":"2024-02-12T17:03:58.692691+0000","last_became_active":"2024-02-12T17:03:50.574370+0000","last_became_peered":"2024-02-12T17:03:50.574370+0000","last_unstale":"2024-02-12T17:03:58.692691+0000","last_undegraded":"2024-02-12T17:03:58.692691+0000","last_fullsized":"2024-02-12T17:03:58.692691+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:57.950284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:00:57.950284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685453+0000","last_change":"2024-02-12T17:03:50.561433+0000","last_active":"2024-02-12T17:03:58.685453+0000","last_peered":"2024-02-12T17:03:58.685453+0000","last_clean":"2024-02-12T17:03:58.685453+0000","last_became_active":"2024-02-12T17:03:50.558953+0000","last_became_peered":"2024-02-12T17:03:50.558953+0000","last_unstale":"2024-02-12T17:03:58.685453+0000","last_undegraded":"2024-02-12T17:03:58.685453+0000","last_fullsized":"2024-02-12T17:03:58.685453+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":318,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687727+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:03:58.687727+0000","last_peered":"2024-02-12T17:03:58.687727+0000","last_clean":"2024-02-12T17:03:58.687727+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:03:58.687727+0000","last_undegraded":"2024-02-12T17:03:58.687727+0000","last_fullsized":"2024-02-12T17:03:58.687727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":342,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678650+0000","last_change":"2024-02-12T17:03:20.775585+0000","last_active":"2024-02-12T17:03:58.678650+0000","last_peered":"2024-02-12T17:03:58.678650+0000","last_clean":"2024-02-12T17:03:58.678650+0000","last_became_active":"2024-02-12T17:03:20.774779+0000","last_became_peered":"2024-02-12T17:03:20.774779+0000","last_unstale":"2024-02-12T17:03:58.678650+0000","last_undegraded":"2024-02-12T17:03:58.678650+0000","last_fullsized":"2024-02-12T17:03:58.678650+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:25.870916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:01:25.870916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":229,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.676993+0000","last_change":"2024-02-12T17:03:51.169772+0000","last_active":"2024-02-12T17:03:58.676993+0000","last_peered":"2024-02-12T17:03:58.676993+0000","last_clean":"2024-02-12T17:03:58.676993+0000","last_became_active":"2024-02-12T17:03:50.555602+0000","last_became_peered":"2024-02-12T17:03:50.555602+0000","last_unstale":"2024-02-12T17:03:58.676993+0000","last_undegraded":"2024-02-12T17:03:58.676993+0000","last_fullsized":"2024-02-12T17:03:58.676993+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'26","last_scrub_stamp":"2024-02-12T17:01:43.163724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:43.163724+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":317,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678812+0000","last_change":"2024-02-12T17:03:15.796227+0000","last_active":"2024-02-12T17:03:58.678812+0000","last_peered":"2024-02-12T17:03:58.678812+0000","last_clean":"2024-02-12T17:03:58.678812+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:03:58.678812+0000","last_undegraded":"2024-02-12T17:03:58.678812+0000","last_fullsized":"2024-02-12T17:03:58.678812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"352'12103","reported_seq":12783,"reported_epoch":352,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:03.662505+0000","last_change":"2024-02-12T17:03:50.706566+0000","last_active":"2024-02-12T17:04:03.662505+0000","last_peered":"2024-02-12T17:04:03.662505+0000","last_clean":"2024-02-12T17:03:19.701993+0000","last_became_active":"2024-02-12T17:03:50.639685+0000","last_became_peered":"2024-02-12T17:03:50.639685+0000","last_unstale":"2024-02-12T17:04:03.662505+0000","last_undegraded":"2024-02-12T17:04:03.662505+0000","last_fullsized":"2024-02-12T17:04:03.662505+0000","mapping_epoch":347,"log_start":"336'8767","ondisk_log_start":"336'8767","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3336,"ondisk_log_size":3336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":334561280,"num_objects":5105,"num_object_clones":0,"num_object_copies":15315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5105,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":550656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":19791872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":5105}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":267,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.688047+0000","last_change":"2024-02-12T17:03:50.563771+0000","last_active":"2024-02-12T17:03:58.688047+0000","last_peered":"2024-02-12T17:03:58.688047+0000","last_clean":"2024-02-12T17:03:58.688047+0000","last_became_active":"2024-02-12T17:03:50.563433+0000","last_became_peered":"2024-02-12T17:03:50.563433+0000","last_unstale":"2024-02-12T17:03:58.688047+0000","last_undegraded":"2024-02-12T17:03:58.688047+0000","last_fullsized":"2024-02-12T17:03:58.688047+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":129,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.673998+0000","last_change":"2024-02-12T17:03:50.558523+0000","last_active":"2024-02-12T17:03:58.673998+0000","last_peered":"2024-02-12T17:03:58.673998+0000","last_clean":"2024-02-12T17:03:58.673998+0000","last_became_active":"2024-02-12T17:03:50.558112+0000","last_became_peered":"2024-02-12T17:03:50.558112+0000","last_unstale":"2024-02-12T17:03:58.673998+0000","last_undegraded":"2024-02-12T17:03:58.673998+0000","last_fullsized":"2024-02-12T17:03:58.673998+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.941776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":348,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678493+0000","last_change":"2024-02-12T17:03:50.776630+0000","last_active":"2024-02-12T17:03:58.678493+0000","last_peered":"2024-02-12T17:03:58.678493+0000","last_clean":"2024-02-12T17:03:58.678493+0000","last_became_active":"2024-02-12T17:03:50.564881+0000","last_became_peered":"2024-02-12T17:03:50.564881+0000","last_unstale":"2024-02-12T17:03:58.678493+0000","last_undegraded":"2024-02-12T17:03:58.678493+0000","last_fullsized":"2024-02-12T17:03:58.678493+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":334,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.681481+0000","last_change":"2024-02-12T17:03:24.771579+0000","last_active":"2024-02-12T17:03:58.681481+0000","last_peered":"2024-02-12T17:03:58.681481+0000","last_clean":"2024-02-12T17:03:58.681481+0000","last_became_active":"2024-02-12T17:03:20.780926+0000","last_became_peered":"2024-02-12T17:03:20.780926+0000","last_unstale":"2024-02-12T17:03:58.681481+0000","last_undegraded":"2024-02-12T17:03:58.681481+0000","last_fullsized":"2024-02-12T17:03:58.681481+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":341,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.692621+0000","last_change":"2024-02-12T17:02:33.737620+0000","last_active":"2024-02-12T17:03:58.692621+0000","last_peered":"2024-02-12T17:03:58.692621+0000","last_clean":"2024-02-12T17:03:58.692621+0000","last_became_active":"2024-02-12T16:58:14.391221+0000","last_became_peered":"2024-02-12T16:58:14.391221+0000","last_unstale":"2024-02-12T17:03:58.692621+0000","last_undegraded":"2024-02-12T17:03:58.692621+0000","last_fullsized":"2024-02-12T17:03:58.692621+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"352'12305","reported_seq":13244,"reported_epoch":352,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:04:03.593754+0000","last_change":"2024-02-12T17:03:52.285317+0000","last_active":"2024-02-12T17:04:03.593754+0000","last_peered":"2024-02-12T17:04:03.593754+0000","last_clean":"2024-02-12T17:03:19.726721+0000","last_became_active":"2024-02-12T17:03:21.907545+0000","last_became_peered":"2024-02-12T17:03:21.907545+0000","last_unstale":"2024-02-12T17:04:03.593754+0000","last_undegraded":"2024-02-12T17:04:03.593754+0000","last_fullsized":"2024-02-12T17:04:03.593754+0000","mapping_epoch":341,"log_start":"334'8593","ondisk_log_start":"334'8593","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3712,"ondisk_log_size":3712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":339410944,"num_objects":5179,"num_object_clones":0,"num_object_copies":15537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4812,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12305,"num_write_kb":559488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":28442624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,0],"avail_no_missing":["7","0","4"],"object_location_counts":[{"shards":"0,4,7","objects":5179}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":78,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679996+0000","last_change":"2024-02-12T17:03:20.779625+0000","last_active":"2024-02-12T17:03:58.679996+0000","last_peered":"2024-02-12T17:03:58.679996+0000","last_clean":"2024-02-12T17:03:58.679996+0000","last_became_active":"2024-02-12T17:03:20.779126+0000","last_became_peered":"2024-02-12T17:03:20.779126+0000","last_unstale":"2024-02-12T17:03:58.679996+0000","last_undegraded":"2024-02-12T17:03:58.679996+0000","last_fullsized":"2024-02-12T17:03:58.679996+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:48.617913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:02:48.617913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":330,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.680202+0000","last_change":"2024-02-12T17:03:20.776794+0000","last_active":"2024-02-12T17:03:58.680202+0000","last_peered":"2024-02-12T17:03:58.680202+0000","last_clean":"2024-02-12T17:03:58.680202+0000","last_became_active":"2024-02-12T17:03:20.771211+0000","last_became_peered":"2024-02-12T17:03:20.771211+0000","last_unstale":"2024-02-12T17:03:58.680202+0000","last_undegraded":"2024-02-12T17:03:58.680202+0000","last_fullsized":"2024-02-12T17:03:58.680202+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.212949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.212949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":330,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687815+0000","last_change":"2024-02-12T17:03:21.658125+0000","last_active":"2024-02-12T17:03:58.687815+0000","last_peered":"2024-02-12T17:03:58.687815+0000","last_clean":"2024-02-12T17:03:58.687815+0000","last_became_active":"2024-02-12T17:03:20.769223+0000","last_became_peered":"2024-02-12T17:03:20.769223+0000","last_unstale":"2024-02-12T17:03:58.687815+0000","last_undegraded":"2024-02-12T17:03:58.687815+0000","last_fullsized":"2024-02-12T17:03:58.687815+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:02:44.688402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:44.688402+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":111,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.676576+0000","last_change":"2024-02-12T17:03:15.810501+0000","last_active":"2024-02-12T17:03:58.676576+0000","last_peered":"2024-02-12T17:03:58.676576+0000","last_clean":"2024-02-12T17:03:58.676576+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:03:58.676576+0000","last_undegraded":"2024-02-12T17:03:58.676576+0000","last_fullsized":"2024-02-12T17:03:58.676576+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"352'11859","reported_seq":11916,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:03.535427+0000","last_change":"2024-02-12T17:01:13.886816+0000","last_active":"2024-02-12T17:04:03.535427+0000","last_peered":"2024-02-12T17:04:03.535427+0000","last_clean":"2024-02-12T17:04:03.535427+0000","last_became_active":"2024-02-12T17:01:13.886430+0000","last_became_peered":"2024-02-12T17:01:13.886430+0000","last_unstale":"2024-02-12T17:04:03.535427+0000","last_undegraded":"2024-02-12T17:04:03.535427+0000","last_fullsized":"2024-02-12T17:04:03.535427+0000","mapping_epoch":304,"log_start":"334'8239","ondisk_log_start":"334'8239","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3620,"ondisk_log_size":3620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":323616792,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11859,"num_write_kb":537473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685499+0000","last_change":"2024-02-12T17:03:50.550528+0000","last_active":"2024-02-12T17:03:58.685499+0000","last_peered":"2024-02-12T17:03:58.685499+0000","last_clean":"2024-02-12T17:03:58.685499+0000","last_became_active":"2024-02-12T17:03:50.550166+0000","last_became_peered":"2024-02-12T17:03:50.550166+0000","last_unstale":"2024-02-12T17:03:58.685499+0000","last_undegraded":"2024-02-12T17:03:58.685499+0000","last_fullsized":"2024-02-12T17:03:58.685499+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.676284+0000","last_change":"2024-02-12T17:03:50.552622+0000","last_active":"2024-02-12T17:03:58.676284+0000","last_peered":"2024-02-12T17:03:58.676284+0000","last_clean":"2024-02-12T17:03:58.676284+0000","last_became_active":"2024-02-12T17:03:50.552066+0000","last_became_peered":"2024-02-12T17:03:50.552066+0000","last_unstale":"2024-02-12T17:03:58.676284+0000","last_undegraded":"2024-02-12T17:03:58.676284+0000","last_fullsized":"2024-02-12T17:03:58.676284+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":32,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685034+0000","last_change":"2024-02-12T17:03:54.620306+0000","last_active":"2024-02-12T17:03:58.685034+0000","last_peered":"2024-02-12T17:03:58.685034+0000","last_clean":"2024-02-12T17:03:58.685034+0000","last_became_active":"2024-02-12T17:03:50.556763+0000","last_became_peered":"2024-02-12T17:03:50.556763+0000","last_unstale":"2024-02-12T17:03:58.685034+0000","last_undegraded":"2024-02-12T17:03:58.685034+0000","last_fullsized":"2024-02-12T17:03:58.685034+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":352,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.674072+0000","last_change":"2024-02-12T17:03:50.580875+0000","last_active":"2024-02-12T17:03:58.674072+0000","last_peered":"2024-02-12T17:03:58.674072+0000","last_clean":"2024-02-12T17:03:58.674072+0000","last_became_active":"2024-02-12T17:03:50.580321+0000","last_became_peered":"2024-02-12T17:03:50.580321+0000","last_unstale":"2024-02-12T17:03:58.674072+0000","last_undegraded":"2024-02-12T17:03:58.674072+0000","last_fullsized":"2024-02-12T17:03:58.674072+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"352'12260","reported_seq":13265,"reported_epoch":352,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:04:03.538680+0000","last_change":"2024-02-12T17:03:51.715067+0000","last_active":"2024-02-12T17:04:03.538680+0000","last_peered":"2024-02-12T17:04:03.538680+0000","last_clean":"2024-02-12T17:03:19.745457+0000","last_became_active":"2024-02-12T17:03:50.621163+0000","last_became_peered":"2024-02-12T17:03:50.621163+0000","last_unstale":"2024-02-12T17:04:03.538680+0000","last_undegraded":"2024-02-12T17:04:03.538680+0000","last_fullsized":"2024-02-12T17:04:03.538680+0000","mapping_epoch":347,"log_start":"333'8317","ondisk_log_start":"333'8317","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'5831","last_scrub_stamp":"2024-02-12T17:02:24.719414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.719414+0000","log_size":3943,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":341966848,"num_objects":5218,"num_object_clones":0,"num_object_copies":15654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5091,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12260,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":29687808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[4,7,0],"avail_no_missing":["4","0","5","7"],"object_location_counts":[{"shards":"0,4,7","objects":5218}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":342,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679592+0000","last_change":"2024-02-12T17:03:20.882647+0000","last_active":"2024-02-12T17:03:58.679592+0000","last_peered":"2024-02-12T17:03:58.679592+0000","last_clean":"2024-02-12T17:03:58.679592+0000","last_became_active":"2024-02-12T17:03:20.766117+0000","last_became_peered":"2024-02-12T17:03:20.766117+0000","last_unstale":"2024-02-12T17:03:58.679592+0000","last_undegraded":"2024-02-12T17:03:58.679592+0000","last_fullsized":"2024-02-12T17:03:58.679592+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:03:17.811656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.811656+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":120,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679534+0000","last_change":"2024-02-12T17:03:50.563493+0000","last_active":"2024-02-12T17:03:58.679534+0000","last_peered":"2024-02-12T17:03:58.679534+0000","last_clean":"2024-02-12T17:03:58.679534+0000","last_became_active":"2024-02-12T17:03:50.562556+0000","last_became_peered":"2024-02-12T17:03:50.562556+0000","last_unstale":"2024-02-12T17:03:58.679534+0000","last_undegraded":"2024-02-12T17:03:58.679534+0000","last_fullsized":"2024-02-12T17:03:58.679534+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.562425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.562425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":380,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687001+0000","last_change":"2024-02-12T17:03:23.141960+0000","last_active":"2024-02-12T17:03:58.687001+0000","last_peered":"2024-02-12T17:03:58.687001+0000","last_clean":"2024-02-12T17:03:58.687001+0000","last_became_active":"2024-02-12T17:03:20.776619+0000","last_became_peered":"2024-02-12T17:03:20.776619+0000","last_unstale":"2024-02-12T17:03:58.687001+0000","last_undegraded":"2024-02-12T17:03:58.687001+0000","last_fullsized":"2024-02-12T17:03:58.687001+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"348'13","reported_seq":22,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:01.731906+0000","last_change":"2024-02-12T17:03:52.637102+0000","last_active":"2024-02-12T17:04:01.731906+0000","last_peered":"2024-02-12T17:04:01.731906+0000","last_clean":"2024-02-12T17:04:01.731906+0000","last_became_active":"2024-02-12T17:03:50.551135+0000","last_became_peered":"2024-02-12T17:03:50.551135+0000","last_unstale":"2024-02-12T17:04:01.731906+0000","last_undegraded":"2024-02-12T17:04:01.731906+0000","last_fullsized":"2024-02-12T17:04:01.731906+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"352'11885","reported_seq":11971,"reported_epoch":352,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:03.592334+0000","last_change":"2024-02-12T17:03:52.638188+0000","last_active":"2024-02-12T17:04:03.592334+0000","last_peered":"2024-02-12T17:04:03.592334+0000","last_clean":"2024-02-12T17:03:19.746645+0000","last_became_active":"2024-02-12T17:03:50.650695+0000","last_became_peered":"2024-02-12T17:03:50.650695+0000","last_unstale":"2024-02-12T17:04:03.592334+0000","last_undegraded":"2024-02-12T17:04:03.592334+0000","last_fullsized":"2024-02-12T17:04:03.592334+0000","mapping_epoch":347,"log_start":"334'8253","ondisk_log_start":"334'8253","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3632,"ondisk_log_size":3632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":329973760,"num_objects":5035,"num_object_clones":0,"num_object_copies":15105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5035,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11885,"num_write_kb":541440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,5,0],"avail_no_missing":["7","0","2","5"],"object_location_counts":[{"shards":"0,5,7","objects":5035}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686018+0000","last_change":"2024-02-12T17:03:50.559588+0000","last_active":"2024-02-12T17:03:58.686018+0000","last_peered":"2024-02-12T17:03:58.686018+0000","last_clean":"2024-02-12T17:03:58.686018+0000","last_became_active":"2024-02-12T17:03:50.559277+0000","last_became_peered":"2024-02-12T17:03:50.559277+0000","last_unstale":"2024-02-12T17:03:58.686018+0000","last_undegraded":"2024-02-12T17:03:58.686018+0000","last_fullsized":"2024-02-12T17:03:58.686018+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.811153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.811153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"261'14","reported_seq":437,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:01.749147+0000","last_change":"2024-02-12T16:59:12.225158+0000","last_active":"2024-02-12T17:04:01.749147+0000","last_peered":"2024-02-12T17:04:01.749147+0000","last_clean":"2024-02-12T17:04:01.749147+0000","last_became_active":"2024-02-12T16:59:12.224830+0000","last_became_peered":"2024-02-12T16:59:12.224830+0000","last_unstale":"2024-02-12T17:04:01.749147+0000","last_undegraded":"2024-02-12T17:04:01.749147+0000","last_fullsized":"2024-02-12T17:04:01.749147+0000","mapping_epoch":260,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687214+0000","last_change":"2024-02-12T17:03:20.774823+0000","last_active":"2024-02-12T17:03:58.687214+0000","last_peered":"2024-02-12T17:03:58.687214+0000","last_clean":"2024-02-12T17:03:58.687214+0000","last_became_active":"2024-02-12T17:03:20.774127+0000","last_became_peered":"2024-02-12T17:03:20.774127+0000","last_unstale":"2024-02-12T17:03:58.687214+0000","last_undegraded":"2024-02-12T17:03:58.687214+0000","last_fullsized":"2024-02-12T17:03:58.687214+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":26,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686728+0000","last_change":"2024-02-12T17:03:52.630637+0000","last_active":"2024-02-12T17:03:58.686728+0000","last_peered":"2024-02-12T17:03:58.686728+0000","last_clean":"2024-02-12T17:03:58.686728+0000","last_became_active":"2024-02-12T17:03:50.549627+0000","last_became_peered":"2024-02-12T17:03:50.549627+0000","last_unstale":"2024-02-12T17:03:58.686728+0000","last_undegraded":"2024-02-12T17:03:58.686728+0000","last_fullsized":"2024-02-12T17:03:58.686728+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"352'12000","reported_seq":12653,"reported_epoch":352,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:03.596863+0000","last_change":"2024-02-12T17:03:50.662715+0000","last_active":"2024-02-12T17:04:03.596863+0000","last_peered":"2024-02-12T17:04:03.596863+0000","last_clean":"2024-02-12T17:03:19.738005+0000","last_became_active":"2024-02-12T17:03:50.628646+0000","last_became_peered":"2024-02-12T17:03:50.628646+0000","last_unstale":"2024-02-12T17:04:03.596863+0000","last_undegraded":"2024-02-12T17:04:03.596863+0000","last_fullsized":"2024-02-12T17:04:03.596863+0000","mapping_epoch":347,"log_start":"334'8283","ondisk_log_start":"334'8283","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3717,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330956800,"num_objects":5050,"num_object_clones":0,"num_object_copies":15150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5050,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12000,"num_write_kb":545600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":18743296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[7,2,0],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":5050}],"blocked_by":[],"up_primary":3,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":335,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.689861+0000","last_change":"2024-02-12T17:03:20.775021+0000","last_active":"2024-02-12T17:03:58.689861+0000","last_peered":"2024-02-12T17:03:58.689861+0000","last_clean":"2024-02-12T17:03:58.689861+0000","last_became_active":"2024-02-12T17:03:20.773536+0000","last_became_peered":"2024-02-12T17:03:20.773536+0000","last_unstale":"2024-02-12T17:03:58.689861+0000","last_undegraded":"2024-02-12T17:03:58.689861+0000","last_fullsized":"2024-02-12T17:03:58.689861+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"208'12","reported_seq":487,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:01.748404+0000","last_change":"2024-02-12T17:02:20.866677+0000","last_active":"2024-02-12T17:04:01.748404+0000","last_peered":"2024-02-12T17:04:01.748404+0000","last_clean":"2024-02-12T17:04:01.748404+0000","last_became_active":"2024-02-12T16:57:46.177517+0000","last_became_peered":"2024-02-12T16:57:46.177517+0000","last_unstale":"2024-02-12T17:04:01.748404+0000","last_undegraded":"2024-02-12T17:04:01.748404+0000","last_fullsized":"2024-02-12T17:04:01.748404+0000","mapping_epoch":97,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-12T17:02:20.866591+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:02:20.866591+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685451+0000","last_change":"2024-02-12T17:03:50.550425+0000","last_active":"2024-02-12T17:03:58.685451+0000","last_peered":"2024-02-12T17:03:58.685451+0000","last_clean":"2024-02-12T17:03:58.685451+0000","last_became_active":"2024-02-12T17:03:50.549672+0000","last_became_peered":"2024-02-12T17:03:50.549672+0000","last_unstale":"2024-02-12T17:03:58.685451+0000","last_undegraded":"2024-02-12T17:03:58.685451+0000","last_fullsized":"2024-02-12T17:03:58.685451+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.863530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.863530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":30,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686260+0000","last_change":"2024-02-12T17:03:54.460681+0000","last_active":"2024-02-12T17:03:58.686260+0000","last_peered":"2024-02-12T17:03:58.686260+0000","last_clean":"2024-02-12T17:03:58.686260+0000","last_became_active":"2024-02-12T17:03:50.555226+0000","last_became_peered":"2024-02-12T17:03:50.555226+0000","last_unstale":"2024-02-12T17:03:58.686260+0000","last_undegraded":"2024-02-12T17:03:58.686260+0000","last_fullsized":"2024-02-12T17:03:58.686260+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:01:41.694875+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:01:41.694875+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"352'12090","reported_seq":12155,"reported_epoch":352,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:03.534882+0000","last_change":"2024-02-12T17:03:51.698804+0000","last_active":"2024-02-12T17:04:03.534882+0000","last_peered":"2024-02-12T17:04:03.534882+0000","last_clean":"2024-02-12T17:03:49.455800+0000","last_became_active":"2024-02-12T17:03:51.638628+0000","last_became_peered":"2024-02-12T17:03:51.638628+0000","last_unstale":"2024-02-12T17:04:03.534882+0000","last_undegraded":"2024-02-12T17:04:03.534882+0000","last_fullsized":"2024-02-12T17:04:03.534882+0000","mapping_epoch":348,"log_start":"334'8435","ondisk_log_start":"334'8435","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3655,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":331350016,"num_objects":5056,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5056,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12090,"num_write_kb":548672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[2,7,4],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":5056}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":62,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.688038+0000","last_change":"2024-02-12T17:01:16.909471+0000","last_active":"2024-02-12T17:03:58.688038+0000","last_peered":"2024-02-12T17:03:58.688038+0000","last_clean":"2024-02-12T17:03:58.688038+0000","last_became_active":"2024-02-12T17:01:16.909039+0000","last_became_peered":"2024-02-12T17:01:16.909039+0000","last_unstale":"2024-02-12T17:03:58.688038+0000","last_undegraded":"2024-02-12T17:03:58.688038+0000","last_fullsized":"2024-02-12T17:03:58.688038+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"308'11","reported_seq":114,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:01.717913+0000","last_change":"2024-02-12T17:02:31.547056+0000","last_active":"2024-02-12T17:04:01.717913+0000","last_peered":"2024-02-12T17:04:01.717913+0000","last_clean":"2024-02-12T17:04:01.717913+0000","last_became_active":"2024-02-12T17:01:16.912683+0000","last_became_peered":"2024-02-12T17:01:16.912683+0000","last_unstale":"2024-02-12T17:04:01.717913+0000","last_undegraded":"2024-02-12T17:04:01.717913+0000","last_fullsized":"2024-02-12T17:04:01.717913+0000","mapping_epoch":307,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'11","last_scrub_stamp":"2024-02-12T17:02:31.546965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:31.546965+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.685543+0000","last_change":"2024-02-12T17:03:50.589204+0000","last_active":"2024-02-12T17:03:58.685543+0000","last_peered":"2024-02-12T17:03:58.685543+0000","last_clean":"2024-02-12T17:03:58.685543+0000","last_became_active":"2024-02-12T17:03:50.588844+0000","last_became_peered":"2024-02-12T17:03:50.588844+0000","last_unstale":"2024-02-12T17:03:58.685543+0000","last_undegraded":"2024-02-12T17:03:58.685543+0000","last_fullsized":"2024-02-12T17:03:58.685543+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.888942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.888942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":26,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686165+0000","last_change":"2024-02-12T17:03:58.166479+0000","last_active":"2024-02-12T17:03:58.686165+0000","last_peered":"2024-02-12T17:03:58.686165+0000","last_clean":"2024-02-12T17:03:58.686165+0000","last_became_active":"2024-02-12T17:03:50.567802+0000","last_became_peered":"2024-02-12T17:03:50.567802+0000","last_unstale":"2024-02-12T17:03:58.686165+0000","last_undegraded":"2024-02-12T17:03:58.686165+0000","last_fullsized":"2024-02-12T17:03:58.686165+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"352'12104","reported_seq":12160,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:03.624067+0000","last_change":"2024-02-12T17:01:16.913292+0000","last_active":"2024-02-12T17:04:03.624067+0000","last_peered":"2024-02-12T17:04:03.624067+0000","last_clean":"2024-02-12T17:04:03.624067+0000","last_became_active":"2024-02-12T17:01:16.912605+0000","last_became_peered":"2024-02-12T17:01:16.912605+0000","last_unstale":"2024-02-12T17:04:03.624067+0000","last_undegraded":"2024-02-12T17:04:03.624067+0000","last_fullsized":"2024-02-12T17:04:03.624067+0000","mapping_epoch":307,"log_start":"331'6800","ondisk_log_start":"331'6800","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":5304,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330301440,"num_objects":5040,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12104,"num_write_kb":548608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686690+0000","last_change":"2024-02-12T17:03:50.550752+0000","last_active":"2024-02-12T17:03:58.686690+0000","last_peered":"2024-02-12T17:03:58.686690+0000","last_clean":"2024-02-12T17:03:58.686690+0000","last_became_active":"2024-02-12T17:03:50.550310+0000","last_became_peered":"2024-02-12T17:03:50.550310+0000","last_unstale":"2024-02-12T17:03:58.686690+0000","last_undegraded":"2024-02-12T17:03:58.686690+0000","last_fullsized":"2024-02-12T17:03:58.686690+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":193,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.675760+0000","last_change":"2024-02-12T17:03:19.084640+0000","last_active":"2024-02-12T17:03:58.675760+0000","last_peered":"2024-02-12T17:03:58.675760+0000","last_clean":"2024-02-12T17:03:58.675760+0000","last_became_active":"2024-02-12T16:59:09.247447+0000","last_became_peered":"2024-02-12T16:59:09.247447+0000","last_unstale":"2024-02-12T17:03:58.675760+0000","last_undegraded":"2024-02-12T17:03:58.675760+0000","last_fullsized":"2024-02-12T17:03:58.675760+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:19.084549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.084549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":321,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.679287+0000","last_change":"2024-02-12T17:01:27.859045+0000","last_active":"2024-02-12T17:03:58.679287+0000","last_peered":"2024-02-12T17:03:58.679287+0000","last_clean":"2024-02-12T17:03:58.679287+0000","last_became_active":"2024-02-12T16:49:39.103862+0000","last_became_peered":"2024-02-12T16:49:39.103862+0000","last_unstale":"2024-02-12T17:03:58.679287+0000","last_undegraded":"2024-02-12T17:03:58.679287+0000","last_fullsized":"2024-02-12T17:03:58.679287+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.858955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.858955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":39,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.689308+0000","last_change":"2024-02-12T17:03:21.667636+0000","last_active":"2024-02-12T17:03:58.689308+0000","last_peered":"2024-02-12T17:03:58.689308+0000","last_clean":"2024-02-12T17:03:58.689308+0000","last_became_active":"2024-02-12T17:03:20.773021+0000","last_became_peered":"2024-02-12T17:03:20.773021+0000","last_unstale":"2024-02-12T17:03:58.689308+0000","last_undegraded":"2024-02-12T17:03:58.689308+0000","last_fullsized":"2024-02-12T17:03:58.689308+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:01:35.160145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:35.160145+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"352'11869","reported_seq":11935,"reported_epoch":352,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:03.650988+0000","last_change":"2024-02-12T17:03:51.694169+0000","last_active":"2024-02-12T17:04:03.650988+0000","last_peered":"2024-02-12T17:04:03.650988+0000","last_clean":"2024-02-12T17:03:49.522755+0000","last_became_active":"2024-02-12T17:03:51.663049+0000","last_became_peered":"2024-02-12T17:03:51.663049+0000","last_unstale":"2024-02-12T17:04:03.650988+0000","last_undegraded":"2024-02-12T17:04:03.650988+0000","last_fullsized":"2024-02-12T17:04:03.650988+0000","mapping_epoch":348,"log_start":"335'8476","ondisk_log_start":"335'8476","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3393,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":326172672,"num_objects":4977,"num_object_clones":0,"num_object_copies":14931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4977,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11869,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":4977}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":346,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678401+0000","last_change":"2024-02-12T17:03:50.565273+0000","last_active":"2024-02-12T17:03:58.678401+0000","last_peered":"2024-02-12T17:03:58.678401+0000","last_clean":"2024-02-12T17:03:58.678401+0000","last_became_active":"2024-02-12T17:03:50.564942+0000","last_became_peered":"2024-02-12T17:03:50.564942+0000","last_unstale":"2024-02-12T17:03:58.678401+0000","last_undegraded":"2024-02-12T17:03:58.678401+0000","last_fullsized":"2024-02-12T17:03:58.678401+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":257,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.675829+0000","last_change":"2024-02-12T17:03:28.770200+0000","last_active":"2024-02-12T17:03:58.675829+0000","last_peered":"2024-02-12T17:03:58.675829+0000","last_clean":"2024-02-12T17:03:58.675829+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:03:58.675829+0000","last_undegraded":"2024-02-12T17:03:58.675829+0000","last_fullsized":"2024-02-12T17:03:58.675829+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":74,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.689098+0000","last_change":"2024-02-12T17:03:50.552102+0000","last_active":"2024-02-12T17:03:58.689098+0000","last_peered":"2024-02-12T17:03:58.689098+0000","last_clean":"2024-02-12T17:03:58.689098+0000","last_became_active":"2024-02-12T17:03:50.551804+0000","last_became_peered":"2024-02-12T17:03:50.551804+0000","last_unstale":"2024-02-12T17:03:58.689098+0000","last_undegraded":"2024-02-12T17:03:58.689098+0000","last_fullsized":"2024-02-12T17:03:58.689098+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.794228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":259,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678632+0000","last_change":"2024-02-12T17:03:52.389245+0000","last_active":"2024-02-12T17:03:58.678632+0000","last_peered":"2024-02-12T17:03:58.678632+0000","last_clean":"2024-02-12T17:03:58.678632+0000","last_became_active":"2024-02-12T17:03:50.574562+0000","last_became_peered":"2024-02-12T17:03:50.574562+0000","last_unstale":"2024-02-12T17:03:58.678632+0000","last_undegraded":"2024-02-12T17:03:58.678632+0000","last_fullsized":"2024-02-12T17:03:58.678632+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:02:45.594719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:45.594719+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"352'12179","reported_seq":16409,"reported_epoch":352,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:03.690333+0000","last_change":"2024-02-12T17:03:50.580528+0000","last_active":"2024-02-12T17:04:03.690333+0000","last_peered":"2024-02-12T17:04:03.690333+0000","last_clean":"2024-02-12T17:03:19.721273+0000","last_became_active":"2024-02-12T17:03:21.956889+0000","last_became_peered":"2024-02-12T17:03:21.956889+0000","last_unstale":"2024-02-12T17:04:03.690333+0000","last_undegraded":"2024-02-12T17:04:03.690333+0000","last_fullsized":"2024-02-12T17:04:03.690333+0000","mapping_epoch":341,"log_start":"335'8718","ondisk_log_start":"335'8718","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3461,"ondisk_log_size":3461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":339148800,"num_objects":5175,"num_object_clones":0,"num_object_copies":15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3434,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12179,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2079,"num_bytes_recovered":136249344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":5175}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":32,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.680069+0000","last_change":"2024-02-12T17:03:50.578027+0000","last_active":"2024-02-12T17:03:58.680069+0000","last_peered":"2024-02-12T17:03:58.680069+0000","last_clean":"2024-02-12T17:03:58.680069+0000","last_became_active":"2024-02-12T17:03:50.577698+0000","last_became_peered":"2024-02-12T17:03:50.577698+0000","last_unstale":"2024-02-12T17:03:58.680069+0000","last_undegraded":"2024-02-12T17:03:58.680069+0000","last_fullsized":"2024-02-12T17:03:58.680069+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.986137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.986137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":287,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.673497+0000","last_change":"2024-02-12T17:03:50.553374+0000","last_active":"2024-02-12T17:03:58.673497+0000","last_peered":"2024-02-12T17:03:58.673497+0000","last_clean":"2024-02-12T17:03:58.673497+0000","last_became_active":"2024-02-12T17:03:50.552924+0000","last_became_peered":"2024-02-12T17:03:50.552924+0000","last_unstale":"2024-02-12T17:03:58.673497+0000","last_undegraded":"2024-02-12T17:03:58.673497+0000","last_fullsized":"2024-02-12T17:03:58.673497+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":18,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686167+0000","last_change":"2024-02-12T17:03:51.530474+0000","last_active":"2024-02-12T17:03:58.686167+0000","last_peered":"2024-02-12T17:03:58.686167+0000","last_clean":"2024-02-12T17:03:58.686167+0000","last_became_active":"2024-02-12T17:03:50.549642+0000","last_became_peered":"2024-02-12T17:03:50.549642+0000","last_unstale":"2024-02-12T17:03:58.686167+0000","last_undegraded":"2024-02-12T17:03:58.686167+0000","last_fullsized":"2024-02-12T17:03:58.686167+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:00:50.756578+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.756578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":334,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.678196+0000","last_change":"2024-02-12T17:03:50.905199+0000","last_active":"2024-02-12T17:03:58.678196+0000","last_peered":"2024-02-12T17:03:58.678196+0000","last_clean":"2024-02-12T17:03:58.678196+0000","last_became_active":"2024-02-12T17:03:50.558938+0000","last_became_peered":"2024-02-12T17:03:50.558938+0000","last_unstale":"2024-02-12T17:03:58.678196+0000","last_undegraded":"2024-02-12T17:03:58.678196+0000","last_fullsized":"2024-02-12T17:03:58.678196+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:28.096708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.096708+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"352'12319","reported_seq":12375,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-02-12T17:04:03.619400+0000","last_change":"2024-02-12T17:01:13.886545+0000","last_active":"2024-02-12T17:04:03.619400+0000","last_peered":"2024-02-12T17:04:03.619400+0000","last_clean":"2024-02-12T17:04:03.619400+0000","last_became_active":"2024-02-12T17:01:13.886238+0000","last_became_peered":"2024-02-12T17:01:13.886238+0000","last_unstale":"2024-02-12T17:04:03.619400+0000","last_undegraded":"2024-02-12T17:04:03.619400+0000","last_fullsized":"2024-02-12T17:04:03.619400+0000","mapping_epoch":304,"log_start":"331'7000","ondisk_log_start":"331'7000","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":5319,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":338100224,"num_objects":5159,"num_object_clones":0,"num_object_copies":15477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12319,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":321,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.690991+0000","last_change":"2024-02-12T17:01:26.804003+0000","last_active":"2024-02-12T17:03:58.690991+0000","last_peered":"2024-02-12T17:03:58.690991+0000","last_clean":"2024-02-12T17:03:58.690991+0000","last_became_active":"2024-02-12T16:59:09.261034+0000","last_became_peered":"2024-02-12T16:59:09.261034+0000","last_unstale":"2024-02-12T17:03:58.690991+0000","last_undegraded":"2024-02-12T17:03:58.690991+0000","last_fullsized":"2024-02-12T17:03:58.690991+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:26.803903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.803903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.686809+0000","last_change":"2024-02-12T17:03:50.560958+0000","last_active":"2024-02-12T17:03:58.686809+0000","last_peered":"2024-02-12T17:03:58.686809+0000","last_clean":"2024-02-12T17:03:58.686809+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:03:58.686809+0000","last_undegraded":"2024-02-12T17:03:58.686809+0000","last_fullsized":"2024-02-12T17:03:58.686809+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:01.789464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:01.789464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":396,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-02-12T17:03:58.687585+0000","last_change":"2024-02-12T17:02:43.665354+0000","last_active":"2024-02-12T17:03:58.687585+0000","last_peered":"2024-02-12T17:03:58.687585+0000","last_clean":"2024-02-12T17:03:58.687585+0000","last_became_active":"2024-02-12T16:59:09.280858+0000","last_became_peered":"2024-02-12T16:59:09.280858+0000","last_unstale":"2024-02-12T17:03:58.687585+0000","last_undegraded":"2024-02-12T17:03:58.687585+0000","last_fullsized":"2024-02-12T17:03:58.687585+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:02:43.665248+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.665248+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"349'15","reported_seq":553,"reported_epoch":352,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:01.719220+0000","last_change":"2024-02-12T17:03:52.392983+0000","last_active":"2024-02-12T17:04:01.719220+0000","last_peered":"2024-02-12T17:04:01.719220+0000","last_clean":"2024-02-12T17:03:47.426268+0000","last_became_active":"2024-02-12T17:03:51.637467+0000","last_became_peered":"2024-02-12T17:03:51.637467+0000","last_unstale":"2024-02-12T17:04:01.719220+0000","last_undegraded":"2024-02-12T17:04:01.719220+0000","last_fullsized":"2024-02-12T17:04:01.719220+0000","mapping_epoch":348,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-02-12T17:00:49.304784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.304784+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,2],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5333319704,"num_objects":81381,"num_object_clones":0,"num_object_copies":244143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":62412,"num_objects_unfound":0,"num_objects_dirty":81381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193795,"num_write_kb":8805569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5543,"num_bytes_recovered":362872832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16324243456,"data_stored":16324231240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64141,"ondisk_log_size":64141,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":173,"num_read_kb":168,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1095,"num_read_kb":852,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":9924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234160,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4305688,"kb_used_data":3707792,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":89439464,"statfs":{"total":95995035648,"available":91586011136,"internally_reserved":0,"allocated":3796779008,"data_stored":3796358746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":3.5419999999999998,"5min":2.1789999999999998,"15min":1.619},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.375,"5min":16.759,"15min":36.439},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.6799999999999999,"5min":2.4820000000000002,"15min":1.736},"min":{"1min":0.23100000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.8689999999999998,"5min":17.844000000000001,"15min":36.490000000000002},"last":2.9860000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":2.262,"5min":1.8580000000000001,"15min":1.3089999999999999},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.20899999999999999},"max":{"1min":6.3479999999999999,"5min":11.048999999999999,"15min":12.032},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.8480000000000001,"5min":2.194,"15min":1.399},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":5.8949999999999996,"5min":17.324999999999999,"15min":17.324999999999999},"last":3.008}]},{"osd":2,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":3.6949999999999998,"5min":1.8660000000000001,"15min":1.284},"min":{"1min":0.249,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.4,"5min":12.4,"15min":12.4},"last":0.44600000000000001},{"interface":"front","average":{"1min":2.2930000000000001,"5min":1.897,"15min":1.452},"min":{"1min":0.26400000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":6.3540000000000001,"5min":11.071999999999999,"15min":20.338000000000001},"last":1.5109999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":3.702,"5min":1.9890000000000001,"15min":1.5660000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":13.608000000000001,"5min":13.608000000000001,"15min":20.433},"last":0.47199999999999998},{"interface":"front","average":{"1min":3.6349999999999998,"5min":2.2639999999999998,"15min":1.7569999999999999},"min":{"1min":0.34300000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":31.152000000000001,"5min":31.152000000000001,"15min":31.152000000000001},"last":1.5600000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":4.9829999999999997,"5min":2.6200000000000001,"15min":1.5820000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":19.030000000000001,"5min":19.030000000000001,"15min":50.406999999999996},"last":3.1909999999999998},{"interface":"front","average":{"1min":4.2670000000000003,"5min":2.9420000000000002,"15min":1.8260000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.23400000000000001},"max":{"1min":16.867000000000001,"5min":16.867000000000001,"15min":50.168999999999997},"last":8.9550000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":5.274,"5min":3.0990000000000002,"15min":1.5740000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.24199999999999999},"max":{"1min":19.702000000000002,"5min":19.702000000000002,"15min":19.702000000000002},"last":3.234},{"interface":"front","average":{"1min":2.875,"5min":2.8719999999999999,"15min":1.609},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26200000000000001},"max":{"1min":11.122,"5min":18.651,"15min":18.651},"last":8.9890000000000008}]},{"osd":6,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":2.5609999999999999,"15min":1.4530000000000001},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":10.522,"5min":17.234999999999999,"15min":17.234999999999999},"last":3.0230000000000001},{"interface":"front","average":{"1min":3.9660000000000002,"5min":2.2709999999999999,"15min":1.411},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":13.676,"5min":13.676,"15min":13.676},"last":0.65000000000000002}]}]},{"osd":6,"up_from":0,"seq":193273528501,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738578,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4214652,"kb_used_data":3620852,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":89530500,"statfs":{"total":95995035648,"available":91679232000,"internally_reserved":0,"allocated":3707752448,"data_stored":3707424233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.76600000000000001,"15min":1.2170000000000001},"min":{"1min":0.189,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.8730000000000002,"5min":13.013,"15min":23.510999999999999},"last":2.2309999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.84599999999999997,"15min":1.1479999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":2.2629999999999999,"5min":5.0449999999999999,"15min":10.186999999999999},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.94599999999999995,"15min":1.3759999999999999},"min":{"1min":0.20999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.496,"5min":11.122999999999999,"15min":16.521000000000001},"last":3.7759999999999998},{"interface":"front","average":{"1min":1.361,"5min":1,"15min":1.2529999999999999},"min":{"1min":0.253,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.5999999999999996,"5min":4.984,"15min":10.340999999999999},"last":2.5059999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.121,"15min":1.514},"min":{"1min":0.29099999999999998,"5min":0.26100000000000001,"15min":0.19400000000000001},"max":{"1min":6.3230000000000004,"5min":7.0339999999999998,"15min":21.100000000000001},"last":2.528},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.2509999999999999,"15min":1.522},"min":{"1min":0.26300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":8.1980000000000004,"5min":11.976000000000001,"15min":16.446000000000002},"last":2.3340000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:03:41 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.746,"15min":1.4490000000000001},"min":{"1min":0.34200000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.9379999999999997,"5min":16.888999999999999,"15min":16.888999999999999},"last":2.2650000000000001},{"interface":"front","average":{"1min":1.8620000000000001,"5min":1.909,"15min":1.427},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28399999999999997},"max":{"1min":9.1020000000000003,"5min":16.899999999999999,"15min":16.899999999999999},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:03:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8759999999999999,"5min":2.5,"15min":1.863},"min":{"1min":0.433,"5min":0.30199999999999999,"15min":0.26400000000000001},"max":{"1min":5.9390000000000001,"5min":50.404000000000003,"15min":50.404000000000003},"last":2.4580000000000002},{"interface":"front","average":{"1min":1.6100000000000001,"5min":2.3570000000000002,"15min":1.752},"min":{"1min":0.28799999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.476,"5min":49.979999999999997,"15min":49.979999999999997},"last":2.2919999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:03:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.6060000000000001,"15min":1.4830000000000001},"min":{"1min":0.39300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.0739999999999998,"5min":9.8460000000000001,"15min":17.907},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.1469999999999998,"5min":1.9750000000000001,"15min":1.75},"min":{"1min":0.38500000000000001,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":6.8380000000000001,"5min":9.6739999999999995,"15min":21.062000000000001},"last":2.496}]},{"osd":7,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.881,"15min":1.496},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.32400000000000001},"max":{"1min":3.4409999999999998,"5min":12.058,"15min":16.175000000000001},"last":2.3130000000000002},{"interface":"front","average":{"1min":2.1280000000000001,"5min":2.5339999999999998,"15min":1.613},"min":{"1min":0.45100000000000001,"5min":0.39900000000000002,"15min":0.33600000000000002},"max":{"1min":7.0960000000000001,"5min":34.722999999999999,"15min":34.722999999999999},"last":0.502}]}]},{"osd":2,"up_from":18,"seq":77309411527,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3205948,"kb_used_data":2620340,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":90539204,"statfs":{"total":95995035648,"available":92712144896,"internally_reserved":0,"allocated":2683228160,"data_stored":2682836764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:03:06 2024","interfaces":[{"interface":"back","average":{"1min":2.7949999999999999,"5min":1.2529999999999999,"15min":1.5329999999999999},"min":{"1min":0.19400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":36.100999999999999,"5min":36.100999999999999,"15min":36.100999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":3.5059999999999998,"5min":1.6359999999999999,"15min":1.46},"min":{"1min":0.23599999999999999,"5min":0.193,"15min":0.158},"max":{"1min":36.299999999999997,"5min":36.299999999999997,"15min":36.299999999999997},"last":0.247}]},{"osd":1,"last update":"Mon Feb 12 17:03:06 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.98999999999999999,"15min":1.4730000000000001},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.191},"max":{"1min":2.556,"5min":6.5819999999999999,"15min":17.739000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.0700000000000001,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":4.1699999999999999,"5min":4.3099999999999996,"15min":15.766999999999999},"last":1.1679999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1399999999999999,"15min":1.7070000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23499999999999999},"max":{"1min":4.1180000000000003,"5min":5.1239999999999997,"15min":51.579999999999998},"last":1.131},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1160000000000001,"15min":1.4470000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23200000000000001},"max":{"1min":4.5449999999999999,"5min":4.5449999999999999,"15min":48.988999999999997},"last":0.499}]},{"osd":4,"last update":"Mon Feb 12 17:03:59 2024","interfaces":[{"interface":"back","average":{"1min":2.383,"5min":1.964,"15min":1.6850000000000001},"min":{"1min":0.35999999999999999,"5min":0.253,"15min":0.253},"max":{"1min":18.34,"5min":18.658999999999999,"15min":18.658999999999999},"last":1.105},{"interface":"front","average":{"1min":2.5139999999999998,"5min":2.2909999999999999,"15min":1.6899999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":18.355,"5min":18.355,"15min":18.355},"last":0.90300000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:03:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.6679999999999999,"15min":1.589},"min":{"1min":0.33000000000000002,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":3.359,"5min":10.407999999999999,"15min":15.699},"last":1.147},{"interface":"front","average":{"1min":1.0760000000000001,"5min":2.0489999999999999,"15min":1.72},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.27700000000000002},"max":{"1min":2.9350000000000001,"5min":13.207000000000001,"15min":13.207000000000001},"last":2.024}]},{"osd":6,"last update":"Mon Feb 12 17:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.8759999999999999,"15min":1.8200000000000001},"min":{"1min":0.33000000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.6310000000000002,"5min":16.739000000000001,"15min":17.782},"last":2.2200000000000002},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.8049999999999999,"15min":1.6160000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29799999999999999},"max":{"1min":5.9080000000000004,"5min":8.7850000000000001,"15min":13.446999999999999},"last":0.87}]},{"osd":7,"last update":"Mon Feb 12 17:03:06 2024","interfaces":[{"interface":"back","average":{"1min":2.1360000000000001,"5min":2.0529999999999999,"15min":1.6619999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.32600000000000001},"max":{"1min":6.1299999999999999,"5min":8.8520000000000003,"15min":13.473000000000001},"last":2.3290000000000002},{"interface":"front","average":{"1min":1.863,"5min":2.1509999999999998,"15min":1.748},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":6.4340000000000002,"5min":16.768000000000001,"15min":16.768000000000001},"last":0.84699999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149891,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":418400,"kb_used_data":108888,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93326752,"statfs":{"total":95995035648,"available":95566594048,"internally_reserved":0,"allocated":111501312,"data_stored":111073893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:03:31 2024","interfaces":[{"interface":"back","average":{"1min":3.355,"5min":1.175,"15min":1.522},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":58.451999999999998,"5min":58.451999999999998,"15min":58.451999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":3.327,"5min":1.494,"15min":1.5009999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":57.497999999999998,"5min":57.497999999999998,"15min":57.497999999999998},"last":0.24199999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.92100000000000004,"15min":1.282},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.216},"max":{"1min":2.7480000000000002,"5min":9.1370000000000005,"15min":13.94},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.72999999999999998,"15min":1.202},"min":{"1min":0.25800000000000001,"5min":0.254,"15min":0.19500000000000001},"max":{"1min":3.5899999999999999,"5min":5.468,"15min":14.962},"last":0.438}]},{"osd":2,"last update":"Mon Feb 12 17:03:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.86399999999999999,"15min":1.466},"min":{"1min":0.27700000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.109,"5min":5.4189999999999996,"15min":49.112000000000002},"last":0.5},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.73099999999999998,"15min":1.5860000000000001},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.9889999999999999,"5min":3.8940000000000001,"15min":51.997999999999998},"last":0.40899999999999997}]},{"osd":4,"last update":"Mon Feb 12 17:03:47 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":2.1299999999999999,"15min":1.6499999999999999},"min":{"1min":0.44500000000000001,"5min":0.32600000000000001,"15min":0.25900000000000001},"max":{"1min":3.3519999999999999,"5min":11.047000000000001,"15min":19.010000000000002},"last":2.5609999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.8700000000000001,"15min":1.5029999999999999},"min":{"1min":0.34999999999999998,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":3.1840000000000002,"5min":19.033999999999999,"15min":19.033999999999999},"last":6.0990000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":2.3149999999999999,"15min":1.6930000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":14.236000000000001,"5min":16.076000000000001,"15min":19.882999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.9359999999999999,"5min":2.5529999999999999,"15min":1.72},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33300000000000002},"max":{"1min":13.927,"5min":15.595000000000001,"15min":15.595000000000001},"last":1.3600000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.629,"15min":1.335},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":3.8660000000000001,"5min":12.856999999999999,"15min":12.856999999999999},"last":2.052},{"interface":"front","average":{"1min":1.448,"5min":1.7150000000000001,"15min":1.587},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.27500000000000002},"max":{"1min":3.976,"5min":12.635999999999999,"15min":19.925999999999998},"last":0.85299999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:03:47 2024","interfaces":[{"interface":"back","average":{"1min":2.8660000000000001,"5min":2.415,"15min":1.8140000000000001},"min":{"1min":0.497,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":9.8420000000000005,"5min":12.695,"15min":12.891},"last":0.81499999999999995},{"interface":"front","average":{"1min":3.5449999999999999,"5min":2.3109999999999999,"15min":1.79},"min":{"1min":0.55300000000000005,"5min":0.39700000000000002,"15min":0.32800000000000001},"max":{"1min":25.838999999999999,"5min":25.838999999999999,"15min":25.838999999999999},"last":1.1899999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953663,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2608508,"kb_used_data":2010484,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":91136644,"statfs":{"total":95995035648,"available":93323923456,"internally_reserved":0,"allocated":2058735616,"data_stored":2058343594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:03:46 2024","interfaces":[{"interface":"back","average":{"1min":3.3620000000000001,"5min":2.331,"15min":1.6479999999999999},"min":{"1min":0.20999999999999999,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":12.874000000000001,"5min":12.874000000000001,"15min":38.179000000000002},"last":4.3019999999999996},{"interface":"front","average":{"1min":1.454,"5min":2.331,"15min":1.6719999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.3239999999999998,"5min":12.281000000000001,"15min":38.151000000000003},"last":4.9299999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:03:46 2024","interfaces":[{"interface":"back","average":{"1min":2.524,"5min":2.339,"15min":1.534},"min":{"1min":0.30599999999999999,"5min":0.251,"15min":0.218},"max":{"1min":9.6940000000000008,"5min":11.077999999999999,"15min":11.077999999999999},"last":3.3239999999999998},{"interface":"front","average":{"1min":3.5640000000000001,"5min":2.3719999999999999,"15min":1.5760000000000001},"min":{"1min":0.307,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":12.936999999999999,"5min":12.936999999999999,"15min":12.936999999999999},"last":4.3460000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:03:46 2024","interfaces":[{"interface":"back","average":{"1min":3.5630000000000002,"5min":2.2509999999999999,"15min":1.5720000000000001},"min":{"1min":0.35499999999999998,"5min":0.25800000000000001,"15min":0.22500000000000001},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":4.3239999999999998},{"interface":"front","average":{"1min":1.8560000000000001,"5min":2.3479999999999999,"15min":1.5940000000000001},"min":{"1min":0.26900000000000002,"5min":0.245,"15min":0.224},"max":{"1min":5.3799999999999999,"5min":12.246,"15min":12.246},"last":2.7010000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:03:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":2.3919999999999999,"15min":1.9219999999999999},"min":{"1min":0.40300000000000002,"5min":0.28899999999999998,"15min":0.26700000000000002},"max":{"1min":5.3490000000000002,"5min":12.324,"15min":38.170999999999999},"last":4.899},{"interface":"front","average":{"1min":2.4990000000000001,"5min":2.444,"15min":1.7709999999999999},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":9.7289999999999992,"5min":10.865,"15min":38.311999999999998},"last":3.3010000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:03:11 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":3.7229999999999999,"15min":2.0059999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.24099999999999999},"max":{"1min":8.3070000000000004,"5min":42.420000000000002,"15min":79.573999999999998},"last":3.3679999999999999},{"interface":"front","average":{"1min":2.3610000000000002,"5min":4.2530000000000001,"15min":2.1030000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":7.7869999999999999,"5min":45.204999999999998,"15min":80.497},"last":4.3650000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:03:39 2024","interfaces":[{"interface":"back","average":{"1min":2.7389999999999999,"5min":2.8650000000000002,"15min":1.9430000000000001},"min":{"1min":0.44800000000000001,"5min":0.28699999999999998,"15min":0.255},"max":{"1min":12.265000000000001,"5min":12.343999999999999,"15min":104.907},"last":2.7309999999999999},{"interface":"front","average":{"1min":2.4900000000000002,"5min":3.149,"15min":2.0659999999999998},"min":{"1min":0.49299999999999999,"5min":0.30499999999999999,"15min":0.26500000000000001},"max":{"1min":9.8000000000000007,"5min":13.739000000000001,"15min":105.395},"last":3.3450000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:03:54 2024","interfaces":[{"interface":"back","average":{"1min":4.6070000000000002,"5min":3.004,"15min":1.649},"min":{"1min":0.41099999999999998,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":15.836,"5min":15.836,"15min":15.836},"last":6.3559999999999999},{"interface":"front","average":{"1min":3.3799999999999999,"5min":3.23,"15min":1.806},"min":{"1min":0.45000000000000001,"5min":0.33500000000000002,"15min":0.28799999999999998},"max":{"1min":11.223000000000001,"5min":21.079000000000001,"15min":27.773},"last":6.2439999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757435,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4893268,"kb_used_data":3992396,"kb_used_omap":0,"kb_used_meta":900672,"kb_avail":88851884,"statfs":{"total":95995035648,"available":90984329216,"internally_reserved":0,"allocated":4088213504,"data_stored":4087797394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922288128},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":2.5449999999999999,"15min":1.591},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20300000000000001},"max":{"1min":6.6040000000000001,"5min":46.365000000000002,"15min":46.365000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.53,"5min":3,"15min":1.839},"min":{"1min":0.20699999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.6900000000000004,"5min":46.134,"15min":46.134},"last":0.73399999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.8959999999999999,"15min":1.446},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.23200000000000001},"max":{"1min":8.1099999999999994,"5min":8.1099999999999994,"15min":16.655000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.4570000000000001,"5min":2.258,"15min":1.5980000000000001},"min":{"1min":0.23799999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.7359999999999998,"5min":13.130000000000001,"15min":16.712},"last":0.69099999999999995}]},{"osd":2,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":2.464,"15min":1.6970000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":7.9050000000000002,"5min":11.15,"15min":18.218},"last":0.48299999999999998},{"interface":"front","average":{"1min":2.2160000000000002,"5min":2.2200000000000002,"15min":1.6160000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":8.0470000000000006,"5min":9.1199999999999992,"15min":18.327000000000002},"last":1.7230000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":2.2320000000000002,"15min":1.6040000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.6319999999999997,"5min":11.122,"15min":18.251000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.8979999999999999,"5min":2.097,"15min":1.5149999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":20.263000000000002},"last":1.9510000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":2.9020000000000001,"15min":1.3720000000000001},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":6.8650000000000002,"5min":11.880000000000001,"15min":11.880000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.4139999999999999,"5min":3.3260000000000001,"15min":1.573},"min":{"1min":0.32100000000000001,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":7.0010000000000003,"5min":20.550000000000001,"15min":20.550000000000001},"last":0.28499999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:03:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":2.6440000000000001,"15min":1.4350000000000001},"min":{"1min":0.39000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.6219999999999999,"5min":12.281000000000001,"15min":12.281000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.6870000000000001,"5min":2.7669999999999999,"15min":1.546},"min":{"1min":0.35399999999999998,"5min":0.28100000000000003,"15min":0.26700000000000002},"max":{"1min":5.8300000000000001,"5min":14.323,"15min":22.035},"last":0.314}]},{"osd":7,"last update":"Mon Feb 12 17:03:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4100000000000001,"5min":3.0329999999999999,"15min":1.4890000000000001},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.27500000000000002},"max":{"1min":8.1999999999999993,"5min":19.184999999999999,"15min":19.184999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.633,"5min":2.8359999999999999,"15min":1.448},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":9.7409999999999997,"5min":12.253,"15min":12.253},"last":1.605}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3676569600,"data_stored":3676569600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2670592000,"data_stored":2670592000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111345664,"data_stored":111345664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2045972480,"data_stored":2045968408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4048162816,"data_stored":4048158744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3771600896,"data_stored":3771596824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:04:06.091 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-12T17:04:06.091 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.meta pg_num 22 2024-02-12T17:04:06.184 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:06.184 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:06.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:06.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:04.251524+0000 mgr.y (mgr.24481) 548 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:06.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:04.633731+0000 mgr.y (mgr.24481) 549 : cluster [DBG] pgmap v707: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 5.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 62413/244719 objects misplaced (25.504%); 13 MiB/s, 220 objects/s recovering 2024-02-12T17:04:06.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:04.940816+0000 mgr.y (mgr.24481) 550 : audit [DBG] from='client.15636 v1:172.21.15.33:0/1460949401' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:04:06.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:04.986175+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.33:0/2902161555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:06.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:04.251524+0000 mgr.y (mgr.24481) 548 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:06.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:04.633731+0000 mgr.y (mgr.24481) 549 : cluster [DBG] pgmap v707: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 5.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 62413/244719 objects misplaced (25.504%); 13 MiB/s, 220 objects/s recovering 2024-02-12T17:04:06.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:04.940816+0000 mgr.y (mgr.24481) 550 : audit [DBG] from='client.15636 v1:172.21.15.33:0/1460949401' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:04:06.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:04.986175+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.33:0/2902161555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:06.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:04.251524+0000 mgr.y (mgr.24481) 548 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:06.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:04.633731+0000 mgr.y (mgr.24481) 549 : cluster [DBG] pgmap v707: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 5.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 62413/244719 objects misplaced (25.504%); 13 MiB/s, 220 objects/s recovering 2024-02-12T17:04:06.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:04.940816+0000 mgr.y (mgr.24481) 550 : audit [DBG] from='client.15636 v1:172.21.15.33:0/1460949401' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:04:06.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:04.986175+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.33:0/2902161555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:06.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:06.357 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:06.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:06.494 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:06.537 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:07.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:06.634213+0000 mgr.y (mgr.24481) 551 : cluster [DBG] pgmap v708: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 5.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 62413/244719 objects misplaced (25.504%); 12 MiB/s, 195 objects/s recovering 2024-02-12T17:04:07.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:06.634213+0000 mgr.y (mgr.24481) 551 : cluster [DBG] pgmap v708: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 5.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 62413/244719 objects misplaced (25.504%); 12 MiB/s, 195 objects/s recovering 2024-02-12T17:04:07.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:06.634213+0000 mgr.y (mgr.24481) 551 : cluster [DBG] pgmap v708: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 5.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 62413/244719 objects misplaced (25.504%); 12 MiB/s, 195 objects/s recovering 2024-02-12T17:04:07.823 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:07.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:07.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:08.003 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:08.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:08.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:08.236 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:08.276 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:08.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:08.383 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:08.465 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:08.525 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:08.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:06.650721+0000 osd.0 (osd.0) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:08.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:06.650772+0000 osd.0 (osd.0) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:08.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:07.310550+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e353: 8 total, 8 up, 6 in 2024-02-12T17:04:08.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:06.650721+0000 osd.0 (osd.0) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:08.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:06.650772+0000 osd.0 (osd.0) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:08.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:07.310550+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e353: 8 total, 8 up, 6 in 2024-02-12T17:04:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:08.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:08.536 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:06.650721+0000 osd.0 (osd.0) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:08.537 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:06.650772+0000 osd.0 (osd.0) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:08.537 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:07.310550+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e353: 8 total, 8 up, 6 in 2024-02-12T17:04:08.637 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:08.682 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:08.711 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:08.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:08.817 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:08.857 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:08.973 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:04:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:09.320 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:09.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:09.427 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:09.475 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:09.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:08.634788+0000 mgr.y (mgr.24481) 552 : cluster [DBG] pgmap v710: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 777 op/s; 60780/239370 objects misplaced (25.392%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T17:04:09.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:08.634788+0000 mgr.y (mgr.24481) 552 : cluster [DBG] pgmap v710: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 777 op/s; 60780/239370 objects misplaced (25.392%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T17:04:09.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:08.634788+0000 mgr.y (mgr.24481) 552 : cluster [DBG] pgmap v710: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 777 op/s; 60780/239370 objects misplaced (25.392%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T17:04:09.743 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:09.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:09.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:09.890 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:10.164 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:04:10.164 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:199234 9.4 9:2504a2ad:::benchmark_data_smithi033_131576_object61644:head [delete] snapc 0=[] ondisk+write+known_if_redirected e353)", 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:04:10.136973+0000", 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0090977769999999996, 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091273540000000007, 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 199234 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.136973+0000", 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.136973+0000", 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:04:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.136971+0000", 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6680000000000001e-06 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.136978+0000", 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2779000000000001e-05 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.136991+0000", 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3379999999999999e-06 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.136999+0000", 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9790000000000003e-05 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.137048+0000", 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015526099999999999 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.137204+0000", 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019082100000000001 2024-02-12T17:04:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,5", 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.137394+0000", 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0076453199999999997 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145040+0000", 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5210000000000008e-06 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145049+0000", 2024-02-12T17:04:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060658000000000001 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145656+0000", 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086826850000000007 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.15459.0:199236 9.6 e353/341 9:60049fd7:::benchmark_data_smithi033_131576_object61646:head v 353'12526, mlcod=353'12523)", 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:04:10.141079+0000", 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0049920590000000001, 2024-02-12T17:04:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050945770000000003, 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.141079+0000", 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.141079+0000", 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.141078+0000", 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1541e-05 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.141089+0000", 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6760000000000002e-06 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:04:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.141099+0000", 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2939999999999996e-06 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.141104+0000", 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5043999999999998e-05 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.141149+0000", 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5219999999999999e-05 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.141174+0000", 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5445000000000005e-05 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.15459.0:199239 9.e e353/341 9:7178c87b:::benchmark_data_smithi033_131576_object61649:head v 353'12643, mlcod=353'12641)", 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:04:10.144272+0000", 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0017987879999999999, 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019394239999999999, 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:04:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.144272+0000", 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.144272+0000", 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.144271+0000", 2024-02-12T17:04:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1447999999999999e-05 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.144282+0000", 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9279999999999999e-06 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.144291+0000", 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2830000000000003e-06 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.144295+0000", 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0179000000000002e-05 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.144345+0000", 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3661000000000001e-05 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.144389+0000", 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011695499999999999 2024-02-12T17:04:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.15459.0:199243 9.6 e353/341 9:6814c035:::benchmark_data_smithi033_131576_object61653:head v 353'12527, mlcod=353'12524)", 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:04:10.145052+0000", 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.001018737, 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001195301, 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:04:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145052+0000", 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145052+0000", 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145051+0000", 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3879999999999997e-06 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145058+0000", 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0430000000000008e-06 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145066+0000", 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0830000000000003e-06 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145071+0000", 2024-02-12T17:04:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5717000000000003e-05 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145117+0000", 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3895000000000001e-05 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:04:10.145141+0000", 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8366000000000001e-05 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:04:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:04:10.182 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 4 2024-02-12T17:04:10.182 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:04:10.582 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:10.607 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:04:10.880 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:04:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:10] "GET /metrics HTTP/1.1" 200 176146 "" "Prometheus/2.33.4" 2024-02-12T17:04:10.881 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:10.532481+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? v1:172.21.15.33:0/3359589078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:10.881 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:10.583034+0000 mon.a (mon.0) 1388 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:04:10.881 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:10.532481+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? v1:172.21.15.33:0/3359589078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:10.882 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:10.583034+0000 mon.a (mon.0) 1388 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:04:10.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:10.532481+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? v1:172.21.15.33:0/3359589078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:10.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:10.583034+0000 mon.a (mon.0) 1388 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:04:11.158 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:11.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:11.178 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T17:04:11.265 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:11.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:11.370 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:11.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:11.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:11.474 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:11.548 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:11.598 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:11.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:11.691 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:10.592760+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? v1:172.21.15.33:0/3359589078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:04:11.691 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:10.592814+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e354: 8 total, 8 up, 6 in 2024-02-12T17:04:11.691 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:10.635257+0000 mgr.y (mgr.24481) 553 : cluster [DBG] pgmap v712: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 58439/232131 objects misplaced (25.175%); 13 MiB/s, 214 objects/s recovering 2024-02-12T17:04:11.691 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:10.592760+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? v1:172.21.15.33:0/3359589078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:04:11.692 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:10.592814+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e354: 8 total, 8 up, 6 in 2024-02-12T17:04:11.692 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:10.635257+0000 mgr.y (mgr.24481) 553 : cluster [DBG] pgmap v712: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 58439/232131 objects misplaced (25.175%); 13 MiB/s, 214 objects/s recovering 2024-02-12T17:04:11.705 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:11.777 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:11.808 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:11.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:11.915 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:11.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:10.592760+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? v1:172.21.15.33:0/3359589078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:04:11.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:10.592814+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e354: 8 total, 8 up, 6 in 2024-02-12T17:04:11.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:10.635257+0000 mgr.y (mgr.24481) 553 : cluster [DBG] pgmap v712: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 58439/232131 objects misplaced (25.175%); 13 MiB/s, 214 objects/s recovering 2024-02-12T17:04:12.007 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:12.034 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:12.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:12.147 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:12.199 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:04:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:04:13.766 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:13.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:13.872 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:13.918 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:14.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:12.635717+0000 mgr.y (mgr.24481) 554 : cluster [DBG] pgmap v713: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 524 op/s; 58439/232131 objects misplaced (25.175%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T17:04:14.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:12.970702+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:14.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:12.635717+0000 mgr.y (mgr.24481) 554 : cluster [DBG] pgmap v713: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 524 op/s; 58439/232131 objects misplaced (25.175%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T17:04:14.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:12.970702+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:14.029 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:14.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:14.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:14.181 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:12.635717+0000 mgr.y (mgr.24481) 554 : cluster [DBG] pgmap v713: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 524 op/s; 58439/232131 objects misplaced (25.175%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T17:04:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:12.970702+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:14.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:04:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:04:14.800 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:14.844 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:14.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:14.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:15.029 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:15.071 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:15.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:15.176 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:15.224 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:15.767 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:04:16.016 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 6 pg_num to 22 2024-02-12T17:04:16.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:14.263515+0000 mgr.y (mgr.24481) 555 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:14.636500+0000 mgr.y (mgr.24481) 556 : cluster [DBG] pgmap v714: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 54503/219792 objects misplaced (24.798%); 13 MiB/s, 207 objects/s recovering 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:15.118640+0000 mon.c (mon.2) 198 : audit [INF] from='client.? v1:172.21.15.33:0/959749131' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:15.120863+0000 mon.a (mon.0) 1392 : audit [INF] from='client.35603 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:15.692999+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.33:0/1223784328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:14.263515+0000 mgr.y (mgr.24481) 555 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:14.636500+0000 mgr.y (mgr.24481) 556 : cluster [DBG] pgmap v714: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 54503/219792 objects misplaced (24.798%); 13 MiB/s, 207 objects/s recovering 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:15.118640+0000 mon.c (mon.2) 198 : audit [INF] from='client.? v1:172.21.15.33:0/959749131' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:15.120863+0000 mon.a (mon.0) 1392 : audit [INF] from='client.35603 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:04:16.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:15.692999+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.33:0/1223784328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:16.499 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:14.263515+0000 mgr.y (mgr.24481) 555 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:16.499 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:14.636500+0000 mgr.y (mgr.24481) 556 : cluster [DBG] pgmap v714: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 54503/219792 objects misplaced (24.798%); 13 MiB/s, 207 objects/s recovering 2024-02-12T17:04:16.500 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:15.118640+0000 mon.c (mon.2) 198 : audit [INF] from='client.? v1:172.21.15.33:0/959749131' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:04:16.500 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:15.120863+0000 mon.a (mon.0) 1392 : audit [INF] from='client.35603 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:04:16.500 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:15.692999+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.33:0/1223784328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:16.586 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:04:16.586 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:04:16.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:04:16.588 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:04:17.029 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:17.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:17.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:17.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:15.998735+0000 mon.a (mon.0) 1393 : audit [INF] from='client.35603 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:04:17.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:15.998815+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e355: 8 total, 8 up, 6 in 2024-02-12T17:04:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:16.637216+0000 mon.c (mon.2) 200 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:16.637996+0000 mon.a (mon.0) 1395 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:15.998735+0000 mon.a (mon.0) 1393 : audit [INF] from='client.35603 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:04:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:15.998815+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e355: 8 total, 8 up, 6 in 2024-02-12T17:04:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:16.637216+0000 mon.c (mon.2) 200 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:16.637996+0000 mon.a (mon.0) 1395 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:15.998735+0000 mon.a (mon.0) 1393 : audit [INF] from='client.35603 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:04:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:15.998815+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e355: 8 total, 8 up, 6 in 2024-02-12T17:04:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:16.637216+0000 mon.c (mon.2) 200 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:16.637996+0000 mon.a (mon.0) 1395 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:17.652 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:17.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:17.758 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:17.808 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:17.916 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:17.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:18.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:18.096 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:18.138 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:18.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:18.246 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:16.636905+0000 mgr.y (mgr.24481) 557 : cluster [DBG] pgmap v716: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 816 op/s; 54503/219792 objects misplaced (24.798%); 12 MiB/s, 192 objects/s recovering 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:17.295401+0000 mon.a (mon.0) 1396 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:17.295481+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e356: 8 total, 8 up, 6 in 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:17.487947+0000 mon.a (mon.0) 1398 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:17.700905+0000 mon.c (mon.2) 201 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:16.636905+0000 mgr.y (mgr.24481) 557 : cluster [DBG] pgmap v716: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 816 op/s; 54503/219792 objects misplaced (24.798%); 12 MiB/s, 192 objects/s recovering 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:17.295401+0000 mon.a (mon.0) 1396 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:17.295481+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e356: 8 total, 8 up, 6 in 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:17.487947+0000 mon.a (mon.0) 1398 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:17.700905+0000 mon.c (mon.2) 201 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:04:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:16.636905+0000 mgr.y (mgr.24481) 557 : cluster [DBG] pgmap v716: 138 pgs: 7 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 124 active+clean; 4.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 816 op/s; 54503/219792 objects misplaced (24.798%); 12 MiB/s, 192 objects/s recovering 2024-02-12T17:04:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:17.295401+0000 mon.a (mon.0) 1396 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:04:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:17.295481+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e356: 8 total, 8 up, 6 in 2024-02-12T17:04:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:17.487947+0000 mon.a (mon.0) 1398 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:17.700905+0000 mon.c (mon.2) 201 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:04:18.612 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:04:18.612 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:04:18.874 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:18.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:18.982 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:19.029 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:19.349 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:19.350 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:19.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:19.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T17:04:19.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:19.515 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:19.544 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:19.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:19.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:19.670 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:04:19.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:18.321492+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-02-12T17:04:19.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:18.467490+0000 mon.c (mon.2) 202 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:04:19.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:18.468990+0000 mon.c (mon.2) 203 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:04:19.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:18.481167+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:19.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:18.637708+0000 mgr.y (mgr.24481) 558 : cluster [DBG] pgmap v719: 138 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 123 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 971 op/s; 52959/214662 objects misplaced (24.671%); 10 MiB/s, 163 objects/s recovering 2024-02-12T17:04:19.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:18.638238+0000 mon.c (mon.2) 204 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:04:19.704 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:18.640763+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:04:19.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:18.321492+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-02-12T17:04:19.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:18.467490+0000 mon.c (mon.2) 202 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:04:19.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:18.468990+0000 mon.c (mon.2) 203 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:04:19.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:18.481167+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:18.637708+0000 mgr.y (mgr.24481) 558 : cluster [DBG] pgmap v719: 138 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 123 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 971 op/s; 52959/214662 objects misplaced (24.671%); 10 MiB/s, 163 objects/s recovering 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:18.638238+0000 mon.c (mon.2) 204 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:18.640763+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:18.321492+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:18.467490+0000 mon.c (mon.2) 202 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:18.468990+0000 mon.c (mon.2) 203 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:18.481167+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:18.637708+0000 mgr.y (mgr.24481) 558 : cluster [DBG] pgmap v719: 138 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 123 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 971 op/s; 52959/214662 objects misplaced (24.671%); 10 MiB/s, 163 objects/s recovering 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:18.638238+0000 mon.c (mon.2) 204 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:04:19.711 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:18.640763+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:04:19.712 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:19.738 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:19.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:19.841 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:19.893 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:19.952 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:19.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:19.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:20.058 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:20.110 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:20.152 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:20.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:20.256 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:20.299 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:20.335 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:20.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:20.396 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:20.442 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:20.493 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:20.641 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:19.492196+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T17:04:20.642 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:19.492256+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-02-12T17:04:20.642 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:19.492196+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T17:04:20.642 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:19.492256+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-02-12T17:04:20.975 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:20.977 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:04:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:20] "GET /metrics HTTP/1.1" 200 176139 "" "Prometheus/2.33.4" 2024-02-12T17:04:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:19.492196+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T17:04:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:19.492256+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-02-12T17:04:21.763 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:21.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:20.521025+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e359: 8 total, 8 up, 6 in 2024-02-12T17:04:21.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:20.638243+0000 mgr.y (mgr.24481) 559 : cluster [DBG] pgmap v722: 138 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 123 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 992 op/s; 50754/207894 objects misplaced (24.413%); 13 MiB/s, 203 objects/s recovering 2024-02-12T17:04:21.763 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:20.521025+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e359: 8 total, 8 up, 6 in 2024-02-12T17:04:21.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:20.638243+0000 mgr.y (mgr.24481) 559 : cluster [DBG] pgmap v722: 138 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 123 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 992 op/s; 50754/207894 objects misplaced (24.413%); 13 MiB/s, 203 objects/s recovering 2024-02-12T17:04:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:21.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:21.869 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:21.919 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:21.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:20.521025+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e359: 8 total, 8 up, 6 in 2024-02-12T17:04:21.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:20.638243+0000 mgr.y (mgr.24481) 559 : cluster [DBG] pgmap v722: 138 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 123 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 992 op/s; 50754/207894 objects misplaced (24.413%); 13 MiB/s, 203 objects/s recovering 2024-02-12T17:04:22.066 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:22.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:22.170 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:22.235 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:22.265 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:22.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:22.385 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:22.494 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:22.558 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:04:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:22.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:22.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:22.681 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:22.725 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:22.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:04:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:04:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:21.535125+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e360: 8 total, 8 up, 6 in 2024-02-12T17:04:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:22.373288+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.33:0/662681831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:04:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:22.373972+0000 mon.a (mon.0) 1406 : audit [INF] from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:04:23.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:21.535125+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e360: 8 total, 8 up, 6 in 2024-02-12T17:04:23.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:22.373288+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.33:0/662681831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:04:23.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:22.373972+0000 mon.a (mon.0) 1406 : audit [INF] from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:04:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:21.535125+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e360: 8 total, 8 up, 6 in 2024-02-12T17:04:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:22.373288+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.33:0/662681831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:04:23.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:22.373972+0000 mon.a (mon.0) 1406 : audit [INF] from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:04:23.091 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:23.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:23.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:23.273 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:23.322 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:23.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7, 3] out_osds: [1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:04:23.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:04:23.326 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-02-12T17:04:23.326 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T17:04:23.326 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T17:04:23.326 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-02-12T17:04:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:23.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:23.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:23.493 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:23.531 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:23.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:23.638 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:22.532165+0000 mon.a (mon.0) 1407 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:22.545314+0000 mon.a (mon.0) 1408 : audit [INF] from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:22.545388+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e361: 8 total, 8 up, 6 in 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:22.638711+0000 mgr.y (mgr.24481) 560 : cluster [DBG] pgmap v725: 137 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 122 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 50754/207894 objects misplaced (24.413%); 11 MiB/s, 173 objects/s recovering 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:22.639104+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:22.640261+0000 mon.a (mon.0) 1410 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:22.532165+0000 mon.a (mon.0) 1407 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:22.545314+0000 mon.a (mon.0) 1408 : audit [INF] from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:04:23.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:22.545388+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e361: 8 total, 8 up, 6 in 2024-02-12T17:04:23.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:22.638711+0000 mgr.y (mgr.24481) 560 : cluster [DBG] pgmap v725: 137 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 122 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 50754/207894 objects misplaced (24.413%); 11 MiB/s, 173 objects/s recovering 2024-02-12T17:04:23.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:22.639104+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:04:23.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:22.640261+0000 mon.a (mon.0) 1410 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:04:23.682 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:23.869 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:04:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:22.532165+0000 mon.a (mon.0) 1407 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:23.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:22.545314+0000 mon.a (mon.0) 1408 : audit [INF] from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:04:23.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:22.545388+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e361: 8 total, 8 up, 6 in 2024-02-12T17:04:23.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:22.638711+0000 mgr.y (mgr.24481) 560 : cluster [DBG] pgmap v725: 137 pgs: 7 active+remapped+backfill_toofull, 1 active+recovering, 3 active+remapped+backfill_wait, 4 active+remapped+backfilling, 122 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 50754/207894 objects misplaced (24.413%); 11 MiB/s, 173 objects/s recovering 2024-02-12T17:04:23.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:22.639104+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:04:23.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:22.640261+0000 mon.a (mon.0) 1410 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:04:24.350 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:24.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:24.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:24.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:04:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:04:24.533 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:24.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:23.421423+0000 osd.2 (osd.2) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:24.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:23.421484+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:24.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:23.558533+0000 mon.a (mon.0) 1411 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T17:04:24.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:23.558612+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e362: 8 total, 8 up, 6 in 2024-02-12T17:04:24.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:23.421423+0000 osd.2 (osd.2) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:24.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:23.421484+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:24.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:23.558533+0000 mon.a (mon.0) 1411 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T17:04:24.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:23.558612+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e362: 8 total, 8 up, 6 in 2024-02-12T17:04:24.749 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:23.421423+0000 osd.2 (osd.2) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:24.749 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:23.421484+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:24.749 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:23.558533+0000 mon.a (mon.0) 1411 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T17:04:24.749 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:23.558612+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e362: 8 total, 8 up, 6 in 2024-02-12T17:04:24.908 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:04:24.908 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:04:25.262 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:25.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:25.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:25.562 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:25.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:25.670 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:25.714 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:04:25.720 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:25.751 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:25.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:25.861 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:25.967 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:24.169951+0000 osd.5 (osd.5) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:25.967 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:24.169979+0000 osd.5 (osd.5) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:25.967 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:24.284075+0000 mgr.y (mgr.24481) 561 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:25.967 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:24.572391+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e363: 8 total, 8 up, 6 in 2024-02-12T17:04:25.967 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:24.641216+0000 mgr.y (mgr.24481) 562 : cluster [DBG] pgmap v728: 137 pgs: 2 peering, 5 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 4.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s; 47020/196629 objects misplaced (23.913%); 16 MiB/s, 250 objects/s recovering 2024-02-12T17:04:25.972 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:25.993 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:04:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "" 2024-02-12T17:04:25.993 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:04:26.019 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:26.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:26.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:24.169951+0000 osd.5 (osd.5) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:26.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:24.169979+0000 osd.5 (osd.5) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:26.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:24.284075+0000 mgr.y (mgr.24481) 561 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:26.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:24.572391+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e363: 8 total, 8 up, 6 in 2024-02-12T17:04:26.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:24.641216+0000 mgr.y (mgr.24481) 562 : cluster [DBG] pgmap v728: 137 pgs: 2 peering, 5 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 4.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s; 47020/196629 objects misplaced (23.913%); 16 MiB/s, 250 objects/s recovering 2024-02-12T17:04:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:24.169951+0000 osd.5 (osd.5) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:24.169979+0000 osd.5 (osd.5) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:24.284075+0000 mgr.y (mgr.24481) 561 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:24.572391+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e363: 8 total, 8 up, 6 in 2024-02-12T17:04:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:24.641216+0000 mgr.y (mgr.24481) 562 : cluster [DBG] pgmap v728: 137 pgs: 2 peering, 5 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 4.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s; 47020/196629 objects misplaced (23.913%); 16 MiB/s, 250 objects/s recovering 2024-02-12T17:04:26.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:26.183 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:26.670 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:25.139038+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:26.670 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:25.139082+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:26.670 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:25.570336+0000 mon.a (mon.0) 1414 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:04:26.671 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:25.581161+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e364: 8 total, 8 up, 6 in 2024-02-12T17:04:26.671 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:26.382743+0000 mon.a (mon.0) 1416 : audit [DBG] from='client.? v1:172.21.15.33:0/3708643817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:26.745 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:26.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:26.777 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T17:04:26.850 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:25.139038+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:25.139082+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:25.570336+0000 mon.a (mon.0) 1414 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:25.581161+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e364: 8 total, 8 up, 6 in 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:26.382743+0000 mon.a (mon.0) 1416 : audit [DBG] from='client.? v1:172.21.15.33:0/3708643817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:25.139038+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:25.139082+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:25.570336+0000 mon.a (mon.0) 1414 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:04:26.855 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:25.581161+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e364: 8 total, 8 up, 6 in 2024-02-12T17:04:26.856 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:26.382743+0000 mon.a (mon.0) 1416 : audit [DBG] from='client.? v1:172.21.15.33:0/3708643817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:04:26.881 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:04:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:04:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:04:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:04:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:04:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:04:26.882 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:04:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:26.971 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:26.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:27.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:27.145 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:27.186 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:27.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:27.291 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:27.339 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:27.547 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:27.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:27.624 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:27.655 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:27.674 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:27.726 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:27.775 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:27.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:26.592661+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e365: 8 total, 8 up, 6 in 2024-02-12T17:04:27.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:26.641716+0000 mgr.y (mgr.24481) 563 : cluster [DBG] pgmap v731: 136 pgs: 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 4.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s; 47020/196629 objects misplaced (23.913%); 16 MiB/s, 250 objects/s recovering 2024-02-12T17:04:27.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:26.642078+0000 mon.c (mon.2) 206 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:04:27.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:26.643164+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:04:27.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:26.592661+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e365: 8 total, 8 up, 6 in 2024-02-12T17:04:27.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:26.641716+0000 mgr.y (mgr.24481) 563 : cluster [DBG] pgmap v731: 136 pgs: 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 4.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s; 47020/196629 objects misplaced (23.913%); 16 MiB/s, 250 objects/s recovering 2024-02-12T17:04:27.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:26.642078+0000 mon.c (mon.2) 206 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:04:27.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:26.643164+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:04:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:27.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:27.883 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:27.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:26.592661+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e365: 8 total, 8 up, 6 in 2024-02-12T17:04:27.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:26.641716+0000 mgr.y (mgr.24481) 563 : cluster [DBG] pgmap v731: 136 pgs: 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 4.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s; 47020/196629 objects misplaced (23.913%); 16 MiB/s, 250 objects/s recovering 2024-02-12T17:04:27.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:26.642078+0000 mon.c (mon.2) 206 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:04:27.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:26.643164+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:04:28.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:27.602054+0000 mon.a (mon.0) 1419 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T17:04:28.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:27.602133+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e366: 8 total, 8 up, 6 in 2024-02-12T17:04:29.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:27.602054+0000 mon.a (mon.0) 1419 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T17:04:29.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:27.602133+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e366: 8 total, 8 up, 6 in 2024-02-12T17:04:29.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:27.602054+0000 mon.a (mon.0) 1419 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T17:04:29.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:27.602133+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e366: 8 total, 8 up, 6 in 2024-02-12T17:04:29.100 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:29.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:29.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:29.364 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:29.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:29.472 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:29.512 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:29.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:04:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:29.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:29.664 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:28.635116+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e367: 8 total, 8 up, 6 in 2024-02-12T17:04:29.757 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:28.635116+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e367: 8 total, 8 up, 6 in 2024-02-12T17:04:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:28.642258+0000 mgr.y (mgr.24481) 564 : cluster [DBG] pgmap v734: 136 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 3.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 396 op/s; 45563/191865 objects misplaced (23.747%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T17:04:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:28.635116+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e367: 8 total, 8 up, 6 in 2024-02-12T17:04:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:28.642258+0000 mgr.y (mgr.24481) 564 : cluster [DBG] pgmap v734: 136 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 3.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 396 op/s; 45563/191865 objects misplaced (23.747%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T17:04:29.820 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:29.852 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:29.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:29.955 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:29.991 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:29.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:28.642258+0000 mgr.y (mgr.24481) 564 : cluster [DBG] pgmap v734: 136 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 121 active+clean; 3.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 396 op/s; 45563/191865 objects misplaced (23.747%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T17:04:30.142 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:30.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:30.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:30.279 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:30.305 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:30.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:30.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:30.439 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:30.667 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:29.664392+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e368: 8 total, 8 up, 6 in 2024-02-12T17:04:30.667 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:30.642883+0000 mon.c (mon.2) 207 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:04:30.667 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:30.643763+0000 mon.a (mon.0) 1423 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:04:30.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:29.664392+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e368: 8 total, 8 up, 6 in 2024-02-12T17:04:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:30.642883+0000 mon.c (mon.2) 207 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:04:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:30.643763+0000 mon.a (mon.0) 1423 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:04:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:29.664392+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e368: 8 total, 8 up, 6 in 2024-02-12T17:04:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:30.642883+0000 mon.c (mon.2) 207 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:04:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:30.643763+0000 mon.a (mon.0) 1423 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:04:31.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:04:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:30] "GET /metrics HTTP/1.1" 200 176139 "" "Prometheus/2.33.4" 2024-02-12T17:04:31.699 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7, 3] out_osds: [1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:04:31.699 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:04:31.699 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 4, 5, 7, 3] 2024-02-12T17:04:31.700 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd out 4 2024-02-12T17:04:31.801 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:31.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:31.951 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:31.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:30.642644+0000 mgr.y (mgr.24481) 565 : cluster [DBG] pgmap v736: 135 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 1 clean+premerge+peered, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 915 op/s; 43347/185523 objects misplaced (23.365%); 16 MiB/s, 257 objects/s recovering 2024-02-12T17:04:31.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:30.664989+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T17:04:31.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:30.665059+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e369: 8 total, 8 up, 6 in 2024-02-12T17:04:31.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:30.642644+0000 mgr.y (mgr.24481) 565 : cluster [DBG] pgmap v736: 135 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 1 clean+premerge+peered, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 915 op/s; 43347/185523 objects misplaced (23.365%); 16 MiB/s, 257 objects/s recovering 2024-02-12T17:04:31.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:30.664989+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T17:04:31.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:30.665059+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e369: 8 total, 8 up, 6 in 2024-02-12T17:04:31.997 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:30.642644+0000 mgr.y (mgr.24481) 565 : cluster [DBG] pgmap v736: 135 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 1 clean+premerge+peered, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 915 op/s; 43347/185523 objects misplaced (23.365%); 16 MiB/s, 257 objects/s recovering 2024-02-12T17:04:31.997 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:30.664989+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T17:04:31.997 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:30.665059+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e369: 8 total, 8 up, 6 in 2024-02-12T17:04:32.002 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:32.264 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:32.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:32.369 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:32.515 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:32.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:32.609 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:04:32.609 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:04:32.609 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-02-12T17:04:32.609 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:04:32.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:32.674 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:32.730 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:04:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:04:32.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:31.682556+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e370: 8 total, 8 up, 6 in 2024-02-12T17:04:32.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:32.490313+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:32.936 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:32.948 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:31.682556+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e370: 8 total, 8 up, 6 in 2024-02-12T17:04:32.948 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:32.490313+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:32.948 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:31.682556+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e370: 8 total, 8 up, 6 in 2024-02-12T17:04:32.948 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:32.490313+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:32.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:33.050 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:33.095 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:33.469 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:33.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:33.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T17:04:33.577 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:33.622 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:33.662 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:33.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:33.721 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:33.721 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:33.771 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:33.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:32.643191+0000 mgr.y (mgr.24481) 566 : cluster [DBG] pgmap v739: 135 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 1 clean+premerge+peered, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 528 op/s; 43347/185523 objects misplaced (23.365%); 14 MiB/s, 231 objects/s recovering 2024-02-12T17:04:33.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:32.736612+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e371: 8 total, 8 up, 6 in 2024-02-12T17:04:33.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:33.135994+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.33:0/3085915659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:33.776 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:33.136469+0000 mon.a (mon.0) 1429 : audit [INF] from='client.25480 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:33.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:32.643191+0000 mgr.y (mgr.24481) 566 : cluster [DBG] pgmap v739: 135 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 1 clean+premerge+peered, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 528 op/s; 43347/185523 objects misplaced (23.365%); 14 MiB/s, 231 objects/s recovering 2024-02-12T17:04:33.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:32.736612+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e371: 8 total, 8 up, 6 in 2024-02-12T17:04:33.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:33.135994+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.33:0/3085915659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:33.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:33.136469+0000 mon.a (mon.0) 1429 : audit [INF] from='client.25480 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:33.794 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:04:33.830 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:33.867 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:33.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:33.971 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:34.013 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:34.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:32.643191+0000 mgr.y (mgr.24481) 566 : cluster [DBG] pgmap v739: 135 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 1 clean+premerge+peered, 3 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 528 op/s; 43347/185523 objects misplaced (23.365%); 14 MiB/s, 231 objects/s recovering 2024-02-12T17:04:34.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:32.736612+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e371: 8 total, 8 up, 6 in 2024-02-12T17:04:34.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:33.135994+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.33:0/3085915659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:34.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:33.136469+0000 mon.a (mon.0) 1429 : audit [INF] from='client.25480 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:04:34.379 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:34.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:34.487 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:34.525 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:34.728 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:34.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:34.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:04:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:04:34.835 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:34.868 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:35.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:33.737547+0000 mon.a (mon.0) 1430 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:33.744178+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25480 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:33.744243+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e372: 8 total, 8 up, 6 in 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:34.644067+0000 mon.c (mon.2) 208 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:34.645001+0000 mon.a (mon.0) 1433 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:33.737547+0000 mon.a (mon.0) 1430 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:33.744178+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25480 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:33.744243+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e372: 8 total, 8 up, 6 in 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:34.644067+0000 mon.c (mon.2) 208 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:04:35.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:34.645001+0000 mon.a (mon.0) 1433 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:04:35.248 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:33.737547+0000 mon.a (mon.0) 1430 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:35.249 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:33.744178+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25480 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:04:35.249 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:33.744243+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e372: 8 total, 8 up, 6 in 2024-02-12T17:04:35.249 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:34.644067+0000 mon.c (mon.2) 208 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:04:35.249 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:34.645001+0000 mon.a (mon.0) 1433 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:04:35.524 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:35.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:35.629 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:35.669 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:35.714 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:35.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:35.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:35.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:35.860 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:36.011 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:34.294475+0000 mgr.y (mgr.24481) 567 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:36.011 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:34.643762+0000 mgr.y (mgr.24481) 568 : cluster [DBG] pgmap v742: 134 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 40270/176112 objects misplaced (22.866%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:04:36.011 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:34.776577+0000 mon.a (mon.0) 1434 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T17:04:36.011 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:34.776653+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e373: 8 total, 8 up, 6 in 2024-02-12T17:04:36.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:34.294475+0000 mgr.y (mgr.24481) 567 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:36.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:34.643762+0000 mgr.y (mgr.24481) 568 : cluster [DBG] pgmap v742: 134 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 40270/176112 objects misplaced (22.866%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:04:36.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:34.776577+0000 mon.a (mon.0) 1434 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T17:04:36.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:34.776653+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e373: 8 total, 8 up, 6 in 2024-02-12T17:04:36.016 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:36.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:36.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:36.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:36.203 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:34.294475+0000 mgr.y (mgr.24481) 567 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:36.204 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:34.643762+0000 mgr.y (mgr.24481) 568 : cluster [DBG] pgmap v742: 134 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 40270/176112 objects misplaced (22.866%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:04:36.204 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:34.776577+0000 mon.a (mon.0) 1434 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T17:04:36.204 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:34.776653+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e373: 8 total, 8 up, 6 in 2024-02-12T17:04:36.206 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:36.244 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:36.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:36.352 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:36.410 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:36.438 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:36.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:36.549 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:36.594 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:36.749 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:36.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:36.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:37.006 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:37.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:37.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:35.797125+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e374: 8 total, 8 up, 6 in 2024-02-12T17:04:37.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:35.797125+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e374: 8 total, 8 up, 6 in 2024-02-12T17:04:37.116 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:37.168 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:37.169 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:35.797125+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e374: 8 total, 8 up, 6 in 2024-02-12T17:04:37.205 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:37.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:37.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:37.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:37.367 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:37.392 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:37.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:37.414 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:04:37.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:37.563 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:37.650 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:37.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:37.753 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:37.811 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:37.849 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:37.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:37.957 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:38.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:36.644167+0000 mgr.y (mgr.24481) 569 : cluster [DBG] pgmap v745: 134 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 40270/176112 objects misplaced (22.866%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:04:38.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:36.814485+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e375: 8 total, 8 up, 6 in 2024-02-12T17:04:38.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:36.644167+0000 mgr.y (mgr.24481) 569 : cluster [DBG] pgmap v745: 134 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 40270/176112 objects misplaced (22.866%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:04:38.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:36.814485+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e375: 8 total, 8 up, 6 in 2024-02-12T17:04:38.022 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:38.134 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:38.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:38.147 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:38.150 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:04:38.223 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:04:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:36.644167+0000 mgr.y (mgr.24481) 569 : cluster [DBG] pgmap v745: 134 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 40270/176112 objects misplaced (22.866%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:04:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:36.814485+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e375: 8 total, 8 up, 6 in 2024-02-12T17:04:38.248 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:38.268 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:04:38.281 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:04:38.318 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:38.337 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:04:38.359 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:38.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:38.391 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:04:38.432 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:04:38.466 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:04:38.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:04:38.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:04:38.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:04:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:04:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:04:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:04:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:04:38.472 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:04:38.474 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:38.567 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:38.630 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:38.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:38.738 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:38.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:04:38.305+0000 7fd96a9de700 -1 mon.a@0(leader).osd e376 definitely_dead 0 2024-02-12T17:04:38.777 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:38.844 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:37.826421+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e376: 8 total, 8 up, 6 in 2024-02-12T17:04:38.844 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:38.104669+0000 mon.a (mon.0) 1439 : audit [DBG] from='client.? v1:172.21.15.33:0/4019453256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:38.844 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:38.305928+0000 mon.c (mon.2) 209 : audit [INF] from='client.? v1:172.21.15.33:0/1711768794' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:04:38.844 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:38.306509+0000 mon.a (mon.0) 1440 : audit [INF] from='client.35621 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:04:38.844 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:38.306619+0000 mon.a (mon.0) 1441 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-12T17:04:38.844 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:38.645094+0000 mon.c (mon.2) 210 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:04:38.844 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:38.646012+0000 mon.a (mon.0) 1442 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:04:38.852 INFO:teuthology.orchestra.run.smithi033.stderr:marked out osd.4. 2024-02-12T17:04:38.857 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:38.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:38.969 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:39.004 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:39.024 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:39.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:39.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:37.826421+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e376: 8 total, 8 up, 6 in 2024-02-12T17:04:39.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:38.104669+0000 mon.a (mon.0) 1439 : audit [DBG] from='client.? v1:172.21.15.33:0/4019453256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:39.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:38.305928+0000 mon.c (mon.2) 209 : audit [INF] from='client.? v1:172.21.15.33:0/1711768794' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:04:39.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:38.306509+0000 mon.a (mon.0) 1440 : audit [INF] from='client.35621 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:04:39.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:38.306619+0000 mon.a (mon.0) 1441 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-12T17:04:39.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:38.645094+0000 mon.c (mon.2) 210 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:04:39.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:38.646012+0000 mon.a (mon.0) 1442 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:04:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:37.826421+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e376: 8 total, 8 up, 6 in 2024-02-12T17:04:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:38.104669+0000 mon.a (mon.0) 1439 : audit [DBG] from='client.? v1:172.21.15.33:0/4019453256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:38.305928+0000 mon.c (mon.2) 209 : audit [INF] from='client.? v1:172.21.15.33:0/1711768794' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:04:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:38.306509+0000 mon.a (mon.0) 1440 : audit [INF] from='client.35621 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:04:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:38.306619+0000 mon.a (mon.0) 1441 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-12T17:04:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:38.645094+0000 mon.c (mon.2) 210 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:04:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:38.646012+0000 mon.a (mon.0) 1442 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:04:39.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:39.157 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:39.183 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:39.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:39.286 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:39.313 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:39.415 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:39.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:39.519 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:39.570 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:39.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:39.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:39.705 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:39.727 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:39.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:39.831 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:39.839 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:38.644740+0000 mgr.y (mgr.24481) 570 : cluster [DBG] pgmap v748: 133 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 38821/171513 objects misplaced (22.634%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:04:39.840 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:38.840434+0000 mon.a (mon.0) 1443 : audit [INF] from='client.35621 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-12T17:04:39.840 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:38.840495+0000 mon.a (mon.0) 1444 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:04:39.840 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:38.840528+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e377: 8 total, 8 up, 5 in 2024-02-12T17:04:39.873 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:38.644740+0000 mgr.y (mgr.24481) 570 : cluster [DBG] pgmap v748: 133 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 38821/171513 objects misplaced (22.634%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:04:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:38.840434+0000 mon.a (mon.0) 1443 : audit [INF] from='client.35621 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-12T17:04:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:38.840495+0000 mon.a (mon.0) 1444 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:04:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:38.840528+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e377: 8 total, 8 up, 5 in 2024-02-12T17:04:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:38.644740+0000 mgr.y (mgr.24481) 570 : cluster [DBG] pgmap v748: 133 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 38821/171513 objects misplaced (22.634%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:04:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:38.840434+0000 mon.a (mon.0) 1443 : audit [INF] from='client.35621 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-12T17:04:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:38.840495+0000 mon.a (mon.0) 1444 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:04:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:38.840528+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e377: 8 total, 8 up, 5 in 2024-02-12T17:04:40.539 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:40.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:40.644 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:40.705 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:40.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:40.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:40.840 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:39.845483+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e378: 8 total, 8 up, 5 in 2024-02-12T17:04:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:39.846036+0000 osd.4 (osd.4) 164 : cluster [INF] 9.c continuing backfill to osd.3 from (338'9837,376'13948] 9:31e3e2c0:::benchmark_data_smithi033_131576_object109027:head to 376'13948 2024-02-12T17:04:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:39.852651+0000 osd.2 (osd.2) 162 : cluster [INF] 9.1 continuing backfill to osd.3 from (338'9665,376'13742] MIN to 376'13742 2024-02-12T17:04:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:39.857485+0000 osd.7 (osd.7) 187 : cluster [DBG] 9.e starting backfill to osd.3 from (0'0,0'0] MAX to 376'13954 2024-02-12T17:04:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:39.869462+0000 osd.7 (osd.7) 188 : cluster [INF] 9.e continuing backfill to osd.5 from (338'9993,376'13954] 9:72cdcc03:::benchmark_data_smithi033_131576_object92369:head to 376'13954 2024-02-12T17:04:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:39.845483+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e378: 8 total, 8 up, 5 in 2024-02-12T17:04:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:39.846036+0000 osd.4 (osd.4) 164 : cluster [INF] 9.c continuing backfill to osd.3 from (338'9837,376'13948] 9:31e3e2c0:::benchmark_data_smithi033_131576_object109027:head to 376'13948 2024-02-12T17:04:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:39.852651+0000 osd.2 (osd.2) 162 : cluster [INF] 9.1 continuing backfill to osd.3 from (338'9665,376'13742] MIN to 376'13742 2024-02-12T17:04:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:39.857485+0000 osd.7 (osd.7) 187 : cluster [DBG] 9.e starting backfill to osd.3 from (0'0,0'0] MAX to 376'13954 2024-02-12T17:04:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:39.869462+0000 osd.7 (osd.7) 188 : cluster [INF] 9.e continuing backfill to osd.5 from (338'9993,376'13954] 9:72cdcc03:::benchmark_data_smithi033_131576_object92369:head to 376'13954 2024-02-12T17:04:41.029 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:40] "GET /metrics HTTP/1.1" 200 176161 "" "Prometheus/2.33.4" 2024-02-12T17:04:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:39.845483+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e378: 8 total, 8 up, 5 in 2024-02-12T17:04:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:39.846036+0000 osd.4 (osd.4) 164 : cluster [INF] 9.c continuing backfill to osd.3 from (338'9837,376'13948] 9:31e3e2c0:::benchmark_data_smithi033_131576_object109027:head to 376'13948 2024-02-12T17:04:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:39.852651+0000 osd.2 (osd.2) 162 : cluster [INF] 9.1 continuing backfill to osd.3 from (338'9665,376'13742] MIN to 376'13742 2024-02-12T17:04:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:39.857485+0000 osd.7 (osd.7) 187 : cluster [DBG] 9.e starting backfill to osd.3 from (0'0,0'0] MAX to 376'13954 2024-02-12T17:04:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:39.869462+0000 osd.7 (osd.7) 188 : cluster [INF] 9.e continuing backfill to osd.5 from (338'9993,376'13954] 9:72cdcc03:::benchmark_data_smithi033_131576_object92369:head to 376'13954 2024-02-12T17:04:41.503 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:04:41.503 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:04:41.761 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:42.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:40.645174+0000 mgr.y (mgr.24481) 571 : cluster [DBG] pgmap v751: 133 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 37346/165792 objects misplaced (22.526%); 8.2 MiB/s, 131 objects/s recovering 2024-02-12T17:04:42.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:40.855188+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e379: 8 total, 8 up, 5 in 2024-02-12T17:04:42.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:40.855369+0000 osd.7 (osd.7) 189 : cluster [INF] 5.9 continuing backfill to osd.3 from (65'8,349'15] MIN to 349'15 2024-02-12T17:04:42.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:40.855581+0000 osd.5 (osd.5) 183 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 208'12 2024-02-12T17:04:42.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:40.856504+0000 osd.5 (osd.5) 184 : cluster [DBG] 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 261'14 2024-02-12T17:04:42.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:40.857854+0000 osd.7 (osd.7) 190 : cluster [DBG] 9.d starting backfill to osd.2 from (0'0,0'0] MAX to 376'13415 2024-02-12T17:04:42.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:40.865716+0000 osd.0 (osd.0) 179 : cluster [INF] 9.4 continuing backfill to osd.3 from (331'6700,376'13636] MIN to 376'13636 2024-02-12T17:04:42.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:40.645174+0000 mgr.y (mgr.24481) 571 : cluster [DBG] pgmap v751: 133 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 37346/165792 objects misplaced (22.526%); 8.2 MiB/s, 131 objects/s recovering 2024-02-12T17:04:42.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:40.855188+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e379: 8 total, 8 up, 5 in 2024-02-12T17:04:42.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:40.855369+0000 osd.7 (osd.7) 189 : cluster [INF] 5.9 continuing backfill to osd.3 from (65'8,349'15] MIN to 349'15 2024-02-12T17:04:42.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:40.855581+0000 osd.5 (osd.5) 183 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 208'12 2024-02-12T17:04:42.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:40.856504+0000 osd.5 (osd.5) 184 : cluster [DBG] 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 261'14 2024-02-12T17:04:42.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:40.857854+0000 osd.7 (osd.7) 190 : cluster [DBG] 9.d starting backfill to osd.2 from (0'0,0'0] MAX to 376'13415 2024-02-12T17:04:42.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:40.865716+0000 osd.0 (osd.0) 179 : cluster [INF] 9.4 continuing backfill to osd.3 from (331'6700,376'13636] MIN to 376'13636 2024-02-12T17:04:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:40.645174+0000 mgr.y (mgr.24481) 571 : cluster [DBG] pgmap v751: 133 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 120 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 37346/165792 objects misplaced (22.526%); 8.2 MiB/s, 131 objects/s recovering 2024-02-12T17:04:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:40.855188+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e379: 8 total, 8 up, 5 in 2024-02-12T17:04:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:40.855369+0000 osd.7 (osd.7) 189 : cluster [INF] 5.9 continuing backfill to osd.3 from (65'8,349'15] MIN to 349'15 2024-02-12T17:04:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:40.855581+0000 osd.5 (osd.5) 183 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 208'12 2024-02-12T17:04:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:40.856504+0000 osd.5 (osd.5) 184 : cluster [DBG] 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 261'14 2024-02-12T17:04:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:40.857854+0000 osd.7 (osd.7) 190 : cluster [DBG] 9.d starting backfill to osd.2 from (0'0,0'0] MAX to 376'13415 2024-02-12T17:04:42.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:40.865716+0000 osd.0 (osd.0) 179 : cluster [INF] 9.4 continuing backfill to osd.3 from (331'6700,376'13636] MIN to 376'13636 2024-02-12T17:04:42.550 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T17:04:42.778 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:04:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw-foo-a[65833]: debug 2024-02-12T17:04:42.431+0000 7efdb0cdd700 -1 RGWWatcher::handle_error cookie 94508044027008 err (110) Connection timed out 2024-02-12T17:04:42.896 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:41.867114+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e380: 8 total, 8 up, 5 in 2024-02-12T17:04:42.897 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:42.650106+0000 mon.c (mon.2) 211 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:04:42.897 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:42.651473+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:04:42.897 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:04:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:04:42.913 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:42.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:43.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:04:43.112 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:43.149 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:04:43.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:41.867114+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e380: 8 total, 8 up, 5 in 2024-02-12T17:04:43.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:42.650106+0000 mon.c (mon.2) 211 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:04:43.154 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:42.651473+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:04:43.155 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:41.867114+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e380: 8 total, 8 up, 5 in 2024-02-12T17:04:43.155 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:42.650106+0000 mon.c (mon.2) 211 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:04:43.155 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:42.651473+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:04:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:43.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:04:43.258 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:43.303 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:43.394 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:04:43.465 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:43.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:43.528 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:43.580 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:43.634 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:43.799 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:43.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:43.905 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:43.975 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:44.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:44.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:44.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:42.649727+0000 mgr.y (mgr.24481) 572 : cluster [DBG] pgmap v754: 132 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 119 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1022 B/s rd, 0 B/s wr, 477 op/s; 37346/165792 objects misplaced (22.526%); 5.5 MiB/s, 88 objects/s recovering 2024-02-12T17:04:44.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:42.893784+0000 mon.a (mon.0) 1450 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T17:04:44.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:42.893878+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e381: 8 total, 8 up, 5 in 2024-02-12T17:04:44.206 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:42.649727+0000 mgr.y (mgr.24481) 572 : cluster [DBG] pgmap v754: 132 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 119 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1022 B/s rd, 0 B/s wr, 477 op/s; 37346/165792 objects misplaced (22.526%); 5.5 MiB/s, 88 objects/s recovering 2024-02-12T17:04:44.206 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:42.893784+0000 mon.a (mon.0) 1450 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T17:04:44.206 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:42.893878+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e381: 8 total, 8 up, 5 in 2024-02-12T17:04:44.210 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:42.649727+0000 mgr.y (mgr.24481) 572 : cluster [DBG] pgmap v754: 132 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 119 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1022 B/s rd, 0 B/s wr, 477 op/s; 37346/165792 objects misplaced (22.526%); 5.5 MiB/s, 88 objects/s recovering 2024-02-12T17:04:44.210 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:42.893784+0000 mon.a (mon.0) 1450 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T17:04:44.210 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:42.893878+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e381: 8 total, 8 up, 5 in 2024-02-12T17:04:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:44.258 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:44.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:44.365 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:44.493 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:04:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:04:44.589 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:44.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:44.694 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:44.764 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:44.799 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:44.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:44.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:44.910 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:04:44.910 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:04:44.910 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.control 2024-02-12T17:04:44.910 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.control by 10 min 8 2024-02-12T17:04:44.910 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:04:44.955 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:45.032 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:43.901025+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e382: 8 total, 8 up, 5 in 2024-02-12T17:04:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:45.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:45.133 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:45.190 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:45.235 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:45.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:43.901025+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e382: 8 total, 8 up, 5 in 2024-02-12T17:04:45.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:43.901025+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e382: 8 total, 8 up, 5 in 2024-02-12T17:04:45.348 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:45.392 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:45.601 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:45.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:45.710 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:45.791 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:45.814 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:45.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:45.922 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:45.979 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:46.025 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:04:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:46.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:04:46.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:46.180 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:46.183 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:44.310696+0000 mgr.y (mgr.24481) 573 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:46.184 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:44.650627+0000 mgr.y (mgr.24481) 574 : cluster [DBG] pgmap v757: 132 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 2 peering, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 115 active+clean; 3.2 GiB data, 13 GiB used, 435 GiB / 447 GiB avail; 766 B/s rd, 0 B/s wr, 579 op/s; 38623/158841 objects misplaced (24.316%); 13 MiB/s, 242 objects/s recovering 2024-02-12T17:04:46.184 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:44.916458+0000 mon.a (mon.0) 1453 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:04:46.184 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:44.934394+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e383: 8 total, 8 up, 5 in 2024-02-12T17:04:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:46.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:46.223 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:44.310696+0000 mgr.y (mgr.24481) 573 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:46.223 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:44.650627+0000 mgr.y (mgr.24481) 574 : cluster [DBG] pgmap v757: 132 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 2 peering, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 115 active+clean; 3.2 GiB data, 13 GiB used, 435 GiB / 447 GiB avail; 766 B/s rd, 0 B/s wr, 579 op/s; 38623/158841 objects misplaced (24.316%); 13 MiB/s, 242 objects/s recovering 2024-02-12T17:04:46.224 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:44.916458+0000 mon.a (mon.0) 1453 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:04:46.224 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:44.934394+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e383: 8 total, 8 up, 5 in 2024-02-12T17:04:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:44.310696+0000 mgr.y (mgr.24481) 573 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:44.650627+0000 mgr.y (mgr.24481) 574 : cluster [DBG] pgmap v757: 132 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 2 peering, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 115 active+clean; 3.2 GiB data, 13 GiB used, 435 GiB / 447 GiB avail; 766 B/s rd, 0 B/s wr, 579 op/s; 38623/158841 objects misplaced (24.316%); 13 MiB/s, 242 objects/s recovering 2024-02-12T17:04:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:44.916458+0000 mon.a (mon.0) 1453 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:04:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:44.934394+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e383: 8 total, 8 up, 5 in 2024-02-12T17:04:46.312 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:46.368 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:46.496 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:46.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:46.578 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:46.578 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:04:46.578 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:04:46.579 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:46.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:46.654 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:46.686 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:46.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:46.792 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:46.843 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:47.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:45.988349+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e384: 8 total, 8 up, 5 in 2024-02-12T17:04:47.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:46.291643+0000 mon.a (mon.0) 1456 : audit [INF] from='client.? v1:172.21.15.33:0/1563376345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:04:47.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:46.297601+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e385: 8 total, 8 up, 5 in 2024-02-12T17:04:47.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:46.655032+0000 mon.c (mon.2) 212 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:04:47.115 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:46.662886+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:04:47.117 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:45.988349+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e384: 8 total, 8 up, 5 in 2024-02-12T17:04:47.117 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:46.291643+0000 mon.a (mon.0) 1456 : audit [INF] from='client.? v1:172.21.15.33:0/1563376345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:04:47.117 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:46.297601+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e385: 8 total, 8 up, 5 in 2024-02-12T17:04:47.117 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:46.655032+0000 mon.c (mon.2) 212 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:04:47.117 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:46.662886+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:04:47.336 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:04:47.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:45.988349+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e384: 8 total, 8 up, 5 in 2024-02-12T17:04:47.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:46.291643+0000 mon.a (mon.0) 1456 : audit [INF] from='client.? v1:172.21.15.33:0/1563376345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:04:47.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:46.297601+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e385: 8 total, 8 up, 5 in 2024-02-12T17:04:47.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:46.655032+0000 mon.c (mon.2) 212 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:04:47.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:46.662886+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:04:48.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:46.654630+0000 mgr.y (mgr.24481) 575 : cluster [DBG] pgmap v761: 131 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 peering, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 115 active+clean; 3.2 GiB data, 13 GiB used, 435 GiB / 447 GiB avail; 817 B/s rd, 0 B/s wr, 617 op/s; 38623/158841 objects misplaced (24.316%); 14 MiB/s, 258 objects/s recovering 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:47.294911+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:47.305261+0000 mon.a (mon.0) 1460 : audit [INF] from='client.? v1:172.21.15.33:0/1563376345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:47.305328+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:47.305399+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e386: 8 total, 8 up, 5 in 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:46.654630+0000 mgr.y (mgr.24481) 575 : cluster [DBG] pgmap v761: 131 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 peering, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 115 active+clean; 3.2 GiB data, 13 GiB used, 435 GiB / 447 GiB avail; 817 B/s rd, 0 B/s wr, 617 op/s; 38623/158841 objects misplaced (24.316%); 14 MiB/s, 258 objects/s recovering 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:47.294911+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:47.305261+0000 mon.a (mon.0) 1460 : audit [INF] from='client.? v1:172.21.15.33:0/1563376345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:47.305328+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T17:04:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:47.305399+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e386: 8 total, 8 up, 5 in 2024-02-12T17:04:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:46.654630+0000 mgr.y (mgr.24481) 575 : cluster [DBG] pgmap v761: 131 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 peering, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 115 active+clean; 3.2 GiB data, 13 GiB used, 435 GiB / 447 GiB avail; 817 B/s rd, 0 B/s wr, 617 op/s; 38623/158841 objects misplaced (24.316%); 14 MiB/s, 258 objects/s recovering 2024-02-12T17:04:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:47.294911+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:04:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:47.305261+0000 mon.a (mon.0) 1460 : audit [INF] from='client.? v1:172.21.15.33:0/1563376345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:04:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:47.305328+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T17:04:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:47.305399+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e386: 8 total, 8 up, 5 in 2024-02-12T17:04:49.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:48.317840+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e387: 8 total, 8 up, 5 in 2024-02-12T17:04:49.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:48.655387+0000 mgr.y (mgr.24481) 576 : cluster [DBG] pgmap v764: 131 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 2 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 472 op/s; 36809/153564 objects misplaced (23.970%); 1.9 MiB/s, 46 objects/s recovering 2024-02-12T17:04:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:48.317840+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e387: 8 total, 8 up, 5 in 2024-02-12T17:04:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:48.655387+0000 mgr.y (mgr.24481) 576 : cluster [DBG] pgmap v764: 131 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 2 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 472 op/s; 36809/153564 objects misplaced (23.970%); 1.9 MiB/s, 46 objects/s recovering 2024-02-12T17:04:49.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:48.317840+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e387: 8 total, 8 up, 5 in 2024-02-12T17:04:49.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:48.655387+0000 mgr.y (mgr.24481) 576 : cluster [DBG] pgmap v764: 131 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 2 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 472 op/s; 36809/153564 objects misplaced (23.970%); 1.9 MiB/s, 46 objects/s recovering 2024-02-12T17:04:50.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:49.332226+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e388: 8 total, 8 up, 5 in 2024-02-12T17:04:50.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:49.332226+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e388: 8 total, 8 up, 5 in 2024-02-12T17:04:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:49.332226+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e388: 8 total, 8 up, 5 in 2024-02-12T17:04:51.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:04:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:50] "GET /metrics HTTP/1.1" 200 176162 "" "Prometheus/2.33.4" 2024-02-12T17:04:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:50.357120+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e389: 8 total, 8 up, 5 in 2024-02-12T17:04:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:50.655823+0000 mgr.y (mgr.24481) 577 : cluster [DBG] pgmap v767: 130 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s; 35551/146922 objects misplaced (24.197%); 2.7 MiB/s, 58 objects/s recovering 2024-02-12T17:04:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:50.656124+0000 mon.c (mon.2) 213 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:04:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:50.657118+0000 mon.a (mon.0) 1466 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:04:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:51.303424+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T17:04:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:51.303488+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e390: 8 total, 8 up, 5 in 2024-02-12T17:04:51.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:50.357120+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e389: 8 total, 8 up, 5 in 2024-02-12T17:04:51.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:50.655823+0000 mgr.y (mgr.24481) 577 : cluster [DBG] pgmap v767: 130 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s; 35551/146922 objects misplaced (24.197%); 2.7 MiB/s, 58 objects/s recovering 2024-02-12T17:04:51.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:50.656124+0000 mon.c (mon.2) 213 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:04:51.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:50.657118+0000 mon.a (mon.0) 1466 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:04:51.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:51.303424+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T17:04:51.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:51.303488+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e390: 8 total, 8 up, 5 in 2024-02-12T17:04:51.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:50.357120+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e389: 8 total, 8 up, 5 in 2024-02-12T17:04:51.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:50.655823+0000 mgr.y (mgr.24481) 577 : cluster [DBG] pgmap v767: 130 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s; 35551/146922 objects misplaced (24.197%); 2.7 MiB/s, 58 objects/s recovering 2024-02-12T17:04:51.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:50.656124+0000 mon.c (mon.2) 213 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:04:51.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:50.657118+0000 mon.a (mon.0) 1466 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:04:51.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:51.303424+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T17:04:51.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:51.303488+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e390: 8 total, 8 up, 5 in 2024-02-12T17:04:51.966 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:51.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:52.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:52.169 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:52.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:52.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:52.322 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:52.372 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:52.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:52.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:52.560 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:52.596 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:52.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:52.707 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:52.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:04:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:04:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:04:52.741 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:53.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:52.338282+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e391: 8 total, 8 up, 5 in 2024-02-12T17:04:53.745 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:52.656636+0000 mgr.y (mgr.24481) 578 : cluster [DBG] pgmap v770: 130 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 554 op/s; 35551/146922 objects misplaced (24.197%); 944 KiB/s, 14 objects/s recovering 2024-02-12T17:04:53.746 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:53.033323+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:53.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:52.338282+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e391: 8 total, 8 up, 5 in 2024-02-12T17:04:53.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:52.656636+0000 mgr.y (mgr.24481) 578 : cluster [DBG] pgmap v770: 130 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 554 op/s; 35551/146922 objects misplaced (24.197%); 944 KiB/s, 14 objects/s recovering 2024-02-12T17:04:53.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:53.033323+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:52.338282+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e391: 8 total, 8 up, 5 in 2024-02-12T17:04:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:52.656636+0000 mgr.y (mgr.24481) 578 : cluster [DBG] pgmap v770: 130 pgs: 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 1 clean+premerge+peered, 7 active+remapped+backfill_wait, 3 active+remapped+backfilling, 114 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 554 op/s; 35551/146922 objects misplaced (24.197%); 944 KiB/s, 14 objects/s recovering 2024-02-12T17:04:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:53.033323+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:54.359 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:53.352255+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e392: 8 total, 8 up, 5 in 2024-02-12T17:04:54.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:04:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:04:54.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:53.352255+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e392: 8 total, 8 up, 5 in 2024-02-12T17:04:54.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:53.352255+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e392: 8 total, 8 up, 5 in 2024-02-12T17:04:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:54.322492+0000 mgr.y (mgr.24481) 579 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:54.374666+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e393: 8 total, 8 up, 5 in 2024-02-12T17:04:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:54.657036+0000 mgr.y (mgr.24481) 580 : cluster [DBG] pgmap v773: 129 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 37523/134757 objects misplaced (27.845%); 3.3 MiB/s, 57 objects/s recovering 2024-02-12T17:04:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:54.657370+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:54.658372+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:55.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:54.322492+0000 mgr.y (mgr.24481) 579 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:55.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:54.374666+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e393: 8 total, 8 up, 5 in 2024-02-12T17:04:55.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:54.657036+0000 mgr.y (mgr.24481) 580 : cluster [DBG] pgmap v773: 129 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 37523/134757 objects misplaced (27.845%); 3.3 MiB/s, 57 objects/s recovering 2024-02-12T17:04:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:54.657370+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:55.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:54.658372+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:54.322492+0000 mgr.y (mgr.24481) 579 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:04:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:54.374666+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e393: 8 total, 8 up, 5 in 2024-02-12T17:04:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:54.657036+0000 mgr.y (mgr.24481) 580 : cluster [DBG] pgmap v773: 129 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 37523/134757 objects misplaced (27.845%); 3.3 MiB/s, 57 objects/s recovering 2024-02-12T17:04:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:54.657370+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:55.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:54.658372+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:04:56.171 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T17:04:56.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:55.386939+0000 mon.a (mon.0) 1474 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:04:56.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:55.387004+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e394: 8 total, 8 up, 5 in 2024-02-12T17:04:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:55.386939+0000 mon.a (mon.0) 1474 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:04:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:55.387004+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e394: 8 total, 8 up, 5 in 2024-02-12T17:04:56.606 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:56.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:56.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:04:56.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:55.386939+0000 mon.a (mon.0) 1474 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:04:56.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:55.387004+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e394: 8 total, 8 up, 5 in 2024-02-12T17:04:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:57.128 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:04:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:57.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:04:57.237 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:04:57.291 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:57.383 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:04:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:57.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:04:57.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:57.495 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:04:57.570 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:57.628 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:57.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:57.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:57.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:57.751 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:56.390394+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e395: 8 total, 8 up, 5 in 2024-02-12T17:04:57.751 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:56.657577+0000 mgr.y (mgr.24481) 581 : cluster [DBG] pgmap v776: 129 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 37523/134757 objects misplaced (27.845%); 3.3 MiB/s, 57 objects/s recovering 2024-02-12T17:04:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:56.390394+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e395: 8 total, 8 up, 5 in 2024-02-12T17:04:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:56.657577+0000 mgr.y (mgr.24481) 581 : cluster [DBG] pgmap v776: 129 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 37523/134757 objects misplaced (27.845%); 3.3 MiB/s, 57 objects/s recovering 2024-02-12T17:04:57.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:56.390394+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e395: 8 total, 8 up, 5 in 2024-02-12T17:04:57.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:56.657577+0000 mgr.y (mgr.24481) 581 : cluster [DBG] pgmap v776: 129 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 37523/134757 objects misplaced (27.845%); 3.3 MiB/s, 57 objects/s recovering 2024-02-12T17:04:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:57.881 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:57.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:57.917 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:04:57.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:58.035 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:58.059 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:58.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:58.153 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:04:58.162 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:04:58.166 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:04:58.231 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:04:58.234 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:58.264 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:04:58.287 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:04:58.290 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:04:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:58.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:04:58.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:04:58.358 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:04:58.393 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:04:58.399 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:04:58.434 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:04:58.445 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:58.478 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:04:58.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:57.419507+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e396: 8 total, 8 up, 5 in 2024-02-12T17:04:58.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:58.058724+0000 mon.a (mon.0) 1478 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:58.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:58.129385+0000 mgr.y (mgr.24481) 582 : audit [DBG] from='client.15690 v1:172.21.15.33:0/587176985' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:04:58.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:04:58.181229+0000 mon.a (mon.0) 1479 : audit [DBG] from='client.? v1:172.21.15.33:0/3205372144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:58.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:04:58.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:04:58.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:04:58.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:04:58.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:04:58.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:04:58.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:04:58.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:04:58.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:57.419507+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e396: 8 total, 8 up, 5 in 2024-02-12T17:04:58.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:58.058724+0000 mon.a (mon.0) 1478 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:58.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:58.129385+0000 mgr.y (mgr.24481) 582 : audit [DBG] from='client.15690 v1:172.21.15.33:0/587176985' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:04:58.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:04:58.181229+0000 mon.a (mon.0) 1479 : audit [DBG] from='client.? v1:172.21.15.33:0/3205372144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:58.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:57.419507+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e396: 8 total, 8 up, 5 in 2024-02-12T17:04:58.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:58.058724+0000 mon.a (mon.0) 1478 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:04:58.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:58.129385+0000 mgr.y (mgr.24481) 582 : audit [DBG] from='client.15690 v1:172.21.15.33:0/587176985' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:04:58.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:04:58.181229+0000 mon.a (mon.0) 1479 : audit [DBG] from='client.? v1:172.21.15.33:0/3205372144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:04:58.921 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:04:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:58.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:04:59.026 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:04:59.106 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:59.147 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:04:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:59.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:04:59.253 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:04:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:59.370 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:04:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:04:59.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:04:59.474 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:04:59.519 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:04:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:58.440557+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e397: 8 total, 8 up, 5 in 2024-02-12T17:04:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:04:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:04:58.658186+0000 mgr.y (mgr.24481) 583 : cluster [DBG] pgmap v779: 128 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 116 op/s; 36858/133353 objects misplaced (27.639%) 2024-02-12T17:04:59.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:58.440557+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e397: 8 total, 8 up, 5 in 2024-02-12T17:04:59.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:04:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:04:58.658186+0000 mgr.y (mgr.24481) 583 : cluster [DBG] pgmap v779: 128 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 116 op/s; 36858/133353 objects misplaced (27.639%) 2024-02-12T17:04:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:58.440557+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e397: 8 total, 8 up, 5 in 2024-02-12T17:04:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:04:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:04:58.658186+0000 mgr.y (mgr.24481) 583 : cluster [DBG] pgmap v779: 128 pgs: 1 peering, 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 116 op/s; 36858/133353 objects misplaced (27.639%) 2024-02-12T17:04:59.840 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":777,"stamp":"2024-02-12T17:04:57.434146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2931168956,"num_objects":44919,"num_object_clones":0,"num_object_copies":134757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":37523,"num_objects_unfound":0,"num_objects_dirty":44919,"num_whiteouts":0,"num_read":1338,"num_read_kb":1090,"num_write":231163,"num_write_kb":8805610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13443,"num_bytes_recovered":836058550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70417,"ondisk_log_size":70417,"up":384,"acting":384,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":401,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":11711432,"kb_used_data":8110304,"kb_used_omap":0,"kb_used_meta":3601024,"kb_avail":457014328,"statfs":{"total":479975178240,"available":467982671872,"internally_reserved":0,"allocated":8304951296,"data_stored":8302519182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3687448576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-265748480,"num_objects":-4055,"num_object_clones":0,"num_object_copies":-12165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1972,"num_objects_unfound":0,"num_objects_dirty":-4055,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4055,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":13959168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.080144"},"pg_stats":[{"pgid":"2.1f","version":"0'0","reported_seq":81,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356775+0000","last_change":"2024-02-12T17:04:39.916313+0000","last_active":"2024-02-12T17:04:52.356775+0000","last_peered":"2024-02-12T17:04:52.356775+0000","last_clean":"2024-02-12T17:04:52.356775+0000","last_became_active":"2024-02-12T17:04:39.916010+0000","last_became_peered":"2024-02-12T17:04:39.916010+0000","last_unstale":"2024-02-12T17:04:52.356775+0000","last_undegraded":"2024-02-12T17:04:52.356775+0000","last_fullsized":"2024-02-12T17:04:52.356775+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.732293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.732293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":70,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346781+0000","last_change":"2024-02-12T17:03:52.280899+0000","last_active":"2024-02-12T17:04:52.346781+0000","last_peered":"2024-02-12T17:04:52.346781+0000","last_clean":"2024-02-12T17:04:52.346781+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:04:52.346781+0000","last_undegraded":"2024-02-12T17:04:52.346781+0000","last_fullsized":"2024-02-12T17:04:52.346781+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:01:26.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.047527+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":54,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345512+0000","last_change":"2024-02-12T17:03:50.545988+0000","last_active":"2024-02-12T17:04:52.345512+0000","last_peered":"2024-02-12T17:04:52.345512+0000","last_clean":"2024-02-12T17:04:52.345512+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:04:52.345512+0000","last_undegraded":"2024-02-12T17:04:52.345512+0000","last_fullsized":"2024-02-12T17:04:52.345512+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.176431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":445,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.343463+0000","last_change":"2024-02-12T17:04:41.353781+0000","last_active":"2024-02-12T17:04:52.343463+0000","last_peered":"2024-02-12T17:04:52.343463+0000","last_clean":"2024-02-12T17:04:52.343463+0000","last_became_active":"2024-02-12T17:04:39.873280+0000","last_became_peered":"2024-02-12T17:04:39.873280+0000","last_unstale":"2024-02-12T17:04:52.343463+0000","last_undegraded":"2024-02-12T17:04:52.343463+0000","last_fullsized":"2024-02-12T17:04:52.343463+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:01:45.154164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":387,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359503+0000","last_change":"2024-02-12T17:03:50.565273+0000","last_active":"2024-02-12T17:04:52.359503+0000","last_peered":"2024-02-12T17:04:52.359503+0000","last_clean":"2024-02-12T17:04:52.359503+0000","last_became_active":"2024-02-12T17:03:50.562938+0000","last_became_peered":"2024-02-12T17:03:50.562938+0000","last_unstale":"2024-02-12T17:04:52.359503+0000","last_undegraded":"2024-02-12T17:04:52.359503+0000","last_fullsized":"2024-02-12T17:04:52.359503+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:37.813008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:01:37.813008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":75,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344311+0000","last_change":"2024-02-12T17:04:42.260018+0000","last_active":"2024-02-12T17:04:52.344311+0000","last_peered":"2024-02-12T17:04:52.344311+0000","last_clean":"2024-02-12T17:04:52.344311+0000","last_became_active":"2024-02-12T17:04:39.889725+0000","last_became_peered":"2024-02-12T17:04:39.889725+0000","last_unstale":"2024-02-12T17:04:52.344311+0000","last_undegraded":"2024-02-12T17:04:52.344311+0000","last_fullsized":"2024-02-12T17:04:52.344311+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":361,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.355996+0000","last_change":"2024-02-12T17:02:34.502434+0000","last_active":"2024-02-12T17:04:52.355996+0000","last_peered":"2024-02-12T17:04:52.355996+0000","last_clean":"2024-02-12T17:04:52.355996+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:04:52.355996+0000","last_undegraded":"2024-02-12T17:04:52.355996+0000","last_fullsized":"2024-02-12T17:04:52.355996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.502332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":83,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345526+0000","last_change":"2024-02-12T17:04:40.923654+0000","last_active":"2024-02-12T17:04:52.345526+0000","last_peered":"2024-02-12T17:04:52.345526+0000","last_clean":"2024-02-12T17:04:52.345526+0000","last_became_active":"2024-02-12T17:04:39.865216+0000","last_became_peered":"2024-02-12T17:04:39.865216+0000","last_unstale":"2024-02-12T17:04:52.345526+0000","last_undegraded":"2024-02-12T17:04:52.345526+0000","last_fullsized":"2024-02-12T17:04:52.345526+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":63,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344139+0000","last_change":"2024-02-12T17:04:39.890400+0000","last_active":"2024-02-12T17:04:52.344139+0000","last_peered":"2024-02-12T17:04:52.344139+0000","last_clean":"2024-02-12T17:04:52.344139+0000","last_became_active":"2024-02-12T17:04:39.887235+0000","last_became_peered":"2024-02-12T17:04:39.887235+0000","last_unstale":"2024-02-12T17:04:52.344139+0000","last_undegraded":"2024-02-12T17:04:52.344139+0000","last_fullsized":"2024-02-12T17:04:52.344139+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":87,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344199+0000","last_change":"2024-02-12T17:04:40.319567+0000","last_active":"2024-02-12T17:04:52.344199+0000","last_peered":"2024-02-12T17:04:52.344199+0000","last_clean":"2024-02-12T17:04:52.344199+0000","last_became_active":"2024-02-12T17:04:39.864612+0000","last_became_peered":"2024-02-12T17:04:39.864612+0000","last_unstale":"2024-02-12T17:04:52.344199+0000","last_undegraded":"2024-02-12T17:04:52.344199+0000","last_fullsized":"2024-02-12T17:04:52.344199+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":29,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.355512+0000","last_change":"2024-02-12T17:04:39.888727+0000","last_active":"2024-02-12T17:04:52.355512+0000","last_peered":"2024-02-12T17:04:52.355512+0000","last_clean":"2024-02-12T17:04:52.355512+0000","last_became_active":"2024-02-12T17:04:39.888398+0000","last_became_peered":"2024-02-12T17:04:39.888398+0000","last_unstale":"2024-02-12T17:04:52.355512+0000","last_undegraded":"2024-02-12T17:04:52.355512+0000","last_fullsized":"2024-02-12T17:04:52.355512+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.157357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.157357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":95,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344402+0000","last_change":"2024-02-12T17:04:41.433996+0000","last_active":"2024-02-12T17:04:52.344402+0000","last_peered":"2024-02-12T17:04:52.344402+0000","last_clean":"2024-02-12T17:04:52.344402+0000","last_became_active":"2024-02-12T17:04:39.872998+0000","last_became_peered":"2024-02-12T17:04:39.872998+0000","last_unstale":"2024-02-12T17:04:52.344402+0000","last_undegraded":"2024-02-12T17:04:52.344402+0000","last_fullsized":"2024-02-12T17:04:52.344402+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":54,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.347086+0000","last_change":"2024-02-12T17:03:50.533143+0000","last_active":"2024-02-12T17:04:52.347086+0000","last_peered":"2024-02-12T17:04:52.347086+0000","last_clean":"2024-02-12T17:04:52.347086+0000","last_became_active":"2024-02-12T17:03:50.532233+0000","last_became_peered":"2024-02-12T17:03:50.532233+0000","last_unstale":"2024-02-12T17:04:52.347086+0000","last_undegraded":"2024-02-12T17:04:52.347086+0000","last_fullsized":"2024-02-12T17:04:52.347086+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":248,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.372793+0000","last_change":"2024-02-12T17:03:50.670847+0000","last_active":"2024-02-12T17:04:52.372793+0000","last_peered":"2024-02-12T17:04:52.372793+0000","last_clean":"2024-02-12T17:04:52.372793+0000","last_became_active":"2024-02-12T17:03:50.570118+0000","last_became_peered":"2024-02-12T17:03:50.570118+0000","last_unstale":"2024-02-12T17:04:52.372793+0000","last_undegraded":"2024-02-12T17:04:52.372793+0000","last_fullsized":"2024-02-12T17:04:52.372793+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:34.738385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.738385+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":367,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357590+0000","last_change":"2024-02-12T17:04:39.880788+0000","last_active":"2024-02-12T17:04:52.357590+0000","last_peered":"2024-02-12T17:04:52.357590+0000","last_clean":"2024-02-12T17:04:52.357590+0000","last_became_active":"2024-02-12T17:04:39.880094+0000","last_became_peered":"2024-02-12T17:04:39.880094+0000","last_unstale":"2024-02-12T17:04:52.357590+0000","last_undegraded":"2024-02-12T17:04:52.357590+0000","last_fullsized":"2024-02-12T17:04:52.357590+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":406,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356491+0000","last_change":"2024-02-12T17:03:51.528465+0000","last_active":"2024-02-12T17:04:52.356491+0000","last_peered":"2024-02-12T17:04:52.356491+0000","last_clean":"2024-02-12T17:04:52.356491+0000","last_became_active":"2024-02-12T17:03:50.576714+0000","last_became_peered":"2024-02-12T17:03:50.576714+0000","last_unstale":"2024-02-12T17:04:52.356491+0000","last_undegraded":"2024-02-12T17:04:52.356491+0000","last_fullsized":"2024-02-12T17:04:52.356491+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:01:42.076584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.076584+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":63,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344333+0000","last_change":"2024-02-12T17:04:39.877511+0000","last_active":"2024-02-12T17:04:52.344333+0000","last_peered":"2024-02-12T17:04:52.344333+0000","last_clean":"2024-02-12T17:04:52.344333+0000","last_became_active":"2024-02-12T17:04:39.877005+0000","last_became_peered":"2024-02-12T17:04:39.877005+0000","last_unstale":"2024-02-12T17:04:52.344333+0000","last_undegraded":"2024-02-12T17:04:52.344333+0000","last_fullsized":"2024-02-12T17:04:52.344333+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":379,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.354371+0000","last_change":"2024-02-12T17:04:39.889899+0000","last_active":"2024-02-12T17:04:52.354371+0000","last_peered":"2024-02-12T17:04:52.354371+0000","last_clean":"2024-02-12T17:04:52.354371+0000","last_became_active":"2024-02-12T17:04:39.889158+0000","last_became_peered":"2024-02-12T17:04:39.889158+0000","last_unstale":"2024-02-12T17:04:52.354371+0000","last_undegraded":"2024-02-12T17:04:52.354371+0000","last_fullsized":"2024-02-12T17:04:52.354371+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:51.269648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.269648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":164,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.365434+0000","last_change":"2024-02-12T17:04:39.885628+0000","last_active":"2024-02-12T17:04:52.365434+0000","last_peered":"2024-02-12T17:04:52.365434+0000","last_clean":"2024-02-12T17:04:52.365434+0000","last_became_active":"2024-02-12T17:04:39.884769+0000","last_became_peered":"2024-02-12T17:04:39.884769+0000","last_unstale":"2024-02-12T17:04:52.365434+0000","last_undegraded":"2024-02-12T17:04:52.365434+0000","last_fullsized":"2024-02-12T17:04:52.365434+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":66,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346972+0000","last_change":"2024-02-12T17:03:59.035349+0000","last_active":"2024-02-12T17:04:52.346972+0000","last_peered":"2024-02-12T17:04:52.346972+0000","last_clean":"2024-02-12T17:04:52.346972+0000","last_became_active":"2024-02-12T17:03:50.639321+0000","last_became_peered":"2024-02-12T17:03:50.639321+0000","last_unstale":"2024-02-12T17:04:52.346972+0000","last_undegraded":"2024-02-12T17:04:52.346972+0000","last_fullsized":"2024-02-12T17:04:52.346972+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":359,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.355742+0000","last_change":"2024-02-12T17:04:39.887129+0000","last_active":"2024-02-12T17:04:52.355742+0000","last_peered":"2024-02-12T17:04:52.355742+0000","last_clean":"2024-02-12T17:04:52.355742+0000","last_became_active":"2024-02-12T17:04:39.886041+0000","last_became_peered":"2024-02-12T17:04:39.886041+0000","last_unstale":"2024-02-12T17:04:52.355742+0000","last_undegraded":"2024-02-12T17:04:52.355742+0000","last_fullsized":"2024-02-12T17:04:52.355742+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":147,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.355166+0000","last_change":"2024-02-12T17:04:39.888701+0000","last_active":"2024-02-12T17:04:52.355166+0000","last_peered":"2024-02-12T17:04:52.355166+0000","last_clean":"2024-02-12T17:04:52.355166+0000","last_became_active":"2024-02-12T17:04:39.888396+0000","last_became_peered":"2024-02-12T17:04:39.888396+0000","last_unstale":"2024-02-12T17:04:52.355166+0000","last_undegraded":"2024-02-12T17:04:52.355166+0000","last_fullsized":"2024-02-12T17:04:52.355166+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.020845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.020845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":385,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359949+0000","last_change":"2024-02-12T17:03:50.551444+0000","last_active":"2024-02-12T17:04:52.359949+0000","last_peered":"2024-02-12T17:04:52.359949+0000","last_clean":"2024-02-12T17:04:52.359949+0000","last_became_active":"2024-02-12T17:03:50.550931+0000","last_became_peered":"2024-02-12T17:03:50.550931+0000","last_unstale":"2024-02-12T17:04:52.359949+0000","last_undegraded":"2024-02-12T17:04:52.359949+0000","last_fullsized":"2024-02-12T17:04:52.359949+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":75,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.343607+0000","last_change":"2024-02-12T17:04:40.315768+0000","last_active":"2024-02-12T17:04:52.343607+0000","last_peered":"2024-02-12T17:04:52.343607+0000","last_clean":"2024-02-12T17:04:52.343607+0000","last_became_active":"2024-02-12T17:04:39.865088+0000","last_became_peered":"2024-02-12T17:04:39.865088+0000","last_unstale":"2024-02-12T17:04:52.343607+0000","last_undegraded":"2024-02-12T17:04:52.343607+0000","last_fullsized":"2024-02-12T17:04:52.343607+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.938805+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":54,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346862+0000","last_change":"2024-02-12T17:03:50.545703+0000","last_active":"2024-02-12T17:04:52.346862+0000","last_peered":"2024-02-12T17:04:52.346862+0000","last_clean":"2024-02-12T17:04:52.346862+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:04:52.346862+0000","last_undegraded":"2024-02-12T17:04:52.346862+0000","last_fullsized":"2024-02-12T17:04:52.346862+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.051528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.051528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":314,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.358978+0000","last_change":"2024-02-12T17:03:50.577696+0000","last_active":"2024-02-12T17:04:52.358978+0000","last_peered":"2024-02-12T17:04:52.358978+0000","last_clean":"2024-02-12T17:04:52.358978+0000","last_became_active":"2024-02-12T17:03:50.577250+0000","last_became_peered":"2024-02-12T17:03:50.577250+0000","last_unstale":"2024-02-12T17:04:52.358978+0000","last_undegraded":"2024-02-12T17:04:52.358978+0000","last_fullsized":"2024-02-12T17:04:52.358978+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":219,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357462+0000","last_change":"2024-02-12T17:03:20.768326+0000","last_active":"2024-02-12T17:04:52.357462+0000","last_peered":"2024-02-12T17:04:52.357462+0000","last_clean":"2024-02-12T17:04:52.357462+0000","last_became_active":"2024-02-12T17:03:20.767943+0000","last_became_peered":"2024-02-12T17:03:20.767943+0000","last_unstale":"2024-02-12T17:04:52.357462+0000","last_undegraded":"2024-02-12T17:04:52.357462+0000","last_fullsized":"2024-02-12T17:04:52.357462+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.858411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.858411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":359,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.361375+0000","last_change":"2024-02-12T17:04:40.575962+0000","last_active":"2024-02-12T17:04:52.361375+0000","last_peered":"2024-02-12T17:04:52.361375+0000","last_clean":"2024-02-12T17:04:52.361375+0000","last_became_active":"2024-02-12T17:04:39.883548+0000","last_became_peered":"2024-02-12T17:04:39.883548+0000","last_unstale":"2024-02-12T17:04:52.361375+0000","last_undegraded":"2024-02-12T17:04:52.361375+0000","last_fullsized":"2024-02-12T17:04:52.361375+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:16.064966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.064966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":301,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357745+0000","last_change":"2024-02-12T17:04:39.877393+0000","last_active":"2024-02-12T17:04:52.357745+0000","last_peered":"2024-02-12T17:04:52.357745+0000","last_clean":"2024-02-12T17:04:52.357745+0000","last_became_active":"2024-02-12T17:04:39.876910+0000","last_became_peered":"2024-02-12T17:04:39.876910+0000","last_unstale":"2024-02-12T17:04:52.357745+0000","last_undegraded":"2024-02-12T17:04:52.357745+0000","last_fullsized":"2024-02-12T17:04:52.357745+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.019813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.019813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":63,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344230+0000","last_change":"2024-02-12T17:04:39.890984+0000","last_active":"2024-02-12T17:04:52.344230+0000","last_peered":"2024-02-12T17:04:52.344230+0000","last_clean":"2024-02-12T17:04:52.344230+0000","last_became_active":"2024-02-12T17:04:39.888436+0000","last_became_peered":"2024-02-12T17:04:39.888436+0000","last_unstale":"2024-02-12T17:04:52.344230+0000","last_undegraded":"2024-02-12T17:04:52.344230+0000","last_fullsized":"2024-02-12T17:04:52.344230+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:33.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:01:33.149132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":54,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345852+0000","last_change":"2024-02-12T17:03:50.559509+0000","last_active":"2024-02-12T17:04:52.345852+0000","last_peered":"2024-02-12T17:04:52.345852+0000","last_clean":"2024-02-12T17:04:52.345852+0000","last_became_active":"2024-02-12T17:03:50.558598+0000","last_became_peered":"2024-02-12T17:03:50.558598+0000","last_unstale":"2024-02-12T17:04:52.345852+0000","last_undegraded":"2024-02-12T17:04:52.345852+0000","last_fullsized":"2024-02-12T17:04:52.345852+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":168,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.358278+0000","last_change":"2024-02-12T17:03:22.527733+0000","last_active":"2024-02-12T17:04:52.358278+0000","last_peered":"2024-02-12T17:04:52.358278+0000","last_clean":"2024-02-12T17:04:52.358278+0000","last_became_active":"2024-02-12T17:03:20.774315+0000","last_became_peered":"2024-02-12T17:03:20.774315+0000","last_unstale":"2024-02-12T17:04:52.358278+0000","last_undegraded":"2024-02-12T17:04:52.358278+0000","last_fullsized":"2024-02-12T17:04:52.358278+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:00:53.792980+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.792980+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"348'11","reported_seq":72,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.434016+0000","last_change":"2024-02-12T17:03:54.814978+0000","last_active":"2024-02-12T17:04:52.434016+0000","last_peered":"2024-02-12T17:04:52.434016+0000","last_clean":"2024-02-12T17:04:52.434016+0000","last_became_active":"2024-02-12T17:03:50.558517+0000","last_became_peered":"2024-02-12T17:03:50.558517+0000","last_unstale":"2024-02-12T17:04:52.434016+0000","last_undegraded":"2024-02-12T17:04:52.434016+0000","last_fullsized":"2024-02-12T17:04:52.434016+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-12T17:02:17.689218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.689218+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":142,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344088+0000","last_change":"2024-02-12T17:04:39.890091+0000","last_active":"2024-02-12T17:04:52.344088+0000","last_peered":"2024-02-12T17:04:52.344088+0000","last_clean":"2024-02-12T17:04:52.344088+0000","last_became_active":"2024-02-12T17:04:39.886343+0000","last_became_peered":"2024-02-12T17:04:39.886343+0000","last_unstale":"2024-02-12T17:04:52.344088+0000","last_undegraded":"2024-02-12T17:04:52.344088+0000","last_fullsized":"2024-02-12T17:04:52.344088+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.953891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.953891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":457,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357381+0000","last_change":"2024-02-12T17:03:50.690222+0000","last_active":"2024-02-12T17:04:52.357381+0000","last_peered":"2024-02-12T17:04:52.357381+0000","last_clean":"2024-02-12T17:04:52.357381+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:04:52.357381+0000","last_undegraded":"2024-02-12T17:04:52.357381+0000","last_fullsized":"2024-02-12T17:04:52.357381+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:31.827632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:31.827632+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"380'12","reported_seq":600,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.435934+0000","last_change":"2024-02-12T17:04:40.578496+0000","last_active":"2024-02-12T17:04:52.435934+0000","last_peered":"2024-02-12T17:04:52.435934+0000","last_clean":"2024-02-12T17:04:52.435934+0000","last_became_active":"2024-02-12T17:04:39.887044+0000","last_became_peered":"2024-02-12T17:04:39.887044+0000","last_unstale":"2024-02-12T17:04:52.435934+0000","last_undegraded":"2024-02-12T17:04:52.435934+0000","last_fullsized":"2024-02-12T17:04:52.435934+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'9","last_scrub_stamp":"2024-02-12T17:00:49.846432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.846432+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":376,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.358680+0000","last_change":"2024-02-12T17:03:50.577996+0000","last_active":"2024-02-12T17:04:52.358680+0000","last_peered":"2024-02-12T17:04:52.358680+0000","last_clean":"2024-02-12T17:04:52.358680+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:04:52.358680+0000","last_undegraded":"2024-02-12T17:04:52.358680+0000","last_fullsized":"2024-02-12T17:04:52.358680+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:34.874363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:34.874363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":66,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344482+0000","last_change":"2024-02-12T17:03:55.244602+0000","last_active":"2024-02-12T17:04:52.344482+0000","last_peered":"2024-02-12T17:04:52.344482+0000","last_clean":"2024-02-12T17:04:52.344482+0000","last_became_active":"2024-02-12T17:03:50.556444+0000","last_became_peered":"2024-02-12T17:03:50.556444+0000","last_unstale":"2024-02-12T17:04:52.344482+0000","last_undegraded":"2024-02-12T17:04:52.344482+0000","last_fullsized":"2024-02-12T17:04:52.344482+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":372,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359435+0000","last_change":"2024-02-12T17:04:39.889382+0000","last_active":"2024-02-12T17:04:52.359435+0000","last_peered":"2024-02-12T17:04:52.359435+0000","last_clean":"2024-02-12T17:04:52.359435+0000","last_became_active":"2024-02-12T17:04:39.889055+0000","last_became_peered":"2024-02-12T17:04:39.889055+0000","last_unstale":"2024-02-12T17:04:52.359435+0000","last_undegraded":"2024-02-12T17:04:52.359435+0000","last_fullsized":"2024-02-12T17:04:52.359435+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.187738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":315,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357969+0000","last_change":"2024-02-12T17:04:39.884622+0000","last_active":"2024-02-12T17:04:52.357969+0000","last_peered":"2024-02-12T17:04:52.357969+0000","last_clean":"2024-02-12T17:04:52.357969+0000","last_became_active":"2024-02-12T17:04:39.884162+0000","last_became_peered":"2024-02-12T17:04:39.884162+0000","last_unstale":"2024-02-12T17:04:52.357969+0000","last_undegraded":"2024-02-12T17:04:52.357969+0000","last_fullsized":"2024-02-12T17:04:52.357969+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:32.127593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:32.127593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":62,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346856+0000","last_change":"2024-02-12T17:03:53.944994+0000","last_active":"2024-02-12T17:04:52.346856+0000","last_peered":"2024-02-12T17:04:52.346856+0000","last_clean":"2024-02-12T17:04:52.346856+0000","last_became_active":"2024-02-12T17:03:50.555238+0000","last_became_peered":"2024-02-12T17:03:50.555238+0000","last_unstale":"2024-02-12T17:04:52.346856+0000","last_undegraded":"2024-02-12T17:04:52.346856+0000","last_fullsized":"2024-02-12T17:04:52.346856+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-02-12T17:00:53.269107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.269107+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":29,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356072+0000","last_change":"2024-02-12T17:04:39.936874+0000","last_active":"2024-02-12T17:04:52.356072+0000","last_peered":"2024-02-12T17:04:52.356072+0000","last_clean":"2024-02-12T17:04:52.356072+0000","last_became_active":"2024-02-12T17:04:39.936538+0000","last_became_peered":"2024-02-12T17:04:39.936538+0000","last_unstale":"2024-02-12T17:04:52.356072+0000","last_undegraded":"2024-02-12T17:04:52.356072+0000","last_fullsized":"2024-02-12T17:04:52.356072+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:43.982623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.982623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":279,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345570+0000","last_change":"2024-02-12T17:04:39.867059+0000","last_active":"2024-02-12T17:04:52.345570+0000","last_peered":"2024-02-12T17:04:52.345570+0000","last_clean":"2024-02-12T17:04:52.345570+0000","last_became_active":"2024-02-12T17:04:39.866732+0000","last_became_peered":"2024-02-12T17:04:39.866732+0000","last_unstale":"2024-02-12T17:04:52.345570+0000","last_undegraded":"2024-02-12T17:04:52.345570+0000","last_fullsized":"2024-02-12T17:04:52.345570+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.824194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.824194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":64,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345596+0000","last_change":"2024-02-12T17:03:55.514799+0000","last_active":"2024-02-12T17:04:52.345596+0000","last_peered":"2024-02-12T17:04:52.345596+0000","last_clean":"2024-02-12T17:04:52.345596+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:04:52.345596+0000","last_undegraded":"2024-02-12T17:04:52.345596+0000","last_fullsized":"2024-02-12T17:04:52.345596+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:00:53.936664+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.936664+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"392'14260","reported_seq":14377,"reported_epoch":392,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:53.650321+0000","last_change":"2024-02-12T17:04:40.958720+0000","last_active":"2024-02-12T17:04:53.650321+0000","last_peered":"2024-02-12T17:04:53.650321+0000","last_clean":"2024-02-12T17:03:49.505679+0000","last_became_active":"2024-02-12T17:04:40.888742+0000","last_became_peered":"2024-02-12T17:04:40.888742+0000","last_unstale":"2024-02-12T17:04:53.650321+0000","last_undegraded":"2024-02-12T17:04:53.650321+0000","last_fullsized":"2024-02-12T17:04:53.650321+0000","mapping_epoch":378,"log_start":"331'6700","ondisk_log_start":"331'6700","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":7560,"ondisk_log_size":7560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":184287232,"num_objects":2812,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14260,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":2812}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":81,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344228+0000","last_change":"2024-02-12T17:04:39.890573+0000","last_active":"2024-02-12T17:04:52.344228+0000","last_peered":"2024-02-12T17:04:52.344228+0000","last_clean":"2024-02-12T17:04:52.344228+0000","last_became_active":"2024-02-12T17:04:39.889283+0000","last_became_peered":"2024-02-12T17:04:39.889283+0000","last_unstale":"2024-02-12T17:04:52.344228+0000","last_undegraded":"2024-02-12T17:04:52.344228+0000","last_fullsized":"2024-02-12T17:04:52.344228+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:28.912872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.912872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":91,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356862+0000","last_change":"2024-02-12T17:04:40.578504+0000","last_active":"2024-02-12T17:04:52.356862+0000","last_peered":"2024-02-12T17:04:52.356862+0000","last_clean":"2024-02-12T17:04:52.356862+0000","last_became_active":"2024-02-12T17:04:39.890440+0000","last_became_peered":"2024-02-12T17:04:39.890440+0000","last_unstale":"2024-02-12T17:04:52.356862+0000","last_undegraded":"2024-02-12T17:04:52.356862+0000","last_fullsized":"2024-02-12T17:04:52.356862+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:02:17.066331+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.066331+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":87,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.347154+0000","last_change":"2024-02-12T17:04:42.701462+0000","last_active":"2024-02-12T17:04:52.347154+0000","last_peered":"2024-02-12T17:04:52.347154+0000","last_clean":"2024-02-12T17:04:52.347154+0000","last_became_active":"2024-02-12T17:04:39.891351+0000","last_became_peered":"2024-02-12T17:04:39.891351+0000","last_unstale":"2024-02-12T17:04:52.347154+0000","last_undegraded":"2024-02-12T17:04:52.347154+0000","last_fullsized":"2024-02-12T17:04:52.347154+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":286,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357357+0000","last_change":"2024-02-12T17:04:39.882820+0000","last_active":"2024-02-12T17:04:52.357357+0000","last_peered":"2024-02-12T17:04:52.357357+0000","last_clean":"2024-02-12T17:04:52.357357+0000","last_became_active":"2024-02-12T17:04:39.880493+0000","last_became_peered":"2024-02-12T17:04:39.880493+0000","last_unstale":"2024-02-12T17:04:52.357357+0000","last_undegraded":"2024-02-12T17:04:52.357357+0000","last_fullsized":"2024-02-12T17:04:52.357357+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:25.815816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.815816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"392'14675","reported_seq":24362,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-02-12T17:04:53.755100+0000","last_change":"2024-02-12T17:04:30.711484+0000","last_active":"2024-02-12T17:04:53.755100+0000","last_peered":"2024-02-12T17:04:53.755100+0000","last_clean":"2024-02-12T17:04:53.755100+0000","last_became_active":"2024-02-12T17:04:30.678545+0000","last_became_peered":"2024-02-12T17:04:30.678545+0000","last_unstale":"2024-02-12T17:04:53.755100+0000","last_undegraded":"2024-02-12T17:04:53.755100+0000","last_fullsized":"2024-02-12T17:04:53.755100+0000","mapping_epoch":368,"log_start":"346'11652","ondisk_log_start":"346'11652","created":304,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3023,"ondisk_log_size":3023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14675,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4785,"num_bytes_recovered":313589760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":368,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345200+0000","last_change":"2024-02-12T17:03:50.533429+0000","last_active":"2024-02-12T17:04:52.345200+0000","last_peered":"2024-02-12T17:04:52.345200+0000","last_clean":"2024-02-12T17:04:52.345200+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:04:52.345200+0000","last_undegraded":"2024-02-12T17:04:52.345200+0000","last_fullsized":"2024-02-12T17:04:52.345200+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346131+0000","last_change":"2024-02-12T17:02:30.758581+0000","last_active":"2024-02-12T17:04:52.346131+0000","last_peered":"2024-02-12T17:04:52.346131+0000","last_clean":"2024-02-12T17:04:52.346131+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:04:52.346131+0000","last_undegraded":"2024-02-12T17:04:52.346131+0000","last_fullsized":"2024-02-12T17:04:52.346131+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.758520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.758520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":292,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359065+0000","last_change":"2024-02-12T17:04:41.560789+0000","last_active":"2024-02-12T17:04:52.359065+0000","last_peered":"2024-02-12T17:04:52.359065+0000","last_clean":"2024-02-12T17:04:52.359065+0000","last_became_active":"2024-02-12T17:04:39.883521+0000","last_became_peered":"2024-02-12T17:04:39.883521+0000","last_unstale":"2024-02-12T17:04:52.359065+0000","last_undegraded":"2024-02-12T17:04:52.359065+0000","last_fullsized":"2024-02-12T17:04:52.359065+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"348'12","reported_seq":510,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.435807+0000","last_change":"2024-02-12T17:03:51.176776+0000","last_active":"2024-02-12T17:04:52.435807+0000","last_peered":"2024-02-12T17:04:52.435807+0000","last_clean":"2024-02-12T17:04:52.435807+0000","last_became_active":"2024-02-12T17:03:50.577424+0000","last_became_peered":"2024-02-12T17:03:50.577424+0000","last_unstale":"2024-02-12T17:04:52.435807+0000","last_undegraded":"2024-02-12T17:04:52.435807+0000","last_fullsized":"2024-02-12T17:04:52.435807+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"392'14441","reported_seq":15232,"reported_epoch":392,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:04:53.647547+0000","last_change":"2024-02-12T17:04:50.118563+0000","last_active":"2024-02-12T17:04:53.647547+0000","last_peered":"2024-02-12T17:04:53.647547+0000","last_clean":"2024-02-12T17:03:19.725881+0000","last_became_active":"2024-02-12T17:03:50.651275+0000","last_became_peered":"2024-02-12T17:03:50.651275+0000","last_unstale":"2024-02-12T17:04:53.647547+0000","last_undegraded":"2024-02-12T17:04:53.647547+0000","last_fullsized":"2024-02-12T17:04:53.647547+0000","mapping_epoch":347,"log_start":"346'11390","ondisk_log_start":"346'11390","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3051,"ondisk_log_size":3051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":176750592,"num_objects":2697,"num_object_clones":0,"num_object_copies":8091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2659,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14441,"num_write_kb":548416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":21954560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,0],"avail_no_missing":["7","0","4","5"],"object_location_counts":[{"shards":"0,5,7","objects":2697}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":63,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345246+0000","last_change":"2024-02-12T17:04:39.888977+0000","last_active":"2024-02-12T17:04:52.345246+0000","last_peered":"2024-02-12T17:04:52.345246+0000","last_clean":"2024-02-12T17:04:52.345246+0000","last_became_active":"2024-02-12T17:04:39.888051+0000","last_became_peered":"2024-02-12T17:04:39.888051+0000","last_unstale":"2024-02-12T17:04:52.345246+0000","last_undegraded":"2024-02-12T17:04:52.345246+0000","last_fullsized":"2024-02-12T17:04:52.345246+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":390,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.358754+0000","last_change":"2024-02-12T17:03:50.549240+0000","last_active":"2024-02-12T17:04:52.358754+0000","last_peered":"2024-02-12T17:04:52.358754+0000","last_clean":"2024-02-12T17:04:52.358754+0000","last_became_active":"2024-02-12T17:03:50.548926+0000","last_became_peered":"2024-02-12T17:03:50.548926+0000","last_unstale":"2024-02-12T17:04:52.358754+0000","last_undegraded":"2024-02-12T17:04:52.358754+0000","last_fullsized":"2024-02-12T17:04:52.358754+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:42.892563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.892563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"348'55","reported_seq":90,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.434205+0000","last_change":"2024-02-12T17:03:58.596193+0000","last_active":"2024-02-12T17:04:52.434205+0000","last_peered":"2024-02-12T17:04:52.434205+0000","last_clean":"2024-02-12T17:04:52.434205+0000","last_became_active":"2024-02-12T17:03:50.557154+0000","last_became_peered":"2024-02-12T17:03:50.557154+0000","last_unstale":"2024-02-12T17:04:52.434205+0000","last_undegraded":"2024-02-12T17:04:52.434205+0000","last_fullsized":"2024-02-12T17:04:52.434205+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":308,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359578+0000","last_change":"2024-02-12T17:04:39.887098+0000","last_active":"2024-02-12T17:04:52.359578+0000","last_peered":"2024-02-12T17:04:52.359578+0000","last_clean":"2024-02-12T17:04:52.359578+0000","last_became_active":"2024-02-12T17:04:39.886564+0000","last_became_peered":"2024-02-12T17:04:39.886564+0000","last_unstale":"2024-02-12T17:04:52.359578+0000","last_undegraded":"2024-02-12T17:04:52.359578+0000","last_fullsized":"2024-02-12T17:04:52.359578+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"392'14578","reported_seq":15790,"reported_epoch":392,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:53.743289+0000","last_change":"2024-02-12T17:04:39.884779+0000","last_active":"2024-02-12T17:04:53.743289+0000","last_peered":"2024-02-12T17:04:53.743289+0000","last_clean":"2024-02-12T17:03:49.453764+0000","last_became_active":"2024-02-12T17:03:51.655919+0000","last_became_peered":"2024-02-12T17:03:51.655919+0000","last_unstale":"2024-02-12T17:04:53.743289+0000","last_undegraded":"2024-02-12T17:04:53.743289+0000","last_fullsized":"2024-02-12T17:04:53.743289+0000","mapping_epoch":348,"log_start":"346'11497","ondisk_log_start":"346'11497","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3081,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":183762944,"num_objects":2804,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2465,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14578,"num_write_kb":556224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":36175872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":2804}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":379,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359694+0000","last_change":"2024-02-12T17:03:50.566072+0000","last_active":"2024-02-12T17:04:52.359694+0000","last_peered":"2024-02-12T17:04:52.359694+0000","last_clean":"2024-02-12T17:04:52.359694+0000","last_became_active":"2024-02-12T17:03:50.564700+0000","last_became_peered":"2024-02-12T17:03:50.564700+0000","last_unstale":"2024-02-12T17:04:52.359694+0000","last_undegraded":"2024-02-12T17:04:52.359694+0000","last_fullsized":"2024-02-12T17:04:52.359694+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:04.085827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:04.085827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":338,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.766838+0000","last_change":"2024-02-12T17:01:16.913133+0000","last_active":"2024-02-12T17:04:52.766838+0000","last_peered":"2024-02-12T17:04:52.766838+0000","last_clean":"2024-02-12T17:04:52.766838+0000","last_became_active":"2024-02-12T17:01:16.912041+0000","last_became_peered":"2024-02-12T17:01:16.912041+0000","last_unstale":"2024-02-12T17:04:52.766838+0000","last_undegraded":"2024-02-12T17:04:52.766838+0000","last_fullsized":"2024-02-12T17:04:52.766838+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:00:52.838852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:52.838852+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":222,"num_read_kb":217,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.347049+0000","last_change":"2024-02-12T17:03:50.533363+0000","last_active":"2024-02-12T17:04:52.347049+0000","last_peered":"2024-02-12T17:04:52.347049+0000","last_clean":"2024-02-12T17:04:52.347049+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:04:52.347049+0000","last_undegraded":"2024-02-12T17:04:52.347049+0000","last_fullsized":"2024-02-12T17:04:52.347049+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:38.142602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:38.142602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":324,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.365510+0000","last_change":"2024-02-12T17:03:51.175255+0000","last_active":"2024-02-12T17:04:52.365510+0000","last_peered":"2024-02-12T17:04:52.365510+0000","last_clean":"2024-02-12T17:04:52.365510+0000","last_became_active":"2024-02-12T17:03:50.573814+0000","last_became_peered":"2024-02-12T17:03:50.573814+0000","last_unstale":"2024-02-12T17:04:52.365510+0000","last_undegraded":"2024-02-12T17:04:52.365510+0000","last_fullsized":"2024-02-12T17:04:52.365510+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":306,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.360022+0000","last_change":"2024-02-12T17:01:16.909653+0000","last_active":"2024-02-12T17:04:52.360022+0000","last_peered":"2024-02-12T17:04:52.360022+0000","last_clean":"2024-02-12T17:04:52.360022+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:04:52.360022+0000","last_undegraded":"2024-02-12T17:04:52.360022+0000","last_fullsized":"2024-02-12T17:04:52.360022+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"392'14352","reported_seq":14784,"reported_epoch":392,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:04:53.644897+0000","last_change":"2024-02-12T17:04:40.746118+0000","last_active":"2024-02-12T17:04:53.644897+0000","last_peered":"2024-02-12T17:04:53.644897+0000","last_clean":"2024-02-12T17:03:49.506422+0000","last_became_active":"2024-02-12T17:03:51.676105+0000","last_became_peered":"2024-02-12T17:03:51.676105+0000","last_unstale":"2024-02-12T17:04:53.644897+0000","last_undegraded":"2024-02-12T17:04:53.644897+0000","last_fullsized":"2024-02-12T17:04:53.644897+0000","mapping_epoch":348,"log_start":"331'6800","ondisk_log_start":"331'6800","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":7552,"ondisk_log_size":7552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":185335808,"num_objects":2828,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2682,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14352,"num_write_kb":549760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":10616832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":2828}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":398,"reported_epoch":392,"state":"peering","last_fresh":"2024-02-12T17:04:53.376923+0000","last_change":"2024-02-12T17:04:53.377150+0000","last_active":"2024-02-12T17:04:51.326573+0000","last_peered":"2024-02-12T17:04:52.395311+0000","last_clean":"2024-02-12T17:04:52.395311+0000","last_became_active":"2024-02-12T17:04:39.883522+0000","last_became_peered":"2024-02-12T17:04:52.385957+0000","last_unstale":"2024-02-12T17:04:53.376923+0000","last_undegraded":"2024-02-12T17:04:53.376923+0000","last_fullsized":"2024-02-12T17:04:53.376923+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:57.950284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:00:57.950284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345284+0000","last_change":"2024-02-12T17:04:39.885990+0000","last_active":"2024-02-12T17:04:52.345284+0000","last_peered":"2024-02-12T17:04:52.345284+0000","last_clean":"2024-02-12T17:04:52.345284+0000","last_became_active":"2024-02-12T17:04:39.885173+0000","last_became_peered":"2024-02-12T17:04:39.885173+0000","last_unstale":"2024-02-12T17:04:52.345284+0000","last_undegraded":"2024-02-12T17:04:52.345284+0000","last_fullsized":"2024-02-12T17:04:52.345284+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":358,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.361145+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:04:52.361145+0000","last_peered":"2024-02-12T17:04:52.361145+0000","last_clean":"2024-02-12T17:04:52.361145+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:04:52.361145+0000","last_undegraded":"2024-02-12T17:04:52.361145+0000","last_fullsized":"2024-02-12T17:04:52.361145+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":391,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.354201+0000","last_change":"2024-02-12T17:04:39.915666+0000","last_active":"2024-02-12T17:04:52.354201+0000","last_peered":"2024-02-12T17:04:52.354201+0000","last_clean":"2024-02-12T17:04:52.354201+0000","last_became_active":"2024-02-12T17:04:39.915392+0000","last_became_peered":"2024-02-12T17:04:39.915392+0000","last_unstale":"2024-02-12T17:04:52.354201+0000","last_undegraded":"2024-02-12T17:04:52.354201+0000","last_fullsized":"2024-02-12T17:04:52.354201+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:25.870916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:01:25.870916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":140,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359839+0000","last_change":"2024-02-12T17:04:40.745958+0000","last_active":"2024-02-12T17:04:52.359839+0000","last_peered":"2024-02-12T17:04:52.359839+0000","last_clean":"2024-02-12T17:04:52.359839+0000","last_became_active":"2024-02-12T17:04:39.889039+0000","last_became_peered":"2024-02-12T17:04:39.889039+0000","last_unstale":"2024-02-12T17:04:52.359839+0000","last_undegraded":"2024-02-12T17:04:52.359839+0000","last_fullsized":"2024-02-12T17:04:52.359839+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'26","last_scrub_stamp":"2024-02-12T17:01:43.163724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:43.163724+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":357,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356111+0000","last_change":"2024-02-12T17:03:15.796227+0000","last_active":"2024-02-12T17:04:52.356111+0000","last_peered":"2024-02-12T17:04:52.356111+0000","last_clean":"2024-02-12T17:04:52.356111+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:04:52.356111+0000","last_undegraded":"2024-02-12T17:04:52.356111+0000","last_fullsized":"2024-02-12T17:04:52.356111+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"392'14483","reported_seq":15208,"reported_epoch":392,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:53.791648+0000","last_change":"2024-02-12T17:04:40.576566+0000","last_active":"2024-02-12T17:04:53.791648+0000","last_peered":"2024-02-12T17:04:53.791648+0000","last_clean":"2024-02-12T17:03:19.701993+0000","last_became_active":"2024-02-12T17:03:50.639685+0000","last_became_peered":"2024-02-12T17:03:50.639685+0000","last_unstale":"2024-02-12T17:04:53.791648+0000","last_undegraded":"2024-02-12T17:04:53.791648+0000","last_fullsized":"2024-02-12T17:04:53.791648+0000","mapping_epoch":347,"log_start":"346'11449","ondisk_log_start":"346'11449","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3034,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":178585600,"num_objects":2725,"num_object_clones":0,"num_object_copies":8175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2725,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14483,"num_write_kb":550656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":19791872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":2725}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":316,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.360639+0000","last_change":"2024-02-12T17:04:39.871302+0000","last_active":"2024-02-12T17:04:52.360639+0000","last_peered":"2024-02-12T17:04:52.360639+0000","last_clean":"2024-02-12T17:04:52.360639+0000","last_became_active":"2024-02-12T17:04:39.870979+0000","last_became_peered":"2024-02-12T17:04:39.870979+0000","last_unstale":"2024-02-12T17:04:52.360639+0000","last_undegraded":"2024-02-12T17:04:52.360639+0000","last_fullsized":"2024-02-12T17:04:52.360639+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346615+0000","last_change":"2024-02-12T17:04:39.892450+0000","last_active":"2024-02-12T17:04:52.346615+0000","last_peered":"2024-02-12T17:04:52.346615+0000","last_clean":"2024-02-12T17:04:52.346615+0000","last_became_active":"2024-02-12T17:04:39.892122+0000","last_became_peered":"2024-02-12T17:04:39.892122+0000","last_unstale":"2024-02-12T17:04:52.346615+0000","last_undegraded":"2024-02-12T17:04:52.346615+0000","last_fullsized":"2024-02-12T17:04:52.346615+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.941776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":388,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357163+0000","last_change":"2024-02-12T17:03:50.776630+0000","last_active":"2024-02-12T17:04:52.357163+0000","last_peered":"2024-02-12T17:04:52.357163+0000","last_clean":"2024-02-12T17:04:52.357163+0000","last_became_active":"2024-02-12T17:03:50.564881+0000","last_became_peered":"2024-02-12T17:03:50.564881+0000","last_unstale":"2024-02-12T17:04:52.357163+0000","last_undegraded":"2024-02-12T17:04:52.357163+0000","last_fullsized":"2024-02-12T17:04:52.357163+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":374,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356143+0000","last_change":"2024-02-12T17:03:24.771579+0000","last_active":"2024-02-12T17:04:52.356143+0000","last_peered":"2024-02-12T17:04:52.356143+0000","last_clean":"2024-02-12T17:04:52.356143+0000","last_became_active":"2024-02-12T17:03:20.780926+0000","last_became_peered":"2024-02-12T17:03:20.780926+0000","last_unstale":"2024-02-12T17:04:52.356143+0000","last_undegraded":"2024-02-12T17:04:52.356143+0000","last_fullsized":"2024-02-12T17:04:52.356143+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":390,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.372879+0000","last_change":"2024-02-12T17:04:39.885796+0000","last_active":"2024-02-12T17:04:52.372879+0000","last_peered":"2024-02-12T17:04:52.372879+0000","last_clean":"2024-02-12T17:04:52.372879+0000","last_became_active":"2024-02-12T17:04:39.885162+0000","last_became_peered":"2024-02-12T17:04:39.885162+0000","last_unstale":"2024-02-12T17:04:52.372879+0000","last_undegraded":"2024-02-12T17:04:52.372879+0000","last_fullsized":"2024-02-12T17:04:52.372879+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"392'14598","reported_seq":16407,"reported_epoch":392,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:53.645933+0000","last_change":"2024-02-12T17:04:40.407124+0000","last_active":"2024-02-12T17:04:53.645933+0000","last_peered":"2024-02-12T17:04:53.645933+0000","last_clean":"2024-02-12T17:03:19.726721+0000","last_became_active":"2024-02-12T17:04:39.940046+0000","last_became_peered":"2024-02-12T17:04:39.940046+0000","last_unstale":"2024-02-12T17:04:53.645933+0000","last_undegraded":"2024-02-12T17:04:53.645933+0000","last_fullsized":"2024-02-12T17:04:53.645933+0000","mapping_epoch":377,"log_start":"346'11484","ondisk_log_start":"346'11484","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3114,"ondisk_log_size":3114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":189136896,"num_objects":2886,"num_object_clones":0,"num_object_copies":8658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5291,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14598,"num_write_kb":559488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":55312384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,4,0],"avail_no_missing":["7","0","4"],"object_location_counts":[{"shards":"0,4,7","objects":2886}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":118,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.354681+0000","last_change":"2024-02-12T17:03:20.779625+0000","last_active":"2024-02-12T17:04:52.354681+0000","last_peered":"2024-02-12T17:04:52.354681+0000","last_clean":"2024-02-12T17:04:52.354681+0000","last_became_active":"2024-02-12T17:03:20.779126+0000","last_became_peered":"2024-02-12T17:03:20.779126+0000","last_unstale":"2024-02-12T17:04:52.354681+0000","last_undegraded":"2024-02-12T17:04:52.354681+0000","last_fullsized":"2024-02-12T17:04:52.354681+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:48.617913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:02:48.617913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":379,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359764+0000","last_change":"2024-02-12T17:04:39.886695+0000","last_active":"2024-02-12T17:04:52.359764+0000","last_peered":"2024-02-12T17:04:52.359764+0000","last_clean":"2024-02-12T17:04:52.359764+0000","last_became_active":"2024-02-12T17:04:39.886102+0000","last_became_peered":"2024-02-12T17:04:39.886102+0000","last_unstale":"2024-02-12T17:04:52.359764+0000","last_undegraded":"2024-02-12T17:04:52.359764+0000","last_fullsized":"2024-02-12T17:04:52.359764+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.212949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.212949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":370,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.361237+0000","last_change":"2024-02-12T17:03:21.658125+0000","last_active":"2024-02-12T17:04:52.361237+0000","last_peered":"2024-02-12T17:04:52.361237+0000","last_clean":"2024-02-12T17:04:52.361237+0000","last_became_active":"2024-02-12T17:03:20.769223+0000","last_became_peered":"2024-02-12T17:03:20.769223+0000","last_unstale":"2024-02-12T17:04:52.361237+0000","last_undegraded":"2024-02-12T17:04:52.361237+0000","last_fullsized":"2024-02-12T17:04:52.361237+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:02:44.688402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:44.688402+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":151,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345398+0000","last_change":"2024-02-12T17:03:15.810501+0000","last_active":"2024-02-12T17:04:52.345398+0000","last_peered":"2024-02-12T17:04:52.345398+0000","last_clean":"2024-02-12T17:04:52.345398+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:04:52.345398+0000","last_undegraded":"2024-02-12T17:04:52.345398+0000","last_fullsized":"2024-02-12T17:04:52.345398+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"392'14089","reported_seq":14481,"reported_epoch":392,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:04:53.640785+0000","last_change":"2024-02-12T17:04:41.384911+0000","last_active":"2024-02-12T17:04:53.640785+0000","last_peered":"2024-02-12T17:04:53.640785+0000","last_clean":"2024-02-12T17:04:38.799786+0000","last_became_active":"2024-02-12T17:04:40.925272+0000","last_became_peered":"2024-02-12T17:04:40.925272+0000","last_unstale":"2024-02-12T17:04:53.640785+0000","last_undegraded":"2024-02-12T17:04:53.640785+0000","last_fullsized":"2024-02-12T17:04:53.640785+0000","mapping_epoch":378,"log_start":"345'10945","ondisk_log_start":"345'10945","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3144,"ondisk_log_size":3144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":177471512,"num_objects":2709,"num_object_clones":0,"num_object_copies":8127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2583,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14089,"num_write_kb":537473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":9306112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,4],"avail_no_missing":["7","4","5"],"object_location_counts":[{"shards":"4,5,7","objects":2709}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":63,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345369+0000","last_change":"2024-02-12T17:04:39.886704+0000","last_active":"2024-02-12T17:04:52.345369+0000","last_peered":"2024-02-12T17:04:52.345369+0000","last_clean":"2024-02-12T17:04:52.345369+0000","last_became_active":"2024-02-12T17:04:39.885677+0000","last_became_peered":"2024-02-12T17:04:39.885677+0000","last_unstale":"2024-02-12T17:04:52.345369+0000","last_undegraded":"2024-02-12T17:04:52.345369+0000","last_fullsized":"2024-02-12T17:04:52.345369+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":150,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345107+0000","last_change":"2024-02-12T17:04:39.864989+0000","last_active":"2024-02-12T17:04:52.345107+0000","last_peered":"2024-02-12T17:04:52.345107+0000","last_clean":"2024-02-12T17:04:52.345107+0000","last_became_active":"2024-02-12T17:04:39.864631+0000","last_became_peered":"2024-02-12T17:04:39.864631+0000","last_unstale":"2024-02-12T17:04:52.345107+0000","last_undegraded":"2024-02-12T17:04:52.345107+0000","last_fullsized":"2024-02-12T17:04:52.345107+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":72,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.343969+0000","last_change":"2024-02-12T17:03:54.620306+0000","last_active":"2024-02-12T17:04:52.343969+0000","last_peered":"2024-02-12T17:04:52.343969+0000","last_clean":"2024-02-12T17:04:52.343969+0000","last_became_active":"2024-02-12T17:03:50.556763+0000","last_became_peered":"2024-02-12T17:03:50.556763+0000","last_unstale":"2024-02-12T17:04:52.343969+0000","last_undegraded":"2024-02-12T17:04:52.343969+0000","last_fullsized":"2024-02-12T17:04:52.343969+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":365,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346934+0000","last_change":"2024-02-12T17:04:39.891789+0000","last_active":"2024-02-12T17:04:52.346934+0000","last_peered":"2024-02-12T17:04:52.346934+0000","last_clean":"2024-02-12T17:04:52.346934+0000","last_became_active":"2024-02-12T17:04:39.890884+0000","last_became_peered":"2024-02-12T17:04:39.890884+0000","last_unstale":"2024-02-12T17:04:52.346934+0000","last_undegraded":"2024-02-12T17:04:52.346934+0000","last_fullsized":"2024-02-12T17:04:52.346934+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"392'14608","reported_seq":16716,"reported_epoch":392,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:53.588431+0000","last_change":"2024-02-12T17:04:50.113487+0000","last_active":"2024-02-12T17:04:53.588431+0000","last_peered":"2024-02-12T17:04:53.588431+0000","last_clean":"2024-02-12T17:03:19.745457+0000","last_became_active":"2024-02-12T17:04:39.930523+0000","last_became_peered":"2024-02-12T17:04:39.930523+0000","last_unstale":"2024-02-12T17:04:53.588431+0000","last_undegraded":"2024-02-12T17:04:53.588431+0000","last_fullsized":"2024-02-12T17:04:53.588431+0000","mapping_epoch":377,"log_start":"338'9837","ondisk_log_start":"338'9837","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'5831","last_scrub_stamp":"2024-02-12T17:02:24.719414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.719414+0000","log_size":4771,"ondisk_log_size":4771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":188088320,"num_objects":2870,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5126,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14608,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":979,"num_bytes_recovered":56623104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":2870}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":395,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356735+0000","last_change":"2024-02-12T17:04:40.405970+0000","last_active":"2024-02-12T17:04:52.356735+0000","last_peered":"2024-02-12T17:04:52.356735+0000","last_clean":"2024-02-12T17:04:52.356735+0000","last_became_active":"2024-02-12T17:04:39.889077+0000","last_became_peered":"2024-02-12T17:04:39.889077+0000","last_unstale":"2024-02-12T17:04:52.356735+0000","last_undegraded":"2024-02-12T17:04:52.356735+0000","last_fullsized":"2024-02-12T17:04:52.356735+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:03:17.811656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.811656+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":160,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356939+0000","last_change":"2024-02-12T17:03:50.563493+0000","last_active":"2024-02-12T17:04:52.356939+0000","last_peered":"2024-02-12T17:04:52.356939+0000","last_clean":"2024-02-12T17:04:52.356939+0000","last_became_active":"2024-02-12T17:03:50.562556+0000","last_became_peered":"2024-02-12T17:03:50.562556+0000","last_unstale":"2024-02-12T17:04:52.356939+0000","last_undegraded":"2024-02-12T17:04:52.356939+0000","last_fullsized":"2024-02-12T17:04:52.356939+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.562425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.562425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":445,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.358290+0000","last_change":"2024-02-12T17:04:41.124664+0000","last_active":"2024-02-12T17:04:52.358290+0000","last_peered":"2024-02-12T17:04:52.358290+0000","last_clean":"2024-02-12T17:04:52.358290+0000","last_became_active":"2024-02-12T17:04:39.884095+0000","last_became_peered":"2024-02-12T17:04:39.884095+0000","last_unstale":"2024-02-12T17:04:52.358290+0000","last_undegraded":"2024-02-12T17:04:52.358290+0000","last_fullsized":"2024-02-12T17:04:52.358290+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"380'16","reported_seq":87,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.433860+0000","last_change":"2024-02-12T17:04:40.143962+0000","last_active":"2024-02-12T17:04:52.433860+0000","last_peered":"2024-02-12T17:04:52.433860+0000","last_clean":"2024-02-12T17:04:52.433860+0000","last_became_active":"2024-02-12T17:04:39.864546+0000","last_became_peered":"2024-02-12T17:04:39.864546+0000","last_unstale":"2024-02-12T17:04:52.433860+0000","last_undegraded":"2024-02-12T17:04:52.433860+0000","last_fullsized":"2024-02-12T17:04:52.433860+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"392'14116","reported_seq":14549,"reported_epoch":392,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:04:53.645787+0000","last_change":"2024-02-12T17:04:40.579124+0000","last_active":"2024-02-12T17:04:53.645787+0000","last_peered":"2024-02-12T17:04:53.645787+0000","last_clean":"2024-02-12T17:03:19.746645+0000","last_became_active":"2024-02-12T17:03:50.650695+0000","last_became_peered":"2024-02-12T17:03:50.650695+0000","last_unstale":"2024-02-12T17:04:53.645787+0000","last_undegraded":"2024-02-12T17:04:53.645787+0000","last_fullsized":"2024-02-12T17:04:53.645787+0000","mapping_epoch":347,"log_start":"345'11012","ondisk_log_start":"345'11012","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3104,"ondisk_log_size":3104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":183762944,"num_objects":2804,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2673,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14116,"num_write_kb":541440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":9961472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,5,0],"avail_no_missing":["7","0","2","5"],"object_location_counts":[{"shards":"0,5,7","objects":2804}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":72,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344213+0000","last_change":"2024-02-12T17:04:33.794769+0000","last_active":"2024-02-12T17:04:52.344213+0000","last_peered":"2024-02-12T17:04:52.344213+0000","last_clean":"2024-02-12T17:04:52.344213+0000","last_became_active":"2024-02-12T17:04:33.794457+0000","last_became_peered":"2024-02-12T17:04:33.794457+0000","last_unstale":"2024-02-12T17:04:52.344213+0000","last_undegraded":"2024-02-12T17:04:52.344213+0000","last_fullsized":"2024-02-12T17:04:52.344213+0000","mapping_epoch":347,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.811153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.811153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"380'16","reported_seq":498,"reported_epoch":391,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:52.436291+0000","last_change":"2024-02-12T17:04:40.884599+0000","last_active":"2024-02-12T17:04:52.436291+0000","last_peered":"2024-02-12T17:04:52.436291+0000","last_clean":"2024-02-12T17:04:37.846594+0000","last_became_active":"2024-02-12T17:04:40.884237+0000","last_became_peered":"2024-02-12T17:04:40.884237+0000","last_unstale":"2024-02-12T17:04:52.436291+0000","last_undegraded":"2024-02-12T17:04:52.436291+0000","last_fullsized":"2024-02-12T17:04:52.436291+0000","mapping_epoch":378,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,4],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.353894+0000","last_change":"2024-02-12T17:04:39.890077+0000","last_active":"2024-02-12T17:04:52.353894+0000","last_peered":"2024-02-12T17:04:52.353894+0000","last_clean":"2024-02-12T17:04:52.353894+0000","last_became_active":"2024-02-12T17:04:39.889588+0000","last_became_peered":"2024-02-12T17:04:39.889588+0000","last_unstale":"2024-02-12T17:04:52.353894+0000","last_undegraded":"2024-02-12T17:04:52.353894+0000","last_fullsized":"2024-02-12T17:04:52.353894+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":87,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345340+0000","last_change":"2024-02-12T17:04:41.814348+0000","last_active":"2024-02-12T17:04:52.345340+0000","last_peered":"2024-02-12T17:04:52.345340+0000","last_clean":"2024-02-12T17:04:52.345340+0000","last_became_active":"2024-02-12T17:04:39.888473+0000","last_became_peered":"2024-02-12T17:04:39.888473+0000","last_unstale":"2024-02-12T17:04:52.345340+0000","last_undegraded":"2024-02-12T17:04:52.345340+0000","last_fullsized":"2024-02-12T17:04:52.345340+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"392'14227","reported_seq":15172,"reported_epoch":392,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:53.634508+0000","last_change":"2024-02-12T17:04:40.406489+0000","last_active":"2024-02-12T17:04:53.634508+0000","last_peered":"2024-02-12T17:04:53.634508+0000","last_clean":"2024-02-12T17:03:19.738005+0000","last_became_active":"2024-02-12T17:03:50.628646+0000","last_became_peered":"2024-02-12T17:03:50.628646+0000","last_unstale":"2024-02-12T17:04:53.634508+0000","last_undegraded":"2024-02-12T17:04:53.634508+0000","last_fullsized":"2024-02-12T17:04:53.634508+0000","mapping_epoch":347,"log_start":"346'11151","ondisk_log_start":"346'11151","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3076,"ondisk_log_size":3076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2729,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14227,"num_write_kb":545600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":26804224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[7,2,0],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":2823}],"blocked_by":[],"up_primary":3,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":402,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.364223+0000","last_change":"2024-02-12T17:04:39.887836+0000","last_active":"2024-02-12T17:04:52.364223+0000","last_peered":"2024-02-12T17:04:52.364223+0000","last_clean":"2024-02-12T17:04:52.364223+0000","last_became_active":"2024-02-12T17:04:39.887488+0000","last_became_peered":"2024-02-12T17:04:39.887488+0000","last_unstale":"2024-02-12T17:04:52.364223+0000","last_undegraded":"2024-02-12T17:04:52.364223+0000","last_fullsized":"2024-02-12T17:04:52.364223+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"381'15","reported_seq":564,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.435578+0000","last_change":"2024-02-12T17:04:42.920681+0000","last_active":"2024-02-12T17:04:52.435578+0000","last_peered":"2024-02-12T17:04:52.435578+0000","last_clean":"2024-02-12T17:04:52.435578+0000","last_became_active":"2024-02-12T17:04:42.920371+0000","last_became_peered":"2024-02-12T17:04:42.920371+0000","last_unstale":"2024-02-12T17:04:52.435578+0000","last_undegraded":"2024-02-12T17:04:52.435578+0000","last_fullsized":"2024-02-12T17:04:52.435578+0000","mapping_epoch":380,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-12T17:02:20.866591+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:02:20.866591+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.354311+0000","last_change":"2024-02-12T17:04:39.886675+0000","last_active":"2024-02-12T17:04:52.354311+0000","last_peered":"2024-02-12T17:04:52.354311+0000","last_clean":"2024-02-12T17:04:52.354311+0000","last_became_active":"2024-02-12T17:04:39.886222+0000","last_became_peered":"2024-02-12T17:04:39.886222+0000","last_unstale":"2024-02-12T17:04:52.354311+0000","last_undegraded":"2024-02-12T17:04:52.354311+0000","last_fullsized":"2024-02-12T17:04:52.354311+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.863530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.863530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":95,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346781+0000","last_change":"2024-02-12T17:04:43.226687+0000","last_active":"2024-02-12T17:04:52.346781+0000","last_peered":"2024-02-12T17:04:52.346781+0000","last_clean":"2024-02-12T17:04:52.346781+0000","last_became_active":"2024-02-12T17:04:39.892266+0000","last_became_peered":"2024-02-12T17:04:39.892266+0000","last_unstale":"2024-02-12T17:04:52.346781+0000","last_undegraded":"2024-02-12T17:04:52.346781+0000","last_fullsized":"2024-02-12T17:04:52.346781+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:01:41.694875+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:01:41.694875+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"392'14371","reported_seq":14491,"reported_epoch":392,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:53.481963+0000","last_change":"2024-02-12T17:04:39.929912+0000","last_active":"2024-02-12T17:04:53.481963+0000","last_peered":"2024-02-12T17:04:53.481963+0000","last_clean":"2024-02-12T17:03:49.455800+0000","last_became_active":"2024-02-12T17:04:39.870350+0000","last_became_peered":"2024-02-12T17:04:39.870350+0000","last_unstale":"2024-02-12T17:04:53.481963+0000","last_undegraded":"2024-02-12T17:04:53.481963+0000","last_fullsized":"2024-02-12T17:04:53.481963+0000","mapping_epoch":377,"log_start":"343'10713","ondisk_log_start":"343'10713","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3658,"ondisk_log_size":3658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":181862400,"num_objects":2775,"num_object_clones":0,"num_object_copies":8325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2775,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14371,"num_write_kb":548672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[2,7,4],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":2775}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":129,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.359744+0000","last_change":"2024-02-12T17:04:39.884597+0000","last_active":"2024-02-12T17:04:52.359744+0000","last_peered":"2024-02-12T17:04:52.359744+0000","last_clean":"2024-02-12T17:04:52.359744+0000","last_became_active":"2024-02-12T17:04:39.884295+0000","last_became_peered":"2024-02-12T17:04:39.884295+0000","last_unstale":"2024-02-12T17:04:52.359744+0000","last_undegraded":"2024-02-12T17:04:52.359744+0000","last_fullsized":"2024-02-12T17:04:52.359744+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"308'11","reported_seq":164,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.437833+0000","last_change":"2024-02-12T17:02:31.547056+0000","last_active":"2024-02-12T17:04:52.437833+0000","last_peered":"2024-02-12T17:04:52.437833+0000","last_clean":"2024-02-12T17:04:52.437833+0000","last_became_active":"2024-02-12T17:01:16.912683+0000","last_became_peered":"2024-02-12T17:01:16.912683+0000","last_unstale":"2024-02-12T17:04:52.437833+0000","last_undegraded":"2024-02-12T17:04:52.437833+0000","last_fullsized":"2024-02-12T17:04:52.437833+0000","mapping_epoch":307,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'11","last_scrub_stamp":"2024-02-12T17:02:31.546965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:31.546965+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":54,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345441+0000","last_change":"2024-02-12T17:03:50.589204+0000","last_active":"2024-02-12T17:04:52.345441+0000","last_peered":"2024-02-12T17:04:52.345441+0000","last_clean":"2024-02-12T17:04:52.345441+0000","last_became_active":"2024-02-12T17:03:50.588844+0000","last_became_peered":"2024-02-12T17:03:50.588844+0000","last_unstale":"2024-02-12T17:04:52.345441+0000","last_undegraded":"2024-02-12T17:04:52.345441+0000","last_fullsized":"2024-02-12T17:04:52.345441+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.888942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.888942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":87,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344301+0000","last_change":"2024-02-12T17:04:42.611814+0000","last_active":"2024-02-12T17:04:52.344301+0000","last_peered":"2024-02-12T17:04:52.344301+0000","last_clean":"2024-02-12T17:04:52.344301+0000","last_became_active":"2024-02-12T17:04:39.891435+0000","last_became_peered":"2024-02-12T17:04:39.891435+0000","last_unstale":"2024-02-12T17:04:52.344301+0000","last_undegraded":"2024-02-12T17:04:52.344301+0000","last_fullsized":"2024-02-12T17:04:52.344301+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"392'14371","reported_seq":14467,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-02-12T17:04:53.627232+0000","last_change":"2024-02-12T17:01:16.913292+0000","last_active":"2024-02-12T17:04:53.627232+0000","last_peered":"2024-02-12T17:04:53.627232+0000","last_clean":"2024-02-12T17:04:53.627232+0000","last_became_active":"2024-02-12T17:01:16.912605+0000","last_became_peered":"2024-02-12T17:01:16.912605+0000","last_unstale":"2024-02-12T17:04:53.627232+0000","last_undegraded":"2024-02-12T17:04:53.627232+0000","last_fullsized":"2024-02-12T17:04:53.627232+0000","mapping_epoch":307,"log_start":"331'6800","ondisk_log_start":"331'6800","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":7571,"ondisk_log_size":7571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":181731328,"num_objects":2773,"num_object_clones":0,"num_object_copies":8319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14371,"num_write_kb":548608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":81,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.346789+0000","last_change":"2024-02-12T17:04:39.875038+0000","last_active":"2024-02-12T17:04:52.346789+0000","last_peered":"2024-02-12T17:04:52.346789+0000","last_clean":"2024-02-12T17:04:52.346789+0000","last_became_active":"2024-02-12T17:04:39.874748+0000","last_became_peered":"2024-02-12T17:04:39.874748+0000","last_unstale":"2024-02-12T17:04:52.346789+0000","last_undegraded":"2024-02-12T17:04:52.346789+0000","last_fullsized":"2024-02-12T17:04:52.346789+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":106,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345850+0000","last_change":"2024-02-12T17:04:39.870194+0000","last_active":"2024-02-12T17:04:52.345850+0000","last_peered":"2024-02-12T17:04:52.345850+0000","last_clean":"2024-02-12T17:04:52.345850+0000","last_became_active":"2024-02-12T17:04:39.869843+0000","last_became_peered":"2024-02-12T17:04:39.869843+0000","last_unstale":"2024-02-12T17:04:52.345850+0000","last_undegraded":"2024-02-12T17:04:52.345850+0000","last_fullsized":"2024-02-12T17:04:52.345850+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:19.084549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.084549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":370,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.356379+0000","last_change":"2024-02-12T17:04:39.888763+0000","last_active":"2024-02-12T17:04:52.356379+0000","last_peered":"2024-02-12T17:04:52.356379+0000","last_clean":"2024-02-12T17:04:52.356379+0000","last_became_active":"2024-02-12T17:04:39.888466+0000","last_became_peered":"2024-02-12T17:04:39.888466+0000","last_unstale":"2024-02-12T17:04:52.356379+0000","last_undegraded":"2024-02-12T17:04:52.356379+0000","last_fullsized":"2024-02-12T17:04:52.356379+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.858955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.858955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":106,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357088+0000","last_change":"2024-02-12T17:04:41.383385+0000","last_active":"2024-02-12T17:04:52.357088+0000","last_peered":"2024-02-12T17:04:52.357088+0000","last_clean":"2024-02-12T17:04:52.357088+0000","last_became_active":"2024-02-12T17:04:39.887688+0000","last_became_peered":"2024-02-12T17:04:39.887688+0000","last_unstale":"2024-02-12T17:04:52.357088+0000","last_undegraded":"2024-02-12T17:04:52.357088+0000","last_fullsized":"2024-02-12T17:04:52.357088+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:01:35.160145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:35.160145+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"392'14112","reported_seq":14222,"reported_epoch":392,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:04:53.750544+0000","last_change":"2024-02-12T17:04:41.128855+0000","last_active":"2024-02-12T17:04:53.750544+0000","last_peered":"2024-02-12T17:04:53.750544+0000","last_clean":"2024-02-12T17:03:49.522755+0000","last_became_active":"2024-02-12T17:03:51.663049+0000","last_became_peered":"2024-02-12T17:03:51.663049+0000","last_unstale":"2024-02-12T17:04:53.750544+0000","last_undegraded":"2024-02-12T17:04:53.750544+0000","last_fullsized":"2024-02-12T17:04:53.750544+0000","mapping_epoch":348,"log_start":"345'11004","ondisk_log_start":"345'11004","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3108,"ondisk_log_size":3108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179175424,"num_objects":2734,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2734,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14112,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":2734}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":404,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.354035+0000","last_change":"2024-02-12T17:04:50.393748+0000","last_active":"2024-02-12T17:04:52.354035+0000","last_peered":"2024-02-12T17:04:52.354035+0000","last_clean":"2024-02-12T17:04:52.354035+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:04:52.354035+0000","last_undegraded":"2024-02-12T17:04:52.354035+0000","last_fullsized":"2024-02-12T17:04:52.354035+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":297,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345683+0000","last_change":"2024-02-12T17:03:28.770200+0000","last_active":"2024-02-12T17:04:52.345683+0000","last_peered":"2024-02-12T17:04:52.345683+0000","last_clean":"2024-02-12T17:04:52.345683+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:04:52.345683+0000","last_undegraded":"2024-02-12T17:04:52.345683+0000","last_fullsized":"2024-02-12T17:04:52.345683+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":114,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.360842+0000","last_change":"2024-02-12T17:03:50.552102+0000","last_active":"2024-02-12T17:04:52.360842+0000","last_peered":"2024-02-12T17:04:52.360842+0000","last_clean":"2024-02-12T17:04:52.360842+0000","last_became_active":"2024-02-12T17:03:50.551804+0000","last_became_peered":"2024-02-12T17:03:50.551804+0000","last_unstale":"2024-02-12T17:04:52.360842+0000","last_undegraded":"2024-02-12T17:04:52.360842+0000","last_fullsized":"2024-02-12T17:04:52.360842+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.794228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":299,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.355596+0000","last_change":"2024-02-12T17:03:52.389245+0000","last_active":"2024-02-12T17:04:52.355596+0000","last_peered":"2024-02-12T17:04:52.355596+0000","last_clean":"2024-02-12T17:04:52.355596+0000","last_became_active":"2024-02-12T17:03:50.574562+0000","last_became_peered":"2024-02-12T17:03:50.574562+0000","last_unstale":"2024-02-12T17:04:52.355596+0000","last_undegraded":"2024-02-12T17:04:52.355596+0000","last_fullsized":"2024-02-12T17:04:52.355596+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:02:45.594719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:45.594719+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"392'14509","reported_seq":23049,"reported_epoch":392,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:53.761891+0000","last_change":"2024-02-12T17:04:41.364381+0000","last_active":"2024-02-12T17:04:53.761891+0000","last_peered":"2024-02-12T17:04:53.761891+0000","last_clean":"2024-02-12T17:03:19.721273+0000","last_became_active":"2024-02-12T17:03:21.956889+0000","last_became_peered":"2024-02-12T17:03:21.956889+0000","last_unstale":"2024-02-12T17:04:53.761891+0000","last_undegraded":"2024-02-12T17:04:53.761891+0000","last_fullsized":"2024-02-12T17:04:53.761891+0000","mapping_epoch":341,"log_start":"346'11411","ondisk_log_start":"346'11411","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3098,"ondisk_log_size":3098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":186449920,"num_objects":2845,"num_object_clones":0,"num_object_copies":8535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":267,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14509,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4210,"num_bytes_recovered":275906560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":2845}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":90,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.354764+0000","last_change":"2024-02-12T17:04:46.024414+0000","last_active":"2024-02-12T17:04:52.354764+0000","last_peered":"2024-02-12T17:04:52.354764+0000","last_clean":"2024-02-12T17:04:52.354764+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:04:52.354764+0000","last_undegraded":"2024-02-12T17:04:52.354764+0000","last_fullsized":"2024-02-12T17:04:52.354764+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.986137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.986137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":300,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344524+0000","last_change":"2024-02-12T17:04:39.886902+0000","last_active":"2024-02-12T17:04:52.344524+0000","last_peered":"2024-02-12T17:04:52.344524+0000","last_clean":"2024-02-12T17:04:52.344524+0000","last_became_active":"2024-02-12T17:04:39.886403+0000","last_became_peered":"2024-02-12T17:04:39.886403+0000","last_unstale":"2024-02-12T17:04:52.344524+0000","last_undegraded":"2024-02-12T17:04:52.344524+0000","last_fullsized":"2024-02-12T17:04:52.344524+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":71,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344287+0000","last_change":"2024-02-12T17:04:41.654733+0000","last_active":"2024-02-12T17:04:52.344287+0000","last_peered":"2024-02-12T17:04:52.344287+0000","last_clean":"2024-02-12T17:04:52.344287+0000","last_became_active":"2024-02-12T17:04:39.889999+0000","last_became_peered":"2024-02-12T17:04:39.889999+0000","last_unstale":"2024-02-12T17:04:52.344287+0000","last_undegraded":"2024-02-12T17:04:52.344287+0000","last_fullsized":"2024-02-12T17:04:52.344287+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:00:50.756578+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.756578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":347,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.344013+0000","last_change":"2024-02-12T17:04:42.178594+0000","last_active":"2024-02-12T17:04:52.344013+0000","last_peered":"2024-02-12T17:04:52.344013+0000","last_clean":"2024-02-12T17:04:52.344013+0000","last_became_active":"2024-02-12T17:04:39.886809+0000","last_became_peered":"2024-02-12T17:04:39.886809+0000","last_unstale":"2024-02-12T17:04:52.344013+0000","last_undegraded":"2024-02-12T17:04:52.344013+0000","last_fullsized":"2024-02-12T17:04:52.344013+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:28.096708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.096708+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"392'14659","reported_seq":14755,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-02-12T17:04:53.588701+0000","last_change":"2024-02-12T17:01:13.886545+0000","last_active":"2024-02-12T17:04:53.588701+0000","last_peered":"2024-02-12T17:04:53.588701+0000","last_clean":"2024-02-12T17:04:53.588701+0000","last_became_active":"2024-02-12T17:01:13.886238+0000","last_became_peered":"2024-02-12T17:01:13.886238+0000","last_unstale":"2024-02-12T17:04:53.588701+0000","last_undegraded":"2024-02-12T17:04:53.588701+0000","last_fullsized":"2024-02-12T17:04:53.588701+0000","mapping_epoch":304,"log_start":"331'7000","ondisk_log_start":"331'7000","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":7659,"ondisk_log_size":7659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":184745984,"num_objects":2819,"num_object_clones":0,"num_object_copies":8457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14659,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":379,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.358541+0000","last_change":"2024-02-12T17:04:41.899891+0000","last_active":"2024-02-12T17:04:52.358541+0000","last_peered":"2024-02-12T17:04:52.358541+0000","last_clean":"2024-02-12T17:04:52.358541+0000","last_became_active":"2024-02-12T17:04:41.899598+0000","last_became_peered":"2024-02-12T17:04:41.899598+0000","last_unstale":"2024-02-12T17:04:52.358541+0000","last_undegraded":"2024-02-12T17:04:52.358541+0000","last_fullsized":"2024-02-12T17:04:52.358541+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:26.803903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.803903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":54,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.345435+0000","last_change":"2024-02-12T17:03:50.560958+0000","last_active":"2024-02-12T17:04:52.345435+0000","last_peered":"2024-02-12T17:04:52.345435+0000","last_clean":"2024-02-12T17:04:52.345435+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:04:52.345435+0000","last_undegraded":"2024-02-12T17:04:52.345435+0000","last_fullsized":"2024-02-12T17:04:52.345435+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:01.789464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:01.789464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":461,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-12T17:04:52.357443+0000","last_change":"2024-02-12T17:04:40.522253+0000","last_active":"2024-02-12T17:04:52.357443+0000","last_peered":"2024-02-12T17:04:52.357443+0000","last_clean":"2024-02-12T17:04:52.357443+0000","last_became_active":"2024-02-12T17:04:39.882260+0000","last_became_peered":"2024-02-12T17:04:39.882260+0000","last_unstale":"2024-02-12T17:04:52.357443+0000","last_undegraded":"2024-02-12T17:04:52.357443+0000","last_fullsized":"2024-02-12T17:04:52.357443+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:02:43.665248+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.665248+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"380'17","reported_seq":616,"reported_epoch":391,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:04:52.435677+0000","last_change":"2024-02-12T17:04:40.872865+0000","last_active":"2024-02-12T17:04:52.435677+0000","last_peered":"2024-02-12T17:04:52.435677+0000","last_clean":"2024-02-12T17:03:47.426268+0000","last_became_active":"2024-02-12T17:04:40.872513+0000","last_became_peered":"2024-02-12T17:04:40.872513+0000","last_unstale":"2024-02-12T17:04:52.435677+0000","last_undegraded":"2024-02-12T17:04:52.435677+0000","last_fullsized":"2024-02-12T17:04:52.435677+0000","mapping_epoch":378,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-02-12T17:00:49.304784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.304784+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,2,4],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2931163160,"num_objects":44727,"num_object_clones":0,"num_object_copies":134181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":37521,"num_objects_unfound":0,"num_objects_dirty":44727,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":230449,"num_write_kb":8805569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12877,"num_bytes_recovered":836042752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9348722688,"data_stored":9348710472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69604,"ondisk_log_size":69604,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":6},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":222,"num_read_kb":217,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1093,"num_read_kb":850,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":11090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234170,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2763600,"kb_used_data":2165704,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":90981552,"statfs":{"total":95995035648,"available":93165109248,"internally_reserved":0,"allocated":2217680896,"data_stored":2217154797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":3.5419999999999998,"5min":2.1789999999999998,"15min":1.619},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.375,"5min":16.759,"15min":36.439},"last":2.23},{"interface":"front","average":{"1min":1.6799999999999999,"5min":2.4820000000000002,"15min":1.736},"min":{"1min":0.23100000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.8689999999999998,"5min":17.844000000000001,"15min":36.490000000000002},"last":1.494}]},{"osd":1,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":2.262,"5min":1.8580000000000001,"15min":1.3089999999999999},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.20899999999999999},"max":{"1min":6.3479999999999999,"5min":11.048999999999999,"15min":12.032},"last":6.0730000000000004},{"interface":"front","average":{"1min":1.8480000000000001,"5min":2.194,"15min":1.399},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":5.8949999999999996,"5min":17.324999999999999,"15min":17.324999999999999},"last":1.5229999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":3.6949999999999998,"5min":1.8660000000000001,"15min":1.284},"min":{"1min":0.249,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.4,"5min":12.4,"15min":12.4},"last":2.3010000000000002},{"interface":"front","average":{"1min":2.2930000000000001,"5min":1.897,"15min":1.452},"min":{"1min":0.26400000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":6.3540000000000001,"5min":11.071999999999999,"15min":20.338000000000001},"last":6.1559999999999997}]},{"osd":3,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":3.702,"5min":1.9890000000000001,"15min":1.5660000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":13.608000000000001,"5min":13.608000000000001,"15min":20.433},"last":2.2799999999999998},{"interface":"front","average":{"1min":3.6349999999999998,"5min":2.2639999999999998,"15min":1.7569999999999999},"min":{"1min":0.34300000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":31.152000000000001,"5min":31.152000000000001,"15min":31.152000000000001},"last":6.1299999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":4.9829999999999997,"5min":2.6200000000000001,"15min":1.5820000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":19.030000000000001,"5min":19.030000000000001,"15min":50.406999999999996},"last":2.3170000000000002},{"interface":"front","average":{"1min":4.2670000000000003,"5min":2.9420000000000002,"15min":1.8260000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.23400000000000001},"max":{"1min":16.867000000000001,"5min":16.867000000000001,"15min":50.168999999999997},"last":6.1079999999999997}]},{"osd":5,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":5.274,"5min":3.0990000000000002,"15min":1.5740000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.24199999999999999},"max":{"1min":19.702000000000002,"5min":19.702000000000002,"15min":19.702000000000002},"last":2.206},{"interface":"front","average":{"1min":2.875,"5min":2.8719999999999999,"15min":1.609},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26200000000000001},"max":{"1min":11.122,"5min":18.651,"15min":18.651},"last":6.2800000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:03:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":2.5609999999999999,"15min":1.4530000000000001},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":10.522,"5min":17.234999999999999,"15min":17.234999999999999},"last":1.4550000000000001},{"interface":"front","average":{"1min":3.9660000000000002,"5min":2.2709999999999999,"15min":1.411},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":13.676,"5min":13.676,"15min":13.676},"last":2.2610000000000001}]}]},{"osd":6,"up_from":0,"seq":193273528511,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738588,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2914452,"kb_used_data":2010060,"kb_used_omap":0,"kb_used_meta":904384,"kb_avail":90830700,"statfs":{"total":95995035648,"available":93010636800,"internally_reserved":0,"allocated":2058301440,"data_stored":2057898513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":926089216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:04:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.83999999999999997,"15min":1.2549999999999999},"min":{"1min":0.221,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.2469999999999999,"5min":13.013,"15min":23.510999999999999},"last":2.1819999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.92600000000000005,"15min":1.222},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":16.16,"5min":16.16,"15min":16.16},"last":2.8679999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.89100000000000001,"15min":1.4019999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.7759999999999998,"5min":11.122999999999999,"15min":16.521000000000001},"last":4.6849999999999996},{"interface":"front","average":{"1min":1.04,"5min":0.99299999999999999,"15min":1.296},"min":{"1min":0.20799999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.3209999999999997,"5min":6.3209999999999997,"15min":10.340999999999999},"last":5.0979999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.141,"15min":1.5569999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.19400000000000001},"max":{"1min":6.923,"5min":7.0339999999999998,"15min":21.100000000000001},"last":2.9079999999999999},{"interface":"front","average":{"1min":1.9770000000000001,"5min":1.484,"15min":1.623},"min":{"1min":0.26200000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":16.759,"5min":16.759,"15min":16.759},"last":2.3340000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:04:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.8009999999999999,"15min":1.5349999999999999},"min":{"1min":0.42699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.6849999999999996,"5min":16.888999999999999,"15min":16.888999999999999},"last":3.6920000000000002},{"interface":"front","average":{"1min":1.778,"5min":1.8500000000000001,"15min":1.5069999999999999},"min":{"1min":0.40100000000000002,"5min":0.28799999999999998,"15min":0.28399999999999997},"max":{"1min":4.9320000000000004,"5min":16.899999999999999,"15min":16.899999999999999},"last":4.9779999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:04:13 2024","interfaces":[{"interface":"back","average":{"1min":2.2789999999999999,"5min":2.48,"15min":1.976},"min":{"1min":0.48199999999999998,"5min":0.30199999999999999,"15min":0.26400000000000001},"max":{"1min":7.423,"5min":50.404000000000003,"15min":50.404000000000003},"last":1.8560000000000001},{"interface":"front","average":{"1min":2.4180000000000001,"5min":2.407,"15min":1.8740000000000001},"min":{"1min":0.29599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":12.666,"5min":49.979999999999997,"15min":49.979999999999997},"last":2.1099999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:04:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.542,"15min":1.534},"min":{"1min":0.35199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.7510000000000003,"5min":9.8460000000000001,"15min":17.907},"last":2.9590000000000001},{"interface":"front","average":{"1min":2.137,"5min":2.0619999999999998,"15min":1.8540000000000001},"min":{"1min":0.41599999999999998,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":7.7510000000000003,"5min":9.6739999999999995,"15min":21.062000000000001},"last":3.0139999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:04:08 2024","interfaces":[{"interface":"back","average":{"1min":3.117,"5min":2.1560000000000001,"15min":1.6639999999999999},"min":{"1min":0.48799999999999999,"5min":0.39100000000000001,"15min":0.32400000000000001},"max":{"1min":19.373000000000001,"5min":19.373000000000001,"15min":19.373000000000001},"last":7.6180000000000003},{"interface":"front","average":{"1min":1.8220000000000001,"5min":2.496,"15min":1.6950000000000001},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.33600000000000002},"max":{"1min":7.9390000000000001,"5min":34.722999999999999,"15min":34.722999999999999},"last":5.008}]}]},{"osd":2,"up_from":18,"seq":77309411537,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2349920,"kb_used_data":1457240,"kb_used_omap":0,"kb_used_meta":892672,"kb_avail":91395232,"statfs":{"total":95995035648,"available":93588717568,"internally_reserved":0,"allocated":1492213760,"data_stored":1491735307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914096128},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.1060000000000001,"15min":1.5509999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.6269999999999998,"5min":36.100999999999999,"15min":36.100999999999999},"last":0.246},{"interface":"front","average":{"1min":0.443,"5min":1.3939999999999999,"15min":1.464},"min":{"1min":0.215,"5min":0.193,"15min":0.158},"max":{"1min":1.7310000000000001,"5min":36.299999999999997,"15min":36.299999999999997},"last":0.71699999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.79100000000000004,"15min":1.4830000000000001},"min":{"1min":0.23799999999999999,"5min":0.215,"15min":0.191},"max":{"1min":1.419,"5min":6.5819999999999999,"15min":17.739000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.92700000000000005,"15min":1.5269999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.1679999999999999,"5min":4.3099999999999996,"15min":15.766999999999999},"last":0.28000000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:04:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.048,"15min":1.734},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.23499999999999999},"max":{"1min":1.9370000000000001,"5min":5.1239999999999997,"15min":51.579999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1,"15min":1.468},"min":{"1min":0.26400000000000001,"5min":0.23699999999999999,"15min":0.23200000000000001},"max":{"1min":4.0499999999999998,"5min":4.5449999999999999,"15min":48.988999999999997},"last":0.68200000000000005}]},{"osd":4,"last update":"Mon Feb 12 17:03:59 2024","interfaces":[{"interface":"back","average":{"1min":2.383,"5min":1.964,"15min":1.6850000000000001},"min":{"1min":0.35999999999999999,"5min":0.253,"15min":0.253},"max":{"1min":18.34,"5min":18.658999999999999,"15min":18.658999999999999},"last":0.31},{"interface":"front","average":{"1min":2.5139999999999998,"5min":2.2909999999999999,"15min":1.6899999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":18.355,"5min":18.355,"15min":18.355},"last":0.78100000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:04:08 2024","interfaces":[{"interface":"back","average":{"1min":3.4649999999999999,"5min":2.0099999999999998,"15min":1.77},"min":{"1min":0.35899999999999999,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":32.57,"5min":32.57,"15min":32.57},"last":1.645},{"interface":"front","average":{"1min":4.5380000000000003,"5min":2.4649999999999999,"15min":1.9830000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":31.905999999999999,"5min":31.905999999999999,"15min":31.905999999999999},"last":0.41999999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:04:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.597,"15min":1.891},"min":{"1min":0.46700000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.6669999999999998,"5min":7.6669999999999998,"15min":17.782},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.218,"5min":1.8,"15min":1.6479999999999999},"min":{"1min":0.39000000000000001,"5min":0.35499999999999998,"15min":0.29799999999999999},"max":{"1min":4.008,"5min":8.7850000000000001,"15min":13.446999999999999},"last":1.7809999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:04:08 2024","interfaces":[{"interface":"back","average":{"1min":6.7229999999999999,"5min":2.9409999999999998,"15min":2.0449999999999999},"min":{"1min":0.46500000000000002,"5min":0.33700000000000002,"15min":0.32600000000000001},"max":{"1min":45.468000000000004,"5min":45.468000000000004,"15min":45.468000000000004},"last":7.3920000000000003},{"interface":"front","average":{"1min":7.6079999999999997,"5min":3.1850000000000001,"15min":2.2130000000000001},"min":{"1min":0.40400000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":46.656999999999996,"5min":46.656999999999996,"15min":46.656999999999996},"last":0.38400000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149901,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563776,"kb_used_data":254200,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93181376,"statfs":{"total":95995035648,"available":95417729024,"internally_reserved":0,"allocated":260300800,"data_stored":259790956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:04:32 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.232,"15min":1.5489999999999999},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":5.415,"5min":58.451999999999998,"15min":58.451999999999998},"last":0.218},{"interface":"front","average":{"1min":0.83099999999999996,"5min":1.369,"15min":1.5289999999999999},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.6349999999999998,"5min":57.497999999999998,"15min":57.497999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.93500000000000005,"15min":1.3240000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.216},"max":{"1min":3.6499999999999999,"5min":4.782,"15min":13.94},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.75600000000000001,"15min":1.2370000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":4.7709999999999999,"5min":4.7709999999999999,"15min":14.962},"last":0.47599999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:04:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.872,"15min":1.5},"min":{"1min":0.33200000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.431,"5min":4.6079999999999997,"15min":49.112000000000002},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.81999999999999995,"15min":1.6240000000000001},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.9420000000000002,"5min":4.9420000000000002,"15min":51.997999999999998},"last":0.42699999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.722,"15min":1.7070000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":4.8710000000000004,"5min":11.047000000000001,"15min":19.010000000000002},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.903,"5min":2.0790000000000002,"15min":1.655},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":10.250999999999999,"5min":19.033999999999999,"15min":19.033999999999999},"last":0.44}]},{"osd":5,"last update":"Mon Feb 12 17:04:14 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.1699999999999999,"15min":1.837},"min":{"1min":0.434,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":14.117000000000001,"5min":16.076000000000001,"15min":19.882999999999999},"last":4.2249999999999996},{"interface":"front","average":{"1min":2.3260000000000001,"5min":2.5569999999999999,"15min":1.833},"min":{"1min":0.40899999999999997,"5min":0.34699999999999998,"15min":0.33300000000000002},"max":{"1min":8.3689999999999998,"5min":15.595000000000001,"15min":15.595000000000001},"last":12.502000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":1.5169999999999999,"15min":1.425},"min":{"1min":0.48399999999999999,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":5.3129999999999997,"5min":6.6150000000000002,"15min":12.856999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.748,"15min":1.6240000000000001},"min":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.27500000000000002},"max":{"1min":4.9729999999999999,"5min":12.635999999999999,"15min":19.925999999999998},"last":0.46500000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:04:51 2024","interfaces":[{"interface":"back","average":{"1min":1.994,"5min":2.2989999999999999,"15min":1.907},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":5.585,"5min":12.695,"15min":12.891},"last":5.585},{"interface":"front","average":{"1min":2.617,"5min":2.4529999999999998,"15min":1.923},"min":{"1min":0.49099999999999999,"5min":0.40799999999999997,"15min":0.32800000000000001},"max":{"1min":5.1369999999999996,"5min":25.838999999999999,"15min":25.838999999999999},"last":3.9740000000000002}]}]},{"osd":4,"up_from":0,"seq":137438953673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757445,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3119684,"kb_used_data":2223100,"kb_used_omap":0,"kb_used_meta":896576,"kb_avail":90625468,"statfs":{"total":95995035648,"available":92800479232,"internally_reserved":0,"allocated":2276454400,"data_stored":2275939609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":918093824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":2.2890000000000001,"15min":1.6200000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20300000000000001},"max":{"1min":5.8070000000000004,"5min":46.365000000000002,"15min":46.365000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":2.1800000000000002,"5min":2.6850000000000001,"15min":1.952},"min":{"1min":0.28499999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0709999999999997,"5min":46.134,"15min":46.134},"last":1.6799999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.7210000000000001,"15min":1.4930000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":3.9329999999999998,"5min":8.1099999999999994,"15min":16.655000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":2.1360000000000001,"5min":2.1779999999999999,"15min":1.7070000000000001},"min":{"1min":0.315,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.0990000000000002,"5min":13.130000000000001,"15min":16.712},"last":1.708}]},{"osd":2,"last update":"Mon Feb 12 17:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":2.0649999999999999,"15min":1.7270000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.2549999999999999,"5min":11.15,"15min":18.218},"last":0.245},{"interface":"front","average":{"1min":1.3049999999999999,"5min":2.0230000000000001,"15min":1.6699999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":3.968,"5min":9.1199999999999992,"15min":18.327000000000002},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Feb 12 17:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.9550000000000001,"15min":1.627},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":3.4049999999999998,"5min":11.122,"15min":18.251000000000001},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.9079999999999999,"15min":1.571},"min":{"1min":0.309,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.903,"5min":8.0790000000000006,"15min":20.263000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Feb 12 17:04:11 2024","interfaces":[{"interface":"back","average":{"1min":2.9209999999999998,"5min":2.819,"15min":1.542},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":8.0399999999999991,"5min":11.382,"15min":11.880000000000001},"last":2.0600000000000001},{"interface":"front","average":{"1min":2.3330000000000002,"5min":2.8799999999999999,"15min":1.7030000000000001},"min":{"1min":0.28499999999999998,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":10.327,"5min":20.550000000000001,"15min":20.550000000000001},"last":2.9089999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:04:39 2024","interfaces":[{"interface":"back","average":{"1min":3,"5min":2.5819999999999999,"15min":1.6060000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.26400000000000001},"max":{"1min":12.121,"5min":12.281000000000001,"15min":12.281000000000001},"last":1.728},{"interface":"front","average":{"1min":3.1720000000000002,"5min":2.802,"15min":1.728},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":11.087,"5min":14.323,"15min":22.035},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:04:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8130000000000002,"5min":3.0459999999999998,"15min":1.645},"min":{"1min":0.31900000000000001,"5min":0.29499999999999998,"15min":0.27500000000000002},"max":{"1min":11.275,"5min":19.184999999999999,"15min":19.184999999999999},"last":6.9589999999999996},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.984,"15min":1.6599999999999999},"min":{"1min":0.38100000000000001,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":8.8729999999999993,"5min":12.253,"15min":12.253},"last":2.2639999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2024538112,"data_stored":2024538112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1474625536,"data_stored":1474625536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258473984,"data_stored":258473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1163726848,"data_stored":1163722776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2241925120,"data_stored":2241921048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2185433088,"data_stored":2185429016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:04:59.845 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-12T17:04:59.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.control pg_num 12 2024-02-12T17:05:00.495 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:00.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:00.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:00.662 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:00.729 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:00.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:00.834 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:00.889 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:01.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:05:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:00] "GET /metrics HTTP/1.1" 200 176162 "" "Prometheus/2.33.4" 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.15459.0:236277 9.2 e397/347 9:41f7a646:::benchmark_data_smithi033_131576_object98687:head v 397'14579, mlcod=397'14575)", 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:01.140024+0000", 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0040385830000000001, 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040669210000000003, 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:05:01.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:01.140024+0000", 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:01.140024+0000", 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:01.140023+0000", 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0862999999999999e-05 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:01.140034+0000", 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2342e-05 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:01.140046+0000", 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9609999999999999e-06 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:01.140053+0000", 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5204000000000002e-05 2024-02-12T17:05:01.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:01.140108+0000", 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1118999999999999e-05 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:01.140129+0000", 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104985 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:01.393 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:01.394 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:05:01.394 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 1 2024-02-12T17:05:01.394 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:05:01.657 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:01.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:01.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:01.837 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:01.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:01.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:05:01.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:00.660013+0000 mgr.y (mgr.24481) 584 : cluster [DBG] pgmap v780: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 971 B/s rd, 0 B/s wr, 737 op/s; 34098/123102 objects misplaced (27.699%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:05:01.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:01.148914+0000 osd.3 (osd.3) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:01.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:01.148972+0000 osd.3 (osd.3) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:01.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:01.308200+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e398: 8 total, 8 up, 5 in 2024-02-12T17:05:02.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:00.660013+0000 mgr.y (mgr.24481) 584 : cluster [DBG] pgmap v780: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 971 B/s rd, 0 B/s wr, 737 op/s; 34098/123102 objects misplaced (27.699%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:05:02.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:01.148914+0000 osd.3 (osd.3) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:02.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:01.148972+0000 osd.3 (osd.3) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:02.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:01.308200+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e398: 8 total, 8 up, 5 in 2024-02-12T17:05:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:00.660013+0000 mgr.y (mgr.24481) 584 : cluster [DBG] pgmap v780: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 971 B/s rd, 0 B/s wr, 737 op/s; 34098/123102 objects misplaced (27.699%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:05:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:01.148914+0000 osd.3 (osd.3) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:01.148972+0000 osd.3 (osd.3) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:01.308200+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e398: 8 total, 8 up, 5 in 2024-02-12T17:05:02.044 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:05:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:02.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:05:02.106 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:02.108 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:02.139 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T17:05:02.204 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:02.284 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:02.354 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:02.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:02.463 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:02.509 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:02.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:05:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:05:03.347 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:05:04.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:02.660470+0000 mgr.y (mgr.24481) 585 : cluster [DBG] pgmap v782: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 648 op/s; 34098/123102 objects misplaced (27.699%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T17:05:04.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:02.660470+0000 mgr.y (mgr.24481) 585 : cluster [DBG] pgmap v782: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 648 op/s; 34098/123102 objects misplaced (27.699%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T17:05:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:02.660470+0000 mgr.y (mgr.24481) 585 : cluster [DBG] pgmap v782: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 648 op/s; 34098/123102 objects misplaced (27.699%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T17:05:04.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:05:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:05:05.288 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:05.298 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:05.319 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:05.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:05.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:05:05.503 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:05:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:05.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:05:05.652 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:05:05.720 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:05.754 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:05:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:05.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:05:05.790 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:04.333481+0000 mgr.y (mgr.24481) 586 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:04.422081+0000 osd.7 (osd.7) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:04.422134+0000 osd.7 (osd.7) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:04.661114+0000 mgr.y (mgr.24481) 587 : cluster [DBG] pgmap v783: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.05k op/s; 30727/111966 objects misplaced (27.443%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:04.766598+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e399: 8 total, 8 up, 5 in 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:05.106670+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.33:0/2871624160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:04.333481+0000 mgr.y (mgr.24481) 586 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:04.422081+0000 osd.7 (osd.7) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:04.422134+0000 osd.7 (osd.7) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:05.848 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:04.661114+0000 mgr.y (mgr.24481) 587 : cluster [DBG] pgmap v783: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.05k op/s; 30727/111966 objects misplaced (27.443%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T17:05:05.849 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:04.766598+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e399: 8 total, 8 up, 5 in 2024-02-12T17:05:05.849 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:05.106670+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.33:0/2871624160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:05.860 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:05.930 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:05.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:06.058 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:06.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:06.244 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:06.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:04.333481+0000 mgr.y (mgr.24481) 586 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:06.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:04.422081+0000 osd.7 (osd.7) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:06.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:04.422134+0000 osd.7 (osd.7) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:06.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:04.661114+0000 mgr.y (mgr.24481) 587 : cluster [DBG] pgmap v783: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.05k op/s; 30727/111966 objects misplaced (27.443%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T17:05:06.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:04.766598+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e399: 8 total, 8 up, 5 in 2024-02-12T17:05:06.245 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:05.106670+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.33:0/2871624160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:06.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:06.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:06.399 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:06.436 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:06.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:06.544 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:07.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:05.522253+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:07.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:05.522312+0000 osd.6 (osd.6) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:07.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:05.764743+0000 mon.a (mon.0) 1484 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:05:07.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:05.779749+0000 mon.a (mon.0) 1485 : audit [INF] from='client.? v1:172.21.15.33:0/2871624160' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:07.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:05.779795+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e400: 8 total, 8 up, 5 in 2024-02-12T17:05:07.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:05.522253+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:07.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:05.522312+0000 osd.6 (osd.6) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:07.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:05.764743+0000 mon.a (mon.0) 1484 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:05:07.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:05.779749+0000 mon.a (mon.0) 1485 : audit [INF] from='client.? v1:172.21.15.33:0/2871624160' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:07.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:05.779795+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e400: 8 total, 8 up, 5 in 2024-02-12T17:05:07.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:05.522253+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:07.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:05.522312+0000 osd.6 (osd.6) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:07.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:05.764743+0000 mon.a (mon.0) 1484 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:05:07.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:05.779749+0000 mon.a (mon.0) 1485 : audit [INF] from='client.? v1:172.21.15.33:0/2871624160' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:07.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:05.779795+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e400: 8 total, 8 up, 5 in 2024-02-12T17:05:07.464 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:07.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:07.571 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:07.615 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:07.983 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:07.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:08.090 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:08.118 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:06.661505+0000 mgr.y (mgr.24481) 588 : cluster [DBG] pgmap v786: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 619 op/s; 30727/111966 objects misplaced (27.443%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:08.118 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:07.101528+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.33:0/1113300489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]: dispatch 2024-02-12T17:05:08.119 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:07.102146+0000 mon.a (mon.0) 1487 : audit [INF] from='client.35648 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]: dispatch 2024-02-12T17:05:08.119 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:07.278354+0000 osd.4 (osd.4) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:08.119 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:07.278426+0000 osd.4 (osd.4) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:08.119 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:06.661505+0000 mgr.y (mgr.24481) 588 : cluster [DBG] pgmap v786: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 619 op/s; 30727/111966 objects misplaced (27.443%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:08.119 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:07.101528+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.33:0/1113300489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]: dispatch 2024-02-12T17:05:08.119 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:07.102146+0000 mon.a (mon.0) 1487 : audit [INF] from='client.35648 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]: dispatch 2024-02-12T17:05:08.119 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:07.278354+0000 osd.4 (osd.4) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:08.120 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:07.278426+0000 osd.4 (osd.4) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:08.129 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:08.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:06.661505+0000 mgr.y (mgr.24481) 588 : cluster [DBG] pgmap v786: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 619 op/s; 30727/111966 objects misplaced (27.443%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:08.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:07.101528+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.33:0/1113300489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]: dispatch 2024-02-12T17:05:08.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:07.102146+0000 mon.a (mon.0) 1487 : audit [INF] from='client.35648 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]: dispatch 2024-02-12T17:05:08.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:07.278354+0000 osd.4 (osd.4) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:08.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:07.278426+0000 osd.4 (osd.4) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:08.272 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 5 pg_num to 12 2024-02-12T17:05:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:07.783205+0000 mon.a (mon.0) 1488 : audit [INF] from='client.35648 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]': finished 2024-02-12T17:05:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:07.783294+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e401: 8 total, 8 up, 5 in 2024-02-12T17:05:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:08.662669+0000 mon.c (mon.2) 216 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-12T17:05:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:08.666926+0000 mon.a (mon.0) 1490 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-12T17:05:09.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:07.783205+0000 mon.a (mon.0) 1488 : audit [INF] from='client.35648 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]': finished 2024-02-12T17:05:09.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:07.783294+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e401: 8 total, 8 up, 5 in 2024-02-12T17:05:09.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:08.662669+0000 mon.c (mon.2) 216 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-12T17:05:09.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:08.666926+0000 mon.a (mon.0) 1490 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-12T17:05:09.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:07.783205+0000 mon.a (mon.0) 1488 : audit [INF] from='client.35648 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "12"}]': finished 2024-02-12T17:05:09.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:07.783294+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e401: 8 total, 8 up, 5 in 2024-02-12T17:05:09.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:08.662669+0000 mon.c (mon.2) 216 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-12T17:05:09.246 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:08.666926+0000 mon.a (mon.0) 1490 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-12T17:05:09.982 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:05:10.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:08.662209+0000 mgr.y (mgr.24481) 589 : cluster [DBG] pgmap v788: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.2 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 895 op/s; 29335/106986 objects misplaced (27.419%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:10.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:08.813227+0000 mon.a (mon.0) 1491 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]': finished 2024-02-12T17:05:10.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:08.813305+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e402: 8 total, 8 up, 5 in 2024-02-12T17:05:10.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:08.662209+0000 mgr.y (mgr.24481) 589 : cluster [DBG] pgmap v788: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.2 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 895 op/s; 29335/106986 objects misplaced (27.419%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:10.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:08.813227+0000 mon.a (mon.0) 1491 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]': finished 2024-02-12T17:05:10.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:08.813305+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e402: 8 total, 8 up, 5 in 2024-02-12T17:05:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:08.662209+0000 mgr.y (mgr.24481) 589 : cluster [DBG] pgmap v788: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.2 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 895 op/s; 29335/106986 objects misplaced (27.419%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:08.813227+0000 mon.a (mon.0) 1491 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "12"}]': finished 2024-02-12T17:05:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:08.813305+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e402: 8 total, 8 up, 5 in 2024-02-12T17:05:10.992 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:05:10.992 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:05:10.992 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:05:10.992 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:05:10.993 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:05:11.108 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:05:11.128 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:05:11.157 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:05:11.191 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:05:11.217 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:05:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:09.823547+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e403: 8 total, 8 up, 5 in 2024-02-12T17:05:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:09.834903+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.f starting backfill to osd.0 from (0'0,0'0] MAX to 380'16 2024-02-12T17:05:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:09.835015+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.f starting backfill to osd.7 from (0'0,0'0] MAX to 380'16 2024-02-12T17:05:11.246 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:05:11.272 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:05:11.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:05:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:10] "GET /metrics HTTP/1.1" 200 176195 "" "Prometheus/2.33.4" 2024-02-12T17:05:11.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:09.823547+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e403: 8 total, 8 up, 5 in 2024-02-12T17:05:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:09.834903+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.f starting backfill to osd.0 from (0'0,0'0] MAX to 380'16 2024-02-12T17:05:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:09.835015+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.f starting backfill to osd.7 from (0'0,0'0] MAX to 380'16 2024-02-12T17:05:11.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:09.823547+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e403: 8 total, 8 up, 5 in 2024-02-12T17:05:11.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:09.834903+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.f starting backfill to osd.0 from (0'0,0'0] MAX to 380'16 2024-02-12T17:05:11.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:09.835015+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.f starting backfill to osd.7 from (0'0,0'0] MAX to 380'16 2024-02-12T17:05:11.293 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:05:11.297 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:05:11.297 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:05:11.297 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:05:11.297 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:05:11.297 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:05:11.297 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:05:11.297 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:05:11.297 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:05:11.553 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:11.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:11.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:11.664 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:11.720 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:11.750 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:11.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:11.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:11.969 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:11.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:12.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:10.662695+0000 mgr.y (mgr.24481) 590 : cluster [DBG] pgmap v791: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.0 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s; 27287/100509 objects misplaced (27.149%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:05:12.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:10.870750+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e404: 8 total, 8 up, 5 in 2024-02-12T17:05:12.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:10.890756+0000 osd.5 (osd.5) 187 : cluster [DBG] 5.e starting backfill to osd.7 from (0'0,0'0] MAX to 381'15 2024-02-12T17:05:12.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:10.892769+0000 osd.7 (osd.7) 193 : cluster [DBG] 5.d starting backfill to osd.3 from (0'0,0'0] MAX to 308'11 2024-02-12T17:05:12.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:11.074503+0000 mon.b (mon.1) 80 : audit [DBG] from='client.? v1:172.21.15.33:0/1535993604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:12.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:11.311795+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e405: 8 total, 8 up, 5 in 2024-02-12T17:05:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:10.662695+0000 mgr.y (mgr.24481) 590 : cluster [DBG] pgmap v791: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.0 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s; 27287/100509 objects misplaced (27.149%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:05:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:10.870750+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e404: 8 total, 8 up, 5 in 2024-02-12T17:05:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:10.890756+0000 osd.5 (osd.5) 187 : cluster [DBG] 5.e starting backfill to osd.7 from (0'0,0'0] MAX to 381'15 2024-02-12T17:05:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:10.892769+0000 osd.7 (osd.7) 193 : cluster [DBG] 5.d starting backfill to osd.3 from (0'0,0'0] MAX to 308'11 2024-02-12T17:05:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:11.074503+0000 mon.b (mon.1) 80 : audit [DBG] from='client.? v1:172.21.15.33:0/1535993604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:11.311795+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e405: 8 total, 8 up, 5 in 2024-02-12T17:05:12.079 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:12.150 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:12.194 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:12.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:12.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:10.662695+0000 mgr.y (mgr.24481) 590 : cluster [DBG] pgmap v791: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.0 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s; 27287/100509 objects misplaced (27.149%); 2.9 MiB/s, 46 objects/s recovering 2024-02-12T17:05:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:10.870750+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e404: 8 total, 8 up, 5 in 2024-02-12T17:05:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:10.890756+0000 osd.5 (osd.5) 187 : cluster [DBG] 5.e starting backfill to osd.7 from (0'0,0'0] MAX to 381'15 2024-02-12T17:05:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:10.892769+0000 osd.7 (osd.7) 193 : cluster [DBG] 5.d starting backfill to osd.3 from (0'0,0'0] MAX to 308'11 2024-02-12T17:05:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:11.074503+0000 mon.b (mon.1) 80 : audit [DBG] from='client.? v1:172.21.15.33:0/1535993604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:11.311795+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e405: 8 total, 8 up, 5 in 2024-02-12T17:05:12.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:12.372 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:12.432 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:12.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:12.538 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:05:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:05:12.632 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:05:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:05:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:12.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:05:12.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:12.764 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T17:05:12.789 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:12.792 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:12.823 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:12.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:12.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:12.978 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:13.016 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:13.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:13.125 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:13.172 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:13.227 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:13.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:13.337 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:13.412 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:12.326669+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e406: 8 total, 8 up, 5 in 2024-02-12T17:05:13.412 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:12.663139+0000 mgr.y (mgr.24481) 591 : cluster [DBG] pgmap v795: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.0 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 27287/100509 objects misplaced (27.149%) 2024-02-12T17:05:13.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:12.326669+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e406: 8 total, 8 up, 5 in 2024-02-12T17:05:13.413 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:12.663139+0000 mgr.y (mgr.24481) 591 : cluster [DBG] pgmap v795: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.0 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 27287/100509 objects misplaced (27.149%) 2024-02-12T17:05:13.419 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:12.326669+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e406: 8 total, 8 up, 5 in 2024-02-12T17:05:13.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:12.663139+0000 mgr.y (mgr.24481) 591 : cluster [DBG] pgmap v795: 128 pgs: 4 active+remapped+backfill_toofull, 7 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 2.0 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 27287/100509 objects misplaced (27.149%) 2024-02-12T17:05:13.785 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:13.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:13.889 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:13.944 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:14.330 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:05:14.341 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:14.520 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:14.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:14.625 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:14.681 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:14.705 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:14.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:14.736 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:14.664269+0000 mon.c (mon.2) 217 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T17:05:14.736 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:14.667803+0000 mon.a (mon.0) 1497 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T17:05:14.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:05:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:05:14.818 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:14.866 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:14.941 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:14.943 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:14.664269+0000 mon.c (mon.2) 217 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T17:05:14.944 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:14.667803+0000 mon.a (mon.0) 1497 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T17:05:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:14.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:14.947 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:14.664269+0000 mon.c (mon.2) 217 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T17:05:14.947 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:14.667803+0000 mon.a (mon.0) 1497 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T17:05:15.046 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:15.128 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:05:15.128 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:05:15.129 INFO:tasks.thrashosds.thrasher:Growing pool default.rgw.control 2024-02-12T17:05:15.129 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:05:15.129 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:15.234 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:15.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:15.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:15.439 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:15.675 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:15.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:15.784 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:15.838 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:14.349432+0000 mgr.y (mgr.24481) 592 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:14.663886+0000 mgr.y (mgr.24481) 593 : cluster [DBG] pgmap v796: 128 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 1.8 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 772 op/s; 23845/89310 objects misplaced (26.699%); 5.4 MiB/s, 87 objects/s recovering 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:14.722115+0000 mon.a (mon.0) 1498 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:14.749614+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:14.749685+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e407: 8 total, 8 up, 5 in 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:14.349432+0000 mgr.y (mgr.24481) 592 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:14.663886+0000 mgr.y (mgr.24481) 593 : cluster [DBG] pgmap v796: 128 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 1.8 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 772 op/s; 23845/89310 objects misplaced (26.699%); 5.4 MiB/s, 87 objects/s recovering 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:14.722115+0000 mon.a (mon.0) 1498 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:05:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:14.749614+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-12T17:05:16.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:14.749685+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e407: 8 total, 8 up, 5 in 2024-02-12T17:05:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:14.349432+0000 mgr.y (mgr.24481) 592 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:14.663886+0000 mgr.y (mgr.24481) 593 : cluster [DBG] pgmap v796: 128 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 1.8 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 772 op/s; 23845/89310 objects misplaced (26.699%); 5.4 MiB/s, 87 objects/s recovering 2024-02-12T17:05:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:14.722115+0000 mon.a (mon.0) 1498 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:05:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:14.749614+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-12T17:05:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:14.749685+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e407: 8 total, 8 up, 5 in 2024-02-12T17:05:16.503 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:16.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:16.511 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:16.567 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:16.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:16.659 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:16.761 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:15.747760+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.33:0/2066471646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:05:16.761 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:15.758125+0000 mon.a (mon.0) 1501 : audit [INF] from='client.35654 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:05:16.761 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:15.766658+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e408: 8 total, 8 up, 5 in 2024-02-12T17:05:16.796 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:05:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:16.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:16.914 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:16.974 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:17.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:15.747760+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.33:0/2066471646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:05:17.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:15.758125+0000 mon.a (mon.0) 1501 : audit [INF] from='client.35654 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:05:17.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:15.766658+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e408: 8 total, 8 up, 5 in 2024-02-12T17:05:17.051 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:17.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:17.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:15.747760+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.33:0/2066471646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:05:17.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:15.758125+0000 mon.a (mon.0) 1501 : audit [INF] from='client.35654 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:05:17.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:15.766658+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e408: 8 total, 8 up, 5 in 2024-02-12T17:05:17.255 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:17.313 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:05:17.313 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:05:17.321 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:17.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:17.427 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:17.484 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:17.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:17.660 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:17.709 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:18.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:16.665078+0000 mgr.y (mgr.24481) 594 : cluster [DBG] pgmap v799: 128 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 1.8 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 956 B/s rd, 0 B/s wr, 698 op/s; 23845/89310 objects misplaced (26.699%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T17:05:18.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:16.760221+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:18.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:16.780691+0000 mon.a (mon.0) 1504 : audit [INF] from='client.35654 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:05:18.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:16.780773+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e409: 8 total, 8 up, 5 in 2024-02-12T17:05:18.006 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:16.665078+0000 mgr.y (mgr.24481) 594 : cluster [DBG] pgmap v799: 128 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 1.8 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 956 B/s rd, 0 B/s wr, 698 op/s; 23845/89310 objects misplaced (26.699%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T17:05:18.006 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:16.760221+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:18.006 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:16.780691+0000 mon.a (mon.0) 1504 : audit [INF] from='client.35654 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:05:18.006 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:16.780773+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e409: 8 total, 8 up, 5 in 2024-02-12T17:05:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:16.665078+0000 mgr.y (mgr.24481) 594 : cluster [DBG] pgmap v799: 128 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 114 active+clean; 1.8 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 956 B/s rd, 0 B/s wr, 698 op/s; 23845/89310 objects misplaced (26.699%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T17:05:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:16.760221+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:16.780691+0000 mon.a (mon.0) 1504 : audit [INF] from='client.35654 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:05:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:16.780773+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e409: 8 total, 8 up, 5 in 2024-02-12T17:05:18.501 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T17:05:19.073 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:19.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:19.084 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:17.788656+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e410: 8 total, 8 up, 5 in 2024-02-12T17:05:19.084 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:18.499225+0000 mon.c (mon.2) 219 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:05:19.084 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:18.666169+0000 mon.c (mon.2) 220 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-12T17:05:19.084 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:18.668029+0000 mon.a (mon.0) 1507 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-12T17:05:19.084 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:17.788656+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e410: 8 total, 8 up, 5 in 2024-02-12T17:05:19.084 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:18.499225+0000 mon.c (mon.2) 219 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:05:19.084 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:18.666169+0000 mon.c (mon.2) 220 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-12T17:05:19.084 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:18.668029+0000 mon.a (mon.0) 1507 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-12T17:05:19.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:05:19.228 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:17.788656+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e410: 8 total, 8 up, 5 in 2024-02-12T17:05:19.228 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:18.499225+0000 mon.c (mon.2) 219 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:05:19.229 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:18.666169+0000 mon.c (mon.2) 220 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-12T17:05:19.229 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:18.668029+0000 mon.a (mon.0) 1507 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-12T17:05:19.231 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:19.269 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:05:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:19.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:05:19.373 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:19.434 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:18.665796+0000 mgr.y (mgr.24481) 595 : cluster [DBG] pgmap v802: 127 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 1.7 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 410 op/s; 22467/84375 objects misplaced (26.628%) 2024-02-12T17:05:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:18.823042+0000 mon.a (mon.0) 1508 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-12T17:05:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:18.823088+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e411: 8 total, 8 up, 5 in 2024-02-12T17:05:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:19.188259+0000 mon.a (mon.0) 1510 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:18.665796+0000 mgr.y (mgr.24481) 595 : cluster [DBG] pgmap v802: 127 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 1.7 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 410 op/s; 22467/84375 objects misplaced (26.628%) 2024-02-12T17:05:20.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:18.823042+0000 mon.a (mon.0) 1508 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-12T17:05:20.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:18.823088+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e411: 8 total, 8 up, 5 in 2024-02-12T17:05:20.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:19.188259+0000 mon.a (mon.0) 1510 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:20.228 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:20.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:18.665796+0000 mgr.y (mgr.24481) 595 : cluster [DBG] pgmap v802: 127 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 1.7 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 410 op/s; 22467/84375 objects misplaced (26.628%) 2024-02-12T17:05:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:18.823042+0000 mon.a (mon.0) 1508 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-12T17:05:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:18.823088+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e411: 8 total, 8 up, 5 in 2024-02-12T17:05:20.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:19.188259+0000 mon.a (mon.0) 1510 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:20.335 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:05:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:20.446 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:05:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:20.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:05:20.551 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:20.632 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:20.745 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:20.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:20.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:20.912 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:20.949 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:20.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:21.059 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:21.096 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:21.239 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:05:21.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:19.843197+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e412: 8 total, 8 up, 5 in 2024-02-12T17:05:21.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:19.913580+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:05:21.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:19.919002+0000 mon.c (mon.2) 222 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:05:21.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:19.935433+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:21.260 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:05:21.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:19.843197+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e412: 8 total, 8 up, 5 in 2024-02-12T17:05:21.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:19.913580+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:05:21.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:19.919002+0000 mon.c (mon.2) 222 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:05:21.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:19.935433+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:21.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:19.843197+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e412: 8 total, 8 up, 5 in 2024-02-12T17:05:21.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:19.913580+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:05:21.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:19.919002+0000 mon.c (mon.2) 222 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:05:21.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:19.935433+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:21.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:05:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:20] "GET /metrics HTTP/1.1" 200 176206 "" "Prometheus/2.33.4" 2024-02-12T17:05:21.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:05:22.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:20.666299+0000 mgr.y (mgr.24481) 596 : cluster [DBG] pgmap v805: 127 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 1.6 GiB data, 8.3 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 945 op/s; 20529/77976 objects misplaced (26.327%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:22.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:20.875734+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e413: 8 total, 8 up, 5 in 2024-02-12T17:05:22.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:21.325530+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e414: 8 total, 8 up, 5 in 2024-02-12T17:05:22.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:20.666299+0000 mgr.y (mgr.24481) 596 : cluster [DBG] pgmap v805: 127 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 1.6 GiB data, 8.3 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 945 op/s; 20529/77976 objects misplaced (26.327%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:22.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:20.875734+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e413: 8 total, 8 up, 5 in 2024-02-12T17:05:22.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:21.325530+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e414: 8 total, 8 up, 5 in 2024-02-12T17:05:22.171 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:22.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:20.666299+0000 mgr.y (mgr.24481) 596 : cluster [DBG] pgmap v805: 127 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 1.6 GiB data, 8.3 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 945 op/s; 20529/77976 objects misplaced (26.327%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:20.875734+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e413: 8 total, 8 up, 5 in 2024-02-12T17:05:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:21.325530+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e414: 8 total, 8 up, 5 in 2024-02-12T17:05:22.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:22.351 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:22.383 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:22.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:22.495 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:22.560 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:22.600 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:22.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:22.643 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":806,"stamp":"2024-02-12T17:05:20.873864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1690769084,"num_objects":25991,"num_object_clones":0,"num_object_copies":77973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20529,"num_objects_unfound":0,"num_objects_dirty":25991,"num_whiteouts":0,"num_read":1363,"num_read_kb":1115,"num_write":250090,"num_write_kb":8805610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14779,"num_bytes_recovered":923549110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76732,"ondisk_log_size":76732,"up":378,"acting":378,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":8652144,"kb_used_data":4765192,"kb_used_omap":0,"kb_used_meta":3886400,"kb_avail":460073616,"statfs":{"total":479975178240,"available":471115382784,"internally_reserved":0,"allocated":4879556608,"data_stored":4876656908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3979673600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-247595008,"num_objects":-3778,"num_object_clones":0,"num_object_copies":-11334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3316,"num_objects_unfound":0,"num_objects_dirty":-3778,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3778,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":14942208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.101298"},"pg_stats":[{"pgid":"2.1f","version":"0'0","reported_seq":99,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785914+0000","last_change":"2024-02-12T17:04:39.916313+0000","last_active":"2024-02-12T17:05:16.785914+0000","last_peered":"2024-02-12T17:05:16.785914+0000","last_clean":"2024-02-12T17:05:16.785914+0000","last_became_active":"2024-02-12T17:04:39.916010+0000","last_became_peered":"2024-02-12T17:04:39.916010+0000","last_unstale":"2024-02-12T17:05:16.785914+0000","last_undegraded":"2024-02-12T17:05:16.785914+0000","last_fullsized":"2024-02-12T17:05:16.785914+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.732293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.732293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":88,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786289+0000","last_change":"2024-02-12T17:03:52.280899+0000","last_active":"2024-02-12T17:05:16.786289+0000","last_peered":"2024-02-12T17:05:16.786289+0000","last_clean":"2024-02-12T17:05:16.786289+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:05:16.786289+0000","last_undegraded":"2024-02-12T17:05:16.786289+0000","last_fullsized":"2024-02-12T17:05:16.786289+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:01:26.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.047527+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":72,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782906+0000","last_change":"2024-02-12T17:03:50.545988+0000","last_active":"2024-02-12T17:05:16.782906+0000","last_peered":"2024-02-12T17:05:16.782906+0000","last_clean":"2024-02-12T17:05:16.782906+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:05:16.782906+0000","last_undegraded":"2024-02-12T17:05:16.782906+0000","last_fullsized":"2024-02-12T17:05:16.782906+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.176431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":463,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781122+0000","last_change":"2024-02-12T17:04:41.353781+0000","last_active":"2024-02-12T17:05:16.781122+0000","last_peered":"2024-02-12T17:05:16.781122+0000","last_clean":"2024-02-12T17:05:16.781122+0000","last_became_active":"2024-02-12T17:04:39.873280+0000","last_became_peered":"2024-02-12T17:04:39.873280+0000","last_unstale":"2024-02-12T17:05:16.781122+0000","last_undegraded":"2024-02-12T17:05:16.781122+0000","last_fullsized":"2024-02-12T17:05:16.781122+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:01:45.154164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":405,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784558+0000","last_change":"2024-02-12T17:03:50.565273+0000","last_active":"2024-02-12T17:05:16.784558+0000","last_peered":"2024-02-12T17:05:16.784558+0000","last_clean":"2024-02-12T17:05:16.784558+0000","last_became_active":"2024-02-12T17:03:50.562938+0000","last_became_peered":"2024-02-12T17:03:50.562938+0000","last_unstale":"2024-02-12T17:05:16.784558+0000","last_undegraded":"2024-02-12T17:05:16.784558+0000","last_fullsized":"2024-02-12T17:05:16.784558+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:37.813008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:01:37.813008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":93,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784007+0000","last_change":"2024-02-12T17:04:42.260018+0000","last_active":"2024-02-12T17:05:16.784007+0000","last_peered":"2024-02-12T17:05:16.784007+0000","last_clean":"2024-02-12T17:05:16.784007+0000","last_became_active":"2024-02-12T17:04:39.889725+0000","last_became_peered":"2024-02-12T17:04:39.889725+0000","last_unstale":"2024-02-12T17:05:16.784007+0000","last_undegraded":"2024-02-12T17:05:16.784007+0000","last_fullsized":"2024-02-12T17:05:16.784007+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":379,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785857+0000","last_change":"2024-02-12T17:02:34.502434+0000","last_active":"2024-02-12T17:05:16.785857+0000","last_peered":"2024-02-12T17:05:16.785857+0000","last_clean":"2024-02-12T17:05:16.785857+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:05:16.785857+0000","last_undegraded":"2024-02-12T17:05:16.785857+0000","last_fullsized":"2024-02-12T17:05:16.785857+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.502332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":101,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786585+0000","last_change":"2024-02-12T17:04:40.923654+0000","last_active":"2024-02-12T17:05:16.786585+0000","last_peered":"2024-02-12T17:05:16.786585+0000","last_clean":"2024-02-12T17:05:16.786585+0000","last_became_active":"2024-02-12T17:04:39.865216+0000","last_became_peered":"2024-02-12T17:04:39.865216+0000","last_unstale":"2024-02-12T17:05:16.786585+0000","last_undegraded":"2024-02-12T17:05:16.786585+0000","last_fullsized":"2024-02-12T17:05:16.786585+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":81,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783910+0000","last_change":"2024-02-12T17:04:39.890400+0000","last_active":"2024-02-12T17:05:16.783910+0000","last_peered":"2024-02-12T17:05:16.783910+0000","last_clean":"2024-02-12T17:05:16.783910+0000","last_became_active":"2024-02-12T17:04:39.887235+0000","last_became_peered":"2024-02-12T17:04:39.887235+0000","last_unstale":"2024-02-12T17:05:16.783910+0000","last_undegraded":"2024-02-12T17:05:16.783910+0000","last_fullsized":"2024-02-12T17:05:16.783910+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":105,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782628+0000","last_change":"2024-02-12T17:04:40.319567+0000","last_active":"2024-02-12T17:05:16.782628+0000","last_peered":"2024-02-12T17:05:16.782628+0000","last_clean":"2024-02-12T17:05:16.782628+0000","last_became_active":"2024-02-12T17:04:39.864612+0000","last_became_peered":"2024-02-12T17:04:39.864612+0000","last_unstale":"2024-02-12T17:05:16.782628+0000","last_undegraded":"2024-02-12T17:05:16.782628+0000","last_fullsized":"2024-02-12T17:05:16.782628+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":47,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786755+0000","last_change":"2024-02-12T17:04:39.888727+0000","last_active":"2024-02-12T17:05:16.786755+0000","last_peered":"2024-02-12T17:05:16.786755+0000","last_clean":"2024-02-12T17:05:16.786755+0000","last_became_active":"2024-02-12T17:04:39.888398+0000","last_became_peered":"2024-02-12T17:04:39.888398+0000","last_unstale":"2024-02-12T17:05:16.786755+0000","last_undegraded":"2024-02-12T17:05:16.786755+0000","last_fullsized":"2024-02-12T17:05:16.786755+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.157357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.157357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":113,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781953+0000","last_change":"2024-02-12T17:04:41.433996+0000","last_active":"2024-02-12T17:05:16.781953+0000","last_peered":"2024-02-12T17:05:16.781953+0000","last_clean":"2024-02-12T17:05:16.781953+0000","last_became_active":"2024-02-12T17:04:39.872998+0000","last_became_peered":"2024-02-12T17:04:39.872998+0000","last_unstale":"2024-02-12T17:05:16.781953+0000","last_undegraded":"2024-02-12T17:05:16.781953+0000","last_fullsized":"2024-02-12T17:05:16.781953+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":72,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.787514+0000","last_change":"2024-02-12T17:03:50.533143+0000","last_active":"2024-02-12T17:05:16.787514+0000","last_peered":"2024-02-12T17:05:16.787514+0000","last_clean":"2024-02-12T17:05:16.787514+0000","last_became_active":"2024-02-12T17:03:50.532233+0000","last_became_peered":"2024-02-12T17:03:50.532233+0000","last_unstale":"2024-02-12T17:05:16.787514+0000","last_undegraded":"2024-02-12T17:05:16.787514+0000","last_fullsized":"2024-02-12T17:05:16.787514+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":266,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.780143+0000","last_change":"2024-02-12T17:03:50.670847+0000","last_active":"2024-02-12T17:05:16.780143+0000","last_peered":"2024-02-12T17:05:16.780143+0000","last_clean":"2024-02-12T17:05:16.780143+0000","last_became_active":"2024-02-12T17:03:50.570118+0000","last_became_peered":"2024-02-12T17:03:50.570118+0000","last_unstale":"2024-02-12T17:05:16.780143+0000","last_undegraded":"2024-02-12T17:05:16.780143+0000","last_fullsized":"2024-02-12T17:05:16.780143+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:34.738385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.738385+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":385,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.779280+0000","last_change":"2024-02-12T17:04:39.880788+0000","last_active":"2024-02-12T17:05:16.779280+0000","last_peered":"2024-02-12T17:05:16.779280+0000","last_clean":"2024-02-12T17:05:16.779280+0000","last_became_active":"2024-02-12T17:04:39.880094+0000","last_became_peered":"2024-02-12T17:04:39.880094+0000","last_unstale":"2024-02-12T17:05:16.779280+0000","last_undegraded":"2024-02-12T17:05:16.779280+0000","last_fullsized":"2024-02-12T17:05:16.779280+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":424,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785639+0000","last_change":"2024-02-12T17:03:51.528465+0000","last_active":"2024-02-12T17:05:16.785639+0000","last_peered":"2024-02-12T17:05:16.785639+0000","last_clean":"2024-02-12T17:05:16.785639+0000","last_became_active":"2024-02-12T17:03:50.576714+0000","last_became_peered":"2024-02-12T17:03:50.576714+0000","last_unstale":"2024-02-12T17:05:16.785639+0000","last_undegraded":"2024-02-12T17:05:16.785639+0000","last_fullsized":"2024-02-12T17:05:16.785639+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:01:42.076584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.076584+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":81,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783989+0000","last_change":"2024-02-12T17:04:39.877511+0000","last_active":"2024-02-12T17:05:16.783989+0000","last_peered":"2024-02-12T17:05:16.783989+0000","last_clean":"2024-02-12T17:05:16.783989+0000","last_became_active":"2024-02-12T17:04:39.877005+0000","last_became_peered":"2024-02-12T17:04:39.877005+0000","last_unstale":"2024-02-12T17:05:16.783989+0000","last_undegraded":"2024-02-12T17:05:16.783989+0000","last_fullsized":"2024-02-12T17:05:16.783989+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":17,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781720+0000","last_change":"2024-02-12T17:05:09.845951+0000","last_active":"2024-02-12T17:05:16.781720+0000","last_peered":"2024-02-12T17:05:16.781720+0000","last_clean":"2024-02-12T17:05:16.781720+0000","last_became_active":"2024-02-12T17:05:09.845606+0000","last_became_peered":"2024-02-12T17:05:09.845606+0000","last_unstale":"2024-02-12T17:05:16.781720+0000","last_undegraded":"2024-02-12T17:05:16.781720+0000","last_fullsized":"2024-02-12T17:05:16.781720+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:51.269648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.269648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":182,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.779882+0000","last_change":"2024-02-12T17:04:39.885628+0000","last_active":"2024-02-12T17:05:16.779882+0000","last_peered":"2024-02-12T17:05:16.779882+0000","last_clean":"2024-02-12T17:05:16.779882+0000","last_became_active":"2024-02-12T17:04:39.884769+0000","last_became_peered":"2024-02-12T17:04:39.884769+0000","last_unstale":"2024-02-12T17:05:16.779882+0000","last_undegraded":"2024-02-12T17:05:16.779882+0000","last_fullsized":"2024-02-12T17:05:16.779882+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":84,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.788553+0000","last_change":"2024-02-12T17:03:59.035349+0000","last_active":"2024-02-12T17:05:16.788553+0000","last_peered":"2024-02-12T17:05:16.788553+0000","last_clean":"2024-02-12T17:05:16.788553+0000","last_became_active":"2024-02-12T17:03:50.639321+0000","last_became_peered":"2024-02-12T17:03:50.639321+0000","last_unstale":"2024-02-12T17:05:16.788553+0000","last_undegraded":"2024-02-12T17:05:16.788553+0000","last_fullsized":"2024-02-12T17:05:16.788553+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":377,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786900+0000","last_change":"2024-02-12T17:04:39.887129+0000","last_active":"2024-02-12T17:05:16.786900+0000","last_peered":"2024-02-12T17:05:16.786900+0000","last_clean":"2024-02-12T17:05:16.786900+0000","last_became_active":"2024-02-12T17:04:39.886041+0000","last_became_peered":"2024-02-12T17:04:39.886041+0000","last_unstale":"2024-02-12T17:05:16.786900+0000","last_undegraded":"2024-02-12T17:05:16.786900+0000","last_fullsized":"2024-02-12T17:05:16.786900+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":143,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.787392+0000","last_change":"2024-02-12T17:05:09.842144+0000","last_active":"2024-02-12T17:05:16.787392+0000","last_peered":"2024-02-12T17:05:16.787392+0000","last_clean":"2024-02-12T17:05:16.787392+0000","last_became_active":"2024-02-12T17:05:09.841868+0000","last_became_peered":"2024-02-12T17:05:09.841868+0000","last_unstale":"2024-02-12T17:05:16.787392+0000","last_undegraded":"2024-02-12T17:05:16.787392+0000","last_fullsized":"2024-02-12T17:05:16.787392+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.020845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:45.225996+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.020845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":403,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.777289+0000","last_change":"2024-02-12T17:03:50.551444+0000","last_active":"2024-02-12T17:05:16.777289+0000","last_peered":"2024-02-12T17:05:16.777289+0000","last_clean":"2024-02-12T17:05:16.777289+0000","last_became_active":"2024-02-12T17:03:50.550931+0000","last_became_peered":"2024-02-12T17:03:50.550931+0000","last_unstale":"2024-02-12T17:05:16.777289+0000","last_undegraded":"2024-02-12T17:05:16.777289+0000","last_fullsized":"2024-02-12T17:05:16.777289+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":93,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781298+0000","last_change":"2024-02-12T17:04:40.315768+0000","last_active":"2024-02-12T17:05:16.781298+0000","last_peered":"2024-02-12T17:05:16.781298+0000","last_clean":"2024-02-12T17:05:16.781298+0000","last_became_active":"2024-02-12T17:04:39.865088+0000","last_became_peered":"2024-02-12T17:04:39.865088+0000","last_unstale":"2024-02-12T17:05:16.781298+0000","last_undegraded":"2024-02-12T17:05:16.781298+0000","last_fullsized":"2024-02-12T17:05:16.781298+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.938805+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":72,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786438+0000","last_change":"2024-02-12T17:03:50.545703+0000","last_active":"2024-02-12T17:05:16.786438+0000","last_peered":"2024-02-12T17:05:16.786438+0000","last_clean":"2024-02-12T17:05:16.786438+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:05:16.786438+0000","last_undegraded":"2024-02-12T17:05:16.786438+0000","last_fullsized":"2024-02-12T17:05:16.786438+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.051528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.051528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":17,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.791303+0000","last_change":"2024-02-12T17:05:09.851083+0000","last_active":"2024-02-12T17:05:16.791303+0000","last_peered":"2024-02-12T17:05:16.791303+0000","last_clean":"2024-02-12T17:05:16.791303+0000","last_became_active":"2024-02-12T17:05:09.850722+0000","last_became_peered":"2024-02-12T17:05:09.850722+0000","last_unstale":"2024-02-12T17:05:16.791303+0000","last_undegraded":"2024-02-12T17:05:16.791303+0000","last_fullsized":"2024-02-12T17:05:16.791303+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:47.004467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:56:47.004467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":237,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.776903+0000","last_change":"2024-02-12T17:03:20.768326+0000","last_active":"2024-02-12T17:05:16.776903+0000","last_peered":"2024-02-12T17:05:16.776903+0000","last_clean":"2024-02-12T17:05:16.776903+0000","last_became_active":"2024-02-12T17:03:20.767943+0000","last_became_peered":"2024-02-12T17:03:20.767943+0000","last_unstale":"2024-02-12T17:05:16.776903+0000","last_undegraded":"2024-02-12T17:05:16.776903+0000","last_fullsized":"2024-02-12T17:05:16.776903+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.858411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.858411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":377,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.773733+0000","last_change":"2024-02-12T17:04:40.575962+0000","last_active":"2024-02-12T17:05:16.773733+0000","last_peered":"2024-02-12T17:05:16.773733+0000","last_clean":"2024-02-12T17:05:16.773733+0000","last_became_active":"2024-02-12T17:04:39.883548+0000","last_became_peered":"2024-02-12T17:04:39.883548+0000","last_unstale":"2024-02-12T17:05:16.773733+0000","last_undegraded":"2024-02-12T17:05:16.773733+0000","last_fullsized":"2024-02-12T17:05:16.773733+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:16.064966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.064966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":319,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.779428+0000","last_change":"2024-02-12T17:04:39.877393+0000","last_active":"2024-02-12T17:05:16.779428+0000","last_peered":"2024-02-12T17:05:16.779428+0000","last_clean":"2024-02-12T17:05:16.779428+0000","last_became_active":"2024-02-12T17:04:39.876910+0000","last_became_peered":"2024-02-12T17:04:39.876910+0000","last_unstale":"2024-02-12T17:05:16.779428+0000","last_undegraded":"2024-02-12T17:05:16.779428+0000","last_fullsized":"2024-02-12T17:05:16.779428+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.019813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.019813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":138,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.773397+0000","last_change":"2024-02-12T17:05:09.851755+0000","last_active":"2024-02-12T17:05:16.773397+0000","last_peered":"2024-02-12T17:05:16.773397+0000","last_clean":"2024-02-12T17:05:16.773397+0000","last_became_active":"2024-02-12T17:05:09.851412+0000","last_became_peered":"2024-02-12T17:05:09.851412+0000","last_unstale":"2024-02-12T17:05:16.773397+0000","last_undegraded":"2024-02-12T17:05:16.773397+0000","last_fullsized":"2024-02-12T17:05:16.773397+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:33.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:01:33.149132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":72,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785894+0000","last_change":"2024-02-12T17:03:50.559509+0000","last_active":"2024-02-12T17:05:16.785894+0000","last_peered":"2024-02-12T17:05:16.785894+0000","last_clean":"2024-02-12T17:05:16.785894+0000","last_became_active":"2024-02-12T17:03:50.558598+0000","last_became_peered":"2024-02-12T17:03:50.558598+0000","last_unstale":"2024-02-12T17:05:16.785894+0000","last_undegraded":"2024-02-12T17:05:16.785894+0000","last_fullsized":"2024-02-12T17:05:16.785894+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":186,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.776155+0000","last_change":"2024-02-12T17:03:22.527733+0000","last_active":"2024-02-12T17:05:16.776155+0000","last_peered":"2024-02-12T17:05:16.776155+0000","last_clean":"2024-02-12T17:05:16.776155+0000","last_became_active":"2024-02-12T17:03:20.774315+0000","last_became_peered":"2024-02-12T17:03:20.774315+0000","last_unstale":"2024-02-12T17:05:16.776155+0000","last_undegraded":"2024-02-12T17:05:16.776155+0000","last_fullsized":"2024-02-12T17:05:16.776155+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:00:53.792980+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.792980+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":160,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783820+0000","last_change":"2024-02-12T17:04:39.890091+0000","last_active":"2024-02-12T17:05:16.783820+0000","last_peered":"2024-02-12T17:05:16.783820+0000","last_clean":"2024-02-12T17:05:16.783820+0000","last_became_active":"2024-02-12T17:04:39.886343+0000","last_became_peered":"2024-02-12T17:04:39.886343+0000","last_unstale":"2024-02-12T17:05:16.783820+0000","last_undegraded":"2024-02-12T17:05:16.783820+0000","last_fullsized":"2024-02-12T17:05:16.783820+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.953891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.953891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":475,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.776069+0000","last_change":"2024-02-12T17:03:50.690222+0000","last_active":"2024-02-12T17:05:16.776069+0000","last_peered":"2024-02-12T17:05:16.776069+0000","last_clean":"2024-02-12T17:05:16.776069+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:05:16.776069+0000","last_undegraded":"2024-02-12T17:05:16.776069+0000","last_fullsized":"2024-02-12T17:05:16.776069+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:31.827632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:31.827632+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":394,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783365+0000","last_change":"2024-02-12T17:03:50.577996+0000","last_active":"2024-02-12T17:05:16.783365+0000","last_peered":"2024-02-12T17:05:16.783365+0000","last_clean":"2024-02-12T17:05:16.783365+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:05:16.783365+0000","last_undegraded":"2024-02-12T17:05:16.783365+0000","last_fullsized":"2024-02-12T17:05:16.783365+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:34.874363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:34.874363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":84,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785804+0000","last_change":"2024-02-12T17:03:55.244602+0000","last_active":"2024-02-12T17:05:16.785804+0000","last_peered":"2024-02-12T17:05:16.785804+0000","last_clean":"2024-02-12T17:05:16.785804+0000","last_became_active":"2024-02-12T17:03:50.556444+0000","last_became_peered":"2024-02-12T17:03:50.556444+0000","last_unstale":"2024-02-12T17:05:16.785804+0000","last_undegraded":"2024-02-12T17:05:16.785804+0000","last_fullsized":"2024-02-12T17:05:16.785804+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":390,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784511+0000","last_change":"2024-02-12T17:04:39.889382+0000","last_active":"2024-02-12T17:05:16.784511+0000","last_peered":"2024-02-12T17:05:16.784511+0000","last_clean":"2024-02-12T17:05:16.784511+0000","last_became_active":"2024-02-12T17:04:39.889055+0000","last_became_peered":"2024-02-12T17:04:39.889055+0000","last_unstale":"2024-02-12T17:05:16.784511+0000","last_undegraded":"2024-02-12T17:05:16.784511+0000","last_fullsized":"2024-02-12T17:05:16.784511+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.187738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":333,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.776170+0000","last_change":"2024-02-12T17:04:39.884622+0000","last_active":"2024-02-12T17:05:16.776170+0000","last_peered":"2024-02-12T17:05:16.776170+0000","last_clean":"2024-02-12T17:05:16.776170+0000","last_became_active":"2024-02-12T17:04:39.884162+0000","last_became_peered":"2024-02-12T17:04:39.884162+0000","last_unstale":"2024-02-12T17:05:16.776170+0000","last_undegraded":"2024-02-12T17:05:16.776170+0000","last_fullsized":"2024-02-12T17:05:16.776170+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:32.127593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:32.127593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":80,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783644+0000","last_change":"2024-02-12T17:03:53.944994+0000","last_active":"2024-02-12T17:05:16.783644+0000","last_peered":"2024-02-12T17:05:16.783644+0000","last_clean":"2024-02-12T17:05:16.783644+0000","last_became_active":"2024-02-12T17:03:50.555238+0000","last_became_peered":"2024-02-12T17:03:50.555238+0000","last_unstale":"2024-02-12T17:05:16.783644+0000","last_undegraded":"2024-02-12T17:05:16.783644+0000","last_fullsized":"2024-02-12T17:05:16.783644+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-02-12T17:00:53.269107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.269107+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":47,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785985+0000","last_change":"2024-02-12T17:04:39.936874+0000","last_active":"2024-02-12T17:05:16.785985+0000","last_peered":"2024-02-12T17:05:16.785985+0000","last_clean":"2024-02-12T17:05:16.785985+0000","last_became_active":"2024-02-12T17:04:39.936538+0000","last_became_peered":"2024-02-12T17:04:39.936538+0000","last_unstale":"2024-02-12T17:05:16.785985+0000","last_undegraded":"2024-02-12T17:05:16.785985+0000","last_fullsized":"2024-02-12T17:05:16.785985+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:43.982623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.982623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":297,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.793413+0000","last_change":"2024-02-12T17:04:39.867059+0000","last_active":"2024-02-12T17:05:16.793413+0000","last_peered":"2024-02-12T17:05:16.793413+0000","last_clean":"2024-02-12T17:05:16.793413+0000","last_became_active":"2024-02-12T17:04:39.866732+0000","last_became_peered":"2024-02-12T17:04:39.866732+0000","last_unstale":"2024-02-12T17:05:16.793413+0000","last_undegraded":"2024-02-12T17:05:16.793413+0000","last_fullsized":"2024-02-12T17:05:16.793413+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.824194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.824194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":82,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782943+0000","last_change":"2024-02-12T17:03:55.514799+0000","last_active":"2024-02-12T17:05:16.782943+0000","last_peered":"2024-02-12T17:05:16.782943+0000","last_clean":"2024-02-12T17:05:16.782943+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:05:16.782943+0000","last_undegraded":"2024-02-12T17:05:16.782943+0000","last_fullsized":"2024-02-12T17:05:16.782943+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:00:53.936664+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.936664+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"410'15491","reported_seq":15823,"reported_epoch":410,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:05:18.668491+0000","last_change":"2024-02-12T17:05:10.179097+0000","last_active":"2024-02-12T17:05:18.668491+0000","last_peered":"2024-02-12T17:05:18.668491+0000","last_clean":"2024-02-12T17:03:49.505679+0000","last_became_active":"2024-02-12T17:04:40.888742+0000","last_became_peered":"2024-02-12T17:04:40.888742+0000","last_unstale":"2024-02-12T17:05:18.668491+0000","last_undegraded":"2024-02-12T17:05:18.668491+0000","last_fullsized":"2024-02-12T17:05:18.668491+0000","mapping_epoch":378,"log_start":"331'6700","ondisk_log_start":"331'6700","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":8791,"ondisk_log_size":8791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103612416,"num_objects":1581,"num_object_clones":0,"num_object_copies":4743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1497,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15491,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":6422528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":1581}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":99,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783933+0000","last_change":"2024-02-12T17:04:39.890573+0000","last_active":"2024-02-12T17:05:16.783933+0000","last_peered":"2024-02-12T17:05:16.783933+0000","last_clean":"2024-02-12T17:05:16.783933+0000","last_became_active":"2024-02-12T17:04:39.889283+0000","last_became_peered":"2024-02-12T17:04:39.889283+0000","last_unstale":"2024-02-12T17:05:16.783933+0000","last_undegraded":"2024-02-12T17:05:16.783933+0000","last_fullsized":"2024-02-12T17:05:16.783933+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:28.912872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.912872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":109,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785964+0000","last_change":"2024-02-12T17:04:40.578504+0000","last_active":"2024-02-12T17:05:16.785964+0000","last_peered":"2024-02-12T17:05:16.785964+0000","last_clean":"2024-02-12T17:05:16.785964+0000","last_became_active":"2024-02-12T17:04:39.890440+0000","last_became_peered":"2024-02-12T17:04:39.890440+0000","last_unstale":"2024-02-12T17:05:16.785964+0000","last_undegraded":"2024-02-12T17:05:16.785964+0000","last_fullsized":"2024-02-12T17:05:16.785964+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:02:17.066331+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.066331+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":105,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.787593+0000","last_change":"2024-02-12T17:04:42.701462+0000","last_active":"2024-02-12T17:05:16.787593+0000","last_peered":"2024-02-12T17:05:16.787593+0000","last_clean":"2024-02-12T17:05:16.787593+0000","last_became_active":"2024-02-12T17:04:39.891351+0000","last_became_peered":"2024-02-12T17:04:39.891351+0000","last_unstale":"2024-02-12T17:05:16.787593+0000","last_undegraded":"2024-02-12T17:05:16.787593+0000","last_fullsized":"2024-02-12T17:05:16.787593+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":304,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.779069+0000","last_change":"2024-02-12T17:04:39.882820+0000","last_active":"2024-02-12T17:05:16.779069+0000","last_peered":"2024-02-12T17:05:16.779069+0000","last_clean":"2024-02-12T17:05:16.779069+0000","last_became_active":"2024-02-12T17:04:39.880493+0000","last_became_peered":"2024-02-12T17:04:39.880493+0000","last_unstale":"2024-02-12T17:05:16.779069+0000","last_undegraded":"2024-02-12T17:05:16.779069+0000","last_fullsized":"2024-02-12T17:05:16.779069+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:25.815816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.815816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"410'15853","reported_seq":25558,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-12T17:05:18.273254+0000","last_change":"2024-02-12T17:04:30.711484+0000","last_active":"2024-02-12T17:05:18.273254+0000","last_peered":"2024-02-12T17:05:18.273254+0000","last_clean":"2024-02-12T17:05:18.273254+0000","last_became_active":"2024-02-12T17:04:30.678545+0000","last_became_peered":"2024-02-12T17:04:30.678545+0000","last_unstale":"2024-02-12T17:05:18.273254+0000","last_undegraded":"2024-02-12T17:05:18.273254+0000","last_fullsized":"2024-02-12T17:05:18.273254+0000","mapping_epoch":368,"log_start":"354'12771","ondisk_log_start":"354'12771","created":304,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3082,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":107806720,"num_objects":1645,"num_object_clones":0,"num_object_copies":4935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15853,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4785,"num_bytes_recovered":313589760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":386,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.793081+0000","last_change":"2024-02-12T17:03:50.533429+0000","last_active":"2024-02-12T17:05:16.793081+0000","last_peered":"2024-02-12T17:05:16.793081+0000","last_clean":"2024-02-12T17:05:16.793081+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:05:16.793081+0000","last_undegraded":"2024-02-12T17:05:16.793081+0000","last_fullsized":"2024-02-12T17:05:16.793081+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":219,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.797054+0000","last_change":"2024-02-12T17:02:30.758581+0000","last_active":"2024-02-12T17:05:16.797054+0000","last_peered":"2024-02-12T17:05:16.797054+0000","last_clean":"2024-02-12T17:05:16.797054+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:05:16.797054+0000","last_undegraded":"2024-02-12T17:05:16.797054+0000","last_fullsized":"2024-02-12T17:05:16.797054+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.758520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.758520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":310,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783578+0000","last_change":"2024-02-12T17:04:41.560789+0000","last_active":"2024-02-12T17:05:16.783578+0000","last_peered":"2024-02-12T17:05:16.783578+0000","last_clean":"2024-02-12T17:05:16.783578+0000","last_became_active":"2024-02-12T17:04:39.883521+0000","last_became_peered":"2024-02-12T17:04:39.883521+0000","last_unstale":"2024-02-12T17:05:16.783578+0000","last_undegraded":"2024-02-12T17:05:16.783578+0000","last_fullsized":"2024-02-12T17:05:16.783578+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"348'12","reported_seq":533,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:17.443120+0000","last_change":"2024-02-12T17:03:51.176776+0000","last_active":"2024-02-12T17:05:17.443120+0000","last_peered":"2024-02-12T17:05:17.443120+0000","last_clean":"2024-02-12T17:05:17.443120+0000","last_became_active":"2024-02-12T17:03:50.577424+0000","last_became_peered":"2024-02-12T17:03:50.577424+0000","last_unstale":"2024-02-12T17:05:17.443120+0000","last_undegraded":"2024-02-12T17:05:17.443120+0000","last_fullsized":"2024-02-12T17:05:17.443120+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"410'15647","reported_seq":16820,"reported_epoch":410,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:05:18.537318+0000","last_change":"2024-02-12T17:05:09.869381+0000","last_active":"2024-02-12T17:05:18.537318+0000","last_peered":"2024-02-12T17:05:18.537318+0000","last_clean":"2024-02-12T17:03:19.725881+0000","last_became_active":"2024-02-12T17:03:50.651275+0000","last_became_peered":"2024-02-12T17:03:50.651275+0000","last_unstale":"2024-02-12T17:05:18.537318+0000","last_undegraded":"2024-02-12T17:05:18.537318+0000","last_fullsized":"2024-02-12T17:05:18.537318+0000","mapping_epoch":347,"log_start":"354'12490","ondisk_log_start":"354'12490","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3157,"ondisk_log_size":3157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97714176,"num_objects":1491,"num_object_clones":0,"num_object_copies":4473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1356,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15647,"num_write_kb":548416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":33619968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,0],"avail_no_missing":["7","0","4","5"],"object_location_counts":[{"shards":"0,5,7","objects":1491}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":99,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782752+0000","last_change":"2024-02-12T17:04:58.474896+0000","last_active":"2024-02-12T17:05:16.782752+0000","last_peered":"2024-02-12T17:05:16.782752+0000","last_clean":"2024-02-12T17:05:16.782752+0000","last_became_active":"2024-02-12T17:04:58.474504+0000","last_became_peered":"2024-02-12T17:04:58.474504+0000","last_unstale":"2024-02-12T17:05:16.782752+0000","last_undegraded":"2024-02-12T17:05:16.782752+0000","last_fullsized":"2024-02-12T17:05:16.782752+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":408,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783395+0000","last_change":"2024-02-12T17:03:50.549240+0000","last_active":"2024-02-12T17:05:16.783395+0000","last_peered":"2024-02-12T17:05:16.783395+0000","last_clean":"2024-02-12T17:05:16.783395+0000","last_became_active":"2024-02-12T17:03:50.548926+0000","last_became_peered":"2024-02-12T17:03:50.548926+0000","last_unstale":"2024-02-12T17:05:16.783395+0000","last_undegraded":"2024-02-12T17:05:16.783395+0000","last_fullsized":"2024-02-12T17:05:16.783395+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:42.892563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.892563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"348'55","reported_seq":113,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:17.440013+0000","last_change":"2024-02-12T17:03:58.596193+0000","last_active":"2024-02-12T17:05:17.440013+0000","last_peered":"2024-02-12T17:05:17.440013+0000","last_clean":"2024-02-12T17:05:17.440013+0000","last_became_active":"2024-02-12T17:03:50.557154+0000","last_became_peered":"2024-02-12T17:03:50.557154+0000","last_unstale":"2024-02-12T17:05:17.440013+0000","last_undegraded":"2024-02-12T17:05:17.440013+0000","last_fullsized":"2024-02-12T17:05:17.440013+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":346,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-12T17:05:17.831222+0000","last_change":"2024-02-12T17:05:17.817363+0000","last_active":"2024-02-12T17:05:17.831222+0000","last_peered":"2024-02-12T17:05:17.831222+0000","last_clean":"2024-02-12T17:05:17.831222+0000","last_became_active":"2024-02-12T17:05:17.816743+0000","last_became_peered":"2024-02-12T17:05:17.816743+0000","last_unstale":"2024-02-12T17:05:17.831222+0000","last_undegraded":"2024-02-12T17:05:17.831222+0000","last_fullsized":"2024-02-12T17:05:17.831222+0000","mapping_epoch":377,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"410'15811","reported_seq":17042,"reported_epoch":410,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:18.283367+0000","last_change":"2024-02-12T17:04:39.884779+0000","last_active":"2024-02-12T17:05:18.283367+0000","last_peered":"2024-02-12T17:05:18.283367+0000","last_clean":"2024-02-12T17:03:49.453764+0000","last_became_active":"2024-02-12T17:03:51.655919+0000","last_became_peered":"2024-02-12T17:03:51.655919+0000","last_unstale":"2024-02-12T17:05:18.283367+0000","last_undegraded":"2024-02-12T17:05:18.283367+0000","last_fullsized":"2024-02-12T17:05:18.283367+0000","mapping_epoch":348,"log_start":"354'12697","ondisk_log_start":"354'12697","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3114,"ondisk_log_size":3114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1374,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15811,"num_write_kb":556224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":36175872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":1571}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":397,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784653+0000","last_change":"2024-02-12T17:03:50.566072+0000","last_active":"2024-02-12T17:05:16.784653+0000","last_peered":"2024-02-12T17:05:16.784653+0000","last_clean":"2024-02-12T17:05:16.784653+0000","last_became_active":"2024-02-12T17:03:50.564700+0000","last_became_peered":"2024-02-12T17:03:50.564700+0000","last_unstale":"2024-02-12T17:05:16.784653+0000","last_undegraded":"2024-02-12T17:05:16.784653+0000","last_fullsized":"2024-02-12T17:05:16.784653+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:04.085827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:04.085827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":381,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-12T17:05:17.801187+0000","last_change":"2024-02-12T17:01:16.913133+0000","last_active":"2024-02-12T17:05:17.801187+0000","last_peered":"2024-02-12T17:05:17.801187+0000","last_clean":"2024-02-12T17:05:17.801187+0000","last_became_active":"2024-02-12T17:01:16.912041+0000","last_became_peered":"2024-02-12T17:01:16.912041+0000","last_unstale":"2024-02-12T17:05:17.801187+0000","last_undegraded":"2024-02-12T17:05:17.801187+0000","last_fullsized":"2024-02-12T17:05:17.801187+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:00:52.838852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:52.838852+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":247,"num_read_kb":242,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.787458+0000","last_change":"2024-02-12T17:03:50.533363+0000","last_active":"2024-02-12T17:05:16.787458+0000","last_peered":"2024-02-12T17:05:16.787458+0000","last_clean":"2024-02-12T17:05:16.787458+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:05:16.787458+0000","last_undegraded":"2024-02-12T17:05:16.787458+0000","last_fullsized":"2024-02-12T17:05:16.787458+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:38.142602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:38.142602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":342,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.779967+0000","last_change":"2024-02-12T17:03:51.175255+0000","last_active":"2024-02-12T17:05:16.779967+0000","last_peered":"2024-02-12T17:05:16.779967+0000","last_clean":"2024-02-12T17:05:16.779967+0000","last_became_active":"2024-02-12T17:03:50.573814+0000","last_became_peered":"2024-02-12T17:03:50.573814+0000","last_unstale":"2024-02-12T17:05:16.779967+0000","last_undegraded":"2024-02-12T17:05:16.779967+0000","last_fullsized":"2024-02-12T17:05:16.779967+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":324,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.777362+0000","last_change":"2024-02-12T17:01:16.909653+0000","last_active":"2024-02-12T17:05:16.777362+0000","last_peered":"2024-02-12T17:05:16.777362+0000","last_clean":"2024-02-12T17:05:16.777362+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:05:16.777362+0000","last_undegraded":"2024-02-12T17:05:16.777362+0000","last_fullsized":"2024-02-12T17:05:16.777362+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"410'15584","reported_seq":16616,"reported_epoch":410,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:05:18.683283+0000","last_change":"2024-02-12T17:04:40.746118+0000","last_active":"2024-02-12T17:05:18.683283+0000","last_peered":"2024-02-12T17:05:18.683283+0000","last_clean":"2024-02-12T17:03:49.506422+0000","last_became_active":"2024-02-12T17:03:51.676105+0000","last_became_peered":"2024-02-12T17:03:51.676105+0000","last_unstale":"2024-02-12T17:05:18.683283+0000","last_undegraded":"2024-02-12T17:05:18.683283+0000","last_fullsized":"2024-02-12T17:05:18.683283+0000","mapping_epoch":348,"log_start":"331'6800","ondisk_log_start":"331'6800","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":8784,"ondisk_log_size":8784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104595456,"num_objects":1596,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1298,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15584,"num_write_kb":549760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":29687808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":1596}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":424,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.780316+0000","last_change":"2024-02-12T17:04:54.415608+0000","last_active":"2024-02-12T17:05:16.780316+0000","last_peered":"2024-02-12T17:05:16.780316+0000","last_clean":"2024-02-12T17:05:16.780316+0000","last_became_active":"2024-02-12T17:04:54.415431+0000","last_became_peered":"2024-02-12T17:04:54.415431+0000","last_unstale":"2024-02-12T17:05:16.780316+0000","last_undegraded":"2024-02-12T17:05:16.780316+0000","last_fullsized":"2024-02-12T17:05:16.780316+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:57.950284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:00:57.950284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":81,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786296+0000","last_change":"2024-02-12T17:04:39.885990+0000","last_active":"2024-02-12T17:05:16.786296+0000","last_peered":"2024-02-12T17:05:16.786296+0000","last_clean":"2024-02-12T17:05:16.786296+0000","last_became_active":"2024-02-12T17:04:39.885173+0000","last_became_peered":"2024-02-12T17:04:39.885173+0000","last_unstale":"2024-02-12T17:05:16.786296+0000","last_undegraded":"2024-02-12T17:05:16.786296+0000","last_fullsized":"2024-02-12T17:05:16.786296+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":376,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.773516+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:05:16.773516+0000","last_peered":"2024-02-12T17:05:16.773516+0000","last_clean":"2024-02-12T17:05:16.773516+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:05:16.773516+0000","last_undegraded":"2024-02-12T17:05:16.773516+0000","last_fullsized":"2024-02-12T17:05:16.773516+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":409,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782198+0000","last_change":"2024-02-12T17:04:39.915666+0000","last_active":"2024-02-12T17:05:16.782198+0000","last_peered":"2024-02-12T17:05:16.782198+0000","last_clean":"2024-02-12T17:05:16.782198+0000","last_became_active":"2024-02-12T17:04:39.915392+0000","last_became_peered":"2024-02-12T17:04:39.915392+0000","last_unstale":"2024-02-12T17:05:16.782198+0000","last_undegraded":"2024-02-12T17:05:16.782198+0000","last_fullsized":"2024-02-12T17:05:16.782198+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:25.870916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:01:25.870916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":158,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784753+0000","last_change":"2024-02-12T17:04:40.745958+0000","last_active":"2024-02-12T17:05:16.784753+0000","last_peered":"2024-02-12T17:05:16.784753+0000","last_clean":"2024-02-12T17:05:16.784753+0000","last_became_active":"2024-02-12T17:04:39.889039+0000","last_became_peered":"2024-02-12T17:04:39.889039+0000","last_unstale":"2024-02-12T17:05:16.784753+0000","last_undegraded":"2024-02-12T17:05:16.784753+0000","last_fullsized":"2024-02-12T17:05:16.784753+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'26","last_scrub_stamp":"2024-02-12T17:01:43.163724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:43.163724+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":375,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786068+0000","last_change":"2024-02-12T17:03:15.796227+0000","last_active":"2024-02-12T17:05:16.786068+0000","last_peered":"2024-02-12T17:05:16.786068+0000","last_clean":"2024-02-12T17:05:16.786068+0000","last_became_active":"2024-02-12T16:49:43.201008+0000","last_became_peered":"2024-02-12T16:49:43.201008+0000","last_unstale":"2024-02-12T17:05:16.786068+0000","last_undegraded":"2024-02-12T17:05:16.786068+0000","last_fullsized":"2024-02-12T17:05:16.786068+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"410'15581","reported_seq":16324,"reported_epoch":410,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:18.243894+0000","last_change":"2024-02-12T17:04:40.576566+0000","last_active":"2024-02-12T17:05:18.243894+0000","last_peered":"2024-02-12T17:05:18.243894+0000","last_clean":"2024-02-12T17:03:19.701993+0000","last_became_active":"2024-02-12T17:03:50.639685+0000","last_became_peered":"2024-02-12T17:03:50.639685+0000","last_unstale":"2024-02-12T17:05:18.243894+0000","last_undegraded":"2024-02-12T17:05:18.243894+0000","last_fullsized":"2024-02-12T17:05:18.243894+0000","mapping_epoch":347,"log_start":"354'12549","ondisk_log_start":"354'12549","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3032,"ondisk_log_size":3032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":4881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1627,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15581,"num_write_kb":550656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":19791872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":1627}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":334,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.773365+0000","last_change":"2024-02-12T17:04:39.871302+0000","last_active":"2024-02-12T17:05:16.773365+0000","last_peered":"2024-02-12T17:05:16.773365+0000","last_clean":"2024-02-12T17:05:16.773365+0000","last_became_active":"2024-02-12T17:04:39.870979+0000","last_became_peered":"2024-02-12T17:04:39.870979+0000","last_unstale":"2024-02-12T17:05:16.773365+0000","last_undegraded":"2024-02-12T17:05:16.773365+0000","last_fullsized":"2024-02-12T17:05:16.773365+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":160,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785991+0000","last_change":"2024-02-12T17:04:39.892450+0000","last_active":"2024-02-12T17:05:16.785991+0000","last_peered":"2024-02-12T17:05:16.785991+0000","last_clean":"2024-02-12T17:05:16.785991+0000","last_became_active":"2024-02-12T17:04:39.892122+0000","last_became_peered":"2024-02-12T17:04:39.892122+0000","last_unstale":"2024-02-12T17:05:16.785991+0000","last_undegraded":"2024-02-12T17:05:16.785991+0000","last_fullsized":"2024-02-12T17:05:16.785991+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.941776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":406,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782034+0000","last_change":"2024-02-12T17:03:50.776630+0000","last_active":"2024-02-12T17:05:16.782034+0000","last_peered":"2024-02-12T17:05:16.782034+0000","last_clean":"2024-02-12T17:05:16.782034+0000","last_became_active":"2024-02-12T17:03:50.564881+0000","last_became_peered":"2024-02-12T17:03:50.564881+0000","last_unstale":"2024-02-12T17:05:16.782034+0000","last_undegraded":"2024-02-12T17:05:16.782034+0000","last_fullsized":"2024-02-12T17:05:16.782034+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":392,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784893+0000","last_change":"2024-02-12T17:03:24.771579+0000","last_active":"2024-02-12T17:05:16.784893+0000","last_peered":"2024-02-12T17:05:16.784893+0000","last_clean":"2024-02-12T17:05:16.784893+0000","last_became_active":"2024-02-12T17:03:20.780926+0000","last_became_peered":"2024-02-12T17:03:20.780926+0000","last_unstale":"2024-02-12T17:05:16.784893+0000","last_undegraded":"2024-02-12T17:05:16.784893+0000","last_fullsized":"2024-02-12T17:05:16.784893+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":408,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.780234+0000","last_change":"2024-02-12T17:04:39.885796+0000","last_active":"2024-02-12T17:05:16.780234+0000","last_peered":"2024-02-12T17:05:16.780234+0000","last_clean":"2024-02-12T17:05:16.780234+0000","last_became_active":"2024-02-12T17:04:39.885162+0000","last_became_peered":"2024-02-12T17:04:39.885162+0000","last_unstale":"2024-02-12T17:05:16.780234+0000","last_undegraded":"2024-02-12T17:05:16.780234+0000","last_fullsized":"2024-02-12T17:05:16.780234+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"410'15809","reported_seq":17637,"reported_epoch":410,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:05:18.643019+0000","last_change":"2024-02-12T17:05:10.407523+0000","last_active":"2024-02-12T17:05:18.643019+0000","last_peered":"2024-02-12T17:05:18.643019+0000","last_clean":"2024-02-12T17:03:19.726721+0000","last_became_active":"2024-02-12T17:04:39.940046+0000","last_became_peered":"2024-02-12T17:04:39.940046+0000","last_unstale":"2024-02-12T17:05:18.643019+0000","last_undegraded":"2024-02-12T17:05:18.643019+0000","last_fullsized":"2024-02-12T17:05:18.643019+0000","mapping_epoch":377,"log_start":"353'12584","ondisk_log_start":"353'12584","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3225,"ondisk_log_size":3225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109772800,"num_objects":1675,"num_object_clones":0,"num_object_copies":5025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3078,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15809,"num_write_kb":559488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":55312384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,4,0],"avail_no_missing":["7","0","4"],"object_location_counts":[{"shards":"0,4,7","objects":1675}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":136,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782195+0000","last_change":"2024-02-12T17:03:20.779625+0000","last_active":"2024-02-12T17:05:16.782195+0000","last_peered":"2024-02-12T17:05:16.782195+0000","last_clean":"2024-02-12T17:05:16.782195+0000","last_became_active":"2024-02-12T17:03:20.779126+0000","last_became_peered":"2024-02-12T17:03:20.779126+0000","last_unstale":"2024-02-12T17:05:16.782195+0000","last_undegraded":"2024-02-12T17:05:16.782195+0000","last_fullsized":"2024-02-12T17:05:16.782195+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:48.617913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:02:48.617913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":397,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784707+0000","last_change":"2024-02-12T17:04:39.886695+0000","last_active":"2024-02-12T17:05:16.784707+0000","last_peered":"2024-02-12T17:05:16.784707+0000","last_clean":"2024-02-12T17:05:16.784707+0000","last_became_active":"2024-02-12T17:04:39.886102+0000","last_became_peered":"2024-02-12T17:04:39.886102+0000","last_unstale":"2024-02-12T17:05:16.784707+0000","last_undegraded":"2024-02-12T17:05:16.784707+0000","last_fullsized":"2024-02-12T17:05:16.784707+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.212949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.212949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":388,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.773592+0000","last_change":"2024-02-12T17:03:21.658125+0000","last_active":"2024-02-12T17:05:16.773592+0000","last_peered":"2024-02-12T17:05:16.773592+0000","last_clean":"2024-02-12T17:05:16.773592+0000","last_became_active":"2024-02-12T17:03:20.769223+0000","last_became_peered":"2024-02-12T17:03:20.769223+0000","last_unstale":"2024-02-12T17:05:16.773592+0000","last_undegraded":"2024-02-12T17:05:16.773592+0000","last_fullsized":"2024-02-12T17:05:16.773592+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:02:44.688402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:44.688402+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":169,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.791489+0000","last_change":"2024-02-12T17:03:15.810501+0000","last_active":"2024-02-12T17:05:16.791489+0000","last_peered":"2024-02-12T17:05:16.791489+0000","last_clean":"2024-02-12T17:05:16.791489+0000","last_became_active":"2024-02-12T16:59:09.271093+0000","last_became_peered":"2024-02-12T16:59:09.271093+0000","last_unstale":"2024-02-12T17:05:16.791489+0000","last_undegraded":"2024-02-12T17:05:16.791489+0000","last_fullsized":"2024-02-12T17:05:16.791489+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"410'15191","reported_seq":16171,"reported_epoch":410,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:05:18.603399+0000","last_change":"2024-02-12T17:05:09.874394+0000","last_active":"2024-02-12T17:05:18.603399+0000","last_peered":"2024-02-12T17:05:18.603399+0000","last_clean":"2024-02-12T17:04:38.799786+0000","last_became_active":"2024-02-12T17:04:40.925272+0000","last_became_peered":"2024-02-12T17:04:40.925272+0000","last_unstale":"2024-02-12T17:05:18.603399+0000","last_undegraded":"2024-02-12T17:05:18.603399+0000","last_fullsized":"2024-02-12T17:05:18.603399+0000","mapping_epoch":378,"log_start":"353'12063","ondisk_log_start":"353'12063","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3128,"ondisk_log_size":3128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105250840,"num_objects":1607,"num_object_clones":0,"num_object_copies":4821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1320,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15191,"num_write_kb":537473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":27852800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,4],"avail_no_missing":["7","4","5"],"object_location_counts":[{"shards":"4,5,7","objects":1607}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":81,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786405+0000","last_change":"2024-02-12T17:04:39.886704+0000","last_active":"2024-02-12T17:05:16.786405+0000","last_peered":"2024-02-12T17:05:16.786405+0000","last_clean":"2024-02-12T17:05:16.786405+0000","last_became_active":"2024-02-12T17:04:39.885677+0000","last_became_peered":"2024-02-12T17:04:39.885677+0000","last_unstale":"2024-02-12T17:05:16.786405+0000","last_undegraded":"2024-02-12T17:05:16.786405+0000","last_fullsized":"2024-02-12T17:05:16.786405+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":168,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782433+0000","last_change":"2024-02-12T17:04:39.864989+0000","last_active":"2024-02-12T17:05:16.782433+0000","last_peered":"2024-02-12T17:05:16.782433+0000","last_clean":"2024-02-12T17:05:16.782433+0000","last_became_active":"2024-02-12T17:04:39.864631+0000","last_became_peered":"2024-02-12T17:04:39.864631+0000","last_unstale":"2024-02-12T17:05:16.782433+0000","last_undegraded":"2024-02-12T17:05:16.782433+0000","last_fullsized":"2024-02-12T17:05:16.782433+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":90,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781826+0000","last_change":"2024-02-12T17:03:54.620306+0000","last_active":"2024-02-12T17:05:16.781826+0000","last_peered":"2024-02-12T17:05:16.781826+0000","last_clean":"2024-02-12T17:05:16.781826+0000","last_became_active":"2024-02-12T17:03:50.556763+0000","last_became_peered":"2024-02-12T17:03:50.556763+0000","last_unstale":"2024-02-12T17:05:16.781826+0000","last_undegraded":"2024-02-12T17:05:16.781826+0000","last_fullsized":"2024-02-12T17:05:16.781826+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":383,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.787273+0000","last_change":"2024-02-12T17:04:39.891789+0000","last_active":"2024-02-12T17:05:16.787273+0000","last_peered":"2024-02-12T17:05:16.787273+0000","last_clean":"2024-02-12T17:05:16.787273+0000","last_became_active":"2024-02-12T17:04:39.890884+0000","last_became_peered":"2024-02-12T17:04:39.890884+0000","last_unstale":"2024-02-12T17:05:16.787273+0000","last_undegraded":"2024-02-12T17:05:16.787273+0000","last_fullsized":"2024-02-12T17:05:16.787273+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"410'15849","reported_seq":17975,"reported_epoch":410,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:18.598013+0000","last_change":"2024-02-12T17:04:50.113487+0000","last_active":"2024-02-12T17:05:18.598013+0000","last_peered":"2024-02-12T17:05:18.598013+0000","last_clean":"2024-02-12T17:03:19.745457+0000","last_became_active":"2024-02-12T17:04:39.930523+0000","last_became_peered":"2024-02-12T17:04:39.930523+0000","last_unstale":"2024-02-12T17:05:18.598013+0000","last_undegraded":"2024-02-12T17:05:18.598013+0000","last_fullsized":"2024-02-12T17:05:18.598013+0000","mapping_epoch":377,"log_start":"338'9837","ondisk_log_start":"338'9837","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'5831","last_scrub_stamp":"2024-02-12T17:02:24.719414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.719414+0000","log_size":6012,"ondisk_log_size":6012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2864,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15849,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":979,"num_bytes_recovered":56623104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":1629}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":413,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784586+0000","last_change":"2024-02-12T17:04:40.405970+0000","last_active":"2024-02-12T17:05:16.784586+0000","last_peered":"2024-02-12T17:05:16.784586+0000","last_clean":"2024-02-12T17:05:16.784586+0000","last_became_active":"2024-02-12T17:04:39.889077+0000","last_became_peered":"2024-02-12T17:04:39.889077+0000","last_unstale":"2024-02-12T17:05:16.784586+0000","last_undegraded":"2024-02-12T17:05:16.784586+0000","last_fullsized":"2024-02-12T17:05:16.784586+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:03:17.811656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.811656+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":178,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786005+0000","last_change":"2024-02-12T17:03:50.563493+0000","last_active":"2024-02-12T17:05:16.786005+0000","last_peered":"2024-02-12T17:05:16.786005+0000","last_clean":"2024-02-12T17:05:16.786005+0000","last_became_active":"2024-02-12T17:03:50.562556+0000","last_became_peered":"2024-02-12T17:03:50.562556+0000","last_unstale":"2024-02-12T17:05:16.786005+0000","last_undegraded":"2024-02-12T17:05:16.786005+0000","last_fullsized":"2024-02-12T17:05:16.786005+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.562425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.562425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":463,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.776557+0000","last_change":"2024-02-12T17:04:41.124664+0000","last_active":"2024-02-12T17:05:16.776557+0000","last_peered":"2024-02-12T17:05:16.776557+0000","last_clean":"2024-02-12T17:05:16.776557+0000","last_became_active":"2024-02-12T17:04:39.884095+0000","last_became_peered":"2024-02-12T17:04:39.884095+0000","last_unstale":"2024-02-12T17:05:16.776557+0000","last_undegraded":"2024-02-12T17:05:16.776557+0000","last_fullsized":"2024-02-12T17:05:16.776557+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"380'16","reported_seq":110,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:17.439642+0000","last_change":"2024-02-12T17:04:40.143962+0000","last_active":"2024-02-12T17:05:17.439642+0000","last_peered":"2024-02-12T17:05:17.439642+0000","last_clean":"2024-02-12T17:05:17.439642+0000","last_became_active":"2024-02-12T17:04:39.864546+0000","last_became_peered":"2024-02-12T17:04:39.864546+0000","last_unstale":"2024-02-12T17:05:17.439642+0000","last_undegraded":"2024-02-12T17:05:17.439642+0000","last_fullsized":"2024-02-12T17:05:17.439642+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"410'15342","reported_seq":16375,"reported_epoch":410,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:05:18.656731+0000","last_change":"2024-02-12T17:04:40.579124+0000","last_active":"2024-02-12T17:05:18.656731+0000","last_peered":"2024-02-12T17:05:18.656731+0000","last_clean":"2024-02-12T17:03:19.746645+0000","last_became_active":"2024-02-12T17:03:50.650695+0000","last_became_peered":"2024-02-12T17:03:50.650695+0000","last_unstale":"2024-02-12T17:05:18.656731+0000","last_undegraded":"2024-02-12T17:05:18.656731+0000","last_fullsized":"2024-02-12T17:05:18.656731+0000","mapping_epoch":347,"log_start":"353'12212","ondisk_log_start":"353'12212","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3130,"ondisk_log_size":3130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103415808,"num_objects":1578,"num_object_clones":0,"num_object_copies":4734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1285,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15342,"num_write_kb":541440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":29032448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,5,0],"avail_no_missing":["7","0","2","5"],"object_location_counts":[{"shards":"0,5,7","objects":1578}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":90,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781819+0000","last_change":"2024-02-12T17:04:33.794769+0000","last_active":"2024-02-12T17:05:16.781819+0000","last_peered":"2024-02-12T17:05:16.781819+0000","last_clean":"2024-02-12T17:05:16.781819+0000","last_became_active":"2024-02-12T17:04:33.794457+0000","last_became_peered":"2024-02-12T17:04:33.794457+0000","last_unstale":"2024-02-12T17:05:16.781819+0000","last_undegraded":"2024-02-12T17:05:16.781819+0000","last_fullsized":"2024-02-12T17:05:16.781819+0000","mapping_epoch":347,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.811153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.811153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"406'18","reported_seq":548,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:17.442794+0000","last_change":"2024-02-12T17:05:11.317793+0000","last_active":"2024-02-12T17:05:17.442794+0000","last_peered":"2024-02-12T17:05:17.442794+0000","last_clean":"2024-02-12T17:05:17.442794+0000","last_became_active":"2024-02-12T17:05:11.317402+0000","last_became_peered":"2024-02-12T17:05:11.317402+0000","last_unstale":"2024-02-12T17:05:17.442794+0000","last_undegraded":"2024-02-12T17:05:17.442794+0000","last_fullsized":"2024-02-12T17:05:17.442794+0000","mapping_epoch":404,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":42,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781789+0000","last_change":"2024-02-12T17:04:39.890077+0000","last_active":"2024-02-12T17:05:16.781789+0000","last_peered":"2024-02-12T17:05:16.781789+0000","last_clean":"2024-02-12T17:05:16.781789+0000","last_became_active":"2024-02-12T17:04:39.889588+0000","last_became_peered":"2024-02-12T17:04:39.889588+0000","last_unstale":"2024-02-12T17:05:16.781789+0000","last_undegraded":"2024-02-12T17:05:16.781789+0000","last_fullsized":"2024-02-12T17:05:16.781789+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":105,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782801+0000","last_change":"2024-02-12T17:04:41.814348+0000","last_active":"2024-02-12T17:05:16.782801+0000","last_peered":"2024-02-12T17:05:16.782801+0000","last_clean":"2024-02-12T17:05:16.782801+0000","last_became_active":"2024-02-12T17:04:39.888473+0000","last_became_peered":"2024-02-12T17:04:39.888473+0000","last_unstale":"2024-02-12T17:05:16.782801+0000","last_undegraded":"2024-02-12T17:05:16.782801+0000","last_fullsized":"2024-02-12T17:05:16.782801+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"410'15414","reported_seq":16378,"reported_epoch":410,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:05:18.679013+0000","last_change":"2024-02-12T17:05:10.406749+0000","last_active":"2024-02-12T17:05:18.679013+0000","last_peered":"2024-02-12T17:05:18.679013+0000","last_clean":"2024-02-12T17:03:19.738005+0000","last_became_active":"2024-02-12T17:03:50.628646+0000","last_became_peered":"2024-02-12T17:03:50.628646+0000","last_unstale":"2024-02-12T17:05:18.679013+0000","last_undegraded":"2024-02-12T17:05:18.679013+0000","last_fullsized":"2024-02-12T17:05:18.679013+0000","mapping_epoch":347,"log_start":"353'12269","ondisk_log_start":"353'12269","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3145,"ondisk_log_size":3145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":107216896,"num_objects":1636,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1580,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15414,"num_write_kb":545600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":26804224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[7,2,0],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":1636}],"blocked_by":[],"up_primary":3,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":420,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.777131+0000","last_change":"2024-02-12T17:04:39.887836+0000","last_active":"2024-02-12T17:05:16.777131+0000","last_peered":"2024-02-12T17:05:16.777131+0000","last_clean":"2024-02-12T17:05:16.777131+0000","last_became_active":"2024-02-12T17:04:39.887488+0000","last_became_peered":"2024-02-12T17:04:39.887488+0000","last_unstale":"2024-02-12T17:05:16.777131+0000","last_undegraded":"2024-02-12T17:05:16.777131+0000","last_fullsized":"2024-02-12T17:05:16.777131+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"406'18","reported_seq":614,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:17.444532+0000","last_change":"2024-02-12T17:05:12.353020+0000","last_active":"2024-02-12T17:05:17.444532+0000","last_peered":"2024-02-12T17:05:17.444532+0000","last_clean":"2024-02-12T17:05:17.444532+0000","last_became_active":"2024-02-12T17:05:12.352616+0000","last_became_peered":"2024-02-12T17:05:12.352616+0000","last_unstale":"2024-02-12T17:05:17.444532+0000","last_undegraded":"2024-02-12T17:05:17.444532+0000","last_fullsized":"2024-02-12T17:05:17.444532+0000","mapping_epoch":405,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-12T17:02:20.866591+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:02:20.866591+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":42,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781972+0000","last_change":"2024-02-12T17:04:39.886675+0000","last_active":"2024-02-12T17:05:16.781972+0000","last_peered":"2024-02-12T17:05:16.781972+0000","last_clean":"2024-02-12T17:05:16.781972+0000","last_became_active":"2024-02-12T17:04:39.886222+0000","last_became_peered":"2024-02-12T17:04:39.886222+0000","last_unstale":"2024-02-12T17:05:16.781972+0000","last_undegraded":"2024-02-12T17:05:16.781972+0000","last_fullsized":"2024-02-12T17:05:16.781972+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.863530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.863530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":113,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783479+0000","last_change":"2024-02-12T17:04:43.226687+0000","last_active":"2024-02-12T17:05:16.783479+0000","last_peered":"2024-02-12T17:05:16.783479+0000","last_clean":"2024-02-12T17:05:16.783479+0000","last_became_active":"2024-02-12T17:04:39.892266+0000","last_became_peered":"2024-02-12T17:04:39.892266+0000","last_unstale":"2024-02-12T17:05:16.783479+0000","last_undegraded":"2024-02-12T17:05:16.783479+0000","last_fullsized":"2024-02-12T17:05:16.783479+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:01:41.694875+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:01:41.694875+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"410'15518","reported_seq":15659,"reported_epoch":410,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:05:18.281270+0000","last_change":"2024-02-12T17:05:09.931525+0000","last_active":"2024-02-12T17:05:18.281270+0000","last_peered":"2024-02-12T17:05:18.281270+0000","last_clean":"2024-02-12T17:03:49.455800+0000","last_became_active":"2024-02-12T17:04:39.870350+0000","last_became_peered":"2024-02-12T17:04:39.870350+0000","last_unstale":"2024-02-12T17:05:18.281270+0000","last_undegraded":"2024-02-12T17:05:18.281270+0000","last_fullsized":"2024-02-12T17:05:18.281270+0000","mapping_epoch":377,"log_start":"351'11925","ondisk_log_start":"351'11925","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3593,"ondisk_log_size":3593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1628,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15518,"num_write_kb":548672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[2,7,4],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":1628}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":147,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.777073+0000","last_change":"2024-02-12T17:04:39.884597+0000","last_active":"2024-02-12T17:05:16.777073+0000","last_peered":"2024-02-12T17:05:16.777073+0000","last_clean":"2024-02-12T17:05:16.777073+0000","last_became_active":"2024-02-12T17:04:39.884295+0000","last_became_peered":"2024-02-12T17:04:39.884295+0000","last_unstale":"2024-02-12T17:05:16.777073+0000","last_undegraded":"2024-02-12T17:05:16.777073+0000","last_fullsized":"2024-02-12T17:05:16.777073+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"406'13","reported_seq":197,"reported_epoch":409,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:17.456428+0000","last_change":"2024-02-12T17:05:10.902647+0000","last_active":"2024-02-12T17:05:17.456428+0000","last_peered":"2024-02-12T17:05:17.456428+0000","last_clean":"2024-02-12T17:05:07.829488+0000","last_became_active":"2024-02-12T17:05:10.902307+0000","last_became_peered":"2024-02-12T17:05:10.902307+0000","last_unstale":"2024-02-12T17:05:17.456428+0000","last_undegraded":"2024-02-12T17:05:17.456428+0000","last_fullsized":"2024-02-12T17:05:17.456428+0000","mapping_epoch":403,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'11","last_scrub_stamp":"2024-02-12T17:02:31.546965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:31.546965+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,2,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":72,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786490+0000","last_change":"2024-02-12T17:03:50.589204+0000","last_active":"2024-02-12T17:05:16.786490+0000","last_peered":"2024-02-12T17:05:16.786490+0000","last_clean":"2024-02-12T17:05:16.786490+0000","last_became_active":"2024-02-12T17:03:50.588844+0000","last_became_peered":"2024-02-12T17:03:50.588844+0000","last_unstale":"2024-02-12T17:05:16.786490+0000","last_undegraded":"2024-02-12T17:05:16.786490+0000","last_fullsized":"2024-02-12T17:05:16.786490+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.888942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.888942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":105,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781901+0000","last_change":"2024-02-12T17:04:42.611814+0000","last_active":"2024-02-12T17:05:16.781901+0000","last_peered":"2024-02-12T17:05:16.781901+0000","last_clean":"2024-02-12T17:05:16.781901+0000","last_became_active":"2024-02-12T17:04:39.891435+0000","last_became_peered":"2024-02-12T17:04:39.891435+0000","last_unstale":"2024-02-12T17:05:16.781901+0000","last_undegraded":"2024-02-12T17:05:16.781901+0000","last_fullsized":"2024-02-12T17:05:16.781901+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"410'15559","reported_seq":15673,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-12T17:05:18.626791+0000","last_change":"2024-02-12T17:01:16.913292+0000","last_active":"2024-02-12T17:05:18.626791+0000","last_peered":"2024-02-12T17:05:18.626791+0000","last_clean":"2024-02-12T17:05:18.626791+0000","last_became_active":"2024-02-12T17:01:16.912605+0000","last_became_peered":"2024-02-12T17:01:16.912605+0000","last_unstale":"2024-02-12T17:05:18.626791+0000","last_undegraded":"2024-02-12T17:05:18.626791+0000","last_fullsized":"2024-02-12T17:05:18.626791+0000","mapping_epoch":307,"log_start":"331'6800","ondisk_log_start":"331'6800","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":8759,"ondisk_log_size":8759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103874560,"num_objects":1585,"num_object_clones":0,"num_object_copies":4755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15559,"num_write_kb":548608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":99,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783562+0000","last_change":"2024-02-12T17:04:39.875038+0000","last_active":"2024-02-12T17:05:16.783562+0000","last_peered":"2024-02-12T17:05:16.783562+0000","last_clean":"2024-02-12T17:05:16.783562+0000","last_became_active":"2024-02-12T17:04:39.874748+0000","last_became_peered":"2024-02-12T17:04:39.874748+0000","last_unstale":"2024-02-12T17:05:16.783562+0000","last_undegraded":"2024-02-12T17:05:16.783562+0000","last_fullsized":"2024-02-12T17:05:16.783562+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":102,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786297+0000","last_change":"2024-02-12T17:05:09.854431+0000","last_active":"2024-02-12T17:05:16.786297+0000","last_peered":"2024-02-12T17:05:16.786297+0000","last_clean":"2024-02-12T17:05:16.786297+0000","last_became_active":"2024-02-12T17:05:09.854112+0000","last_became_peered":"2024-02-12T17:05:09.854112+0000","last_unstale":"2024-02-12T17:05:16.786297+0000","last_undegraded":"2024-02-12T17:05:16.786297+0000","last_fullsized":"2024-02-12T17:05:16.786297+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:19.084549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.084549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":388,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.784361+0000","last_change":"2024-02-12T17:04:39.888763+0000","last_active":"2024-02-12T17:05:16.784361+0000","last_peered":"2024-02-12T17:05:16.784361+0000","last_clean":"2024-02-12T17:05:16.784361+0000","last_became_active":"2024-02-12T17:04:39.888466+0000","last_became_peered":"2024-02-12T17:04:39.888466+0000","last_unstale":"2024-02-12T17:05:16.784361+0000","last_undegraded":"2024-02-12T17:05:16.784361+0000","last_fullsized":"2024-02-12T17:05:16.784361+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.858955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.858955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":124,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.776823+0000","last_change":"2024-02-12T17:04:41.383385+0000","last_active":"2024-02-12T17:05:16.776823+0000","last_peered":"2024-02-12T17:05:16.776823+0000","last_clean":"2024-02-12T17:05:16.776823+0000","last_became_active":"2024-02-12T17:04:39.887688+0000","last_became_peered":"2024-02-12T17:04:39.887688+0000","last_unstale":"2024-02-12T17:05:16.776823+0000","last_undegraded":"2024-02-12T17:05:16.776823+0000","last_fullsized":"2024-02-12T17:05:16.776823+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:01:35.160145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:35.160145+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"410'15226","reported_seq":15356,"reported_epoch":410,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:05:18.295931+0000","last_change":"2024-02-12T17:05:11.130524+0000","last_active":"2024-02-12T17:05:18.295931+0000","last_peered":"2024-02-12T17:05:18.295931+0000","last_clean":"2024-02-12T17:03:49.522755+0000","last_became_active":"2024-02-12T17:03:51.663049+0000","last_became_peered":"2024-02-12T17:03:51.663049+0000","last_unstale":"2024-02-12T17:05:18.295931+0000","last_undegraded":"2024-02-12T17:05:18.295931+0000","last_fullsized":"2024-02-12T17:05:18.295931+0000","mapping_epoch":348,"log_start":"353'12135","ondisk_log_start":"353'12135","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3091,"ondisk_log_size":3091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1620,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15226,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":1620}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":422,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.781948+0000","last_change":"2024-02-12T17:04:50.393748+0000","last_active":"2024-02-12T17:05:16.781948+0000","last_peered":"2024-02-12T17:05:16.781948+0000","last_clean":"2024-02-12T17:05:16.781948+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:05:16.781948+0000","last_undegraded":"2024-02-12T17:05:16.781948+0000","last_fullsized":"2024-02-12T17:05:16.781948+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":315,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.796713+0000","last_change":"2024-02-12T17:03:28.770200+0000","last_active":"2024-02-12T17:05:16.796713+0000","last_peered":"2024-02-12T17:05:16.796713+0000","last_clean":"2024-02-12T17:05:16.796713+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:05:16.796713+0000","last_undegraded":"2024-02-12T17:05:16.796713+0000","last_fullsized":"2024-02-12T17:05:16.796713+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":132,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.773553+0000","last_change":"2024-02-12T17:03:50.552102+0000","last_active":"2024-02-12T17:05:16.773553+0000","last_peered":"2024-02-12T17:05:16.773553+0000","last_clean":"2024-02-12T17:05:16.773553+0000","last_became_active":"2024-02-12T17:03:50.551804+0000","last_became_peered":"2024-02-12T17:03:50.551804+0000","last_unstale":"2024-02-12T17:05:16.773553+0000","last_undegraded":"2024-02-12T17:05:16.773553+0000","last_fullsized":"2024-02-12T17:05:16.773553+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.794228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":317,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.786817+0000","last_change":"2024-02-12T17:03:52.389245+0000","last_active":"2024-02-12T17:05:16.786817+0000","last_peered":"2024-02-12T17:05:16.786817+0000","last_clean":"2024-02-12T17:05:16.786817+0000","last_became_active":"2024-02-12T17:03:50.574562+0000","last_became_peered":"2024-02-12T17:03:50.574562+0000","last_unstale":"2024-02-12T17:05:16.786817+0000","last_undegraded":"2024-02-12T17:05:16.786817+0000","last_fullsized":"2024-02-12T17:05:16.786817+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:02:45.594719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:45.594719+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"410'15680","reported_seq":24642,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-12T17:05:18.268634+0000","last_change":"2024-02-12T17:05:11.339984+0000","last_active":"2024-02-12T17:05:18.268634+0000","last_peered":"2024-02-12T17:05:18.268634+0000","last_clean":"2024-02-12T17:05:18.268634+0000","last_became_active":"2024-02-12T17:05:11.322885+0000","last_became_peered":"2024-02-12T17:05:11.322885+0000","last_unstale":"2024-02-12T17:05:18.268634+0000","last_undegraded":"2024-02-12T17:05:18.268634+0000","last_fullsized":"2024-02-12T17:05:18.268634+0000","mapping_epoch":404,"log_start":"354'12611","ondisk_log_start":"354'12611","created":304,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109707264,"num_objects":1674,"num_object_clones":0,"num_object_copies":5022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15680,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4404,"num_bytes_recovered":288620544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":108,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782268+0000","last_change":"2024-02-12T17:04:46.024414+0000","last_active":"2024-02-12T17:05:16.782268+0000","last_peered":"2024-02-12T17:05:16.782268+0000","last_clean":"2024-02-12T17:05:16.782268+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:05:16.782268+0000","last_undegraded":"2024-02-12T17:05:16.782268+0000","last_fullsized":"2024-02-12T17:05:16.782268+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.986137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.986137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":318,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.785957+0000","last_change":"2024-02-12T17:04:39.886902+0000","last_active":"2024-02-12T17:05:16.785957+0000","last_peered":"2024-02-12T17:05:16.785957+0000","last_clean":"2024-02-12T17:05:16.785957+0000","last_became_active":"2024-02-12T17:04:39.886403+0000","last_became_peered":"2024-02-12T17:04:39.886403+0000","last_unstale":"2024-02-12T17:05:16.785957+0000","last_undegraded":"2024-02-12T17:05:16.785957+0000","last_fullsized":"2024-02-12T17:05:16.785957+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":89,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782605+0000","last_change":"2024-02-12T17:04:41.654733+0000","last_active":"2024-02-12T17:05:16.782605+0000","last_peered":"2024-02-12T17:05:16.782605+0000","last_clean":"2024-02-12T17:05:16.782605+0000","last_became_active":"2024-02-12T17:04:39.889999+0000","last_became_peered":"2024-02-12T17:04:39.889999+0000","last_unstale":"2024-02-12T17:05:16.782605+0000","last_undegraded":"2024-02-12T17:05:16.782605+0000","last_fullsized":"2024-02-12T17:05:16.782605+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:00:50.756578+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.756578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":365,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783770+0000","last_change":"2024-02-12T17:04:42.178594+0000","last_active":"2024-02-12T17:05:16.783770+0000","last_peered":"2024-02-12T17:05:16.783770+0000","last_clean":"2024-02-12T17:05:16.783770+0000","last_became_active":"2024-02-12T17:04:39.886809+0000","last_became_peered":"2024-02-12T17:04:39.886809+0000","last_unstale":"2024-02-12T17:05:16.783770+0000","last_undegraded":"2024-02-12T17:05:16.783770+0000","last_fullsized":"2024-02-12T17:05:16.783770+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:28.096708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.096708+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"410'15821","reported_seq":15935,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-12T17:05:18.597894+0000","last_change":"2024-02-12T17:01:13.886545+0000","last_active":"2024-02-12T17:05:18.597894+0000","last_peered":"2024-02-12T17:05:18.597894+0000","last_clean":"2024-02-12T17:05:18.597894+0000","last_became_active":"2024-02-12T17:01:13.886238+0000","last_became_peered":"2024-02-12T17:01:13.886238+0000","last_unstale":"2024-02-12T17:05:18.597894+0000","last_undegraded":"2024-02-12T17:05:18.597894+0000","last_fullsized":"2024-02-12T17:05:18.597894+0000","mapping_epoch":304,"log_start":"331'7000","ondisk_log_start":"331'7000","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":8821,"ondisk_log_size":8821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":108593152,"num_objects":1657,"num_object_clones":0,"num_object_copies":4971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15821,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":397,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.783312+0000","last_change":"2024-02-12T17:04:41.899891+0000","last_active":"2024-02-12T17:05:16.783312+0000","last_peered":"2024-02-12T17:05:16.783312+0000","last_clean":"2024-02-12T17:05:16.783312+0000","last_became_active":"2024-02-12T17:04:41.899598+0000","last_became_peered":"2024-02-12T17:04:41.899598+0000","last_unstale":"2024-02-12T17:05:16.783312+0000","last_undegraded":"2024-02-12T17:05:16.783312+0000","last_fullsized":"2024-02-12T17:05:16.783312+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:26.803903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.803903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":72,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.782832+0000","last_change":"2024-02-12T17:03:50.560958+0000","last_active":"2024-02-12T17:05:16.782832+0000","last_peered":"2024-02-12T17:05:16.782832+0000","last_clean":"2024-02-12T17:05:16.782832+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:05:16.782832+0000","last_undegraded":"2024-02-12T17:05:16.782832+0000","last_fullsized":"2024-02-12T17:05:16.782832+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:01.789464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:01.789464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":479,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-12T17:05:16.779147+0000","last_change":"2024-02-12T17:04:40.522253+0000","last_active":"2024-02-12T17:05:16.779147+0000","last_peered":"2024-02-12T17:05:16.779147+0000","last_clean":"2024-02-12T17:05:16.779147+0000","last_became_active":"2024-02-12T17:04:39.882260+0000","last_became_peered":"2024-02-12T17:04:39.882260+0000","last_unstale":"2024-02-12T17:05:16.779147+0000","last_undegraded":"2024-02-12T17:05:16.779147+0000","last_fullsized":"2024-02-12T17:05:16.779147+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:02:43.665248+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.665248+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"380'17","reported_seq":642,"reported_epoch":409,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:17.456499+0000","last_change":"2024-02-12T17:05:09.874904+0000","last_active":"2024-02-12T17:05:17.456499+0000","last_peered":"2024-02-12T17:05:17.456499+0000","last_clean":"2024-02-12T17:03:47.426268+0000","last_became_active":"2024-02-12T17:04:40.872513+0000","last_became_peered":"2024-02-12T17:04:40.872513+0000","last_unstale":"2024-02-12T17:05:17.456499+0000","last_undegraded":"2024-02-12T17:05:17.456499+0000","last_fullsized":"2024-02-12T17:05:17.456499+0000","mapping_epoch":378,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-02-12T17:00:49.304784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.304784+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,2,4],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1690763288,"num_objects":25800,"num_object_clones":0,"num_object_copies":77400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20527,"num_objects_unfound":0,"num_objects_dirty":25800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249376,"num_write_kb":8805569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14212,"num_bytes_recovered":923533312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5525286912,"data_stored":5525274696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75933,"ondisk_log_size":75933,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":6},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":247,"num_read_kb":242,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1093,"num_read_kb":850,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":11090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":20,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":60,"acting":60,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234175,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2154932,"kb_used_data":1263084,"kb_used_omap":0,"kb_used_meta":891456,"kb_avail":91590220,"statfs":{"total":95995035648,"available":93788385280,"internally_reserved":0,"allocated":1293398016,"data_stored":1292777371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912850944},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":3.7330000000000001,"5min":2.6800000000000002,"15min":1.837},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":8.6120000000000001,"5min":16.759,"15min":36.439},"last":3.879},{"interface":"front","average":{"1min":2.7919999999999998,"5min":2.423,"15min":1.887},"min":{"1min":0.38500000000000001,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":8.0800000000000001,"5min":17.844000000000001,"15min":36.490000000000002},"last":0.25}]},{"osd":1,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":4.5330000000000004,"5min":2.2370000000000001,"15min":1.575},"min":{"1min":0.41599999999999998,"5min":0.23400000000000001,"15min":0.20899999999999999},"max":{"1min":9.6980000000000004,"5min":9.6980000000000004,"15min":12.032},"last":0.98999999999999999},{"interface":"front","average":{"1min":3.1760000000000002,"5min":2.3380000000000001,"15min":1.5740000000000001},"min":{"1min":0.42599999999999999,"5min":0.25800000000000001,"15min":0.22700000000000001},"max":{"1min":8.1170000000000009,"5min":17.324999999999999,"15min":17.324999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":2.9369999999999998,"5min":2.2029999999999998,"15min":1.448},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":6.7290000000000001,"5min":12.4,"15min":12.4},"last":3.782},{"interface":"front","average":{"1min":4.4740000000000002,"5min":2.274,"15min":1.716},"min":{"1min":0.28000000000000003,"5min":0.26400000000000001,"15min":0.23000000000000001},"max":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":20.338000000000001},"last":1.02}]},{"osd":3,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":3.5369999999999999,"5min":2.3889999999999998,"15min":1.762},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.6489999999999991,"5min":13.608000000000001,"15min":20.433},"last":3.8130000000000002},{"interface":"front","average":{"1min":4.6260000000000003,"5min":2.6480000000000001,"15min":2.028},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.25600000000000001},"max":{"1min":9.7200000000000006,"5min":31.152000000000001,"15min":31.152000000000001},"last":1.9810000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":4.8440000000000003,"5min":3.177,"15min":1.8720000000000001},"min":{"1min":0.435,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":10.138,"5min":19.030000000000001,"15min":50.406999999999996},"last":3.899},{"interface":"front","average":{"1min":5.5979999999999999,"5min":3.2109999999999999,"15min":2.1640000000000001},"min":{"1min":0.33500000000000002,"5min":0.27000000000000002,"15min":0.23400000000000001},"max":{"1min":11.191000000000001,"5min":16.867000000000001,"15min":50.168999999999997},"last":1.1020000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":5.9130000000000003,"5min":3.8319999999999999,"15min":1.9319999999999999},"min":{"1min":0.39300000000000002,"5min":0.26800000000000002,"15min":0.24199999999999999},"max":{"1min":14.074,"5min":19.702000000000002,"15min":19.702000000000002},"last":3.8580000000000001},{"interface":"front","average":{"1min":6.1769999999999996,"5min":3.2229999999999999,"15min":1.9850000000000001},"min":{"1min":0.308,"5min":0.28499999999999998,"15min":0.26200000000000001},"max":{"1min":17.585999999999999,"5min":18.651,"15min":18.651},"last":1.321}]},{"osd":6,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":3.8919999999999999,"5min":2.8399999999999999,"15min":1.6779999999999999},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":8.1839999999999993,"5min":17.234999999999999,"15min":17.234999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.6389999999999998,"5min":2.681,"15min":1.6160000000000001},"min":{"1min":0.40899999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.5549999999999997,"5min":13.676,"15min":13.676},"last":3.8359999999999999}]}]},{"osd":6,"up_from":0,"seq":193273528516,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738593,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2068972,"kb_used_data":1172708,"kb_used_omap":0,"kb_used_meta":896192,"kb_avail":91676180,"statfs":{"total":95995035648,"available":93876408320,"internally_reserved":0,"allocated":1200852992,"data_stored":1200360663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917700608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:04:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.83999999999999997,"15min":1.2549999999999999},"min":{"1min":0.221,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.2469999999999999,"5min":13.013,"15min":23.510999999999999},"last":2.1600000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.92600000000000005,"15min":1.222},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":16.16,"5min":16.16,"15min":16.16},"last":2.2669999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":0.82299999999999995,"15min":1.464},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.7750000000000004,"5min":5.7750000000000004,"15min":16.521000000000001},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.1100000000000001,"15min":1.381},"min":{"1min":0.26800000000000002,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":9.827,"5min":9.827,"15min":10.340999999999999},"last":3.419}]},{"osd":3,"last update":"Mon Feb 12 17:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.141,"15min":1.5569999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.19400000000000001},"max":{"1min":6.923,"5min":7.0339999999999998,"15min":21.100000000000001},"last":1.946},{"interface":"front","average":{"1min":1.9770000000000001,"5min":1.484,"15min":1.623},"min":{"1min":0.26200000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":16.759,"5min":16.759,"15min":16.759},"last":1.4410000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:04:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.8009999999999999,"15min":1.5349999999999999},"min":{"1min":0.42699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.6849999999999996,"5min":16.888999999999999,"15min":16.888999999999999},"last":1.47},{"interface":"front","average":{"1min":1.778,"5min":1.8500000000000001,"15min":1.5069999999999999},"min":{"1min":0.40100000000000002,"5min":0.28799999999999998,"15min":0.28399999999999997},"max":{"1min":4.9320000000000004,"5min":16.899999999999999,"15min":16.899999999999999},"last":2.0859999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:05:16 2024","interfaces":[{"interface":"back","average":{"1min":2.726,"5min":2.5619999999999998,"15min":2.1200000000000001},"min":{"1min":0.38300000000000001,"5min":0.30199999999999999,"15min":0.26400000000000001},"max":{"1min":10.146000000000001,"5min":50.404000000000003,"15min":50.404000000000003},"last":1.542},{"interface":"front","average":{"1min":3.0449999999999999,"5min":2.4100000000000001,"15min":2.0369999999999999},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.3550000000000004,"5min":49.979999999999997,"15min":49.979999999999997},"last":1.514}]},{"osd":6,"last update":"Mon Feb 12 17:04:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.542,"15min":1.534},"min":{"1min":0.35199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.7510000000000003,"5min":9.8460000000000001,"15min":17.907},"last":2.1320000000000001},{"interface":"front","average":{"1min":2.137,"5min":2.0619999999999998,"15min":1.8540000000000001},"min":{"1min":0.41599999999999998,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":7.7510000000000003,"5min":9.6739999999999995,"15min":21.062000000000001},"last":1.9870000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":3.5699999999999998,"5min":2.375,"15min":1.863},"min":{"1min":0.498,"5min":0.40699999999999997,"15min":0.32400000000000001},"max":{"1min":12.836,"5min":19.373000000000001,"15min":19.373000000000001},"last":1.492},{"interface":"front","average":{"1min":2.8370000000000002,"5min":2.3050000000000002,"15min":1.8440000000000001},"min":{"1min":0.442,"5min":0.39900000000000002,"15min":0.33600000000000002},"max":{"1min":6.3550000000000004,"5min":15.550000000000001,"15min":34.722999999999999},"last":2.1120000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411542,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753796,"kb_used_data":861052,"kb_used_omap":0,"kb_used_meta":892672,"kb_avail":91991356,"statfs":{"total":95995035648,"available":94199148544,"internally_reserved":0,"allocated":881717248,"data_stored":881144249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914096128},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1220000000000001,"15min":1.589},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.8339999999999996,"5min":36.100999999999999,"15min":36.100999999999999},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.859,"5min":1.5049999999999999,"15min":1.5569999999999999},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.158},"max":{"1min":7.548,"5min":36.299999999999997,"15min":36.299999999999997},"last":0.29599999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.64600000000000002,"15min":1.49},"min":{"1min":0.247,"5min":0.215,"15min":0.191},"max":{"1min":1.3560000000000001,"5min":5.0890000000000004,"15min":17.739000000000001},"last":0.246},{"interface":"front","average":{"1min":1.4950000000000001,"5min":0.97899999999999998,"15min":1.591},"min":{"1min":0.28000000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":7.8609999999999998,"5min":7.8609999999999998,"15min":15.766999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:04:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.048,"15min":1.734},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.23499999999999999},"max":{"1min":1.9370000000000001,"5min":5.1239999999999997,"15min":51.579999999999998},"last":3.6339999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1,"15min":1.468},"min":{"1min":0.26400000000000001,"5min":0.23699999999999999,"15min":0.23200000000000001},"max":{"1min":4.0499999999999998,"5min":4.5449999999999999,"15min":48.988999999999997},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Feb 12 17:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.7809999999999999,"15min":1.7430000000000001},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":5.8559999999999999,"5min":18.658999999999999,"15min":18.658999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.8680000000000001,"5min":2.1720000000000002,"15min":1.7749999999999999},"min":{"1min":0.38,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":6.2220000000000004,"5min":18.355,"15min":18.355},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":2.0699999999999998,"15min":1.877},"min":{"1min":0.308,"5min":0.308,"15min":0.27300000000000002},"max":{"1min":10.728999999999999,"5min":32.57,"15min":32.57},"last":6.3029999999999999},{"interface":"front","average":{"1min":2.6309999999999998,"5min":2.3999999999999999,"15min":2.1179999999999999},"min":{"1min":0.40400000000000003,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":9.8420000000000005,"5min":31.905999999999999,"15min":31.905999999999999},"last":0.79500000000000004}]},{"osd":6,"last update":"Mon Feb 12 17:04:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.597,"15min":1.891},"min":{"1min":0.46700000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.6669999999999998,"5min":7.6669999999999998,"15min":17.782},"last":2.3399999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.8,"15min":1.6479999999999999},"min":{"1min":0.39000000000000001,"5min":0.35499999999999998,"15min":0.29799999999999999},"max":{"1min":4.008,"5min":8.7850000000000001,"15min":13.446999999999999},"last":10.6}]},{"osd":7,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":4.1719999999999997,"5min":3.2090000000000001,"15min":2.258},"min":{"1min":0.57999999999999996,"5min":0.33700000000000002,"15min":0.32600000000000001},"max":{"1min":10.754,"5min":45.468000000000004,"15min":45.468000000000004},"last":2.4630000000000001},{"interface":"front","average":{"1min":2.3969999999999998,"5min":2.9870000000000001,"15min":2.331},"min":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":6.7000000000000002,"5min":46.656999999999996,"15min":46.656999999999996},"last":2.6269999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149906,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":496740,"kb_used_data":187228,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93248412,"statfs":{"total":95995035648,"available":95486373888,"internally_reserved":0,"allocated":191721472,"data_stored":191117082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:04:32 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.232,"15min":1.5489999999999999},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":5.415,"5min":58.451999999999998,"15min":58.451999999999998},"last":2.1019999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":1.369,"15min":1.5289999999999999},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.6349999999999998,"5min":57.497999999999998,"15min":57.497999999999998},"last":2.7589999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.93500000000000005,"15min":1.3240000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.216},"max":{"1min":3.6499999999999999,"5min":4.782,"15min":13.94},"last":2.7360000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.75600000000000001,"15min":1.2370000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":4.7709999999999999,"5min":4.7709999999999999,"15min":14.962},"last":0.501}]},{"osd":2,"last update":"Mon Feb 12 17:04:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.872,"15min":1.5},"min":{"1min":0.33200000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.431,"5min":4.6079999999999997,"15min":49.112000000000002},"last":2.6379999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.81999999999999995,"15min":1.6240000000000001},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.9420000000000002,"5min":4.9420000000000002,"15min":51.997999999999998},"last":2.6949999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.722,"15min":1.7070000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":4.8710000000000004,"5min":11.047000000000001,"15min":19.010000000000002},"last":3.1720000000000002},{"interface":"front","average":{"1min":2.903,"5min":2.0790000000000002,"15min":1.655},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":10.250999999999999,"5min":19.033999999999999,"15min":19.033999999999999},"last":2.6640000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:05:16 2024","interfaces":[{"interface":"back","average":{"1min":2.2130000000000001,"5min":2.048,"15min":1.9419999999999999},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.9589999999999996,"5min":14.236000000000001,"15min":19.882999999999999},"last":2.129},{"interface":"front","average":{"1min":3.4590000000000001,"5min":2.423,"15min":2.0209999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.33300000000000002},"max":{"1min":12.502000000000001,"5min":13.927,"15min":15.595000000000001},"last":2.278}]},{"osd":6,"last update":"Mon Feb 12 17:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":1.5169999999999999,"15min":1.425},"min":{"1min":0.48399999999999999,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":5.3129999999999997,"5min":6.6150000000000002,"15min":12.856999999999999},"last":2.944},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.748,"15min":1.6240000000000001},"min":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.27500000000000002},"max":{"1min":4.9729999999999999,"5min":12.635999999999999,"15min":19.925999999999998},"last":2.161}]},{"osd":7,"last update":"Mon Feb 12 17:04:51 2024","interfaces":[{"interface":"back","average":{"1min":1.994,"5min":2.2989999999999999,"15min":1.907},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":5.585,"5min":12.695,"15min":12.891},"last":4.1210000000000004},{"interface":"front","average":{"1min":2.617,"5min":2.4529999999999998,"15min":1.923},"min":{"1min":0.49099999999999999,"5min":0.40799999999999997,"15min":0.32800000000000001},"max":{"1min":5.1369999999999996,"5min":25.838999999999999,"15min":25.838999999999999},"last":2.2509999999999999}]}]},{"osd":4,"up_from":0,"seq":137438953678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757450,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2177704,"kb_used_data":1281120,"kb_used_omap":0,"kb_used_meta":896576,"kb_avail":91567448,"statfs":{"total":95995035648,"available":93765066752,"internally_reserved":0,"allocated":1311866880,"data_stored":1311257543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":918093824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":2.202,"15min":1.7430000000000001},"min":{"1min":0.23899999999999999,"5min":0.20399999999999999,"15min":0.20300000000000001},"max":{"1min":10.971,"5min":46.365000000000002,"15min":46.365000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.5419999999999998,"15min":2.101},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":12.140000000000001,"5min":46.134,"15min":46.134},"last":8.1400000000000006}]},{"osd":1,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":2.573,"5min":1.802,"15min":1.6299999999999999},"min":{"1min":0.33000000000000002,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":11.19,"5min":11.19,"15min":16.655000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":3.1739999999999999,"5min":2.1789999999999998,"15min":1.8839999999999999},"min":{"1min":0.26300000000000001,"5min":0.23799999999999999,"15min":0.21199999999999999},"max":{"1min":12.154,"5min":13.130000000000001,"15min":16.712},"last":8.1159999999999997}]},{"osd":2,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":3.1400000000000001,"5min":2.04,"15min":1.8999999999999999},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":11.007999999999999,"5min":11.15,"15min":18.218},"last":0.46000000000000002},{"interface":"front","average":{"1min":2.2669999999999999,"5min":1.998,"15min":1.788},"min":{"1min":0.32500000000000001,"5min":0.25,"15min":0.25},"max":{"1min":6.0839999999999996,"5min":9.1199999999999992,"15min":18.327000000000002},"last":0.83299999999999996}]},{"osd":3,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8860000000000001,"5min":1.923,"15min":1.784},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":11.045,"5min":11.122,"15min":18.251000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":2.4449999999999998,"5min":1.925,"15min":1.7},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.1600000000000001,"5min":8.0790000000000006,"15min":20.263000000000002},"last":0.872}]},{"osd":4,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.6850000000000001,"15min":1.754},"min":{"1min":0.308,"5min":0.26400000000000001,"15min":0.22},"max":{"1min":12.656000000000001,"5min":12.656000000000001,"15min":12.656000000000001},"last":8.1620000000000008},{"interface":"front","average":{"1min":3.528,"5min":2.6309999999999998,"15min":1.913},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.24299999999999999},"max":{"1min":11.068,"5min":20.550000000000001,"15min":20.550000000000001},"last":0.26300000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:04:39 2024","interfaces":[{"interface":"back","average":{"1min":3,"5min":2.5819999999999999,"15min":1.6060000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.26400000000000001},"max":{"1min":12.121,"5min":12.281000000000001,"15min":12.281000000000001},"last":8.0760000000000005},{"interface":"front","average":{"1min":3.1720000000000002,"5min":2.802,"15min":1.728},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":11.087,"5min":14.323,"15min":22.035},"last":0.36699999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":4.0869999999999997,"5min":3.1920000000000002,"15min":1.8859999999999999},"min":{"1min":0.375,"5min":0.29499999999999998,"15min":0.27500000000000002},"max":{"1min":13.647,"5min":19.184999999999999,"15min":19.184999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":4.7640000000000002,"5min":3.1869999999999998,"15min":1.9450000000000001},"min":{"1min":0.38800000000000001,"5min":0.30499999999999999,"15min":0.26800000000000002},"max":{"1min":20.274000000000001,"5min":20.274000000000001,"15min":20.274000000000001},"last":8.1839999999999993}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1166999552,"data_stored":1166999552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":872153088,"data_stored":872153088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190185472,"data_stored":190185472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":725028864,"data_stored":725024792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1307578368,"data_stored":1307574296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1263341568,"data_stored":1263337496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:05:22.648 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T17:05:22.648 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.control pg_num 22 2024-02-12T17:05:22.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:05:22.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:05:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:05:22.777 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:22.813 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:05:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:22.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:05:22.918 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:05:22.989 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:23.038 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:05:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:23.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:21.203389+0000 mgr.y (mgr.24481) 597 : audit [DBG] from='client.35657 v1:172.21.15.33:0/4126012586' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:05:23.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:22.667280+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-12T17:05:23.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:22.669623+0000 mon.a (mon.0) 1515 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-12T17:05:23.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:05:23.154 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:05:23.157 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:23.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:21.203389+0000 mgr.y (mgr.24481) 597 : audit [DBG] from='client.35657 v1:172.21.15.33:0/4126012586' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:05:23.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:22.667280+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-12T17:05:23.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:22.669623+0000 mon.a (mon.0) 1515 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-12T17:05:23.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:21.203389+0000 mgr.y (mgr.24481) 597 : audit [DBG] from='client.35657 v1:172.21.15.33:0/4126012586' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:05:23.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:22.667280+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-12T17:05:23.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:22.669623+0000 mon.a (mon.0) 1515 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-12T17:05:23.178 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:23.188 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:23.217 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:05:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:23.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:05:23.326 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:05:23.381 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:23.420 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:05:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:23.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:05:23.529 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:23.633 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:23.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:23.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:23.927 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:23.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:23.947 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:22.666763+0000 mgr.y (mgr.24481) 598 : cluster [DBG] pgmap v808: 126 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 1.6 GiB data, 8.3 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 534 op/s; 20529/77973 objects misplaced (26.328%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:23.948 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:22.908220+0000 mon.a (mon.0) 1516 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-12T17:05:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:22.666763+0000 mgr.y (mgr.24481) 598 : cluster [DBG] pgmap v808: 126 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 1.6 GiB data, 8.3 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 534 op/s; 20529/77973 objects misplaced (26.328%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:22.908220+0000 mon.a (mon.0) 1516 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-12T17:05:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:22.908305+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e415: 8 total, 8 up, 5 in 2024-02-12T17:05:24.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:22.666763+0000 mgr.y (mgr.24481) 598 : cluster [DBG] pgmap v808: 126 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 1.6 GiB data, 8.3 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 534 op/s; 20529/77973 objects misplaced (26.328%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:24.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:22.908220+0000 mon.a (mon.0) 1516 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-12T17:05:24.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:22.908305+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e415: 8 total, 8 up, 5 in 2024-02-12T17:05:24.032 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:24.086 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:24.150 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:05:24.190 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:05:24.193 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:24.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:24.237 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:05:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:22.908305+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e415: 8 total, 8 up, 5 in 2024-02-12T17:05:24.269 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:05:24.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:24.312 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:05:24.343 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:05:24.354 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:24.374 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:05:24.393 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:24.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:24.417 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:05:24.431 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:05:24.432 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:05:24.432 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:05:24.432 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:05:24.432 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:05:24.432 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:05:24.432 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:05:24.432 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:05:24.500 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:24.568 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:24.745 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:05:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:05:25.140 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:25.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:23.919241+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e416: 8 total, 8 up, 5 in 2024-02-12T17:05:25.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:24.107542+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? v1:172.21.15.33:0/2724453823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:25.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:25.146 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:23.919241+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e416: 8 total, 8 up, 5 in 2024-02-12T17:05:25.146 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:24.107542+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? v1:172.21.15.33:0/2724453823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:25.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:23.919241+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e416: 8 total, 8 up, 5 in 2024-02-12T17:05:25.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:24.107542+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? v1:172.21.15.33:0/2724453823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:25.246 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:25.302 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:25.356 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:25.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:25.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:25.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:25.516 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:25.598 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:25.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:25.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:25.760 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:26.092 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:24.364669+0000 mgr.y (mgr.24481) 599 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:24.667286+0000 mgr.y (mgr.24481) 600 : cluster [DBG] pgmap v811: 126 pgs: 2 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 17312/66789 objects misplaced (25.920%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:24.968602+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e417: 8 total, 8 up, 5 in 2024-02-12T17:05:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:24.364669+0000 mgr.y (mgr.24481) 599 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:24.667286+0000 mgr.y (mgr.24481) 600 : cluster [DBG] pgmap v811: 126 pgs: 2 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 17312/66789 objects misplaced (25.920%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:26.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:24.968602+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e417: 8 total, 8 up, 5 in 2024-02-12T17:05:26.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:24.364669+0000 mgr.y (mgr.24481) 599 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:26.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:24.667286+0000 mgr.y (mgr.24481) 600 : cluster [DBG] pgmap v811: 126 pgs: 2 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 17312/66789 objects misplaced (25.920%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:26.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:24.968602+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e417: 8 total, 8 up, 5 in 2024-02-12T17:05:27.120 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:25.982946+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e418: 8 total, 8 up, 5 in 2024-02-12T17:05:27.120 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:26.674765+0000 mon.c (mon.2) 225 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-12T17:05:27.121 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:26.676439+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-12T17:05:27.121 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:25.982946+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e418: 8 total, 8 up, 5 in 2024-02-12T17:05:27.121 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:26.674765+0000 mon.c (mon.2) 225 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-12T17:05:27.121 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:26.676439+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-12T17:05:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:25.982946+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e418: 8 total, 8 up, 5 in 2024-02-12T17:05:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:26.674765+0000 mon.c (mon.2) 225 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-12T17:05:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:26.676439+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-12T17:05:27.251 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:27.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:27.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:27.363 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:27.426 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:27.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:05:27.993 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:26.667703+0000 mgr.y (mgr.24481) 601 : cluster [DBG] pgmap v814: 125 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 17312/66789 objects misplaced (25.920%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:27.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:27.361170+0000 mon.a (mon.0) 1522 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-12T17:05:27.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:27.361233+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e419: 8 total, 8 up, 5 in 2024-02-12T17:05:28.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:26.667703+0000 mgr.y (mgr.24481) 601 : cluster [DBG] pgmap v814: 125 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 17312/66789 objects misplaced (25.920%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:28.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:27.361170+0000 mon.a (mon.0) 1522 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-12T17:05:28.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:27.361233+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e419: 8 total, 8 up, 5 in 2024-02-12T17:05:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:26.667703+0000 mgr.y (mgr.24481) 601 : cluster [DBG] pgmap v814: 125 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 17312/66789 objects misplaced (25.920%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:27.361170+0000 mon.a (mon.0) 1522 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-12T17:05:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:27.361233+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e419: 8 total, 8 up, 5 in 2024-02-12T17:05:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:28.101481+0000 osd.0 (osd.0) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:28.101518+0000 osd.0 (osd.0) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:28.366660+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e420: 8 total, 8 up, 5 in 2024-02-12T17:05:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:28.668354+0000 mgr.y (mgr.24481) 602 : cluster [DBG] pgmap v817: 125 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 413 op/s; 16005/61824 objects misplaced (25.888%) 2024-02-12T17:05:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:28.101481+0000 osd.0 (osd.0) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:28.101518+0000 osd.0 (osd.0) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:28.366660+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e420: 8 total, 8 up, 5 in 2024-02-12T17:05:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:28.668354+0000 mgr.y (mgr.24481) 602 : cluster [DBG] pgmap v817: 125 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 413 op/s; 16005/61824 objects misplaced (25.888%) 2024-02-12T17:05:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:28.101481+0000 osd.0 (osd.0) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:28.101518+0000 osd.0 (osd.0) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:28.366660+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e420: 8 total, 8 up, 5 in 2024-02-12T17:05:29.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:28.668354+0000 mgr.y (mgr.24481) 602 : cluster [DBG] pgmap v817: 125 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.2 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 413 op/s; 16005/61824 objects misplaced (25.888%) 2024-02-12T17:05:30.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:29.393990+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e421: 8 total, 8 up, 5 in 2024-02-12T17:05:30.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:29.393990+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e421: 8 total, 8 up, 5 in 2024-02-12T17:05:30.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:29.393990+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e421: 8 total, 8 up, 5 in 2024-02-12T17:05:31.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:05:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:30] "GET /metrics HTTP/1.1" 200 176206 "" "Prometheus/2.33.4" 2024-02-12T17:05:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:30.238420+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:30.238476+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:30.398206+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e422: 8 total, 8 up, 5 in 2024-02-12T17:05:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:30.673703+0000 mgr.y (mgr.24481) 603 : cluster [DBG] pgmap v820: 124 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 1 clean+premerge+peered, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 1.1 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s; 14139/55230 objects misplaced (25.600%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:05:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:30.674031+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-12T17:05:31.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:30.675036+0000 mon.a (mon.0) 1527 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-12T17:05:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:30.238420+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:30.238476+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:30.398206+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e422: 8 total, 8 up, 5 in 2024-02-12T17:05:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:30.673703+0000 mgr.y (mgr.24481) 603 : cluster [DBG] pgmap v820: 124 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 1 clean+premerge+peered, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 1.1 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s; 14139/55230 objects misplaced (25.600%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:05:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:30.674031+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-12T17:05:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:30.675036+0000 mon.a (mon.0) 1527 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-12T17:05:31.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:30.238420+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:31.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:30.238476+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:31.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:30.398206+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e422: 8 total, 8 up, 5 in 2024-02-12T17:05:31.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:30.673703+0000 mgr.y (mgr.24481) 603 : cluster [DBG] pgmap v820: 124 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 1 clean+premerge+peered, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 1.1 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s; 14139/55230 objects misplaced (25.600%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:05:31.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:30.674031+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-12T17:05:31.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:30.675036+0000 mon.a (mon.0) 1527 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-12T17:05:32.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:31.496128+0000 mon.a (mon.0) 1528 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-12T17:05:32.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:31.496198+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e423: 8 total, 8 up, 5 in 2024-02-12T17:05:32.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:32.162348+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? v1:172.21.15.33:0/2258104617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:32.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:31.496128+0000 mon.a (mon.0) 1528 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-12T17:05:32.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:31.496198+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e423: 8 total, 8 up, 5 in 2024-02-12T17:05:32.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:32.162348+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? v1:172.21.15.33:0/2258104617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:32.819 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:05:32.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:05:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:05:32.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:31.496128+0000 mon.a (mon.0) 1528 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-12T17:05:32.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:31.496198+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e423: 8 total, 8 up, 5 in 2024-02-12T17:05:32.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:32.162348+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? v1:172.21.15.33:0/2258104617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:33.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:32.490340+0000 mon.a (mon.0) 1531 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:33.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:32.494180+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? v1:172.21.15.33:0/2258104617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:33.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:32.494243+0000 mon.a (mon.0) 1533 : cluster [DBG] osdmap e424: 8 total, 8 up, 5 in 2024-02-12T17:05:33.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:32.674219+0000 mgr.y (mgr.24481) 604 : cluster [DBG] pgmap v823: 124 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 1 clean+premerge+peered, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 1.1 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 549 op/s; 14139/55230 objects misplaced (25.600%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:05:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:32.490340+0000 mon.a (mon.0) 1531 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:33.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:32.494180+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? v1:172.21.15.33:0/2258104617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:33.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:32.494243+0000 mon.a (mon.0) 1533 : cluster [DBG] osdmap e424: 8 total, 8 up, 5 in 2024-02-12T17:05:33.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:32.674219+0000 mgr.y (mgr.24481) 604 : cluster [DBG] pgmap v823: 124 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 1 clean+premerge+peered, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 1.1 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 549 op/s; 14139/55230 objects misplaced (25.600%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:05:33.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:32.490340+0000 mon.a (mon.0) 1531 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:33.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:32.494180+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? v1:172.21.15.33:0/2258104617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:33.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:32.494243+0000 mon.a (mon.0) 1533 : cluster [DBG] osdmap e424: 8 total, 8 up, 5 in 2024-02-12T17:05:33.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:32.674219+0000 mgr.y (mgr.24481) 604 : cluster [DBG] pgmap v823: 124 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 1 clean+premerge+peered, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 1.1 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 549 op/s; 14139/55230 objects misplaced (25.600%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T17:05:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:34.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:34.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:34.414 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:34.617 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:34.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:34.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:33.512691+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e425: 8 total, 8 up, 5 in 2024-02-12T17:05:34.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:05:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:05:34.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:33.512691+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e425: 8 total, 8 up, 5 in 2024-02-12T17:05:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:33.512691+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e425: 8 total, 8 up, 5 in 2024-02-12T17:05:34.799 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:34.843 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:34.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:34.954 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:35.025 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:35.397 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:35.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:35.504 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:35.551 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:34.374890+0000 mgr.y (mgr.24481) 605 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:35.551 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:34.518228+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e426: 8 total, 8 up, 5 in 2024-02-12T17:05:35.552 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:34.674694+0000 mgr.y (mgr.24481) 606 : cluster [DBG] pgmap v826: 123 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 897 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 10965/43635 objects misplaced (25.129%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:35.552 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:34.679032+0000 mon.c (mon.2) 227 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:35.552 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:34.680589+0000 mon.a (mon.0) 1536 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:35.552 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:34.374890+0000 mgr.y (mgr.24481) 605 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:35.552 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:34.518228+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e426: 8 total, 8 up, 5 in 2024-02-12T17:05:35.552 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:34.674694+0000 mgr.y (mgr.24481) 606 : cluster [DBG] pgmap v826: 123 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 897 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 10965/43635 objects misplaced (25.129%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:35.552 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:34.679032+0000 mon.c (mon.2) 227 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:35.552 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:34.680589+0000 mon.a (mon.0) 1536 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:35.580 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:34.374890+0000 mgr.y (mgr.24481) 605 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:34.518228+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e426: 8 total, 8 up, 5 in 2024-02-12T17:05:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:34.674694+0000 mgr.y (mgr.24481) 606 : cluster [DBG] pgmap v826: 123 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 897 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 10965/43635 objects misplaced (25.129%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:34.679032+0000 mon.c (mon.2) 227 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:34.680589+0000 mon.a (mon.0) 1536 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:36.009 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:36.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:35.545463+0000 mon.a (mon.0) 1537 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-12T17:05:36.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:35.545533+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e427: 8 total, 8 up, 5 in 2024-02-12T17:05:36.785 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:36.334396+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e428: 8 total, 8 up, 5 in 2024-02-12T17:05:36.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:35.545463+0000 mon.a (mon.0) 1537 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-12T17:05:36.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:35.545533+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e427: 8 total, 8 up, 5 in 2024-02-12T17:05:36.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:36.334396+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e428: 8 total, 8 up, 5 in 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.15459.0:263218 9.8 e428/348 9:1dc53c1c:::benchmark_data_smithi033_131576_object125628:head v 428'16419, mlcod=428'16417)", 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:36.763213+0000", 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.020400279, 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020431436000000001, 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:05:36.835 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.763213+0000", 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.763213+0000", 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.763213+0000", 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6899999999999997e-06 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.763218+0000", 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5990000000000001e-06 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.763224+0000", 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1300000000000003e-06 2024-02-12T17:05:36.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.763228+0000", 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014281499999999999 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.763371+0000", 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6294e-05 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.763387+0000", 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000173955 2024-02-12T17:05:36.838 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.15459.0:263223 9.8 e428/348 9:1aa871f7:::benchmark_data_smithi033_131576_object125633:head v 428'16420, mlcod=428'16417)", 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:36.775131+0000", 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0084825649999999992, 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085988650000000007, 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:05:36.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775131+0000", 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775131+0000", 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775130+0000", 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7980000000000002e-06 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775136+0000", 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1090000000000003e-06 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775142+0000", 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2320000000000003e-06 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775146+0000", 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1618000000000003e-05 2024-02-12T17:05:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775238+0000", 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2317999999999999e-05 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775260+0000", 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012942100000000001 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.15459.0:263233 9.8 e428/348 9:1558920c:::benchmark_data_smithi033_131576_object125643:head v 428'16421, mlcod=428'16417)", 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:36.775157+0000", 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0084567989999999992, 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086458490000000006, 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775157+0000", 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:05:36.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775157+0000", 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775156+0000", 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5160000000000002e-06 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775162+0000", 2024-02-12T17:05:36.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3380000000000004e-06 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775167+0000", 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9780000000000005e-06 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.775175+0000", 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004733396 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:05:36.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.779908+0000", 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1617999999999999e-05 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.779930+0000", 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047732790000000001 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "pg_backfill(progress 9.8 e 428/428 lb 9:160852c8:::benchmark_data_smithi033_131576_object127930:head)", 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:36.783632+0000", 2024-02-12T17:05:36.846 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4294967295.9999814, 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023984500000000001, 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783632+0000", 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783632+0000", 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:05:36.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783631+0000", 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1749000000000001e-05 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783642+0000", 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2395000000000001e-05 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783655+0000", 2024-02-12T17:05:36.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6089999999999997e-06 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783661+0000", 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9159e-05 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(9.8 428/348 [PushOp(9:1609ac33:::benchmark_data_smithi033_131576_object128796:head, version: 333'8027, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:1609ac33:::benchmark_data_smithi033_131576_object128796:head@333'8027, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:36.783677+0000", 2024-02-12T17:05:36.849 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4294967295.9999366, 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027678700000000002, 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783677+0000", 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783677+0000", 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:05:36.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783676+0000", 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0514e-05 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783707+0000", 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5554999999999999e-05 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783722+0000", 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0332e-05 2024-02-12T17:05:36.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:36.783733+0000", 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5708999999999998e-05 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 5 2024-02-12T17:05:36.852 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:05:36.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:35.545463+0000 mon.a (mon.0) 1537 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-12T17:05:36.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:35.545533+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e427: 8 total, 8 up, 5 in 2024-02-12T17:05:36.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:36.334396+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e428: 8 total, 8 up, 5 in 2024-02-12T17:05:37.294 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:05:37.423 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 5 pg_num to 22 2024-02-12T17:05:37.774 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:37.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:37.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:36.675219+0000 mgr.y (mgr.24481) 607 : cluster [DBG] pgmap v829: 123 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 897 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 10965/43635 objects misplaced (25.129%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:37.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:37.155923+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.33:0/1048907176' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:05:37.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:37.370417+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? v1:172.21.15.33:0/1048907176' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:05:37.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:37.370475+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e429: 8 total, 8 up, 5 in 2024-02-12T17:05:37.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:36.675219+0000 mgr.y (mgr.24481) 607 : cluster [DBG] pgmap v829: 123 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 897 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 10965/43635 objects misplaced (25.129%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:37.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:37.155923+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.33:0/1048907176' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:05:37.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:37.370417+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? v1:172.21.15.33:0/1048907176' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:05:37.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:37.370475+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e429: 8 total, 8 up, 5 in 2024-02-12T17:05:37.882 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:37.950 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:37.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:36.675219+0000 mgr.y (mgr.24481) 607 : cluster [DBG] pgmap v829: 123 pgs: 1 peering, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 897 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 10965/43635 objects misplaced (25.129%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:37.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:37.155923+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.33:0/1048907176' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:05:37.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:37.370417+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? v1:172.21.15.33:0/1048907176' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:05:37.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:37.370475+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e429: 8 total, 8 up, 5 in 2024-02-12T17:05:38.778 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T17:05:39.016 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:39.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:39.119 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:39.167 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:39.234 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:39.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:39.340 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:39.420 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:39.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:38.373157+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e430: 8 total, 8 up, 5 in 2024-02-12T17:05:39.496 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:38.676766+0000 mgr.y (mgr.24481) 608 : cluster [DBG] pgmap v832: 122 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 793 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 414 op/s; 9604/38664 objects misplaced (24.840%) 2024-02-12T17:05:39.496 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:38.678395+0000 mon.c (mon.2) 228 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:39.496 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:38.678535+0000 mon.c (mon.2) 229 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:39.496 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:38.679796+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:39.496 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:38.680058+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:38.373157+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e430: 8 total, 8 up, 5 in 2024-02-12T17:05:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:38.676766+0000 mgr.y (mgr.24481) 608 : cluster [DBG] pgmap v832: 122 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 793 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 414 op/s; 9604/38664 objects misplaced (24.840%) 2024-02-12T17:05:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:38.678395+0000 mon.c (mon.2) 228 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:38.678535+0000 mon.c (mon.2) 229 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:38.679796+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:38.680058+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:38.373157+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e430: 8 total, 8 up, 5 in 2024-02-12T17:05:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:38.676766+0000 mgr.y (mgr.24481) 608 : cluster [DBG] pgmap v832: 122 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 793 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 414 op/s; 9604/38664 objects misplaced (24.840%) 2024-02-12T17:05:39.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:38.678395+0000 mon.c (mon.2) 228 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:39.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:38.678535+0000 mon.c (mon.2) 229 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:39.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:38.679796+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:39.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:38.680058+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-02-12T17:05:39.583 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:39.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:39.687 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:05:39.787 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:39.823 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:05:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:39.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:05:39.930 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:39.989 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:40.206 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:40.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:40.286 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:40.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:40.382 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:40.527 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:40.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:40.542 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:40.639 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:39.393657+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:05:40.639 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:39.393739+0000 mon.a (mon.0) 1547 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]': finished 2024-02-12T17:05:40.640 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:39.393802+0000 mon.a (mon.0) 1548 : cluster [DBG] osdmap e431: 8 total, 8 up, 5 in 2024-02-12T17:05:40.640 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:39.766914+0000 osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:40.640 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:39.766963+0000 osd.5 (osd.5) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:40.640 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:39.393657+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:05:40.640 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:39.393739+0000 mon.a (mon.0) 1547 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]': finished 2024-02-12T17:05:40.640 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:39.393802+0000 mon.a (mon.0) 1548 : cluster [DBG] osdmap e431: 8 total, 8 up, 5 in 2024-02-12T17:05:40.640 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:39.766914+0000 osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:40.640 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:39.766963+0000 osd.5 (osd.5) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:40.640 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:40.733 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:39.393657+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:05:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:39.393739+0000 mon.a (mon.0) 1547 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "16"}]': finished 2024-02-12T17:05:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:39.393802+0000 mon.a (mon.0) 1548 : cluster [DBG] osdmap e431: 8 total, 8 up, 5 in 2024-02-12T17:05:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:39.766914+0000 osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:40.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:39.766963+0000 osd.5 (osd.5) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:40.780 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:40.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:40.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:40.962 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:41.029 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:05:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:40] "GET /metrics HTTP/1.1" 200 176221 "" "Prometheus/2.33.4" 2024-02-12T17:05:41.305 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:05:41.305 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:05:41.305 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:05:41.307 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:05:41.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:05:41.405 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:05:41.424 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:41.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:41.437 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:05:41.480 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:05:41.492 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:05:41.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:05:41.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:05:41.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:05:41.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:05:41.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:05:41.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:05:41.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:05:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:40.404535+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e432: 8 total, 8 up, 5 in 2024-02-12T17:05:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:40.678082+0000 mgr.y (mgr.24481) 609 : cluster [DBG] pgmap v835: 128 pgs: 6 unknown, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 665 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s; 7882/32484 objects misplaced (24.264%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:40.678472+0000 mon.c (mon.2) 230 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:40.680180+0000 mon.a (mon.0) 1550 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:41.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:41.126345+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.33:0/82436760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:41.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:40.404535+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e432: 8 total, 8 up, 5 in 2024-02-12T17:05:41.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:40.678082+0000 mgr.y (mgr.24481) 609 : cluster [DBG] pgmap v835: 128 pgs: 6 unknown, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 665 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s; 7882/32484 objects misplaced (24.264%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:41.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:40.678472+0000 mon.c (mon.2) 230 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:41.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:40.680180+0000 mon.a (mon.0) 1550 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:41.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:41.126345+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.33:0/82436760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:41.531 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:41.609 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:41.751 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:40.404535+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e432: 8 total, 8 up, 5 in 2024-02-12T17:05:41.751 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:40.678082+0000 mgr.y (mgr.24481) 609 : cluster [DBG] pgmap v835: 128 pgs: 6 unknown, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 665 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s; 7882/32484 objects misplaced (24.264%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:41.751 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:40.678472+0000 mon.c (mon.2) 230 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:41.751 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:40.680180+0000 mon.a (mon.0) 1550 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:05:41.751 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:41.126345+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.33:0/82436760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:42.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:41.459949+0000 mon.a (mon.0) 1551 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:05:42.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:41.460008+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e433: 8 total, 8 up, 5 in 2024-02-12T17:05:42.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:41.470484+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:42.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:41.473637+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:42.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:41.473726+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:42.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:05:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:05:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:41.459949+0000 mon.a (mon.0) 1551 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:05:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:41.460008+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e433: 8 total, 8 up, 5 in 2024-02-12T17:05:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:41.470484+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:41.473637+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:41.473726+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:41.459949+0000 mon.a (mon.0) 1551 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:05:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:41.460008+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e433: 8 total, 8 up, 5 in 2024-02-12T17:05:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:41.470484+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:41.473637+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:42.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:41.473726+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 406'18 2024-02-12T17:05:43.360 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:05:43.360 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:05:43.360 INFO:tasks.thrashosds.thrasher:Shrinking pool iscsi 2024-02-12T17:05:43.360 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool iscsi by 10 min 8 2024-02-12T17:05:43.360 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:05:43.581 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:43.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:43.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:43.736 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:42.488310+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e434: 8 total, 8 up, 5 in 2024-02-12T17:05:43.736 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:42.678574+0000 mgr.y (mgr.24481) 610 : cluster [DBG] pgmap v838: 128 pgs: 6 unknown, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 665 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 516 op/s; 7882/32484 objects misplaced (24.264%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:43.736 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:42.488310+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e434: 8 total, 8 up, 5 in 2024-02-12T17:05:43.736 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:42.678574+0000 mgr.y (mgr.24481) 610 : cluster [DBG] pgmap v838: 128 pgs: 6 unknown, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 665 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 516 op/s; 7882/32484 objects misplaced (24.264%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:43.741 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:43.951 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:43.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:43.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:42.488310+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e434: 8 total, 8 up, 5 in 2024-02-12T17:05:43.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:42.678574+0000 mgr.y (mgr.24481) 610 : cluster [DBG] pgmap v838: 128 pgs: 6 unknown, 3 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 108 active+clean; 665 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 516 op/s; 7882/32484 objects misplaced (24.264%); 3.6 MiB/s, 56 objects/s recovering 2024-02-12T17:05:44.068 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:44.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:43.511744+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e435: 8 total, 8 up, 5 in 2024-02-12T17:05:44.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:43.538141+0000 osd.2 (osd.2) 165 : cluster [DBG] 5.14 starting backfill to osd.3 from (0'0,0'0] MAX to 414'14 2024-02-12T17:05:44.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:43.538665+0000 osd.2 (osd.2) 166 : cluster [DBG] 5.15 starting backfill to osd.5 from (0'0,0'0] MAX to 410'15 2024-02-12T17:05:44.540 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:43.538764+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.10 starting backfill to osd.7 from (0'0,0'0] MAX to 430'18 2024-02-12T17:05:44.713 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:05:44.713 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:05:44.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:43.511744+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e435: 8 total, 8 up, 5 in 2024-02-12T17:05:44.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:43.538141+0000 osd.2 (osd.2) 165 : cluster [DBG] 5.14 starting backfill to osd.3 from (0'0,0'0] MAX to 414'14 2024-02-12T17:05:44.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:43.538665+0000 osd.2 (osd.2) 166 : cluster [DBG] 5.15 starting backfill to osd.5 from (0'0,0'0] MAX to 410'15 2024-02-12T17:05:44.744 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:43.538764+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.10 starting backfill to osd.7 from (0'0,0'0] MAX to 430'18 2024-02-12T17:05:44.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:05:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:05:45.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:43.511744+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e435: 8 total, 8 up, 5 in 2024-02-12T17:05:45.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:43.538141+0000 osd.2 (osd.2) 165 : cluster [DBG] 5.14 starting backfill to osd.3 from (0'0,0'0] MAX to 414'14 2024-02-12T17:05:45.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:43.538665+0000 osd.2 (osd.2) 166 : cluster [DBG] 5.15 starting backfill to osd.5 from (0'0,0'0] MAX to 410'15 2024-02-12T17:05:45.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:43.538764+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.10 starting backfill to osd.7 from (0'0,0'0] MAX to 430'18 2024-02-12T17:05:45.486 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T17:05:45.617 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:05:45.663 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:44.391416+0000 mgr.y (mgr.24481) 611 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:44.559269+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e436: 8 total, 8 up, 5 in 2024-02-12T17:05:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:44.679740+0000 mgr.y (mgr.24481) 612 : cluster [DBG] pgmap v841: 128 pgs: 3 remapped+peering, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 435 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 918 op/s; 4958/21477 objects misplaced (23.085%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:44.853002+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.33:0/208031539' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:05:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:44.855444+0000 mon.a (mon.0) 1556 : audit [INF] from='client.35678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:05:45.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:44.391416+0000 mgr.y (mgr.24481) 611 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:45.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:44.559269+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e436: 8 total, 8 up, 5 in 2024-02-12T17:05:45.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:44.679740+0000 mgr.y (mgr.24481) 612 : cluster [DBG] pgmap v841: 128 pgs: 3 remapped+peering, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 435 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 918 op/s; 4958/21477 objects misplaced (23.085%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:45.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:44.853002+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.33:0/208031539' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:05:45.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:44.855444+0000 mon.a (mon.0) 1556 : audit [INF] from='client.35678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:05:45.858 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:45.902 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:44.391416+0000 mgr.y (mgr.24481) 611 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:45.902 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:44.559269+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e436: 8 total, 8 up, 5 in 2024-02-12T17:05:45.902 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:44.679740+0000 mgr.y (mgr.24481) 612 : cluster [DBG] pgmap v841: 128 pgs: 3 remapped+peering, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 435 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 918 op/s; 4958/21477 objects misplaced (23.085%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:45.902 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:44.853002+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.33:0/208031539' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:05:45.902 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:44.855444+0000 mon.a (mon.0) 1556 : audit [INF] from='client.35678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:05:45.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:46.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:46.055 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:46.092 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:46.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:46.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:05:46.274 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:46.312 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:05:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:46.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:05:46.419 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:46.477 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:45.552515+0000 mon.a (mon.0) 1557 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:45.552827+0000 mon.a (mon.0) 1558 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:45.576423+0000 mon.a (mon.0) 1559 : audit [INF] from='client.35678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:45.576481+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e437: 8 total, 8 up, 5 in 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:46.333707+0000 mon.a (mon.0) 1561 : cluster [DBG] osdmap e438: 8 total, 8 up, 5 in 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:45.552515+0000 mon.a (mon.0) 1557 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:45.552827+0000 mon.a (mon.0) 1558 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:45.576423+0000 mon.a (mon.0) 1559 : audit [INF] from='client.35678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:45.576481+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e437: 8 total, 8 up, 5 in 2024-02-12T17:05:46.576 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:46.333707+0000 mon.a (mon.0) 1561 : cluster [DBG] osdmap e438: 8 total, 8 up, 5 in 2024-02-12T17:05:46.701 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:46.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:46.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:46.836 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:46.846 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:46.849 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:46.898 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:46.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:46.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:45.552515+0000 mon.a (mon.0) 1557 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-12T17:05:46.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:45.552827+0000 mon.a (mon.0) 1558 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:05:46.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:45.576423+0000 mon.a (mon.0) 1559 : audit [INF] from='client.35678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:05:46.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:45.576481+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e437: 8 total, 8 up, 5 in 2024-02-12T17:05:46.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:46.333707+0000 mon.a (mon.0) 1561 : cluster [DBG] osdmap e438: 8 total, 8 up, 5 in 2024-02-12T17:05:47.007 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:47.069 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:47.284 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:47.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:47.394 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:47.479 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:47.596 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:47.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:47.704 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:47.758 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:47.939 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:05:47.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:46.681650+0000 mgr.y (mgr.24481) 613 : cluster [DBG] pgmap v844: 128 pgs: 3 remapped+peering, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 435 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 4958/21477 objects misplaced (23.085%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:47.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:47.515624+0000 mon.a (mon.0) 1562 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:47.992 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:46.681650+0000 mgr.y (mgr.24481) 613 : cluster [DBG] pgmap v844: 128 pgs: 3 remapped+peering, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 435 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 4958/21477 objects misplaced (23.085%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:47.992 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:47.515624+0000 mon.a (mon.0) 1562 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:48.320 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:46.681650+0000 mgr.y (mgr.24481) 613 : cluster [DBG] pgmap v844: 128 pgs: 3 remapped+peering, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 4 active+remapped+backfilling, 110 active+clean; 435 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 4958/21477 objects misplaced (23.085%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T17:05:48.320 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:47.515624+0000 mon.a (mon.0) 1562 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:05:48.321 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 17:05:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T17:05:47+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-02-12T17:05:48.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:05:49.055 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:49.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:49.162 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:49.221 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:49.583 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:49.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:49.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:05:49.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:48.682451+0000 mgr.y (mgr.24481) 614 : cluster [DBG] pgmap v845: 128 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 334 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 992 B/s rd, 0 B/s wr, 758 op/s; 3641/16626 objects misplaced (21.899%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:05:49.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:48.682451+0000 mgr.y (mgr.24481) 614 : cluster [DBG] pgmap v845: 128 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 334 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 992 B/s rd, 0 B/s wr, 758 op/s; 3641/16626 objects misplaced (21.899%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:05:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:49.894 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:05:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:49.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:05:50.004 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:50.066 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:50.127 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:50.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:50.231 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:50.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:48.682451+0000 mgr.y (mgr.24481) 614 : cluster [DBG] pgmap v845: 128 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 112 active+clean; 334 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 992 B/s rd, 0 B/s wr, 758 op/s; 3641/16626 objects misplaced (21.899%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:05:50.303 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:50.322 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:50.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:50.429 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:50.493 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:50.557 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:05:50.557 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:05:50.797 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:05:50.804 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:05:50.831 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:50.832 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:51.080 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:51.085 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:50.757029+0000 mon.a (mon.0) 1563 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-12T17:05:51.086 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:05:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:50] "GET /metrics HTTP/1.1" 200 176300 "" "Prometheus/2.33.4" 2024-02-12T17:05:51.086 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:50.757029+0000 mon.a (mon.0) 1563 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-12T17:05:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:51.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:51.188 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:51.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:50.757029+0000 mon.a (mon.0) 1563 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-12T17:05:51.255 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:51.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T17:05:51.819 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:51.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:51.927 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:51.971 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:50.683120+0000 mgr.y (mgr.24481) 615 : cluster [DBG] pgmap v846: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 204 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 836 B/s rd, 0 B/s wr, 606 op/s; 2154/10368 objects misplaced (20.775%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:51.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:50.766236+0000 mgr.y (mgr.24481) 616 : audit [DBG] from='client.15729 v1:172.21.15.33:0/339688277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:05:52.068 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:52.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:52.082 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:50.683120+0000 mgr.y (mgr.24481) 615 : cluster [DBG] pgmap v846: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 204 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 836 B/s rd, 0 B/s wr, 606 op/s; 2154/10368 objects misplaced (20.775%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:52.082 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:50.766236+0000 mgr.y (mgr.24481) 616 : audit [DBG] from='client.15729 v1:172.21.15.33:0/339688277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:05:52.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:50.683120+0000 mgr.y (mgr.24481) 615 : cluster [DBG] pgmap v846: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 204 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 836 B/s rd, 0 B/s wr, 606 op/s; 2154/10368 objects misplaced (20.775%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:05:52.082 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:50.766236+0000 mgr.y (mgr.24481) 616 : audit [DBG] from='client.15729 v1:172.21.15.33:0/339688277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:05:52.173 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:52.228 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:52.585 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":846,"stamp":"2024-02-12T17:05:50.682652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213849788,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2154,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":1393,"num_read_kb":1145,"num_write":272626,"num_write_kb":8805610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16159,"num_bytes_recovered":1013792206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82899,"ondisk_log_size":82899,"up":384,"acting":384,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":401,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":4659856,"kb_used_data":769256,"kb_used_omap":0,"kb_used_meta":3890496,"kb_avail":464065904,"statfs":{"total":479975178240,"available":475203485696,"internally_reserved":0,"allocated":787718144,"data_stored":784112913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3983867904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-242679808,"num_objects":-3703,"num_object_clones":0,"num_object_copies":-11109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2804,"num_objects_unfound":0,"num_objects_dirty":-3703,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3703,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15269888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.117248"},"pg_stats":[{"pgid":"2.1f","version":"0'0","reported_seq":127,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.584979+0000","last_change":"2024-02-12T17:04:39.916313+0000","last_active":"2024-02-12T17:05:45.584979+0000","last_peered":"2024-02-12T17:05:45.584979+0000","last_clean":"2024-02-12T17:05:45.584979+0000","last_became_active":"2024-02-12T17:04:39.916010+0000","last_became_peered":"2024-02-12T17:04:39.916010+0000","last_unstale":"2024-02-12T17:05:45.584979+0000","last_undegraded":"2024-02-12T17:05:45.584979+0000","last_fullsized":"2024-02-12T17:05:45.584979+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.732293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.732293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":116,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589331+0000","last_change":"2024-02-12T17:03:52.280899+0000","last_active":"2024-02-12T17:05:45.589331+0000","last_peered":"2024-02-12T17:05:45.589331+0000","last_clean":"2024-02-12T17:05:45.589331+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:05:45.589331+0000","last_undegraded":"2024-02-12T17:05:45.589331+0000","last_fullsized":"2024-02-12T17:05:45.589331+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:01:26.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.047527+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":100,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.595719+0000","last_change":"2024-02-12T17:03:50.545988+0000","last_active":"2024-02-12T17:05:45.595719+0000","last_peered":"2024-02-12T17:05:45.595719+0000","last_clean":"2024-02-12T17:05:45.595719+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:05:45.595719+0000","last_undegraded":"2024-02-12T17:05:45.595719+0000","last_fullsized":"2024-02-12T17:05:45.595719+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.176431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":491,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589199+0000","last_change":"2024-02-12T17:04:41.353781+0000","last_active":"2024-02-12T17:05:45.589199+0000","last_peered":"2024-02-12T17:05:45.589199+0000","last_clean":"2024-02-12T17:05:45.589199+0000","last_became_active":"2024-02-12T17:04:39.873280+0000","last_became_peered":"2024-02-12T17:04:39.873280+0000","last_unstale":"2024-02-12T17:05:45.589199+0000","last_undegraded":"2024-02-12T17:05:45.589199+0000","last_fullsized":"2024-02-12T17:05:45.589199+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:01:45.154164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":433,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.587689+0000","last_change":"2024-02-12T17:03:50.565273+0000","last_active":"2024-02-12T17:05:45.587689+0000","last_peered":"2024-02-12T17:05:45.587689+0000","last_clean":"2024-02-12T17:05:45.587689+0000","last_became_active":"2024-02-12T17:03:50.562938+0000","last_became_peered":"2024-02-12T17:03:50.562938+0000","last_unstale":"2024-02-12T17:05:45.587689+0000","last_undegraded":"2024-02-12T17:05:45.587689+0000","last_fullsized":"2024-02-12T17:05:45.587689+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:37.813008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:01:37.813008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":121,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.597794+0000","last_change":"2024-02-12T17:04:42.260018+0000","last_active":"2024-02-12T17:05:45.597794+0000","last_peered":"2024-02-12T17:05:45.597794+0000","last_clean":"2024-02-12T17:05:45.597794+0000","last_became_active":"2024-02-12T17:04:39.889725+0000","last_became_peered":"2024-02-12T17:04:39.889725+0000","last_unstale":"2024-02-12T17:05:45.597794+0000","last_undegraded":"2024-02-12T17:05:45.597794+0000","last_fullsized":"2024-02-12T17:05:45.597794+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T16:57:24.413375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.413375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":407,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.583429+0000","last_change":"2024-02-12T17:02:34.502434+0000","last_active":"2024-02-12T17:05:45.583429+0000","last_peered":"2024-02-12T17:05:45.583429+0000","last_clean":"2024-02-12T17:05:45.583429+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:05:45.583429+0000","last_undegraded":"2024-02-12T17:05:45.583429+0000","last_fullsized":"2024-02-12T17:05:45.583429+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.502332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":129,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.597612+0000","last_change":"2024-02-12T17:04:40.923654+0000","last_active":"2024-02-12T17:05:45.597612+0000","last_peered":"2024-02-12T17:05:45.597612+0000","last_clean":"2024-02-12T17:05:45.597612+0000","last_became_active":"2024-02-12T17:04:39.865216+0000","last_became_peered":"2024-02-12T17:04:39.865216+0000","last_unstale":"2024-02-12T17:05:45.597612+0000","last_undegraded":"2024-02-12T17:05:45.597612+0000","last_fullsized":"2024-02-12T17:05:45.597612+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":109,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.592059+0000","last_change":"2024-02-12T17:04:39.890400+0000","last_active":"2024-02-12T17:05:45.592059+0000","last_peered":"2024-02-12T17:05:45.592059+0000","last_clean":"2024-02-12T17:05:45.592059+0000","last_became_active":"2024-02-12T17:04:39.887235+0000","last_became_peered":"2024-02-12T17:04:39.887235+0000","last_unstale":"2024-02-12T17:05:45.592059+0000","last_undegraded":"2024-02-12T17:05:45.592059+0000","last_fullsized":"2024-02-12T17:05:45.592059+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.724890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":133,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591353+0000","last_change":"2024-02-12T17:04:40.319567+0000","last_active":"2024-02-12T17:05:45.591353+0000","last_peered":"2024-02-12T17:05:45.591353+0000","last_clean":"2024-02-12T17:05:45.591353+0000","last_became_active":"2024-02-12T17:04:39.864612+0000","last_became_peered":"2024-02-12T17:04:39.864612+0000","last_unstale":"2024-02-12T17:05:45.591353+0000","last_undegraded":"2024-02-12T17:05:45.591353+0000","last_fullsized":"2024-02-12T17:05:45.591353+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":75,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582322+0000","last_change":"2024-02-12T17:04:39.888727+0000","last_active":"2024-02-12T17:05:45.582322+0000","last_peered":"2024-02-12T17:05:45.582322+0000","last_clean":"2024-02-12T17:05:45.582322+0000","last_became_active":"2024-02-12T17:04:39.888398+0000","last_became_peered":"2024-02-12T17:04:39.888398+0000","last_unstale":"2024-02-12T17:05:45.582322+0000","last_undegraded":"2024-02-12T17:05:45.582322+0000","last_fullsized":"2024-02-12T17:05:45.582322+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.157357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.157357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":141,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591218+0000","last_change":"2024-02-12T17:04:41.433996+0000","last_active":"2024-02-12T17:05:45.591218+0000","last_peered":"2024-02-12T17:05:45.591218+0000","last_clean":"2024-02-12T17:05:45.591218+0000","last_became_active":"2024-02-12T17:04:39.872998+0000","last_became_peered":"2024-02-12T17:04:39.872998+0000","last_unstale":"2024-02-12T17:05:45.591218+0000","last_undegraded":"2024-02-12T17:05:45.591218+0000","last_fullsized":"2024-02-12T17:05:45.591218+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":100,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589891+0000","last_change":"2024-02-12T17:03:50.533143+0000","last_active":"2024-02-12T17:05:45.589891+0000","last_peered":"2024-02-12T17:05:45.589891+0000","last_clean":"2024-02-12T17:05:45.589891+0000","last_became_active":"2024-02-12T17:03:50.532233+0000","last_became_peered":"2024-02-12T17:03:50.532233+0000","last_unstale":"2024-02-12T17:05:45.589891+0000","last_undegraded":"2024-02-12T17:05:45.589891+0000","last_fullsized":"2024-02-12T17:05:45.589891+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":294,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582237+0000","last_change":"2024-02-12T17:03:50.670847+0000","last_active":"2024-02-12T17:05:45.582237+0000","last_peered":"2024-02-12T17:05:45.582237+0000","last_clean":"2024-02-12T17:05:45.582237+0000","last_became_active":"2024-02-12T17:03:50.570118+0000","last_became_peered":"2024-02-12T17:03:50.570118+0000","last_unstale":"2024-02-12T17:05:45.582237+0000","last_undegraded":"2024-02-12T17:05:45.582237+0000","last_fullsized":"2024-02-12T17:05:45.582237+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:34.738385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.738385+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":413,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581786+0000","last_change":"2024-02-12T17:04:39.880788+0000","last_active":"2024-02-12T17:05:45.581786+0000","last_peered":"2024-02-12T17:05:45.581786+0000","last_clean":"2024-02-12T17:05:45.581786+0000","last_became_active":"2024-02-12T17:04:39.880094+0000","last_became_peered":"2024-02-12T17:04:39.880094+0000","last_unstale":"2024-02-12T17:05:45.581786+0000","last_undegraded":"2024-02-12T17:05:45.581786+0000","last_fullsized":"2024-02-12T17:05:45.581786+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.790302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.790302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":452,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.585548+0000","last_change":"2024-02-12T17:03:51.528465+0000","last_active":"2024-02-12T17:05:45.585548+0000","last_peered":"2024-02-12T17:05:45.585548+0000","last_clean":"2024-02-12T17:05:45.585548+0000","last_became_active":"2024-02-12T17:03:50.576714+0000","last_became_peered":"2024-02-12T17:03:50.576714+0000","last_unstale":"2024-02-12T17:05:45.585548+0000","last_undegraded":"2024-02-12T17:05:45.585548+0000","last_fullsized":"2024-02-12T17:05:45.585548+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:01:42.076584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.076584+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":96,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.634469+0000","last_change":"2024-02-12T17:05:45.618872+0000","last_active":"2024-02-12T17:05:45.634469+0000","last_peered":"2024-02-12T17:05:45.634469+0000","last_clean":"2024-02-12T17:05:45.634469+0000","last_became_active":"2024-02-12T17:05:45.618551+0000","last_became_peered":"2024-02-12T17:05:45.618551+0000","last_unstale":"2024-02-12T17:05:45.634469+0000","last_undegraded":"2024-02-12T17:05:45.634469+0000","last_fullsized":"2024-02-12T17:05:45.634469+0000","mapping_epoch":436,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":109,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.597871+0000","last_change":"2024-02-12T17:04:39.877511+0000","last_active":"2024-02-12T17:05:45.597871+0000","last_peered":"2024-02-12T17:05:45.597871+0000","last_clean":"2024-02-12T17:05:45.597871+0000","last_became_active":"2024-02-12T17:04:39.877005+0000","last_became_peered":"2024-02-12T17:04:39.877005+0000","last_unstale":"2024-02-12T17:05:45.597871+0000","last_undegraded":"2024-02-12T17:05:45.597871+0000","last_fullsized":"2024-02-12T17:05:45.597871+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:31.978839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:31.978839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":210,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582121+0000","last_change":"2024-02-12T17:04:39.885628+0000","last_active":"2024-02-12T17:05:45.582121+0000","last_peered":"2024-02-12T17:05:45.582121+0000","last_clean":"2024-02-12T17:05:45.582121+0000","last_became_active":"2024-02-12T17:04:39.884769+0000","last_became_peered":"2024-02-12T17:04:39.884769+0000","last_unstale":"2024-02-12T17:05:45.582121+0000","last_undegraded":"2024-02-12T17:05:45.582121+0000","last_fullsized":"2024-02-12T17:05:45.582121+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:25.278409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:25.278409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":112,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589689+0000","last_change":"2024-02-12T17:03:59.035349+0000","last_active":"2024-02-12T17:05:45.589689+0000","last_peered":"2024-02-12T17:05:45.589689+0000","last_clean":"2024-02-12T17:05:45.589689+0000","last_became_active":"2024-02-12T17:03:50.639321+0000","last_became_peered":"2024-02-12T17:03:50.639321+0000","last_unstale":"2024-02-12T17:05:45.589689+0000","last_undegraded":"2024-02-12T17:05:45.589689+0000","last_fullsized":"2024-02-12T17:05:45.589689+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":438,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582542+0000","last_change":"2024-02-12T17:05:42.512197+0000","last_active":"2024-02-12T17:05:45.582542+0000","last_peered":"2024-02-12T17:05:45.582542+0000","last_clean":"2024-02-12T17:05:45.582542+0000","last_became_active":"2024-02-12T17:05:42.511859+0000","last_became_peered":"2024-02-12T17:05:42.511859+0000","last_unstale":"2024-02-12T17:05:45.582542+0000","last_undegraded":"2024-02-12T17:05:45.582542+0000","last_fullsized":"2024-02-12T17:05:45.582542+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":405,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582452+0000","last_change":"2024-02-12T17:04:39.887129+0000","last_active":"2024-02-12T17:05:45.582452+0000","last_peered":"2024-02-12T17:05:45.582452+0000","last_clean":"2024-02-12T17:05:45.582452+0000","last_became_active":"2024-02-12T17:04:39.886041+0000","last_became_peered":"2024-02-12T17:04:39.886041+0000","last_unstale":"2024-02-12T17:05:45.582452+0000","last_undegraded":"2024-02-12T17:05:45.582452+0000","last_fullsized":"2024-02-12T17:05:45.582452+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T16:57:28.846760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":431,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.580978+0000","last_change":"2024-02-12T17:03:50.551444+0000","last_active":"2024-02-12T17:05:45.580978+0000","last_peered":"2024-02-12T17:05:45.580978+0000","last_clean":"2024-02-12T17:05:45.580978+0000","last_became_active":"2024-02-12T17:03:50.550931+0000","last_became_peered":"2024-02-12T17:03:50.550931+0000","last_unstale":"2024-02-12T17:05:45.580978+0000","last_undegraded":"2024-02-12T17:05:45.580978+0000","last_fullsized":"2024-02-12T17:05:45.580978+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:24.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T16:57:24.805273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":121,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589350+0000","last_change":"2024-02-12T17:04:40.315768+0000","last_active":"2024-02-12T17:05:45.589350+0000","last_peered":"2024-02-12T17:05:45.589350+0000","last_clean":"2024-02-12T17:05:45.589350+0000","last_became_active":"2024-02-12T17:04:39.865088+0000","last_became_peered":"2024-02-12T17:04:39.865088+0000","last_unstale":"2024-02-12T17:05:45.589350+0000","last_undegraded":"2024-02-12T17:05:45.589350+0000","last_fullsized":"2024-02-12T17:05:45.589350+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.938805+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":224,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582157+0000","last_change":"2024-02-12T17:05:42.544935+0000","last_active":"2024-02-12T17:05:45.582157+0000","last_peered":"2024-02-12T17:05:45.582157+0000","last_clean":"2024-02-12T17:05:45.582157+0000","last_became_active":"2024-02-12T17:05:42.544611+0000","last_became_peered":"2024-02-12T17:05:42.544611+0000","last_unstale":"2024-02-12T17:05:45.582157+0000","last_undegraded":"2024-02-12T17:05:45.582157+0000","last_fullsized":"2024-02-12T17:05:45.582157+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":100,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589470+0000","last_change":"2024-02-12T17:03:50.545703+0000","last_active":"2024-02-12T17:05:45.589470+0000","last_peered":"2024-02-12T17:05:45.589470+0000","last_clean":"2024-02-12T17:05:45.589470+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:05:45.589470+0000","last_undegraded":"2024-02-12T17:05:45.589470+0000","last_fullsized":"2024-02-12T17:05:45.589470+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.051528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.051528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":265,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581251+0000","last_change":"2024-02-12T17:03:20.768326+0000","last_active":"2024-02-12T17:05:45.581251+0000","last_peered":"2024-02-12T17:05:45.581251+0000","last_clean":"2024-02-12T17:05:45.581251+0000","last_became_active":"2024-02-12T17:03:20.767943+0000","last_became_peered":"2024-02-12T17:03:20.767943+0000","last_unstale":"2024-02-12T17:05:45.581251+0000","last_undegraded":"2024-02-12T17:05:45.581251+0000","last_fullsized":"2024-02-12T17:05:45.581251+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.858411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.858411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":405,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.580905+0000","last_change":"2024-02-12T17:04:40.575962+0000","last_active":"2024-02-12T17:05:45.580905+0000","last_peered":"2024-02-12T17:05:45.580905+0000","last_clean":"2024-02-12T17:05:45.580905+0000","last_became_active":"2024-02-12T17:04:39.883548+0000","last_became_peered":"2024-02-12T17:04:39.883548+0000","last_unstale":"2024-02-12T17:05:45.580905+0000","last_undegraded":"2024-02-12T17:05:45.580905+0000","last_fullsized":"2024-02-12T17:05:45.580905+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:16.064966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.064966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":455,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.590609+0000","last_change":"2024-02-12T17:05:42.512845+0000","last_active":"2024-02-12T17:05:45.590609+0000","last_peered":"2024-02-12T17:05:45.590609+0000","last_clean":"2024-02-12T17:05:45.590609+0000","last_became_active":"2024-02-12T17:05:42.512394+0000","last_became_peered":"2024-02-12T17:05:42.512394+0000","last_unstale":"2024-02-12T17:05:45.590609+0000","last_undegraded":"2024-02-12T17:05:45.590609+0000","last_fullsized":"2024-02-12T17:05:45.590609+0000","mapping_epoch":433,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":347,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581935+0000","last_change":"2024-02-12T17:04:39.877393+0000","last_active":"2024-02-12T17:05:45.581935+0000","last_peered":"2024-02-12T17:05:45.581935+0000","last_clean":"2024-02-12T17:05:45.581935+0000","last_became_active":"2024-02-12T17:04:39.876910+0000","last_became_peered":"2024-02-12T17:04:39.876910+0000","last_unstale":"2024-02-12T17:05:45.581935+0000","last_undegraded":"2024-02-12T17:05:45.581935+0000","last_fullsized":"2024-02-12T17:05:45.581935+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.019813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.019813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":100,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591406+0000","last_change":"2024-02-12T17:03:50.559509+0000","last_active":"2024-02-12T17:05:45.591406+0000","last_peered":"2024-02-12T17:05:45.591406+0000","last_clean":"2024-02-12T17:05:45.591406+0000","last_became_active":"2024-02-12T17:03:50.558598+0000","last_became_peered":"2024-02-12T17:03:50.558598+0000","last_unstale":"2024-02-12T17:05:45.591406+0000","last_undegraded":"2024-02-12T17:05:45.591406+0000","last_fullsized":"2024-02-12T17:05:45.591406+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":214,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581209+0000","last_change":"2024-02-12T17:03:22.527733+0000","last_active":"2024-02-12T17:05:45.581209+0000","last_peered":"2024-02-12T17:05:45.581209+0000","last_clean":"2024-02-12T17:05:45.581209+0000","last_became_active":"2024-02-12T17:03:20.774315+0000","last_became_peered":"2024-02-12T17:03:20.774315+0000","last_unstale":"2024-02-12T17:05:45.581209+0000","last_undegraded":"2024-02-12T17:05:45.581209+0000","last_fullsized":"2024-02-12T17:05:45.581209+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:00:53.792980+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.792980+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"435'15","reported_seq":394,"reported_epoch":438,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:46.346108+0000","last_change":"2024-02-12T17:05:43.559297+0000","last_active":"2024-02-12T17:05:46.346108+0000","last_peered":"2024-02-12T17:05:46.346108+0000","last_clean":"2024-02-12T17:05:17.796312+0000","last_became_active":"2024-02-12T17:05:43.558958+0000","last_became_peered":"2024-02-12T17:05:43.558958+0000","last_unstale":"2024-02-12T17:05:46.346108+0000","last_undegraded":"2024-02-12T17:05:46.346108+0000","last_fullsized":"2024-02-12T17:05:46.346108+0000","mapping_epoch":434,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[2,7,5],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":188,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591907+0000","last_change":"2024-02-12T17:04:39.890091+0000","last_active":"2024-02-12T17:05:45.591907+0000","last_peered":"2024-02-12T17:05:45.591907+0000","last_clean":"2024-02-12T17:05:45.591907+0000","last_became_active":"2024-02-12T17:04:39.886343+0000","last_became_peered":"2024-02-12T17:04:39.886343+0000","last_unstale":"2024-02-12T17:05:45.591907+0000","last_undegraded":"2024-02-12T17:05:45.591907+0000","last_fullsized":"2024-02-12T17:05:45.591907+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.953891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.953891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":503,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581162+0000","last_change":"2024-02-12T17:03:50.690222+0000","last_active":"2024-02-12T17:05:45.581162+0000","last_peered":"2024-02-12T17:05:45.581162+0000","last_clean":"2024-02-12T17:05:45.581162+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:05:45.581162+0000","last_undegraded":"2024-02-12T17:05:45.581162+0000","last_fullsized":"2024-02-12T17:05:45.581162+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:31.827632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:31.827632+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"438'17","reported_seq":353,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:46.339239+0000","last_change":"2024-02-12T17:05:45.604659+0000","last_active":"2024-02-12T17:05:46.339239+0000","last_peered":"2024-02-12T17:05:46.339239+0000","last_clean":"2024-02-12T17:05:46.339239+0000","last_became_active":"2024-02-12T17:05:45.604285+0000","last_became_peered":"2024-02-12T17:05:45.604285+0000","last_unstale":"2024-02-12T17:05:46.339239+0000","last_undegraded":"2024-02-12T17:05:46.339239+0000","last_fullsized":"2024-02-12T17:05:46.339239+0000","mapping_epoch":436,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":422,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581632+0000","last_change":"2024-02-12T17:03:50.577996+0000","last_active":"2024-02-12T17:05:45.581632+0000","last_peered":"2024-02-12T17:05:45.581632+0000","last_clean":"2024-02-12T17:05:45.581632+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:05:45.581632+0000","last_undegraded":"2024-02-12T17:05:45.581632+0000","last_fullsized":"2024-02-12T17:05:45.581632+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:34.874363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:34.874363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":112,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591283+0000","last_change":"2024-02-12T17:03:55.244602+0000","last_active":"2024-02-12T17:05:45.591283+0000","last_peered":"2024-02-12T17:05:45.591283+0000","last_clean":"2024-02-12T17:05:45.591283+0000","last_became_active":"2024-02-12T17:03:50.556444+0000","last_became_peered":"2024-02-12T17:03:50.556444+0000","last_unstale":"2024-02-12T17:05:45.591283+0000","last_undegraded":"2024-02-12T17:05:45.591283+0000","last_fullsized":"2024-02-12T17:05:45.591283+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:37.219879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:37.219879+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":418,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.587606+0000","last_change":"2024-02-12T17:04:39.889382+0000","last_active":"2024-02-12T17:05:45.587606+0000","last_peered":"2024-02-12T17:05:45.587606+0000","last_clean":"2024-02-12T17:05:45.587606+0000","last_became_active":"2024-02-12T17:04:39.889055+0000","last_became_peered":"2024-02-12T17:04:39.889055+0000","last_unstale":"2024-02-12T17:05:45.587606+0000","last_undegraded":"2024-02-12T17:05:45.587606+0000","last_fullsized":"2024-02-12T17:05:45.587606+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.187738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":361,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.580223+0000","last_change":"2024-02-12T17:04:39.884622+0000","last_active":"2024-02-12T17:05:45.580223+0000","last_peered":"2024-02-12T17:05:45.580223+0000","last_clean":"2024-02-12T17:05:45.580223+0000","last_became_active":"2024-02-12T17:04:39.884162+0000","last_became_peered":"2024-02-12T17:04:39.884162+0000","last_unstale":"2024-02-12T17:05:45.580223+0000","last_undegraded":"2024-02-12T17:05:45.580223+0000","last_fullsized":"2024-02-12T17:05:45.580223+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:32.127593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:32.127593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":108,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.599895+0000","last_change":"2024-02-12T17:03:53.944994+0000","last_active":"2024-02-12T17:05:45.599895+0000","last_peered":"2024-02-12T17:05:45.599895+0000","last_clean":"2024-02-12T17:05:45.599895+0000","last_became_active":"2024-02-12T17:03:50.555238+0000","last_became_peered":"2024-02-12T17:03:50.555238+0000","last_unstale":"2024-02-12T17:05:45.599895+0000","last_undegraded":"2024-02-12T17:05:45.599895+0000","last_fullsized":"2024-02-12T17:05:45.599895+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-02-12T17:00:53.269107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.269107+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":75,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.583573+0000","last_change":"2024-02-12T17:04:39.936874+0000","last_active":"2024-02-12T17:05:45.583573+0000","last_peered":"2024-02-12T17:05:45.583573+0000","last_clean":"2024-02-12T17:05:45.583573+0000","last_became_active":"2024-02-12T17:04:39.936538+0000","last_became_peered":"2024-02-12T17:04:39.936538+0000","last_unstale":"2024-02-12T17:05:45.583573+0000","last_undegraded":"2024-02-12T17:05:45.583573+0000","last_fullsized":"2024-02-12T17:05:45.583573+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:43.982623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.982623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":325,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.590768+0000","last_change":"2024-02-12T17:04:39.867059+0000","last_active":"2024-02-12T17:05:45.590768+0000","last_peered":"2024-02-12T17:05:45.590768+0000","last_clean":"2024-02-12T17:05:45.590768+0000","last_became_active":"2024-02-12T17:04:39.866732+0000","last_became_peered":"2024-02-12T17:04:39.866732+0000","last_unstale":"2024-02-12T17:05:45.590768+0000","last_undegraded":"2024-02-12T17:05:45.590768+0000","last_fullsized":"2024-02-12T17:05:45.590768+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.824194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.824194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":110,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.595805+0000","last_change":"2024-02-12T17:03:55.514799+0000","last_active":"2024-02-12T17:05:45.595805+0000","last_peered":"2024-02-12T17:05:45.595805+0000","last_clean":"2024-02-12T17:05:45.595805+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:05:45.595805+0000","last_undegraded":"2024-02-12T17:05:45.595805+0000","last_fullsized":"2024-02-12T17:05:45.595805+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:00:53.936664+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.936664+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"438'16903","reported_seq":17963,"reported_epoch":438,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:05:48.698618+0000","last_change":"2024-02-12T17:05:10.179097+0000","last_active":"2024-02-12T17:05:48.698618+0000","last_peered":"2024-02-12T17:05:48.698618+0000","last_clean":"2024-02-12T17:03:49.505679+0000","last_became_active":"2024-02-12T17:04:40.888742+0000","last_became_peered":"2024-02-12T17:04:40.888742+0000","last_unstale":"2024-02-12T17:05:48.698618+0000","last_undegraded":"2024-02-12T17:05:48.698618+0000","last_fullsized":"2024-02-12T17:05:48.698618+0000","mapping_epoch":378,"log_start":"331'6900","ondisk_log_start":"331'6900","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":10003,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11075584,"num_objects":169,"num_object_clones":0,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":72,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":448,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":169}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":127,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.597709+0000","last_change":"2024-02-12T17:04:39.890573+0000","last_active":"2024-02-12T17:05:45.597709+0000","last_peered":"2024-02-12T17:05:45.597709+0000","last_clean":"2024-02-12T17:05:45.597709+0000","last_became_active":"2024-02-12T17:04:39.889283+0000","last_became_peered":"2024-02-12T17:04:39.889283+0000","last_unstale":"2024-02-12T17:05:45.597709+0000","last_undegraded":"2024-02-12T17:05:45.597709+0000","last_fullsized":"2024-02-12T17:05:45.597709+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:28.912872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.912872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":137,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.584986+0000","last_change":"2024-02-12T17:04:40.578504+0000","last_active":"2024-02-12T17:05:45.584986+0000","last_peered":"2024-02-12T17:05:45.584986+0000","last_clean":"2024-02-12T17:05:45.584986+0000","last_became_active":"2024-02-12T17:04:39.890440+0000","last_became_peered":"2024-02-12T17:04:39.890440+0000","last_unstale":"2024-02-12T17:05:45.584986+0000","last_undegraded":"2024-02-12T17:05:45.584986+0000","last_fullsized":"2024-02-12T17:05:45.584986+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:02:17.066331+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.066331+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":133,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589968+0000","last_change":"2024-02-12T17:04:42.701462+0000","last_active":"2024-02-12T17:05:45.589968+0000","last_peered":"2024-02-12T17:05:45.589968+0000","last_clean":"2024-02-12T17:05:45.589968+0000","last_became_active":"2024-02-12T17:04:39.891351+0000","last_became_peered":"2024-02-12T17:04:39.891351+0000","last_unstale":"2024-02-12T17:05:45.589968+0000","last_undegraded":"2024-02-12T17:05:45.589968+0000","last_fullsized":"2024-02-12T17:05:45.589968+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":332,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581557+0000","last_change":"2024-02-12T17:04:39.882820+0000","last_active":"2024-02-12T17:05:45.581557+0000","last_peered":"2024-02-12T17:05:45.581557+0000","last_clean":"2024-02-12T17:05:45.581557+0000","last_became_active":"2024-02-12T17:04:39.880493+0000","last_became_peered":"2024-02-12T17:04:39.880493+0000","last_unstale":"2024-02-12T17:05:45.581557+0000","last_undegraded":"2024-02-12T17:05:45.581557+0000","last_fullsized":"2024-02-12T17:05:45.581557+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:25.815816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.815816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"438'17264","reported_seq":26997,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:48.217944+0000","last_change":"2024-02-12T17:04:30.711484+0000","last_active":"2024-02-12T17:05:48.217944+0000","last_peered":"2024-02-12T17:05:48.217944+0000","last_clean":"2024-02-12T17:05:48.217944+0000","last_became_active":"2024-02-12T17:04:30.678545+0000","last_became_peered":"2024-02-12T17:04:30.678545+0000","last_unstale":"2024-02-12T17:05:48.217944+0000","last_undegraded":"2024-02-12T17:05:48.217944+0000","last_fullsized":"2024-02-12T17:05:48.217944+0000","mapping_epoch":368,"log_start":"382'14219","ondisk_log_start":"382'14219","created":304,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3045,"ondisk_log_size":3045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15335424,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17264,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4785,"num_bytes_recovered":313589760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":414,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.586155+0000","last_change":"2024-02-12T17:03:50.533429+0000","last_active":"2024-02-12T17:05:45.586155+0000","last_peered":"2024-02-12T17:05:45.586155+0000","last_clean":"2024-02-12T17:05:45.586155+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:05:45.586155+0000","last_undegraded":"2024-02-12T17:05:45.586155+0000","last_fullsized":"2024-02-12T17:05:45.586155+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.933952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.592899+0000","last_change":"2024-02-12T17:02:30.758581+0000","last_active":"2024-02-12T17:05:45.592899+0000","last_peered":"2024-02-12T17:05:45.592899+0000","last_clean":"2024-02-12T17:05:45.592899+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:05:45.592899+0000","last_undegraded":"2024-02-12T17:05:45.592899+0000","last_fullsized":"2024-02-12T17:05:45.592899+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.758520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.758520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":338,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582251+0000","last_change":"2024-02-12T17:04:41.560789+0000","last_active":"2024-02-12T17:05:45.582251+0000","last_peered":"2024-02-12T17:05:45.582251+0000","last_clean":"2024-02-12T17:05:45.582251+0000","last_became_active":"2024-02-12T17:04:39.883521+0000","last_became_peered":"2024-02-12T17:04:39.883521+0000","last_unstale":"2024-02-12T17:05:45.582251+0000","last_undegraded":"2024-02-12T17:05:45.582251+0000","last_fullsized":"2024-02-12T17:05:45.582251+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-12T16:57:35.783906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:35.783906+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"433'15","reported_seq":596,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:46.338657+0000","last_change":"2024-02-12T17:05:40.422839+0000","last_active":"2024-02-12T17:05:46.338657+0000","last_peered":"2024-02-12T17:05:46.338657+0000","last_clean":"2024-02-12T17:05:46.338657+0000","last_became_active":"2024-02-12T17:05:40.422494+0000","last_became_peered":"2024-02-12T17:05:40.422494+0000","last_unstale":"2024-02-12T17:05:46.338657+0000","last_undegraded":"2024-02-12T17:05:46.338657+0000","last_fullsized":"2024-02-12T17:05:46.338657+0000","mapping_epoch":347,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"438'16956","reported_seq":18158,"reported_epoch":438,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:05:48.669833+0000","last_change":"2024-02-12T17:05:39.869576+0000","last_active":"2024-02-12T17:05:48.669833+0000","last_peered":"2024-02-12T17:05:48.669833+0000","last_clean":"2024-02-12T17:03:19.725881+0000","last_became_active":"2024-02-12T17:03:50.651275+0000","last_became_peered":"2024-02-12T17:03:50.651275+0000","last_unstale":"2024-02-12T17:05:48.669833+0000","last_undegraded":"2024-02-12T17:05:48.669833+0000","last_fullsized":"2024-02-12T17:05:48.669833+0000","mapping_epoch":347,"log_start":"380'13848","ondisk_log_start":"380'13848","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3108,"ondisk_log_size":3108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11927552,"num_objects":182,"num_object_clones":0,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":170,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16956,"num_write_kb":548416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":33619968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,0],"avail_no_missing":["7","0","4","5"],"object_location_counts":[{"shards":"0,5,7","objects":182}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":127,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.595467+0000","last_change":"2024-02-12T17:04:58.474896+0000","last_active":"2024-02-12T17:05:45.595467+0000","last_peered":"2024-02-12T17:05:45.595467+0000","last_clean":"2024-02-12T17:05:45.595467+0000","last_became_active":"2024-02-12T17:04:58.474504+0000","last_became_peered":"2024-02-12T17:04:58.474504+0000","last_unstale":"2024-02-12T17:05:45.595467+0000","last_undegraded":"2024-02-12T17:05:45.595467+0000","last_fullsized":"2024-02-12T17:05:45.595467+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":436,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581715+0000","last_change":"2024-02-12T17:03:50.549240+0000","last_active":"2024-02-12T17:05:45.581715+0000","last_peered":"2024-02-12T17:05:45.581715+0000","last_clean":"2024-02-12T17:05:45.581715+0000","last_became_active":"2024-02-12T17:03:50.548926+0000","last_became_peered":"2024-02-12T17:03:50.548926+0000","last_unstale":"2024-02-12T17:05:45.581715+0000","last_undegraded":"2024-02-12T17:05:45.581715+0000","last_fullsized":"2024-02-12T17:05:45.581715+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:42.892563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.892563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"348'55","reported_seq":147,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:46.340396+0000","last_change":"2024-02-12T17:03:58.596193+0000","last_active":"2024-02-12T17:05:46.340396+0000","last_peered":"2024-02-12T17:05:46.340396+0000","last_clean":"2024-02-12T17:05:46.340396+0000","last_became_active":"2024-02-12T17:03:50.557154+0000","last_became_peered":"2024-02-12T17:03:50.557154+0000","last_unstale":"2024-02-12T17:05:46.340396+0000","last_undegraded":"2024-02-12T17:05:46.340396+0000","last_fullsized":"2024-02-12T17:05:46.340396+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":387,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.587777+0000","last_change":"2024-02-12T17:05:40.431590+0000","last_active":"2024-02-12T17:05:45.587777+0000","last_peered":"2024-02-12T17:05:45.587777+0000","last_clean":"2024-02-12T17:05:45.587777+0000","last_became_active":"2024-02-12T17:05:40.431265+0000","last_became_peered":"2024-02-12T17:05:40.431265+0000","last_unstale":"2024-02-12T17:05:45.587777+0000","last_undegraded":"2024-02-12T17:05:45.587777+0000","last_fullsized":"2024-02-12T17:05:45.587777+0000","mapping_epoch":377,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"438'17182","reported_seq":18440,"reported_epoch":438,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:48.302536+0000","last_change":"2024-02-12T17:04:39.884779+0000","last_active":"2024-02-12T17:05:48.302536+0000","last_peered":"2024-02-12T17:05:48.302536+0000","last_clean":"2024-02-12T17:03:49.453764+0000","last_became_active":"2024-02-12T17:03:51.655919+0000","last_became_peered":"2024-02-12T17:03:51.655919+0000","last_unstale":"2024-02-12T17:05:48.302536+0000","last_undegraded":"2024-02-12T17:05:48.302536+0000","last_fullsized":"2024-02-12T17:05:48.302536+0000","mapping_epoch":348,"log_start":"383'14142","ondisk_log_start":"383'14142","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3040,"ondisk_log_size":3040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":174,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17182,"num_write_kb":556224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":36175872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":200}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":425,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.587906+0000","last_change":"2024-02-12T17:03:50.566072+0000","last_active":"2024-02-12T17:05:45.587906+0000","last_peered":"2024-02-12T17:05:45.587906+0000","last_clean":"2024-02-12T17:05:45.587906+0000","last_became_active":"2024-02-12T17:03:50.564700+0000","last_became_peered":"2024-02-12T17:03:50.564700+0000","last_unstale":"2024-02-12T17:05:45.587906+0000","last_undegraded":"2024-02-12T17:05:45.587906+0000","last_fullsized":"2024-02-12T17:05:45.587906+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:04.085827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:04.085827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":439,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:47.160493+0000","last_change":"2024-02-12T17:01:16.913133+0000","last_active":"2024-02-12T17:05:47.160493+0000","last_peered":"2024-02-12T17:05:47.160493+0000","last_clean":"2024-02-12T17:05:47.160493+0000","last_became_active":"2024-02-12T17:01:16.912041+0000","last_became_peered":"2024-02-12T17:01:16.912041+0000","last_unstale":"2024-02-12T17:05:47.160493+0000","last_undegraded":"2024-02-12T17:05:47.160493+0000","last_fullsized":"2024-02-12T17:05:47.160493+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:00:52.838852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:52.838852+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":277,"num_read_kb":272,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":100,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589822+0000","last_change":"2024-02-12T17:03:50.533363+0000","last_active":"2024-02-12T17:05:45.589822+0000","last_peered":"2024-02-12T17:05:45.589822+0000","last_clean":"2024-02-12T17:05:45.589822+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:05:45.589822+0000","last_undegraded":"2024-02-12T17:05:45.589822+0000","last_fullsized":"2024-02-12T17:05:45.589822+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:38.142602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:38.142602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":370,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582143+0000","last_change":"2024-02-12T17:03:51.175255+0000","last_active":"2024-02-12T17:05:45.582143+0000","last_peered":"2024-02-12T17:05:45.582143+0000","last_clean":"2024-02-12T17:05:45.582143+0000","last_became_active":"2024-02-12T17:03:50.573814+0000","last_became_peered":"2024-02-12T17:03:50.573814+0000","last_unstale":"2024-02-12T17:05:45.582143+0000","last_undegraded":"2024-02-12T17:05:45.582143+0000","last_fullsized":"2024-02-12T17:05:45.582143+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":352,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581835+0000","last_change":"2024-02-12T17:01:16.909653+0000","last_active":"2024-02-12T17:05:45.581835+0000","last_peered":"2024-02-12T17:05:45.581835+0000","last_clean":"2024-02-12T17:05:45.581835+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:05:45.581835+0000","last_undegraded":"2024-02-12T17:05:45.581835+0000","last_fullsized":"2024-02-12T17:05:45.581835+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:17.691007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T16:57:17.691007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"438'16990","reported_seq":18748,"reported_epoch":438,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:05:48.662293+0000","last_change":"2024-02-12T17:04:40.746118+0000","last_active":"2024-02-12T17:05:48.662293+0000","last_peered":"2024-02-12T17:05:48.662293+0000","last_clean":"2024-02-12T17:03:49.506422+0000","last_became_active":"2024-02-12T17:03:51.676105+0000","last_became_peered":"2024-02-12T17:03:51.676105+0000","last_unstale":"2024-02-12T17:05:48.662293+0000","last_undegraded":"2024-02-12T17:05:48.662293+0000","last_fullsized":"2024-02-12T17:05:48.662293+0000","mapping_epoch":348,"log_start":"331'6900","ondisk_log_start":"331'6900","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":10090,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12451840,"num_objects":190,"num_object_clones":0,"num_object_copies":570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":59,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16990,"num_write_kb":549760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":802,"num_bytes_recovered":52559872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":190}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":452,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582392+0000","last_change":"2024-02-12T17:04:54.415608+0000","last_active":"2024-02-12T17:05:45.582392+0000","last_peered":"2024-02-12T17:05:45.582392+0000","last_clean":"2024-02-12T17:05:45.582392+0000","last_became_active":"2024-02-12T17:04:54.415431+0000","last_became_peered":"2024-02-12T17:04:54.415431+0000","last_unstale":"2024-02-12T17:05:45.582392+0000","last_undegraded":"2024-02-12T17:05:45.582392+0000","last_fullsized":"2024-02-12T17:05:45.582392+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:57.950284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:00:57.950284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":109,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591112+0000","last_change":"2024-02-12T17:04:39.885990+0000","last_active":"2024-02-12T17:05:45.591112+0000","last_peered":"2024-02-12T17:05:45.591112+0000","last_clean":"2024-02-12T17:05:45.591112+0000","last_became_active":"2024-02-12T17:04:39.885173+0000","last_became_peered":"2024-02-12T17:04:39.885173+0000","last_unstale":"2024-02-12T17:05:45.591112+0000","last_undegraded":"2024-02-12T17:05:45.591112+0000","last_fullsized":"2024-02-12T17:05:45.591112+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:56:45.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T16:56:45.218026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":404,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.580728+0000","last_change":"2024-02-12T16:57:21.756288+0000","last_active":"2024-02-12T17:05:45.580728+0000","last_peered":"2024-02-12T17:05:45.580728+0000","last_clean":"2024-02-12T17:05:45.580728+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:05:45.580728+0000","last_undegraded":"2024-02-12T17:05:45.580728+0000","last_fullsized":"2024-02-12T17:05:45.580728+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:21.756200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.756200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":437,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.579008+0000","last_change":"2024-02-12T17:04:39.915666+0000","last_active":"2024-02-12T17:05:45.579008+0000","last_peered":"2024-02-12T17:05:45.579008+0000","last_clean":"2024-02-12T17:05:45.579008+0000","last_became_active":"2024-02-12T17:04:39.915392+0000","last_became_peered":"2024-02-12T17:04:39.915392+0000","last_unstale":"2024-02-12T17:05:45.579008+0000","last_undegraded":"2024-02-12T17:05:45.579008+0000","last_fullsized":"2024-02-12T17:05:45.579008+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:25.870916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:01:25.870916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":186,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.588063+0000","last_change":"2024-02-12T17:04:40.745958+0000","last_active":"2024-02-12T17:05:45.588063+0000","last_peered":"2024-02-12T17:05:45.588063+0000","last_clean":"2024-02-12T17:05:45.588063+0000","last_became_active":"2024-02-12T17:04:39.889039+0000","last_became_peered":"2024-02-12T17:04:39.889039+0000","last_unstale":"2024-02-12T17:05:45.588063+0000","last_undegraded":"2024-02-12T17:05:45.588063+0000","last_fullsized":"2024-02-12T17:05:45.588063+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'26","last_scrub_stamp":"2024-02-12T17:01:43.163724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:43.163724+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":435,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.583656+0000","last_change":"2024-02-12T17:05:40.432925+0000","last_active":"2024-02-12T17:05:45.583656+0000","last_peered":"2024-02-12T17:05:45.583656+0000","last_clean":"2024-02-12T17:05:45.583656+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:05:45.583656+0000","last_undegraded":"2024-02-12T17:05:45.583656+0000","last_fullsized":"2024-02-12T17:05:45.583656+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"438'16994","reported_seq":17764,"reported_epoch":438,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:48.329937+0000","last_change":"2024-02-12T17:04:40.576566+0000","last_active":"2024-02-12T17:05:48.329937+0000","last_peered":"2024-02-12T17:05:48.329937+0000","last_clean":"2024-02-12T17:03:19.701993+0000","last_became_active":"2024-02-12T17:03:50.639685+0000","last_became_peered":"2024-02-12T17:03:50.639685+0000","last_unstale":"2024-02-12T17:05:48.329937+0000","last_undegraded":"2024-02-12T17:05:48.329937+0000","last_fullsized":"2024-02-12T17:05:48.329937+0000","mapping_epoch":347,"log_start":"381'13945","ondisk_log_start":"381'13945","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3049,"ondisk_log_size":3049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14024704,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":214,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16994,"num_write_kb":550656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":19791872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,0],"avail_no_missing":["5","0","2","4"],"object_location_counts":[{"shards":"0,2,5","objects":214}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":362,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581315+0000","last_change":"2024-02-12T17:04:39.871302+0000","last_active":"2024-02-12T17:05:45.581315+0000","last_peered":"2024-02-12T17:05:45.581315+0000","last_clean":"2024-02-12T17:05:45.581315+0000","last_became_active":"2024-02-12T17:04:39.870979+0000","last_became_peered":"2024-02-12T17:04:39.870979+0000","last_unstale":"2024-02-12T17:05:45.581315+0000","last_undegraded":"2024-02-12T17:05:45.581315+0000","last_fullsized":"2024-02-12T17:05:45.581315+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:27.383019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:27.383019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":188,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589068+0000","last_change":"2024-02-12T17:04:39.892450+0000","last_active":"2024-02-12T17:05:45.589068+0000","last_peered":"2024-02-12T17:05:45.589068+0000","last_clean":"2024-02-12T17:05:45.589068+0000","last_became_active":"2024-02-12T17:04:39.892122+0000","last_became_peered":"2024-02-12T17:04:39.892122+0000","last_unstale":"2024-02-12T17:05:45.589068+0000","last_undegraded":"2024-02-12T17:05:45.589068+0000","last_fullsized":"2024-02-12T17:05:45.589068+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.941776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":434,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.578944+0000","last_change":"2024-02-12T17:03:50.776630+0000","last_active":"2024-02-12T17:05:45.578944+0000","last_peered":"2024-02-12T17:05:45.578944+0000","last_clean":"2024-02-12T17:05:45.578944+0000","last_became_active":"2024-02-12T17:03:50.564881+0000","last_became_peered":"2024-02-12T17:03:50.564881+0000","last_unstale":"2024-02-12T17:05:45.578944+0000","last_undegraded":"2024-02-12T17:05:45.578944+0000","last_fullsized":"2024-02-12T17:05:45.578944+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-12T16:57:21.909637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T16:57:21.909637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":420,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.585358+0000","last_change":"2024-02-12T17:03:24.771579+0000","last_active":"2024-02-12T17:05:45.585358+0000","last_peered":"2024-02-12T17:05:45.585358+0000","last_clean":"2024-02-12T17:05:45.585358+0000","last_became_active":"2024-02-12T17:03:20.780926+0000","last_became_peered":"2024-02-12T17:03:20.780926+0000","last_unstale":"2024-02-12T17:05:45.585358+0000","last_undegraded":"2024-02-12T17:05:45.585358+0000","last_fullsized":"2024-02-12T17:05:45.585358+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-12T16:57:33.892181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.892181+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":436,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582313+0000","last_change":"2024-02-12T17:04:39.885796+0000","last_active":"2024-02-12T17:05:45.582313+0000","last_peered":"2024-02-12T17:05:45.582313+0000","last_clean":"2024-02-12T17:05:45.582313+0000","last_became_active":"2024-02-12T17:04:39.885162+0000","last_became_peered":"2024-02-12T17:04:39.885162+0000","last_unstale":"2024-02-12T17:05:45.582313+0000","last_undegraded":"2024-02-12T17:05:45.582313+0000","last_fullsized":"2024-02-12T17:05:45.582313+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"438'17264","reported_seq":19121,"reported_epoch":438,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:05:48.622658+0000","last_change":"2024-02-12T17:05:10.407523+0000","last_active":"2024-02-12T17:05:48.622658+0000","last_peered":"2024-02-12T17:05:48.622658+0000","last_clean":"2024-02-12T17:03:19.726721+0000","last_became_active":"2024-02-12T17:04:39.940046+0000","last_became_peered":"2024-02-12T17:04:39.940046+0000","last_unstale":"2024-02-12T17:05:48.622658+0000","last_undegraded":"2024-02-12T17:05:48.622658+0000","last_fullsized":"2024-02-12T17:05:48.622658+0000","mapping_epoch":377,"log_start":"382'14131","ondisk_log_start":"382'14131","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3133,"ondisk_log_size":3133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":411,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17264,"num_write_kb":559488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":55312384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,4,0],"avail_no_missing":["7","0","4"],"object_location_counts":[{"shards":"0,4,7","objects":220}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":164,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582874+0000","last_change":"2024-02-12T17:03:20.779625+0000","last_active":"2024-02-12T17:05:45.582874+0000","last_peered":"2024-02-12T17:05:45.582874+0000","last_clean":"2024-02-12T17:05:45.582874+0000","last_became_active":"2024-02-12T17:03:20.779126+0000","last_became_peered":"2024-02-12T17:03:20.779126+0000","last_unstale":"2024-02-12T17:05:45.582874+0000","last_undegraded":"2024-02-12T17:05:45.582874+0000","last_fullsized":"2024-02-12T17:05:45.582874+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:48.617913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:02:48.617913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":425,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.587982+0000","last_change":"2024-02-12T17:04:39.886695+0000","last_active":"2024-02-12T17:05:45.587982+0000","last_peered":"2024-02-12T17:05:45.587982+0000","last_clean":"2024-02-12T17:05:45.587982+0000","last_became_active":"2024-02-12T17:04:39.886102+0000","last_became_peered":"2024-02-12T17:04:39.886102+0000","last_unstale":"2024-02-12T17:05:45.587982+0000","last_undegraded":"2024-02-12T17:05:45.587982+0000","last_fullsized":"2024-02-12T17:05:45.587982+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.212949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.212949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":416,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.580815+0000","last_change":"2024-02-12T17:03:21.658125+0000","last_active":"2024-02-12T17:05:45.580815+0000","last_peered":"2024-02-12T17:05:45.580815+0000","last_clean":"2024-02-12T17:05:45.580815+0000","last_became_active":"2024-02-12T17:03:20.769223+0000","last_became_peered":"2024-02-12T17:03:20.769223+0000","last_unstale":"2024-02-12T17:05:45.580815+0000","last_undegraded":"2024-02-12T17:05:45.580815+0000","last_fullsized":"2024-02-12T17:05:45.580815+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:02:44.688402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:44.688402+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":224,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.586406+0000","last_change":"2024-02-12T17:05:40.448536+0000","last_active":"2024-02-12T17:05:45.586406+0000","last_peered":"2024-02-12T17:05:45.586406+0000","last_clean":"2024-02-12T17:05:45.586406+0000","last_became_active":"2024-02-12T17:05:40.448219+0000","last_became_peered":"2024-02-12T17:05:40.448219+0000","last_unstale":"2024-02-12T17:05:45.586406+0000","last_undegraded":"2024-02-12T17:05:45.586406+0000","last_fullsized":"2024-02-12T17:05:45.586406+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"438'16626","reported_seq":18294,"reported_epoch":438,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:05:48.675459+0000","last_change":"2024-02-12T17:05:09.874394+0000","last_active":"2024-02-12T17:05:48.675459+0000","last_peered":"2024-02-12T17:05:48.675459+0000","last_clean":"2024-02-12T17:04:38.799786+0000","last_became_active":"2024-02-12T17:04:40.925272+0000","last_became_peered":"2024-02-12T17:04:40.925272+0000","last_unstale":"2024-02-12T17:05:48.675459+0000","last_undegraded":"2024-02-12T17:05:48.675459+0000","last_fullsized":"2024-02-12T17:05:48.675459+0000","mapping_epoch":378,"log_start":"381'13525","ondisk_log_start":"381'13525","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3101,"ondisk_log_size":3101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11206680,"num_objects":172,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":58,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16626,"num_write_kb":537473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":755,"num_bytes_recovered":49414168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,4],"avail_no_missing":["7","4","5"],"object_location_counts":[{"shards":"4,5,7","objects":172}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":109,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591192+0000","last_change":"2024-02-12T17:04:39.886704+0000","last_active":"2024-02-12T17:05:45.591192+0000","last_peered":"2024-02-12T17:05:45.591192+0000","last_clean":"2024-02-12T17:05:45.591192+0000","last_became_active":"2024-02-12T17:04:39.885677+0000","last_became_peered":"2024-02-12T17:04:39.885677+0000","last_unstale":"2024-02-12T17:05:45.591192+0000","last_undegraded":"2024-02-12T17:05:45.591192+0000","last_fullsized":"2024-02-12T17:05:45.591192+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T16:57:32.780114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":196,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.595306+0000","last_change":"2024-02-12T17:04:39.864989+0000","last_active":"2024-02-12T17:05:45.595306+0000","last_peered":"2024-02-12T17:05:45.595306+0000","last_clean":"2024-02-12T17:05:45.595306+0000","last_became_active":"2024-02-12T17:04:39.864631+0000","last_became_peered":"2024-02-12T17:04:39.864631+0000","last_unstale":"2024-02-12T17:05:45.595306+0000","last_undegraded":"2024-02-12T17:05:45.595306+0000","last_fullsized":"2024-02-12T17:05:45.595306+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":118,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589835+0000","last_change":"2024-02-12T17:03:54.620306+0000","last_active":"2024-02-12T17:05:45.589835+0000","last_peered":"2024-02-12T17:05:45.589835+0000","last_clean":"2024-02-12T17:05:45.589835+0000","last_became_active":"2024-02-12T17:03:50.556763+0000","last_became_peered":"2024-02-12T17:03:50.556763+0000","last_unstale":"2024-02-12T17:05:45.589835+0000","last_undegraded":"2024-02-12T17:05:45.589835+0000","last_fullsized":"2024-02-12T17:05:45.589835+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":438,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.589614+0000","last_change":"2024-02-12T17:05:40.428597+0000","last_active":"2024-02-12T17:05:45.589614+0000","last_peered":"2024-02-12T17:05:45.589614+0000","last_clean":"2024-02-12T17:05:45.589614+0000","last_became_active":"2024-02-12T17:05:40.427915+0000","last_became_peered":"2024-02-12T17:05:40.427915+0000","last_unstale":"2024-02-12T17:05:45.589614+0000","last_undegraded":"2024-02-12T17:05:45.589614+0000","last_fullsized":"2024-02-12T17:05:45.589614+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"438'17261","reported_seq":19416,"reported_epoch":438,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:48.588282+0000","last_change":"2024-02-12T17:04:50.113487+0000","last_active":"2024-02-12T17:05:48.588282+0000","last_peered":"2024-02-12T17:05:48.588282+0000","last_clean":"2024-02-12T17:03:19.745457+0000","last_became_active":"2024-02-12T17:04:39.930523+0000","last_became_peered":"2024-02-12T17:04:39.930523+0000","last_unstale":"2024-02-12T17:05:48.588282+0000","last_undegraded":"2024-02-12T17:05:48.588282+0000","last_fullsized":"2024-02-12T17:05:48.588282+0000","mapping_epoch":377,"log_start":"338'9837","ondisk_log_start":"338'9837","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'5831","last_scrub_stamp":"2024-02-12T17:02:24.719414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.719414+0000","log_size":7424,"ondisk_log_size":7424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":286,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17261,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":979,"num_bytes_recovered":56623104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":217}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":441,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.583650+0000","last_change":"2024-02-12T17:04:40.405970+0000","last_active":"2024-02-12T17:05:45.583650+0000","last_peered":"2024-02-12T17:05:45.583650+0000","last_clean":"2024-02-12T17:05:45.583650+0000","last_became_active":"2024-02-12T17:04:39.889077+0000","last_became_peered":"2024-02-12T17:04:39.889077+0000","last_unstale":"2024-02-12T17:05:45.583650+0000","last_undegraded":"2024-02-12T17:05:45.583650+0000","last_fullsized":"2024-02-12T17:05:45.583650+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:03:17.811656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.811656+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.585061+0000","last_change":"2024-02-12T17:03:50.563493+0000","last_active":"2024-02-12T17:05:45.585061+0000","last_peered":"2024-02-12T17:05:45.585061+0000","last_clean":"2024-02-12T17:05:45.585061+0000","last_became_active":"2024-02-12T17:03:50.562556+0000","last_became_peered":"2024-02-12T17:03:50.562556+0000","last_unstale":"2024-02-12T17:05:45.585061+0000","last_undegraded":"2024-02-12T17:05:45.585061+0000","last_fullsized":"2024-02-12T17:05:45.585061+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.562425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.562425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":491,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.580568+0000","last_change":"2024-02-12T17:04:41.124664+0000","last_active":"2024-02-12T17:05:45.580568+0000","last_peered":"2024-02-12T17:05:45.580568+0000","last_clean":"2024-02-12T17:05:45.580568+0000","last_became_active":"2024-02-12T17:04:39.884095+0000","last_became_peered":"2024-02-12T17:04:39.884095+0000","last_unstale":"2024-02-12T17:05:45.580568+0000","last_undegraded":"2024-02-12T17:05:45.580568+0000","last_fullsized":"2024-02-12T17:05:45.580568+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-12T16:57:18.681488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:18.681488+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"433'19","reported_seq":173,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:46.338889+0000","last_change":"2024-02-12T17:05:40.451102+0000","last_active":"2024-02-12T17:05:46.338889+0000","last_peered":"2024-02-12T17:05:46.338889+0000","last_clean":"2024-02-12T17:05:46.338889+0000","last_became_active":"2024-02-12T17:05:40.450299+0000","last_became_peered":"2024-02-12T17:05:40.450299+0000","last_unstale":"2024-02-12T17:05:46.338889+0000","last_undegraded":"2024-02-12T17:05:46.338889+0000","last_fullsized":"2024-02-12T17:05:46.338889+0000","mapping_epoch":377,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"438'16726","reported_seq":18485,"reported_epoch":438,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:05:48.694189+0000","last_change":"2024-02-12T17:04:40.579124+0000","last_active":"2024-02-12T17:05:48.694189+0000","last_peered":"2024-02-12T17:05:48.694189+0000","last_clean":"2024-02-12T17:03:19.746645+0000","last_became_active":"2024-02-12T17:03:50.650695+0000","last_became_peered":"2024-02-12T17:03:50.650695+0000","last_unstale":"2024-02-12T17:05:48.694189+0000","last_undegraded":"2024-02-12T17:05:48.694189+0000","last_fullsized":"2024-02-12T17:05:48.694189+0000","mapping_epoch":347,"log_start":"382'13636","ondisk_log_start":"382'13636","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3090,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12713984,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16726,"num_write_kb":541440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":794,"num_bytes_recovered":51904512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,5,0],"avail_no_missing":["7","0","2","5"],"object_location_counts":[{"shards":"0,5,7","objects":194}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":118,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591021+0000","last_change":"2024-02-12T17:04:33.794769+0000","last_active":"2024-02-12T17:05:45.591021+0000","last_peered":"2024-02-12T17:05:45.591021+0000","last_clean":"2024-02-12T17:05:45.591021+0000","last_became_active":"2024-02-12T17:04:33.794457+0000","last_became_peered":"2024-02-12T17:04:33.794457+0000","last_unstale":"2024-02-12T17:05:45.591021+0000","last_undegraded":"2024-02-12T17:05:45.591021+0000","last_fullsized":"2024-02-12T17:05:45.591021+0000","mapping_epoch":347,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.811153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.811153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"433'20","reported_seq":594,"reported_epoch":438,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:46.339987+0000","last_change":"2024-02-12T17:05:41.498296+0000","last_active":"2024-02-12T17:05:46.339987+0000","last_peered":"2024-02-12T17:05:46.339987+0000","last_clean":"2024-02-12T17:05:38.402403+0000","last_became_active":"2024-02-12T17:05:41.497919+0000","last_became_peered":"2024-02-12T17:05:41.497919+0000","last_unstale":"2024-02-12T17:05:46.339987+0000","last_undegraded":"2024-02-12T17:05:46.339987+0000","last_fullsized":"2024-02-12T17:05:46.339987+0000","mapping_epoch":432,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:29.805325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T16:57:29.805325+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,0,7],"avail_no_missing":["5","0","7"],"object_location_counts":[{"shards":"0,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":70,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.578858+0000","last_change":"2024-02-12T17:04:39.890077+0000","last_active":"2024-02-12T17:05:45.578858+0000","last_peered":"2024-02-12T17:05:45.578858+0000","last_clean":"2024-02-12T17:05:45.578858+0000","last_became_active":"2024-02-12T17:04:39.889588+0000","last_became_peered":"2024-02-12T17:04:39.889588+0000","last_unstale":"2024-02-12T17:05:45.578858+0000","last_undegraded":"2024-02-12T17:05:45.578858+0000","last_fullsized":"2024-02-12T17:05:45.578858+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T16:57:33.806968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":133,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.595563+0000","last_change":"2024-02-12T17:04:41.814348+0000","last_active":"2024-02-12T17:05:45.595563+0000","last_peered":"2024-02-12T17:05:45.595563+0000","last_clean":"2024-02-12T17:05:45.595563+0000","last_became_active":"2024-02-12T17:04:39.888473+0000","last_became_peered":"2024-02-12T17:04:39.888473+0000","last_unstale":"2024-02-12T17:05:45.595563+0000","last_undegraded":"2024-02-12T17:05:45.595563+0000","last_fullsized":"2024-02-12T17:05:45.595563+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-12T16:57:26.266525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.266525+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"438'16824","reported_seq":17817,"reported_epoch":438,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:05:48.640880+0000","last_change":"2024-02-12T17:05:10.406749+0000","last_active":"2024-02-12T17:05:48.640880+0000","last_peered":"2024-02-12T17:05:48.640880+0000","last_clean":"2024-02-12T17:03:19.738005+0000","last_became_active":"2024-02-12T17:03:50.628646+0000","last_became_peered":"2024-02-12T17:03:50.628646+0000","last_unstale":"2024-02-12T17:05:48.640880+0000","last_undegraded":"2024-02-12T17:05:48.640880+0000","last_fullsized":"2024-02-12T17:05:48.640880+0000","mapping_epoch":347,"log_start":"380'13696","ondisk_log_start":"380'13696","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3128,"ondisk_log_size":3128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":219,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":545600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":26804224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[7,2,0],"avail_no_missing":["7","0","2","5"],"object_location_counts":[{"shards":"0,2,7","objects":226}],"blocked_by":[],"up_primary":3,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":448,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581471+0000","last_change":"2024-02-12T17:04:39.887836+0000","last_active":"2024-02-12T17:05:45.581471+0000","last_peered":"2024-02-12T17:05:45.581471+0000","last_clean":"2024-02-12T17:05:45.581471+0000","last_became_active":"2024-02-12T17:04:39.887488+0000","last_became_peered":"2024-02-12T17:04:39.887488+0000","last_unstale":"2024-02-12T17:05:45.581471+0000","last_undegraded":"2024-02-12T17:05:45.581471+0000","last_fullsized":"2024-02-12T17:05:45.581471+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"435'21","reported_seq":675,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:46.338936+0000","last_change":"2024-02-12T17:05:43.541152+0000","last_active":"2024-02-12T17:05:46.338936+0000","last_peered":"2024-02-12T17:05:46.338936+0000","last_clean":"2024-02-12T17:05:46.338936+0000","last_became_active":"2024-02-12T17:05:43.540504+0000","last_became_peered":"2024-02-12T17:05:43.540504+0000","last_unstale":"2024-02-12T17:05:46.338936+0000","last_undegraded":"2024-02-12T17:05:46.338936+0000","last_fullsized":"2024-02-12T17:05:46.338936+0000","mapping_epoch":434,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-12T17:02:20.866591+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:02:20.866591+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":70,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582741+0000","last_change":"2024-02-12T17:04:39.886675+0000","last_active":"2024-02-12T17:05:45.582741+0000","last_peered":"2024-02-12T17:05:45.582741+0000","last_clean":"2024-02-12T17:05:45.582741+0000","last_became_active":"2024-02-12T17:04:39.886222+0000","last_became_peered":"2024-02-12T17:04:39.886222+0000","last_unstale":"2024-02-12T17:05:45.582741+0000","last_undegraded":"2024-02-12T17:05:45.582741+0000","last_fullsized":"2024-02-12T17:05:45.582741+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.863530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.863530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":141,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.599735+0000","last_change":"2024-02-12T17:04:43.226687+0000","last_active":"2024-02-12T17:05:45.599735+0000","last_peered":"2024-02-12T17:05:45.599735+0000","last_clean":"2024-02-12T17:05:45.599735+0000","last_became_active":"2024-02-12T17:04:39.892266+0000","last_became_peered":"2024-02-12T17:04:39.892266+0000","last_unstale":"2024-02-12T17:05:45.599735+0000","last_undegraded":"2024-02-12T17:05:45.599735+0000","last_fullsized":"2024-02-12T17:05:45.599735+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:01:41.694875+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:01:41.694875+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"438'16925","reported_seq":17099,"reported_epoch":438,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:05:48.274878+0000","last_change":"2024-02-12T17:05:39.932637+0000","last_active":"2024-02-12T17:05:48.274878+0000","last_peered":"2024-02-12T17:05:48.274878+0000","last_clean":"2024-02-12T17:03:49.455800+0000","last_became_active":"2024-02-12T17:04:39.870350+0000","last_became_peered":"2024-02-12T17:04:39.870350+0000","last_unstale":"2024-02-12T17:05:48.274878+0000","last_undegraded":"2024-02-12T17:05:48.274878+0000","last_fullsized":"2024-02-12T17:05:48.274878+0000","mapping_epoch":377,"log_start":"368'13328","ondisk_log_start":"368'13328","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3597,"ondisk_log_size":3597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14483456,"num_objects":221,"num_object_clones":0,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":221,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16925,"num_write_kb":548672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[2,7,4],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":221}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":175,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.580759+0000","last_change":"2024-02-12T17:04:39.884597+0000","last_active":"2024-02-12T17:05:45.580759+0000","last_peered":"2024-02-12T17:05:45.580759+0000","last_clean":"2024-02-12T17:05:45.580759+0000","last_became_active":"2024-02-12T17:04:39.884295+0000","last_became_peered":"2024-02-12T17:04:39.884295+0000","last_unstale":"2024-02-12T17:05:45.580759+0000","last_undegraded":"2024-02-12T17:05:45.580759+0000","last_fullsized":"2024-02-12T17:05:45.580759+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:22.329766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:22.329766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":240,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:46.341348+0000","last_change":"2024-02-12T17:05:40.431666+0000","last_active":"2024-02-12T17:05:46.341348+0000","last_peered":"2024-02-12T17:05:46.341348+0000","last_clean":"2024-02-12T17:05:46.341348+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:05:46.341348+0000","last_undegraded":"2024-02-12T17:05:46.341348+0000","last_fullsized":"2024-02-12T17:05:46.341348+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'11","last_scrub_stamp":"2024-02-12T17:02:31.546965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:31.546965+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":100,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591265+0000","last_change":"2024-02-12T17:03:50.589204+0000","last_active":"2024-02-12T17:05:45.591265+0000","last_peered":"2024-02-12T17:05:45.591265+0000","last_clean":"2024-02-12T17:05:45.591265+0000","last_became_active":"2024-02-12T17:03:50.588844+0000","last_became_peered":"2024-02-12T17:03:50.588844+0000","last_unstale":"2024-02-12T17:05:45.591265+0000","last_undegraded":"2024-02-12T17:05:45.591265+0000","last_fullsized":"2024-02-12T17:05:45.591265+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.888942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.888942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":133,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591122+0000","last_change":"2024-02-12T17:04:42.611814+0000","last_active":"2024-02-12T17:05:45.591122+0000","last_peered":"2024-02-12T17:05:45.591122+0000","last_clean":"2024-02-12T17:05:45.591122+0000","last_became_active":"2024-02-12T17:04:39.891435+0000","last_became_peered":"2024-02-12T17:04:39.891435+0000","last_unstale":"2024-02-12T17:05:45.591122+0000","last_undegraded":"2024-02-12T17:05:45.591122+0000","last_fullsized":"2024-02-12T17:05:45.591122+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"438'16965","reported_seq":17107,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:48.654103+0000","last_change":"2024-02-12T17:01:16.913292+0000","last_active":"2024-02-12T17:05:48.654103+0000","last_peered":"2024-02-12T17:05:48.654103+0000","last_clean":"2024-02-12T17:05:48.654103+0000","last_became_active":"2024-02-12T17:01:16.912605+0000","last_became_peered":"2024-02-12T17:01:16.912605+0000","last_unstale":"2024-02-12T17:05:48.654103+0000","last_undegraded":"2024-02-12T17:05:48.654103+0000","last_fullsized":"2024-02-12T17:05:48.654103+0000","mapping_epoch":307,"log_start":"331'6900","ondisk_log_start":"331'6900","created":304,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":10065,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16965,"num_write_kb":548608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":127,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.599819+0000","last_change":"2024-02-12T17:04:39.875038+0000","last_active":"2024-02-12T17:05:45.599819+0000","last_peered":"2024-02-12T17:05:45.599819+0000","last_clean":"2024-02-12T17:05:45.599819+0000","last_became_active":"2024-02-12T17:04:39.874748+0000","last_became_peered":"2024-02-12T17:04:39.874748+0000","last_unstale":"2024-02-12T17:05:45.599819+0000","last_undegraded":"2024-02-12T17:05:45.599819+0000","last_fullsized":"2024-02-12T17:05:45.599819+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:36.798239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T16:57:36.798239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":16,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.586708+0000","last_change":"2024-02-12T17:05:40.443877+0000","last_active":"2024-02-12T17:05:45.586708+0000","last_peered":"2024-02-12T17:05:45.586708+0000","last_clean":"2024-02-12T17:05:45.586708+0000","last_became_active":"2024-02-12T17:05:40.443276+0000","last_became_peered":"2024-02-12T17:05:40.443276+0000","last_unstale":"2024-02-12T17:05:45.586708+0000","last_undegraded":"2024-02-12T17:05:45.586708+0000","last_fullsized":"2024-02-12T17:05:45.586708+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:19.084549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.084549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":416,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.583256+0000","last_change":"2024-02-12T17:04:39.888763+0000","last_active":"2024-02-12T17:05:45.583256+0000","last_peered":"2024-02-12T17:05:45.583256+0000","last_clean":"2024-02-12T17:05:45.583256+0000","last_became_active":"2024-02-12T17:04:39.888466+0000","last_became_peered":"2024-02-12T17:04:39.888466+0000","last_unstale":"2024-02-12T17:05:45.583256+0000","last_undegraded":"2024-02-12T17:05:45.583256+0000","last_fullsized":"2024-02-12T17:05:45.583256+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.858955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.858955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":152,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.580733+0000","last_change":"2024-02-12T17:04:41.383385+0000","last_active":"2024-02-12T17:05:45.580733+0000","last_peered":"2024-02-12T17:05:45.580733+0000","last_clean":"2024-02-12T17:05:45.580733+0000","last_became_active":"2024-02-12T17:04:39.887688+0000","last_became_peered":"2024-02-12T17:04:39.887688+0000","last_unstale":"2024-02-12T17:05:45.580733+0000","last_undegraded":"2024-02-12T17:05:45.580733+0000","last_fullsized":"2024-02-12T17:05:45.580733+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:01:35.160145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:35.160145+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"438'16637","reported_seq":16795,"reported_epoch":438,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:05:48.226133+0000","last_change":"2024-02-12T17:05:11.130524+0000","last_active":"2024-02-12T17:05:48.226133+0000","last_peered":"2024-02-12T17:05:48.226133+0000","last_clean":"2024-02-12T17:03:49.522755+0000","last_became_active":"2024-02-12T17:03:51.663049+0000","last_became_peered":"2024-02-12T17:03:51.663049+0000","last_unstale":"2024-02-12T17:05:48.226133+0000","last_undegraded":"2024-02-12T17:05:48.226133+0000","last_fullsized":"2024-02-12T17:05:48.226133+0000","mapping_epoch":348,"log_start":"379'13509","ondisk_log_start":"379'13509","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3128,"ondisk_log_size":3128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":209,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16637,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":209}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":450,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.578900+0000","last_change":"2024-02-12T17:04:50.393748+0000","last_active":"2024-02-12T17:05:45.578900+0000","last_peered":"2024-02-12T17:05:45.578900+0000","last_clean":"2024-02-12T17:05:45.578900+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:05:45.578900+0000","last_undegraded":"2024-02-12T17:05:45.578900+0000","last_fullsized":"2024-02-12T17:05:45.578900+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:26.862078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T16:57:26.862078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":343,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.587027+0000","last_change":"2024-02-12T17:03:28.770200+0000","last_active":"2024-02-12T17:05:45.587027+0000","last_peered":"2024-02-12T17:05:45.587027+0000","last_clean":"2024-02-12T17:05:45.587027+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:05:45.587027+0000","last_undegraded":"2024-02-12T17:05:45.587027+0000","last_fullsized":"2024-02-12T17:05:45.587027+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:57:30.754652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T16:57:30.754652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":160,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581419+0000","last_change":"2024-02-12T17:03:50.552102+0000","last_active":"2024-02-12T17:05:45.581419+0000","last_peered":"2024-02-12T17:05:45.581419+0000","last_clean":"2024-02-12T17:05:45.581419+0000","last_became_active":"2024-02-12T17:03:50.551804+0000","last_became_peered":"2024-02-12T17:03:50.551804+0000","last_unstale":"2024-02-12T17:05:45.581419+0000","last_undegraded":"2024-02-12T17:05:45.581419+0000","last_fullsized":"2024-02-12T17:05:45.581419+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.794228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":345,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582362+0000","last_change":"2024-02-12T17:03:52.389245+0000","last_active":"2024-02-12T17:05:45.582362+0000","last_peered":"2024-02-12T17:05:45.582362+0000","last_clean":"2024-02-12T17:05:45.582362+0000","last_became_active":"2024-02-12T17:03:50.574562+0000","last_became_peered":"2024-02-12T17:03:50.574562+0000","last_unstale":"2024-02-12T17:05:45.582362+0000","last_undegraded":"2024-02-12T17:05:45.582362+0000","last_fullsized":"2024-02-12T17:05:45.582362+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:02:45.594719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:45.594719+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"438'17139","reported_seq":26129,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:48.318032+0000","last_change":"2024-02-12T17:05:11.339984+0000","last_active":"2024-02-12T17:05:48.318032+0000","last_peered":"2024-02-12T17:05:48.318032+0000","last_clean":"2024-02-12T17:05:48.318032+0000","last_became_active":"2024-02-12T17:05:11.322885+0000","last_became_peered":"2024-02-12T17:05:11.322885+0000","last_unstale":"2024-02-12T17:05:48.318032+0000","last_undegraded":"2024-02-12T17:05:48.318032+0000","last_fullsized":"2024-02-12T17:05:48.318032+0000","mapping_epoch":404,"log_start":"383'14093","ondisk_log_start":"383'14093","created":304,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":3046,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4404,"num_bytes_recovered":288620544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":136,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.582963+0000","last_change":"2024-02-12T17:04:46.024414+0000","last_active":"2024-02-12T17:05:45.582963+0000","last_peered":"2024-02-12T17:05:45.582963+0000","last_clean":"2024-02-12T17:05:45.582963+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:05:45.582963+0000","last_undegraded":"2024-02-12T17:05:45.582963+0000","last_fullsized":"2024-02-12T17:05:45.582963+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.986137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.986137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":346,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.590756+0000","last_change":"2024-02-12T17:04:39.886902+0000","last_active":"2024-02-12T17:05:45.590756+0000","last_peered":"2024-02-12T17:05:45.590756+0000","last_clean":"2024-02-12T17:05:45.590756+0000","last_became_active":"2024-02-12T17:04:39.886403+0000","last_became_peered":"2024-02-12T17:04:39.886403+0000","last_unstale":"2024-02-12T17:05:45.590756+0000","last_undegraded":"2024-02-12T17:05:45.590756+0000","last_fullsized":"2024-02-12T17:05:45.590756+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":117,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591463+0000","last_change":"2024-02-12T17:04:41.654733+0000","last_active":"2024-02-12T17:05:45.591463+0000","last_peered":"2024-02-12T17:05:45.591463+0000","last_clean":"2024-02-12T17:05:45.591463+0000","last_became_active":"2024-02-12T17:04:39.889999+0000","last_became_peered":"2024-02-12T17:04:39.889999+0000","last_unstale":"2024-02-12T17:05:45.591463+0000","last_undegraded":"2024-02-12T17:05:45.591463+0000","last_fullsized":"2024-02-12T17:05:45.591463+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:00:50.756578+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.756578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":393,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.591801+0000","last_change":"2024-02-12T17:04:42.178594+0000","last_active":"2024-02-12T17:05:45.591801+0000","last_peered":"2024-02-12T17:05:45.591801+0000","last_clean":"2024-02-12T17:05:45.591801+0000","last_became_active":"2024-02-12T17:04:39.886809+0000","last_became_peered":"2024-02-12T17:04:39.886809+0000","last_unstale":"2024-02-12T17:05:45.591801+0000","last_undegraded":"2024-02-12T17:05:45.591801+0000","last_fullsized":"2024-02-12T17:05:45.591801+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:28.096708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.096708+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"438'17256","reported_seq":17397,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T17:05:48.673444+0000","last_change":"2024-02-12T17:01:13.886545+0000","last_active":"2024-02-12T17:05:48.673444+0000","last_peered":"2024-02-12T17:05:48.673444+0000","last_clean":"2024-02-12T17:05:48.673444+0000","last_became_active":"2024-02-12T17:01:13.886238+0000","last_became_peered":"2024-02-12T17:01:13.886238+0000","last_unstale":"2024-02-12T17:05:48.673444+0000","last_undegraded":"2024-02-12T17:05:48.673444+0000","last_fullsized":"2024-02-12T17:05:48.673444+0000","mapping_epoch":304,"log_start":"331'7200","ondisk_log_start":"331'7200","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:12.866317+0000","last_clean_scrub_stamp":"2024-02-12T17:01:12.866317+0000","log_size":10056,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14548992,"num_objects":222,"num_object_clones":0,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17256,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":425,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581546+0000","last_change":"2024-02-12T17:04:41.899891+0000","last_active":"2024-02-12T17:05:45.581546+0000","last_peered":"2024-02-12T17:05:45.581546+0000","last_clean":"2024-02-12T17:05:45.581546+0000","last_became_active":"2024-02-12T17:04:41.899598+0000","last_became_peered":"2024-02-12T17:04:41.899598+0000","last_unstale":"2024-02-12T17:05:45.581546+0000","last_undegraded":"2024-02-12T17:05:45.581546+0000","last_fullsized":"2024-02-12T17:05:45.581546+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:26.803903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.803903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":100,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.595643+0000","last_change":"2024-02-12T17:03:50.560958+0000","last_active":"2024-02-12T17:05:45.595643+0000","last_peered":"2024-02-12T17:05:45.595643+0000","last_clean":"2024-02-12T17:05:45.595643+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:05:45.595643+0000","last_undegraded":"2024-02-12T17:05:45.595643+0000","last_fullsized":"2024-02-12T17:05:45.595643+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:01.789464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:01.789464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":507,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T17:05:45.581642+0000","last_change":"2024-02-12T17:04:40.522253+0000","last_active":"2024-02-12T17:05:45.581642+0000","last_peered":"2024-02-12T17:05:45.581642+0000","last_clean":"2024-02-12T17:05:45.581642+0000","last_became_active":"2024-02-12T17:04:39.882260+0000","last_became_peered":"2024-02-12T17:04:39.882260+0000","last_unstale":"2024-02-12T17:05:45.581642+0000","last_undegraded":"2024-02-12T17:05:45.581642+0000","last_fullsized":"2024-02-12T17:05:45.581642+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:02:43.665248+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.665248+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"380'17","reported_seq":676,"reported_epoch":438,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:05:46.341911+0000","last_change":"2024-02-12T17:05:09.874904+0000","last_active":"2024-02-12T17:05:46.341911+0000","last_peered":"2024-02-12T17:05:46.341911+0000","last_clean":"2024-02-12T17:03:47.426268+0000","last_became_active":"2024-02-12T17:04:40.872513+0000","last_became_peered":"2024-02-12T17:04:40.872513+0000","last_unstale":"2024-02-12T17:05:46.341911+0000","last_undegraded":"2024-02-12T17:05:46.341911+0000","last_fullsized":"2024-02-12T17:05:46.341911+0000","mapping_epoch":378,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-02-12T17:00:49.304784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.304784+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,2,4],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":213843992,"num_objects":3264,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":271912,"num_write_kb":8805569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15590,"num_bytes_recovered":1013776408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":893206528,"data_stored":893190240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82103,"ondisk_log_size":82103,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":6},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":277,"num_read_kb":272,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1093,"num_read_kb":850,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":11090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234181,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1073220,"kb_used_data":181756,"kb_used_omap":0,"kb_used_meta":891456,"kb_avail":92671932,"statfs":{"total":95995035648,"available":94896058368,"internally_reserved":0,"allocated":186118144,"data_stored":185358236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912850944},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":3.7330000000000001,"5min":2.6800000000000002,"15min":1.837},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":8.6120000000000001,"5min":16.759,"15min":36.439},"last":4.5060000000000002},{"interface":"front","average":{"1min":2.7919999999999998,"5min":2.423,"15min":1.887},"min":{"1min":0.38500000000000001,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":8.0800000000000001,"5min":17.844000000000001,"15min":36.490000000000002},"last":6.3879999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":4.5330000000000004,"5min":2.2370000000000001,"15min":1.575},"min":{"1min":0.41599999999999998,"5min":0.23400000000000001,"15min":0.20899999999999999},"max":{"1min":9.6980000000000004,"5min":9.6980000000000004,"15min":12.032},"last":2.617},{"interface":"front","average":{"1min":3.1760000000000002,"5min":2.3380000000000001,"15min":1.5740000000000001},"min":{"1min":0.42599999999999999,"5min":0.25800000000000001,"15min":0.22700000000000001},"max":{"1min":8.1170000000000009,"5min":17.324999999999999,"15min":17.324999999999999},"last":6.3220000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":2.9369999999999998,"5min":2.2029999999999998,"15min":1.448},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":6.7290000000000001,"5min":12.4,"15min":12.4},"last":0.27100000000000002},{"interface":"front","average":{"1min":4.4740000000000002,"5min":2.274,"15min":1.716},"min":{"1min":0.28000000000000003,"5min":0.26400000000000001,"15min":0.23000000000000001},"max":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":20.338000000000001},"last":4.742}]},{"osd":3,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":3.5369999999999999,"5min":2.3889999999999998,"15min":1.762},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.6489999999999991,"5min":13.608000000000001,"15min":20.433},"last":2.6859999999999999},{"interface":"front","average":{"1min":4.6260000000000003,"5min":2.6480000000000001,"15min":2.028},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.25600000000000001},"max":{"1min":9.7200000000000006,"5min":31.152000000000001,"15min":31.152000000000001},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":4.8440000000000003,"5min":3.177,"15min":1.8720000000000001},"min":{"1min":0.435,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":10.138,"5min":19.030000000000001,"15min":50.406999999999996},"last":0.246},{"interface":"front","average":{"1min":5.5979999999999999,"5min":3.2109999999999999,"15min":2.1640000000000001},"min":{"1min":0.33500000000000002,"5min":0.27000000000000002,"15min":0.23400000000000001},"max":{"1min":11.191000000000001,"5min":16.867000000000001,"15min":50.168999999999997},"last":0.308}]},{"osd":5,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":5.9130000000000003,"5min":3.8319999999999999,"15min":1.9319999999999999},"min":{"1min":0.39300000000000002,"5min":0.26800000000000002,"15min":0.24199999999999999},"max":{"1min":14.074,"5min":19.702000000000002,"15min":19.702000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":6.1769999999999996,"5min":3.2229999999999999,"15min":1.9850000000000001},"min":{"1min":0.308,"5min":0.28499999999999998,"15min":0.26200000000000001},"max":{"1min":17.585999999999999,"5min":18.651,"15min":18.651},"last":0.376}]},{"osd":6,"last update":"Mon Feb 12 17:04:57 2024","interfaces":[{"interface":"back","average":{"1min":3.8919999999999999,"5min":2.8399999999999999,"15min":1.6779999999999999},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":8.1839999999999993,"5min":17.234999999999999,"15min":17.234999999999999},"last":1.4630000000000001},{"interface":"front","average":{"1min":3.6389999999999998,"5min":2.681,"15min":1.6160000000000001},"min":{"1min":0.40899999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.5549999999999997,"5min":13.676,"15min":13.676},"last":1.4890000000000001}]}]},{"osd":6,"up_from":0,"seq":193273528522,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738599,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1108488,"kb_used_data":216384,"kb_used_omap":0,"kb_used_meta":892096,"kb_avail":92636664,"statfs":{"total":95995035648,"available":94859943936,"internally_reserved":0,"allocated":221577216,"data_stored":220940480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913506304},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:05:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3119999999999998,"5min":1.046,"15min":1.3819999999999999},"min":{"1min":0.29699999999999999,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":6.2939999999999996,"5min":6.2939999999999996,"15min":23.510999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":2.8439999999999999,"5min":1.2729999999999999,"15min":1.385},"min":{"1min":0.27700000000000002,"5min":0.23000000000000001,"15min":0.193},"max":{"1min":8.2400000000000002,"5min":16.16,"15min":16.16},"last":10.009}]},{"osd":2,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":0.82299999999999995,"15min":1.464},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.7750000000000004,"5min":5.7750000000000004,"15min":16.521000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.1100000000000001,"15min":1.381},"min":{"1min":0.26800000000000002,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":9.827,"5min":9.827,"15min":10.340999999999999},"last":3.4929999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:05:32 2024","interfaces":[{"interface":"back","average":{"1min":2.1680000000000001,"5min":1.296,"15min":1.673},"min":{"1min":0.33000000000000002,"5min":0.23499999999999999,"15min":0.19400000000000001},"max":{"1min":5.7670000000000003,"5min":6.923,"15min":21.100000000000001},"last":9.8819999999999997},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.5109999999999999,"15min":1.73},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":6.4249999999999998,"5min":16.759,"15min":16.759},"last":0.45100000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:05:46 2024","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":1.55,"15min":1.641},"min":{"1min":0.28599999999999998,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":7.8639999999999999,"5min":7.8639999999999999,"15min":16.888999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":2.9390000000000001,"5min":1.7889999999999999,"15min":1.6639999999999999},"min":{"1min":0.33100000000000002,"5min":0.28799999999999998,"15min":0.28399999999999997},"max":{"1min":7.8540000000000001,"5min":9.1020000000000003,"15min":16.899999999999999},"last":3.5329999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:05:16 2024","interfaces":[{"interface":"back","average":{"1min":2.726,"5min":2.5619999999999998,"15min":2.1200000000000001},"min":{"1min":0.38300000000000001,"5min":0.30199999999999999,"15min":0.26400000000000001},"max":{"1min":10.146000000000001,"5min":50.404000000000003,"15min":50.404000000000003},"last":9.1769999999999996},{"interface":"front","average":{"1min":3.0449999999999999,"5min":2.4100000000000001,"15min":2.0369999999999999},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.3550000000000004,"5min":49.979999999999997,"15min":49.979999999999997},"last":19.751999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:05:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6920000000000002,"5min":1.585,"15min":1.675},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.29999999999999999},"max":{"1min":5.6849999999999996,"5min":7.4489999999999998,"15min":17.907},"last":8.9570000000000007},{"interface":"front","average":{"1min":2.069,"5min":1.8919999999999999,"15min":1.954},"min":{"1min":0.39300000000000002,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":4.4740000000000002,"5min":7.7510000000000003,"15min":21.062000000000001},"last":3.351}]},{"osd":7,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":3.5699999999999998,"5min":2.375,"15min":1.863},"min":{"1min":0.498,"5min":0.40699999999999997,"15min":0.32400000000000001},"max":{"1min":12.836,"5min":19.373000000000001,"15min":19.373000000000001},"last":9.3480000000000008},{"interface":"front","average":{"1min":2.8370000000000002,"5min":2.3050000000000002,"15min":1.8440000000000001},"min":{"1min":0.442,"5min":0.39900000000000002,"15min":0.33600000000000002},"max":{"1min":6.3550000000000004,"5min":15.550000000000001,"15min":34.722999999999999},"last":8.923}]}]},{"osd":2,"up_from":18,"seq":77309411548,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039704,"kb_used_data":134736,"kb_used_omap":0,"kb_used_meta":904960,"kb_avail":92705448,"statfs":{"total":95995035648,"available":94930378752,"internally_reserved":0,"allocated":137969664,"data_stored":137253330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":926679040},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1220000000000001,"15min":1.589},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.8339999999999996,"5min":36.100999999999999,"15min":36.100999999999999},"last":0.309},{"interface":"front","average":{"1min":1.859,"5min":1.5049999999999999,"15min":1.5569999999999999},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.158},"max":{"1min":7.548,"5min":36.299999999999997,"15min":36.299999999999997},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.64600000000000002,"15min":1.49},"min":{"1min":0.247,"5min":0.215,"15min":0.191},"max":{"1min":1.3560000000000001,"5min":5.0890000000000004,"15min":17.739000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.4950000000000001,"5min":0.97899999999999998,"15min":1.591},"min":{"1min":0.28000000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":7.8609999999999998,"5min":7.8609999999999998,"15min":15.766999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:05:41 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.1459999999999999,"15min":1.7929999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":8.0449999999999999,"5min":8.0449999999999999,"15min":51.579999999999998},"last":0.379},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.149,"15min":1.55},"min":{"1min":0.32400000000000001,"5min":0.23699999999999999,"15min":0.23200000000000001},"max":{"1min":5.4180000000000001,"5min":5.4180000000000001,"15min":48.988999999999997},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.7809999999999999,"15min":1.7430000000000001},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":5.8559999999999999,"5min":18.658999999999999,"15min":18.658999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.8680000000000001,"5min":2.1720000000000002,"15min":1.7749999999999999},"min":{"1min":0.38,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":6.2220000000000004,"5min":18.355,"15min":18.355},"last":0.437}]},{"osd":5,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":2.0699999999999998,"15min":1.877},"min":{"1min":0.308,"5min":0.308,"15min":0.27300000000000002},"max":{"1min":10.728999999999999,"5min":32.57,"15min":32.57},"last":3.7109999999999999},{"interface":"front","average":{"1min":2.6309999999999998,"5min":2.3999999999999999,"15min":2.1179999999999999},"min":{"1min":0.40400000000000003,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":9.8420000000000005,"5min":31.905999999999999,"15min":31.905999999999999},"last":1.28}]},{"osd":6,"last update":"Mon Feb 12 17:05:41 2024","interfaces":[{"interface":"back","average":{"1min":2.2650000000000001,"5min":1.494,"15min":2.0009999999999999},"min":{"1min":0.45600000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.7199999999999998,"5min":7.6669999999999998,"15min":17.782},"last":0.53000000000000003},{"interface":"front","average":{"1min":2.5939999999999999,"5min":1.621,"15min":1.7729999999999999},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.29799999999999999},"max":{"1min":10.6,"5min":10.6,"15min":13.446999999999999},"last":0.46500000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":4.1719999999999997,"5min":3.2090000000000001,"15min":2.258},"min":{"1min":0.57999999999999996,"5min":0.33700000000000002,"15min":0.32600000000000001},"max":{"1min":10.754,"5min":45.468000000000004,"15min":45.468000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.3969999999999998,"5min":2.9870000000000001,"15min":2.331},"min":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":6.7000000000000002,"5min":46.656999999999996,"15min":46.656999999999996},"last":0.55100000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149912,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347636,"kb_used_data":46252,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93397516,"statfs":{"total":95995035648,"available":95639056384,"internally_reserved":0,"allocated":47362048,"data_stored":46618395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:05:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.401,"15min":1.6319999999999999},"min":{"1min":0.20899999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":6.2190000000000003,"5min":58.451999999999998,"15min":58.451999999999998},"last":0.37},{"interface":"front","average":{"1min":3.5459999999999998,"5min":1.8200000000000001,"15min":1.7350000000000001},"min":{"1min":0.251,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":19.748999999999999,"5min":57.497999999999998,"15min":57.497999999999998},"last":3.0960000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:05:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6240000000000001,"5min":1.1970000000000001,"15min":1.4710000000000001},"min":{"1min":0.28100000000000003,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":11.537000000000001,"5min":11.537000000000001,"15min":13.94},"last":3.0630000000000002},{"interface":"front","average":{"1min":1.5680000000000001,"5min":0.97399999999999998,"15min":1.3120000000000001},"min":{"1min":0.29399999999999998,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":10.278,"5min":10.278,"15min":14.962},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:05:34 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.153,"15min":1.611},"min":{"1min":0.30199999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.302,"5min":10.302,"15min":49.112000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":0.91900000000000004,"15min":1.6839999999999999},"min":{"1min":0.23699999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.4960000000000004,"5min":6.4960000000000004,"15min":51.997999999999998},"last":0.39900000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.722,"15min":1.7070000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":4.8710000000000004,"5min":11.047000000000001,"15min":19.010000000000002},"last":4.0110000000000001},{"interface":"front","average":{"1min":2.903,"5min":2.0790000000000002,"15min":1.655},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":10.250999999999999,"5min":19.033999999999999,"15min":19.033999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Mon Feb 12 17:05:16 2024","interfaces":[{"interface":"back","average":{"1min":2.2130000000000001,"5min":2.048,"15min":1.9419999999999999},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.9589999999999996,"5min":14.236000000000001,"15min":19.882999999999999},"last":8.1850000000000005},{"interface":"front","average":{"1min":3.4590000000000001,"5min":2.423,"15min":2.0209999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.33300000000000002},"max":{"1min":12.502000000000001,"5min":13.927,"15min":15.595000000000001},"last":3.0339999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:05:34 2024","interfaces":[{"interface":"back","average":{"1min":2.548,"5min":1.665,"15min":1.5529999999999999},"min":{"1min":0.39500000000000002,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":10.212999999999999,"5min":10.212999999999999,"15min":12.856999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":1.6819999999999999},"min":{"1min":0.40100000000000002,"5min":0.34999999999999998,"15min":0.27500000000000002},"max":{"1min":8.5039999999999996,"5min":12.635999999999999,"15min":19.925999999999998},"last":8.1489999999999991}]},{"osd":7,"last update":"Mon Feb 12 17:04:51 2024","interfaces":[{"interface":"back","average":{"1min":1.994,"5min":2.2989999999999999,"15min":1.907},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":5.585,"5min":12.695,"15min":12.891},"last":3.1539999999999999},{"interface":"front","average":{"1min":2.617,"5min":2.4529999999999998,"15min":1.923},"min":{"1min":0.49099999999999999,"5min":0.40799999999999997,"15min":0.32800000000000001},"max":{"1min":5.1369999999999996,"5min":25.838999999999999,"15min":25.838999999999999},"last":3.5139999999999998}]}]},{"osd":4,"up_from":0,"seq":137438953684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757456,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090808,"kb_used_data":190128,"kb_used_omap":0,"kb_used_meta":900672,"kb_avail":92654344,"statfs":{"total":95995035648,"available":94878048256,"internally_reserved":0,"allocated":194691072,"data_stored":193942472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922288128},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":2.202,"15min":1.7430000000000001},"min":{"1min":0.23899999999999999,"5min":0.20399999999999999,"15min":0.20300000000000001},"max":{"1min":10.971,"5min":46.365000000000002,"15min":46.365000000000002},"last":0.23699999999999999},{"interface":"front","average":{"1min":2.7229999999999999,"5min":2.5419999999999998,"15min":2.101},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":12.140000000000001,"5min":46.134,"15min":46.134},"last":5.4859999999999998}]},{"osd":1,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":2.573,"5min":1.802,"15min":1.6299999999999999},"min":{"1min":0.33000000000000002,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":11.19,"5min":11.19,"15min":16.655000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":3.1739999999999999,"5min":2.1789999999999998,"15min":1.8839999999999999},"min":{"1min":0.26300000000000001,"5min":0.23799999999999999,"15min":0.21199999999999999},"max":{"1min":12.154,"5min":13.130000000000001,"15min":16.712},"last":5.556}]},{"osd":2,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":3.1400000000000001,"5min":2.04,"15min":1.8999999999999999},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":11.007999999999999,"5min":11.15,"15min":18.218},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.2669999999999999,"5min":1.998,"15min":1.788},"min":{"1min":0.32500000000000001,"5min":0.25,"15min":0.25},"max":{"1min":6.0839999999999996,"5min":9.1199999999999992,"15min":18.327000000000002},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8860000000000001,"5min":1.923,"15min":1.784},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":11.045,"5min":11.122,"15min":18.251000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":2.4449999999999998,"5min":1.925,"15min":1.7},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.1600000000000001,"5min":8.0790000000000006,"15min":20.263000000000002},"last":4.2380000000000004}]},{"osd":4,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.6850000000000001,"15min":1.754},"min":{"1min":0.308,"5min":0.26400000000000001,"15min":0.22},"max":{"1min":12.656000000000001,"5min":12.656000000000001,"15min":12.656000000000001},"last":5.5190000000000001},{"interface":"front","average":{"1min":3.528,"5min":2.6309999999999998,"15min":1.913},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.24299999999999999},"max":{"1min":11.068,"5min":20.550000000000001,"15min":20.550000000000001},"last":2.4769999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:05:44 2024","interfaces":[{"interface":"back","average":{"1min":3.044,"5min":2.2429999999999999,"15min":1.78},"min":{"1min":0.33200000000000002,"5min":0.29199999999999998,"15min":0.26400000000000001},"max":{"1min":10.497,"5min":12.121,"15min":12.281000000000001},"last":5.5949999999999998},{"interface":"front","average":{"1min":2.1200000000000001,"5min":2.2519999999999998,"15min":1.8400000000000001},"min":{"1min":0.314,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":9.5020000000000007,"5min":13.916,"15min":22.035},"last":0.41099999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:05:11 2024","interfaces":[{"interface":"back","average":{"1min":4.0869999999999997,"5min":3.1920000000000002,"15min":1.8859999999999999},"min":{"1min":0.375,"5min":0.29499999999999998,"15min":0.27500000000000002},"max":{"1min":13.647,"5min":19.184999999999999,"15min":19.184999999999999},"last":2.0379999999999998},{"interface":"front","average":{"1min":4.7640000000000002,"5min":3.1869999999999998,"15min":1.9450000000000001},"min":{"1min":0.38800000000000001,"5min":0.30499999999999999,"15min":0.26800000000000002},"max":{"1min":20.274000000000001,"5min":20.274000000000001,"15min":20.274000000000001},"last":5.8040000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186843136,"data_stored":186843136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122687488,"data_stored":122683416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44302336,"data_stored":44302336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":198774784,"data_stored":198770712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":184553472,"data_stored":184549400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":156045312,"data_stored":156041240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:05:52.590 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -7, currently 3 2024-02-12T17:05:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:52.591 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:52.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:05:52.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:52.831 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:05:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:05:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:05:52.831 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:51.331461+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:53.068 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:53.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:53.107 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:51.331517+0000 osd.1 (osd.1) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:53.107 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:52.336574+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e439: 8 total, 8 up, 5 in 2024-02-12T17:05:53.173 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:53.208 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:53.274 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:51.331461+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:53.274 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:51.331517+0000 osd.1 (osd.1) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:53.274 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:52.336574+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e439: 8 total, 8 up, 5 in 2024-02-12T17:05:53.275 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:51.331461+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:05:53.275 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:51.331517+0000 osd.1 (osd.1) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:05:53.275 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:52.336574+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e439: 8 total, 8 up, 5 in 2024-02-12T17:05:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:53.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:53.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:53.417 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:05:53.455 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:05:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:53.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:05:53.842 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:53.883 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:54.004 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Removed 137587 objects 2024-02-12T17:05:54.004 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Clean up completed and total clean up time :174.487 2024-02-12T17:05:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:52.683532+0000 mgr.y (mgr.24481) 617 : cluster [DBG] pgmap v848: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 204 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 721 B/s rd, 0 B/s wr, 522 op/s; 2154/10368 objects misplaced (20.775%); 2.1 MiB/s, 32 objects/s recovering 2024-02-12T17:05:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:53.340702+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e440: 8 total, 8 up, 5 in 2024-02-12T17:05:54.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:52.683532+0000 mgr.y (mgr.24481) 617 : cluster [DBG] pgmap v848: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 204 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 721 B/s rd, 0 B/s wr, 522 op/s; 2154/10368 objects misplaced (20.775%); 2.1 MiB/s, 32 objects/s recovering 2024-02-12T17:05:54.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:53.340702+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e440: 8 total, 8 up, 5 in 2024-02-12T17:05:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:54.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:54.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:54.193 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:54.220 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:54.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:52.683532+0000 mgr.y (mgr.24481) 617 : cluster [DBG] pgmap v848: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 113 active+clean; 204 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 721 B/s rd, 0 B/s wr, 522 op/s; 2154/10368 objects misplaced (20.775%); 2.1 MiB/s, 32 objects/s recovering 2024-02-12T17:05:54.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:53.340702+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e440: 8 total, 8 up, 5 in 2024-02-12T17:05:54.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:05:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:05:54.767 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:05:54.768 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:05:55.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T17:05:55.480 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:05:55.496 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:05:55.513 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:54.308873+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:54.311801+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:54.343217+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e441: 8 total, 8 up, 5 in 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:54.403173+0000 mgr.y (mgr.24481) 618 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:54.683959+0000 mgr.y (mgr.24481) 619 : cluster [DBG] pgmap v851: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 117 active+clean; 19 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 841 op/s; 132/1506 objects misplaced (8.765%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:55.024082+0000 osd.5 (osd.5) 193 : cluster [DBG] 2.18 scrub starts 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:55.027182+0000 osd.5 (osd.5) 194 : cluster [DBG] 2.18 scrub ok 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:54.308873+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:54.311801+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:05:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:54.343217+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e441: 8 total, 8 up, 5 in 2024-02-12T17:05:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:54.403173+0000 mgr.y (mgr.24481) 618 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:54.683959+0000 mgr.y (mgr.24481) 619 : cluster [DBG] pgmap v851: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 117 active+clean; 19 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 841 op/s; 132/1506 objects misplaced (8.765%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T17:05:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:55.024082+0000 osd.5 (osd.5) 193 : cluster [DBG] 2.18 scrub starts 2024-02-12T17:05:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:55.027182+0000 osd.5 (osd.5) 194 : cluster [DBG] 2.18 scrub ok 2024-02-12T17:05:55.531 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:05:55.549 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:05:55.568 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:05:55.570 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:55.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:55.587 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:05:55.605 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:05:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:05:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:05:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:05:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:05:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:05:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:05:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:05:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:05:55.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:55.707 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:05:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:54.308873+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:05:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:54.311801+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:05:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:54.343217+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e441: 8 total, 8 up, 5 in 2024-02-12T17:05:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:54.403173+0000 mgr.y (mgr.24481) 618 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:05:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:54.683959+0000 mgr.y (mgr.24481) 619 : cluster [DBG] pgmap v851: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 117 active+clean; 19 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 841 op/s; 132/1506 objects misplaced (8.765%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T17:05:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:55.024082+0000 osd.5 (osd.5) 193 : cluster [DBG] 2.18 scrub starts 2024-02-12T17:05:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:55.027182+0000 osd.5 (osd.5) 194 : cluster [DBG] 2.18 scrub ok 2024-02-12T17:05:55.910 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:55.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:56.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:05:56.044 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:56.361 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:05:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:55.341287+0000 osd.3 (osd.3) 119 : cluster [DBG] 4.e scrub starts 2024-02-12T17:05:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:55.344990+0000 osd.3 (osd.3) 120 : cluster [DBG] 4.e scrub ok 2024-02-12T17:05:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:55.349379+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e442: 8 total, 8 up, 5 in 2024-02-12T17:05:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:55.460735+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.33:0/3967814551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:55.841906+0000 osd.7 (osd.7) 194 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:05:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:55.847921+0000 osd.7 (osd.7) 195 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:05:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:55.887552+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.33:0/778849601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:55.887860+0000 mon.a (mon.0) 1568 : audit [INF] from='client.35690 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.051058+0000 osd.5 (osd.5) 195 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:05:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.053967+0000 osd.5 (osd.5) 196 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:55.341287+0000 osd.3 (osd.3) 119 : cluster [DBG] 4.e scrub starts 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:55.344990+0000 osd.3 (osd.3) 120 : cluster [DBG] 4.e scrub ok 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:55.349379+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e442: 8 total, 8 up, 5 in 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:55.460735+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.33:0/3967814551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:55.841906+0000 osd.7 (osd.7) 194 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:55.847921+0000 osd.7 (osd.7) 195 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:55.887552+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.33:0/778849601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:55.887860+0000 mon.a (mon.0) 1568 : audit [INF] from='client.35690 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.051058+0000 osd.5 (osd.5) 195 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:05:56.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.053967+0000 osd.5 (osd.5) 196 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:55.341287+0000 osd.3 (osd.3) 119 : cluster [DBG] 4.e scrub starts 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:55.344990+0000 osd.3 (osd.3) 120 : cluster [DBG] 4.e scrub ok 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:55.349379+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e442: 8 total, 8 up, 5 in 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:55.460735+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.33:0/3967814551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:55.841906+0000 osd.7 (osd.7) 194 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:55.847921+0000 osd.7 (osd.7) 195 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:55.887552+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.33:0/778849601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:55.887860+0000 mon.a (mon.0) 1568 : audit [INF] from='client.35690 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.051058+0000 osd.5 (osd.5) 195 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:05:56.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.053967+0000 osd.5 (osd.5) 196 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:05:57.094 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:05:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:57.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:05:57.199 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:57.227 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:57.246 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:57.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:57.350 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:05:57.381 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:57.469 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:05:57.470 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:05:57.470 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:05:57.470 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:05:57.470 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:05:57.479 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:05:57.482 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.344913+0000 mon.a (mon.0) 1569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:05:57.482 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:05:56.355250+0000 mon.a (mon.0) 1570 : audit [INF] from='client.35690 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:57.482 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.355372+0000 mon.a (mon.0) 1571 : cluster [DBG] osdmap e443: 8 total, 8 up, 5 in 2024-02-12T17:05:57.482 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.384889+0000 osd.3 (osd.3) 121 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:05:57.482 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.388359+0000 osd.3 (osd.3) 122 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:05:57.482 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.684525+0000 mgr.y (mgr.24481) 620 : cluster [DBG] pgmap v854: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 117 active+clean; 19 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 739 op/s; 132/1506 objects misplaced (8.765%); 3.0 MiB/s, 48 objects/s recovering 2024-02-12T17:05:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:57.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:05:57.584 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:57.591 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:05:57.591 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:05:57.591 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-12T17:05:57.591 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T17:05:57.613 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.344913+0000 mon.a (mon.0) 1569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:05:57.613 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:05:56.355250+0000 mon.a (mon.0) 1570 : audit [INF] from='client.35690 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:57.613 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.355372+0000 mon.a (mon.0) 1571 : cluster [DBG] osdmap e443: 8 total, 8 up, 5 in 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.384889+0000 osd.3 (osd.3) 121 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.388359+0000 osd.3 (osd.3) 122 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.684525+0000 mgr.y (mgr.24481) 620 : cluster [DBG] pgmap v854: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 117 active+clean; 19 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 739 op/s; 132/1506 objects misplaced (8.765%); 3.0 MiB/s, 48 objects/s recovering 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.344913+0000 mon.a (mon.0) 1569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:05:56.355250+0000 mon.a (mon.0) 1570 : audit [INF] from='client.35690 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.355372+0000 mon.a (mon.0) 1571 : cluster [DBG] osdmap e443: 8 total, 8 up, 5 in 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.384889+0000 osd.3 (osd.3) 121 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.388359+0000 osd.3 (osd.3) 122 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:05:57.614 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.684525+0000 mgr.y (mgr.24481) 620 : cluster [DBG] pgmap v854: 128 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 117 active+clean; 19 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 739 op/s; 132/1506 objects misplaced (8.765%); 3.0 MiB/s, 48 objects/s recovering 2024-02-12T17:05:57.616 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:57.637 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:57.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:05:57.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:05:57.769 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:57.894 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:57.929 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-02-12T17:05:57.929 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-02-12T17:05:58.117 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T17:05:58.117 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:05:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:58.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:05:58.221 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:05:58.251 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:05:58.400 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:05:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:58.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:05:58.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:05:58.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.778262+0000 osd.0 (osd.0) 182 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:05:58.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:56.852117+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:05:58.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:57.077155+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.e deep-scrub starts 2024-02-12T17:05:58.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:57.077967+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.e deep-scrub ok 2024-02-12T17:05:58.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:57.886719+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:05:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:57.889951+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:05:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.778262+0000 osd.0 (osd.0) 182 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:05:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:56.852117+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:05:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:57.077155+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.e deep-scrub starts 2024-02-12T17:05:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:57.077967+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.e deep-scrub ok 2024-02-12T17:05:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:57.886719+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:05:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:57.889951+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:05:58.533 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:58.645 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.778262+0000 osd.0 (osd.0) 182 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:05:58.645 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:56.852117+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:05:58.645 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:57.077155+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.e deep-scrub starts 2024-02-12T17:05:58.645 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:57.077967+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.e deep-scrub ok 2024-02-12T17:05:58.645 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:57.886719+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:05:58.645 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:57.889951+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:05:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:05:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:58.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:05:58.663 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:05:58.757 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:05:58.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:05:58.789 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:58.813 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:05:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:58.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:05:58.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:05:58.950 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:59.099 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:05:59.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:05:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:58.685076+0000 mgr.y (mgr.24481) 621 : cluster [DBG] pgmap v855: 128 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 124 active+clean; 5.1 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 957 B/s rd, 0 B/s wr, 382 op/s; 42/822 objects misplaced (5.109%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:05:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:05:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:58.685076+0000 mgr.y (mgr.24481) 621 : cluster [DBG] pgmap v855: 128 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 124 active+clean; 5.1 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 957 B/s rd, 0 B/s wr, 382 op/s; 42/822 objects misplaced (5.109%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:05:59.648 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:05:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:58.685076+0000 mgr.y (mgr.24481) 621 : cluster [DBG] pgmap v855: 128 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 124 active+clean; 5.1 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 957 B/s rd, 0 B/s wr, 382 op/s; 42/822 objects misplaced (5.109%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:05:59.701 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:05:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:59.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:05:59.807 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:05:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:05:59.986 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:05:59.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:05:59.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:00.090 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:00.119 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:00.238 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:06:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:06:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:06:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:06:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85057 8.b 8:df0e10c0:::benchmark_data_smithi033_105085_object85056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-12T17:06:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.130368+0000", 2024-02-12T17:06:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 494.10362588700002, 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.117763128, 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85057 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130368+0000", 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130368+0000", 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130366+0000", 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.603e-05 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130392+0000", 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7159999999999997e-06 2024-02-12T17:06:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130399+0000", 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130404+0000", 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00032313900000000001 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130727+0000", 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124036 2024-02-12T17:06:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.130851+0000", 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017825299999999999 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.131029+0000", 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024958350000000001 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133525+0000", 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062191540000000002 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.139744+0000", 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017399532999999998 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157144+0000", 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0638e-05 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157154+0000", 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0026200919999999 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159774+0000", 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0857e-05 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159785+0000", 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.088274985 2024-02-12T17:06:00.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248060+0000", 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0550999999999996e-05 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248131+0000", 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.117763128 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85060 8.b 8:d0c504d4:::benchmark_data_smithi033_105085_object85059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.133233+0000", 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 494.10076000999999, 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1475086289999998, 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85060 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133233+0000", 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133233+0000", 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133230+0000", 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6168000000000003e-05 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133266+0000", 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1382999999999999e-05 2024-02-12T17:06:00.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133278+0000", 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7170000000000002e-06 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133284+0000", 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025424699999999998 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133538+0000", 2024-02-12T17:06:00.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112871 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133651+0000", 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015755700000000001 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.133809+0000", 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023381417000000002 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157190+0000", 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7670000000000003e-06 2024-02-12T17:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157198+0000", 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.002615364 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159813+0000", 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1559999999999998e-06 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159821+0000", 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0883366249999999 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248157+0000", 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044692669999999999 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for degraded object", 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.252626+0000", 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028011846999999999 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.280638+0000", 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103669 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.280742+0000", 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1475086289999998 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85062 8.b 8:d12d443b:::benchmark_data_smithi033_105085_object85061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.141630+0000", 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 494.09236317199998, 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.139506715, 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.252 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85062 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141630+0000", 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141630+0000", 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141628+0000", 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4357999999999998e-05 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141672+0000", 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2106000000000001e-05 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141684+0000", 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7390000000000002e-06 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141691+0000", 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3009e-05 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141744+0000", 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012559 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.141869+0000", 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016855700000000001 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.142038+0000", 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015187798000000001 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157226+0000", 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8449999999999997e-06 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.157233+0000", 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.002611798 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159844+0000", 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1400000000000002e-06 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159852+0000", 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0928147509999999 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.252666+0000", 2024-02-12T17:06:00.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038766899999999999 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for degraded object", 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253054+0000", 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028008100000000001 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.281062+0000", 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4779999999999999e-05 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.281137+0000", 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.139506715 2024-02-12T17:06:00.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85069 8.b 8:d240be51:::benchmark_data_smithi033_105085_object85068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.166508+0000", 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 494.06748554699999, 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1456601989999999, 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85069 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.259 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166508+0000", 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166508+0000", 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166506+0000", 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7187000000000002e-05 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166543+0000", 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6649999999999998e-06 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166553+0000", 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3049999999999998e-06 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166559+0000", 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124027 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166683+0000", 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0033e-05 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.166693+0000", 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.993179433 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159873+0000", 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.328e-06 2024-02-12T17:06:00.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.159880+0000", 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0931984909999999 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253079+0000", 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013465999999999999 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253213+0000", 2024-02-12T17:06:00.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051193199999999999 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,7", 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253725+0000", 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0084671769999999993 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.262192+0000", 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047363939999999997 2024-02-12T17:06:00.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.266929+0000", 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030863169999999999 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.270015+0000", 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.041951342000000003 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.311966+0000", 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4244000000000004e-05 2024-02-12T17:06:00.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.312031+0000", 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013736999999999999 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.312168+0000", 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1456601989999999 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220735 8.b 8:dc5cc4b6:::benchmark_data_smithi033_105085_object109077:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.266 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.748274+0000", 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.48571978799998, 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99742273999999997, 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220735 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748274+0000", 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748274+0000", 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748271+0000", 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1210000000000007e-06 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748280+0000", 2024-02-12T17:06:00.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6354e-05 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748296+0000", 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4479999999999999e-06 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748302+0000", 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5645999999999999e-05 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748398+0000", 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2930000000000003e-06 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748405+0000", 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97336656499999996 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.721772+0000", 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012956099999999999 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.721901+0000", 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016388700000000001 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.722065+0000", 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0099480499999999999 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732013+0000", 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013501833 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745515+0000", 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5191000000000004e-05 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745610+0000", 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4626000000000003e-05 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745655+0000", 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1374000000000001e-05 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745696+0000", 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99742273999999997 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220736 8.b 8:d7ee9352:::benchmark_data_smithi033_105085_object109078:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.748314+0000", 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.48567935400001, 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99754634200000003, 2024-02-12T17:06:00.273 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220736 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748314+0000", 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748314+0000", 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748313+0000", 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2619999999999999e-06 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748317+0000", 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1290000000000004e-06 2024-02-12T17:06:00.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748322+0000", 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9429999999999999e-06 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748327+0000", 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7467999999999996e-05 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748424+0000", 2024-02-12T17:06:00.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8879999999999996e-06 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.748429+0000", 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97531739399999995 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.723746+0000", 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115288 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.723862+0000", 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013617699999999999 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.723998+0000", 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080225620000000004 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732020+0000", 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013542134000000001 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745563+0000", 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000208486 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745771+0000", 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7753999999999997e-05 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745819+0000", 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1597000000000002e-05 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745860+0000", 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99754634200000003 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220747 8.b 8:dc732349:::benchmark_data_smithi033_105085_object109089:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.764408+0000", 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.469585833, 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98163293600000001, 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.280 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220747 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764408+0000", 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764408+0000", 2024-02-12T17:06:00.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764405+0000", 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6539999999999999e-06 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764413+0000", 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7130000000000005e-06 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764421+0000", 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0069999999999997e-06 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.764426+0000", 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071895100000000001 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.765145+0000", 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-06 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.765153+0000", 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96046406799999995 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.725617+0000", 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000105274 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.725722+0000", 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014608899999999999 2024-02-12T17:06:00.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.725868+0000", 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061591099999999998 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732027+0000", 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013887171 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745915+0000", 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3433999999999998e-05 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745958+0000", 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3881999999999998e-05 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746002+0000", 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8677999999999998e-05 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746040+0000", 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98163293600000001 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220785 8.b 8:daf40cf0:::benchmark_data_smithi033_105085_object109127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.797112+0000", 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.43688136999998, 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94916868899999995, 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220785 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797112+0000", 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797112+0000", 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:00.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797111+0000", 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3400000000000002e-06 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797115+0000", 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3419999999999998e-06 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797120+0000", 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6969999999999999e-06 2024-02-12T17:06:00.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797124+0000", 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8279999999999999e-05 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797172+0000", 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6473999999999998e-05 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797188+0000", 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.92994460499999998 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.727133+0000", 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013374799999999999 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.727267+0000", 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012754699999999999 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.727394+0000", 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072825829999999996 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734677+0000", 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011405406999999999 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746082+0000", 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114867 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746197+0000", 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3798000000000003e-05 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746241+0000", 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9749999999999997e-05 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746281+0000", 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94916868899999995 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.293 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220787 8.b 8:d18a4d72:::benchmark_data_smithi033_105085_object109129:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.797229+0000", 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.43676412799999, 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94926104300000003, 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220787 2024-02-12T17:06:00.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797229+0000", 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797229+0000", 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797229+0000", 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4499999999999998e-06 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797231+0000", 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5850000000000001e-06 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797236+0000", 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5480000000000002e-06 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797239+0000", 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000140223 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797380+0000", 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6250000000000004e-06 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.797385+0000", 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.931185033 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.728570+0000", 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106504 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.728677+0000", 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014063899999999999 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.728817+0000", 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058666860000000003 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734684+0000", 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011638766 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746323+0000", 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6386999999999998e-05 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746409+0000", 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8921000000000002e-05 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746448+0000", 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2188000000000003e-05 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746490+0000", 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94926104300000003 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220835 8.b 8:dd40dfe1:::benchmark_data_smithi033_105085_object109177:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.300 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.845315+0000", 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.38867790500001, 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.90127469299999996, 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220835 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845315+0000", 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845315+0000", 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845314+0000", 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0217999999999999e-05 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845324+0000", 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1071e-05 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845335+0000", 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5480000000000004e-06 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.845342+0000", 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051822960000000003 2024-02-12T17:06:00.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850525+0000", 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1790000000000008e-06 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850533+0000", 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87922209299999998 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.729755+0000", 2024-02-12T17:06:00.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0649000000000002e-05 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.729816+0000", 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7392999999999995e-05 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.729903+0000", 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047881829999999997 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734691+0000", 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011659630000000001 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746351+0000", 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173017 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746524+0000", 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5839e-05 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746560+0000", 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0505e-05 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746590+0000", 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.90127469299999996 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220840 8.b 8:de442c69:::benchmark_data_smithi033_105085_object109182:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.307 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.850406+0000", 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.38358755600001, 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.89630858400000002, 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220840 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850406+0000", 2024-02-12T17:06:00.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850406+0000", 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850404+0000", 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1919000000000001e-05 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850416+0000", 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0019999999999995e-06 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850425+0000", 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9190000000000004e-06 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850431+0000", 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121835 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850553+0000", 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2240000000000003e-06 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.850558+0000", 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.880290087 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.730848+0000", 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111928 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.730960+0000", 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000146126 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.731106+0000", 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061862439999999996 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737292+0000", 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090883839999999997 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746381+0000", 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026667999999999999 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746648+0000", 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6155999999999997e-05 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746684+0000", 2024-02-12T17:06:00.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0743e-05 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746714+0000", 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.89630858400000002 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220880 8.b 8:d0ab2051:::benchmark_data_smithi033_105085_object109222:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.900313+0000", 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.33368032099997, 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84649108299999998, 2024-02-12T17:06:00.314 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220880 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900313+0000", 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900313+0000", 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900311+0000", 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5899999999999997e-06 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900320+0000", 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1103e-05 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900331+0000", 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.959e-06 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900337+0000", 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7706e-05 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900385+0000", 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.469e-06 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.900392+0000", 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83167645999999995 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732069+0000", 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114713 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732184+0000", 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013649699999999999 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.732320+0000", 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0049788640000000004 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737299+0000", 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0093221889999999998 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746621+0000", 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012349600000000001 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746745+0000", 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0648000000000002e-05 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746775+0000", 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8821000000000001e-05 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746804+0000", 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84649108299999998 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220883 8.b 8:d17dac6e:::benchmark_data_smithi033_105085_object109225:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.904632+0000", 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.32936132100002, 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84231522400000003, 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220883 2024-02-12T17:06:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904632+0000", 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904632+0000", 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904630+0000", 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4369999999999996e-06 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904640+0000", 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0437e-05 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904650+0000", 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3119999999999999e-06 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904657+0000", 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5430999999999999e-05 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904702+0000", 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7610000000000007e-06 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.904710+0000", 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82888116999999994 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.733591+0000", 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000110407 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.733701+0000", 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014007599999999999 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.733841+0000", 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064799719999999996 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740321+0000", 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065120630000000002 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746833+0000", 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2596000000000003e-05 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746886+0000", 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1346000000000001e-05 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746917+0000", 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9869000000000002e-05 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746947+0000", 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84231522400000003 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220885 8.b 8:d01c3d69:::benchmark_data_smithi033_105085_object109227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.907595+0000", 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.32639872999999, 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83944173600000005, 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220885 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907595+0000", 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907595+0000", 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907593+0000", 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3400000000000003e-06 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907599+0000", 2024-02-12T17:06:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1269999999999992e-06 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907608+0000", 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6330000000000001e-06 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907614+0000", 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796000000000003e-05 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907654+0000", 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3150000000000003e-06 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.907661+0000", 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.827071259 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734732+0000", 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000104963 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734837+0000", 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000139192 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.734977+0000", 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053569009999999999 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740333+0000", 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065265089999999998 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746860+0000", 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116649 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.746977+0000", 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0667999999999998e-05 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747007+0000", 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9127000000000001e-05 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747036+0000", 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83944173600000005 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220895 8.b 8:d966b5f5:::benchmark_data_smithi033_105085_object109237:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.919760+0000", 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.31423385699998, 2024-02-12T17:06:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82739850299999995, 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220895 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919760+0000", 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919760+0000", 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919757+0000", 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6240000000000008e-06 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919766+0000", 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5795e-05 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919782+0000", 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.64e-06 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919788+0000", 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5522999999999998e-05 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919834+0000", 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.592e-06 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.919839+0000", 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.81639381799999999 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.736233+0000", 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5377999999999994e-05 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.736329+0000", 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4993000000000005e-05 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.736404+0000", 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.007085727 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.743489+0000", 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035814969999999999 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747071+0000", 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6551999999999999e-05 2024-02-12T17:06:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747097+0000", 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1009000000000001e-05 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747128+0000", 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9694999999999999e-05 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747158+0000", 2024-02-12T17:06:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82739850299999995 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220897 8.b 8:d66e080d:::benchmark_data_smithi033_105085_object109239:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.926673+0000", 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.30732011200001, 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82061067099999996, 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220897 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926673+0000", 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926673+0000", 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926671+0000", 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6340000000000007e-06 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926679+0000", 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3509999999999999e-05 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926693+0000", 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7900000000000002e-06 2024-02-12T17:06:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926700+0000", 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2910000000000001e-05 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926743+0000", 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0366999999999999e-05 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.926753+0000", 2024-02-12T17:06:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.810602445 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737355+0000", 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010326 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737459+0000", 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013548499999999999 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.737594+0000", 2024-02-12T17:06:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058989669999999998 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.743493+0000", 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037014119999999998 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747195+0000", 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7673000000000001e-05 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747222+0000", 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1764999999999998e-05 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747254+0000", 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9954000000000002e-05 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747284+0000", 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.82061067099999996 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220902 8.b 8:d534fbf5:::benchmark_data_smithi033_105085_object109244:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.931051+0000", 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.30294202699997, 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.816350519, 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220902 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931051+0000", 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931051+0000", 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931048+0000", 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0699999999999999e-05 2024-02-12T17:06:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931059+0000", 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.925e-05 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931078+0000", 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3809999999999999e-06 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931086+0000", 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4478999999999998e-05 2024-02-12T17:06:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931141+0000", 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8519999999999999e-06 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.931151+0000", 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.80772856699999995 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.738879+0000", 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5386e-05 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.738945+0000", 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6019000000000004e-05 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.739031+0000", 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0060882009999999997 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745119+0000", 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021945689999999999 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747314+0000", 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6559000000000001e-05 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747340+0000", 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1337000000000002e-05 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747371+0000", 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0459000000000001e-05 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747402+0000", 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.816350519 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220903 8.b 8:d138e6c7:::benchmark_data_smithi033_105085_object109245:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:45.932215+0000", 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.301778824, 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.81530424899999998, 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220903 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932215+0000", 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932215+0000", 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932212+0000", 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3659999999999992e-06 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932221+0000", 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4065e-05 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932235+0000", 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4860000000000002e-06 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932241+0000", 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1485e-05 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932283+0000", 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8480000000000003e-06 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:45.932290+0000", 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.80808374500000002 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740373+0000", 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010814700000000001 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740482+0000", 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013815 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.740620+0000", 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045471419999999997 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745167+0000", 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022638350000000001 2024-02-12T17:06:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747431+0000", 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6512999999999998e-05 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747457+0000", 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1942e-05 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747489+0000", 2024-02-12T17:06:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9643999999999999e-05 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747519+0000", 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.81530424899999998 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:220958 8.b 8:dc590d84:::benchmark_data_smithi033_105085_object109300:head [delete] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:46.036007+0000", 2024-02-12T17:06:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.19798634, 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.71162536899999995, 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 220958 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036007+0000", 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036007+0000", 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036005+0000", 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9109999999999993e-06 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036012+0000", 2024-02-12T17:06:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2116000000000001e-05 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036025+0000", 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0010000000000004e-06 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036032+0000", 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9883999999999999e-05 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036081+0000", 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5270000000000004e-06 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.036091+0000", 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.70584844700000005 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.741939+0000", 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7015999999999996e-05 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.742016+0000", 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4719000000000001e-05 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.742091+0000", 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00343203 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.745523+0000", 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020237580000000001 2024-02-12T17:06:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747547+0000", 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5474e-05 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747572+0000", 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0948000000000003e-05 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747603+0000", 2024-02-12T17:06:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9077e-05 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:46.747632+0000", 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.71162536899999995 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183798 9.c 9:3861ca0a:::benchmark_data_smithi033_131576_object46208:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.520521+0000", 2024-02-12T17:06:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.71347198199999, 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.131731695, 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183798 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520521+0000", 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520521+0000", 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520519+0000", 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4889999999999995e-06 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520527+0000", 2024-02-12T17:06:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2181000000000001e-05 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520539+0000", 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5869999999999998e-06 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520546+0000", 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6119000000000007e-05 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520632+0000", 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012252199999999999 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520755+0000", 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014270799999999999 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,5,7", 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.520897+0000", 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010872414 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.531770+0000", 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002294679 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.534064+0000", 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029693449 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.563758+0000", 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0434999999999999e-05 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.563768+0000", 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0809046310000001 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.644673+0000", 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012612600000000001 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.644799+0000", 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00736675 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.652166+0000", 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7241e-05 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.652253+0000", 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.131731695 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:00.376 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:06:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:00.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:00.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:00.503 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:58.780939+0000 osd.0 (osd.0) 184 : cluster [DBG] 5.b scrub starts 2024-02-12T17:06:00.503 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:58.811518+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.b scrub ok 2024-02-12T17:06:00.503 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:59.354429+0000 mon.a (mon.0) 1572 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:06:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:00.524 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:00.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:00.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:58.780939+0000 osd.0 (osd.0) 184 : cluster [DBG] 5.b scrub starts 2024-02-12T17:06:00.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:58.811518+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.b scrub ok 2024-02-12T17:06:00.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:59.354429+0000 mon.a (mon.0) 1572 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:06:00.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:58.780939+0000 osd.0 (osd.0) 184 : cluster [DBG] 5.b scrub starts 2024-02-12T17:06:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:58.811518+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.b scrub ok 2024-02-12T17:06:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:59.354429+0000 mon.a (mon.0) 1572 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:06:00.563 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:00.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:00.656 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:00.903 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:00.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:00.922 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T17:06:00.973 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:06:00.973 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:06:00.973 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 443, 2024-02-12T17:06:00.973 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2024-02-12T16:43:50.282468+0000", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2024-02-12T17:05:56.344666+0000", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2024-02-12T16:49:30.933575+0000", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2024-02-12T17:04:38.823636+0000", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5799936, 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 18, 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T17:06:00.974 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 9, 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 8, 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "luminous", 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "pacific", 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "device_health_metrics", 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:47:33.056945+0000", 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:00.975 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2024-02-12T17:06:00.976 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "71", 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:00.977 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:00.978 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:00.979 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_devicehealth": {} 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".rgw.root", 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:37.647343+0000", 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:00.980 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:06:00.981 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "59", 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:00.982 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:00.983 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:00.984 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "iscsi", 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:38.076200+0000", 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:00.985 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 3, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 3, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 3, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 3, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 3, 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:00.986 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "60", 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:00.988 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:00.989 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 4, 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.log", 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:39.589565+0000", 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:00.990 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:00.991 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "61", 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:00.992 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:00.993 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:00.994 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 5, 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.control", 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:41.623856+0000", 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:00.995 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "5.10", 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 429, 2024-02-12T17:06:00.996 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 428, 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 428, 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "380'16" 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "433", 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "429", 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "431", 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:00.997 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:00.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:00.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 6, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.meta", 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:43.693806+0000", 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:01.000 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "6.16", 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 396, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 395, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 395, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "396", 2024-02-12T17:06:01.001 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "396", 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "394", 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:01.002 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:01.003 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_bias": 4 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.004 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 9, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_2", 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T17:01:12.591563+0000", 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:01.005 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "310", 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:01.006 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:01.007 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:01.008 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "rados": {} 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.009 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7", 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0.2548370361328125, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 8, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 438, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.010 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:01.011 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2024-02-12T17:06:01.012 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/2545411201", 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6802/2545411201", 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6804/2545411201", 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6803/2545411201", 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.013 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24", 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 13, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 198, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.014 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6805/1723231708", 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6806/1723231708", 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6808/1723231708", 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6807/1723231708", 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c", 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:06:01.017 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0, 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 18, 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 440, 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:01.018 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3126580659", 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6810/3126580659", 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6812/3126580659", 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6811/3126580659", 2024-02-12T17:06:01.020 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9", 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 26, 2024-02-12T17:06:01.021 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 441, 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.022 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:01.023 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6813/3941314957", 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6814/3941314957", 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6816/3941314957", 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6815/3941314957", 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:01.024 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0498129c-861c-4f7e-b8ba-219949da3130", 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.54083251953125, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 32, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 440, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6800", 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6801", 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6803", 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6802", 2024-02-12T17:06:01.027 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6800/544126779", 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6801/544126779", 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6803/544126779", 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6802/544126779", 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:06:01.028 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50", 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 38, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 441, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6804", 2024-02-12T17:06:01.029 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6805", 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.030 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6807", 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6806", 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.031 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6804/3229884005", 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6805/3229884005", 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6807/3229884005", 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6806/3229884005", 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821", 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:06:01.032 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 45, 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 248, 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6808", 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6809", 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6811", 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:06:01.034 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6810", 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6808/507305558", 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6809/507305558", 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6811/507305558", 2024-02-12T17:06:01.035 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6810/507305558", 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66", 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:01.036 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 54, 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 441, 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6812", 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6813", 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6815", 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6814", 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6812/3401977873", 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6813/3401977873", 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6815/3401977873", 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6814/3401977873", 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:01.039 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:28.101520+0000", 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:51.331519+0000", 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:01.041 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:30.238493+0000", 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:01.148974+0000", 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.042 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:07.278429+0000", 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:01.043 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:39.766966+0000", 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:05.522314+0000", 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.044 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:04.422136+0000", 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [ 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.8", 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: "mappings": [ 2024-02-12T17:06:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: "from": 7, 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: "to": 2 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [ 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "9.1", 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: 4 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.046 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": { 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/2829464018": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/1835128904": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2277321750": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2116602769": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3967118778": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2298361247": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/218227980": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2070582473": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/4291575956": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3157562110": "2024-02-13T16:44:22.727241+0000" 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2024-02-12T17:06:01.047 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "ruleset-failure-domain": "osd", 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [], 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2024-02-12T17:06:01.048 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2024-02-12T17:06:01.049 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2024-02-12T17:06:01.049 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2024-02-12T17:06:01.049 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2024-02-12T17:06:01.049 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2024-02-12T17:06:01.049 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:01.049 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:06:01.049 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:01.074 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:01.093 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:01.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:01.196 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:01.224 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:01.237 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:01.247 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:01.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:01.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:06:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:00] "GET /metrics HTTP/1.1" 200 176300 "" "Prometheus/2.33.4" 2024-02-12T17:06:01.351 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:01.384 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:01.475 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:59.781098+0000 osd.0 (osd.0) 186 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:06:01.475 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:05:59.784627+0000 osd.0 (osd.0) 187 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:06:01.475 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:00.685618+0000 mgr.y (mgr.24481) 622 : cluster [DBG] pgmap v856: 128 pgs: 1 active+remapped+backfill_toofull, 127 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 806 B/s rd, 0 B/s wr, 49 op/s; 182 KiB/s, 3 objects/s recovering 2024-02-12T17:06:01.475 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:00.970842+0000 mon.a (mon.0) 1573 : audit [DBG] from='client.? v1:172.21.15.33:0/2517823263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:06:01.476 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:01.335462+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e444: 8 total, 8 up, 5 in 2024-02-12T17:06:01.634 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:01.637 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:59.781098+0000 osd.0 (osd.0) 186 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:06:01.637 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:05:59.784627+0000 osd.0 (osd.0) 187 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:06:01.637 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:00.685618+0000 mgr.y (mgr.24481) 622 : cluster [DBG] pgmap v856: 128 pgs: 1 active+remapped+backfill_toofull, 127 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 806 B/s rd, 0 B/s wr, 49 op/s; 182 KiB/s, 3 objects/s recovering 2024-02-12T17:06:01.637 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:00.970842+0000 mon.a (mon.0) 1573 : audit [DBG] from='client.? v1:172.21.15.33:0/2517823263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:06:01.637 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:01.335462+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e444: 8 total, 8 up, 5 in 2024-02-12T17:06:01.637 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:59.781098+0000 osd.0 (osd.0) 186 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:06:01.637 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:05:59.784627+0000 osd.0 (osd.0) 187 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:06:01.637 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:00.685618+0000 mgr.y (mgr.24481) 622 : cluster [DBG] pgmap v856: 128 pgs: 1 active+remapped+backfill_toofull, 127 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 806 B/s rd, 0 B/s wr, 49 op/s; 182 KiB/s, 3 objects/s recovering 2024-02-12T17:06:01.638 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:00.970842+0000 mon.a (mon.0) 1573 : audit [DBG] from='client.? v1:172.21.15.33:0/2517823263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:06:01.638 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:01.335462+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e444: 8 total, 8 up, 5 in 2024-02-12T17:06:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:01.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:01.739 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:01.772 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:02.120 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:02.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:02.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:02.251 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:01.452164+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:06:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:01.455965+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:06:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:01.901745+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.0 deep-scrub starts 2024-02-12T17:06:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:01.905925+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.0 deep-scrub ok 2024-02-12T17:06:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:02.044089+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:06:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:02.044879+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:06:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:01.452164+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:06:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:01.455965+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:06:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:01.901745+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.0 deep-scrub starts 2024-02-12T17:06:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:01.905925+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.0 deep-scrub ok 2024-02-12T17:06:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:02.044089+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:06:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:02.044879+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:06:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:02.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:02.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:02.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:06:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:06:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:01.452164+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:06:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:01.455965+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:06:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:01.901745+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.0 deep-scrub starts 2024-02-12T17:06:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:01.905925+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.0 deep-scrub ok 2024-02-12T17:06:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:02.044089+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:06:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:02.044879+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:06:02.741 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:03.063 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:06:03.063 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:06:03.064 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-02-12T17:06:03.064 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:06:03.446 INFO:tasks.thrashosds.thrasher:j is {'epoch': 443, 'fsid': 'cf6ad642-c9c5-11ee-95b9-87774f69a715', 'created': '2024-02-12T16:43:50.282468+0000', 'modified': '2024-02-12T17:05:56.344666+0000', 'last_up_change': '2024-02-12T16:49:30.933575+0000', 'last_in_change': '2024-02-12T17:04:38.823636+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T16:47:33.056945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-12T16:49:37.647343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-12T16:49:38.076200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-12T16:49:39.589565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-12T16:49:41.623856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.10', 'ready_epoch': 429, 'last_epoch_started': 428, 'last_epoch_clean': 428, 'source_version': "0'0", 'target_version': "380'16"}, 'last_change': '433', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '429', 'last_force_op_resend_preluminous': '431', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-12T16:49:43.693806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '6.16', 'ready_epoch': 396, 'last_epoch_started': 395, 'last_epoch_clean': 395, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '396', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '396', 'last_force_op_resend_preluminous': '394', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-12T17:01:12.591563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '310', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '5dd172eb-ee9d-4700-9b80-60f91c5f2cb7', 'up': 1, 'in': 1, 'weight': 0.2548370361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 438, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 2545411201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6802', 'nonce': 2545411201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6804', 'nonce': 2545411201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 2545411201}]}, 'public_addr': '172.21.15.33:6801/2545411201', 'cluster_addr': '172.21.15.33:6802/2545411201', 'heartbeat_back_addr': '172.21.15.33:6804/2545411201', 'heartbeat_front_addr': '172.21.15.33:6803/2545411201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71231bc4-1716-431d-bd88-f6a3cad0bd24', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1723231708}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6806', 'nonce': 1723231708}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6808', 'nonce': 1723231708}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1723231708}]}, 'public_addr': '172.21.15.33:6805/1723231708', 'cluster_addr': '172.21.15.33:6806/1723231708', 'heartbeat_back_addr': '172.21.15.33:6808/1723231708', 'heartbeat_front_addr': '172.21.15.33:6807/1723231708', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8a9c7127-d17c-4c44-b666-4211006de47c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 440, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3126580659}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6810', 'nonce': 3126580659}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6812', 'nonce': 3126580659}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3126580659}]}, 'public_addr': '172.21.15.33:6809/3126580659', 'cluster_addr': '172.21.15.33:6810/3126580659', 'heartbeat_back_addr': '172.21.15.33:6812/3126580659', 'heartbeat_front_addr': '172.21.15.33:6811/3126580659', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3cc40671-c8f7-412c-a0f1-987f48a016b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3941314957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6814', 'nonce': 3941314957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6816', 'nonce': 3941314957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3941314957}]}, 'public_addr': '172.21.15.33:6813/3941314957', 'cluster_addr': '172.21.15.33:6814/3941314957', 'heartbeat_back_addr': '172.21.15.33:6816/3941314957', 'heartbeat_front_addr': '172.21.15.33:6815/3941314957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0498129c-861c-4f7e-b8ba-219949da3130', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.54083251953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 440, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6800', 'nonce': 544126779}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 544126779}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 544126779}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 544126779}]}, 'public_addr': '172.21.15.94:6800/544126779', 'cluster_addr': '172.21.15.94:6801/544126779', 'heartbeat_back_addr': '172.21.15.94:6803/544126779', 'heartbeat_front_addr': '172.21.15.94:6802/544126779', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '05718a1e-1164-4b54-a493-23c1cd3adb50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 3229884005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 3229884005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 3229884005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 3229884005}]}, 'public_addr': '172.21.15.94:6804/3229884005', 'cluster_addr': '172.21.15.94:6805/3229884005', 'heartbeat_back_addr': '172.21.15.94:6807/3229884005', 'heartbeat_front_addr': '172.21.15.94:6806/3229884005', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dc48640f-8f27-405c-a9a4-cf1e3e6c8821', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 507305558}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 507305558}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 507305558}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 507305558}]}, 'public_addr': '172.21.15.94:6808/507305558', 'cluster_addr': '172.21.15.94:6809/507305558', 'heartbeat_back_addr': '172.21.15.94:6811/507305558', 'heartbeat_front_addr': '172.21.15.94:6810/507305558', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7d55aa3f-06b6-4fc3-97fd-62b535f78b66', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3401977873}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 3401977873}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 3401977873}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 3401977873}]}, 'public_addr': '172.21.15.94:6812/3401977873', 'cluster_addr': '172.21.15.94:6813/3401977873', 'heartbeat_back_addr': '172.21.15.94:6815/3401977873', 'heartbeat_front_addr': '172.21.15.94:6814/3401977873', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:05:28.101520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:05:51.331519+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:05:30.238493+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:05:01.148974+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:05:07.278429+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:05:39.766966+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:05:05.522314+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:05:04.422136+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.8', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_temp': [{'pgid': '9.1', 'osds': [2, 7, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.33:6800/2829464018': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/1835128904': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2277321750': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2116602769': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:6800/3967118778': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/2298361247': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/218227980': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/2070582473': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/4291575956': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:6800/3157562110': '2024-02-13T16:44:22.727241+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T17:06:03.446 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.8 2024-02-12T17:06:03.446 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd rm-pg-upmap-items 2.8 2024-02-12T17:06:03.733 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:03.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:03.737 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:01.753571+0000 osd.0 (osd.0) 188 : cluster [DBG] 9.0 scrub starts 2024-02-12T17:06:03.737 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:01.780429+0000 osd.0 (osd.0) 189 : cluster [DBG] 9.0 scrub ok 2024-02-12T17:06:03.737 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:02.492840+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:03.737 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:02.686088+0000 mgr.y (mgr.24481) 623 : cluster [DBG] pgmap v858: 128 pgs: 1 active+remapped+backfill_toofull, 127 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 697 B/s rd, 0 B/s wr, 42 op/s; 157 KiB/s, 2 objects/s recovering 2024-02-12T17:06:03.737 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:01.753571+0000 osd.0 (osd.0) 188 : cluster [DBG] 9.0 scrub starts 2024-02-12T17:06:03.737 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:01.780429+0000 osd.0 (osd.0) 189 : cluster [DBG] 9.0 scrub ok 2024-02-12T17:06:03.737 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:02.492840+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:03.738 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:02.686088+0000 mgr.y (mgr.24481) 623 : cluster [DBG] pgmap v858: 128 pgs: 1 active+remapped+backfill_toofull, 127 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 697 B/s rd, 0 B/s wr, 42 op/s; 157 KiB/s, 2 objects/s recovering 2024-02-12T17:06:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:01.753571+0000 osd.0 (osd.0) 188 : cluster [DBG] 9.0 scrub starts 2024-02-12T17:06:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:01.780429+0000 osd.0 (osd.0) 189 : cluster [DBG] 9.0 scrub ok 2024-02-12T17:06:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:02.492840+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:02.686088+0000 mgr.y (mgr.24481) 623 : cluster [DBG] pgmap v858: 128 pgs: 1 active+remapped+backfill_toofull, 127 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 697 B/s rd, 0 B/s wr, 42 op/s; 157 KiB/s, 2 objects/s recovering 2024-02-12T17:06:03.781 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:03.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T17:06:03.836 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:03.864 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:03.882 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:03.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:03.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:04.015 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:04.034 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:04.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:04.062 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:04.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:04.165 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:04.250 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:04.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:04.416 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:04.445 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:04.511 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:06:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:04.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:06:04.570 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:02.741445+0000 osd.0 (osd.0) 190 : cluster [DBG] 2.10 deep-scrub starts 2024-02-12T17:06:04.570 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:02.744487+0000 osd.0 (osd.0) 191 : cluster [DBG] 2.10 deep-scrub ok 2024-02-12T17:06:04.571 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:03.486438+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:06:04.571 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:03.551620+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:06:04.571 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:03.908389+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:06:04.571 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:03.911367+0000 osd.7 (osd.7) 201 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:06:04.571 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:03.927025+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.33:0/2030336849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:04.571 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:03.927419+0000 mon.a (mon.0) 1576 : audit [INF] from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:04.571 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:04.037094+0000 osd.5 (osd.5) 201 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:06:04.571 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:04.040551+0000 osd.5 (osd.5) 202 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:06:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:04.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:04.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:04.703 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:02.741445+0000 osd.0 (osd.0) 190 : cluster [DBG] 2.10 deep-scrub starts 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:02.744487+0000 osd.0 (osd.0) 191 : cluster [DBG] 2.10 deep-scrub ok 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:03.486438+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:03.551620+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:03.908389+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:03.911367+0000 osd.7 (osd.7) 201 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:03.927025+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.33:0/2030336849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:03.927419+0000 mon.a (mon.0) 1576 : audit [INF] from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:04.037094+0000 osd.5 (osd.5) 201 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:04.040551+0000 osd.5 (osd.5) 202 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:02.741445+0000 osd.0 (osd.0) 190 : cluster [DBG] 2.10 deep-scrub starts 2024-02-12T17:06:04.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:02.744487+0000 osd.0 (osd.0) 191 : cluster [DBG] 2.10 deep-scrub ok 2024-02-12T17:06:04.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:03.486438+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:06:04.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:03.551620+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:06:04.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:03.908389+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:06:04.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:03.911367+0000 osd.7 (osd.7) 201 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:06:04.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:03.927025+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.33:0/2030336849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:04.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:03.927419+0000 mon.a (mon.0) 1576 : audit [INF] from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:04.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:04.037094+0000 osd.5 (osd.5) 201 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:06:04.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:04.040551+0000 osd.5 (osd.5) 202 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:06:05.527 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_2' removed 2024-02-12T17:06:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:04.408659+0000 mgr.y (mgr.24481) 624 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:04.495779+0000 mon.a (mon.0) 1577 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:04.500608+0000 mon.a (mon.0) 1578 : audit [INF] from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:06:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:04.500671+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e445: 8 total, 8 up, 5 in 2024-02-12T17:06:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:04.537904+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.33:0/514410227' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:06:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:04.538216+0000 mon.a (mon.0) 1580 : audit [INF] from='client.35708 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:06:05.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:04.686515+0000 mgr.y (mgr.24481) 625 : cluster [DBG] pgmap v860: 128 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 126 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 39 op/s; 144 KiB/s, 2 objects/s recovering 2024-02-12T17:06:05.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:04.408659+0000 mgr.y (mgr.24481) 624 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:05.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:04.495779+0000 mon.a (mon.0) 1577 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:05.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:04.500608+0000 mon.a (mon.0) 1578 : audit [INF] from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:06:05.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:04.500671+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e445: 8 total, 8 up, 5 in 2024-02-12T17:06:05.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:04.537904+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.33:0/514410227' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:04.538216+0000 mon.a (mon.0) 1580 : audit [INF] from='client.35708 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:04.686515+0000 mgr.y (mgr.24481) 625 : cluster [DBG] pgmap v860: 128 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 126 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 39 op/s; 144 KiB/s, 2 objects/s recovering 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:04.408659+0000 mgr.y (mgr.24481) 624 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:04.495779+0000 mon.a (mon.0) 1577 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:04.500608+0000 mon.a (mon.0) 1578 : audit [INF] from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:04.500671+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e445: 8 total, 8 up, 5 in 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:04.537904+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.33:0/514410227' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:04.538216+0000 mon.a (mon.0) 1580 : audit [INF] from='client.35708 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:06:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:04.686515+0000 mgr.y (mgr.24481) 625 : cluster [DBG] pgmap v860: 128 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 126 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 39 op/s; 144 KiB/s, 2 objects/s recovering 2024-02-12T17:06:05.789 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:06:05.789 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:06:05.789 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:06:05.789 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:06:05.789 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:06:06.606 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:04.745049+0000 osd.0 (osd.0) 192 : cluster [DBG] 9.5 scrub starts 2024-02-12T17:06:06.607 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:05.512447+0000 mon.a (mon.0) 1581 : audit [INF] from='client.35708 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:06:06.607 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:05.512525+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e446: 8 total, 8 up, 5 in 2024-02-12T17:06:06.607 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:06.332216+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:06:06.625 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T17:06:06.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:04.745049+0000 osd.0 (osd.0) 192 : cluster [DBG] 9.5 scrub starts 2024-02-12T17:06:06.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:05.512447+0000 mon.a (mon.0) 1581 : audit [INF] from='client.35708 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:06:06.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:05.512525+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e446: 8 total, 8 up, 5 in 2024-02-12T17:06:06.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:06.332216+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:06:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:04.745049+0000 osd.0 (osd.0) 192 : cluster [DBG] 9.5 scrub starts 2024-02-12T17:06:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:05.512447+0000 mon.a (mon.0) 1581 : audit [INF] from='client.35708 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:06:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:05.512525+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e446: 8 total, 8 up, 5 in 2024-02-12T17:06:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:06.332216+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:06:06.894 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:07.423 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:06:07.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:06.686827+0000 mgr.y (mgr.24481) 626 : cluster [DBG] pgmap v862: 112 pgs: 1 active+clean+scrubbing, 111 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 682 B/s rd, 0 op/s 2024-02-12T17:06:07.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:06.686827+0000 mgr.y (mgr.24481) 626 : cluster [DBG] pgmap v862: 112 pgs: 1 active+clean+scrubbing, 111 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 682 B/s rd, 0 op/s 2024-02-12T17:06:07.900 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:06.686827+0000 mgr.y (mgr.24481) 626 : cluster [DBG] pgmap v862: 112 pgs: 1 active+clean+scrubbing, 111 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 682 B/s rd, 0 op/s 2024-02-12T17:06:08.286 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:08.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:08.390 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:08.422 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:08.736 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:06:08.736 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:06:08.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:06:08.736 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:06:08.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.736 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85071 8.3 8:cbed544b:::benchmark_data_smithi033_105085_object85070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e207)", 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.169767+0000", 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 502.56243050799998, 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.086660556, 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85071 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169767+0000", 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169767+0000", 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999919 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169759+0000", 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1277999999999999e-05 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169811+0000", 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2286000000000001e-05 2024-02-12T17:06:08.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169823+0000", 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1550000000000007e-06 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169831+0000", 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0877999999999997e-05 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169872+0000", 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0469999999999996e-06 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.169879+0000", 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99304094300000001 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162920+0000", 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5599999999999994e-06 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162928+0000", 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0707229229999999 2024-02-12T17:06:08.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.233651+0000", 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020893259999999999 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.235741+0000", 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000485759 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,6", 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.236226+0000", 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014595634999999999 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.250822+0000", 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3120999999999997e-05 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.250855+0000", 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002971998 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253827+0000", 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024091210000000002 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.256236+0000", 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7637000000000003e-05 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.256304+0000", 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012393300000000001 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.256428+0000", 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.086660556 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85082 8.0 8:07fce887:::benchmark_data_smithi033_105085_object85081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.188697+0000", 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 502.543500639, 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1147261230000001, 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85082 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188697+0000", 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188697+0000", 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188695+0000", 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8671999999999997e-05 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188754+0000", 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3957999999999999e-05 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188768+0000", 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3999999999999997e-06 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188774+0000", 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8887e-05 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188833+0000", 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.85e-06 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.188842+0000", 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97349081199999998 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162333+0000", 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0910000000000001e-05 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162343+0000", 2024-02-12T17:06:08.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.114529681 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.276873+0000", 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000146785 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.277020+0000", 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047932699999999999 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.277499+0000", 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013512467 2024-02-12T17:06:08.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.291012+0000", 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0050795270000000003 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.296091+0000", 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013251560000000001 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.297416+0000", 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058099359999999999 2024-02-12T17:06:08.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303226+0000", 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3219999999999994e-05 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303290+0000", 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013372899999999999 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303423+0000", 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1147261230000001 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85085 8.3 8:c095f601:::benchmark_data_smithi033_105085_object85084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.199890+0000", 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 502.53230798700002, 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0657017529999999, 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85085 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.751 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199890+0000", 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199890+0000", 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199888+0000", 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3984000000000001e-05 2024-02-12T17:06:08.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199972+0000", 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0386e-05 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199983+0000", 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6250000000000001e-06 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.199989+0000", 2024-02-12T17:06:08.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0538000000000002e-05 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.200050+0000", 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6120000000000005e-06 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.200059+0000", 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96288834499999998 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162948+0000", 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9970000000000001e-06 2024-02-12T17:06:08.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162955+0000", 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.084958568 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.247913+0000", 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015372300000000001 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248067+0000", 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022328699999999999 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,6", 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.248290+0000", 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026044319999999998 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.250895+0000", 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00055957100000000003 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.251454+0000", 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024053130000000001 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.253860+0000", 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011626877000000001 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.265486+0000", 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1525e-05 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.265528+0000", 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3641999999999998e-05 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.265592+0000", 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0657017529999999 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85090 8.0 8:00c4cb6e:::benchmark_data_smithi033_105085_object85089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.236897+0000", 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 502.49530066800003, 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.071817995, 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85090 2024-02-12T17:06:08.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236897+0000", 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236897+0000", 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236895+0000", 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6634000000000001e-05 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236942+0000", 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193999999999999e-05 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236952+0000", 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9799999999999998e-06 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.236957+0000", 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7144000000000001e-05 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.237004+0000", 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0240000000000003e-06 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.237010+0000", 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.92535760499999997 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162368+0000", 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6899999999999992e-06 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162375+0000", 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.123740913 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.286116+0000", 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000141197 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.286258+0000", 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018553800000000001 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.286443+0000", 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0046879440000000003 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.291131+0000", 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052767229999999997 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.296408+0000", 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070267460000000004 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303435+0000", 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051164299999999999 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.308551+0000", 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2330000000000002e-05 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.308603+0000", 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111811 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.308715+0000", 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.071817995 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:85096 8.0 8:0b744df4:::benchmark_data_smithi033_105085_object85095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:57:46.253913+0000", 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 502.47828442000002, 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.065851597, 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.765 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85096 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253913+0000", 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253913+0000", 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253911+0000", 2024-02-12T17:06:08.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4963000000000002e-05 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253966+0000", 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2769999999999994e-06 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253975+0000", 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9950000000000002e-06 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.253981+0000", 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.333e-05 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.254025+0000", 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2459999999999997e-06 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:46.254032+0000", 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.90836611599999995 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162398+0000", 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5900000000000002e-06 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:47.162406+0000", 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.126113074 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.288519+0000", 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013297999999999999 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.288652+0000", 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019030100000000001 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T17:06:08.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.288842+0000", 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002323276 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.291165+0000", 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053399099999999998 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.296505+0000", 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069363690000000004 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.303441+0000", 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016137531 2024-02-12T17:06:08.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.319579+0000", 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6953000000000002e-05 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.319636+0000", 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000129016 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:57:48.319765+0000", 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.065851597 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146034 8.0 8:0210f4aa:::benchmark_data_smithi033_105085_object34376:head [delete] snapc 0=[] ondisk+write+known_if_redirected e256)", 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.200477+0000", 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 420.53172112999999, 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1397366739999999, 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146034 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.772 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200477+0000", 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200477+0000", 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200476+0000", 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.525e-06 2024-02-12T17:06:08.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200479+0000", 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0209999999999999e-06 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200484+0000", 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.703e-06 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.200487+0000", 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028473560000000001 2024-02-12T17:06:08.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.203335+0000", 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8774000000000002e-05 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.203414+0000", 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5388000000000002e-05 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,4,6", 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.203509+0000", 2024-02-12T17:06:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079368089999999995 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.211446+0000", 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177801 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.211624+0000", 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000399214 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.212023+0000", 2024-02-12T17:06:08.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013163431 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.225186+0000", 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2659e-05 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.225199+0000", 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.114932679 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340132+0000", 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1829000000000005e-05 2024-02-12T17:06:08.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.340213+0000", 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1397366739999999 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146067 8.3 8:c3c181c2:::benchmark_data_smithi033_105085_object34409:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.273536+0000", 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 420.45866207799997, 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1236668910000001, 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.778 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146067 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273536+0000", 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273536+0000", 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:08.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273534+0000", 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2813000000000001e-05 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273547+0000", 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2418999999999999e-05 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273559+0000", 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4520000000000002e-06 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273566+0000", 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1288999999999997e-05 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273607+0000", 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1859999999999992e-06 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.273615+0000", 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0830521850000001 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.356667+0000", 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2581e-05 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.356760+0000", 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00033919000000000001 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.357099+0000", 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010171671 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367271+0000", 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052693300000000005 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367798+0000", 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367836+0000", 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036862040000000002 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371522+0000", 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025472465999999999 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.396994+0000", 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8136e-05 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397052+0000", 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015023900000000001 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397203+0000", 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1236668910000001 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146069 8.3 8:ccab45b3:::benchmark_data_smithi033_105085_object34411:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.277018+0000", 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 420.45517948000003, 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.120418567, 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.785 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146069 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277018+0000", 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277018+0000", 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277016+0000", 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1019999999999999e-06 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277024+0000", 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2464e-05 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277036+0000", 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3720000000000001e-06 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277042+0000", 2024-02-12T17:06:08.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900999999999999e-05 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277093+0000", 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3749999999999992e-06 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277103+0000", 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0849561649999999 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.362059+0000", 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5056999999999996e-05 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.362134+0000", 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9012999999999995e-05 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.362233+0000", 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0050455099999999996 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367278+0000", 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037674560000000002 2024-02-12T17:06:08.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371046+0000", 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000510445 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371556+0000", 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057177419999999996 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377274+0000", 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019977175999999999 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397251+0000", 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0614e-05 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397302+0000", 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000135062 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397437+0000", 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.120418567 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146070 8.3 8:c3719ad9:::benchmark_data_smithi033_105085_object34412:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.277252+0000", 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 420.45494551100001, 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1204189790000001, 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146070 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277252+0000", 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277252+0000", 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277251+0000", 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6280000000000002e-06 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277257+0000", 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0902e-05 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277268+0000", 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9390000000000002e-06 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277273+0000", 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7271999999999998e-05 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277311+0000", 2024-02-12T17:06:08.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8779999999999997e-06 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.277316+0000", 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0858321420000001 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.363148+0000", 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114474 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.363262+0000", 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014188800000000001 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.363404+0000", 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038811050000000001 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367285+0000", 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037957440000000002 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371081+0000", 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00058387300000000005 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371665+0000", 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056816669999999996 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377347+0000", 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020137176 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397484+0000", 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397531+0000", 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014016300000000001 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397671+0000", 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1204189790000001 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146071 8.3 8:c37e2b9c:::benchmark_data_smithi033_105085_object34413:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.278557+0000", 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 420.453640458, 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.11933178, 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.798 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146071 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278557+0000", 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278557+0000", 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278556+0000", 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278561+0000", 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4619999999999999e-06 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278568+0000", 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.506e-06 2024-02-12T17:06:08.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278572+0000", 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3637999999999999e-05 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278605+0000", 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0549999999999996e-06 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.278611+0000", 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.086035463 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.364647+0000", 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0767999999999997e-05 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.364718+0000", 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9730999999999999e-05 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.364807+0000", 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024845140000000002 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.367292+0000", 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043985839999999997 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.371690+0000", 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00045211699999999998 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.372143+0000", 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052212309999999998 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377364+0000", 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020349597000000001 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397713+0000", 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3578999999999997e-05 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397757+0000", 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013217799999999999 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397889+0000", 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.11933178 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:146074 8.3 8:cb2d083d:::benchmark_data_smithi033_105085_object34416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e257)", 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:08.280792+0000", 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 420.45140622500003, 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1173190850000001, 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:08.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146074 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280792+0000", 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280792+0000", 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280791+0000", 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1910000000000002e-06 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280795+0000", 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0710000000000002e-06 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280800+0000", 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2890000000000002e-06 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280803+0000", 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4401999999999999e-05 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280848+0000", 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2479999999999996e-06 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:08.280856+0000", 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.085028415 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.365884+0000", 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112866 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.365997+0000", 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015273899999999999 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.366150+0000", 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020551240000000002 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.368205+0000", 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040942050000000001 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.372299+0000", 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047219779999999999 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377021+0000", 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00036072399999999998 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.377382+0000", 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020553024 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397935+0000", 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4613000000000001e-05 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.397979+0000", 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013121699999999999 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:09.398111+0000", 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1173190850000001 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183785 9.7 9:e8886c22:::benchmark_data_smithi033_131576_object46195:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.502793+0000", 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.229404523, 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1740439839999999, 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:08.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183785 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502793+0000", 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502793+0000", 2024-02-12T17:06:08.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502792+0000", 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2820000000000002e-06 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502798+0000", 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2590000000000006e-06 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502807+0000", 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6879999999999996e-06 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502813+0000", 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4116000000000005e-05 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502907+0000", 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012017 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.503027+0000", 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013745900000000001 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.503165+0000", 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002240578 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.505405+0000", 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058207989999999998 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511226+0000", 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040592560999999999 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551819+0000", 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3529999999999997e-06 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551826+0000", 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99712210199999995 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548948+0000", 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2199999999999998e-06 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548957+0000", 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.127834097 2024-02-12T17:06:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676791+0000", 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5977999999999999e-05 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676837+0000", 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1740439839999999 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183788 9.a 9:5036b012:::benchmark_data_smithi033_131576_object46198:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:08.818 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.511400+0000", 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.22079735899999, 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.167927138, 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183788 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511400+0000", 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511400+0000", 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511398+0000", 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1103e-05 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511409+0000", 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6739e-05 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511426+0000", 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9079999999999995e-06 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511434+0000", 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5638999999999997e-05 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511529+0000", 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115553 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511645+0000", 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012956600000000001 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511774+0000", 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010709754 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522484+0000", 2024-02-12T17:06:08.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4372000000000001e-05 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522499+0000", 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020223532999999998 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.542722+0000", 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1199999999999996e-06 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.542729+0000", 2024-02-12T17:06:08.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.993438987 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.536168+0000", 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6004000000000001e-05 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.536184+0000", 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.143099299 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.679283+0000", 2024-02-12T17:06:08.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4094000000000002e-05 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.679328+0000", 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.167927138 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183790 9.7 9:e6205a30:::benchmark_data_smithi033_131576_object46200:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.511449+0000", 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.22074916400001, 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1655356800000001, 2024-02-12T17:06:08.825 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183790 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511449+0000", 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511449+0000", 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511448+0000", 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0199999999999996e-06 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511452+0000", 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2700000000000001e-06 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511458+0000", 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0380000000000001e-06 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511462+0000", 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023379049999999999 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.513800+0000", 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.513903+0000", 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000119113 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.514022+0000", 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087794180000000006 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522802+0000", 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029046764999999999 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551849+0000", 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.323e-06 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551856+0000", 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99712761900000002 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548984+0000", 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.678e-06 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548991+0000", 2024-02-12T17:06:08.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1279593509999999 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676951+0000", 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3662999999999999e-05 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676984+0000", 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1655356800000001 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.831 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183791 9.f 9:fdb45a54:::benchmark_data_smithi033_131576_object46201:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.511471+0000", 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.22072690300001, 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1655592750000001, 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183791 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511471+0000", 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511471+0000", 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511470+0000", 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9380000000000001e-06 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511473+0000", 2024-02-12T17:06:08.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0540000000000002e-06 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511478+0000", 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5880000000000002e-06 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511482+0000", 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040833400000000004 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.515565+0000", 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7264999999999998e-05 2024-02-12T17:06:08.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.515662+0000", 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011482800000000001 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.515777+0000", 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070314419999999997 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522809+0000", 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028937609999999999 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551746+0000", 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4230000000000005e-06 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551755+0000", 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99540099699999995 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.547156+0000", 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0518e-05 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.547166+0000", 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1298328820000001 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676999+0000", 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.099e-05 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677030+0000", 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1655592750000001 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183795 9.f 9:fc4b72ca:::benchmark_data_smithi033_131576_object46205:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.517291+0000", 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.214907178, 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1597805719999998, 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183795 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517291+0000", 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517291+0000", 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517289+0000", 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5079999999999999e-06 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517296+0000", 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9369999999999993e-06 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517305+0000", 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9850000000000003e-06 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517311+0000", 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00049205400000000002 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517803+0000", 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8632e-05 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517902+0000", 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116575 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.518018+0000", 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047957549999999996 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522814+0000", 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028968608999999999 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551783+0000", 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2039999999999997e-06 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551791+0000", 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99689064500000002 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548681+0000", 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9699999999999994e-06 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548691+0000", 2024-02-12T17:06:08.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.12835124 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677043+0000", 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8580000000000001e-05 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677071+0000", 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1597805719999998 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24355.0:2081 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94508044031616 gen 3] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.519445+0000", 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.212753029, 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.144085775, 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24355", 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2081 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519445+0000", 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519445+0000", 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519444+0000", 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5200000000000002e-06 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519448+0000", 2024-02-12T17:06:08.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1379999999999998e-06 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519453+0000", 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0740000000000003e-06 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519457+0000", 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2222000000000002e-05 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519509+0000", 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3050000000000004e-06 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519516+0000", 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033933252999999997 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.553450+0000", 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8100000000000004e-06 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.553458+0000", 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0242327739999999 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.577691+0000", 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030178699999999998 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for degraded object", 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.577993+0000", 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042871052 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.620864+0000", 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5253000000000005e-05 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.620949+0000", 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042581320999999998 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.663531+0000", 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.144085775 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183800 9.7 9:e9f894db:::benchmark_data_smithi033_131576_object46210:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.526801+0000", 2024-02-12T17:06:08.849 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.20539639899999, 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1920783080000001, 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183800 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526801+0000", 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526801+0000", 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526799+0000", 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0271e-05 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526809+0000", 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1632e-05 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526821+0000", 2024-02-12T17:06:08.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3779999999999998e-06 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526827+0000", 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025060631999999999 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551888+0000", 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4320000000000002e-06 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551895+0000", 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99711852300000003 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.549014+0000", 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.593e-06 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.549021+0000", 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1279905619999999 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677012+0000", 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5311999999999998e-05 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677107+0000", 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010268900000000001 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677210+0000", 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002630837 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.679841+0000", 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00041140600000000001 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680252+0000", 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059192439999999997 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.686171+0000", 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032511917000000001 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718683+0000", 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6761999999999995e-05 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718750+0000", 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000129562 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718880+0000", 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1920783080000001 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183807 9.a 9:57e153ef:::benchmark_data_smithi033_131576_object46217:head [delete] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.566277+0000", 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.16592036399999, 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.142295142, 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183807 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566277+0000", 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566277+0000", 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566275+0000", 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6699999999999997e-06 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566282+0000", 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7689999999999996e-06 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566290+0000", 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.651e-06 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566294+0000", 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4171999999999997e-05 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566329+0000", 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6659999999999996e-06 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566334+0000", 2024-02-12T17:06:08.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97030596000000002 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.536640+0000", 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5310000000000001e-06 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.536649+0000", 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.143557961 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680207+0000", 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3682000000000003e-05 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680290+0000", 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8075e-05 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680368+0000", 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022998889999999998 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.682668+0000", 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028804249999999998 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.685549+0000", 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060336299999999999 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.686152+0000", 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022233121000000002 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.708385+0000", 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7024000000000002e-05 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.708442+0000", 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013035600000000001 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.708573+0000", 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.142295142 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183809 9.f 9:fdff3c78:::benchmark_data_smithi033_131576_object46219:head [delete] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.579023+0000", 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.15317505600001, 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1398916479999999, 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183809 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:08.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579023+0000", 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579023+0000", 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579020+0000", 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0170000000000002e-06 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579030+0000", 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0334e-05 2024-02-12T17:06:08.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579040+0000", 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3850000000000001e-06 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579045+0000", 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0598000000000001e-05 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579076+0000", 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1750000000000002e-06 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579082+0000", 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96971665299999998 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548799+0000", 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9719999999999992e-06 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548807+0000", 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1282766399999999 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677083+0000", 2024-02-12T17:06:08.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9552999999999997e-05 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677143+0000", 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114326 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,4", 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677257+0000", 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033515070000000001 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680609+0000", 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012439339999999999 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.681853+0000", 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043585680000000002 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.686211+0000", 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032530057000000001 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718741+0000", 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899999999999999e-05 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:08.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718789+0000", 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012528800000000001 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718914+0000", 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1398916479999999 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:08.870 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:06:08.871 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:07.510159+0000 mon.a (mon.0) 1584 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:06:08.871 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:07.510159+0000 mon.a (mon.0) 1584 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:06:08.871 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:07.510159+0000 mon.a (mon.0) 1584 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:06:08.871 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:08.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:08.976 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:09.004 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:09.203 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:09.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:09.306 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:09.337 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:09.460 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:09.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:09.472 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T17:06:09.564 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:09.667 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:09.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:09.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:08.687530+0000 mgr.y (mgr.24481) 627 : cluster [DBG] pgmap v863: 112 pgs: 112 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 556 B/s rd, 0 op/s 2024-02-12T17:06:09.771 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:09.775 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:09.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:08.687530+0000 mgr.y (mgr.24481) 627 : cluster [DBG] pgmap v863: 112 pgs: 112 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 556 B/s rd, 0 op/s 2024-02-12T17:06:09.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:08.687530+0000 mgr.y (mgr.24481) 627 : cluster [DBG] pgmap v863: 112 pgs: 112 active+clean; 5.7 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 556 B/s rd, 0 op/s 2024-02-12T17:06:09.800 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:09.998 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:10.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:10.040 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:06:10.058 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:06:10.074 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:06:10.092 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:06:10.102 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:10.110 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:06:10.128 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:06:10.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:10.147 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:06:10.164 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:06:10.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:06:10.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:06:10.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:06:10.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:06:10.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:06:10.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:06:10.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:06:10.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:06:10.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:09.605007+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? v1:172.21.15.33:0/2554288748' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:06:10.524 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:10.021969+0000 mon.a (mon.0) 1586 : audit [DBG] from='client.? v1:172.21.15.33:0/3718825283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:10.533 INFO:teuthology.orchestra.run.smithi033.stderr:clear 2.8 pg_upmap_items mapping 2024-02-12T17:06:10.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:09.605007+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? v1:172.21.15.33:0/2554288748' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:06:10.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:10.021969+0000 mon.a (mon.0) 1586 : audit [DBG] from='client.? v1:172.21.15.33:0/3718825283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:10.876 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:09.605007+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? v1:172.21.15.33:0/2554288748' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:06:10.876 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:10.021969+0000 mon.a (mon.0) 1586 : audit [DBG] from='client.? v1:172.21.15.33:0/3718825283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:11.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:06:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:10] "GET /metrics HTTP/1.1" 200 176303 "" "Prometheus/2.33.4" 2024-02-12T17:06:11.355 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:06:11.356 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:06:11.381 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:11.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:11.485 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:11.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:10.529271+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? v1:172.21.15.33:0/2554288748' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]': finished 2024-02-12T17:06:11.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:10.529333+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e447: 8 total, 8 up, 5 in 2024-02-12T17:06:11.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:10.687987+0000 mgr.y (mgr.24481) 628 : cluster [DBG] pgmap v865: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 662 B/s rd, 0 op/s 2024-02-12T17:06:11.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:11.338439+0000 mon.a (mon.0) 1589 : cluster [DBG] osdmap e448: 8 total, 8 up, 5 in 2024-02-12T17:06:11.619 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:10.529271+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? v1:172.21.15.33:0/2554288748' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]': finished 2024-02-12T17:06:11.619 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:10.529333+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e447: 8 total, 8 up, 5 in 2024-02-12T17:06:11.619 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:10.687987+0000 mgr.y (mgr.24481) 628 : cluster [DBG] pgmap v865: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 662 B/s rd, 0 op/s 2024-02-12T17:06:11.619 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:11.338439+0000 mon.a (mon.0) 1589 : cluster [DBG] osdmap e448: 8 total, 8 up, 5 in 2024-02-12T17:06:11.620 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:11.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:11.639 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T17:06:11.640 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T17:06:11.640 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-02-12T17:06:11.640 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-02-12T17:06:11.708 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:11.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:10.529271+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? v1:172.21.15.33:0/2554288748' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]': finished 2024-02-12T17:06:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:10.529333+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e447: 8 total, 8 up, 5 in 2024-02-12T17:06:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:10.687987+0000 mgr.y (mgr.24481) 628 : cluster [DBG] pgmap v865: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 662 B/s rd, 0 op/s 2024-02-12T17:06:11.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:11.338439+0000 mon.a (mon.0) 1589 : cluster [DBG] osdmap e448: 8 total, 8 up, 5 in 2024-02-12T17:06:11.752 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:12.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T17:06:12.102 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:12.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:12.205 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:12.232 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:12.365 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:12.380 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:12.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:12.417 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:12.484 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:12.635 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:12.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:12.739 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:12.764 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:06:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:06:12.766 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:12.864 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:12.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:12.968 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:12.997 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:13.035 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:13.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:13.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:13.167 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:13.267 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:13.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:13.372 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:13.399 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:14.078 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:06:14.078 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:06:14.116 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:06:14.383 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:12.688486+0000 mgr.y (mgr.24481) 629 : cluster [DBG] pgmap v867: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 713 B/s rd, 0 op/s 2024-02-12T17:06:14.383 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:13.104091+0000 mon.a (mon.0) 1590 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:14.383 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:12.688486+0000 mgr.y (mgr.24481) 629 : cluster [DBG] pgmap v867: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 713 B/s rd, 0 op/s 2024-02-12T17:06:14.384 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:13.104091+0000 mon.a (mon.0) 1590 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:14.414 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:12.688486+0000 mgr.y (mgr.24481) 629 : cluster [DBG] pgmap v867: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 713 B/s rd, 0 op/s 2024-02-12T17:06:14.414 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:13.104091+0000 mon.a (mon.0) 1590 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:14.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:06:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:06:14.763 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T17:06:15.030 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:15.117 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:06:15.118 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:15.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:14.796846+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? v1:172.21.15.33:0/4241226148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:15.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:14.796846+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? v1:172.21.15.33:0/4241226148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:15.354 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:15.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:15.458 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:15.489 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:15.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:14.796846+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? v1:172.21.15.33:0/4241226148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:16.193 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:14.412876+0000 mgr.y (mgr.24481) 630 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:16.193 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:14.689037+0000 mgr.y (mgr.24481) 631 : cluster [DBG] pgmap v868: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:16.193 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:15.105257+0000 mon.a (mon.0) 1592 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:16.194 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:15.112406+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? v1:172.21.15.33:0/4241226148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:06:16.194 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:15.112465+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e449: 8 total, 8 up, 5 in 2024-02-12T17:06:16.355 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:06:16.355 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:06:16.412 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:14.412876+0000 mgr.y (mgr.24481) 630 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:14.689037+0000 mgr.y (mgr.24481) 631 : cluster [DBG] pgmap v868: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:15.105257+0000 mon.a (mon.0) 1592 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:15.112406+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? v1:172.21.15.33:0/4241226148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:15.112465+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e449: 8 total, 8 up, 5 in 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:14.412876+0000 mgr.y (mgr.24481) 630 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:14.689037+0000 mgr.y (mgr.24481) 631 : cluster [DBG] pgmap v868: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:15.105257+0000 mon.a (mon.0) 1592 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:15.112406+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? v1:172.21.15.33:0/4241226148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:06:16.413 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:15.112465+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e449: 8 total, 8 up, 5 in 2024-02-12T17:06:16.441 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:16.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:16.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:16.574 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:16.647 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:16.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:16.710 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:06:16.710 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:06:16.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:16.779 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:16.978 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:16.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:17.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:17.110 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:17.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T17:06:17.363 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_3' created 2024-02-12T17:06:17.575 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:16.169681+0000 osd.3 (osd.3) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:16.169735+0000 osd.3 (osd.3) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:16.340167+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e450: 8 total, 8 up, 5 in 2024-02-12T17:06:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:16.516683+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? v1:172.21.15.33:0/2549321366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-12T17:06:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:16.689352+0000 mgr.y (mgr.24481) 632 : cluster [DBG] pgmap v871: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:06:17.761 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:17.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:16.169681+0000 osd.3 (osd.3) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:17.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:16.169735+0000 osd.3 (osd.3) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:17.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:16.340167+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e450: 8 total, 8 up, 5 in 2024-02-12T17:06:17.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:16.516683+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? v1:172.21.15.33:0/2549321366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-12T17:06:17.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:16.689352+0000 mgr.y (mgr.24481) 632 : cluster [DBG] pgmap v871: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:06:17.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:16.169681+0000 osd.3 (osd.3) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:17.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:16.169735+0000 osd.3 (osd.3) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:17.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:16.340167+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e450: 8 total, 8 up, 5 in 2024-02-12T17:06:17.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:16.516683+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? v1:172.21.15.33:0/2549321366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-12T17:06:17.764 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:16.689352+0000 mgr.y (mgr.24481) 632 : cluster [DBG] pgmap v871: 112 pgs: 112 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:06:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:17.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:17.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:17.894 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:17.913 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:17.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:18.017 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:18.046 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:18.104 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:06:18.196 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:18.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:18.217 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-02-12T17:06:18.301 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:18.330 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:18.483 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:18.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:18.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:18.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:17.356583+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.33:0/2549321366' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-12T17:06:18.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:17.356672+0000 mon.a (mon.0) 1598 : cluster [DBG] osdmap e451: 8 total, 8 up, 5 in 2024-02-12T17:06:18.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:17.490155+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:18.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:17.356583+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.33:0/2549321366' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-12T17:06:18.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:17.356672+0000 mon.a (mon.0) 1598 : cluster [DBG] osdmap e451: 8 total, 8 up, 5 in 2024-02-12T17:06:18.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:17.490155+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:18.617 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:17.356583+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.33:0/2549321366' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-12T17:06:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:17.356672+0000 mon.a (mon.0) 1598 : cluster [DBG] osdmap e451: 8 total, 8 up, 5 in 2024-02-12T17:06:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:17.490155+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:18.740 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:18.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:18.767 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:18.774 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:06:18.796 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:06:18.815 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:06:18.832 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:06:18.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:18.851 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:06:18.868 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:06:18.873 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:18.887 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:06:18.905 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:06:18.906 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:06:18.906 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:06:18.906 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:06:18.907 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:06:18.907 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:06:18.907 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:06:18.907 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:06:18.907 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:06:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:19.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:19.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:19.204 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:19.383 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:06:19.383 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:06:19.383 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-02-12T17:06:19.383 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:06:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:19.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:19.505 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:19.534 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:19.678 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:18.359719+0000 mon.a (mon.0) 1600 : cluster [DBG] osdmap e452: 8 total, 8 up, 5 in 2024-02-12T17:06:19.678 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:18.689805+0000 mgr.y (mgr.24481) 633 : cluster [DBG] pgmap v874: 128 pgs: 5 creating+peering, 6 unknown, 117 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail 2024-02-12T17:06:19.678 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:18.754504+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.33:0/3877741093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:19.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:18.359719+0000 mon.a (mon.0) 1600 : cluster [DBG] osdmap e452: 8 total, 8 up, 5 in 2024-02-12T17:06:19.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:18.689805+0000 mgr.y (mgr.24481) 633 : cluster [DBG] pgmap v874: 128 pgs: 5 creating+peering, 6 unknown, 117 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail 2024-02-12T17:06:19.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:18.754504+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.33:0/3877741093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:18.359719+0000 mon.a (mon.0) 1600 : cluster [DBG] osdmap e452: 8 total, 8 up, 5 in 2024-02-12T17:06:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:18.689805+0000 mgr.y (mgr.24481) 633 : cluster [DBG] pgmap v874: 128 pgs: 5 creating+peering, 6 unknown, 117 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail 2024-02-12T17:06:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:18.754504+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.33:0/3877741093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:19.986 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T17:06:20.146 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:20.269 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:20.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:19.363092+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e453: 8 total, 8 up, 5 in 2024-02-12T17:06:20.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:19.941661+0000 mon.c (mon.2) 237 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:06:20.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:20.196331+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:20.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:19.363092+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e453: 8 total, 8 up, 5 in 2024-02-12T17:06:20.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:19.941661+0000 mon.c (mon.2) 237 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:06:20.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:20.196331+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:19.363092+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e453: 8 total, 8 up, 5 in 2024-02-12T17:06:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:19.941661+0000 mon.c (mon.2) 237 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:06:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:20.196331+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:20.784 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:20.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:20.889 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:21.116 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:21.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:21.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:21.247 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:21.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:21.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:21.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:06:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:20] "GET /metrics HTTP/1.1" 200 176418 "" "Prometheus/2.33.4" 2024-02-12T17:06:21.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:21.395 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:20.639542+0000 mon.c (mon.2) 238 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:06:21.396 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:20.640244+0000 mon.c (mon.2) 239 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:06:21.396 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:20.644605+0000 mon.a (mon.0) 1603 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:21.396 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:20.690295+0000 mgr.y (mgr.24481) 634 : cluster [DBG] pgmap v876: 128 pgs: 5 creating+peering, 123 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:06:21.396 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:21.342171+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e454: 8 total, 8 up, 5 in 2024-02-12T17:06:21.397 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:21.520 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:21.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:21.623 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:21.654 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:20.639542+0000 mon.c (mon.2) 238 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:06:21.654 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:20.640244+0000 mon.c (mon.2) 239 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:06:21.654 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:20.644605+0000 mon.a (mon.0) 1603 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:21.655 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:20.690295+0000 mgr.y (mgr.24481) 634 : cluster [DBG] pgmap v876: 128 pgs: 5 creating+peering, 123 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:06:21.655 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:21.342171+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e454: 8 total, 8 up, 5 in 2024-02-12T17:06:21.655 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:20.639542+0000 mon.c (mon.2) 238 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:06:21.655 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:20.640244+0000 mon.c (mon.2) 239 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:06:21.655 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:20.644605+0000 mon.a (mon.0) 1603 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:21.655 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:20.690295+0000 mgr.y (mgr.24481) 634 : cluster [DBG] pgmap v876: 128 pgs: 5 creating+peering, 123 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-12T17:06:21.655 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:21.342171+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e454: 8 total, 8 up, 5 in 2024-02-12T17:06:21.657 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:21.810 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:06:21.858 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:21.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:21.964 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:21.984 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:06:21.984 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:06:21.984 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:06:21.984 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:06:21.984 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:06:21.992 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:22.192 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:22.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:22.295 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:22.327 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:22.649 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:06:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:06:22.665 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-02-12T17:06:22.691 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:21.922844+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.33:0/1206646872' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:06:22.691 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:21.923171+0000 mon.a (mon.0) 1605 : audit [INF] from='client.35744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:06:22.691 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:21.922844+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.33:0/1206646872' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:06:22.691 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:21.923171+0000 mon.a (mon.0) 1605 : audit [INF] from='client.35744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:06:22.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T17:06:22.775 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:22.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:22.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:22.904 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:21.922844+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.33:0/1206646872' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:06:22.904 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:21.923171+0000 mon.a (mon.0) 1605 : audit [INF] from='client.35744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:06:22.906 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:22.928 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:22.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:23.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:23.037 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:06:23.060 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:23.672 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:06:23.950 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:22.248678+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:22.248727+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:22.660795+0000 mon.a (mon.0) 1606 : audit [INF] from='client.35744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:06:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:22.660875+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e455: 8 total, 8 up, 5 in 2024-02-12T17:06:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:22.690822+0000 mgr.y (mgr.24481) 635 : cluster [DBG] pgmap v879: 128 pgs: 5 creating+peering, 123 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:23.109221+0000 mon.c (mon.2) 241 : audit [INF] from='client.? v1:172.21.15.33:0/1128284873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:06:23.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:23.109530+0000 mon.a (mon.0) 1608 : audit [INF] from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:06:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:22.248678+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:22.248727+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:22.660795+0000 mon.a (mon.0) 1606 : audit [INF] from='client.35744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:06:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:22.660875+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e455: 8 total, 8 up, 5 in 2024-02-12T17:06:24.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:22.690822+0000 mgr.y (mgr.24481) 635 : cluster [DBG] pgmap v879: 128 pgs: 5 creating+peering, 123 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:23.109221+0000 mon.c (mon.2) 241 : audit [INF] from='client.? v1:172.21.15.33:0/1128284873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:23.109530+0000 mon.a (mon.0) 1608 : audit [INF] from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:22.248678+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:22.248727+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:22.660795+0000 mon.a (mon.0) 1606 : audit [INF] from='client.35744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:22.660875+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e455: 8 total, 8 up, 5 in 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:22.690822+0000 mgr.y (mgr.24481) 635 : cluster [DBG] pgmap v879: 128 pgs: 5 creating+peering, 123 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:23.109221+0000 mon.c (mon.2) 241 : audit [INF] from='client.? v1:172.21.15.33:0/1128284873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:06:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:23.109530+0000 mon.a (mon.0) 1608 : audit [INF] from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:06:24.111 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:24.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:24.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:24.243 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:24.263 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:24.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:24.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:24.399 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:24.662 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:06:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:06:24.750 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:24.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:24.760 DEBUG:teuthology.orchestra.run.smithi033:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-02-12T17:06:24.760 INFO:tasks.thrashosds.thrasher:Shrinking pool device_health_metrics 2024-02-12T17:06:24.760 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool device_health_metrics by 10 min 8 2024-02-12T17:06:24.761 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:06:24.796 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T17:06:24.796 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T17:06:24.847 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:hints = 1 2024-02-12T17:06:24.847 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T17:06:24.847 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object prefix: benchmark_data_smithi033_160181 2024-02-12T17:06:24.847 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:06:24.847 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 0 0 0 0 0 0 - 0 2024-02-12T17:06:24.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:24.892 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:24.950 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:23.565958+0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:24.950 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:23.566038+0000 osd.6 (osd.6) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:24.950 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:23.653818+0000 mon.a (mon.0) 1609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:24.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:23.664165+0000 mon.a (mon.0) 1610 : audit [INF] from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:06:24.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:23.664229+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e456: 8 total, 8 up, 5 in 2024-02-12T17:06:24.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:23.670229+0000 osd.0 (osd.0) 193 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:06:24.951 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:23.673118+0000 osd.0 (osd.0) 194 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:06:24.953 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:24.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:24.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:24.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:23.565958+0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:24.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:23.566038+0000 osd.6 (osd.6) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:24.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:23.653818+0000 mon.a (mon.0) 1609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:24.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:23.664165+0000 mon.a (mon.0) 1610 : audit [INF] from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:06:24.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:23.664229+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e456: 8 total, 8 up, 5 in 2024-02-12T17:06:24.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:23.670229+0000 osd.0 (osd.0) 193 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:06:24.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:23.673118+0000 osd.0 (osd.0) 194 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:06:24.962 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:23.565958+0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:24.962 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:23.566038+0000 osd.6 (osd.6) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:24.962 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:23.653818+0000 mon.a (mon.0) 1609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:24.962 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:23.664165+0000 mon.a (mon.0) 1610 : audit [INF] from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:06:24.962 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:23.664229+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e456: 8 total, 8 up, 5 in 2024-02-12T17:06:24.962 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:23.670229+0000 osd.0 (osd.0) 193 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:06:24.962 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:23.673118+0000 osd.0 (osd.0) 194 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:06:25.058 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:25.099 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:25.258 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:25.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:25.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:25.411 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:25.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:25.561 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:06:25.561 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:06:25.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:06:25.561 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:06:25.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(8.5 272/258 [PushReplyOp(8:a1294c77:::benchmark_data_smithi033_105085_object111213:head)])", 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T16:59:41.217937+0000", 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 404.33820623899999, 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.090223213999999996, 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217937+0000", 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217937+0000", 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217937+0000", 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9809999999999997e-06 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217941+0000", 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2790000000000004e-06 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217947+0000", 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6910000000000002e-06 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.217953+0000", 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089966955000000001 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.307920+0000", 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024068900000000001 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T16:59:41.308160+0000", 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.090223213999999996 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(8.5 287/258 [PushReplyOp(8:a4c25a62:::benchmark_data_smithi033_105085_object101867:head)])", 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:24.801704+0000", 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 360.75443982000002, 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086370809000000007, 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801704+0000", 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801704+0000", 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801703+0000", 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8499999999999996e-06 2024-02-12T17:06:25.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801710+0000", 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9939999999999997e-06 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801716+0000", 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.75e-06 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.801722+0000", 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086105397 2024-02-12T17:06:25.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.887827+0000", 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024725700000000002 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:24.888074+0000", 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086370809000000007 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:222593 8.5 8:ad1bdf22:::benchmark_data_smithi033_105085_object110935:head [delete] snapc 0=[] ondisk+write+known_if_redirected e297)", 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:49.841125+0000", 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.71501881299997, 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0090313639999999, 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 222593 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.841125+0000", 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.841125+0000", 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.841121+0000", 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.096e-05 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.841132+0000", 2024-02-12T17:06:25.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9763999999999999e-05 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.841152+0000", 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4570000000000005e-06 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.841160+0000", 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000102903 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.841263+0000", 2024-02-12T17:06:25.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0729999999999994e-06 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.841272+0000", 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99804128199999997 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.839314+0000", 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013038599999999999 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.839444+0000", 2024-02-12T17:06:25.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015689999999999999 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4", 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.839601+0000", 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082159969999999992 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.847817+0000", 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002196773 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850014+0000", 2024-02-12T17:06:25.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6063999999999996e-05 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850080+0000", 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8886e-05 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850119+0000", 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7221000000000002e-05 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850156+0000", 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0090313639999999 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:222618 8.5 8:af61ff8a:::benchmark_data_smithi033_105085_object110960:head [delete] snapc 0=[] ondisk+write+known_if_redirected e297)", 2024-02-12T17:06:25.575 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:49.863209+0000", 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.69293469600001, 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98706797599999996, 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 222618 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863209+0000", 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863209+0000", 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863206+0000", 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2058999999999999e-05 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863218+0000", 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5063e-05 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863233+0000", 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5969999999999997e-06 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863239+0000", 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8109000000000002e-05 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863297+0000", 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5019999999999992e-06 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863306+0000", 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97851094800000005 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.841817+0000", 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8023000000000007e-05 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.841915+0000", 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3638000000000001e-05 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4", 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.841998+0000", 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076886790000000003 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.849687+0000", 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000360226 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850047+0000", 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156341 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850204+0000", 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7935999999999997e-05 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850242+0000", 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4965999999999997e-05 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850277+0000", 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98706797599999996 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:222620 8.5 8:ae6ff204:::benchmark_data_smithi033_105085_object110962:head [delete] snapc 0=[] ondisk+write+known_if_redirected e297)", 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:49.863256+0000", 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.69288724500001, 2024-02-12T17:06:25.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98712413200000004, 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 222620 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863256+0000", 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863256+0000", 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863255+0000", 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7160000000000001e-06 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863259+0000", 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1180000000000003e-06 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863266+0000", 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.78e-06 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863271+0000", 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018165299999999999 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863453+0000", 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1720000000000001e-06 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.863460+0000", 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98050193299999999 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.843962+0000", 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011538400000000001 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.844077+0000", 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015937199999999999 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4", 2024-02-12T17:06:25.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.844236+0000", 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054564569999999996 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.849693+0000", 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060721199999999999 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850300+0000", 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7833999999999997e-05 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850338+0000", 2024-02-12T17:06:25.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2478999999999999e-05 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850360+0000", 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9836000000000001e-05 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850380+0000", 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98712413200000004 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:222636 8.5 8:ad9b56ae:::benchmark_data_smithi033_105085_object110978:head [delete] snapc 0=[] ondisk+write+known_if_redirected e297)", 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:49.870597+0000", 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.68554667900003, 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98018621299999997, 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 222636 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.870597+0000", 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.870597+0000", 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.870594+0000", 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0260000000000007e-06 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.870602+0000", 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3637e-05 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.870616+0000", 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8510000000000001e-06 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.870623+0000", 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6489000000000002e-05 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.870669+0000", 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0940000000000008e-06 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.870677+0000", 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97538088999999994 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.846058+0000", 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7416000000000006e-05 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.846136+0000", 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011257800000000001 2024-02-12T17:06:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4", 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.846248+0000", 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034978299999999999 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.849746+0000", 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00057026500000000003 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850316+0000", 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038253000000000001 2024-02-12T17:06:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850699+0000", 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5751999999999998e-05 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850745+0000", 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8209e-05 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850783+0000", 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98018621299999997 2024-02-12T17:06:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.35042.0:222659 8.5 8:a82969d3:::benchmark_data_smithi033_105085_object111001:head [delete] snapc 0=[] ondisk+write+known_if_redirected e297)", 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:00:49.931127+0000", 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.62501675999999, 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.921646093, 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.35042", 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/650336487", 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 222659 2024-02-12T17:06:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.931127+0000", 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.931127+0000", 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.931124+0000", 2024-02-12T17:06:25.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8909999999999998e-06 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.931131+0000", 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2323999999999999e-05 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.931144+0000", 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5359999999999998e-06 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.931150+0000", 2024-02-12T17:06:25.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5413000000000005e-05 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.931216+0000", 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5980000000000003e-06 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:49.931225+0000", 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.91662095300000002 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.847846+0000", 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000108653 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.847955+0000", 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000140836 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4", 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.848096+0000", 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024934240000000002 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.850589+0000", 2024-02-12T17:06:25.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016085159999999999 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.852198+0000", 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051705999999999996 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.852715+0000", 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1503e-05 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.852746+0000", 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6537e-05 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:00:50.852773+0000", 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.921646093 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:147771 9.e 9:717b8b63:::benchmark_data_smithi033_131576_object10181:head [delete] snapc 0=[] ondisk+write+known_if_redirected e336)", 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:06.897736+0000", 2024-02-12T17:06:25.601 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 198.65840706399999, 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.097459618999999997, 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 147771 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897736+0000", 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897736+0000", 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897735+0000", 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4469999999999997e-06 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897740+0000", 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4569000000000001e-05 2024-02-12T17:06:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897755+0000", 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.691e-06 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897761+0000", 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0040999999999998e-05 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897801+0000", 2024-02-12T17:06:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7900999999999999e-05 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897869+0000", 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010100800000000001 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4", 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.897970+0000", 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0039893699999999999 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.901960+0000", 2024-02-12T17:06:25.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124048 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.902084+0000", 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.093025338999999999 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.995109+0000", 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3871000000000001e-05 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.995153+0000", 2024-02-12T17:06:25.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2979999999999998e-05 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.995196+0000", 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.097459618999999997 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:147774 9.2 9:403637fa:::benchmark_data_smithi033_131576_object10184:head [delete] snapc 0=[] ondisk+write+known_if_redirected e336)", 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:06.900149+0000", 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 198.65599462899999, 2024-02-12T17:06:25.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.090527270000000007, 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 147774 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900149+0000", 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900149+0000", 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900148+0000", 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9679999999999997e-06 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900153+0000", 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6560000000000004e-06 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900162+0000", 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2559999999999999e-06 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900168+0000", 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8108000000000001e-05 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900217+0000", 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5055000000000005e-05 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900302+0000", 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2354000000000006e-05 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2", 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.900374+0000", 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001602664 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.901977+0000", 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002163136 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.904140+0000", 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086461495999999999 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.990601+0000", 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8603000000000003e-05 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.990640+0000", 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6071000000000001e-05 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:06.990676+0000", 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.090527270000000007 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:180475 9.d 9:b93f71ab:::benchmark_data_smithi033_131576_object42885:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:44.755766+0000", 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.80037759999999, 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086103842999999999, 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.613 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 180475 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.755766+0000", 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.755766+0000", 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:25.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.755765+0000", 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3950000000000003e-06 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.755769+0000", 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0700000000000003e-06 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.755775+0000", 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4789999999999998e-06 2024-02-12T17:06:25.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.755780+0000", 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024643540000000002 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.758244+0000", 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0754999999999994e-05 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.758335+0000", 2024-02-12T17:06:25.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000110751 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 4,5", 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.758446+0000", 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056536620000000003 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.764099+0000", 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01088883 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.774988+0000", 2024-02-12T17:06:25.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.066806306999999995 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.841794+0000", 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9267999999999997e-05 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.841834+0000", 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6001000000000003e-05 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:44.841870+0000", 2024-02-12T17:06:25.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.086103842999999999 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:181880 9.9 9:9dbc7b7a:::benchmark_data_smithi033_131576_object44290:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:46.732465+0000", 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.82367853900001, 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.098221137, 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 181880 2024-02-12T17:06:25.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.732465+0000", 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.732465+0000", 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.732463+0000", 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0589999999999997e-06 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.732470+0000", 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3850000000000008e-06 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.732479+0000", 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2629999999999997e-06 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.732483+0000", 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091196300000000001 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.733395+0000", 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7670999999999995e-05 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.733483+0000", 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010042700000000001 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,5", 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.733584+0000", 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015141670000000001 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.735098+0000", 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015372719999999999 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.736635+0000", 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015994477999999999 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.752629+0000", 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.077874709 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.830504+0000", 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.092e-05 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.830555+0000", 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013085399999999999 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.830686+0000", 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.098221137 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:181881 9.3 9:c0be1fda:::benchmark_data_smithi033_131576_object44291:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:46.735270+0000", 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.820873414, 2024-02-12T17:06:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.094448797000000001, 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 181881 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.735270+0000", 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.735270+0000", 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.735268+0000", 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0909999999999998e-06 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.735274+0000", 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6290000000000001e-06 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.735282+0000", 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.735286+0000", 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00087232300000000004 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.736158+0000", 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000133382 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.736292+0000", 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000161395 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,5", 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.736453+0000", 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011915443 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.748369+0000", 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8773000000000001e-05 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.748387+0000", 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0156000000000001e-05 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.748397+0000", 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047671190000000002 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.796069+0000", 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1814999999999998e-05 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.796120+0000", 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033598281000000001 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.829719+0000", 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.094448797000000001 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.632 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:181883 9.d 9:bd8923f6:::benchmark_data_smithi033_131576_object44293:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:46.738027+0000", 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.818116489, 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.093059046000000006, 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 181883 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738027+0000", 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738027+0000", 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738025+0000", 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3199999999999999e-06 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738031+0000", 2024-02-12T17:06:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4850000000000003e-06 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738038+0000", 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2619999999999998e-06 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738043+0000", 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023407899999999999 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738277+0000", 2024-02-12T17:06:25.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4386000000000001e-05 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738372+0000", 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00029766500000000002 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 4,5", 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.738670+0000", 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092770230000000006 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.747947+0000", 2024-02-12T17:06:25.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0046185619999999997 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.752565+0000", 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.078444486999999993 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.831010+0000", 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9604000000000002e-05 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.831049+0000", 2024-02-12T17:06:25.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6699000000000002e-05 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:46.831086+0000", 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.093059046000000006 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24355.0:2083 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94508044033920 gen 6] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.519335+0000", 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 156.036808517, 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1633644109999999, 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24355", 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2083 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519335+0000", 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519335+0000", 2024-02-12T17:06:25.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519333+0000", 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3970000000000002e-06 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519339+0000", 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9990000000000008e-06 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519346+0000", 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.699e-06 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519352+0000", 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017726999999999999 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519529+0000", 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8419999999999999e-06 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-02-12T17:06:25.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519536+0000", 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034153570000000001 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.553690+0000", 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0160000000000008e-06 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.553698+0000", 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.022487199 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.576185+0000", 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8740000000000007e-06 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.576195+0000", 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0615738669999999 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.637769+0000", 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120844 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.637890+0000", 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.044809557999999999 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.682699+0000", 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1633644109999999 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183796 9.9 9:9b9d44ff:::benchmark_data_smithi033_131576_object46206:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.519476+0000", 2024-02-12T17:06:25.644 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 156.03666721299999, 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.163018455, 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183796 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519476+0000", 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519476+0000", 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519475+0000", 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8999999999999997e-06 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519480+0000", 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1370000000000002e-06 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519488+0000", 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1850000000000001e-06 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519494+0000", 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0197000000000004e-05 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519584+0000", 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012856599999999999 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519713+0000", 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015817699999999999 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,5", 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519871+0000", 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0096412979999999995 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.529512+0000", 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082318930000000005 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.537744+0000", 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-05 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.537754+0000", 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1369527939999999 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.674707+0000", 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000119757 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.674827+0000", 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075961400000000004 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.682423+0000", 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1493000000000001e-05 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.682495+0000", 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.163018455 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.650 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183799 9.3 9:c6d8c0ff:::benchmark_data_smithi033_131576_object46209:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.521648+0000", 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 156.03449524199999, 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1647798149999999, 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183799 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521648+0000", 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521648+0000", 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521647+0000", 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8599999999999998e-06 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521653+0000", 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1588e-05 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521664+0000", 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3470000000000004e-06 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521670+0000", 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056849500000000002 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522239+0000", 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120591 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522360+0000", 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015625200000000001 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,5", 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522516+0000", 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014359825 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.536876+0000", 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8340000000000001e-06 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.536882+0000", 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1419698709999999 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.678852+0000", 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012575800000000001 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.678978+0000", 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.007399203 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.686377+0000", 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0636000000000001e-05 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.686428+0000", 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1647798149999999 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.656 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183801 9.9 9:96e80f0d:::benchmark_data_smithi033_131576_object46211:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.526679+0000", 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 156.02946476400001, 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1562766440000001, 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183801 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526679+0000", 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526679+0000", 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526677+0000", 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8290000000000002e-06 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526683+0000", 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4666e-05 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526698+0000", 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9229999999999996e-06 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526705+0000", 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2431000000000001e-05 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526747+0000", 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6534999999999997e-05 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526834+0000", 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010707 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,5", 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526941+0000", 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010838854 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.537780+0000", 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0490000000000004e-06 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.537788+0000", 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.137077535 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.674865+0000", 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010332600000000001 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:06:25.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.674969+0000", 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079168629999999997 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.682886+0000", 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9560000000000005e-05 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.682955+0000", 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1562766440000001 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.662 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183806 9.2 9:401f2b8c:::benchmark_data_smithi033_131576_object46216:head [delete] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.566880+0000", 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.98926324499999, 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1056333119999999, 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183806 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566880+0000", 2024-02-12T17:06:25.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566880+0000", 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566878+0000", 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.486e-06 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566885+0000", 2024-02-12T17:06:25.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7730000000000005e-06 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566894+0000", 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4620000000000001e-06 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566900+0000", 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494170000000001 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.573450+0000", 2024-02-12T17:06:25.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3419999999999997e-06 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.573459+0000", 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.085010405 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.658470+0000", 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012053300000000001 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.658590+0000", 2024-02-12T17:06:25.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015071 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,3", 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.658741+0000", 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006408815 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.665150+0000", 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8158000000000001e-05 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.665208+0000", 2024-02-12T17:06:25.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045370510000000003 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.669745+0000", 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025681050000000002 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672313+0000", 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1764e-05 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672375+0000", 2024-02-12T17:06:25.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138634 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672513+0000", 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1056333119999999 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183808 9.2 9:4994a024:::benchmark_data_smithi033_131576_object46218:head [delete] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.566913+0000", 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.989230741, 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.105857358, 2024-02-12T17:06:25.669 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183808 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566913+0000", 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566913+0000", 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566912+0000", 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1810000000000003e-06 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566916+0000", 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4219999999999999e-06 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566921+0000", 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4379999999999997e-06 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566926+0000", 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065533589999999999 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.573479+0000", 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3410000000000002e-06 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.573484+0000", 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.087181733 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.660666+0000", 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012873100000000001 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.660795+0000", 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015155499999999999 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,3", 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.660946+0000", 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042465819999999996 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.665193+0000", 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1631e-05 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.665215+0000", 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045630260000000004 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.669778+0000", 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027785520000000001 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672556+0000", 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1248999999999996e-05 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672617+0000", 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000152478 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672770+0000", 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.105857358 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24355.0:2085 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94508044033920 gen 7] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:50.561581+0000", 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.99456193699999, 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1212566980000001, 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:06:25.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24355", 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2085 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561581+0000", 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561581+0000", 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561579+0000", 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9589999999999995e-06 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561587+0000", 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1561e-05 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561598+0000", 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2279999999999998e-06 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561606+0000", 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016483583999999999 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.578090+0000", 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1689999999999992e-06 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.578098+0000", 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0600229800000001 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.638121+0000", 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7311000000000007e-05 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.638189+0000", 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.044649568000000001 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.682838+0000", 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1212566980000001 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.680 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:06:25.681 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:06:25.681 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:06:25.681 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:25.849 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 1 16 3286 3270 204.196 204.375 0.00992707 0.00486846 2024-02-12T17:06:25.862 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:25.864 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:24.421596+0000 mgr.y (mgr.24481) 636 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:25.864 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:24.650061+0000 osd.0 (osd.0) 195 : cluster [DBG] 5.c scrub starts 2024-02-12T17:06:25.864 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:24.653024+0000 osd.0 (osd.0) 196 : cluster [DBG] 5.c scrub ok 2024-02-12T17:06:25.864 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:24.691297+0000 mgr.y (mgr.24481) 637 : cluster [DBG] pgmap v881: 128 pgs: 128 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 960 B/s rd, 0 op/s 2024-02-12T17:06:25.864 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:24.972355+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:06:25.864 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:24.976041+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:06:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:25.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:25.966 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:26.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:24.421596+0000 mgr.y (mgr.24481) 636 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:26.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:24.650061+0000 osd.0 (osd.0) 195 : cluster [DBG] 5.c scrub starts 2024-02-12T17:06:26.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:24.653024+0000 osd.0 (osd.0) 196 : cluster [DBG] 5.c scrub ok 2024-02-12T17:06:26.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:24.691297+0000 mgr.y (mgr.24481) 637 : cluster [DBG] pgmap v881: 128 pgs: 128 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 960 B/s rd, 0 op/s 2024-02-12T17:06:26.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:24.972355+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:06:26.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:24.976041+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:06:26.015 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:24.421596+0000 mgr.y (mgr.24481) 636 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:26.015 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:24.650061+0000 osd.0 (osd.0) 195 : cluster [DBG] 5.c scrub starts 2024-02-12T17:06:26.015 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:24.653024+0000 osd.0 (osd.0) 196 : cluster [DBG] 5.c scrub ok 2024-02-12T17:06:26.015 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:24.691297+0000 mgr.y (mgr.24481) 637 : cluster [DBG] pgmap v881: 128 pgs: 128 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 960 B/s rd, 0 op/s 2024-02-12T17:06:26.015 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:24.972355+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:06:26.015 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:24.976041+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:06:26.015 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:26.151 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:26.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:26.255 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:26.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T17:06:26.299 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:26.331 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:26.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:26.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:06:26.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:26.470 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:26.475 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:26.494 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:26.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:26.579 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:06:26.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:26.611 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:06:26.629 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:26.646 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:06:26.651 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:26.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:26.673 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:25.127605+0000 osd.5 (osd.5) 203 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:25.129186+0000 osd.5 (osd.5) 204 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:25.564551+0000 osd.4 (osd.4) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:25.564599+0000 osd.4 (osd.4) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:25.673252+0000 osd.0 (osd.0) 197 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:25.676712+0000 osd.0 (osd.0) 198 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:25.677562+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e457: 8 total, 8 up, 5 in 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:26.346338+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e458: 8 total, 8 up, 5 in 2024-02-12T17:06:26.682 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:26.533360+0000 mon.b (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.33:0/1845218785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:26.720 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:06:26.751 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:06:26.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:26.774 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:06:26.789 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:26.811 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:06:26.815 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:26.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:06:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:06:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:06:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:06:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:06:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:06:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:06:26.821 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:06:26.848 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 2 16 5642 5626 175.727 147.25 0.0135114 0.0056616 2024-02-12T17:06:26.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:25.127605+0000 osd.5 (osd.5) 203 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:25.129186+0000 osd.5 (osd.5) 204 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:25.564551+0000 osd.4 (osd.4) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:25.564599+0000 osd.4 (osd.4) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:25.673252+0000 osd.0 (osd.0) 197 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:25.676712+0000 osd.0 (osd.0) 198 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:25.677562+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e457: 8 total, 8 up, 5 in 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:26.346338+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e458: 8 total, 8 up, 5 in 2024-02-12T17:06:26.959 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:26.533360+0000 mon.b (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.33:0/1845218785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:26.961 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:26.987 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:26.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:25.127605+0000 osd.5 (osd.5) 203 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:06:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:25.129186+0000 osd.5 (osd.5) 204 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:06:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:25.564551+0000 osd.4 (osd.4) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:25.564599+0000 osd.4 (osd.4) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:25.673252+0000 osd.0 (osd.0) 197 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:06:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:25.676712+0000 osd.0 (osd.0) 198 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:06:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:25.677562+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e457: 8 total, 8 up, 5 in 2024-02-12T17:06:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:26.346338+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e458: 8 total, 8 up, 5 in 2024-02-12T17:06:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:26.533360+0000 mon.b (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.33:0/1845218785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:27.091 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:27.142 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:27.849 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 3 16 8426 8410 175.125 174 0.00469537 0.00570213 2024-02-12T17:06:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:26.447330+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.f scrub starts 2024-02-12T17:06:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:26.583647+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.f scrub ok 2024-02-12T17:06:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:26.697685+0000 mgr.y (mgr.24481) 638 : cluster [DBG] pgmap v884: 128 pgs: 128 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:26.447330+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.f scrub starts 2024-02-12T17:06:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:26.583647+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.f scrub ok 2024-02-12T17:06:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:26.697685+0000 mgr.y (mgr.24481) 638 : cluster [DBG] pgmap v884: 128 pgs: 128 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:26.447330+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.f scrub starts 2024-02-12T17:06:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:26.583647+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.f scrub ok 2024-02-12T17:06:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:26.697685+0000 mgr.y (mgr.24481) 638 : cluster [DBG] pgmap v884: 128 pgs: 128 active+clean; 5.7 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:06:28.849 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 4 16 11335 11319 176.792 181.812 0.00323103 0.00563846 2024-02-12T17:06:29.849 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 5 16 13912 13896 173.643 161.062 0.00392636 0.0057506 2024-02-12T17:06:30.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:28.068386+0000 osd.5 (osd.5) 205 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:06:30.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:28.073126+0000 osd.5 (osd.5) 206 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:06:30.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:28.698334+0000 mgr.y (mgr.24481) 639 : cluster [DBG] pgmap v885: 128 pgs: 128 active+clean; 401 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-12T17:06:30.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:29.473527+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.1b scrub starts 2024-02-12T17:06:30.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:29.479604+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.1b scrub ok 2024-02-12T17:06:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:28.068386+0000 osd.5 (osd.5) 205 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:06:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:28.073126+0000 osd.5 (osd.5) 206 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:06:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:28.698334+0000 mgr.y (mgr.24481) 639 : cluster [DBG] pgmap v885: 128 pgs: 128 active+clean; 401 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-12T17:06:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:29.473527+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.1b scrub starts 2024-02-12T17:06:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:29.479604+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.1b scrub ok 2024-02-12T17:06:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:28.068386+0000 osd.5 (osd.5) 205 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:06:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:28.073126+0000 osd.5 (osd.5) 206 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:06:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:28.698334+0000 mgr.y (mgr.24481) 639 : cluster [DBG] pgmap v885: 128 pgs: 128 active+clean; 401 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-12T17:06:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:29.473527+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.1b scrub starts 2024-02-12T17:06:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:29.479604+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.1b scrub ok 2024-02-12T17:06:30.850 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 6 16 16151 16135 167.991 139.938 0.00608103 0.00594587 2024-02-12T17:06:31.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:06:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:30] "GET /metrics HTTP/1.1" 200 176418 "" "Prometheus/2.33.4" 2024-02-12T17:06:31.851 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 7 16 18673 18657 166.493 157.625 0.00676888 0.00600092 2024-02-12T17:06:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:30.698791+0000 mgr.y (mgr.24481) 640 : cluster [DBG] pgmap v886: 128 pgs: 128 active+clean; 655 MiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 1.4 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-12T17:06:32.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:30.698791+0000 mgr.y (mgr.24481) 640 : cluster [DBG] pgmap v886: 128 pgs: 128 active+clean; 655 MiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 1.4 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-12T17:06:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:30.698791+0000 mgr.y (mgr.24481) 640 : cluster [DBG] pgmap v886: 128 pgs: 128 active+clean; 655 MiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 1.4 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-12T17:06:32.780 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:06:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:06:32.851 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 8 16 20823 20807 162.476 134.375 0.00511587 0.00614844 2024-02-12T17:06:33.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:31.065922+0000 osd.5 (osd.5) 207 : cluster [DBG] 2.17 scrub starts 2024-02-12T17:06:33.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:31.071542+0000 osd.5 (osd.5) 208 : cluster [DBG] 2.17 scrub ok 2024-02-12T17:06:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:31.065922+0000 osd.5 (osd.5) 207 : cluster [DBG] 2.17 scrub starts 2024-02-12T17:06:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:31.071542+0000 osd.5 (osd.5) 208 : cluster [DBG] 2.17 scrub ok 2024-02-12T17:06:33.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:31.065922+0000 osd.5 (osd.5) 207 : cluster [DBG] 2.17 scrub starts 2024-02-12T17:06:33.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:31.071542+0000 osd.5 (osd.5) 208 : cluster [DBG] 2.17 scrub ok 2024-02-12T17:06:33.832 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:33.851 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 9 16 22931 22915 159.062 131.75 0.00664487 0.00627628 2024-02-12T17:06:33.870 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:34.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:32.699297+0000 mgr.y (mgr.24481) 641 : cluster [DBG] pgmap v887: 128 pgs: 128 active+clean; 655 MiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-12T17:06:34.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:32.974579+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.f scrub starts 2024-02-12T17:06:34.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:32.977630+0000 osd.5 (osd.5) 210 : cluster [DBG] 5.f scrub ok 2024-02-12T17:06:34.152 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:32.699297+0000 mgr.y (mgr.24481) 641 : cluster [DBG] pgmap v887: 128 pgs: 128 active+clean; 655 MiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-12T17:06:34.152 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:32.974579+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.f scrub starts 2024-02-12T17:06:34.152 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:32.977630+0000 osd.5 (osd.5) 210 : cluster [DBG] 5.f scrub ok 2024-02-12T17:06:34.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:32.699297+0000 mgr.y (mgr.24481) 641 : cluster [DBG] pgmap v887: 128 pgs: 128 active+clean; 655 MiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-12T17:06:34.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:32.974579+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.f scrub starts 2024-02-12T17:06:34.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:32.977630+0000 osd.5 (osd.5) 210 : cluster [DBG] 5.f scrub ok 2024-02-12T17:06:34.499 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:34.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:34.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:34.649 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:34.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:06:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:06:34.852 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 10 16 24933 24917 155.657 125.125 0.0108133 0.00641786 2024-02-12T17:06:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:34.376171+0000 osd.3 (osd.3) 133 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:06:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:34.381758+0000 osd.3 (osd.3) 134 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:06:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:34.376171+0000 osd.3 (osd.3) 133 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:06:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:34.381758+0000 osd.3 (osd.3) 134 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:06:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:34.376171+0000 osd.3 (osd.3) 133 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:06:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:34.381758+0000 osd.3 (osd.3) 134 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:06:35.310 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:35.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:35.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:35.466 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:35.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:06:35.854 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 11 16 27007 26991 153.288 129.625 0.0115561 0.00651635 2024-02-12T17:06:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:34.431521+0000 mgr.y (mgr.24481) 642 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:34.703735+0000 mgr.y (mgr.24481) 643 : cluster [DBG] pgmap v888: 128 pgs: 128 active+clean; 1.4 GiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-02-12T17:06:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:34.978797+0000 osd.5 (osd.5) 211 : cluster [DBG] 2.16 scrub starts 2024-02-12T17:06:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:34.981894+0000 osd.5 (osd.5) 212 : cluster [DBG] 2.16 scrub ok 2024-02-12T17:06:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:35.403399+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:06:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:35.412984+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:06:36.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:34.431521+0000 mgr.y (mgr.24481) 642 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:36.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:34.703735+0000 mgr.y (mgr.24481) 643 : cluster [DBG] pgmap v888: 128 pgs: 128 active+clean; 1.4 GiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-02-12T17:06:36.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:34.978797+0000 osd.5 (osd.5) 211 : cluster [DBG] 2.16 scrub starts 2024-02-12T17:06:36.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:34.981894+0000 osd.5 (osd.5) 212 : cluster [DBG] 2.16 scrub ok 2024-02-12T17:06:36.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:35.403399+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:06:36.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:35.412984+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:06:36.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:34.431521+0000 mgr.y (mgr.24481) 642 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:36.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:34.703735+0000 mgr.y (mgr.24481) 643 : cluster [DBG] pgmap v888: 128 pgs: 128 active+clean; 1.4 GiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-02-12T17:06:36.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:34.978797+0000 osd.5 (osd.5) 211 : cluster [DBG] 2.16 scrub starts 2024-02-12T17:06:36.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:34.981894+0000 osd.5 (osd.5) 212 : cluster [DBG] 2.16 scrub ok 2024-02-12T17:06:36.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:35.403399+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:06:36.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:35.412984+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:06:36.733 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:36.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:36.839 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:36.852 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 12 16 28680 28664 149.228 104.562 0.00710969 0.00669582 2024-02-12T17:06:36.881 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:36.914 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:36.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:37.025 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:35.061356+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:06:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:35.066857+0000 osd.7 (osd.7) 207 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:06:37.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:35.061356+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:06:37.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:35.066857+0000 osd.7 (osd.7) 207 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:06:37.061 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:35.061356+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:06:37.062 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:35.066857+0000 osd.7 (osd.7) 207 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:06:37.064 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:37.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:37.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:37.238 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:37.264 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:37.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:37.368 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:37.433 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:37.463 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:37.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:37.568 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:37.598 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:37.602 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:37.620 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:37.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:37.652 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:06:37.652 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:06:37.725 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:37.778 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:37.818 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:37.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:37.852 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 13 16 30431 30415 146.168 109.438 0.0119533 0.00683479 2024-02-12T17:06:37.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:37.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:36.704714+0000 mgr.y (mgr.24481) 644 : cluster [DBG] pgmap v889: 128 pgs: 128 active+clean; 1.4 GiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 988 B/s rd, 134 MiB/s wr, 2.15k op/s 2024-02-12T17:06:37.981 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:36.704714+0000 mgr.y (mgr.24481) 644 : cluster [DBG] pgmap v889: 128 pgs: 128 active+clean; 1.4 GiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 988 B/s rd, 134 MiB/s wr, 2.15k op/s 2024-02-12T17:06:37.985 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:37.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:36.704714+0000 mgr.y (mgr.24481) 644 : cluster [DBG] pgmap v889: 128 pgs: 128 active+clean; 1.4 GiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 988 B/s rd, 134 MiB/s wr, 2.15k op/s 2024-02-12T17:06:38.853 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 14 16 32213 32197 143.683 111.375 0.00971704 0.00695379 2024-02-12T17:06:38.864 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T17:06:39.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:37.062967+0000 osd.7 (osd.7) 208 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:06:39.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:37.116602+0000 osd.7 (osd.7) 209 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:06:39.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:37.062967+0000 osd.7 (osd.7) 208 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:06:39.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:37.116602+0000 osd.7 (osd.7) 209 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:06:39.194 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:06:39.209 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:06:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:37.062967+0000 osd.7 (osd.7) 208 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:06:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:37.116602+0000 osd.7 (osd.7) 209 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:06:39.253 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:39.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:39.363 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:39.419 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:39.458 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:39.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:39.563 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:39.620 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:39.705 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:39.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:39.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:39.853 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 15 16 33644 33628 140.063 89.4375 0.00538477 0.00712891 2024-02-12T17:06:39.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:38.705285+0000 mgr.y (mgr.24481) 645 : cluster [DBG] pgmap v890: 128 pgs: 128 active+clean; 1.7 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 852 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-12T17:06:39.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:38.923775+0000 osd.5 (osd.5) 213 : cluster [DBG] 2.c scrub starts 2024-02-12T17:06:39.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:38.925647+0000 osd.5 (osd.5) 214 : cluster [DBG] 2.c scrub ok 2024-02-12T17:06:39.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:39.022639+0000 mgr.y (mgr.24481) 646 : audit [DBG] from='client.25615 v1:172.21.15.33:0/4167990197' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:06:39.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:39.413928+0000 mon.c (mon.2) 242 : audit [INF] from='client.? v1:172.21.15.33:0/989440358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:39.865 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:39.414318+0000 mon.a (mon.0) 1614 : audit [INF] from='client.35768 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:39.865 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:38.705285+0000 mgr.y (mgr.24481) 645 : cluster [DBG] pgmap v890: 128 pgs: 128 active+clean; 1.7 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 852 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-12T17:06:39.865 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:38.923775+0000 osd.5 (osd.5) 213 : cluster [DBG] 2.c scrub starts 2024-02-12T17:06:39.865 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:38.925647+0000 osd.5 (osd.5) 214 : cluster [DBG] 2.c scrub ok 2024-02-12T17:06:39.865 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:39.022639+0000 mgr.y (mgr.24481) 646 : audit [DBG] from='client.25615 v1:172.21.15.33:0/4167990197' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:06:39.866 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:39.413928+0000 mon.c (mon.2) 242 : audit [INF] from='client.? v1:172.21.15.33:0/989440358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:39.866 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:39.414318+0000 mon.a (mon.0) 1614 : audit [INF] from='client.35768 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:39.866 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:06:39.867 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:38.705285+0000 mgr.y (mgr.24481) 645 : cluster [DBG] pgmap v890: 128 pgs: 128 active+clean; 1.7 GiB data, 8.9 GiB used, 438 GiB / 447 GiB avail; 852 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-12T17:06:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:38.923775+0000 osd.5 (osd.5) 213 : cluster [DBG] 2.c scrub starts 2024-02-12T17:06:40.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:38.925647+0000 osd.5 (osd.5) 214 : cluster [DBG] 2.c scrub ok 2024-02-12T17:06:40.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:39.022639+0000 mgr.y (mgr.24481) 646 : audit [DBG] from='client.25615 v1:172.21.15.33:0/4167990197' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:06:40.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:39.413928+0000 mon.c (mon.2) 242 : audit [INF] from='client.? v1:172.21.15.33:0/989440358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:40.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:39.414318+0000 mon.a (mon.0) 1614 : audit [INF] from='client.35768 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:06:40.858 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 16 16 35398 35382 138.148 109.625 0.00617936 0.00723251 2024-02-12T17:06:41.041 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:41.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:41.053 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":890,"stamp":"2024-02-12T17:06:38.704858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1848120996,"num_objects":28392,"num_object_clones":0,"num_object_copies":85176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28392,"num_whiteouts":0,"num_read":1437,"num_read_kb":1189,"num_write":28914,"num_write_kb":1804841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":15798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29000,"ondisk_log_size":29000,"up":384,"acting":384,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":384,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":9331784,"kb_used_data":5457632,"kb_used_omap":0,"kb_used_meta":3874112,"kb_avail":459393976,"statfs":{"total":479975178240,"available":470419431424,"internally_reserved":0,"allocated":5588615168,"data_stored":5584444173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3967090688},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1848115200,"num_objects":28200,"num_object_clones":0,"num_object_copies":84600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28200,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":28200,"num_write_kb":1804800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007412"},"pg_stats":[{"pgid":"2.1f","version":"0'0","reported_seq":146,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686705+0000","last_change":"2024-02-12T17:04:39.916313+0000","last_active":"2024-02-12T17:06:25.686705+0000","last_peered":"2024-02-12T17:06:25.686705+0000","last_clean":"2024-02-12T17:06:25.686705+0000","last_became_active":"2024-02-12T17:04:39.916010+0000","last_became_peered":"2024-02-12T17:04:39.916010+0000","last_unstale":"2024-02-12T17:06:25.686705+0000","last_undegraded":"2024-02-12T17:06:25.686705+0000","last_fullsized":"2024-02-12T17:06:25.686705+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.732293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.732293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":135,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684429+0000","last_change":"2024-02-12T17:03:52.280899+0000","last_active":"2024-02-12T17:06:25.684429+0000","last_peered":"2024-02-12T17:06:25.684429+0000","last_clean":"2024-02-12T17:06:25.684429+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:06:25.684429+0000","last_undegraded":"2024-02-12T17:06:25.684429+0000","last_fullsized":"2024-02-12T17:06:25.684429+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:01:26.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.047527+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":119,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686990+0000","last_change":"2024-02-12T17:03:50.545988+0000","last_active":"2024-02-12T17:06:25.686990+0000","last_peered":"2024-02-12T17:06:25.686990+0000","last_clean":"2024-02-12T17:06:25.686990+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:06:25.686990+0000","last_undegraded":"2024-02-12T17:06:25.686990+0000","last_fullsized":"2024-02-12T17:06:25.686990+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.176431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":510,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683841+0000","last_change":"2024-02-12T17:04:41.353781+0000","last_active":"2024-02-12T17:06:25.683841+0000","last_peered":"2024-02-12T17:06:25.683841+0000","last_clean":"2024-02-12T17:06:25.683841+0000","last_became_active":"2024-02-12T17:04:39.873280+0000","last_became_peered":"2024-02-12T17:04:39.873280+0000","last_unstale":"2024-02-12T17:06:25.683841+0000","last_undegraded":"2024-02-12T17:06:25.683841+0000","last_fullsized":"2024-02-12T17:06:25.683841+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:01:45.154164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":452,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686161+0000","last_change":"2024-02-12T17:03:50.565273+0000","last_active":"2024-02-12T17:06:25.686161+0000","last_peered":"2024-02-12T17:06:25.686161+0000","last_clean":"2024-02-12T17:06:25.686161+0000","last_became_active":"2024-02-12T17:03:50.562938+0000","last_became_peered":"2024-02-12T17:03:50.562938+0000","last_unstale":"2024-02-12T17:06:25.686161+0000","last_undegraded":"2024-02-12T17:06:25.686161+0000","last_fullsized":"2024-02-12T17:06:25.686161+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:37.813008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:01:37.813008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":145,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685677+0000","last_change":"2024-02-12T17:05:56.388435+0000","last_active":"2024-02-12T17:06:25.685677+0000","last_peered":"2024-02-12T17:06:25.685677+0000","last_clean":"2024-02-12T17:06:25.685677+0000","last_became_active":"2024-02-12T17:04:39.889725+0000","last_became_peered":"2024-02-12T17:04:39.889725+0000","last_unstale":"2024-02-12T17:06:25.685677+0000","last_undegraded":"2024-02-12T17:06:25.685677+0000","last_fullsized":"2024-02-12T17:06:25.685677+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7","last_scrub_stamp":"2024-02-12T17:05:56.388366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.388366+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":426,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685599+0000","last_change":"2024-02-12T17:02:34.502434+0000","last_active":"2024-02-12T17:06:25.685599+0000","last_peered":"2024-02-12T17:06:25.685599+0000","last_clean":"2024-02-12T17:06:25.685599+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:06:25.685599+0000","last_undegraded":"2024-02-12T17:06:25.685599+0000","last_fullsized":"2024-02-12T17:06:25.685599+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.502332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":148,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684629+0000","last_change":"2024-02-12T17:04:40.923654+0000","last_active":"2024-02-12T17:06:25.684629+0000","last_peered":"2024-02-12T17:06:25.684629+0000","last_clean":"2024-02-12T17:06:25.684629+0000","last_became_active":"2024-02-12T17:04:39.865216+0000","last_became_peered":"2024-02-12T17:04:39.865216+0000","last_unstale":"2024-02-12T17:06:25.684629+0000","last_undegraded":"2024-02-12T17:06:25.684629+0000","last_fullsized":"2024-02-12T17:06:25.684629+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":133,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:29.479686+0000","last_change":"2024-02-12T17:06:29.479686+0000","last_active":"2024-02-12T17:06:29.479686+0000","last_peered":"2024-02-12T17:06:29.479686+0000","last_clean":"2024-02-12T17:06:29.479686+0000","last_became_active":"2024-02-12T17:04:39.887235+0000","last_became_peered":"2024-02-12T17:04:39.887235+0000","last_unstale":"2024-02-12T17:06:29.479686+0000","last_undegraded":"2024-02-12T17:06:29.479686+0000","last_fullsized":"2024-02-12T17:06:29.479686+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:29.479610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T17:06:29.479610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":152,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686787+0000","last_change":"2024-02-12T17:04:40.319567+0000","last_active":"2024-02-12T17:06:25.686787+0000","last_peered":"2024-02-12T17:06:25.686787+0000","last_clean":"2024-02-12T17:06:25.686787+0000","last_became_active":"2024-02-12T17:04:39.864612+0000","last_became_peered":"2024-02-12T17:04:39.864612+0000","last_unstale":"2024-02-12T17:06:25.686787+0000","last_undegraded":"2024-02-12T17:06:25.686787+0000","last_fullsized":"2024-02-12T17:06:25.686787+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":94,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685513+0000","last_change":"2024-02-12T17:04:39.888727+0000","last_active":"2024-02-12T17:06:25.685513+0000","last_peered":"2024-02-12T17:06:25.685513+0000","last_clean":"2024-02-12T17:06:25.685513+0000","last_became_active":"2024-02-12T17:04:39.888398+0000","last_became_peered":"2024-02-12T17:04:39.888398+0000","last_unstale":"2024-02-12T17:06:25.685513+0000","last_undegraded":"2024-02-12T17:06:25.685513+0000","last_fullsized":"2024-02-12T17:06:25.685513+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.157357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.157357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686839+0000","last_change":"2024-02-12T17:04:41.433996+0000","last_active":"2024-02-12T17:06:25.686839+0000","last_peered":"2024-02-12T17:06:25.686839+0000","last_clean":"2024-02-12T17:06:25.686839+0000","last_became_active":"2024-02-12T17:04:39.872998+0000","last_became_peered":"2024-02-12T17:04:39.872998+0000","last_unstale":"2024-02-12T17:06:25.686839+0000","last_undegraded":"2024-02-12T17:06:25.686839+0000","last_fullsized":"2024-02-12T17:06:25.686839+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":119,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684877+0000","last_change":"2024-02-12T17:03:50.533143+0000","last_active":"2024-02-12T17:06:25.684877+0000","last_peered":"2024-02-12T17:06:25.684877+0000","last_clean":"2024-02-12T17:06:25.684877+0000","last_became_active":"2024-02-12T17:03:50.532233+0000","last_became_peered":"2024-02-12T17:03:50.532233+0000","last_unstale":"2024-02-12T17:06:25.684877+0000","last_undegraded":"2024-02-12T17:06:25.684877+0000","last_fullsized":"2024-02-12T17:06:25.684877+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":314,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684125+0000","last_change":"2024-02-12T17:03:50.670847+0000","last_active":"2024-02-12T17:06:25.684125+0000","last_peered":"2024-02-12T17:06:25.684125+0000","last_clean":"2024-02-12T17:06:25.684125+0000","last_became_active":"2024-02-12T17:03:50.570118+0000","last_became_peered":"2024-02-12T17:03:50.570118+0000","last_unstale":"2024-02-12T17:06:25.684125+0000","last_undegraded":"2024-02-12T17:06:25.684125+0000","last_fullsized":"2024-02-12T17:06:25.684125+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:34.738385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.738385+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":438,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682584+0000","last_change":"2024-02-12T17:05:55.027256+0000","last_active":"2024-02-12T17:06:25.682584+0000","last_peered":"2024-02-12T17:06:25.682584+0000","last_clean":"2024-02-12T17:06:25.682584+0000","last_became_active":"2024-02-12T17:04:39.880094+0000","last_became_peered":"2024-02-12T17:04:39.880094+0000","last_unstale":"2024-02-12T17:06:25.682584+0000","last_undegraded":"2024-02-12T17:06:25.682584+0000","last_fullsized":"2024-02-12T17:06:25.682584+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:55.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.027189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":471,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686599+0000","last_change":"2024-02-12T17:03:51.528465+0000","last_active":"2024-02-12T17:06:25.686599+0000","last_peered":"2024-02-12T17:06:25.686599+0000","last_clean":"2024-02-12T17:06:25.686599+0000","last_became_active":"2024-02-12T17:03:50.576714+0000","last_became_peered":"2024-02-12T17:03:50.576714+0000","last_unstale":"2024-02-12T17:06:25.686599+0000","last_undegraded":"2024-02-12T17:06:25.686599+0000","last_fullsized":"2024-02-12T17:06:25.686599+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:01:42.076584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.076584+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":115,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684902+0000","last_change":"2024-02-12T17:05:45.618872+0000","last_active":"2024-02-12T17:06:25.684902+0000","last_peered":"2024-02-12T17:06:25.684902+0000","last_clean":"2024-02-12T17:06:25.684902+0000","last_became_active":"2024-02-12T17:05:45.618551+0000","last_became_peered":"2024-02-12T17:05:45.618551+0000","last_unstale":"2024-02-12T17:06:25.684902+0000","last_undegraded":"2024-02-12T17:06:25.684902+0000","last_fullsized":"2024-02-12T17:06:25.684902+0000","mapping_epoch":436,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":133,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685752+0000","last_change":"2024-02-12T17:06:03.551775+0000","last_active":"2024-02-12T17:06:25.685752+0000","last_peered":"2024-02-12T17:06:25.685752+0000","last_clean":"2024-02-12T17:06:25.685752+0000","last_became_active":"2024-02-12T17:04:39.877005+0000","last_became_peered":"2024-02-12T17:04:39.877005+0000","last_unstale":"2024-02-12T17:06:25.685752+0000","last_undegraded":"2024-02-12T17:06:25.685752+0000","last_fullsized":"2024-02-12T17:06:25.685752+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":235,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:31.071643+0000","last_change":"2024-02-12T17:06:31.071643+0000","last_active":"2024-02-12T17:06:31.071643+0000","last_peered":"2024-02-12T17:06:31.071643+0000","last_clean":"2024-02-12T17:06:31.071643+0000","last_became_active":"2024-02-12T17:04:39.884769+0000","last_became_peered":"2024-02-12T17:04:39.884769+0000","last_unstale":"2024-02-12T17:06:31.071643+0000","last_undegraded":"2024-02-12T17:06:31.071643+0000","last_fullsized":"2024-02-12T17:06:31.071643+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:31.071549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:31.071549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":131,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684724+0000","last_change":"2024-02-12T17:03:59.035349+0000","last_active":"2024-02-12T17:06:25.684724+0000","last_peered":"2024-02-12T17:06:25.684724+0000","last_clean":"2024-02-12T17:06:25.684724+0000","last_became_active":"2024-02-12T17:03:50.639321+0000","last_became_peered":"2024-02-12T17:03:50.639321+0000","last_unstale":"2024-02-12T17:06:25.684724+0000","last_undegraded":"2024-02-12T17:06:25.684724+0000","last_fullsized":"2024-02-12T17:06:25.684724+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":457,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686016+0000","last_change":"2024-02-12T17:05:42.512197+0000","last_active":"2024-02-12T17:06:25.686016+0000","last_peered":"2024-02-12T17:06:25.686016+0000","last_clean":"2024-02-12T17:06:25.686016+0000","last_became_active":"2024-02-12T17:05:42.511859+0000","last_became_peered":"2024-02-12T17:05:42.511859+0000","last_unstale":"2024-02-12T17:06:25.686016+0000","last_undegraded":"2024-02-12T17:06:25.686016+0000","last_fullsized":"2024-02-12T17:06:25.686016+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":429,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685681+0000","last_change":"2024-02-12T17:06:03.911480+0000","last_active":"2024-02-12T17:06:25.685681+0000","last_peered":"2024-02-12T17:06:25.685681+0000","last_clean":"2024-02-12T17:06:25.685681+0000","last_became_active":"2024-02-12T17:04:39.886041+0000","last_became_peered":"2024-02-12T17:04:39.886041+0000","last_unstale":"2024-02-12T17:06:25.685681+0000","last_undegraded":"2024-02-12T17:06:25.685681+0000","last_fullsized":"2024-02-12T17:06:25.685681+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.911375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.911375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":456,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:34.981969+0000","last_change":"2024-02-12T17:06:34.981969+0000","last_active":"2024-02-12T17:06:34.981969+0000","last_peered":"2024-02-12T17:06:34.981969+0000","last_clean":"2024-02-12T17:06:34.981969+0000","last_became_active":"2024-02-12T17:03:50.550931+0000","last_became_peered":"2024-02-12T17:03:50.550931+0000","last_unstale":"2024-02-12T17:06:34.981969+0000","last_undegraded":"2024-02-12T17:06:34.981969+0000","last_fullsized":"2024-02-12T17:06:34.981969+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:34.981898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.981898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":145,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:35.413066+0000","last_change":"2024-02-12T17:06:35.413066+0000","last_active":"2024-02-12T17:06:35.413066+0000","last_peered":"2024-02-12T17:06:35.413066+0000","last_clean":"2024-02-12T17:06:35.413066+0000","last_became_active":"2024-02-12T17:04:39.865088+0000","last_became_peered":"2024-02-12T17:04:39.865088+0000","last_unstale":"2024-02-12T17:06:35.413066+0000","last_undegraded":"2024-02-12T17:06:35.413066+0000","last_fullsized":"2024-02-12T17:06:35.413066+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":244,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683146+0000","last_change":"2024-02-12T17:05:42.544935+0000","last_active":"2024-02-12T17:06:25.683146+0000","last_peered":"2024-02-12T17:06:25.683146+0000","last_clean":"2024-02-12T17:06:25.683146+0000","last_became_active":"2024-02-12T17:05:42.544611+0000","last_became_peered":"2024-02-12T17:05:42.544611+0000","last_unstale":"2024-02-12T17:06:25.683146+0000","last_undegraded":"2024-02-12T17:06:25.683146+0000","last_fullsized":"2024-02-12T17:06:25.683146+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":119,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684567+0000","last_change":"2024-02-12T17:03:50.545703+0000","last_active":"2024-02-12T17:06:25.684567+0000","last_peered":"2024-02-12T17:06:25.684567+0000","last_clean":"2024-02-12T17:06:25.684567+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:06:25.684567+0000","last_undegraded":"2024-02-12T17:06:25.684567+0000","last_fullsized":"2024-02-12T17:06:25.684567+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.051528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.051528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":285,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683195+0000","last_change":"2024-02-12T17:03:20.768326+0000","last_active":"2024-02-12T17:06:25.683195+0000","last_peered":"2024-02-12T17:06:25.683195+0000","last_clean":"2024-02-12T17:06:25.683195+0000","last_became_active":"2024-02-12T17:03:20.767943+0000","last_became_peered":"2024-02-12T17:03:20.767943+0000","last_unstale":"2024-02-12T17:06:25.683195+0000","last_undegraded":"2024-02-12T17:06:25.683195+0000","last_fullsized":"2024-02-12T17:06:25.683195+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.858411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.858411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":425,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682943+0000","last_change":"2024-02-12T17:04:40.575962+0000","last_active":"2024-02-12T17:06:25.682943+0000","last_peered":"2024-02-12T17:06:25.682943+0000","last_clean":"2024-02-12T17:06:25.682943+0000","last_became_active":"2024-02-12T17:04:39.883548+0000","last_became_peered":"2024-02-12T17:04:39.883548+0000","last_unstale":"2024-02-12T17:06:25.682943+0000","last_undegraded":"2024-02-12T17:06:25.682943+0000","last_fullsized":"2024-02-12T17:06:25.682943+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:16.064966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.064966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685130+0000","last_change":"2024-02-12T17:05:42.512845+0000","last_active":"2024-02-12T17:06:25.685130+0000","last_peered":"2024-02-12T17:06:25.685130+0000","last_clean":"2024-02-12T17:06:25.685130+0000","last_became_active":"2024-02-12T17:05:42.512394+0000","last_became_peered":"2024-02-12T17:05:42.512394+0000","last_unstale":"2024-02-12T17:06:25.685130+0000","last_undegraded":"2024-02-12T17:06:25.685130+0000","last_fullsized":"2024-02-12T17:06:25.685130+0000","mapping_epoch":433,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":367,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682742+0000","last_change":"2024-02-12T17:04:39.877393+0000","last_active":"2024-02-12T17:06:25.682742+0000","last_peered":"2024-02-12T17:06:25.682742+0000","last_clean":"2024-02-12T17:06:25.682742+0000","last_became_active":"2024-02-12T17:04:39.876910+0000","last_became_peered":"2024-02-12T17:04:39.876910+0000","last_unstale":"2024-02-12T17:06:25.682742+0000","last_undegraded":"2024-02-12T17:06:25.682742+0000","last_fullsized":"2024-02-12T17:06:25.682742+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.019813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.019813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":119,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686995+0000","last_change":"2024-02-12T17:03:50.559509+0000","last_active":"2024-02-12T17:06:25.686995+0000","last_peered":"2024-02-12T17:06:25.686995+0000","last_clean":"2024-02-12T17:06:25.686995+0000","last_became_active":"2024-02-12T17:03:50.558598+0000","last_became_peered":"2024-02-12T17:03:50.558598+0000","last_unstale":"2024-02-12T17:06:25.686995+0000","last_undegraded":"2024-02-12T17:06:25.686995+0000","last_fullsized":"2024-02-12T17:06:25.686995+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":234,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682297+0000","last_change":"2024-02-12T17:03:22.527733+0000","last_active":"2024-02-12T17:06:25.682297+0000","last_peered":"2024-02-12T17:06:25.682297+0000","last_clean":"2024-02-12T17:06:25.682297+0000","last_became_active":"2024-02-12T17:03:20.774315+0000","last_became_peered":"2024-02-12T17:03:20.774315+0000","last_unstale":"2024-02-12T17:06:25.682297+0000","last_undegraded":"2024-02-12T17:06:25.682297+0000","last_fullsized":"2024-02-12T17:06:25.682297+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:00:53.792980+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.792980+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":435,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:32.456651+0000","last_change":"2024-02-12T17:05:55.414790+0000","last_active":"2024-02-12T17:06:32.456651+0000","last_peered":"2024-02-12T17:06:32.456651+0000","last_clean":"2024-02-12T17:06:32.456651+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:06:32.456651+0000","last_undegraded":"2024-02-12T17:06:32.456651+0000","last_fullsized":"2024-02-12T17:06:32.456651+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":207,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685413+0000","last_change":"2024-02-12T17:04:39.890091+0000","last_active":"2024-02-12T17:06:25.685413+0000","last_peered":"2024-02-12T17:06:25.685413+0000","last_clean":"2024-02-12T17:06:25.685413+0000","last_became_active":"2024-02-12T17:04:39.886343+0000","last_became_peered":"2024-02-12T17:04:39.886343+0000","last_unstale":"2024-02-12T17:06:25.685413+0000","last_undegraded":"2024-02-12T17:06:25.685413+0000","last_fullsized":"2024-02-12T17:06:25.685413+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:02.953891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:02.953891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":523,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683127+0000","last_change":"2024-02-12T17:03:50.690222+0000","last_active":"2024-02-12T17:06:25.683127+0000","last_peered":"2024-02-12T17:06:25.683127+0000","last_clean":"2024-02-12T17:06:25.683127+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:06:25.683127+0000","last_undegraded":"2024-02-12T17:06:25.683127+0000","last_fullsized":"2024-02-12T17:06:25.683127+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:31.827632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:31.827632+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"438'17","reported_seq":383,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:32.455744+0000","last_change":"2024-02-12T17:05:45.604659+0000","last_active":"2024-02-12T17:06:32.455744+0000","last_peered":"2024-02-12T17:06:32.455744+0000","last_clean":"2024-02-12T17:06:32.455744+0000","last_became_active":"2024-02-12T17:05:45.604285+0000","last_became_peered":"2024-02-12T17:05:45.604285+0000","last_unstale":"2024-02-12T17:06:32.455744+0000","last_undegraded":"2024-02-12T17:06:32.455744+0000","last_fullsized":"2024-02-12T17:06:32.455744+0000","mapping_epoch":436,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":442,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682696+0000","last_change":"2024-02-12T17:03:50.577996+0000","last_active":"2024-02-12T17:06:25.682696+0000","last_peered":"2024-02-12T17:06:25.682696+0000","last_clean":"2024-02-12T17:06:25.682696+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:06:25.682696+0000","last_undegraded":"2024-02-12T17:06:25.682696+0000","last_fullsized":"2024-02-12T17:06:25.682696+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:34.874363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:34.874363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":136,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:34.381885+0000","last_change":"2024-02-12T17:06:34.381885+0000","last_active":"2024-02-12T17:06:34.381885+0000","last_peered":"2024-02-12T17:06:34.381885+0000","last_clean":"2024-02-12T17:06:34.381885+0000","last_became_active":"2024-02-12T17:03:50.556444+0000","last_became_peered":"2024-02-12T17:03:50.556444+0000","last_unstale":"2024-02-12T17:06:34.381885+0000","last_undegraded":"2024-02-12T17:06:34.381885+0000","last_fullsized":"2024-02-12T17:06:34.381885+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":437,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686067+0000","last_change":"2024-02-12T17:04:39.889382+0000","last_active":"2024-02-12T17:06:25.686067+0000","last_peered":"2024-02-12T17:06:25.686067+0000","last_clean":"2024-02-12T17:06:25.686067+0000","last_became_active":"2024-02-12T17:04:39.889055+0000","last_became_peered":"2024-02-12T17:04:39.889055+0000","last_unstale":"2024-02-12T17:06:25.686067+0000","last_undegraded":"2024-02-12T17:06:25.686067+0000","last_fullsized":"2024-02-12T17:06:25.686067+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:00:58.187738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":381,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682358+0000","last_change":"2024-02-12T17:04:39.884622+0000","last_active":"2024-02-12T17:06:25.682358+0000","last_peered":"2024-02-12T17:06:25.682358+0000","last_clean":"2024-02-12T17:06:25.682358+0000","last_became_active":"2024-02-12T17:04:39.884162+0000","last_became_peered":"2024-02-12T17:04:39.884162+0000","last_unstale":"2024-02-12T17:06:25.682358+0000","last_undegraded":"2024-02-12T17:06:25.682358+0000","last_fullsized":"2024-02-12T17:06:25.682358+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:32.127593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:32.127593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"278'12","reported_seq":127,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.689187+0000","last_change":"2024-02-12T17:03:53.944994+0000","last_active":"2024-02-12T17:06:25.689187+0000","last_peered":"2024-02-12T17:06:25.689187+0000","last_clean":"2024-02-12T17:06:25.689187+0000","last_became_active":"2024-02-12T17:03:50.555238+0000","last_became_peered":"2024-02-12T17:03:50.555238+0000","last_unstale":"2024-02-12T17:06:25.689187+0000","last_undegraded":"2024-02-12T17:06:25.689187+0000","last_fullsized":"2024-02-12T17:06:25.689187+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-02-12T17:00:53.269107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.269107+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":94,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685737+0000","last_change":"2024-02-12T17:04:39.936874+0000","last_active":"2024-02-12T17:06:25.685737+0000","last_peered":"2024-02-12T17:06:25.685737+0000","last_clean":"2024-02-12T17:06:25.685737+0000","last_became_active":"2024-02-12T17:04:39.936538+0000","last_became_peered":"2024-02-12T17:04:39.936538+0000","last_unstale":"2024-02-12T17:06:25.685737+0000","last_undegraded":"2024-02-12T17:06:25.685737+0000","last_fullsized":"2024-02-12T17:06:25.685737+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:43.982623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.982623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":350,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684847+0000","last_change":"2024-02-12T17:06:02.744602+0000","last_active":"2024-02-12T17:06:25.684847+0000","last_peered":"2024-02-12T17:06:25.684847+0000","last_clean":"2024-02-12T17:06:25.684847+0000","last_became_active":"2024-02-12T17:04:39.866732+0000","last_became_peered":"2024-02-12T17:04:39.866732+0000","last_unstale":"2024-02-12T17:06:25.684847+0000","last_undegraded":"2024-02-12T17:06:25.684847+0000","last_fullsized":"2024-02-12T17:06:25.684847+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.744493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":129,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.687084+0000","last_change":"2024-02-12T17:03:55.514799+0000","last_active":"2024-02-12T17:06:25.687084+0000","last_peered":"2024-02-12T17:06:25.687084+0000","last_clean":"2024-02-12T17:06:25.687084+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:06:25.687084+0000","last_undegraded":"2024-02-12T17:06:25.687084+0000","last_fullsized":"2024-02-12T17:06:25.687084+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:00:53.936664+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-02-12T16:51:17.050077+0000","last_clean_scrub_stamp":"2024-02-12T17:00:53.936664+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"458'1916","reported_seq":1931,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.538736+0000","last_change":"2024-02-12T17:06:18.392616+0000","last_active":"2024-02-12T17:06:38.538736+0000","last_peered":"2024-02-12T17:06:38.538736+0000","last_clean":"2024-02-12T17:06:38.538736+0000","last_became_active":"2024-02-12T17:06:18.392104+0000","last_became_peered":"2024-02-12T17:06:18.392104+0000","last_unstale":"2024-02-12T17:06:38.538736+0000","last_undegraded":"2024-02-12T17:06:38.538736+0000","last_fullsized":"2024-02-12T17:06:38.538736+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1916,"ondisk_log_size":1916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":125566976,"num_objects":1916,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1916,"num_write_kb":122624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":146,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685589+0000","last_change":"2024-02-12T17:04:39.890573+0000","last_active":"2024-02-12T17:06:25.685589+0000","last_peered":"2024-02-12T17:06:25.685589+0000","last_clean":"2024-02-12T17:06:25.685589+0000","last_became_active":"2024-02-12T17:04:39.889283+0000","last_became_peered":"2024-02-12T17:04:39.889283+0000","last_unstale":"2024-02-12T17:06:25.685589+0000","last_undegraded":"2024-02-12T17:06:25.685589+0000","last_fullsized":"2024-02-12T17:06:25.685589+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:28.912872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.912872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":156,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686846+0000","last_change":"2024-02-12T17:04:40.578504+0000","last_active":"2024-02-12T17:06:25.686846+0000","last_peered":"2024-02-12T17:06:25.686846+0000","last_clean":"2024-02-12T17:06:25.686846+0000","last_became_active":"2024-02-12T17:04:39.890440+0000","last_became_peered":"2024-02-12T17:04:39.890440+0000","last_unstale":"2024-02-12T17:06:25.686846+0000","last_undegraded":"2024-02-12T17:06:25.686846+0000","last_fullsized":"2024-02-12T17:06:25.686846+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:02:17.066331+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.066331+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":152,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684954+0000","last_change":"2024-02-12T17:04:42.701462+0000","last_active":"2024-02-12T17:06:25.684954+0000","last_peered":"2024-02-12T17:06:25.684954+0000","last_clean":"2024-02-12T17:06:25.684954+0000","last_became_active":"2024-02-12T17:04:39.891351+0000","last_became_peered":"2024-02-12T17:04:39.891351+0000","last_unstale":"2024-02-12T17:06:25.684954+0000","last_undegraded":"2024-02-12T17:06:25.684954+0000","last_fullsized":"2024-02-12T17:06:25.684954+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":352,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682751+0000","last_change":"2024-02-12T17:04:39.882820+0000","last_active":"2024-02-12T17:06:25.682751+0000","last_peered":"2024-02-12T17:06:25.682751+0000","last_clean":"2024-02-12T17:06:25.682751+0000","last_became_active":"2024-02-12T17:04:39.880493+0000","last_became_peered":"2024-02-12T17:04:39.880493+0000","last_unstale":"2024-02-12T17:06:25.682751+0000","last_undegraded":"2024-02-12T17:06:25.682751+0000","last_fullsized":"2024-02-12T17:06:25.682751+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:25.815816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.815816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"458'1412","reported_seq":1427,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:33.709643+0000","last_change":"2024-02-12T17:06:18.392774+0000","last_active":"2024-02-12T17:06:33.709643+0000","last_peered":"2024-02-12T17:06:33.709643+0000","last_clean":"2024-02-12T17:06:33.709643+0000","last_became_active":"2024-02-12T17:06:18.392438+0000","last_became_peered":"2024-02-12T17:06:18.392438+0000","last_unstale":"2024-02-12T17:06:33.709643+0000","last_undegraded":"2024-02-12T17:06:33.709643+0000","last_fullsized":"2024-02-12T17:06:33.709643+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1412,"ondisk_log_size":1412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92536832,"num_objects":1412,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1412,"num_write_kb":90368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":439,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684602+0000","last_change":"2024-02-12T17:05:56.852203+0000","last_active":"2024-02-12T17:06:25.684602+0000","last_peered":"2024-02-12T17:06:25.684602+0000","last_clean":"2024-02-12T17:06:25.684602+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:06:25.684602+0000","last_undegraded":"2024-02-12T17:06:25.684602+0000","last_fullsized":"2024-02-12T17:06:25.684602+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:56.852121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.852121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":272,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685134+0000","last_change":"2024-02-12T17:06:23.673240+0000","last_active":"2024-02-12T17:06:25.685134+0000","last_peered":"2024-02-12T17:06:25.685134+0000","last_clean":"2024-02-12T17:06:25.685134+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:06:25.685134+0000","last_undegraded":"2024-02-12T17:06:25.685134+0000","last_fullsized":"2024-02-12T17:06:25.685134+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:23.673128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:23.673128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":363,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683240+0000","last_change":"2024-02-12T17:06:04.040637+0000","last_active":"2024-02-12T17:06:25.683240+0000","last_peered":"2024-02-12T17:06:25.683240+0000","last_clean":"2024-02-12T17:06:25.683240+0000","last_became_active":"2024-02-12T17:04:39.883521+0000","last_became_peered":"2024-02-12T17:04:39.883521+0000","last_unstale":"2024-02-12T17:06:25.683240+0000","last_undegraded":"2024-02-12T17:06:25.683240+0000","last_fullsized":"2024-02-12T17:06:25.683240+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:04.040555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:04.040555+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"433'15","reported_seq":626,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:32.456407+0000","last_change":"2024-02-12T17:05:40.422839+0000","last_active":"2024-02-12T17:06:32.456407+0000","last_peered":"2024-02-12T17:06:32.456407+0000","last_clean":"2024-02-12T17:06:32.456407+0000","last_became_active":"2024-02-12T17:05:40.422494+0000","last_became_peered":"2024-02-12T17:05:40.422494+0000","last_unstale":"2024-02-12T17:06:32.456407+0000","last_undegraded":"2024-02-12T17:06:32.456407+0000","last_fullsized":"2024-02-12T17:06:32.456407+0000","mapping_epoch":347,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"458'1960","reported_seq":1975,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.316657+0000","last_change":"2024-02-12T17:06:18.391956+0000","last_active":"2024-02-12T17:06:38.316657+0000","last_peered":"2024-02-12T17:06:38.316657+0000","last_clean":"2024-02-12T17:06:38.316657+0000","last_became_active":"2024-02-12T17:06:18.391610+0000","last_became_peered":"2024-02-12T17:06:18.391610+0000","last_unstale":"2024-02-12T17:06:38.316657+0000","last_undegraded":"2024-02-12T17:06:38.316657+0000","last_fullsized":"2024-02-12T17:06:38.316657+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1960,"ondisk_log_size":1960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128450560,"num_objects":1960,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1960,"num_write_kb":125440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":146,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686740+0000","last_change":"2024-02-12T17:04:58.474896+0000","last_active":"2024-02-12T17:06:25.686740+0000","last_peered":"2024-02-12T17:06:25.686740+0000","last_clean":"2024-02-12T17:06:25.686740+0000","last_became_active":"2024-02-12T17:04:58.474504+0000","last_became_peered":"2024-02-12T17:04:58.474504+0000","last_unstale":"2024-02-12T17:06:25.686740+0000","last_undegraded":"2024-02-12T17:06:25.686740+0000","last_fullsized":"2024-02-12T17:06:25.686740+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":456,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682883+0000","last_change":"2024-02-12T17:03:50.549240+0000","last_active":"2024-02-12T17:06:25.682883+0000","last_peered":"2024-02-12T17:06:25.682883+0000","last_clean":"2024-02-12T17:06:25.682883+0000","last_became_active":"2024-02-12T17:03:50.548926+0000","last_became_peered":"2024-02-12T17:03:50.548926+0000","last_unstale":"2024-02-12T17:06:25.682883+0000","last_undegraded":"2024-02-12T17:06:25.682883+0000","last_fullsized":"2024-02-12T17:06:25.682883+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:42.892563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.892563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"348'55","reported_seq":176,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:32.456826+0000","last_change":"2024-02-12T17:03:58.596193+0000","last_active":"2024-02-12T17:06:32.456826+0000","last_peered":"2024-02-12T17:06:32.456826+0000","last_clean":"2024-02-12T17:06:32.456826+0000","last_became_active":"2024-02-12T17:03:50.557154+0000","last_became_peered":"2024-02-12T17:03:50.557154+0000","last_unstale":"2024-02-12T17:06:32.456826+0000","last_undegraded":"2024-02-12T17:06:32.456826+0000","last_fullsized":"2024-02-12T17:06:32.456826+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":406,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686460+0000","last_change":"2024-02-12T17:05:40.431590+0000","last_active":"2024-02-12T17:06:25.686460+0000","last_peered":"2024-02-12T17:06:25.686460+0000","last_clean":"2024-02-12T17:06:25.686460+0000","last_became_active":"2024-02-12T17:05:40.431265+0000","last_became_peered":"2024-02-12T17:05:40.431265+0000","last_unstale":"2024-02-12T17:06:25.686460+0000","last_undegraded":"2024-02-12T17:06:25.686460+0000","last_fullsized":"2024-02-12T17:06:25.686460+0000","mapping_epoch":377,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"458'2000","reported_seq":2015,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.331979+0000","last_change":"2024-02-12T17:06:18.392794+0000","last_active":"2024-02-12T17:06:38.331979+0000","last_peered":"2024-02-12T17:06:38.331979+0000","last_clean":"2024-02-12T17:06:38.331979+0000","last_became_active":"2024-02-12T17:06:18.392475+0000","last_became_peered":"2024-02-12T17:06:18.392475+0000","last_unstale":"2024-02-12T17:06:38.331979+0000","last_undegraded":"2024-02-12T17:06:38.331979+0000","last_fullsized":"2024-02-12T17:06:38.331979+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":2000,"ondisk_log_size":2000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":131072000,"num_objects":2000,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2000,"num_write_kb":128000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":444,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686319+0000","last_change":"2024-02-12T17:03:50.566072+0000","last_active":"2024-02-12T17:06:25.686319+0000","last_peered":"2024-02-12T17:06:25.686319+0000","last_clean":"2024-02-12T17:06:25.686319+0000","last_became_active":"2024-02-12T17:03:50.564700+0000","last_became_peered":"2024-02-12T17:03:50.564700+0000","last_unstale":"2024-02-12T17:06:25.686319+0000","last_undegraded":"2024-02-12T17:06:25.686319+0000","last_fullsized":"2024-02-12T17:06:25.686319+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:04.085827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:04.085827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":509,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:32.267615+0000","last_change":"2024-02-12T17:05:59.784735+0000","last_active":"2024-02-12T17:06:32.267615+0000","last_peered":"2024-02-12T17:06:32.267615+0000","last_clean":"2024-02-12T17:06:32.267615+0000","last_became_active":"2024-02-12T17:01:16.912041+0000","last_became_peered":"2024-02-12T17:01:16.912041+0000","last_unstale":"2024-02-12T17:06:32.267615+0000","last_undegraded":"2024-02-12T17:06:32.267615+0000","last_fullsized":"2024-02-12T17:06:32.267615+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:05:59.784636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:59.784636+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":322,"num_read_kb":317,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":119,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684809+0000","last_change":"2024-02-12T17:03:50.533363+0000","last_active":"2024-02-12T17:06:25.684809+0000","last_peered":"2024-02-12T17:06:25.684809+0000","last_clean":"2024-02-12T17:06:25.684809+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:06:25.684809+0000","last_undegraded":"2024-02-12T17:06:25.684809+0000","last_fullsized":"2024-02-12T17:06:25.684809+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:38.142602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:38.142602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":390,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684066+0000","last_change":"2024-02-12T17:03:51.175255+0000","last_active":"2024-02-12T17:06:25.684066+0000","last_peered":"2024-02-12T17:06:25.684066+0000","last_clean":"2024-02-12T17:06:25.684066+0000","last_became_active":"2024-02-12T17:03:50.573814+0000","last_became_peered":"2024-02-12T17:03:50.573814+0000","last_unstale":"2024-02-12T17:06:25.684066+0000","last_undegraded":"2024-02-12T17:06:25.684066+0000","last_fullsized":"2024-02-12T17:06:25.684066+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":377,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683921+0000","last_change":"2024-02-12T17:06:25.129256+0000","last_active":"2024-02-12T17:06:25.683921+0000","last_peered":"2024-02-12T17:06:25.683921+0000","last_clean":"2024-02-12T17:06:25.683921+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:06:25.683921+0000","last_undegraded":"2024-02-12T17:06:25.683921+0000","last_fullsized":"2024-02-12T17:06:25.683921+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:25.129190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:25.129190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"458'1414","reported_seq":1429,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:33.682578+0000","last_change":"2024-02-12T17:06:18.392727+0000","last_active":"2024-02-12T17:06:33.682578+0000","last_peered":"2024-02-12T17:06:33.682578+0000","last_clean":"2024-02-12T17:06:33.682578+0000","last_became_active":"2024-02-12T17:06:18.392427+0000","last_became_peered":"2024-02-12T17:06:18.392427+0000","last_unstale":"2024-02-12T17:06:33.682578+0000","last_undegraded":"2024-02-12T17:06:33.682578+0000","last_fullsized":"2024-02-12T17:06:33.682578+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1414,"ondisk_log_size":1414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92667904,"num_objects":1414,"num_object_clones":0,"num_object_copies":4242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1414,"num_write_kb":90496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":472,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684187+0000","last_change":"2024-02-12T17:04:54.415608+0000","last_active":"2024-02-12T17:06:25.684187+0000","last_peered":"2024-02-12T17:06:25.684187+0000","last_clean":"2024-02-12T17:06:25.684187+0000","last_became_active":"2024-02-12T17:04:54.415431+0000","last_became_peered":"2024-02-12T17:04:54.415431+0000","last_unstale":"2024-02-12T17:06:25.684187+0000","last_undegraded":"2024-02-12T17:06:25.684187+0000","last_fullsized":"2024-02-12T17:06:25.684187+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:57.950284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:00:57.950284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":133,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684294+0000","last_change":"2024-02-12T17:05:54.311879+0000","last_active":"2024-02-12T17:06:25.684294+0000","last_peered":"2024-02-12T17:06:25.684294+0000","last_clean":"2024-02-12T17:06:25.684294+0000","last_became_active":"2024-02-12T17:04:39.885173+0000","last_became_peered":"2024-02-12T17:04:39.885173+0000","last_unstale":"2024-02-12T17:06:25.684294+0000","last_undegraded":"2024-02-12T17:06:25.684294+0000","last_fullsized":"2024-02-12T17:06:25.684294+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:54.311805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:05:54.311805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":429,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682804+0000","last_change":"2024-02-12T17:05:56.054036+0000","last_active":"2024-02-12T17:06:25.682804+0000","last_peered":"2024-02-12T17:06:25.682804+0000","last_clean":"2024-02-12T17:06:25.682804+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:06:25.682804+0000","last_undegraded":"2024-02-12T17:06:25.682804+0000","last_fullsized":"2024-02-12T17:06:25.682804+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:56.053971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.053971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":456,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684769+0000","last_change":"2024-02-12T17:04:39.915666+0000","last_active":"2024-02-12T17:06:25.684769+0000","last_peered":"2024-02-12T17:06:25.684769+0000","last_clean":"2024-02-12T17:06:25.684769+0000","last_became_active":"2024-02-12T17:04:39.915392+0000","last_became_peered":"2024-02-12T17:04:39.915392+0000","last_unstale":"2024-02-12T17:06:25.684769+0000","last_undegraded":"2024-02-12T17:06:25.684769+0000","last_fullsized":"2024-02-12T17:06:25.684769+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:25.870916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:01:25.870916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"278'26","reported_seq":205,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686517+0000","last_change":"2024-02-12T17:04:40.745958+0000","last_active":"2024-02-12T17:06:25.686517+0000","last_peered":"2024-02-12T17:06:25.686517+0000","last_clean":"2024-02-12T17:06:25.686517+0000","last_became_active":"2024-02-12T17:04:39.889039+0000","last_became_peered":"2024-02-12T17:04:39.889039+0000","last_unstale":"2024-02-12T17:06:25.686517+0000","last_undegraded":"2024-02-12T17:06:25.686517+0000","last_fullsized":"2024-02-12T17:06:25.686517+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'26","last_scrub_stamp":"2024-02-12T17:01:43.163724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:43.163724+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":454,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685856+0000","last_change":"2024-02-12T17:05:40.432925+0000","last_active":"2024-02-12T17:06:25.685856+0000","last_peered":"2024-02-12T17:06:25.685856+0000","last_clean":"2024-02-12T17:06:25.685856+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:06:25.685856+0000","last_undegraded":"2024-02-12T17:06:25.685856+0000","last_fullsized":"2024-02-12T17:06:25.685856+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"458'1345","reported_seq":1360,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:33.699597+0000","last_change":"2024-02-12T17:06:18.394208+0000","last_active":"2024-02-12T17:06:33.699597+0000","last_peered":"2024-02-12T17:06:33.699597+0000","last_clean":"2024-02-12T17:06:33.699597+0000","last_became_active":"2024-02-12T17:06:18.393896+0000","last_became_peered":"2024-02-12T17:06:18.393896+0000","last_unstale":"2024-02-12T17:06:33.699597+0000","last_undegraded":"2024-02-12T17:06:33.699597+0000","last_fullsized":"2024-02-12T17:06:33.699597+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1345,"ondisk_log_size":1345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88145920,"num_objects":1345,"num_object_clones":0,"num_object_copies":4035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1345,"num_write_kb":86080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":387,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682610+0000","last_change":"2024-02-12T17:06:02.044976+0000","last_active":"2024-02-12T17:06:25.682610+0000","last_peered":"2024-02-12T17:06:25.682610+0000","last_clean":"2024-02-12T17:06:25.682610+0000","last_became_active":"2024-02-12T17:04:39.870979+0000","last_became_peered":"2024-02-12T17:04:39.870979+0000","last_unstale":"2024-02-12T17:06:25.682610+0000","last_undegraded":"2024-02-12T17:06:25.682610+0000","last_fullsized":"2024-02-12T17:06:25.682610+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":207,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684184+0000","last_change":"2024-02-12T17:04:39.892450+0000","last_active":"2024-02-12T17:06:25.684184+0000","last_peered":"2024-02-12T17:06:25.684184+0000","last_clean":"2024-02-12T17:06:25.684184+0000","last_became_active":"2024-02-12T17:04:39.892122+0000","last_became_peered":"2024-02-12T17:04:39.892122+0000","last_unstale":"2024-02-12T17:06:25.684184+0000","last_undegraded":"2024-02-12T17:06:25.684184+0000","last_fullsized":"2024-02-12T17:06:25.684184+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.941776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":459,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684767+0000","last_change":"2024-02-12T17:06:01.905985+0000","last_active":"2024-02-12T17:06:25.684767+0000","last_peered":"2024-02-12T17:06:25.684767+0000","last_clean":"2024-02-12T17:06:25.684767+0000","last_became_active":"2024-02-12T17:03:50.564881+0000","last_became_peered":"2024-02-12T17:03:50.564881+0000","last_unstale":"2024-02-12T17:06:25.684767+0000","last_undegraded":"2024-02-12T17:06:25.684767+0000","last_fullsized":"2024-02-12T17:06:25.684767+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":444,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685870+0000","last_change":"2024-02-12T17:05:55.847994+0000","last_active":"2024-02-12T17:06:25.685870+0000","last_peered":"2024-02-12T17:06:25.685870+0000","last_clean":"2024-02-12T17:06:25.685870+0000","last_became_active":"2024-02-12T17:03:20.780926+0000","last_became_peered":"2024-02-12T17:03:20.780926+0000","last_unstale":"2024-02-12T17:06:25.685870+0000","last_undegraded":"2024-02-12T17:06:25.685870+0000","last_fullsized":"2024-02-12T17:06:25.685870+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":456,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684173+0000","last_change":"2024-02-12T17:04:39.885796+0000","last_active":"2024-02-12T17:06:25.684173+0000","last_peered":"2024-02-12T17:06:25.684173+0000","last_clean":"2024-02-12T17:06:25.684173+0000","last_became_active":"2024-02-12T17:04:39.885162+0000","last_became_peered":"2024-02-12T17:04:39.885162+0000","last_unstale":"2024-02-12T17:06:25.684173+0000","last_undegraded":"2024-02-12T17:06:25.684173+0000","last_fullsized":"2024-02-12T17:06:25.684173+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"458'1946","reported_seq":1961,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.541738+0000","last_change":"2024-02-12T17:06:18.394587+0000","last_active":"2024-02-12T17:06:38.541738+0000","last_peered":"2024-02-12T17:06:38.541738+0000","last_clean":"2024-02-12T17:06:38.541738+0000","last_became_active":"2024-02-12T17:06:18.394337+0000","last_became_peered":"2024-02-12T17:06:18.394337+0000","last_unstale":"2024-02-12T17:06:38.541738+0000","last_undegraded":"2024-02-12T17:06:38.541738+0000","last_fullsized":"2024-02-12T17:06:38.541738+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1946,"ondisk_log_size":1946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":127533056,"num_objects":1946,"num_object_clones":0,"num_object_copies":5838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1946,"num_write_kb":124544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":183,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686534+0000","last_change":"2024-02-12T17:03:20.779625+0000","last_active":"2024-02-12T17:06:25.686534+0000","last_peered":"2024-02-12T17:06:25.686534+0000","last_clean":"2024-02-12T17:06:25.686534+0000","last_became_active":"2024-02-12T17:03:20.779126+0000","last_became_peered":"2024-02-12T17:03:20.779126+0000","last_unstale":"2024-02-12T17:06:25.686534+0000","last_undegraded":"2024-02-12T17:06:25.686534+0000","last_fullsized":"2024-02-12T17:06:25.686534+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:48.617913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:02:48.617913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":444,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686473+0000","last_change":"2024-02-12T17:04:39.886695+0000","last_active":"2024-02-12T17:06:25.686473+0000","last_peered":"2024-02-12T17:06:25.686473+0000","last_clean":"2024-02-12T17:06:25.686473+0000","last_became_active":"2024-02-12T17:04:39.886102+0000","last_became_peered":"2024-02-12T17:04:39.886102+0000","last_unstale":"2024-02-12T17:06:25.686473+0000","last_undegraded":"2024-02-12T17:06:25.686473+0000","last_fullsized":"2024-02-12T17:06:25.686473+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.212949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.212949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":436,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683085+0000","last_change":"2024-02-12T17:03:21.658125+0000","last_active":"2024-02-12T17:06:25.683085+0000","last_peered":"2024-02-12T17:06:25.683085+0000","last_clean":"2024-02-12T17:06:25.683085+0000","last_became_active":"2024-02-12T17:03:20.769223+0000","last_became_peered":"2024-02-12T17:03:20.769223+0000","last_unstale":"2024-02-12T17:06:25.683085+0000","last_undegraded":"2024-02-12T17:06:25.683085+0000","last_fullsized":"2024-02-12T17:06:25.683085+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:02:44.688402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:44.688402+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":249,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684993+0000","last_change":"2024-02-12T17:06:25.676817+0000","last_active":"2024-02-12T17:06:25.684993+0000","last_peered":"2024-02-12T17:06:25.684993+0000","last_clean":"2024-02-12T17:06:25.684993+0000","last_became_active":"2024-02-12T17:05:40.448219+0000","last_became_peered":"2024-02-12T17:05:40.448219+0000","last_unstale":"2024-02-12T17:06:25.684993+0000","last_undegraded":"2024-02-12T17:06:25.684993+0000","last_fullsized":"2024-02-12T17:06:25.684993+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:25.676720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:25.676720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"458'2022","reported_seq":2037,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.305085+0000","last_change":"2024-02-12T17:06:18.395287+0000","last_active":"2024-02-12T17:06:38.305085+0000","last_peered":"2024-02-12T17:06:38.305085+0000","last_clean":"2024-02-12T17:06:38.305085+0000","last_became_active":"2024-02-12T17:06:18.395022+0000","last_became_peered":"2024-02-12T17:06:18.395022+0000","last_unstale":"2024-02-12T17:06:38.305085+0000","last_undegraded":"2024-02-12T17:06:38.305085+0000","last_fullsized":"2024-02-12T17:06:38.305085+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":2022,"ondisk_log_size":2022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":132513792,"num_objects":2022,"num_object_clones":0,"num_object_copies":6066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":129408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":133,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684344+0000","last_change":"2024-02-12T17:06:01.456051+0000","last_active":"2024-02-12T17:06:25.684344+0000","last_peered":"2024-02-12T17:06:25.684344+0000","last_clean":"2024-02-12T17:06:25.684344+0000","last_became_active":"2024-02-12T17:04:39.885677+0000","last_became_peered":"2024-02-12T17:04:39.885677+0000","last_unstale":"2024-02-12T17:06:25.684344+0000","last_undegraded":"2024-02-12T17:06:25.684344+0000","last_fullsized":"2024-02-12T17:06:25.684344+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:01.455969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.455969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":215,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686587+0000","last_change":"2024-02-12T17:04:39.864989+0000","last_active":"2024-02-12T17:06:25.686587+0000","last_peered":"2024-02-12T17:06:25.686587+0000","last_clean":"2024-02-12T17:06:25.686587+0000","last_became_active":"2024-02-12T17:04:39.864631+0000","last_became_peered":"2024-02-12T17:04:39.864631+0000","last_unstale":"2024-02-12T17:06:25.686587+0000","last_undegraded":"2024-02-12T17:06:25.686587+0000","last_fullsized":"2024-02-12T17:06:25.686587+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":137,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684125+0000","last_change":"2024-02-12T17:03:54.620306+0000","last_active":"2024-02-12T17:06:25.684125+0000","last_peered":"2024-02-12T17:06:25.684125+0000","last_clean":"2024-02-12T17:06:25.684125+0000","last_became_active":"2024-02-12T17:03:50.556763+0000","last_became_peered":"2024-02-12T17:03:50.556763+0000","last_unstale":"2024-02-12T17:06:25.684125+0000","last_undegraded":"2024-02-12T17:06:25.684125+0000","last_fullsized":"2024-02-12T17:06:25.684125+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":457,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684647+0000","last_change":"2024-02-12T17:05:40.428597+0000","last_active":"2024-02-12T17:06:25.684647+0000","last_peered":"2024-02-12T17:06:25.684647+0000","last_clean":"2024-02-12T17:06:25.684647+0000","last_became_active":"2024-02-12T17:05:40.427915+0000","last_became_peered":"2024-02-12T17:05:40.427915+0000","last_unstale":"2024-02-12T17:06:25.684647+0000","last_undegraded":"2024-02-12T17:06:25.684647+0000","last_fullsized":"2024-02-12T17:06:25.684647+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"458'2004","reported_seq":2019,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.528975+0000","last_change":"2024-02-12T17:06:18.392414+0000","last_active":"2024-02-12T17:06:38.528975+0000","last_peered":"2024-02-12T17:06:38.528975+0000","last_clean":"2024-02-12T17:06:38.528975+0000","last_became_active":"2024-02-12T17:06:18.391649+0000","last_became_peered":"2024-02-12T17:06:18.391649+0000","last_unstale":"2024-02-12T17:06:38.528975+0000","last_undegraded":"2024-02-12T17:06:38.528975+0000","last_fullsized":"2024-02-12T17:06:38.528975+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":2004,"ondisk_log_size":2004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":131334144,"num_objects":2004,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":128256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":460,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686094+0000","last_change":"2024-02-12T17:04:40.405970+0000","last_active":"2024-02-12T17:06:25.686094+0000","last_peered":"2024-02-12T17:06:25.686094+0000","last_clean":"2024-02-12T17:06:25.686094+0000","last_became_active":"2024-02-12T17:04:39.889077+0000","last_became_peered":"2024-02-12T17:04:39.889077+0000","last_unstale":"2024-02-12T17:06:25.686094+0000","last_undegraded":"2024-02-12T17:06:25.686094+0000","last_fullsized":"2024-02-12T17:06:25.686094+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:03:17.811656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.811656+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686924+0000","last_change":"2024-02-12T17:03:50.563493+0000","last_active":"2024-02-12T17:06:25.686924+0000","last_peered":"2024-02-12T17:06:25.686924+0000","last_clean":"2024-02-12T17:06:25.686924+0000","last_became_active":"2024-02-12T17:03:50.562556+0000","last_became_peered":"2024-02-12T17:03:50.562556+0000","last_unstale":"2024-02-12T17:06:25.686924+0000","last_undegraded":"2024-02-12T17:06:25.686924+0000","last_fullsized":"2024-02-12T17:06:25.686924+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.562425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.562425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":516,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:28.073224+0000","last_change":"2024-02-12T17:06:28.073224+0000","last_active":"2024-02-12T17:06:28.073224+0000","last_peered":"2024-02-12T17:06:28.073224+0000","last_clean":"2024-02-12T17:06:28.073224+0000","last_became_active":"2024-02-12T17:04:39.884095+0000","last_became_peered":"2024-02-12T17:04:39.884095+0000","last_unstale":"2024-02-12T17:06:28.073224+0000","last_undegraded":"2024-02-12T17:06:28.073224+0000","last_fullsized":"2024-02-12T17:06:28.073224+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'28","last_scrub_stamp":"2024-02-12T17:06:28.073131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:28.073131+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"433'19","reported_seq":202,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:32.456559+0000","last_change":"2024-02-12T17:05:40.451102+0000","last_active":"2024-02-12T17:06:32.456559+0000","last_peered":"2024-02-12T17:06:32.456559+0000","last_clean":"2024-02-12T17:06:32.456559+0000","last_became_active":"2024-02-12T17:05:40.450299+0000","last_became_peered":"2024-02-12T17:05:40.450299+0000","last_unstale":"2024-02-12T17:06:32.456559+0000","last_undegraded":"2024-02-12T17:06:32.456559+0000","last_fullsized":"2024-02-12T17:06:32.456559+0000","mapping_epoch":377,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"458'1416","reported_seq":1431,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:33.711961+0000","last_change":"2024-02-12T17:06:18.394104+0000","last_active":"2024-02-12T17:06:33.711961+0000","last_peered":"2024-02-12T17:06:33.711961+0000","last_clean":"2024-02-12T17:06:33.711961+0000","last_became_active":"2024-02-12T17:06:18.393754+0000","last_became_peered":"2024-02-12T17:06:18.393754+0000","last_unstale":"2024-02-12T17:06:33.711961+0000","last_undegraded":"2024-02-12T17:06:33.711961+0000","last_fullsized":"2024-02-12T17:06:33.711961+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1416,"ondisk_log_size":1416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92798976,"num_objects":1416,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1416,"num_write_kb":90624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":137,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686649+0000","last_change":"2024-02-12T17:04:33.794769+0000","last_active":"2024-02-12T17:06:25.686649+0000","last_peered":"2024-02-12T17:06:25.686649+0000","last_clean":"2024-02-12T17:06:25.686649+0000","last_became_active":"2024-02-12T17:04:33.794457+0000","last_became_peered":"2024-02-12T17:04:33.794457+0000","last_unstale":"2024-02-12T17:06:25.686649+0000","last_undegraded":"2024-02-12T17:06:25.686649+0000","last_fullsized":"2024-02-12T17:06:25.686649+0000","mapping_epoch":347,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.811153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.811153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"443'21","reported_seq":645,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:32.977774+0000","last_change":"2024-02-12T17:06:32.977737+0000","last_active":"2024-02-12T17:06:32.977774+0000","last_peered":"2024-02-12T17:06:32.977774+0000","last_clean":"2024-02-12T17:06:32.977774+0000","last_became_active":"2024-02-12T17:05:55.374445+0000","last_became_peered":"2024-02-12T17:05:55.374445+0000","last_unstale":"2024-02-12T17:06:32.977774+0000","last_undegraded":"2024-02-12T17:06:32.977774+0000","last_fullsized":"2024-02-12T17:06:32.977774+0000","mapping_epoch":441,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'21","last_scrub_stamp":"2024-02-12T17:06:32.977637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:06:32.977637+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":94,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684586+0000","last_change":"2024-02-12T17:06:24.976140+0000","last_active":"2024-02-12T17:06:25.684586+0000","last_peered":"2024-02-12T17:06:25.684586+0000","last_clean":"2024-02-12T17:06:25.684586+0000","last_became_active":"2024-02-12T17:04:39.889588+0000","last_became_peered":"2024-02-12T17:04:39.889588+0000","last_unstale":"2024-02-12T17:06:25.684586+0000","last_undegraded":"2024-02-12T17:06:25.684586+0000","last_fullsized":"2024-02-12T17:06:25.684586+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:24.976048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:06:24.976048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":157,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686836+0000","last_change":"2024-02-12T17:05:55.345164+0000","last_active":"2024-02-12T17:06:25.686836+0000","last_peered":"2024-02-12T17:06:25.686836+0000","last_clean":"2024-02-12T17:06:25.686836+0000","last_became_active":"2024-02-12T17:04:39.888473+0000","last_became_peered":"2024-02-12T17:04:39.888473+0000","last_unstale":"2024-02-12T17:06:25.686836+0000","last_undegraded":"2024-02-12T17:06:25.686836+0000","last_fullsized":"2024-02-12T17:06:25.686836+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:05:55.344999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.344999+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"458'1459","reported_seq":1474,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:33.719455+0000","last_change":"2024-02-12T17:06:18.392670+0000","last_active":"2024-02-12T17:06:33.719455+0000","last_peered":"2024-02-12T17:06:33.719455+0000","last_clean":"2024-02-12T17:06:33.719455+0000","last_became_active":"2024-02-12T17:06:18.392421+0000","last_became_peered":"2024-02-12T17:06:18.392421+0000","last_unstale":"2024-02-12T17:06:33.719455+0000","last_undegraded":"2024-02-12T17:06:33.719455+0000","last_fullsized":"2024-02-12T17:06:33.719455+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1459,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":95617024,"num_objects":1459,"num_object_clones":0,"num_object_copies":4377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1459,"num_write_kb":93376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":468,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683388+0000","last_change":"2024-02-12T17:04:39.887836+0000","last_active":"2024-02-12T17:06:25.683388+0000","last_peered":"2024-02-12T17:06:25.683388+0000","last_clean":"2024-02-12T17:06:25.683388+0000","last_became_active":"2024-02-12T17:04:39.887488+0000","last_became_peered":"2024-02-12T17:04:39.887488+0000","last_unstale":"2024-02-12T17:06:25.683388+0000","last_undegraded":"2024-02-12T17:06:25.683388+0000","last_fullsized":"2024-02-12T17:06:25.683388+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"435'21","reported_seq":705,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:32.455796+0000","last_change":"2024-02-12T17:05:43.541152+0000","last_active":"2024-02-12T17:06:32.455796+0000","last_peered":"2024-02-12T17:06:32.455796+0000","last_clean":"2024-02-12T17:06:32.455796+0000","last_became_active":"2024-02-12T17:05:43.540504+0000","last_became_peered":"2024-02-12T17:05:43.540504+0000","last_unstale":"2024-02-12T17:06:32.455796+0000","last_undegraded":"2024-02-12T17:06:32.455796+0000","last_fullsized":"2024-02-12T17:06:32.455796+0000","mapping_epoch":434,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-12T17:02:20.866591+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:02:20.866591+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":89,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686384+0000","last_change":"2024-02-12T17:04:39.886675+0000","last_active":"2024-02-12T17:06:25.686384+0000","last_peered":"2024-02-12T17:06:25.686384+0000","last_clean":"2024-02-12T17:06:25.686384+0000","last_became_active":"2024-02-12T17:04:39.886222+0000","last_became_peered":"2024-02-12T17:04:39.886222+0000","last_unstale":"2024-02-12T17:06:25.686384+0000","last_undegraded":"2024-02-12T17:06:25.686384+0000","last_fullsized":"2024-02-12T17:06:25.686384+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.863530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.863530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.688733+0000","last_change":"2024-02-12T17:04:43.226687+0000","last_active":"2024-02-12T17:06:25.688733+0000","last_peered":"2024-02-12T17:06:25.688733+0000","last_clean":"2024-02-12T17:06:25.688733+0000","last_became_active":"2024-02-12T17:04:39.892266+0000","last_became_peered":"2024-02-12T17:04:39.892266+0000","last_unstale":"2024-02-12T17:06:25.688733+0000","last_undegraded":"2024-02-12T17:06:25.688733+0000","last_fullsized":"2024-02-12T17:06:25.688733+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:01:41.694875+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:01:41.694875+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"458'1959","reported_seq":1974,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.525892+0000","last_change":"2024-02-12T17:06:18.366111+0000","last_active":"2024-02-12T17:06:38.525892+0000","last_peered":"2024-02-12T17:06:38.525892+0000","last_clean":"2024-02-12T17:06:38.525892+0000","last_became_active":"2024-02-12T17:06:18.365846+0000","last_became_peered":"2024-02-12T17:06:18.365846+0000","last_unstale":"2024-02-12T17:06:38.525892+0000","last_undegraded":"2024-02-12T17:06:38.525892+0000","last_fullsized":"2024-02-12T17:06:38.525892+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1959,"ondisk_log_size":1959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128385024,"num_objects":1959,"num_object_clones":0,"num_object_copies":5877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":200,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683497+0000","last_change":"2024-02-12T17:05:57.078039+0000","last_active":"2024-02-12T17:06:25.683497+0000","last_peered":"2024-02-12T17:06:25.683497+0000","last_clean":"2024-02-12T17:06:25.683497+0000","last_became_active":"2024-02-12T17:04:39.884295+0000","last_became_peered":"2024-02-12T17:04:39.884295+0000","last_unstale":"2024-02-12T17:06:25.683497+0000","last_undegraded":"2024-02-12T17:06:25.683497+0000","last_fullsized":"2024-02-12T17:06:25.683497+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.077971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":268,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:27.454599+0000","last_change":"2024-02-12T17:05:40.431666+0000","last_active":"2024-02-12T17:06:27.454599+0000","last_peered":"2024-02-12T17:06:27.454599+0000","last_clean":"2024-02-12T17:06:27.454599+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:06:27.454599+0000","last_undegraded":"2024-02-12T17:06:27.454599+0000","last_fullsized":"2024-02-12T17:06:27.454599+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'11","last_scrub_stamp":"2024-02-12T17:02:31.546965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:31.546965+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":119,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684529+0000","last_change":"2024-02-12T17:03:50.589204+0000","last_active":"2024-02-12T17:06:25.684529+0000","last_peered":"2024-02-12T17:06:25.684529+0000","last_clean":"2024-02-12T17:06:25.684529+0000","last_became_active":"2024-02-12T17:03:50.588844+0000","last_became_peered":"2024-02-12T17:03:50.588844+0000","last_unstale":"2024-02-12T17:06:25.684529+0000","last_undegraded":"2024-02-12T17:06:25.684529+0000","last_fullsized":"2024-02-12T17:06:25.684529+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.888942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.888942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":152,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686746+0000","last_change":"2024-02-12T17:04:42.611814+0000","last_active":"2024-02-12T17:06:25.686746+0000","last_peered":"2024-02-12T17:06:25.686746+0000","last_clean":"2024-02-12T17:06:25.686746+0000","last_became_active":"2024-02-12T17:04:39.891435+0000","last_became_peered":"2024-02-12T17:04:39.891435+0000","last_unstale":"2024-02-12T17:06:25.686746+0000","last_undegraded":"2024-02-12T17:06:25.686746+0000","last_fullsized":"2024-02-12T17:06:25.686746+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"458'1977","reported_seq":1992,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.336827+0000","last_change":"2024-02-12T17:06:18.392042+0000","last_active":"2024-02-12T17:06:38.336827+0000","last_peered":"2024-02-12T17:06:38.336827+0000","last_clean":"2024-02-12T17:06:38.336827+0000","last_became_active":"2024-02-12T17:06:18.391725+0000","last_became_peered":"2024-02-12T17:06:18.391725+0000","last_unstale":"2024-02-12T17:06:38.336827+0000","last_undegraded":"2024-02-12T17:06:38.336827+0000","last_fullsized":"2024-02-12T17:06:38.336827+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1977,"ondisk_log_size":1977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":5931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":126528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":151,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:26.583756+0000","last_change":"2024-02-12T17:06:26.583756+0000","last_active":"2024-02-12T17:06:26.583756+0000","last_peered":"2024-02-12T17:06:26.583756+0000","last_clean":"2024-02-12T17:06:26.583756+0000","last_became_active":"2024-02-12T17:04:39.874748+0000","last_became_peered":"2024-02-12T17:04:39.874748+0000","last_unstale":"2024-02-12T17:06:26.583756+0000","last_undegraded":"2024-02-12T17:06:26.583756+0000","last_fullsized":"2024-02-12T17:06:26.583756+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":41,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685089+0000","last_change":"2024-02-12T17:06:24.653101+0000","last_active":"2024-02-12T17:06:25.685089+0000","last_peered":"2024-02-12T17:06:25.685089+0000","last_clean":"2024-02-12T17:06:25.685089+0000","last_became_active":"2024-02-12T17:05:40.443276+0000","last_became_peered":"2024-02-12T17:05:40.443276+0000","last_unstale":"2024-02-12T17:06:25.685089+0000","last_undegraded":"2024-02-12T17:06:25.685089+0000","last_fullsized":"2024-02-12T17:06:25.685089+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:24.653029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:06:24.653029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":435,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685548+0000","last_change":"2024-02-12T17:04:39.888763+0000","last_active":"2024-02-12T17:06:25.685548+0000","last_peered":"2024-02-12T17:06:25.685548+0000","last_clean":"2024-02-12T17:06:25.685548+0000","last_became_active":"2024-02-12T17:04:39.888466+0000","last_became_peered":"2024-02-12T17:04:39.888466+0000","last_unstale":"2024-02-12T17:06:25.685548+0000","last_undegraded":"2024-02-12T17:06:25.685548+0000","last_fullsized":"2024-02-12T17:06:25.685548+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:27.858955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:27.858955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":172,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682985+0000","last_change":"2024-02-12T17:04:41.383385+0000","last_active":"2024-02-12T17:06:25.682985+0000","last_peered":"2024-02-12T17:06:25.682985+0000","last_clean":"2024-02-12T17:06:25.682985+0000","last_became_active":"2024-02-12T17:04:39.887688+0000","last_became_peered":"2024-02-12T17:04:39.887688+0000","last_unstale":"2024-02-12T17:06:25.682985+0000","last_undegraded":"2024-02-12T17:06:25.682985+0000","last_fullsized":"2024-02-12T17:06:25.682985+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:01:35.160145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:35.160145+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"458'1961","reported_seq":1976,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.516375+0000","last_change":"2024-02-12T17:06:18.392208+0000","last_active":"2024-02-12T17:06:38.516375+0000","last_peered":"2024-02-12T17:06:38.516375+0000","last_clean":"2024-02-12T17:06:38.516375+0000","last_became_active":"2024-02-12T17:06:18.391699+0000","last_became_peered":"2024-02-12T17:06:18.391699+0000","last_unstale":"2024-02-12T17:06:38.516375+0000","last_undegraded":"2024-02-12T17:06:38.516375+0000","last_fullsized":"2024-02-12T17:06:38.516375+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1961,"ondisk_log_size":1961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128516096,"num_objects":1961,"num_object_clones":0,"num_object_copies":5883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":125504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":476,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684675+0000","last_change":"2024-02-12T17:05:57.890025+0000","last_active":"2024-02-12T17:06:25.684675+0000","last_peered":"2024-02-12T17:06:25.684675+0000","last_clean":"2024-02-12T17:06:25.684675+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:06:25.684675+0000","last_undegraded":"2024-02-12T17:06:25.684675+0000","last_fullsized":"2024-02-12T17:06:25.684675+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.889955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.889955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":368,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684612+0000","last_change":"2024-02-12T17:05:58.811656+0000","last_active":"2024-02-12T17:06:25.684612+0000","last_peered":"2024-02-12T17:06:25.684612+0000","last_clean":"2024-02-12T17:06:25.684612+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:06:25.684612+0000","last_undegraded":"2024-02-12T17:06:25.684612+0000","last_fullsized":"2024-02-12T17:06:25.684612+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:58.811526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:05:58.811526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":180,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.683523+0000","last_change":"2024-02-12T17:03:50.552102+0000","last_active":"2024-02-12T17:06:25.683523+0000","last_peered":"2024-02-12T17:06:25.683523+0000","last_clean":"2024-02-12T17:06:25.683523+0000","last_became_active":"2024-02-12T17:03:50.551804+0000","last_became_peered":"2024-02-12T17:03:50.551804+0000","last_unstale":"2024-02-12T17:06:25.683523+0000","last_undegraded":"2024-02-12T17:06:25.683523+0000","last_fullsized":"2024-02-12T17:06:25.683523+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:00:54.794228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":364,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685598+0000","last_change":"2024-02-12T17:03:52.389245+0000","last_active":"2024-02-12T17:06:25.685598+0000","last_peered":"2024-02-12T17:06:25.685598+0000","last_clean":"2024-02-12T17:06:25.685598+0000","last_became_active":"2024-02-12T17:03:50.574562+0000","last_became_peered":"2024-02-12T17:03:50.574562+0000","last_unstale":"2024-02-12T17:06:25.685598+0000","last_undegraded":"2024-02-12T17:06:25.685598+0000","last_fullsized":"2024-02-12T17:06:25.685598+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:02:45.594719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:45.594719+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"458'1413","reported_seq":1428,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:33.699880+0000","last_change":"2024-02-12T17:06:18.394566+0000","last_active":"2024-02-12T17:06:33.699880+0000","last_peered":"2024-02-12T17:06:33.699880+0000","last_clean":"2024-02-12T17:06:33.699880+0000","last_became_active":"2024-02-12T17:06:18.394280+0000","last_became_peered":"2024-02-12T17:06:18.394280+0000","last_unstale":"2024-02-12T17:06:33.699880+0000","last_undegraded":"2024-02-12T17:06:33.699880+0000","last_fullsized":"2024-02-12T17:06:33.699880+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1413,"ondisk_log_size":1413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92602368,"num_objects":1413,"num_object_clones":0,"num_object_copies":4239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1413,"num_write_kb":90432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":155,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686615+0000","last_change":"2024-02-12T17:04:46.024414+0000","last_active":"2024-02-12T17:06:25.686615+0000","last_peered":"2024-02-12T17:06:25.686615+0000","last_clean":"2024-02-12T17:06:25.686615+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:06:25.686615+0000","last_undegraded":"2024-02-12T17:06:25.686615+0000","last_fullsized":"2024-02-12T17:06:25.686615+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:00:56.986137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:00:56.986137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":365,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.684005+0000","last_change":"2024-02-12T17:04:39.886902+0000","last_active":"2024-02-12T17:06:25.684005+0000","last_peered":"2024-02-12T17:06:25.684005+0000","last_clean":"2024-02-12T17:06:25.684005+0000","last_became_active":"2024-02-12T17:04:39.886403+0000","last_became_peered":"2024-02-12T17:04:39.886403+0000","last_unstale":"2024-02-12T17:06:25.684005+0000","last_undegraded":"2024-02-12T17:06:25.684005+0000","last_fullsized":"2024-02-12T17:06:25.684005+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":136,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686937+0000","last_change":"2024-02-12T17:04:41.654733+0000","last_active":"2024-02-12T17:06:25.686937+0000","last_peered":"2024-02-12T17:06:25.686937+0000","last_clean":"2024-02-12T17:06:25.686937+0000","last_became_active":"2024-02-12T17:04:39.889999+0000","last_became_peered":"2024-02-12T17:04:39.889999+0000","last_unstale":"2024-02-12T17:06:25.686937+0000","last_undegraded":"2024-02-12T17:06:25.686937+0000","last_fullsized":"2024-02-12T17:06:25.686937+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:00:50.756578+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:00:50.756578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":412,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.685336+0000","last_change":"2024-02-12T17:04:42.178594+0000","last_active":"2024-02-12T17:06:25.685336+0000","last_peered":"2024-02-12T17:06:25.685336+0000","last_clean":"2024-02-12T17:06:25.685336+0000","last_became_active":"2024-02-12T17:04:39.886809+0000","last_became_peered":"2024-02-12T17:04:39.886809+0000","last_unstale":"2024-02-12T17:06:25.685336+0000","last_undegraded":"2024-02-12T17:06:25.685336+0000","last_fullsized":"2024-02-12T17:06:25.685336+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:28.096708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.096708+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"458'1996","reported_seq":2011,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:38.331872+0000","last_change":"2024-02-12T17:06:18.395214+0000","last_active":"2024-02-12T17:06:38.331872+0000","last_peered":"2024-02-12T17:06:38.331872+0000","last_clean":"2024-02-12T17:06:38.331872+0000","last_became_active":"2024-02-12T17:06:18.394930+0000","last_became_peered":"2024-02-12T17:06:18.394930+0000","last_unstale":"2024-02-12T17:06:38.331872+0000","last_undegraded":"2024-02-12T17:06:38.331872+0000","last_fullsized":"2024-02-12T17:06:38.331872+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":1996,"ondisk_log_size":1996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":130809856,"num_objects":1996,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1996,"num_write_kb":127744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":445,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682616+0000","last_change":"2024-02-12T17:04:41.899891+0000","last_active":"2024-02-12T17:06:25.682616+0000","last_peered":"2024-02-12T17:06:25.682616+0000","last_clean":"2024-02-12T17:06:25.682616+0000","last_became_active":"2024-02-12T17:04:41.899598+0000","last_became_peered":"2024-02-12T17:04:41.899598+0000","last_unstale":"2024-02-12T17:06:25.682616+0000","last_undegraded":"2024-02-12T17:06:25.682616+0000","last_fullsized":"2024-02-12T17:06:25.682616+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:26.803903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.803903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":119,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.686912+0000","last_change":"2024-02-12T17:03:50.560958+0000","last_active":"2024-02-12T17:06:25.686912+0000","last_peered":"2024-02-12T17:06:25.686912+0000","last_clean":"2024-02-12T17:06:25.686912+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:06:25.686912+0000","last_undegraded":"2024-02-12T17:06:25.686912+0000","last_fullsized":"2024-02-12T17:06:25.686912+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:01.789464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:01.789464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":527,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-12T17:06:25.682473+0000","last_change":"2024-02-12T17:04:40.522253+0000","last_active":"2024-02-12T17:06:25.682473+0000","last_peered":"2024-02-12T17:06:25.682473+0000","last_clean":"2024-02-12T17:06:25.682473+0000","last_became_active":"2024-02-12T17:04:39.882260+0000","last_became_peered":"2024-02-12T17:04:39.882260+0000","last_unstale":"2024-02-12T17:06:25.682473+0000","last_undegraded":"2024-02-12T17:06:25.682473+0000","last_fullsized":"2024-02-12T17:06:25.682473+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:02:43.665248+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.665248+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"443'18","reported_seq":719,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-12T17:06:27.454663+0000","last_change":"2024-02-12T17:05:55.356529+0000","last_active":"2024-02-12T17:06:27.454663+0000","last_peered":"2024-02-12T17:06:27.454663+0000","last_clean":"2024-02-12T17:06:27.454663+0000","last_became_active":"2024-02-12T17:05:55.356115+0000","last_became_peered":"2024-02-12T17:05:55.356115+0000","last_unstale":"2024-02-12T17:06:27.454663+0000","last_undegraded":"2024-02-12T17:06:27.454663+0000","last_fullsized":"2024-02-12T17:06:27.454663+0000","mapping_epoch":441,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-02-12T17:00:49.304784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:00:49.304784+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1848115200,"num_objects":28200,"num_object_clones":0,"num_object_copies":84600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28200,"num_write_kb":1804800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5743640576,"data_stored":5743640576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28200,"ondisk_log_size":28200,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":6},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":322,"num_read_kb":317,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1093,"num_read_kb":850,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":11090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234190,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1742004,"kb_used_data":846444,"kb_used_omap":0,"kb_used_meta":895552,"kb_avail":92003148,"statfs":{"total":95995035648,"available":94211223552,"internally_reserved":0,"allocated":866758656,"data_stored":865887224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917045248},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:05:59 2024","interfaces":[{"interface":"back","average":{"1min":2.492,"5min":2.5150000000000001,"15min":1.9710000000000001},"min":{"1min":0.28399999999999997,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":9.6600000000000001,"5min":12.375,"15min":36.439},"last":3.02},{"interface":"front","average":{"1min":2.5390000000000001,"5min":2.0219999999999998,"15min":2.0219999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.19600000000000001},"max":{"1min":8.5039999999999996,"5min":8.5039999999999996,"15min":36.490000000000002},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:05:59 2024","interfaces":[{"interface":"back","average":{"1min":3.9860000000000002,"5min":2.6240000000000001,"15min":1.8049999999999999},"min":{"1min":0.315,"5min":0.23400000000000001,"15min":0.20899999999999999},"max":{"1min":13.224,"5min":13.224,"15min":13.224},"last":0.64000000000000001},{"interface":"front","average":{"1min":2.226,"5min":2.0960000000000001,"15min":1.6859999999999999},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.22700000000000001},"max":{"1min":10.194000000000001,"5min":10.194000000000001,"15min":17.324999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Mon Feb 12 17:05:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0169999999999999,"5min":2.194,"15min":1.55},"min":{"1min":0.255,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":9.7390000000000008,"5min":12.4,"15min":12.4},"last":2.9950000000000001},{"interface":"front","average":{"1min":4.4139999999999997,"5min":2.742,"15min":1.9750000000000001},"min":{"1min":0.39400000000000002,"5min":0.26400000000000001,"15min":0.23000000000000001},"max":{"1min":13.127000000000001,"5min":13.127000000000001,"15min":20.338000000000001},"last":3.3420000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:05:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3700000000000001,"5min":2.4089999999999998,"15min":1.881},"min":{"1min":0.29499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.641,"5min":13.608000000000001,"15min":20.433},"last":2.9689999999999999},{"interface":"front","average":{"1min":3.887,"5min":2.9870000000000001,"15min":2.2490000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.25600000000000001},"max":{"1min":13.196,"5min":31.152000000000001,"15min":31.152000000000001},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:05:59 2024","interfaces":[{"interface":"back","average":{"1min":2.181,"5min":2.9420000000000002,"15min":1.984},"min":{"1min":0.246,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":12.108000000000001,"5min":19.030000000000001,"15min":50.406999999999996},"last":2.8780000000000001},{"interface":"front","average":{"1min":3.8180000000000001,"5min":3.2829999999999999,"15min":2.383},"min":{"1min":0.28399999999999997,"5min":0.27000000000000002,"15min":0.23400000000000001},"max":{"1min":16.574999999999999,"5min":16.867000000000001,"15min":50.168999999999997},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Feb 12 17:05:59 2024","interfaces":[{"interface":"back","average":{"1min":2.593,"5min":3.4809999999999999,"15min":2.0699999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.24199999999999999},"max":{"1min":12.369999999999999,"5min":19.702000000000002,"15min":19.702000000000002},"last":2.9169999999999998},{"interface":"front","average":{"1min":3.407,"5min":3.1619999999999999,"15min":2.1749999999999998},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.26200000000000001},"max":{"1min":13.166,"5min":17.585999999999999,"15min":18.651},"last":0.71799999999999997}]},{"osd":6,"last update":"Mon Feb 12 17:05:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":2.5169999999999999,"15min":1.782},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27400000000000002},"max":{"1min":8.2420000000000009,"5min":10.522,"15min":17.234999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":3.0920000000000001,"5min":2.714,"15min":1.784},"min":{"1min":0.39700000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":9.7799999999999994,"5min":13.676,"15min":13.676},"last":2.9449999999999998}]}]},{"osd":6,"up_from":0,"seq":193273528532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575085,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738608,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1068284,"kb_used_data":176180,"kb_used_omap":0,"kb_used_meta":892096,"kb_avail":92676868,"statfs":{"total":95995035648,"available":94901112832,"internally_reserved":0,"allocated":180408320,"data_stored":179655988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913506304},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:05:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3119999999999998,"5min":1.046,"15min":1.3819999999999999},"min":{"1min":0.29699999999999999,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":6.2939999999999996,"5min":6.2939999999999996,"15min":23.510999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":2.8439999999999999,"5min":1.2729999999999999,"15min":1.385},"min":{"1min":0.27700000000000002,"5min":0.23000000000000001,"15min":0.193},"max":{"1min":8.2400000000000002,"5min":16.16,"15min":16.16},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:06:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":0.998,"15min":1.542},"min":{"1min":0.26300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.0469999999999997,"5min":6.0469999999999997,"15min":16.521000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":2.1320000000000001,"5min":1.3420000000000001,"15min":1.496},"min":{"1min":0.308,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":6.8940000000000001,"5min":9.827,"15min":10.340999999999999},"last":1.8200000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:06:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.468,"15min":1.7370000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":9.8819999999999997,"5min":9.8819999999999997,"15min":21.100000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.5960000000000001,"15min":1.7669999999999999},"min":{"1min":0.32500000000000001,"5min":0.221,"15min":0.221},"max":{"1min":4.3730000000000002,"5min":16.759,"15min":16.759},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:05:46 2024","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":1.55,"15min":1.641},"min":{"1min":0.28599999999999998,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":7.8639999999999999,"5min":7.8639999999999999,"15min":16.888999999999999},"last":3.851},{"interface":"front","average":{"1min":2.9390000000000001,"5min":1.7889999999999999,"15min":1.6639999999999999},"min":{"1min":0.33100000000000002,"5min":0.28799999999999998,"15min":0.28399999999999997},"max":{"1min":7.8540000000000001,"5min":9.1020000000000003,"15min":16.899999999999999},"last":1.8520000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:06:19 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":2.5640000000000001,"15min":2.2170000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":9.1769999999999996,"5min":50.404000000000003,"15min":50.404000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":3.3050000000000002,"5min":2.8069999999999999,"15min":2.2189999999999999},"min":{"1min":0.441,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":19.751999999999999,"5min":49.979999999999997,"15min":49.979999999999997},"last":0.624}]},{"osd":6,"last update":"Mon Feb 12 17:05:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6920000000000002,"5min":1.585,"15min":1.675},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.29999999999999999},"max":{"1min":5.6849999999999996,"5min":7.4489999999999998,"15min":17.907},"last":1.8720000000000001},{"interface":"front","average":{"1min":2.069,"5min":1.8919999999999999,"15min":1.954},"min":{"1min":0.39300000000000002,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":4.4740000000000002,"5min":7.7510000000000003,"15min":21.062000000000001},"last":3.9449999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:06:13 2024","interfaces":[{"interface":"back","average":{"1min":2.573,"5min":2.3530000000000002,"15min":1.996},"min":{"1min":0.498,"5min":0.438,"15min":0.32400000000000001},"max":{"1min":11.362,"5min":19.373000000000001,"15min":19.373000000000001},"last":3.8140000000000001},{"interface":"front","average":{"1min":2.7090000000000001,"5min":2.1800000000000002,"15min":1.9850000000000001},"min":{"1min":0.441,"5min":0.39900000000000002,"15min":0.33600000000000002},"max":{"1min":8.923,"5min":8.923,"15min":34.722999999999999},"last":1.891}]}]},{"osd":2,"up_from":18,"seq":77309411558,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2717384,"kb_used_data":1824704,"kb_used_omap":0,"kb_used_meta":892672,"kb_avail":91027768,"statfs":{"total":95995035648,"available":93212434432,"internally_reserved":0,"allocated":1868496896,"data_stored":1867669038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914096128},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:06:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.325,"15min":1.661},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":14.471,"5min":36.100999999999999,"15min":36.100999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":2.5209999999999999,"5min":1.839,"15min":1.6910000000000001},"min":{"1min":0.29599999999999999,"5min":0.215,"15min":0.158},"max":{"1min":20.359000000000002,"5min":36.299999999999997,"15min":36.299999999999997},"last":4.8520000000000003}]},{"osd":1,"last update":"Mon Feb 12 17:06:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1000000000000001,"5min":0.95799999999999996,"15min":1.581},"min":{"1min":0.246,"5min":0.22700000000000001,"15min":0.191},"max":{"1min":9.609,"5min":9.609,"15min":17.739000000000001},"last":4.0810000000000004},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.111,"15min":1.6579999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":7.5149999999999997,"5min":7.8609999999999998,"15min":15.766999999999999},"last":0.442}]},{"osd":3,"last update":"Mon Feb 12 17:05:41 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.1459999999999999,"15min":1.7929999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":8.0449999999999999,"5min":8.0449999999999999,"15min":51.579999999999998},"last":8.8209999999999997},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.149,"15min":1.55},"min":{"1min":0.32400000000000001,"5min":0.23699999999999999,"15min":0.23200000000000001},"max":{"1min":5.4180000000000001,"5min":5.4180000000000001,"15min":48.988999999999997},"last":1.552}]},{"osd":4,"last update":"Mon Feb 12 17:05:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3239999999999998,"5min":1.6619999999999999,"15min":1.859},"min":{"1min":0.25600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":10.930999999999999,"5min":18.34,"15min":18.658999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.8220000000000001,"5min":2.0419999999999998,"15min":1.915},"min":{"1min":0.41199999999999998,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":9.1620000000000008,"5min":18.355,"15min":18.355},"last":1.4470000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:06:12 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":2.0609999999999999,"15min":1.9790000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.27300000000000002},"max":{"1min":6.8250000000000002,"5min":32.57,"15min":32.57},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.298,"5min":2.3290000000000002,"15min":2.2320000000000002},"min":{"1min":0.40600000000000003,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":7.3879999999999999,"5min":31.905999999999999,"15min":31.905999999999999},"last":4.1159999999999997}]},{"osd":6,"last update":"Mon Feb 12 17:05:41 2024","interfaces":[{"interface":"back","average":{"1min":2.2650000000000001,"5min":1.494,"15min":2.0009999999999999},"min":{"1min":0.45600000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.7199999999999998,"5min":7.6669999999999998,"15min":17.782},"last":4.1420000000000003},{"interface":"front","average":{"1min":2.5939999999999999,"5min":1.621,"15min":1.7729999999999999},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.29799999999999999},"max":{"1min":10.6,"5min":10.6,"15min":13.446999999999999},"last":1.7589999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:06:12 2024","interfaces":[{"interface":"back","average":{"1min":3.1520000000000001,"5min":3.3999999999999999,"15min":2.4020000000000001},"min":{"1min":0.48099999999999998,"5min":0.33700000000000002,"15min":0.32600000000000001},"max":{"1min":10.888999999999999,"5min":45.468000000000004,"15min":45.468000000000004},"last":2.8580000000000001},{"interface":"front","average":{"1min":2.5099999999999998,"5min":3.0990000000000002,"15min":2.456},"min":{"1min":0.50600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":10.712999999999999,"5min":46.656999999999996,"15min":46.656999999999996},"last":4.1660000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149922,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680360,"kb_used_data":1379040,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92064792,"statfs":{"total":95995035648,"available":94274347008,"internally_reserved":0,"allocated":1412136960,"data_stored":1411277711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.5009999999999999,"15min":1.675},"min":{"1min":0.26900000000000002,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.9079999999999999,"5min":58.451999999999998,"15min":58.451999999999998},"last":1.018},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.9119999999999999,"15min":1.7729999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.452,"5min":57.497999999999998,"15min":57.497999999999998},"last":9.0790000000000006}]},{"osd":1,"last update":"Mon Feb 12 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.296,"15min":1.508},"min":{"1min":0.30599999999999999,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":3.5,"5min":11.537000000000001,"15min":13.94},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.96099999999999997,"15min":1.3160000000000001},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":0.73199999999999998,"5min":10.278,"15min":14.962},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":1.171,"15min":1.6220000000000001},"min":{"1min":0.28899999999999998,"5min":0.24099999999999999,"15min":0.23400000000000001},"max":{"1min":2.008,"5min":10.302,"15min":49.112000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.96099999999999997,"15min":1.708},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.1539999999999999,"5min":6.4960000000000004,"15min":51.997999999999998},"last":5.8460000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:05:52 2024","interfaces":[{"interface":"back","average":{"1min":2.3090000000000002,"5min":1.5940000000000001,"15min":1.8260000000000001},"min":{"1min":0.312,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":6.4880000000000004,"5min":11.047000000000001,"15min":19.010000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.6990000000000001,"15min":1.724},"min":{"1min":0.32600000000000001,"5min":0.32000000000000001,"15min":0.28799999999999998},"max":{"1min":4.5170000000000003,"5min":10.250999999999999,"15min":19.033999999999999},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:06:17 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.9159999999999999,"15min":2.012},"min":{"1min":0.38300000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":8.1850000000000005,"5min":14.236000000000001,"15min":19.882999999999999},"last":1.052},{"interface":"front","average":{"1min":1.6719999999999999,"5min":2.125,"15min":2.0960000000000001},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.33300000000000002},"max":{"1min":6.5250000000000004,"5min":13.927,"15min":15.595000000000001},"last":0.42499999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.5860000000000001,"15min":1.5589999999999999},"min":{"1min":0.39000000000000001,"5min":0.38900000000000001,"15min":0.34100000000000003},"max":{"1min":3.246,"5min":10.212999999999999,"15min":12.856999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.123,"5min":1.2889999999999999,"15min":1.7190000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.27500000000000002},"max":{"1min":8.1489999999999991,"5min":8.5039999999999996,"15min":19.925999999999998},"last":1.0840000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:05:52 2024","interfaces":[{"interface":"back","average":{"1min":5.0099999999999998,"5min":2.5640000000000001,"15min":2.2010000000000001},"min":{"1min":0.38100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":37.719999999999999,"5min":37.719999999999999,"15min":37.719999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":4.883,"5min":2.698,"15min":2.2080000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.32800000000000001},"max":{"1min":38.948,"5min":38.948,"15min":38.948},"last":1.1100000000000001}]}]},{"osd":4,"up_from":0,"seq":137438953694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757466,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2123752,"kb_used_data":1231264,"kb_used_omap":0,"kb_used_meta":892480,"kb_avail":91621400,"statfs":{"total":95995035648,"available":93820313600,"internally_reserved":0,"allocated":1260814336,"data_stored":1259954212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913899520},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:06:11 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.4730000000000001,"15min":1.8140000000000001},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20300000000000001},"max":{"1min":5.4240000000000004,"5min":10.971,"15min":46.365000000000002},"last":1.121},{"interface":"front","average":{"1min":2.2909999999999999,"5min":1.9570000000000001,"15min":2.2240000000000002},"min":{"1min":0.21199999999999999,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":8.1400000000000006,"5min":12.140000000000001,"15min":46.134},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:06:11 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.575,"15min":1.6619999999999999},"min":{"1min":0.26700000000000002,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":6.4820000000000002,"5min":11.19,"15min":16.655000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":2.1160000000000001,"5min":2.0419999999999998,"15min":1.9930000000000001},"min":{"1min":0.254,"5min":0.23799999999999999,"15min":0.21199999999999999},"max":{"1min":8.1159999999999997,"5min":12.154,"15min":16.712},"last":2.169}]},{"osd":2,"last update":"Mon Feb 12 17:06:11 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.641,"15min":1.9590000000000001},"min":{"1min":0.24199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.9029999999999996,"5min":11.007999999999999,"15min":18.218},"last":2.1379999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.7010000000000001,"15min":1.8400000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":7.4850000000000003,"5min":8.0470000000000006,"15min":18.327000000000002},"last":0.749}]},{"osd":3,"last update":"Mon Feb 12 17:06:11 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.5589999999999999,"15min":1.8460000000000001},"min":{"1min":0.30599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.3710000000000004,"5min":11.045,"15min":18.251000000000001},"last":1.149},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.748,"15min":1.772},"min":{"1min":0.30199999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.4509999999999996,"5min":8.0790000000000006,"15min":20.263000000000002},"last":0.34499999999999997}]},{"osd":4,"last update":"Mon Feb 12 17:06:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0659999999999998,"5min":2.367,"15min":1.8640000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.23699999999999999},"max":{"1min":10.939,"5min":12.656000000000001,"15min":12.656000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.9430000000000001,"15min":1.964},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":5.3479999999999999,"5min":11.068,"15min":20.550000000000001},"last":1.0920000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:05:44 2024","interfaces":[{"interface":"back","average":{"1min":3.044,"5min":2.2429999999999999,"15min":1.78},"min":{"1min":0.33200000000000002,"5min":0.29199999999999998,"15min":0.26400000000000001},"max":{"1min":10.497,"5min":12.121,"15min":12.281000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":2.1200000000000001,"5min":2.2519999999999998,"15min":1.8400000000000001},"min":{"1min":0.314,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":9.5020000000000007,"5min":13.916,"15min":22.035},"last":1.0569999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:06:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":2.5430000000000001,"15min":1.946},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5410000000000004,"5min":13.647,"15min":19.184999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.8979999999999999,"5min":2.867,"15min":2.0390000000000001},"min":{"1min":0.35799999999999998,"5min":0.30499999999999999,"15min":0.26800000000000002},"max":{"1min":8.1839999999999993,"5min":20.274000000000001,"15min":20.274000000000001},"last":0.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180944896,"data_stored":180944896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1921449984,"data_stored":1921449984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1423441920,"data_stored":1423441920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1290534912,"data_stored":1290534912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":927268864,"data_stored":927268864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:06:41.059 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-02-12T17:06:41.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:41.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:39.828225+0000 mon.a (mon.0) 1615 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:06:41.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:39.828262+0000 mon.a (mon.0) 1616 : cluster [INF] Cluster is now healthy 2024-02-12T17:06:41.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:39.837380+0000 mon.a (mon.0) 1617 : audit [INF] from='client.35768 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:06:41.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:39.837456+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e459: 8 total, 8 up, 5 in 2024-02-12T17:06:41.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:39.909054+0000 osd.5 (osd.5) 215 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:06:41.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:40.058080+0000 osd.5 (osd.5) 216 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:06:41.164 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:40] "GET /metrics HTTP/1.1" 200 176493 "" "Prometheus/2.33.4" 2024-02-12T17:06:41.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:39.828225+0000 mon.a (mon.0) 1615 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:06:41.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:39.828262+0000 mon.a (mon.0) 1616 : cluster [INF] Cluster is now healthy 2024-02-12T17:06:41.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:39.837380+0000 mon.a (mon.0) 1617 : audit [INF] from='client.35768 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:06:41.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:39.837456+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e459: 8 total, 8 up, 5 in 2024-02-12T17:06:41.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:39.909054+0000 osd.5 (osd.5) 215 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:06:41.164 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:40.058080+0000 osd.5 (osd.5) 216 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:06:41.179 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:39.828225+0000 mon.a (mon.0) 1615 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:06:41.180 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:39.828262+0000 mon.a (mon.0) 1616 : cluster [INF] Cluster is now healthy 2024-02-12T17:06:41.180 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:39.837380+0000 mon.a (mon.0) 1617 : audit [INF] from='client.35768 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:06:41.180 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:39.837456+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e459: 8 total, 8 up, 5 in 2024-02-12T17:06:41.180 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:39.909054+0000 osd.5 (osd.5) 215 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:06:41.180 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:40.058080+0000 osd.5 (osd.5) 216 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:06:41.183 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:41.212 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:41.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:41.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:41.353 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:41.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:41.492 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:41.565 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:41.570 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:41.854 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 17 16 37111 37095 136.32 107.062 0.00604705 0.00733078 2024-02-12T17:06:41.936 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:41.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:42.044 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:42.119 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:42.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:40.710003+0000 mgr.y (mgr.24481) 647 : cluster [DBG] pgmap v892: 128 pgs: 128 active+clean; 1.9 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 1022 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:40.956430+0000 osd.7 (osd.7) 210 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:40.961092+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:41.352812+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e460: 8 total, 8 up, 5 in 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:41.440134+0000 osd.3 (osd.3) 137 : cluster [DBG] 2.d scrub starts 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:41.443664+0000 osd.3 (osd.3) 138 : cluster [DBG] 2.d scrub ok 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:40.710003+0000 mgr.y (mgr.24481) 647 : cluster [DBG] pgmap v892: 128 pgs: 128 active+clean; 1.9 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 1022 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:40.956430+0000 osd.7 (osd.7) 210 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:40.961092+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:06:42.176 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:41.352812+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e460: 8 total, 8 up, 5 in 2024-02-12T17:06:42.177 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:41.440134+0000 osd.3 (osd.3) 137 : cluster [DBG] 2.d scrub starts 2024-02-12T17:06:42.177 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:41.443664+0000 osd.3 (osd.3) 138 : cluster [DBG] 2.d scrub ok 2024-02-12T17:06:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:40.710003+0000 mgr.y (mgr.24481) 647 : cluster [DBG] pgmap v892: 128 pgs: 128 active+clean; 1.9 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 1022 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-12T17:06:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:40.956430+0000 osd.7 (osd.7) 210 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:06:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:40.961092+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:06:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:41.352812+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e460: 8 total, 8 up, 5 in 2024-02-12T17:06:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:41.440134+0000 osd.3 (osd.3) 137 : cluster [DBG] 2.d scrub starts 2024-02-12T17:06:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:41.443664+0000 osd.3 (osd.3) 138 : cluster [DBG] 2.d scrub ok 2024-02-12T17:06:42.306 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:06:42.306 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:06:42.325 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:06:42.360 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:06:42.387 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:06:42.423 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:06:42.454 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:06:42.483 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:06:42.485 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:06:42.485 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:06:42.485 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:06:42.485 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:06:42.485 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:06:42.486 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:06:42.486 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:06:42.486 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:06:42.855 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 18 16 38643 38627 134.066 95.75 0.0126567 0.00745374 2024-02-12T17:06:42.885 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:06:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:06:43.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:42.214605+0000 mon.b (mon.1) 83 : audit [DBG] from='client.? v1:172.21.15.33:0/1325847623' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:43.149 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:42.214605+0000 mon.b (mon.1) 83 : audit [DBG] from='client.? v1:172.21.15.33:0/1325847623' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:42.214605+0000 mon.b (mon.1) 83 : audit [DBG] from='client.? v1:172.21.15.33:0/1325847623' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:43.855 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 19 16 40323 40307 132.536 105 0.00906125 0.00753946 2024-02-12T17:06:43.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:42.710321+0000 mgr.y (mgr.24481) 648 : cluster [DBG] pgmap v894: 128 pgs: 128 active+clean; 1.9 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-12T17:06:43.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:42.997427+0000 osd.7 (osd.7) 212 : cluster [DBG] 6.9 deep-scrub starts 2024-02-12T17:06:43.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:43.014733+0000 osd.7 (osd.7) 213 : cluster [DBG] 6.9 deep-scrub ok 2024-02-12T17:06:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:42.710321+0000 mgr.y (mgr.24481) 648 : cluster [DBG] pgmap v894: 128 pgs: 128 active+clean; 1.9 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-12T17:06:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:42.997427+0000 osd.7 (osd.7) 212 : cluster [DBG] 6.9 deep-scrub starts 2024-02-12T17:06:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:43.014733+0000 osd.7 (osd.7) 213 : cluster [DBG] 6.9 deep-scrub ok 2024-02-12T17:06:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:42.710321+0000 mgr.y (mgr.24481) 648 : cluster [DBG] pgmap v894: 128 pgs: 128 active+clean; 1.9 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-12T17:06:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:42.997427+0000 osd.7 (osd.7) 212 : cluster [DBG] 6.9 deep-scrub starts 2024-02-12T17:06:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:43.014733+0000 osd.7 (osd.7) 213 : cluster [DBG] 6.9 deep-scrub ok 2024-02-12T17:06:44.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:06:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:06:44.855 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:06:44.854512+0000 min lat: 0.00118871 max lat: 0.0907607 avg lat: 0.00762533 2024-02-12T17:06:44.855 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:06:44.855 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 20 16 41964 41948 131.038 102.562 0.00499537 0.00762533 2024-02-12T17:06:44.993 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:44.454149+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.17 deep-scrub starts 2024-02-12T17:06:44.993 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:44.468713+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.17 deep-scrub ok 2024-02-12T17:06:44.993 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:44.454149+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.17 deep-scrub starts 2024-02-12T17:06:44.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:44.468713+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.17 deep-scrub ok 2024-02-12T17:06:44.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:06:44.995 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:45.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:45.101 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:45.134 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:45.162 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:45.164 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:44.454149+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.17 deep-scrub starts 2024-02-12T17:06:45.164 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:44.468713+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.17 deep-scrub ok 2024-02-12T17:06:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:45.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:45.266 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:45.367 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:06:45.367 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:06:45.367 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:06:45.368 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:06:45.368 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:06:45.369 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:45.418 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:45.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:45.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:45.575 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:45.606 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:45.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:45.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:45.759 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:45.855 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 21 16 43305 43289 128.789 83.8125 0.00834097 0.00775821 2024-02-12T17:06:46.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:44.442769+0000 mgr.y (mgr.24481) 649 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:44.711186+0000 mgr.y (mgr.24481) 650 : cluster [DBG] pgmap v895: 128 pgs: 128 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-02-12T17:06:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:44.759804+0000 osd.5 (osd.5) 217 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:06:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:44.765035+0000 osd.5 (osd.5) 218 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:06:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:45.495447+0000 osd.3 (osd.3) 141 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T17:06:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:45.502630+0000 osd.3 (osd.3) 142 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T17:06:46.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:44.442769+0000 mgr.y (mgr.24481) 649 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:46.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:44.711186+0000 mgr.y (mgr.24481) 650 : cluster [DBG] pgmap v895: 128 pgs: 128 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-02-12T17:06:46.037 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:44.759804+0000 osd.5 (osd.5) 217 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:06:46.037 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:44.765035+0000 osd.5 (osd.5) 218 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:06:46.037 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:45.495447+0000 osd.3 (osd.3) 141 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T17:06:46.037 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:45.502630+0000 osd.3 (osd.3) 142 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T17:06:46.060 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:06:46.060 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:06:46.060 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-12T17:06:46.061 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T17:06:46.206 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:46.231 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:46.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:44.442769+0000 mgr.y (mgr.24481) 649 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:44.711186+0000 mgr.y (mgr.24481) 650 : cluster [DBG] pgmap v895: 128 pgs: 128 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-02-12T17:06:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:44.759804+0000 osd.5 (osd.5) 217 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:06:46.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:44.765035+0000 osd.5 (osd.5) 218 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:06:46.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:45.495447+0000 osd.3 (osd.3) 141 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T17:06:46.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:45.502630+0000 osd.3 (osd.3) 142 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T17:06:46.250 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T17:06:46.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:46.373 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:46.395 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:46.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:46.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:46.537 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:46.561 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:46.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:46.665 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:46.699 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:46.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:46.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:06:46.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:06:46.855 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 22 16 44568 44552 126.519 78.9375 0.0131364 0.00789889 2024-02-12T17:06:46.911 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:46.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:45.751801+0000 osd.5 (osd.5) 219 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:06:46.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:45.758479+0000 osd.5 (osd.5) 220 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:06:46.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:45.970954+0000 osd.7 (osd.7) 214 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:06:46.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:46.001880+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:06:46.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:46.532814+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:06:46.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:46.537620+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:06:47.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:45.751801+0000 osd.5 (osd.5) 219 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:06:47.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:45.758479+0000 osd.5 (osd.5) 220 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:06:47.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:45.970954+0000 osd.7 (osd.7) 214 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:06:47.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:46.001880+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:06:47.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:46.532814+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:06:47.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:46.537620+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:06:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:45.751801+0000 osd.5 (osd.5) 219 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:06:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:45.758479+0000 osd.5 (osd.5) 220 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:06:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:45.970954+0000 osd.7 (osd.7) 214 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:06:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:46.001880+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:06:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:46.532814+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:06:47.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:46.537620+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:06:47.289 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:06:47.290 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:47.297 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:47.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:06:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:06:47.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:47.435 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:47.459 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:47.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:47.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:47.597 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:47.621 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:47.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:47.727 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:47.760 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:47.791 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:47.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:47.859 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 23 16 45870 45854 124.556 81.375 0.0125727 0.00802264 2024-02-12T17:06:47.895 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:47.953 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:48.035 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:48.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:48.141 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:48.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:46.711631+0000 mgr.y (mgr.24481) 651 : cluster [DBG] pgmap v896: 128 pgs: 128 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-12T17:06:48.168 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:47.493762+0000 mon.a (mon.0) 1620 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:48.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:46.711631+0000 mgr.y (mgr.24481) 651 : cluster [DBG] pgmap v896: 128 pgs: 128 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-12T17:06:48.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:47.493762+0000 mon.a (mon.0) 1620 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:48.201 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:46.711631+0000 mgr.y (mgr.24481) 651 : cluster [DBG] pgmap v896: 128 pgs: 128 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-12T17:06:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:47.493762+0000 mon.a (mon.0) 1620 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:06:48.562 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:48.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:48.674 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:48.730 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:48.794 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:48.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:48.856 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 24 16 47220 47204 122.882 84.375 0.00893759 0.00813222 2024-02-12T17:06:48.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:06:48.935 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:48.942 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:47.832461+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.a scrub starts 2024-02-12T17:06:48.942 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:47.945294+0000 osd.5 (osd.5) 222 : cluster [DBG] 6.a scrub ok 2024-02-12T17:06:48.943 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:47.832461+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.a scrub starts 2024-02-12T17:06:48.943 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:47.945294+0000 osd.5 (osd.5) 222 : cluster [DBG] 6.a scrub ok 2024-02-12T17:06:48.962 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:06:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:48.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:06:49.067 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:06:49.109 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:49.134 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:06:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:49.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:06:49.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:47.832461+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.a scrub starts 2024-02-12T17:06:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:47.945294+0000 osd.5 (osd.5) 222 : cluster [DBG] 6.a scrub ok 2024-02-12T17:06:49.281 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:49.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:49.414 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:49.478 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:49.856 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 25 16 48719 48703 121.715 93.6875 0.00931955 0.00821063 2024-02-12T17:06:49.936 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:49.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:50.042 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:50.077 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:50.106 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:50.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:50.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:06:50.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:48.713732+0000 mgr.y (mgr.24481) 652 : cluster [DBG] pgmap v897: 128 pgs: 128 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 577 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T17:06:50.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:48.948572+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:50.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:48.948623+0000 osd.2 (osd.2) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:50.255 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:50.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:48.713732+0000 mgr.y (mgr.24481) 652 : cluster [DBG] pgmap v897: 128 pgs: 128 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 577 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T17:06:50.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:48.948572+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:50.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:48.948623+0000 osd.2 (osd.2) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:50.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:48.713732+0000 mgr.y (mgr.24481) 652 : cluster [DBG] pgmap v897: 128 pgs: 128 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 577 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-12T17:06:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:48.948572+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:48.948623+0000 osd.2 (osd.2) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:50.287 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:06:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:50.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:06:50.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:06:50.430 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:50.459 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:06:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:50.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:06:50.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:06:50.614 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:50.856 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 26 16 50138 50122 120.444 88.6875 0.0129871 0.00829808 2024-02-12T17:06:50.968 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:06:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:50] "GET /metrics HTTP/1.1" 200 176504 "" "Prometheus/2.33.4" 2024-02-12T17:06:50.969 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:49.871500+0000 osd.7 (osd.7) 216 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:06:50.969 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:49.876195+0000 osd.7 (osd.7) 217 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:06:50.969 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:49.962486+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e461: 8 total, 8 up, 5 in 2024-02-12T17:06:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:49.871500+0000 osd.7 (osd.7) 216 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:06:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:49.876195+0000 osd.7 (osd.7) 217 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:06:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:49.962486+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e461: 8 total, 8 up, 5 in 2024-02-12T17:06:51.241 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:49.871500+0000 osd.7 (osd.7) 216 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:06:51.241 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:49.876195+0000 osd.7 (osd.7) 217 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:06:51.241 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:49.962486+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e461: 8 total, 8 up, 5 in 2024-02-12T17:06:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:51.830 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:06:51.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:06:51.859 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 27 16 51554 51538 119.261 88.5 0.0103726 0.00838052 2024-02-12T17:06:51.932 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:51.995 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:52.043 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:52.049 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:50.714122+0000 mgr.y (mgr.24481) 653 : cluster [DBG] pgmap v899: 128 pgs: 128 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-02-12T17:06:52.049 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:51.363519+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e462: 8 total, 8 up, 5 in 2024-02-12T17:06:52.049 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:51.619037+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:06:52.049 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:51.622395+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:06:52.050 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:50.714122+0000 mgr.y (mgr.24481) 653 : cluster [DBG] pgmap v899: 128 pgs: 128 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-02-12T17:06:52.050 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:51.363519+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e462: 8 total, 8 up, 5 in 2024-02-12T17:06:52.050 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:51.619037+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:06:52.050 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:51.622395+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:06:52.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:52.151 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:06:52.202 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:50.714122+0000 mgr.y (mgr.24481) 653 : cluster [DBG] pgmap v899: 128 pgs: 128 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-02-12T17:06:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:51.363519+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e462: 8 total, 8 up, 5 in 2024-02-12T17:06:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:51.619037+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:06:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:51.622395+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:06:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:06:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:52.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:06:52.353 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:52.406 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:52.525 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:06:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:06:52.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:06:52.634 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:06:52.687 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:06:52.856 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 28 16 52795 52779 117.772 77.5625 0.0277897 0.00848417 2024-02-12T17:06:52.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:06:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:06:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:06:52.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:51.871836+0000 osd.7 (osd.7) 218 : cluster [DBG] 2.b scrub starts 2024-02-12T17:06:52.997 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:51.877822+0000 osd.7 (osd.7) 219 : cluster [DBG] 2.b scrub ok 2024-02-12T17:06:53.017 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:06:53.017 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 462, 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2024-02-12T16:43:50.282468+0000", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2024-02-12T17:06:51.352769+0000", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2024-02-12T16:49:30.933575+0000", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2024-02-12T17:04:38.823636+0000", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5799936, 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2024-02-12T17:06:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 18, 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 10, 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 8, 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "jewel", 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "pacific", 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2024-02-12T17:06:53.019 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "device_health_metrics", 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:47:33.056945+0000", 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:53.020 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "71", 2024-02-12T17:06:53.021 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:53.022 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:53.023 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:53.024 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_devicehealth": {} 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".rgw.root", 2024-02-12T17:06:53.025 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:37.647343+0000", 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:53.026 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.027 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "59", 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:53.028 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.029 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "iscsi", 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:38.076200+0000", 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:53.031 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 3, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 3, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 3, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 3, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 3, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "60", 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.034 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:53.036 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 4, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.log", 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:39.589565+0000", 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:53.037 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:53.038 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "61", 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:53.039 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:53.040 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:53.041 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 5, 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.control", 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:41.623856+0000", 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:53.042 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "5.10", 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 429, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 428, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 428, 2024-02-12T17:06:53.043 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "380'16" 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "433", 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "429", 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "431", 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:53.044 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:53.045 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:53.046 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 6, 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.meta", 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:43.693806+0000", 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:53.047 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "6.16", 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 396, 2024-02-12T17:06:53.048 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 395, 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 395, 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "396", 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "396", 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "394", 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:53.049 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:53.050 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:53.051 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_bias": 4 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 10, 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_3", 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T17:06:16.954458+0000", 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:06:53.052 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2024-02-12T17:06:53.053 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "455", 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:06:53.054 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:06:53.055 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:06:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "rados": {} 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7", 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:53.057 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0.2548370361328125, 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 8, 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 451, 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2024-02-12T17:06:53.059 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/2545411201", 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6802/2545411201", 2024-02-12T17:06:53.060 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6804/2545411201", 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6803/2545411201", 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24", 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:53.061 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 13, 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 198, 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:53.062 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2024-02-12T17:06:53.063 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.064 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6805/1723231708", 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6806/1723231708", 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6808/1723231708", 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6807/1723231708", 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c", 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:53.065 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0, 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 18, 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 440, 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.066 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.067 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3126580659", 2024-02-12T17:06:53.068 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6810/3126580659", 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6812/3126580659", 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6811/3126580659", 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9", 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:53.069 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 26, 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 451, 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:53.070 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6813/3941314957", 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6814/3941314957", 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6816/3941314957", 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6815/3941314957", 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:53.072 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0498129c-861c-4f7e-b8ba-219949da3130", 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.54083251953125, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 32, 2024-02-12T17:06:53.073 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 440, 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6800", 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.074 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6801", 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6803", 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:53.075 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6802", 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6800/544126779", 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6801/544126779", 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6803/544126779", 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6802/544126779", 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:53.076 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50", 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 38, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 451, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:53.077 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6804", 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6805", 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.078 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6807", 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6806", 2024-02-12T17:06:53.079 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6804/3229884005", 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6805/3229884005", 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6807/3229884005", 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6806/3229884005", 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:06:53.080 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821", 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 45, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 248, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6808", 2024-02-12T17:06:53.081 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6809", 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.082 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6811", 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6810", 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6808/507305558", 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6809/507305558", 2024-02-12T17:06:53.083 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6811/507305558", 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6810/507305558", 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66", 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:06:53.084 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 54, 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 451, 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6812", 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.085 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6813", 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6815", 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.086 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6814", 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6812/3401977873", 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6813/3401977873", 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6815/3401977873", 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6814/3401977873", 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:06:53.087 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:28.101520+0000", 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:06:53.088 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:51.331519+0000", 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:06:48.948626+0000", 2024-02-12T17:06:53.089 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:06:16.169738+0000", 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:53.090 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:06:25.564600+0000", 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:05:39.766966+0000", 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:06:23.566041+0000", 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:06:53.092 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:06:22.248730+0000", 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [], 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [], 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": { 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/2829464018": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/1835128904": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2277321750": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2116602769": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:06:53.093 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3967118778": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2298361247": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/218227980": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2070582473": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/4291575956": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3157562110": "2024-02-13T16:44:22.727241+0000" 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "ruleset-failure-domain": "osd", 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2024-02-12T17:06:53.094 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [], 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:06:53.095 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:06:53.273 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:51.871836+0000 osd.7 (osd.7) 218 : cluster [DBG] 2.b scrub starts 2024-02-12T17:06:53.273 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:51.877822+0000 osd.7 (osd.7) 219 : cluster [DBG] 2.b scrub ok 2024-02-12T17:06:53.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:51.871836+0000 osd.7 (osd.7) 218 : cluster [DBG] 2.b scrub starts 2024-02-12T17:06:53.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:51.877822+0000 osd.7 (osd.7) 219 : cluster [DBG] 2.b scrub ok 2024-02-12T17:06:53.573 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:06:53.573 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:06:53.573 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:06:53.573 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:06:53.573 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:06:53.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:06:53.575 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:06:53.858 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 29 16 53888 53872 116.062 68.3125 0.0165888 0.00861072 2024-02-12T17:06:54.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:52.716133+0000 mgr.y (mgr.24481) 654 : cluster [DBG] pgmap v901: 128 pgs: 128 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 54 MiB/s wr, 872 op/s 2024-02-12T17:06:54.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:52.822854+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.1a scrub starts 2024-02-12T17:06:54.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:52.828441+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.1a scrub ok 2024-02-12T17:06:54.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:53.012655+0000 mon.b (mon.1) 84 : audit [DBG] from='client.? v1:172.21.15.33:0/118355068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:06:54.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:53.184582+0000 mon.a (mon.0) 1623 : audit [DBG] from='client.? v1:172.21.15.33:0/1216029367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:54.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:53.642915+0000 osd.3 (osd.3) 147 : cluster [DBG] 2.e scrub starts 2024-02-12T17:06:54.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:53.648833+0000 osd.3 (osd.3) 148 : cluster [DBG] 2.e scrub ok 2024-02-12T17:06:54.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:53.828578+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.33:0/602015354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:54.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:53.829005+0000 mon.a (mon.0) 1624 : audit [INF] from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:54.019 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:06:54.019 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:06:54.033 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:06:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:52.716133+0000 mgr.y (mgr.24481) 654 : cluster [DBG] pgmap v901: 128 pgs: 128 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 54 MiB/s wr, 872 op/s 2024-02-12T17:06:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:52.822854+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.1a scrub starts 2024-02-12T17:06:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:52.828441+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.1a scrub ok 2024-02-12T17:06:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:53.012655+0000 mon.b (mon.1) 84 : audit [DBG] from='client.? v1:172.21.15.33:0/118355068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:06:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:53.184582+0000 mon.a (mon.0) 1623 : audit [DBG] from='client.? v1:172.21.15.33:0/1216029367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:53.642915+0000 osd.3 (osd.3) 147 : cluster [DBG] 2.e scrub starts 2024-02-12T17:06:54.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:53.648833+0000 osd.3 (osd.3) 148 : cluster [DBG] 2.e scrub ok 2024-02-12T17:06:54.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:53.828578+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.33:0/602015354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:54.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:53.829005+0000 mon.a (mon.0) 1624 : audit [INF] from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:54.454 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:52.716133+0000 mgr.y (mgr.24481) 654 : cluster [DBG] pgmap v901: 128 pgs: 128 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 639 B/s rd, 54 MiB/s wr, 872 op/s 2024-02-12T17:06:54.455 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:52.822854+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.1a scrub starts 2024-02-12T17:06:54.455 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:52.828441+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.1a scrub ok 2024-02-12T17:06:54.455 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:53.012655+0000 mon.b (mon.1) 84 : audit [DBG] from='client.? v1:172.21.15.33:0/118355068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:06:54.455 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:53.184582+0000 mon.a (mon.0) 1623 : audit [DBG] from='client.? v1:172.21.15.33:0/1216029367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:06:54.455 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:53.642915+0000 osd.3 (osd.3) 147 : cluster [DBG] 2.e scrub starts 2024-02-12T17:06:54.455 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:53.648833+0000 osd.3 (osd.3) 148 : cluster [DBG] 2.e scrub ok 2024-02-12T17:06:54.455 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:53.828578+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.33:0/602015354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:54.455 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:53.829005+0000 mon.a (mon.0) 1624 : audit [INF] from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:06:54.456 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:06:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:06:54.858 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 30 16 55309 55293 115.152 88.8125 0.0118496 0.00867946 2024-02-12T17:06:55.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:53.869354+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:06:55.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:53.886333+0000 osd.7 (osd.7) 223 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:06:55.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:53.995261+0000 mon.a (mon.0) 1625 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:55.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:54.012115+0000 mon.a (mon.0) 1626 : audit [INF] from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:06:55.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:54.012171+0000 mon.a (mon.0) 1627 : cluster [DBG] osdmap e463: 8 total, 8 up, 5 in 2024-02-12T17:06:55.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:53.869354+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:06:55.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:53.886333+0000 osd.7 (osd.7) 223 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:06:55.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:53.995261+0000 mon.a (mon.0) 1625 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:55.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:54.012115+0000 mon.a (mon.0) 1626 : audit [INF] from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:06:55.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:54.012171+0000 mon.a (mon.0) 1627 : cluster [DBG] osdmap e463: 8 total, 8 up, 5 in 2024-02-12T17:06:55.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:53.869354+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:06:55.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:53.886333+0000 osd.7 (osd.7) 223 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:06:55.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:53.995261+0000 mon.a (mon.0) 1625 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:06:55.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:54.012115+0000 mon.a (mon.0) 1626 : audit [INF] from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:06:55.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:54.012171+0000 mon.a (mon.0) 1627 : cluster [DBG] osdmap e463: 8 total, 8 up, 5 in 2024-02-12T17:06:55.858 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 31 16 56690 56674 114.221 86.3125 0.00971588 0.00875028 2024-02-12T17:06:56.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:06:54.452486+0000 mgr.y (mgr.24481) 655 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:56.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:54.716538+0000 mgr.y (mgr.24481) 656 : cluster [DBG] pgmap v903: 128 pgs: 128 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-12T17:06:56.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:06:54.452486+0000 mgr.y (mgr.24481) 655 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:56.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:54.716538+0000 mgr.y (mgr.24481) 656 : cluster [DBG] pgmap v903: 128 pgs: 128 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-12T17:06:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:06:54.452486+0000 mgr.y (mgr.24481) 655 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:06:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:54.716538+0000 mgr.y (mgr.24481) 656 : cluster [DBG] pgmap v903: 128 pgs: 128 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-12T17:06:56.858 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 32 16 57955 57939 113.123 79.0625 0.00203098 0.00882946 2024-02-12T17:06:57.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:56.098173+0000 osd.0 (osd.0) 199 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:57.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:56.098228+0000 osd.0 (osd.0) 200 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:57.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:56.359594+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e464: 8 total, 8 up, 5 in 2024-02-12T17:06:57.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:56.716919+0000 mgr.y (mgr.24481) 657 : cluster [DBG] pgmap v905: 128 pgs: 128 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-02-12T17:06:57.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:56.722128+0000 osd.5 (osd.5) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:57.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:56.722188+0000 osd.5 (osd.5) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:57.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:56.098173+0000 osd.0 (osd.0) 199 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:57.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:56.098228+0000 osd.0 (osd.0) 200 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:57.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:56.359594+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e464: 8 total, 8 up, 5 in 2024-02-12T17:06:57.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:56.716919+0000 mgr.y (mgr.24481) 657 : cluster [DBG] pgmap v905: 128 pgs: 128 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-02-12T17:06:57.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:56.722128+0000 osd.5 (osd.5) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:57.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:56.722188+0000 osd.5 (osd.5) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:56.098173+0000 osd.0 (osd.0) 199 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:56.098228+0000 osd.0 (osd.0) 200 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:56.359594+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e464: 8 total, 8 up, 5 in 2024-02-12T17:06:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:56.716919+0000 mgr.y (mgr.24481) 657 : cluster [DBG] pgmap v905: 128 pgs: 128 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-02-12T17:06:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:56.722128+0000 osd.5 (osd.5) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:06:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:56.722188+0000 osd.5 (osd.5) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:06:57.858 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 33 16 59204 59188 112.06 78.0625 0.00769738 0.00891844 2024-02-12T17:06:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:57.374825+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e465: 8 total, 8 up, 5 in 2024-02-12T17:06:58.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:57.374825+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e465: 8 total, 8 up, 5 in 2024-02-12T17:06:58.756 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:57.374825+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e465: 8 total, 8 up, 5 in 2024-02-12T17:06:58.859 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 34 16 60427 60411 111.013 76.4375 0.00571891 0.00900296 2024-02-12T17:06:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:06:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:06:58.717927+0000 mgr.y (mgr.24481) 658 : cluster [DBG] pgmap v907: 128 pgs: 128 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-12T17:06:59.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:06:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:06:58.717927+0000 mgr.y (mgr.24481) 658 : cluster [DBG] pgmap v907: 128 pgs: 128 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-12T17:06:59.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:06:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:06:58.717927+0000 mgr.y (mgr.24481) 658 : cluster [DBG] pgmap v907: 128 pgs: 128 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 853 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-12T17:06:59.858 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 35 15 61673 61658 110.068 77.9375 0.010728 0.00908081 2024-02-12T17:07:00.859 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 36 16 62915 62899 109.164 77.5625 0.0122237 0.00915473 2024-02-12T17:07:01.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:07:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:00] "GET /metrics HTTP/1.1" 200 176504 "" "Prometheus/2.33.4" 2024-02-12T17:07:01.859 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 37 16 64102 64086 108.218 74.1875 0.00484639 0.00923566 2024-02-12T17:07:02.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:00.718993+0000 mgr.y (mgr.24481) 659 : cluster [DBG] pgmap v908: 128 pgs: 128 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 763 B/s rd, 61 MiB/s wr, 970 op/s 2024-02-12T17:07:02.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:01.363397+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e466: 8 total, 8 up, 5 in 2024-02-12T17:07:02.225 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:00.718993+0000 mgr.y (mgr.24481) 659 : cluster [DBG] pgmap v908: 128 pgs: 128 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 763 B/s rd, 61 MiB/s wr, 970 op/s 2024-02-12T17:07:02.225 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:01.363397+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e466: 8 total, 8 up, 5 in 2024-02-12T17:07:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:00.718993+0000 mgr.y (mgr.24481) 659 : cluster [DBG] pgmap v908: 128 pgs: 128 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 763 B/s rd, 61 MiB/s wr, 970 op/s 2024-02-12T17:07:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:01.363397+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e466: 8 total, 8 up, 5 in 2024-02-12T17:07:02.859 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 38 16 65359 65343 107.438 78.5625 0.00893887 0.00930286 2024-02-12T17:07:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:07:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:07:03.008 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:03.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:03.030 INFO:tasks.thrashosds.thrasher:j is {'epoch': 462, 'fsid': 'cf6ad642-c9c5-11ee-95b9-87774f69a715', 'created': '2024-02-12T16:43:50.282468+0000', 'modified': '2024-02-12T17:06:51.352769+0000', 'last_up_change': '2024-02-12T16:49:30.933575+0000', 'last_in_change': '2024-02-12T17:04:38.823636+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T16:47:33.056945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-12T16:49:37.647343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-12T16:49:38.076200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-12T16:49:39.589565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-12T16:49:41.623856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.10', 'ready_epoch': 429, 'last_epoch_started': 428, 'last_epoch_clean': 428, 'source_version': "0'0", 'target_version': "380'16"}, 'last_change': '433', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '429', 'last_force_op_resend_preluminous': '431', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-12T16:49:43.693806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '6.16', 'ready_epoch': 396, 'last_epoch_started': 395, 'last_epoch_clean': 395, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '396', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '396', 'last_force_op_resend_preluminous': '394', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-02-12T17:06:16.954458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '455', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '5dd172eb-ee9d-4700-9b80-60f91c5f2cb7', 'up': 1, 'in': 1, 'weight': 0.2548370361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 451, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 2545411201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6802', 'nonce': 2545411201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6804', 'nonce': 2545411201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 2545411201}]}, 'public_addr': '172.21.15.33:6801/2545411201', 'cluster_addr': '172.21.15.33:6802/2545411201', 'heartbeat_back_addr': '172.21.15.33:6804/2545411201', 'heartbeat_front_addr': '172.21.15.33:6803/2545411201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71231bc4-1716-431d-bd88-f6a3cad0bd24', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1723231708}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6806', 'nonce': 1723231708}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6808', 'nonce': 1723231708}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1723231708}]}, 'public_addr': '172.21.15.33:6805/1723231708', 'cluster_addr': '172.21.15.33:6806/1723231708', 'heartbeat_back_addr': '172.21.15.33:6808/1723231708', 'heartbeat_front_addr': '172.21.15.33:6807/1723231708', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '8a9c7127-d17c-4c44-b666-4211006de47c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 440, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3126580659}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6810', 'nonce': 3126580659}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6812', 'nonce': 3126580659}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3126580659}]}, 'public_addr': '172.21.15.33:6809/3126580659', 'cluster_addr': '172.21.15.33:6810/3126580659', 'heartbeat_back_addr': '172.21.15.33:6812/3126580659', 'heartbeat_front_addr': '172.21.15.33:6811/3126580659', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3cc40671-c8f7-412c-a0f1-987f48a016b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 451, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3941314957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6814', 'nonce': 3941314957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6816', 'nonce': 3941314957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3941314957}]}, 'public_addr': '172.21.15.33:6813/3941314957', 'cluster_addr': '172.21.15.33:6814/3941314957', 'heartbeat_back_addr': '172.21.15.33:6816/3941314957', 'heartbeat_front_addr': '172.21.15.33:6815/3941314957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0498129c-861c-4f7e-b8ba-219949da3130', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.54083251953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 440, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6800', 'nonce': 544126779}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 544126779}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 544126779}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 544126779}]}, 'public_addr': '172.21.15.94:6800/544126779', 'cluster_addr': '172.21.15.94:6801/544126779', 'heartbeat_back_addr': '172.21.15.94:6803/544126779', 'heartbeat_front_addr': '172.21.15.94:6802/544126779', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '05718a1e-1164-4b54-a493-23c1cd3adb50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 451, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 3229884005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 3229884005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 3229884005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 3229884005}]}, 'public_addr': '172.21.15.94:6804/3229884005', 'cluster_addr': '172.21.15.94:6805/3229884005', 'heartbeat_back_addr': '172.21.15.94:6807/3229884005', 'heartbeat_front_addr': '172.21.15.94:6806/3229884005', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dc48640f-8f27-405c-a9a4-cf1e3e6c8821', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 507305558}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 507305558}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 507305558}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 507305558}]}, 'public_addr': '172.21.15.94:6808/507305558', 'cluster_addr': '172.21.15.94:6809/507305558', 'heartbeat_back_addr': '172.21.15.94:6811/507305558', 'heartbeat_front_addr': '172.21.15.94:6810/507305558', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7d55aa3f-06b6-4fc3-97fd-62b535f78b66', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 451, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3401977873}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 3401977873}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 3401977873}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 3401977873}]}, 'public_addr': '172.21.15.94:6812/3401977873', 'cluster_addr': '172.21.15.94:6813/3401977873', 'heartbeat_back_addr': '172.21.15.94:6815/3401977873', 'heartbeat_front_addr': '172.21.15.94:6814/3401977873', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:05:28.101520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:05:51.331519+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:06:48.948626+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:06:16.169738+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:06:25.564600+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:05:39.766966+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T17:06:23.566041+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:06:22.248730+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.33:6800/2829464018': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/1835128904': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2277321750': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2116602769': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:6800/3967118778': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/2298361247': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/218227980': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/2070582473': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/4291575956': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:6800/3157562110': '2024-02-13T16:44:22.727241+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T17:07:03.030 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-02-12T17:07:03.115 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:03.182 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:03.218 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:03.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:03.329 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:03.378 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:03.405 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:03.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:03.511 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:07:03.568 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:03.659 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:07:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:03.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:07:03.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T17:07:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:02.496628+0000 mon.a (mon.0) 1631 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:02.719402+0000 mgr.y (mgr.24481) 660 : cluster [DBG] pgmap v910: 128 pgs: 128 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 804 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:03.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:03.772 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:02.496628+0000 mon.a (mon.0) 1631 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:03.772 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:02.719402+0000 mgr.y (mgr.24481) 660 : cluster [DBG] pgmap v910: 128 pgs: 128 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 804 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:03.774 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:02.496628+0000 mon.a (mon.0) 1631 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:03.774 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:02.719402+0000 mgr.y (mgr.24481) 660 : cluster [DBG] pgmap v910: 128 pgs: 128 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 804 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:03.800 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:03.825 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:03.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:03.861 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 39 16 66663 66647 106.768 81.5 0.0102509 0.00936161 2024-02-12T17:07:03.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:03.960 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:03.988 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:03.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:04.095 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:04.147 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:04.155 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:04.203 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:04.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:04.256 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:07:04.309 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:04.363 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:04.436 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:04.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:04.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:04.576 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:04.601 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:04.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:04.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:07:04.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:07:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:07:04.767 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:04.861 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:07:04.860809+0000 min lat: 0.00118871 max lat: 0.105219 avg lat: 0.00944366 2024-02-12T17:07:04.861 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:07:04.861 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 40 16 67780 67764 105.844 69.8125 0.00786691 0.00944366 2024-02-12T17:07:05.133 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:07:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:05.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:07:05.147 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:07:05.238 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:05.321 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:05.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:05.427 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:05.480 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:05.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:05.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:05.590 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:05.656 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:05.707 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:05.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:05.786 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:05.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:07:05.857 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:05.862 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 41 16 69009 68993 105.133 76.8125 0.00785237 0.00950646 2024-02-12T17:07:06.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:04.462498+0000 mgr.y (mgr.24481) 661 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:06.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:04.726124+0000 mgr.y (mgr.24481) 662 : cluster [DBG] pgmap v911: 128 pgs: 128 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-12T17:07:06.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:04.462498+0000 mgr.y (mgr.24481) 661 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:06.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:04.726124+0000 mgr.y (mgr.24481) 662 : cluster [DBG] pgmap v911: 128 pgs: 128 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-12T17:07:06.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:04.462498+0000 mgr.y (mgr.24481) 661 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:06.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:04.726124+0000 mgr.y (mgr.24481) 662 : cluster [DBG] pgmap v911: 128 pgs: 128 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-12T17:07:06.605 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:07:06.605 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:07:06.863 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 42 16 70246 70230 104.471 77.3125 0.00840192 0.00956743 2024-02-12T17:07:06.988 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:07:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:07.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:07:07.104 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:07.158 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:07.519 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:07.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:07.630 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:07.663 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:06.727883+0000 mgr.y (mgr.24481) 663 : cluster [DBG] pgmap v912: 128 pgs: 128 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-02-12T17:07:07.666 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:07.695 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:07.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:06.727883+0000 mgr.y (mgr.24481) 663 : cluster [DBG] pgmap v912: 128 pgs: 128 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-02-12T17:07:07.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:06.727883+0000 mgr.y (mgr.24481) 663 : cluster [DBG] pgmap v912: 128 pgs: 128 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-02-12T17:07:07.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:07.822 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:07.859 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:07.863 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 43 16 71541 71525 103.923 80.9375 0.0156197 0.00961791 2024-02-12T17:07:08.019 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:08.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:08.031 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:07:08.031 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:07:08.032 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-12T17:07:08.032 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T17:07:08.032 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T17:07:08.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-12T17:07:08.127 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:08.188 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:08.451 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:07:08.597 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T17:07:08.869 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 44 16 72762 72746 103.293 76.3125 0.0109169 0.00967623 2024-02-12T17:07:09.829 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:07:09.864 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 45 16 73913 73897 102.596 71.9375 0.0229057 0.00974192 2024-02-12T17:07:10.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:08.728545+0000 mgr.y (mgr.24481) 664 : cluster [DBG] pgmap v913: 128 pgs: 128 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1022 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-02-12T17:07:10.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:09.476924+0000 mon.c (mon.2) 243 : audit [INF] from='client.? v1:172.21.15.33:0/720757872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:10.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:09.477307+0000 mon.a (mon.0) 1632 : audit [INF] from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:10.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:08.728545+0000 mgr.y (mgr.24481) 664 : cluster [DBG] pgmap v913: 128 pgs: 128 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1022 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-02-12T17:07:10.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:09.476924+0000 mon.c (mon.2) 243 : audit [INF] from='client.? v1:172.21.15.33:0/720757872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:10.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:09.477307+0000 mon.a (mon.0) 1632 : audit [INF] from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:10.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:08.728545+0000 mgr.y (mgr.24481) 664 : cluster [DBG] pgmap v913: 128 pgs: 128 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1022 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-02-12T17:07:10.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:09.476924+0000 mon.c (mon.2) 243 : audit [INF] from='client.? v1:172.21.15.33:0/720757872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:10.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:09.477307+0000 mon.a (mon.0) 1632 : audit [INF] from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:10.864 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 46 16 75197 75181 102.11 80.25 0.023761 0.00978788 2024-02-12T17:07:10.950 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:09.788969+0000 mon.a (mon.0) 1633 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:10.950 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:09.799250+0000 mon.a (mon.0) 1634 : audit [INF] from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:07:10.950 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:09.799381+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e467: 8 total, 8 up, 5 in 2024-02-12T17:07:10.950 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:07:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:10] "GET /metrics HTTP/1.1" 200 176496 "" "Prometheus/2.33.4" 2024-02-12T17:07:10.950 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:09.788969+0000 mon.a (mon.0) 1633 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:10.951 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:09.799250+0000 mon.a (mon.0) 1634 : audit [INF] from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:07:10.951 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:09.799381+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e467: 8 total, 8 up, 5 in 2024-02-12T17:07:10.961 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:10.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:11.067 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:11.133 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:11.170 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:11.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:09.788969+0000 mon.a (mon.0) 1633 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:09.799250+0000 mon.a (mon.0) 1634 : audit [INF] from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:07:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:09.799381+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e467: 8 total, 8 up, 5 in 2024-02-12T17:07:11.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:11.345 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:11.361 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:07:11.393 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:07:11.426 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:07:11.462 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:07:11.490 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:07:11.524 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:07:11.548 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:07:11.596 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:07:11.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:07:11.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:07:11.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:07:11.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:07:11.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:07:11.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:07:11.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:07:11.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:07:11.711 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:11.864 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 47 16 76281 76265 101.379 67.75 0.0106114 0.00985963 2024-02-12T17:07:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:10.728966+0000 mgr.y (mgr.24481) 665 : cluster [DBG] pgmap v915: 128 pgs: 128 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:07:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:11.330160+0000 mon.b (mon.1) 86 : audit [DBG] from='client.? v1:172.21.15.33:0/3803411031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:11.372551+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e468: 8 total, 8 up, 5 in 2024-02-12T17:07:12.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:10.728966+0000 mgr.y (mgr.24481) 665 : cluster [DBG] pgmap v915: 128 pgs: 128 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:07:12.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:11.330160+0000 mon.b (mon.1) 86 : audit [DBG] from='client.? v1:172.21.15.33:0/3803411031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:12.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:11.372551+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e468: 8 total, 8 up, 5 in 2024-02-12T17:07:12.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:10.728966+0000 mgr.y (mgr.24481) 665 : cluster [DBG] pgmap v915: 128 pgs: 128 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-12T17:07:12.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:11.330160+0000 mon.b (mon.1) 86 : audit [DBG] from='client.? v1:172.21.15.33:0/3803411031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:12.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:11.372551+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e468: 8 total, 8 up, 5 in 2024-02-12T17:07:12.712 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:07:12.866 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 48 16 77408 77392 100.732 70.4375 0.0117797 0.00992164 2024-02-12T17:07:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:07:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:07:13.166 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:13.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:13.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:13.275 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:13.341 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:13.514 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:13.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:13.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:13.869 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 49 16 78624 78608 100.226 76 0.0155973 0.0099717 2024-02-12T17:07:13.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:12.729300+0000 mgr.y (mgr.24481) 666 : cluster [DBG] pgmap v917: 128 pgs: 128 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 639 B/s rd, 48 MiB/s wr, 769 op/s 2024-02-12T17:07:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:14.216 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:07:14.216 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:07:14.216 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:07:14.216 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:07:14.216 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:07:14.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:12.729300+0000 mgr.y (mgr.24481) 666 : cluster [DBG] pgmap v917: 128 pgs: 128 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 639 B/s rd, 48 MiB/s wr, 769 op/s 2024-02-12T17:07:14.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:12.729300+0000 mgr.y (mgr.24481) 666 : cluster [DBG] pgmap v917: 128 pgs: 128 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 639 B/s rd, 48 MiB/s wr, 769 op/s 2024-02-12T17:07:14.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:07:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:07:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:14.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:14.868 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 50 16 79680 79664 99.5392 66 0.0189499 0.0100422 2024-02-12T17:07:14.897 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:07:14.926 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:07:15.022 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:15.183 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:07:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:15.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:07:15.288 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:15.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:15.372 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:15.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:15.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:15.438 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:07:15.438 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "" 2024-02-12T17:07:15.438 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:07:15.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:15.521 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:15.754 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T17:07:15.868 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 51 16 80873 80857 99.0495 74.5625 0.0164848 0.0100906 2024-02-12T17:07:16.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:14.472489+0000 mgr.y (mgr.24481) 667 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:16.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:14.730080+0000 mgr.y (mgr.24481) 668 : cluster [DBG] pgmap v918: 128 pgs: 128 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-02-12T17:07:16.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:14.472489+0000 mgr.y (mgr.24481) 667 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:16.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:14.730080+0000 mgr.y (mgr.24481) 668 : cluster [DBG] pgmap v918: 128 pgs: 128 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-02-12T17:07:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:14.472489+0000 mgr.y (mgr.24481) 667 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:14.730080+0000 mgr.y (mgr.24481) 668 : cluster [DBG] pgmap v918: 128 pgs: 128 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-02-12T17:07:16.277 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:16.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:16.344 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:16.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:16.424 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:16.536 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:16.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:16.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:16.715 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:16.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:16.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:16.858 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:16.869 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 52 15 81935 81920 98.4195 66.4375 0.0119183 0.0101545 2024-02-12T17:07:16.881 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:16.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:16.985 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:17.059 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:17.066 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:17.219 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:17.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:17.326 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:17.408 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:17.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:16.730686+0000 mgr.y (mgr.24481) 669 : cluster [DBG] pgmap v919: 128 pgs: 128 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:17.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:16.730686+0000 mgr.y (mgr.24481) 669 : cluster [DBG] pgmap v919: 128 pgs: 128 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:17.555 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:17.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:17.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:17.718 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:16.730686+0000 mgr.y (mgr.24481) 669 : cluster [DBG] pgmap v919: 128 pgs: 128 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:17.870 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 53 16 83070 83054 97.8999 70.875 0.00674003 0.0102102 2024-02-12T17:07:18.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:17.421039+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.33:0/894306010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:07:18.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:17.421493+0000 mon.a (mon.0) 1637 : audit [INF] from='client.25654 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:07:18.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:17.544350+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:18.414 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:17.544421+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:18.418 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:07:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:17.421039+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.33:0/894306010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:07:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:17.421493+0000 mon.a (mon.0) 1637 : audit [INF] from='client.25654 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:07:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:17.544350+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:17.544421+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:18.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:17.421039+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.33:0/894306010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:07:18.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:17.421493+0000 mon.a (mon.0) 1637 : audit [INF] from='client.25654 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:07:18.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:17.544350+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:18.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:17.544421+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:18.870 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 54 16 84308 84292 97.5198 77.375 0.0101 0.0102493 2024-02-12T17:07:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:18.391877+0000 mon.a (mon.0) 1638 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:18.399185+0000 mon.a (mon.0) 1639 : audit [INF] from='client.25654 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:07:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:18.399248+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e469: 8 total, 8 up, 5 in 2024-02-12T17:07:19.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:18.731272+0000 mgr.y (mgr.24481) 670 : cluster [DBG] pgmap v921: 128 pgs: 128 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-12T17:07:19.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:18.750533+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.b scrub starts 2024-02-12T17:07:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:18.391877+0000 mon.a (mon.0) 1638 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:18.399185+0000 mon.a (mon.0) 1639 : audit [INF] from='client.25654 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:07:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:18.399248+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e469: 8 total, 8 up, 5 in 2024-02-12T17:07:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:18.731272+0000 mgr.y (mgr.24481) 670 : cluster [DBG] pgmap v921: 128 pgs: 128 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-12T17:07:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:18.750533+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.b scrub starts 2024-02-12T17:07:19.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:18.391877+0000 mon.a (mon.0) 1638 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:19.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:18.399185+0000 mon.a (mon.0) 1639 : audit [INF] from='client.25654 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:07:19.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:18.399248+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e469: 8 total, 8 up, 5 in 2024-02-12T17:07:19.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:18.731272+0000 mgr.y (mgr.24481) 670 : cluster [DBG] pgmap v921: 128 pgs: 128 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-12T17:07:19.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:18.750533+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.b scrub starts 2024-02-12T17:07:19.871 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 55 16 85552 85536 97.1575 77.75 0.0169251 0.0102876 2024-02-12T17:07:20.208 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:07:20.208 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:07:20.209 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:07:20.210 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:07:20.210 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:07:20.210 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:07:20.210 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:07:20.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:18.845545+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.b scrub ok 2024-02-12T17:07:20.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:19.074275+0000 osd.7 (osd.7) 224 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:07:20.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:19.079150+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:07:20.430 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:19.810725+0000 mon.b (mon.1) 88 : audit [DBG] from='client.? v1:172.21.15.33:0/2748392782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:20.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:18.845545+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.b scrub ok 2024-02-12T17:07:20.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:19.074275+0000 osd.7 (osd.7) 224 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:07:20.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:19.079150+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:07:20.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:19.810725+0000 mon.b (mon.1) 88 : audit [DBG] from='client.? v1:172.21.15.33:0/2748392782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:20.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:18.845545+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.b scrub ok 2024-02-12T17:07:20.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:19.074275+0000 osd.7 (osd.7) 224 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:07:20.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:19.079150+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:07:20.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:19.810725+0000 mon.b (mon.1) 88 : audit [DBG] from='client.? v1:172.21.15.33:0/2748392782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:20.872 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 56 16 86767 86751 96.7786 75.9375 0.0201218 0.0103281 2024-02-12T17:07:21.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:07:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:20] "GET /metrics HTTP/1.1" 200 176503 "" "Prometheus/2.33.4" 2024-02-12T17:07:21.313 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:21.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:21.417 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:21.458 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:21.486 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:21.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:20.552924+0000 osd.0 (osd.0) 201 : cluster [DBG] 5.b scrub starts 2024-02-12T17:07:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:20.649283+0000 mon.c (mon.2) 244 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:07:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:20.731832+0000 mgr.y (mgr.24481) 671 : cluster [DBG] pgmap v922: 128 pgs: 128 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-12T17:07:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:20.761783+0000 osd.3 (osd.3) 151 : cluster [DBG] 6.c scrub starts 2024-02-12T17:07:21.565 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:20.768680+0000 osd.3 (osd.3) 152 : cluster [DBG] 6.c scrub ok 2024-02-12T17:07:21.566 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:20.822848+0000 osd.5 (osd.5) 225 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:07:21.566 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:20.830121+0000 osd.5 (osd.5) 226 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:07:21.566 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:21.375247+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e470: 8 total, 8 up, 5 in 2024-02-12T17:07:21.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:20.552924+0000 osd.0 (osd.0) 201 : cluster [DBG] 5.b scrub starts 2024-02-12T17:07:21.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:20.649283+0000 mon.c (mon.2) 244 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:07:21.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:20.731832+0000 mgr.y (mgr.24481) 671 : cluster [DBG] pgmap v922: 128 pgs: 128 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-12T17:07:21.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:20.761783+0000 osd.3 (osd.3) 151 : cluster [DBG] 6.c scrub starts 2024-02-12T17:07:21.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:20.768680+0000 osd.3 (osd.3) 152 : cluster [DBG] 6.c scrub ok 2024-02-12T17:07:21.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:20.822848+0000 osd.5 (osd.5) 225 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:07:21.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:20.830121+0000 osd.5 (osd.5) 226 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:07:21.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:21.375247+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e470: 8 total, 8 up, 5 in 2024-02-12T17:07:21.591 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:21.627 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:21.656 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:21.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:21.729 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:07:21.729 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:07:21.730 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-02-12T17:07:21.730 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:07:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:20.552924+0000 osd.0 (osd.0) 201 : cluster [DBG] 5.b scrub starts 2024-02-12T17:07:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:20.649283+0000 mon.c (mon.2) 244 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:07:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:20.731832+0000 mgr.y (mgr.24481) 671 : cluster [DBG] pgmap v922: 128 pgs: 128 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-12T17:07:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:20.761783+0000 osd.3 (osd.3) 151 : cluster [DBG] 6.c scrub starts 2024-02-12T17:07:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:20.768680+0000 osd.3 (osd.3) 152 : cluster [DBG] 6.c scrub ok 2024-02-12T17:07:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:20.822848+0000 osd.5 (osd.5) 225 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:07:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:20.830121+0000 osd.5 (osd.5) 226 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:07:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:21.375247+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e470: 8 total, 8 up, 5 in 2024-02-12T17:07:21.762 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:21.826 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:21.870 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:21.871 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 57 16 87913 87897 96.3373 71.625 0.0240558 0.010374 2024-02-12T17:07:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:21.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:21.923 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:07:21.923 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:07:21.979 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:22.014 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:22.043 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:22.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:22.149 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:22.214 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:22.365 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:22.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:22.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:20.629461+0000 osd.0 (osd.0) 202 : cluster [DBG] 5.b scrub ok 2024-02-12T17:07:22.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:21.462633+0000 mon.c (mon.2) 245 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:07:22.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:21.473449+0000 mon.c (mon.2) 246 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:07:22.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:21.487021+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:22.467 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:20.629461+0000 osd.0 (osd.0) 202 : cluster [DBG] 5.b scrub ok 2024-02-12T17:07:22.468 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:21.462633+0000 mon.c (mon.2) 245 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:07:22.468 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:21.473449+0000 mon.c (mon.2) 246 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:07:22.468 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:21.487021+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:22.471 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:22.509 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:22.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:22.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:22.682 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:22.712 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:22.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:22.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:07:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:07:22.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:20.629461+0000 osd.0 (osd.0) 202 : cluster [DBG] 5.b scrub ok 2024-02-12T17:07:22.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:21.462633+0000 mon.c (mon.2) 245 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:07:22.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:21.473449+0000 mon.c (mon.2) 246 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:07:22.763 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:21.487021+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:22.816 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:22.872 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 58 16 89157 89141 96.0169 77.75 0.0101735 0.0104102 2024-02-12T17:07:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:22.972 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:07:23.142 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T17:07:23.152 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:23.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:23.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:23.309 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:23.311 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:23.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:23.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:23.444 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:23.472 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:23.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:23.577 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:23.611 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:22.557976+0000 osd.0 (osd.0) 203 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:07:23.611 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:22.732521+0000 mgr.y (mgr.24481) 672 : cluster [DBG] pgmap v924: 128 pgs: 128 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 44 MiB/s wr, 709 op/s 2024-02-12T17:07:23.611 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:22.782318+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:07:23.611 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:22.791004+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:07:23.614 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:23.646 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:23.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:23.753 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:22.557976+0000 osd.0 (osd.0) 203 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:07:23.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:22.732521+0000 mgr.y (mgr.24481) 672 : cluster [DBG] pgmap v924: 128 pgs: 128 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 44 MiB/s wr, 709 op/s 2024-02-12T17:07:23.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:22.782318+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:07:23.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:22.791004+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:07:23.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:22.557976+0000 osd.0 (osd.0) 203 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:07:23.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:22.732521+0000 mgr.y (mgr.24481) 672 : cluster [DBG] pgmap v924: 128 pgs: 128 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 44 MiB/s wr, 709 op/s 2024-02-12T17:07:23.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:22.782318+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:07:23.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:22.791004+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:07:23.789 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:23.814 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:23.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:23.871 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 59 16 90223 90207 95.5188 66.625 0.0171505 0.0104647 2024-02-12T17:07:23.920 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:23.989 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:24.390 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:07:24.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:22.571421+0000 osd.0 (osd.0) 204 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:07:24.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:07:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:07:24.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:22.571421+0000 osd.0 (osd.0) 204 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:07:24.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:22.571421+0000 osd.0 (osd.0) 204 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:07:24.872 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:07:24.871371+0000 min lat: 0.00118871 max lat: 0.167728 avg lat: 0.0105076 2024-02-12T17:07:24.872 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:07:24.872 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 60 16 91372 91356 95.1237 71.8125 0.0147645 0.0105076 2024-02-12T17:07:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:23.602370+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:07:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:23.791862+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:07:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:24.482489+0000 mgr.y (mgr.24481) 673 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:25.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:24.733116+0000 mgr.y (mgr.24481) 674 : cluster [DBG] pgmap v925: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-12T17:07:25.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:23.602370+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:07:25.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:23.791862+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:07:25.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:24.482489+0000 mgr.y (mgr.24481) 673 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:25.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:24.733116+0000 mgr.y (mgr.24481) 674 : cluster [DBG] pgmap v925: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-12T17:07:25.873 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 61 16 92518 92502 94.7368 71.625 0.0158798 0.0105512 2024-02-12T17:07:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:23.602370+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:07:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:23.791862+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:07:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:24.482489+0000 mgr.y (mgr.24481) 673 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:24.733116+0000 mgr.y (mgr.24481) 674 : cluster [DBG] pgmap v925: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-12T17:07:26.371 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:26.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:26.478 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:07:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:26.572 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:07:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:26.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:07:26.677 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:26.728 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:24.615936+0000 osd.0 (osd.0) 207 : cluster [DBG] 10.5 scrub starts 2024-02-12T17:07:26.728 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:24.615936+0000 osd.0 (osd.0) 207 : cluster [DBG] 10.5 scrub starts 2024-02-12T17:07:26.735 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:26.761 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:26.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:26.869 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:26.874 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 62 16 93588 93572 94.2874 66.875 0.00906383 0.0106009 2024-02-12T17:07:26.926 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:26.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:24.615936+0000 osd.0 (osd.0) 207 : cluster [DBG] 10.5 scrub starts 2024-02-12T17:07:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:27.138 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:27.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:27.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:27.281 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:27.304 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:27.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:27.408 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:27.443 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:27.472 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:27.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:27.523 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:26.733567+0000 mgr.y (mgr.24481) 675 : cluster [DBG] pgmap v926: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-12T17:07:27.578 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:27.635 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:26.733567+0000 mgr.y (mgr.24481) 675 : cluster [DBG] pgmap v926: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-12T17:07:27.753 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:27.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:27.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:26.733567+0000 mgr.y (mgr.24481) 675 : cluster [DBG] pgmap v926: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-12T17:07:27.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:27.873 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 63 16 94743 94727 93.9366 72.1875 0.00937667 0.0106408 2024-02-12T17:07:27.900 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:27.909 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:27.918 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:27.922 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:27.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:28.026 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:28.054 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:28.080 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:28.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:28.185 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:28.244 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:28.314 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:28.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:28.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:07:28.458 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:28.543 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:07:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:28.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:07:28.650 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:28.701 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:28.838 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:28.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:28.844 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:27.706382+0000 osd.0 (osd.0) 208 : cluster [DBG] 10.5 scrub ok 2024-02-12T17:07:28.848 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:27.706382+0000 osd.0 (osd.0) 208 : cluster [DBG] 10.5 scrub ok 2024-02-12T17:07:28.873 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 64 16 95791 95775 93.4923 65.5 0.00867226 0.0106915 2024-02-12T17:07:28.945 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:28.975 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:27.706382+0000 osd.0 (osd.0) 208 : cluster [DBG] 10.5 scrub ok 2024-02-12T17:07:28.978 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:29.000 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:29.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:29.104 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:29.141 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:29.167 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:29.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:29.272 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:29.307 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:07:29.320 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:07:29.489 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:29.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:28.163577+0000 osd.7 (osd.7) 226 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:07:29.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:28.167495+0000 osd.7 (osd.7) 227 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:07:29.647 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:28.734828+0000 mgr.y (mgr.24481) 676 : cluster [DBG] pgmap v927: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-12T17:07:29.873 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 65 16 96942 96926 93.1607 71.9375 0.0117887 0.0107295 2024-02-12T17:07:29.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:28.163577+0000 osd.7 (osd.7) 226 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:07:29.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:28.167495+0000 osd.7 (osd.7) 227 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:07:29.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:28.734828+0000 mgr.y (mgr.24481) 676 : cluster [DBG] pgmap v927: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-12T17:07:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:28.163577+0000 osd.7 (osd.7) 226 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:07:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:28.167495+0000 osd.7 (osd.7) 227 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:07:30.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:28.734828+0000 mgr.y (mgr.24481) 676 : cluster [DBG] pgmap v927: 128 pgs: 1 active+clean+scrubbing, 127 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-12T17:07:30.873 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 66 16 98163 98147 92.9055 76.3125 0.00779089 0.0107588 2024-02-12T17:07:30.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:29.159607+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1d scrub starts 2024-02-12T17:07:30.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:29.165794+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub ok 2024-02-12T17:07:30.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:29.283710+0000 mgr.y (mgr.24481) 677 : audit [DBG] from='client.25666 v1:172.21.15.33:0/1850197321' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:07:31.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:29.159607+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1d scrub starts 2024-02-12T17:07:31.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:29.165794+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub ok 2024-02-12T17:07:31.062 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:29.283710+0000 mgr.y (mgr.24481) 677 : audit [DBG] from='client.25666 v1:172.21.15.33:0/1850197321' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:07:31.062 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:07:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:30] "GET /metrics HTTP/1.1" 200 176503 "" "Prometheus/2.33.4" 2024-02-12T17:07:31.062 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:29.159607+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1d scrub starts 2024-02-12T17:07:31.062 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:29.165794+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub ok 2024-02-12T17:07:31.062 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:29.283710+0000 mgr.y (mgr.24481) 677 : audit [DBG] from='client.25666 v1:172.21.15.33:0/1850197321' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:07:31.873 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 67 16 99389 99373 92.6624 76.625 0.00277427 0.0107836 2024-02-12T17:07:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:30.736800+0000 mgr.y (mgr.24481) 678 : cluster [DBG] pgmap v928: 128 pgs: 128 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:07:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:30.736800+0000 mgr.y (mgr.24481) 678 : cluster [DBG] pgmap v928: 128 pgs: 128 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:07:32.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:30.736800+0000 mgr.y (mgr.24481) 678 : cluster [DBG] pgmap v928: 128 pgs: 128 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:07:32.802 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:32.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:32.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:32.876 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 68 16 100664 100648 92.4686 79.6875 0.00919928 0.0108103 2024-02-12T17:07:32.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:32.943 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:07:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:07:32.945 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:32.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:32.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:33.079 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:33.117 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:33.143 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:33.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:33.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:33.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:33.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:33.419 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:33.814 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:31.901637+0000 osd.3 (osd.3) 153 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:07:33.814 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:31.904777+0000 osd.3 (osd.3) 154 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:07:33.814 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:32.195109+0000 osd.7 (osd.7) 230 : cluster [DBG] 2.f scrub starts 2024-02-12T17:07:33.876 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 69 15 101923 101908 92.2698 78.75 0.00704194 0.0108339 2024-02-12T17:07:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:31.901637+0000 osd.3 (osd.3) 153 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:07:34.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:31.904777+0000 osd.3 (osd.3) 154 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:07:34.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:32.195109+0000 osd.7 (osd.7) 230 : cluster [DBG] 2.f scrub starts 2024-02-12T17:07:34.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:32.199461+0000 osd.7 (osd.7) 231 : cluster [DBG] 2.f scrub ok 2024-02-12T17:07:34.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:32.737195+0000 mgr.y (mgr.24481) 679 : cluster [DBG] pgmap v929: 128 pgs: 128 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 901 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:34.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:31.901637+0000 osd.3 (osd.3) 153 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:07:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:31.904777+0000 osd.3 (osd.3) 154 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:07:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:32.195109+0000 osd.7 (osd.7) 230 : cluster [DBG] 2.f scrub starts 2024-02-12T17:07:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:32.199461+0000 osd.7 (osd.7) 231 : cluster [DBG] 2.f scrub ok 2024-02-12T17:07:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:32.737195+0000 mgr.y (mgr.24481) 679 : cluster [DBG] pgmap v929: 128 pgs: 128 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 901 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:32.199461+0000 osd.7 (osd.7) 231 : cluster [DBG] 2.f scrub ok 2024-02-12T17:07:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:32.737195+0000 mgr.y (mgr.24481) 679 : cluster [DBG] pgmap v929: 128 pgs: 128 active+clean; 5.8 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 901 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:34.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:07:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:07:34.828 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:33.731745+0000 osd.5 (osd.5) 229 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:07:34.828 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:33.734670+0000 osd.5 (osd.5) 230 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:07:34.877 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 70 16 103208 103192 92.0967 80.25 0.0261874 0.0108537 2024-02-12T17:07:34.898 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:35.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:33.731745+0000 osd.5 (osd.5) 229 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:07:35.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:33.734670+0000 osd.5 (osd.5) 230 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:07:35.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:33.731745+0000 osd.5 (osd.5) 229 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:07:35.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:33.734670+0000 osd.5 (osd.5) 230 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:07:35.877 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 71 16 104472 104456 91.9118 79 0.00958653 0.0108762 2024-02-12T17:07:36.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:34.259898+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:36.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:34.259950+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:36.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:34.492506+0000 mgr.y (mgr.24481) 680 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:36.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:34.738668+0000 mgr.y (mgr.24481) 681 : cluster [DBG] pgmap v930: 128 pgs: 128 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-12T17:07:36.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:34.777600+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:07:36.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:34.780670+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:07:36.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:34.828444+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e471: 8 total, 8 up, 5 in 2024-02-12T17:07:36.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:34.259898+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:36.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:34.259950+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:36.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:34.492506+0000 mgr.y (mgr.24481) 680 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:36.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:34.738668+0000 mgr.y (mgr.24481) 681 : cluster [DBG] pgmap v930: 128 pgs: 128 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-12T17:07:36.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:34.777600+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:07:36.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:34.780670+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:07:36.372 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:34.828444+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e471: 8 total, 8 up, 5 in 2024-02-12T17:07:36.373 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:34.259898+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:36.373 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:34.259950+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:36.373 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:34.492506+0000 mgr.y (mgr.24481) 680 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:36.373 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:34.738668+0000 mgr.y (mgr.24481) 681 : cluster [DBG] pgmap v930: 128 pgs: 128 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-12T17:07:36.373 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:34.777600+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:07:36.373 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:34.780670+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:07:36.373 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:34.828444+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e471: 8 total, 8 up, 5 in 2024-02-12T17:07:36.899 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 72 16 105669 105653 91.6517 74.8125 0.0134861 0.0109041 2024-02-12T17:07:37.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:35.793573+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:07:37.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:35.798711+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:07:37.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:35.873861+0000 osd.3 (osd.3) 157 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:07:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:35.884879+0000 osd.3 (osd.3) 158 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:07:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:36.382885+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e472: 8 total, 8 up, 5 in 2024-02-12T17:07:37.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:35.793573+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:07:37.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:35.798711+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:07:37.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:35.873861+0000 osd.3 (osd.3) 157 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:07:37.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:35.884879+0000 osd.3 (osd.3) 158 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:07:37.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:36.382885+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e472: 8 total, 8 up, 5 in 2024-02-12T17:07:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:35.793573+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:07:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:35.798711+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:07:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:35.873861+0000 osd.3 (osd.3) 157 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:07:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:35.884879+0000 osd.3 (osd.3) 158 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:07:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:36.382885+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e472: 8 total, 8 up, 5 in 2024-02-12T17:07:37.898 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 73 16 106933 106917 91.4744 79 0.00952783 0.0109273 2024-02-12T17:07:37.967 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:38.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:38.218 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:36.739080+0000 mgr.y (mgr.24481) 682 : cluster [DBG] pgmap v933: 128 pgs: 128 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:36.833656+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:07:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:36.839338+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:07:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:37.499154+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:07:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:37.504994+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:07:38.251 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:36.739080+0000 mgr.y (mgr.24481) 682 : cluster [DBG] pgmap v933: 128 pgs: 128 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:36.833656+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:07:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:36.839338+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:07:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:37.499154+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:07:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:37.504994+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:07:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:38.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:36.739080+0000 mgr.y (mgr.24481) 682 : cluster [DBG] pgmap v933: 128 pgs: 128 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-12T17:07:38.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:36.833656+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:07:38.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:36.839338+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:07:38.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:37.499154+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:07:38.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:37.504994+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:07:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:38.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:38.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:38.422 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:38.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:38.547 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":927,"stamp":"2024-02-12T17:07:28.734418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6119298724,"num_objects":93565,"num_object_clones":0,"num_object_copies":280695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93565,"num_whiteouts":0,"num_read":1487,"num_read_kb":1239,"num_write":94087,"num_write_kb":5975913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":15798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58483,"ondisk_log_size":58483,"up":384,"acting":384,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":384,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":21831384,"kb_used_data":17936752,"kb_used_omap":0,"kb_used_meta":3894592,"kb_avail":446894376,"statfs":{"total":479975178240,"available":457619841024,"internally_reserved":0,"allocated":18367234048,"data_stored":18362725473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3988062208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":762052608,"num_objects":11628,"num_object_clones":0,"num_object_copies":34884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11628,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11628,"num_write_kb":744192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003562"},"pg_stats":[{"pgid":"2.1f","version":"0'0","reported_seq":158,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408560+0000","last_change":"2024-02-12T17:04:39.916313+0000","last_active":"2024-02-12T17:07:18.408560+0000","last_peered":"2024-02-12T17:07:18.408560+0000","last_clean":"2024-02-12T17:07:18.408560+0000","last_became_active":"2024-02-12T17:04:39.916010+0000","last_became_peered":"2024-02-12T17:04:39.916010+0000","last_unstale":"2024-02-12T17:07:18.408560+0000","last_undegraded":"2024-02-12T17:07:18.408560+0000","last_fullsized":"2024-02-12T17:07:18.408560+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:47.732293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:47.732293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":147,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424534+0000","last_change":"2024-02-12T17:03:52.280899+0000","last_active":"2024-02-12T17:07:18.424534+0000","last_peered":"2024-02-12T17:07:18.424534+0000","last_clean":"2024-02-12T17:07:18.424534+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:07:18.424534+0000","last_undegraded":"2024-02-12T17:07:18.424534+0000","last_fullsized":"2024-02-12T17:07:18.424534+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:01:26.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:26.047527+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":131,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424095+0000","last_change":"2024-02-12T17:03:50.545988+0000","last_active":"2024-02-12T17:07:18.424095+0000","last_peered":"2024-02-12T17:07:18.424095+0000","last_clean":"2024-02-12T17:07:18.424095+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:07:18.424095+0000","last_undegraded":"2024-02-12T17:07:18.424095+0000","last_fullsized":"2024-02-12T17:07:18.424095+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:01:39.176431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":522,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422008+0000","last_change":"2024-02-12T17:04:41.353781+0000","last_active":"2024-02-12T17:07:18.422008+0000","last_peered":"2024-02-12T17:07:18.422008+0000","last_clean":"2024-02-12T17:07:18.422008+0000","last_became_active":"2024-02-12T17:04:39.873280+0000","last_became_peered":"2024-02-12T17:04:39.873280+0000","last_unstale":"2024-02-12T17:07:18.422008+0000","last_undegraded":"2024-02-12T17:07:18.422008+0000","last_fullsized":"2024-02-12T17:07:18.422008+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:01:45.154164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":464,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408843+0000","last_change":"2024-02-12T17:03:50.565273+0000","last_active":"2024-02-12T17:07:18.408843+0000","last_peered":"2024-02-12T17:07:18.408843+0000","last_clean":"2024-02-12T17:07:18.408843+0000","last_became_active":"2024-02-12T17:03:50.562938+0000","last_became_peered":"2024-02-12T17:03:50.562938+0000","last_unstale":"2024-02-12T17:07:18.408843+0000","last_undegraded":"2024-02-12T17:07:18.408843+0000","last_fullsized":"2024-02-12T17:07:18.408843+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:37.813008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:01:37.813008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":157,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423894+0000","last_change":"2024-02-12T17:05:56.388435+0000","last_active":"2024-02-12T17:07:18.423894+0000","last_peered":"2024-02-12T17:07:18.423894+0000","last_clean":"2024-02-12T17:07:18.423894+0000","last_became_active":"2024-02-12T17:04:39.889725+0000","last_became_peered":"2024-02-12T17:04:39.889725+0000","last_unstale":"2024-02-12T17:07:18.423894+0000","last_undegraded":"2024-02-12T17:07:18.423894+0000","last_fullsized":"2024-02-12T17:07:18.423894+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7","last_scrub_stamp":"2024-02-12T17:05:56.388366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.388366+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":438,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408333+0000","last_change":"2024-02-12T17:02:34.502434+0000","last_active":"2024-02-12T17:07:18.408333+0000","last_peered":"2024-02-12T17:07:18.408333+0000","last_clean":"2024-02-12T17:07:18.408333+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:07:18.408333+0000","last_undegraded":"2024-02-12T17:07:18.408333+0000","last_fullsized":"2024-02-12T17:07:18.408333+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.502332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":160,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423041+0000","last_change":"2024-02-12T17:04:40.923654+0000","last_active":"2024-02-12T17:07:18.423041+0000","last_peered":"2024-02-12T17:07:18.423041+0000","last_clean":"2024-02-12T17:07:18.423041+0000","last_became_active":"2024-02-12T17:04:39.865216+0000","last_became_peered":"2024-02-12T17:04:39.865216+0000","last_unstale":"2024-02-12T17:07:18.423041+0000","last_undegraded":"2024-02-12T17:07:18.423041+0000","last_fullsized":"2024-02-12T17:07:18.423041+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":145,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423713+0000","last_change":"2024-02-12T17:06:29.479686+0000","last_active":"2024-02-12T17:07:18.423713+0000","last_peered":"2024-02-12T17:07:18.423713+0000","last_clean":"2024-02-12T17:07:18.423713+0000","last_became_active":"2024-02-12T17:04:39.887235+0000","last_became_peered":"2024-02-12T17:04:39.887235+0000","last_unstale":"2024-02-12T17:07:18.423713+0000","last_undegraded":"2024-02-12T17:07:18.423713+0000","last_fullsized":"2024-02-12T17:07:18.423713+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:29.479610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T17:06:29.479610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":164,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423865+0000","last_change":"2024-02-12T17:04:40.319567+0000","last_active":"2024-02-12T17:07:18.423865+0000","last_peered":"2024-02-12T17:07:18.423865+0000","last_clean":"2024-02-12T17:07:18.423865+0000","last_became_active":"2024-02-12T17:04:39.864612+0000","last_became_peered":"2024-02-12T17:04:39.864612+0000","last_unstale":"2024-02-12T17:07:18.423865+0000","last_undegraded":"2024-02-12T17:07:18.423865+0000","last_fullsized":"2024-02-12T17:07:18.423865+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":111,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408258+0000","last_change":"2024-02-12T17:06:52.828544+0000","last_active":"2024-02-12T17:07:18.408258+0000","last_peered":"2024-02-12T17:07:18.408258+0000","last_clean":"2024-02-12T17:07:18.408258+0000","last_became_active":"2024-02-12T17:04:39.888398+0000","last_became_peered":"2024-02-12T17:04:39.888398+0000","last_unstale":"2024-02-12T17:07:18.408258+0000","last_undegraded":"2024-02-12T17:07:18.408258+0000","last_fullsized":"2024-02-12T17:07:18.408258+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:52.828449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:06:52.828449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":172,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422812+0000","last_change":"2024-02-12T17:04:41.433996+0000","last_active":"2024-02-12T17:07:18.422812+0000","last_peered":"2024-02-12T17:07:18.422812+0000","last_clean":"2024-02-12T17:07:18.422812+0000","last_became_active":"2024-02-12T17:04:39.872998+0000","last_became_peered":"2024-02-12T17:04:39.872998+0000","last_unstale":"2024-02-12T17:07:18.422812+0000","last_undegraded":"2024-02-12T17:07:18.422812+0000","last_fullsized":"2024-02-12T17:07:18.422812+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":131,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.425049+0000","last_change":"2024-02-12T17:03:50.533143+0000","last_active":"2024-02-12T17:07:18.425049+0000","last_peered":"2024-02-12T17:07:18.425049+0000","last_clean":"2024-02-12T17:07:18.425049+0000","last_became_active":"2024-02-12T17:03:50.532233+0000","last_became_peered":"2024-02-12T17:03:50.532233+0000","last_unstale":"2024-02-12T17:07:18.425049+0000","last_undegraded":"2024-02-12T17:07:18.425049+0000","last_fullsized":"2024-02-12T17:07:18.425049+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":326,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415789+0000","last_change":"2024-02-12T17:03:50.670847+0000","last_active":"2024-02-12T17:07:18.415789+0000","last_peered":"2024-02-12T17:07:18.415789+0000","last_clean":"2024-02-12T17:07:18.415789+0000","last_became_active":"2024-02-12T17:03:50.570118+0000","last_became_peered":"2024-02-12T17:03:50.570118+0000","last_unstale":"2024-02-12T17:07:18.415789+0000","last_undegraded":"2024-02-12T17:07:18.415789+0000","last_fullsized":"2024-02-12T17:07:18.415789+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:34.738385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:34.738385+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":450,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413669+0000","last_change":"2024-02-12T17:05:55.027256+0000","last_active":"2024-02-12T17:07:18.413669+0000","last_peered":"2024-02-12T17:07:18.413669+0000","last_clean":"2024-02-12T17:07:18.413669+0000","last_became_active":"2024-02-12T17:04:39.880094+0000","last_became_peered":"2024-02-12T17:04:39.880094+0000","last_unstale":"2024-02-12T17:07:18.413669+0000","last_undegraded":"2024-02-12T17:07:18.413669+0000","last_fullsized":"2024-02-12T17:07:18.413669+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:55.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.027189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":483,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408661+0000","last_change":"2024-02-12T17:03:51.528465+0000","last_active":"2024-02-12T17:07:18.408661+0000","last_peered":"2024-02-12T17:07:18.408661+0000","last_clean":"2024-02-12T17:07:18.408661+0000","last_became_active":"2024-02-12T17:03:50.576714+0000","last_became_peered":"2024-02-12T17:03:50.576714+0000","last_unstale":"2024-02-12T17:07:18.408661+0000","last_undegraded":"2024-02-12T17:07:18.408661+0000","last_fullsized":"2024-02-12T17:07:18.408661+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:01:42.076584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:42.076584+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":127,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408437+0000","last_change":"2024-02-12T17:05:45.618872+0000","last_active":"2024-02-12T17:07:18.408437+0000","last_peered":"2024-02-12T17:07:18.408437+0000","last_clean":"2024-02-12T17:07:18.408437+0000","last_became_active":"2024-02-12T17:05:45.618551+0000","last_became_peered":"2024-02-12T17:05:45.618551+0000","last_unstale":"2024-02-12T17:07:18.408437+0000","last_undegraded":"2024-02-12T17:07:18.408437+0000","last_fullsized":"2024-02-12T17:07:18.408437+0000","mapping_epoch":436,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":145,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423981+0000","last_change":"2024-02-12T17:06:03.551775+0000","last_active":"2024-02-12T17:07:18.423981+0000","last_peered":"2024-02-12T17:07:18.423981+0000","last_clean":"2024-02-12T17:07:18.423981+0000","last_became_active":"2024-02-12T17:04:39.877005+0000","last_became_peered":"2024-02-12T17:04:39.877005+0000","last_unstale":"2024-02-12T17:07:18.423981+0000","last_undegraded":"2024-02-12T17:07:18.423981+0000","last_fullsized":"2024-02-12T17:07:18.423981+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":247,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415681+0000","last_change":"2024-02-12T17:06:31.071643+0000","last_active":"2024-02-12T17:07:18.415681+0000","last_peered":"2024-02-12T17:07:18.415681+0000","last_clean":"2024-02-12T17:07:18.415681+0000","last_became_active":"2024-02-12T17:04:39.884769+0000","last_became_peered":"2024-02-12T17:04:39.884769+0000","last_unstale":"2024-02-12T17:07:18.415681+0000","last_undegraded":"2024-02-12T17:07:18.415681+0000","last_fullsized":"2024-02-12T17:07:18.415681+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:31.071549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:31.071549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":143,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424845+0000","last_change":"2024-02-12T17:03:59.035349+0000","last_active":"2024-02-12T17:07:18.424845+0000","last_peered":"2024-02-12T17:07:18.424845+0000","last_clean":"2024-02-12T17:07:18.424845+0000","last_became_active":"2024-02-12T17:03:50.639321+0000","last_became_peered":"2024-02-12T17:03:50.639321+0000","last_unstale":"2024-02-12T17:07:18.424845+0000","last_undegraded":"2024-02-12T17:07:18.424845+0000","last_fullsized":"2024-02-12T17:07:18.424845+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":469,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.407725+0000","last_change":"2024-02-12T17:05:42.512197+0000","last_active":"2024-02-12T17:07:18.407725+0000","last_peered":"2024-02-12T17:07:18.407725+0000","last_clean":"2024-02-12T17:07:18.407725+0000","last_became_active":"2024-02-12T17:05:42.511859+0000","last_became_peered":"2024-02-12T17:05:42.511859+0000","last_unstale":"2024-02-12T17:07:18.407725+0000","last_undegraded":"2024-02-12T17:07:18.407725+0000","last_fullsized":"2024-02-12T17:07:18.407725+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":441,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408452+0000","last_change":"2024-02-12T17:06:03.911480+0000","last_active":"2024-02-12T17:07:18.408452+0000","last_peered":"2024-02-12T17:07:18.408452+0000","last_clean":"2024-02-12T17:07:18.408452+0000","last_became_active":"2024-02-12T17:04:39.886041+0000","last_became_peered":"2024-02-12T17:04:39.886041+0000","last_unstale":"2024-02-12T17:07:18.408452+0000","last_undegraded":"2024-02-12T17:07:18.408452+0000","last_fullsized":"2024-02-12T17:07:18.408452+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.911375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.911375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":468,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415726+0000","last_change":"2024-02-12T17:06:34.981969+0000","last_active":"2024-02-12T17:07:18.415726+0000","last_peered":"2024-02-12T17:07:18.415726+0000","last_clean":"2024-02-12T17:07:18.415726+0000","last_became_active":"2024-02-12T17:03:50.550931+0000","last_became_peered":"2024-02-12T17:03:50.550931+0000","last_unstale":"2024-02-12T17:07:18.415726+0000","last_undegraded":"2024-02-12T17:07:18.415726+0000","last_fullsized":"2024-02-12T17:07:18.415726+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:34.981898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.981898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":157,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422109+0000","last_change":"2024-02-12T17:06:35.413066+0000","last_active":"2024-02-12T17:07:18.422109+0000","last_peered":"2024-02-12T17:07:18.422109+0000","last_clean":"2024-02-12T17:07:18.422109+0000","last_became_active":"2024-02-12T17:04:39.865088+0000","last_became_peered":"2024-02-12T17:04:39.865088+0000","last_unstale":"2024-02-12T17:07:18.422109+0000","last_undegraded":"2024-02-12T17:07:18.422109+0000","last_fullsized":"2024-02-12T17:07:18.422109+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":256,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.414191+0000","last_change":"2024-02-12T17:05:42.544935+0000","last_active":"2024-02-12T17:07:18.414191+0000","last_peered":"2024-02-12T17:07:18.414191+0000","last_clean":"2024-02-12T17:07:18.414191+0000","last_became_active":"2024-02-12T17:05:42.544611+0000","last_became_peered":"2024-02-12T17:05:42.544611+0000","last_unstale":"2024-02-12T17:07:18.414191+0000","last_undegraded":"2024-02-12T17:07:18.414191+0000","last_fullsized":"2024-02-12T17:07:18.414191+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":131,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424685+0000","last_change":"2024-02-12T17:03:50.545703+0000","last_active":"2024-02-12T17:07:18.424685+0000","last_peered":"2024-02-12T17:07:18.424685+0000","last_clean":"2024-02-12T17:07:18.424685+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:07:18.424685+0000","last_undegraded":"2024-02-12T17:07:18.424685+0000","last_fullsized":"2024-02-12T17:07:18.424685+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:30.051528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:01:30.051528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":302,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:20.830226+0000","last_change":"2024-02-12T17:07:20.830199+0000","last_active":"2024-02-12T17:07:20.830226+0000","last_peered":"2024-02-12T17:07:20.830226+0000","last_clean":"2024-02-12T17:07:20.830226+0000","last_became_active":"2024-02-12T17:03:20.767943+0000","last_became_peered":"2024-02-12T17:03:20.767943+0000","last_unstale":"2024-02-12T17:07:20.830226+0000","last_undegraded":"2024-02-12T17:07:20.830226+0000","last_fullsized":"2024-02-12T17:07:20.830226+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":437,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.414977+0000","last_change":"2024-02-12T17:04:40.575962+0000","last_active":"2024-02-12T17:07:18.414977+0000","last_peered":"2024-02-12T17:07:18.414977+0000","last_clean":"2024-02-12T17:07:18.414977+0000","last_became_active":"2024-02-12T17:04:39.883548+0000","last_became_peered":"2024-02-12T17:04:39.883548+0000","last_unstale":"2024-02-12T17:07:18.414977+0000","last_undegraded":"2024-02-12T17:07:18.414977+0000","last_fullsized":"2024-02-12T17:07:18.414977+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:16.064966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.064966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":486,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423238+0000","last_change":"2024-02-12T17:05:42.512845+0000","last_active":"2024-02-12T17:07:18.423238+0000","last_peered":"2024-02-12T17:07:18.423238+0000","last_clean":"2024-02-12T17:07:18.423238+0000","last_became_active":"2024-02-12T17:05:42.512394+0000","last_became_peered":"2024-02-12T17:05:42.512394+0000","last_unstale":"2024-02-12T17:07:18.423238+0000","last_undegraded":"2024-02-12T17:07:18.423238+0000","last_fullsized":"2024-02-12T17:07:18.423238+0000","mapping_epoch":433,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":384,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413815+0000","last_change":"2024-02-12T17:06:44.765131+0000","last_active":"2024-02-12T17:07:18.413815+0000","last_peered":"2024-02-12T17:07:18.413815+0000","last_clean":"2024-02-12T17:07:18.413815+0000","last_became_active":"2024-02-12T17:04:39.876910+0000","last_became_peered":"2024-02-12T17:04:39.876910+0000","last_unstale":"2024-02-12T17:07:18.413815+0000","last_undegraded":"2024-02-12T17:07:18.413815+0000","last_fullsized":"2024-02-12T17:07:18.413815+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":131,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422978+0000","last_change":"2024-02-12T17:03:50.559509+0000","last_active":"2024-02-12T17:07:18.422978+0000","last_peered":"2024-02-12T17:07:18.422978+0000","last_clean":"2024-02-12T17:07:18.422978+0000","last_became_active":"2024-02-12T17:03:50.558598+0000","last_became_peered":"2024-02-12T17:03:50.558598+0000","last_unstale":"2024-02-12T17:07:18.422978+0000","last_undegraded":"2024-02-12T17:07:18.422978+0000","last_fullsized":"2024-02-12T17:07:18.422978+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":251,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413350+0000","last_change":"2024-02-12T17:06:45.758581+0000","last_active":"2024-02-12T17:07:18.413350+0000","last_peered":"2024-02-12T17:07:18.413350+0000","last_clean":"2024-02-12T17:07:18.413350+0000","last_became_active":"2024-02-12T17:03:20.774315+0000","last_became_peered":"2024-02-12T17:03:20.774315+0000","last_unstale":"2024-02-12T17:07:18.413350+0000","last_undegraded":"2024-02-12T17:07:18.413350+0000","last_fullsized":"2024-02-12T17:07:18.413350+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":457,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.470149+0000","last_change":"2024-02-12T17:05:55.414790+0000","last_active":"2024-02-12T17:07:22.470149+0000","last_peered":"2024-02-12T17:07:22.470149+0000","last_clean":"2024-02-12T17:07:22.470149+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:07:22.470149+0000","last_undegraded":"2024-02-12T17:07:22.470149+0000","last_fullsized":"2024-02-12T17:07:22.470149+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":224,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423549+0000","last_change":"2024-02-12T17:06:51.622488+0000","last_active":"2024-02-12T17:07:18.423549+0000","last_peered":"2024-02-12T17:07:18.423549+0000","last_clean":"2024-02-12T17:07:18.423549+0000","last_became_active":"2024-02-12T17:04:39.886343+0000","last_became_peered":"2024-02-12T17:04:39.886343+0000","last_unstale":"2024-02-12T17:07:18.423549+0000","last_undegraded":"2024-02-12T17:07:18.423549+0000","last_fullsized":"2024-02-12T17:07:18.423549+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.622401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.622401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":541,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:25.757819+0000","last_change":"2024-02-12T17:07:25.763274+0000","last_active":"2024-02-12T17:07:25.757819+0000","last_peered":"2024-02-12T17:07:25.757819+0000","last_clean":"2024-02-12T17:07:25.757819+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:07:25.757819+0000","last_undegraded":"2024-02-12T17:07:25.757819+0000","last_fullsized":"2024-02-12T17:07:25.757819+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:31.827632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:31.827632+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"438'17","reported_seq":405,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.470012+0000","last_change":"2024-02-12T17:05:45.604659+0000","last_active":"2024-02-12T17:07:22.470012+0000","last_peered":"2024-02-12T17:07:22.470012+0000","last_clean":"2024-02-12T17:07:22.470012+0000","last_became_active":"2024-02-12T17:05:45.604285+0000","last_became_peered":"2024-02-12T17:05:45.604285+0000","last_unstale":"2024-02-12T17:07:22.470012+0000","last_undegraded":"2024-02-12T17:07:22.470012+0000","last_fullsized":"2024-02-12T17:07:22.470012+0000","mapping_epoch":436,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":454,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413823+0000","last_change":"2024-02-12T17:03:50.577996+0000","last_active":"2024-02-12T17:07:18.413823+0000","last_peered":"2024-02-12T17:07:18.413823+0000","last_clean":"2024-02-12T17:07:18.413823+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:07:18.413823+0000","last_undegraded":"2024-02-12T17:07:18.413823+0000","last_fullsized":"2024-02-12T17:07:18.413823+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:34.874363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:34.874363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":148,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422899+0000","last_change":"2024-02-12T17:06:34.381885+0000","last_active":"2024-02-12T17:07:18.422899+0000","last_peered":"2024-02-12T17:07:18.422899+0000","last_clean":"2024-02-12T17:07:18.422899+0000","last_became_active":"2024-02-12T17:03:50.556444+0000","last_became_peered":"2024-02-12T17:03:50.556444+0000","last_unstale":"2024-02-12T17:07:18.422899+0000","last_undegraded":"2024-02-12T17:07:18.422899+0000","last_fullsized":"2024-02-12T17:07:18.422899+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":454,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408809+0000","last_change":"2024-02-12T17:06:37.116695+0000","last_active":"2024-02-12T17:07:18.408809+0000","last_peered":"2024-02-12T17:07:18.408809+0000","last_clean":"2024-02-12T17:07:18.408809+0000","last_became_active":"2024-02-12T17:04:39.889055+0000","last_became_peered":"2024-02-12T17:04:39.889055+0000","last_unstale":"2024-02-12T17:07:18.408809+0000","last_undegraded":"2024-02-12T17:07:18.408809+0000","last_fullsized":"2024-02-12T17:07:18.408809+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:37.116609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:06:37.116609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":393,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413569+0000","last_change":"2024-02-12T17:04:39.884622+0000","last_active":"2024-02-12T17:07:18.413569+0000","last_peered":"2024-02-12T17:07:18.413569+0000","last_clean":"2024-02-12T17:07:18.413569+0000","last_became_active":"2024-02-12T17:04:39.884162+0000","last_became_peered":"2024-02-12T17:04:39.884162+0000","last_unstale":"2024-02-12T17:07:18.413569+0000","last_undegraded":"2024-02-12T17:07:18.413569+0000","last_fullsized":"2024-02-12T17:07:18.413569+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:32.127593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:32.127593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"460'15","reported_seq":147,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424157+0000","last_change":"2024-02-12T17:06:44.468829+0000","last_active":"2024-02-12T17:07:18.424157+0000","last_peered":"2024-02-12T17:07:18.424157+0000","last_clean":"2024-02-12T17:07:18.424157+0000","last_became_active":"2024-02-12T17:03:50.555238+0000","last_became_peered":"2024-02-12T17:03:50.555238+0000","last_unstale":"2024-02-12T17:07:18.424157+0000","last_undegraded":"2024-02-12T17:07:18.424157+0000","last_fullsized":"2024-02-12T17:07:18.424157+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":106,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408681+0000","last_change":"2024-02-12T17:04:39.936874+0000","last_active":"2024-02-12T17:07:18.408681+0000","last_peered":"2024-02-12T17:07:18.408681+0000","last_clean":"2024-02-12T17:07:18.408681+0000","last_became_active":"2024-02-12T17:04:39.936538+0000","last_became_peered":"2024-02-12T17:04:39.936538+0000","last_unstale":"2024-02-12T17:07:18.408681+0000","last_undegraded":"2024-02-12T17:07:18.408681+0000","last_fullsized":"2024-02-12T17:07:18.408681+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:43.982623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.982623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":367,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.571520+0000","last_change":"2024-02-12T17:07:22.571520+0000","last_active":"2024-02-12T17:07:22.571520+0000","last_peered":"2024-02-12T17:07:22.571520+0000","last_clean":"2024-02-12T17:07:22.571520+0000","last_became_active":"2024-02-12T17:04:39.866732+0000","last_became_peered":"2024-02-12T17:04:39.866732+0000","last_unstale":"2024-02-12T17:07:22.571520+0000","last_undegraded":"2024-02-12T17:07:22.571520+0000","last_fullsized":"2024-02-12T17:07:22.571520+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":146,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423248+0000","last_change":"2024-02-12T17:06:45.502715+0000","last_active":"2024-02-12T17:07:18.423248+0000","last_peered":"2024-02-12T17:07:18.423248+0000","last_clean":"2024-02-12T17:07:18.423248+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:07:18.423248+0000","last_undegraded":"2024-02-12T17:07:18.423248+0000","last_fullsized":"2024-02-12T17:07:18.423248+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"470'5944","reported_seq":5971,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.587498+0000","last_change":"2024-02-12T17:06:18.392616+0000","last_active":"2024-02-12T17:07:28.587498+0000","last_peered":"2024-02-12T17:07:28.587498+0000","last_clean":"2024-02-12T17:07:28.587498+0000","last_became_active":"2024-02-12T17:06:18.392104+0000","last_became_peered":"2024-02-12T17:06:18.392104+0000","last_unstale":"2024-02-12T17:07:28.587498+0000","last_undegraded":"2024-02-12T17:07:28.587498+0000","last_fullsized":"2024-02-12T17:07:28.587498+0000","mapping_epoch":451,"log_start":"460'2500","ondisk_log_start":"460'2500","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3444,"ondisk_log_size":3444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389545984,"num_objects":5944,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5944,"num_write_kb":380416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":163,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.845696+0000","last_change":"2024-02-12T17:07:18.845664+0000","last_active":"2024-02-12T17:07:18.845696+0000","last_peered":"2024-02-12T17:07:18.845696+0000","last_clean":"2024-02-12T17:07:18.845696+0000","last_became_active":"2024-02-12T17:04:39.889283+0000","last_became_peered":"2024-02-12T17:04:39.889283+0000","last_unstale":"2024-02-12T17:07:18.845696+0000","last_undegraded":"2024-02-12T17:07:18.845696+0000","last_fullsized":"2024-02-12T17:07:18.845696+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":168,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408530+0000","last_change":"2024-02-12T17:04:40.578504+0000","last_active":"2024-02-12T17:07:18.408530+0000","last_peered":"2024-02-12T17:07:18.408530+0000","last_clean":"2024-02-12T17:07:18.408530+0000","last_became_active":"2024-02-12T17:04:39.890440+0000","last_became_peered":"2024-02-12T17:04:39.890440+0000","last_unstale":"2024-02-12T17:07:18.408530+0000","last_undegraded":"2024-02-12T17:07:18.408530+0000","last_fullsized":"2024-02-12T17:07:18.408530+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:02:17.066331+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:02:17.066331+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":164,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.425129+0000","last_change":"2024-02-12T17:04:42.701462+0000","last_active":"2024-02-12T17:07:18.425129+0000","last_peered":"2024-02-12T17:07:18.425129+0000","last_clean":"2024-02-12T17:07:18.425129+0000","last_became_active":"2024-02-12T17:04:39.891351+0000","last_became_peered":"2024-02-12T17:04:39.891351+0000","last_unstale":"2024-02-12T17:07:18.425129+0000","last_undegraded":"2024-02-12T17:07:18.425129+0000","last_fullsized":"2024-02-12T17:07:18.425129+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":364,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413523+0000","last_change":"2024-02-12T17:04:39.882820+0000","last_active":"2024-02-12T17:07:18.413523+0000","last_peered":"2024-02-12T17:07:18.413523+0000","last_clean":"2024-02-12T17:07:18.413523+0000","last_became_active":"2024-02-12T17:04:39.880493+0000","last_became_peered":"2024-02-12T17:04:39.880493+0000","last_unstale":"2024-02-12T17:07:18.413523+0000","last_undegraded":"2024-02-12T17:07:18.413523+0000","last_fullsized":"2024-02-12T17:07:18.413523+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:25.815816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.815816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"470'5558","reported_seq":5585,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:23.720536+0000","last_change":"2024-02-12T17:06:18.392774+0000","last_active":"2024-02-12T17:07:23.720536+0000","last_peered":"2024-02-12T17:07:23.720536+0000","last_clean":"2024-02-12T17:07:23.720536+0000","last_became_active":"2024-02-12T17:06:18.392438+0000","last_became_peered":"2024-02-12T17:06:18.392438+0000","last_unstale":"2024-02-12T17:07:23.720536+0000","last_undegraded":"2024-02-12T17:07:23.720536+0000","last_fullsized":"2024-02-12T17:07:23.720536+0000","mapping_epoch":451,"log_start":"460'2400","ondisk_log_start":"460'2400","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3158,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":364249088,"num_objects":5558,"num_object_clones":0,"num_object_copies":16674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":355712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":453,"reported_epoch":470,"state":"active+clean+scrubbing","last_fresh":"2024-02-12T17:07:23.600846+0000","last_change":"2024-02-12T17:07:23.600846+0000","last_active":"2024-02-12T17:07:23.600846+0000","last_peered":"2024-02-12T17:07:23.600846+0000","last_clean":"2024-02-12T17:07:23.600846+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:07:23.600846+0000","last_undegraded":"2024-02-12T17:07:23.600846+0000","last_fullsized":"2024-02-12T17:07:23.600846+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:56.852121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.852121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":284,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.420704+0000","last_change":"2024-02-12T17:06:23.673240+0000","last_active":"2024-02-12T17:07:18.420704+0000","last_peered":"2024-02-12T17:07:18.420704+0000","last_clean":"2024-02-12T17:07:18.420704+0000","last_became_active":"2024-02-12T16:57:47.186754+0000","last_became_peered":"2024-02-12T16:57:47.186754+0000","last_unstale":"2024-02-12T17:07:18.420704+0000","last_undegraded":"2024-02-12T17:07:18.420704+0000","last_fullsized":"2024-02-12T17:07:18.420704+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:23.673128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:23.673128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":375,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.414269+0000","last_change":"2024-02-12T17:06:04.040637+0000","last_active":"2024-02-12T17:07:18.414269+0000","last_peered":"2024-02-12T17:07:18.414269+0000","last_clean":"2024-02-12T17:07:18.414269+0000","last_became_active":"2024-02-12T17:04:39.883521+0000","last_became_peered":"2024-02-12T17:04:39.883521+0000","last_unstale":"2024-02-12T17:07:18.414269+0000","last_undegraded":"2024-02-12T17:07:18.414269+0000","last_fullsized":"2024-02-12T17:07:18.414269+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:04.040555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:04.040555+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"433'15","reported_seq":648,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.471560+0000","last_change":"2024-02-12T17:05:40.422839+0000","last_active":"2024-02-12T17:07:22.471560+0000","last_peered":"2024-02-12T17:07:22.471560+0000","last_clean":"2024-02-12T17:07:22.471560+0000","last_became_active":"2024-02-12T17:05:40.422494+0000","last_became_peered":"2024-02-12T17:05:40.422494+0000","last_unstale":"2024-02-12T17:07:22.471560+0000","last_undegraded":"2024-02-12T17:07:22.471560+0000","last_fullsized":"2024-02-12T17:07:22.471560+0000","mapping_epoch":347,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"470'5924","reported_seq":5950,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.350790+0000","last_change":"2024-02-12T17:06:18.391956+0000","last_active":"2024-02-12T17:07:28.350790+0000","last_peered":"2024-02-12T17:07:28.350790+0000","last_clean":"2024-02-12T17:07:28.350790+0000","last_became_active":"2024-02-12T17:06:18.391610+0000","last_became_peered":"2024-02-12T17:06:18.391610+0000","last_unstale":"2024-02-12T17:07:28.350790+0000","last_undegraded":"2024-02-12T17:07:28.350790+0000","last_fullsized":"2024-02-12T17:07:28.350790+0000","mapping_epoch":451,"log_start":"460'2600","ondisk_log_start":"460'2600","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3324,"ondisk_log_size":3324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":388235264,"num_objects":5924,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":379136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":158,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422957+0000","last_change":"2024-02-12T17:04:58.474896+0000","last_active":"2024-02-12T17:07:18.422957+0000","last_peered":"2024-02-12T17:07:18.422957+0000","last_clean":"2024-02-12T17:07:18.422957+0000","last_became_active":"2024-02-12T17:04:58.474504+0000","last_became_peered":"2024-02-12T17:04:58.474504+0000","last_unstale":"2024-02-12T17:07:18.422957+0000","last_undegraded":"2024-02-12T17:07:18.422957+0000","last_fullsized":"2024-02-12T17:07:18.422957+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":473,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.791104+0000","last_change":"2024-02-12T17:07:22.791104+0000","last_active":"2024-02-12T17:07:22.791104+0000","last_peered":"2024-02-12T17:07:22.791104+0000","last_clean":"2024-02-12T17:07:22.791104+0000","last_became_active":"2024-02-12T17:03:50.548926+0000","last_became_peered":"2024-02-12T17:03:50.548926+0000","last_unstale":"2024-02-12T17:07:22.791104+0000","last_undegraded":"2024-02-12T17:07:22.791104+0000","last_fullsized":"2024-02-12T17:07:22.791104+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"348'55","reported_seq":198,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.470305+0000","last_change":"2024-02-12T17:03:58.596193+0000","last_active":"2024-02-12T17:07:22.470305+0000","last_peered":"2024-02-12T17:07:22.470305+0000","last_clean":"2024-02-12T17:07:22.470305+0000","last_became_active":"2024-02-12T17:03:50.557154+0000","last_became_peered":"2024-02-12T17:03:50.557154+0000","last_unstale":"2024-02-12T17:07:22.470305+0000","last_undegraded":"2024-02-12T17:07:22.470305+0000","last_fullsized":"2024-02-12T17:07:22.470305+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":418,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408902+0000","last_change":"2024-02-12T17:05:40.431590+0000","last_active":"2024-02-12T17:07:18.408902+0000","last_peered":"2024-02-12T17:07:18.408902+0000","last_clean":"2024-02-12T17:07:18.408902+0000","last_became_active":"2024-02-12T17:05:40.431265+0000","last_became_peered":"2024-02-12T17:05:40.431265+0000","last_unstale":"2024-02-12T17:07:18.408902+0000","last_undegraded":"2024-02-12T17:07:18.408902+0000","last_fullsized":"2024-02-12T17:07:18.408902+0000","mapping_epoch":377,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"470'6052","reported_seq":6079,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.349268+0000","last_change":"2024-02-12T17:06:18.392794+0000","last_active":"2024-02-12T17:07:28.349268+0000","last_peered":"2024-02-12T17:07:28.349268+0000","last_clean":"2024-02-12T17:07:28.349268+0000","last_became_active":"2024-02-12T17:06:18.392475+0000","last_became_peered":"2024-02-12T17:06:18.392475+0000","last_unstale":"2024-02-12T17:07:28.349268+0000","last_undegraded":"2024-02-12T17:07:28.349268+0000","last_fullsized":"2024-02-12T17:07:28.349268+0000","mapping_epoch":451,"log_start":"460'2800","ondisk_log_start":"460'2800","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3252,"ondisk_log_size":3252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":396623872,"num_objects":6052,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":387328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":461,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408952+0000","last_change":"2024-02-12T17:06:46.001986+0000","last_active":"2024-02-12T17:07:18.408952+0000","last_peered":"2024-02-12T17:07:18.408952+0000","last_clean":"2024-02-12T17:07:18.408952+0000","last_became_active":"2024-02-12T17:03:50.564700+0000","last_became_peered":"2024-02-12T17:03:50.564700+0000","last_unstale":"2024-02-12T17:07:18.408952+0000","last_undegraded":"2024-02-12T17:07:18.408952+0000","last_fullsized":"2024-02-12T17:07:18.408952+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.001888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.001888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":571,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.400974+0000","last_change":"2024-02-12T17:05:59.784735+0000","last_active":"2024-02-12T17:07:22.400974+0000","last_peered":"2024-02-12T17:07:22.400974+0000","last_clean":"2024-02-12T17:07:22.400974+0000","last_became_active":"2024-02-12T17:01:16.912041+0000","last_became_peered":"2024-02-12T17:01:16.912041+0000","last_unstale":"2024-02-12T17:07:22.400974+0000","last_undegraded":"2024-02-12T17:07:22.400974+0000","last_fullsized":"2024-02-12T17:07:22.400974+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:05:59.784636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:59.784636+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":367,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":131,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424979+0000","last_change":"2024-02-12T17:03:50.533363+0000","last_active":"2024-02-12T17:07:18.424979+0000","last_peered":"2024-02-12T17:07:18.424979+0000","last_clean":"2024-02-12T17:07:18.424979+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:07:18.424979+0000","last_undegraded":"2024-02-12T17:07:18.424979+0000","last_fullsized":"2024-02-12T17:07:18.424979+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:38.142602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:38.142602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":402,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415736+0000","last_change":"2024-02-12T17:03:51.175255+0000","last_active":"2024-02-12T17:07:18.415736+0000","last_peered":"2024-02-12T17:07:18.415736+0000","last_clean":"2024-02-12T17:07:18.415736+0000","last_became_active":"2024-02-12T17:03:50.573814+0000","last_became_peered":"2024-02-12T17:03:50.573814+0000","last_unstale":"2024-02-12T17:07:18.415736+0000","last_undegraded":"2024-02-12T17:07:18.415736+0000","last_fullsized":"2024-02-12T17:07:18.415736+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":389,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415770+0000","last_change":"2024-02-12T17:06:25.129256+0000","last_active":"2024-02-12T17:07:18.415770+0000","last_peered":"2024-02-12T17:07:18.415770+0000","last_clean":"2024-02-12T17:07:18.415770+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:07:18.415770+0000","last_undegraded":"2024-02-12T17:07:18.415770+0000","last_fullsized":"2024-02-12T17:07:18.415770+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:25.129190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:25.129190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"470'5635","reported_seq":5661,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:23.735747+0000","last_change":"2024-02-12T17:06:18.392727+0000","last_active":"2024-02-12T17:07:23.735747+0000","last_peered":"2024-02-12T17:07:23.735747+0000","last_clean":"2024-02-12T17:07:23.735747+0000","last_became_active":"2024-02-12T17:06:18.392427+0000","last_became_peered":"2024-02-12T17:06:18.392427+0000","last_unstale":"2024-02-12T17:07:23.735747+0000","last_undegraded":"2024-02-12T17:07:23.735747+0000","last_fullsized":"2024-02-12T17:07:23.735747+0000","mapping_epoch":451,"log_start":"460'2400","ondisk_log_start":"460'2400","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3235,"ondisk_log_size":3235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369295360,"num_objects":5635,"num_object_clones":0,"num_object_copies":16905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5635,"num_write_kb":360640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":489,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415881+0000","last_change":"2024-02-12T17:06:40.058164+0000","last_active":"2024-02-12T17:07:18.415881+0000","last_peered":"2024-02-12T17:07:18.415881+0000","last_clean":"2024-02-12T17:07:18.415881+0000","last_became_active":"2024-02-12T17:04:54.415431+0000","last_became_peered":"2024-02-12T17:04:54.415431+0000","last_unstale":"2024-02-12T17:07:18.415881+0000","last_undegraded":"2024-02-12T17:07:18.415881+0000","last_fullsized":"2024-02-12T17:07:18.415881+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.058086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.058086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":145,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422671+0000","last_change":"2024-02-12T17:05:54.311879+0000","last_active":"2024-02-12T17:07:18.422671+0000","last_peered":"2024-02-12T17:07:18.422671+0000","last_clean":"2024-02-12T17:07:18.422671+0000","last_became_active":"2024-02-12T17:04:39.885173+0000","last_became_peered":"2024-02-12T17:04:39.885173+0000","last_unstale":"2024-02-12T17:07:18.422671+0000","last_undegraded":"2024-02-12T17:07:18.422671+0000","last_fullsized":"2024-02-12T17:07:18.422671+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:54.311805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:05:54.311805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":441,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.414864+0000","last_change":"2024-02-12T17:05:56.054036+0000","last_active":"2024-02-12T17:07:18.414864+0000","last_peered":"2024-02-12T17:07:18.414864+0000","last_clean":"2024-02-12T17:07:18.414864+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:07:18.414864+0000","last_undegraded":"2024-02-12T17:07:18.414864+0000","last_fullsized":"2024-02-12T17:07:18.414864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:56.053971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.053971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":473,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408137+0000","last_change":"2024-02-12T17:06:49.876266+0000","last_active":"2024-02-12T17:07:18.408137+0000","last_peered":"2024-02-12T17:07:18.408137+0000","last_clean":"2024-02-12T17:07:18.408137+0000","last_became_active":"2024-02-12T17:04:39.915392+0000","last_became_peered":"2024-02-12T17:04:39.915392+0000","last_unstale":"2024-02-12T17:07:18.408137+0000","last_undegraded":"2024-02-12T17:07:18.408137+0000","last_fullsized":"2024-02-12T17:07:18.408137+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"462'33","reported_seq":229,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.409012+0000","last_change":"2024-02-12T17:06:53.886435+0000","last_active":"2024-02-12T17:07:18.409012+0000","last_peered":"2024-02-12T17:07:18.409012+0000","last_clean":"2024-02-12T17:07:18.409012+0000","last_became_active":"2024-02-12T17:04:39.889039+0000","last_became_peered":"2024-02-12T17:04:39.889039+0000","last_unstale":"2024-02-12T17:07:18.409012+0000","last_undegraded":"2024-02-12T17:07:18.409012+0000","last_fullsized":"2024-02-12T17:07:18.409012+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'33","last_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.886342+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":466,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408501+0000","last_change":"2024-02-12T17:05:40.432925+0000","last_active":"2024-02-12T17:07:18.408501+0000","last_peered":"2024-02-12T17:07:18.408501+0000","last_clean":"2024-02-12T17:07:18.408501+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:07:18.408501+0000","last_undegraded":"2024-02-12T17:07:18.408501+0000","last_fullsized":"2024-02-12T17:07:18.408501+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"470'5609","reported_seq":5636,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:23.588751+0000","last_change":"2024-02-12T17:06:18.394208+0000","last_active":"2024-02-12T17:07:23.588751+0000","last_peered":"2024-02-12T17:07:23.588751+0000","last_clean":"2024-02-12T17:07:23.588751+0000","last_became_active":"2024-02-12T17:06:18.393896+0000","last_became_peered":"2024-02-12T17:06:18.393896+0000","last_unstale":"2024-02-12T17:07:23.588751+0000","last_undegraded":"2024-02-12T17:07:23.588751+0000","last_fullsized":"2024-02-12T17:07:23.588751+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5609,"ondisk_log_size":5609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":367591424,"num_objects":5609,"num_object_clones":0,"num_object_copies":16827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5609,"num_write_kb":358976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":399,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.414955+0000","last_change":"2024-02-12T17:06:02.044976+0000","last_active":"2024-02-12T17:07:18.414955+0000","last_peered":"2024-02-12T17:07:18.414955+0000","last_clean":"2024-02-12T17:07:18.414955+0000","last_became_active":"2024-02-12T17:04:39.870979+0000","last_became_peered":"2024-02-12T17:04:39.870979+0000","last_unstale":"2024-02-12T17:07:18.414955+0000","last_undegraded":"2024-02-12T17:07:18.414955+0000","last_fullsized":"2024-02-12T17:07:18.414955+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":224,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424201+0000","last_change":"2024-02-12T17:06:46.537733+0000","last_active":"2024-02-12T17:07:18.424201+0000","last_peered":"2024-02-12T17:07:18.424201+0000","last_clean":"2024-02-12T17:07:18.424201+0000","last_became_active":"2024-02-12T17:04:39.892122+0000","last_became_peered":"2024-02-12T17:04:39.892122+0000","last_unstale":"2024-02-12T17:07:18.424201+0000","last_undegraded":"2024-02-12T17:07:18.424201+0000","last_fullsized":"2024-02-12T17:07:18.424201+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.537630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.537630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":471,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.407909+0000","last_change":"2024-02-12T17:06:01.905985+0000","last_active":"2024-02-12T17:07:18.407909+0000","last_peered":"2024-02-12T17:07:18.407909+0000","last_clean":"2024-02-12T17:07:18.407909+0000","last_became_active":"2024-02-12T17:03:50.564881+0000","last_became_peered":"2024-02-12T17:03:50.564881+0000","last_unstale":"2024-02-12T17:07:18.407909+0000","last_undegraded":"2024-02-12T17:07:18.407909+0000","last_fullsized":"2024-02-12T17:07:18.407909+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":456,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408397+0000","last_change":"2024-02-12T17:05:55.847994+0000","last_active":"2024-02-12T17:07:18.408397+0000","last_peered":"2024-02-12T17:07:18.408397+0000","last_clean":"2024-02-12T17:07:18.408397+0000","last_became_active":"2024-02-12T17:03:20.780926+0000","last_became_peered":"2024-02-12T17:03:20.780926+0000","last_unstale":"2024-02-12T17:07:18.408397+0000","last_undegraded":"2024-02-12T17:07:18.408397+0000","last_fullsized":"2024-02-12T17:07:18.408397+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":468,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415848+0000","last_change":"2024-02-12T17:04:39.885796+0000","last_active":"2024-02-12T17:07:18.415848+0000","last_peered":"2024-02-12T17:07:18.415848+0000","last_clean":"2024-02-12T17:07:18.415848+0000","last_became_active":"2024-02-12T17:04:39.885162+0000","last_became_peered":"2024-02-12T17:04:39.885162+0000","last_unstale":"2024-02-12T17:07:18.415848+0000","last_undegraded":"2024-02-12T17:07:18.415848+0000","last_fullsized":"2024-02-12T17:07:18.415848+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"470'6042","reported_seq":6069,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.553686+0000","last_change":"2024-02-12T17:06:18.394587+0000","last_active":"2024-02-12T17:07:28.553686+0000","last_peered":"2024-02-12T17:07:28.553686+0000","last_clean":"2024-02-12T17:07:28.553686+0000","last_became_active":"2024-02-12T17:06:18.394337+0000","last_became_peered":"2024-02-12T17:06:18.394337+0000","last_unstale":"2024-02-12T17:07:28.553686+0000","last_undegraded":"2024-02-12T17:07:28.553686+0000","last_fullsized":"2024-02-12T17:07:28.553686+0000","mapping_epoch":451,"log_start":"460'2600","ondisk_log_start":"460'2600","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3442,"ondisk_log_size":3442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":395968512,"num_objects":6042,"num_object_clones":0,"num_object_copies":18126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6042,"num_write_kb":386688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":195,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.407997+0000","last_change":"2024-02-12T17:03:20.779625+0000","last_active":"2024-02-12T17:07:18.407997+0000","last_peered":"2024-02-12T17:07:18.407997+0000","last_clean":"2024-02-12T17:07:18.407997+0000","last_became_active":"2024-02-12T17:03:20.779126+0000","last_became_peered":"2024-02-12T17:03:20.779126+0000","last_unstale":"2024-02-12T17:07:18.407997+0000","last_undegraded":"2024-02-12T17:07:18.407997+0000","last_fullsized":"2024-02-12T17:07:18.407997+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:48.617913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:02:48.617913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":461,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.409001+0000","last_change":"2024-02-12T17:06:40.961157+0000","last_active":"2024-02-12T17:07:18.409001+0000","last_peered":"2024-02-12T17:07:18.409001+0000","last_clean":"2024-02-12T17:07:18.409001+0000","last_became_active":"2024-02-12T17:04:39.886102+0000","last_became_peered":"2024-02-12T17:04:39.886102+0000","last_unstale":"2024-02-12T17:07:18.409001+0000","last_undegraded":"2024-02-12T17:07:18.409001+0000","last_fullsized":"2024-02-12T17:07:18.409001+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.961097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.961097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":448,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.414873+0000","last_change":"2024-02-12T17:03:21.658125+0000","last_active":"2024-02-12T17:07:18.414873+0000","last_peered":"2024-02-12T17:07:18.414873+0000","last_clean":"2024-02-12T17:07:18.414873+0000","last_became_active":"2024-02-12T17:03:20.769223+0000","last_became_peered":"2024-02-12T17:03:20.769223+0000","last_unstale":"2024-02-12T17:07:18.414873+0000","last_undegraded":"2024-02-12T17:07:18.414873+0000","last_fullsized":"2024-02-12T17:07:18.414873+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:02:44.688402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:44.688402+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":261,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.420839+0000","last_change":"2024-02-12T17:06:25.676817+0000","last_active":"2024-02-12T17:07:18.420839+0000","last_peered":"2024-02-12T17:07:18.420839+0000","last_clean":"2024-02-12T17:07:18.420839+0000","last_became_active":"2024-02-12T17:05:40.448219+0000","last_became_peered":"2024-02-12T17:05:40.448219+0000","last_unstale":"2024-02-12T17:07:18.420839+0000","last_undegraded":"2024-02-12T17:07:18.420839+0000","last_fullsized":"2024-02-12T17:07:18.420839+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:25.676720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:25.676720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"470'5970","reported_seq":5997,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.328066+0000","last_change":"2024-02-12T17:06:18.395287+0000","last_active":"2024-02-12T17:07:28.328066+0000","last_peered":"2024-02-12T17:07:28.328066+0000","last_clean":"2024-02-12T17:07:28.328066+0000","last_became_active":"2024-02-12T17:06:18.395022+0000","last_became_peered":"2024-02-12T17:06:18.395022+0000","last_unstale":"2024-02-12T17:07:28.328066+0000","last_undegraded":"2024-02-12T17:07:28.328066+0000","last_fullsized":"2024-02-12T17:07:28.328066+0000","mapping_epoch":451,"log_start":"460'2700","ondisk_log_start":"460'2700","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3270,"ondisk_log_size":3270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391249920,"num_objects":5970,"num_object_clones":0,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5970,"num_write_kb":382080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":145,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422758+0000","last_change":"2024-02-12T17:06:01.456051+0000","last_active":"2024-02-12T17:07:18.422758+0000","last_peered":"2024-02-12T17:07:18.422758+0000","last_clean":"2024-02-12T17:07:18.422758+0000","last_became_active":"2024-02-12T17:04:39.885677+0000","last_became_peered":"2024-02-12T17:04:39.885677+0000","last_unstale":"2024-02-12T17:07:18.422758+0000","last_undegraded":"2024-02-12T17:07:18.422758+0000","last_fullsized":"2024-02-12T17:07:18.422758+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:01.455969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.455969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":227,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422811+0000","last_change":"2024-02-12T17:04:39.864989+0000","last_active":"2024-02-12T17:07:18.422811+0000","last_peered":"2024-02-12T17:07:18.422811+0000","last_clean":"2024-02-12T17:07:18.422811+0000","last_became_active":"2024-02-12T17:04:39.864631+0000","last_became_peered":"2024-02-12T17:04:39.864631+0000","last_unstale":"2024-02-12T17:07:18.422811+0000","last_undegraded":"2024-02-12T17:07:18.422811+0000","last_fullsized":"2024-02-12T17:07:18.422811+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":149,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422265+0000","last_change":"2024-02-12T17:03:54.620306+0000","last_active":"2024-02-12T17:07:18.422265+0000","last_peered":"2024-02-12T17:07:18.422265+0000","last_clean":"2024-02-12T17:07:18.422265+0000","last_became_active":"2024-02-12T17:03:50.556763+0000","last_became_peered":"2024-02-12T17:03:50.556763+0000","last_unstale":"2024-02-12T17:07:18.422265+0000","last_undegraded":"2024-02-12T17:07:18.422265+0000","last_fullsized":"2024-02-12T17:07:18.422265+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":469,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424767+0000","last_change":"2024-02-12T17:05:40.428597+0000","last_active":"2024-02-12T17:07:18.424767+0000","last_peered":"2024-02-12T17:07:18.424767+0000","last_clean":"2024-02-12T17:07:18.424767+0000","last_became_active":"2024-02-12T17:05:40.427915+0000","last_became_peered":"2024-02-12T17:05:40.427915+0000","last_unstale":"2024-02-12T17:07:18.424767+0000","last_undegraded":"2024-02-12T17:07:18.424767+0000","last_fullsized":"2024-02-12T17:07:18.424767+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"470'5967","reported_seq":5990,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.587411+0000","last_change":"2024-02-12T17:06:18.392414+0000","last_active":"2024-02-12T17:07:28.587411+0000","last_peered":"2024-02-12T17:07:28.587411+0000","last_clean":"2024-02-12T17:07:28.587411+0000","last_became_active":"2024-02-12T17:06:18.391649+0000","last_became_peered":"2024-02-12T17:06:18.391649+0000","last_unstale":"2024-02-12T17:07:28.587411+0000","last_undegraded":"2024-02-12T17:07:28.587411+0000","last_fullsized":"2024-02-12T17:07:28.587411+0000","mapping_epoch":451,"log_start":"460'2500","ondisk_log_start":"460'2500","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3467,"ondisk_log_size":3467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391053312,"num_objects":5967,"num_object_clones":0,"num_object_copies":17901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":381888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":472,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408653+0000","last_change":"2024-02-12T17:04:40.405970+0000","last_active":"2024-02-12T17:07:18.408653+0000","last_peered":"2024-02-12T17:07:18.408653+0000","last_clean":"2024-02-12T17:07:18.408653+0000","last_became_active":"2024-02-12T17:04:39.889077+0000","last_became_peered":"2024-02-12T17:04:39.889077+0000","last_unstale":"2024-02-12T17:07:18.408653+0000","last_undegraded":"2024-02-12T17:07:18.408653+0000","last_fullsized":"2024-02-12T17:07:18.408653+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:03:17.811656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.811656+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":237,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408642+0000","last_change":"2024-02-12T17:03:50.563493+0000","last_active":"2024-02-12T17:07:18.408642+0000","last_peered":"2024-02-12T17:07:18.408642+0000","last_clean":"2024-02-12T17:07:18.408642+0000","last_became_active":"2024-02-12T17:03:50.562556+0000","last_became_peered":"2024-02-12T17:03:50.562556+0000","last_unstale":"2024-02-12T17:07:18.408642+0000","last_undegraded":"2024-02-12T17:07:18.408642+0000","last_fullsized":"2024-02-12T17:07:18.408642+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.562425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.562425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":528,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413806+0000","last_change":"2024-02-12T17:06:28.073224+0000","last_active":"2024-02-12T17:07:18.413806+0000","last_peered":"2024-02-12T17:07:18.413806+0000","last_clean":"2024-02-12T17:07:18.413806+0000","last_became_active":"2024-02-12T17:04:39.884095+0000","last_became_peered":"2024-02-12T17:04:39.884095+0000","last_unstale":"2024-02-12T17:07:18.413806+0000","last_undegraded":"2024-02-12T17:07:18.413806+0000","last_fullsized":"2024-02-12T17:07:18.413806+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'28","last_scrub_stamp":"2024-02-12T17:06:28.073131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:28.073131+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"433'19","reported_seq":224,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.470353+0000","last_change":"2024-02-12T17:05:40.451102+0000","last_active":"2024-02-12T17:07:22.470353+0000","last_peered":"2024-02-12T17:07:22.470353+0000","last_clean":"2024-02-12T17:07:22.470353+0000","last_became_active":"2024-02-12T17:05:40.450299+0000","last_became_peered":"2024-02-12T17:05:40.450299+0000","last_unstale":"2024-02-12T17:07:22.470353+0000","last_undegraded":"2024-02-12T17:07:22.470353+0000","last_fullsized":"2024-02-12T17:07:22.470353+0000","mapping_epoch":377,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"470'5544","reported_seq":5571,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:23.729110+0000","last_change":"2024-02-12T17:06:18.394104+0000","last_active":"2024-02-12T17:07:23.729110+0000","last_peered":"2024-02-12T17:07:23.729110+0000","last_clean":"2024-02-12T17:07:23.729110+0000","last_became_active":"2024-02-12T17:06:18.393754+0000","last_became_peered":"2024-02-12T17:06:18.393754+0000","last_unstale":"2024-02-12T17:07:23.729110+0000","last_undegraded":"2024-02-12T17:07:23.729110+0000","last_fullsized":"2024-02-12T17:07:23.729110+0000","mapping_epoch":451,"log_start":"460'2300","ondisk_log_start":"460'2300","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3244,"ondisk_log_size":3244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":363331584,"num_objects":5544,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5544,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":154,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:20.769102+0000","last_change":"2024-02-12T17:07:20.769058+0000","last_active":"2024-02-12T17:07:20.769102+0000","last_peered":"2024-02-12T17:07:20.769102+0000","last_clean":"2024-02-12T17:07:20.769102+0000","last_became_active":"2024-02-12T17:04:33.794457+0000","last_became_peered":"2024-02-12T17:04:33.794457+0000","last_unstale":"2024-02-12T17:07:20.769102+0000","last_undegraded":"2024-02-12T17:07:20.769102+0000","last_fullsized":"2024-02-12T17:07:20.769102+0000","mapping_epoch":347,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"443'21","reported_seq":667,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.469990+0000","last_change":"2024-02-12T17:06:32.977737+0000","last_active":"2024-02-12T17:07:22.469990+0000","last_peered":"2024-02-12T17:07:22.469990+0000","last_clean":"2024-02-12T17:07:22.469990+0000","last_became_active":"2024-02-12T17:05:55.374445+0000","last_became_peered":"2024-02-12T17:05:55.374445+0000","last_unstale":"2024-02-12T17:07:22.469990+0000","last_undegraded":"2024-02-12T17:07:22.469990+0000","last_fullsized":"2024-02-12T17:07:22.469990+0000","mapping_epoch":441,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'21","last_scrub_stamp":"2024-02-12T17:06:32.977637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:06:32.977637+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":106,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.407576+0000","last_change":"2024-02-12T17:06:24.976140+0000","last_active":"2024-02-12T17:07:18.407576+0000","last_peered":"2024-02-12T17:07:18.407576+0000","last_clean":"2024-02-12T17:07:18.407576+0000","last_became_active":"2024-02-12T17:04:39.889588+0000","last_became_peered":"2024-02-12T17:04:39.889588+0000","last_unstale":"2024-02-12T17:07:18.407576+0000","last_undegraded":"2024-02-12T17:07:18.407576+0000","last_fullsized":"2024-02-12T17:07:18.407576+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:24.976048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:06:24.976048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":169,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423053+0000","last_change":"2024-02-12T17:05:55.345164+0000","last_active":"2024-02-12T17:07:18.423053+0000","last_peered":"2024-02-12T17:07:18.423053+0000","last_clean":"2024-02-12T17:07:18.423053+0000","last_became_active":"2024-02-12T17:04:39.888473+0000","last_became_peered":"2024-02-12T17:04:39.888473+0000","last_unstale":"2024-02-12T17:07:18.423053+0000","last_undegraded":"2024-02-12T17:07:18.423053+0000","last_fullsized":"2024-02-12T17:07:18.423053+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:05:55.344999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.344999+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"470'5759","reported_seq":5786,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:23.709115+0000","last_change":"2024-02-12T17:06:18.392670+0000","last_active":"2024-02-12T17:07:23.709115+0000","last_peered":"2024-02-12T17:07:23.709115+0000","last_clean":"2024-02-12T17:07:23.709115+0000","last_became_active":"2024-02-12T17:06:18.392421+0000","last_became_peered":"2024-02-12T17:06:18.392421+0000","last_unstale":"2024-02-12T17:07:23.709115+0000","last_undegraded":"2024-02-12T17:07:23.709115+0000","last_fullsized":"2024-02-12T17:07:23.709115+0000","mapping_epoch":451,"log_start":"460'2600","ondisk_log_start":"460'2600","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3159,"ondisk_log_size":3159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":377421824,"num_objects":5759,"num_object_clones":0,"num_object_copies":17277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":368576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":480,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415466+0000","last_change":"2024-02-12T17:04:39.887836+0000","last_active":"2024-02-12T17:07:18.415466+0000","last_peered":"2024-02-12T17:07:18.415466+0000","last_clean":"2024-02-12T17:07:18.415466+0000","last_became_active":"2024-02-12T17:04:39.887488+0000","last_became_peered":"2024-02-12T17:04:39.887488+0000","last_unstale":"2024-02-12T17:07:18.415466+0000","last_undegraded":"2024-02-12T17:07:18.415466+0000","last_fullsized":"2024-02-12T17:07:18.415466+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"435'21","reported_seq":727,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:22.470045+0000","last_change":"2024-02-12T17:05:43.541152+0000","last_active":"2024-02-12T17:07:22.470045+0000","last_peered":"2024-02-12T17:07:22.470045+0000","last_clean":"2024-02-12T17:07:22.470045+0000","last_became_active":"2024-02-12T17:05:43.540504+0000","last_became_peered":"2024-02-12T17:05:43.540504+0000","last_unstale":"2024-02-12T17:07:22.470045+0000","last_undegraded":"2024-02-12T17:07:22.470045+0000","last_fullsized":"2024-02-12T17:07:22.470045+0000","mapping_epoch":434,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-12T17:02:20.866591+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:02:20.866591+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":106,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:19.079250+0000","last_change":"2024-02-12T17:07:19.079224+0000","last_active":"2024-02-12T17:07:19.079250+0000","last_peered":"2024-02-12T17:07:19.079250+0000","last_clean":"2024-02-12T17:07:19.079250+0000","last_became_active":"2024-02-12T17:04:39.886222+0000","last_became_peered":"2024-02-12T17:04:39.886222+0000","last_unstale":"2024-02-12T17:07:19.079250+0000","last_undegraded":"2024-02-12T17:07:19.079250+0000","last_fullsized":"2024-02-12T17:07:19.079250+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":172,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423991+0000","last_change":"2024-02-12T17:04:43.226687+0000","last_active":"2024-02-12T17:07:18.423991+0000","last_peered":"2024-02-12T17:07:18.423991+0000","last_clean":"2024-02-12T17:07:18.423991+0000","last_became_active":"2024-02-12T17:04:39.892266+0000","last_became_peered":"2024-02-12T17:04:39.892266+0000","last_unstale":"2024-02-12T17:07:18.423991+0000","last_undegraded":"2024-02-12T17:07:18.423991+0000","last_fullsized":"2024-02-12T17:07:18.423991+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:01:41.694875+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:01:41.694875+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"470'5849","reported_seq":5876,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.525176+0000","last_change":"2024-02-12T17:06:18.366111+0000","last_active":"2024-02-12T17:07:28.525176+0000","last_peered":"2024-02-12T17:07:28.525176+0000","last_clean":"2024-02-12T17:07:28.525176+0000","last_became_active":"2024-02-12T17:06:18.365846+0000","last_became_peered":"2024-02-12T17:06:18.365846+0000","last_unstale":"2024-02-12T17:07:28.525176+0000","last_undegraded":"2024-02-12T17:07:28.525176+0000","last_fullsized":"2024-02-12T17:07:28.525176+0000","mapping_epoch":451,"log_start":"460'2400","ondisk_log_start":"460'2400","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3449,"ondisk_log_size":3449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":383320064,"num_objects":5849,"num_object_clones":0,"num_object_copies":17547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5849,"num_write_kb":374336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":212,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415666+0000","last_change":"2024-02-12T17:05:57.078039+0000","last_active":"2024-02-12T17:07:18.415666+0000","last_peered":"2024-02-12T17:07:18.415666+0000","last_clean":"2024-02-12T17:07:18.415666+0000","last_became_active":"2024-02-12T17:04:39.884295+0000","last_became_peered":"2024-02-12T17:04:39.884295+0000","last_unstale":"2024-02-12T17:07:18.415666+0000","last_undegraded":"2024-02-12T17:07:18.415666+0000","last_fullsized":"2024-02-12T17:07:18.415666+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.077971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":290,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:21.383979+0000","last_change":"2024-02-12T17:05:40.431666+0000","last_active":"2024-02-12T17:07:21.383979+0000","last_peered":"2024-02-12T17:07:21.383979+0000","last_clean":"2024-02-12T17:07:21.383979+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:07:21.383979+0000","last_undegraded":"2024-02-12T17:07:21.383979+0000","last_fullsized":"2024-02-12T17:07:21.383979+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'11","last_scrub_stamp":"2024-02-12T17:02:31.546965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:31.546965+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":131,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422947+0000","last_change":"2024-02-12T17:03:50.589204+0000","last_active":"2024-02-12T17:07:18.422947+0000","last_peered":"2024-02-12T17:07:18.422947+0000","last_clean":"2024-02-12T17:07:18.422947+0000","last_became_active":"2024-02-12T17:03:50.588844+0000","last_became_peered":"2024-02-12T17:03:50.588844+0000","last_unstale":"2024-02-12T17:07:18.422947+0000","last_undegraded":"2024-02-12T17:07:18.422947+0000","last_fullsized":"2024-02-12T17:07:18.422947+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:01:29.888942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:01:29.888942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":164,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422720+0000","last_change":"2024-02-12T17:04:42.611814+0000","last_active":"2024-02-12T17:07:18.422720+0000","last_peered":"2024-02-12T17:07:18.422720+0000","last_clean":"2024-02-12T17:07:18.422720+0000","last_became_active":"2024-02-12T17:04:39.891435+0000","last_became_peered":"2024-02-12T17:04:39.891435+0000","last_unstale":"2024-02-12T17:07:18.422720+0000","last_undegraded":"2024-02-12T17:07:18.422720+0000","last_fullsized":"2024-02-12T17:07:18.422720+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"470'6009","reported_seq":6036,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.338466+0000","last_change":"2024-02-12T17:06:18.392042+0000","last_active":"2024-02-12T17:07:28.338466+0000","last_peered":"2024-02-12T17:07:28.338466+0000","last_clean":"2024-02-12T17:07:28.338466+0000","last_became_active":"2024-02-12T17:06:18.391725+0000","last_became_peered":"2024-02-12T17:06:18.391725+0000","last_unstale":"2024-02-12T17:07:28.338466+0000","last_undegraded":"2024-02-12T17:07:28.338466+0000","last_fullsized":"2024-02-12T17:07:28.338466+0000","mapping_epoch":451,"log_start":"460'2700","ondisk_log_start":"460'2700","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3309,"ondisk_log_size":3309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":393805824,"num_objects":6009,"num_object_clones":0,"num_object_copies":18027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6009,"num_write_kb":384576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":163,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424086+0000","last_change":"2024-02-12T17:06:26.583756+0000","last_active":"2024-02-12T17:07:18.424086+0000","last_peered":"2024-02-12T17:07:18.424086+0000","last_clean":"2024-02-12T17:07:18.424086+0000","last_became_active":"2024-02-12T17:04:39.874748+0000","last_became_peered":"2024-02-12T17:04:39.874748+0000","last_unstale":"2024-02-12T17:07:18.424086+0000","last_undegraded":"2024-02-12T17:07:18.424086+0000","last_fullsized":"2024-02-12T17:07:18.424086+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":53,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.420622+0000","last_change":"2024-02-12T17:06:24.653101+0000","last_active":"2024-02-12T17:07:18.420622+0000","last_peered":"2024-02-12T17:07:18.420622+0000","last_clean":"2024-02-12T17:07:18.420622+0000","last_became_active":"2024-02-12T17:05:40.443276+0000","last_became_peered":"2024-02-12T17:05:40.443276+0000","last_unstale":"2024-02-12T17:07:18.420622+0000","last_undegraded":"2024-02-12T17:07:18.420622+0000","last_fullsized":"2024-02-12T17:07:18.420622+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:24.653029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:06:24.653029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":452,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408019+0000","last_change":"2024-02-12T17:06:51.877944+0000","last_active":"2024-02-12T17:07:18.408019+0000","last_peered":"2024-02-12T17:07:18.408019+0000","last_clean":"2024-02-12T17:07:18.408019+0000","last_became_active":"2024-02-12T17:04:39.888466+0000","last_became_peered":"2024-02-12T17:04:39.888466+0000","last_unstale":"2024-02-12T17:07:18.408019+0000","last_undegraded":"2024-02-12T17:07:18.408019+0000","last_fullsized":"2024-02-12T17:07:18.408019+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.877830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.877830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":184,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415050+0000","last_change":"2024-02-12T17:04:41.383385+0000","last_active":"2024-02-12T17:07:18.415050+0000","last_peered":"2024-02-12T17:07:18.415050+0000","last_clean":"2024-02-12T17:07:18.415050+0000","last_became_active":"2024-02-12T17:04:39.887688+0000","last_became_peered":"2024-02-12T17:04:39.887688+0000","last_unstale":"2024-02-12T17:07:18.415050+0000","last_undegraded":"2024-02-12T17:07:18.415050+0000","last_fullsized":"2024-02-12T17:07:18.415050+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:01:35.160145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:35.160145+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"470'5951","reported_seq":5978,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.573608+0000","last_change":"2024-02-12T17:06:18.392208+0000","last_active":"2024-02-12T17:07:28.573608+0000","last_peered":"2024-02-12T17:07:28.573608+0000","last_clean":"2024-02-12T17:07:28.573608+0000","last_became_active":"2024-02-12T17:06:18.391699+0000","last_became_peered":"2024-02-12T17:06:18.391699+0000","last_unstale":"2024-02-12T17:07:28.573608+0000","last_undegraded":"2024-02-12T17:07:28.573608+0000","last_fullsized":"2024-02-12T17:07:28.573608+0000","mapping_epoch":451,"log_start":"460'2500","ondisk_log_start":"460'2500","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3451,"ondisk_log_size":3451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":390004736,"num_objects":5951,"num_object_clones":0,"num_object_copies":17853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":380864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":488,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.407669+0000","last_change":"2024-02-12T17:05:57.890025+0000","last_active":"2024-02-12T17:07:18.407669+0000","last_peered":"2024-02-12T17:07:18.407669+0000","last_clean":"2024-02-12T17:07:18.407669+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:07:18.407669+0000","last_undegraded":"2024-02-12T17:07:18.407669+0000","last_fullsized":"2024-02-12T17:07:18.407669+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.889955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.889955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":385,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:20.629582+0000","last_change":"2024-02-12T17:07:20.629548+0000","last_active":"2024-02-12T17:07:20.629582+0000","last_peered":"2024-02-12T17:07:20.629582+0000","last_clean":"2024-02-12T17:07:20.629582+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:07:20.629582+0000","last_undegraded":"2024-02-12T17:07:20.629582+0000","last_fullsized":"2024-02-12T17:07:20.629582+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.629467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.629467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":197,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.415043+0000","last_change":"2024-02-12T17:06:38.925744+0000","last_active":"2024-02-12T17:07:18.415043+0000","last_peered":"2024-02-12T17:07:18.415043+0000","last_clean":"2024-02-12T17:07:18.415043+0000","last_became_active":"2024-02-12T17:03:50.551804+0000","last_became_peered":"2024-02-12T17:03:50.551804+0000","last_unstale":"2024-02-12T17:07:18.415043+0000","last_undegraded":"2024-02-12T17:07:18.415043+0000","last_fullsized":"2024-02-12T17:07:18.415043+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":376,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408348+0000","last_change":"2024-02-12T17:03:52.389245+0000","last_active":"2024-02-12T17:07:18.408348+0000","last_peered":"2024-02-12T17:07:18.408348+0000","last_clean":"2024-02-12T17:07:18.408348+0000","last_became_active":"2024-02-12T17:03:50.574562+0000","last_became_peered":"2024-02-12T17:03:50.574562+0000","last_unstale":"2024-02-12T17:07:18.408348+0000","last_undegraded":"2024-02-12T17:07:18.408348+0000","last_fullsized":"2024-02-12T17:07:18.408348+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:02:45.594719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:45.594719+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"470'5552","reported_seq":5579,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:23.759547+0000","last_change":"2024-02-12T17:06:18.394566+0000","last_active":"2024-02-12T17:07:23.759547+0000","last_peered":"2024-02-12T17:07:23.759547+0000","last_clean":"2024-02-12T17:07:23.759547+0000","last_became_active":"2024-02-12T17:06:18.394280+0000","last_became_peered":"2024-02-12T17:06:18.394280+0000","last_unstale":"2024-02-12T17:07:23.759547+0000","last_undegraded":"2024-02-12T17:07:23.759547+0000","last_fullsized":"2024-02-12T17:07:23.759547+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5552,"ondisk_log_size":5552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":363855872,"num_objects":5552,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":355328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":172,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.408097+0000","last_change":"2024-02-12T17:06:43.014800+0000","last_active":"2024-02-12T17:07:18.408097+0000","last_peered":"2024-02-12T17:07:18.408097+0000","last_clean":"2024-02-12T17:07:18.408097+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:07:18.408097+0000","last_undegraded":"2024-02-12T17:07:18.408097+0000","last_fullsized":"2024-02-12T17:07:18.408097+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:06:43.014737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":377,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.422325+0000","last_change":"2024-02-12T17:04:39.886902+0000","last_active":"2024-02-12T17:07:18.422325+0000","last_peered":"2024-02-12T17:07:18.422325+0000","last_clean":"2024-02-12T17:07:18.422325+0000","last_became_active":"2024-02-12T17:04:39.886403+0000","last_became_peered":"2024-02-12T17:04:39.886403+0000","last_unstale":"2024-02-12T17:07:18.422325+0000","last_undegraded":"2024-02-12T17:07:18.422325+0000","last_fullsized":"2024-02-12T17:07:18.422325+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":153,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.424026+0000","last_change":"2024-02-12T17:06:41.443780+0000","last_active":"2024-02-12T17:07:18.424026+0000","last_peered":"2024-02-12T17:07:18.424026+0000","last_clean":"2024-02-12T17:07:18.424026+0000","last_became_active":"2024-02-12T17:04:39.889999+0000","last_became_peered":"2024-02-12T17:04:39.889999+0000","last_unstale":"2024-02-12T17:07:18.424026+0000","last_undegraded":"2024-02-12T17:07:18.424026+0000","last_fullsized":"2024-02-12T17:07:18.424026+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":424,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423464+0000","last_change":"2024-02-12T17:04:42.178594+0000","last_active":"2024-02-12T17:07:18.423464+0000","last_peered":"2024-02-12T17:07:18.423464+0000","last_clean":"2024-02-12T17:07:18.423464+0000","last_became_active":"2024-02-12T17:04:39.886809+0000","last_became_peered":"2024-02-12T17:04:39.886809+0000","last_unstale":"2024-02-12T17:07:18.423464+0000","last_undegraded":"2024-02-12T17:07:18.423464+0000","last_fullsized":"2024-02-12T17:07:18.423464+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:01:28.096708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:01:28.096708+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"470'6008","reported_seq":6035,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:28.338927+0000","last_change":"2024-02-12T17:06:18.395214+0000","last_active":"2024-02-12T17:07:28.338927+0000","last_peered":"2024-02-12T17:07:28.338927+0000","last_clean":"2024-02-12T17:07:28.338927+0000","last_became_active":"2024-02-12T17:06:18.394930+0000","last_became_peered":"2024-02-12T17:06:18.394930+0000","last_unstale":"2024-02-12T17:07:28.338927+0000","last_undegraded":"2024-02-12T17:07:28.338927+0000","last_fullsized":"2024-02-12T17:07:28.338927+0000","mapping_epoch":451,"log_start":"460'2700","ondisk_log_start":"460'2700","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3308,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":393740288,"num_objects":6008,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6008,"num_write_kb":384512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":462,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413750+0000","last_change":"2024-02-12T17:06:47.945378+0000","last_active":"2024-02-12T17:07:18.413750+0000","last_peered":"2024-02-12T17:07:18.413750+0000","last_clean":"2024-02-12T17:07:18.413750+0000","last_became_active":"2024-02-12T17:04:41.899598+0000","last_became_peered":"2024-02-12T17:04:41.899598+0000","last_unstale":"2024-02-12T17:07:18.413750+0000","last_undegraded":"2024-02-12T17:07:18.413750+0000","last_fullsized":"2024-02-12T17:07:18.413750+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:47.945301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:47.945301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":136,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.423131+0000","last_change":"2024-02-12T17:06:53.648932+0000","last_active":"2024-02-12T17:07:18.423131+0000","last_peered":"2024-02-12T17:07:18.423131+0000","last_clean":"2024-02-12T17:07:18.423131+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:07:18.423131+0000","last_undegraded":"2024-02-12T17:07:18.423131+0000","last_fullsized":"2024-02-12T17:07:18.423131+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":539,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-02-12T17:07:18.413598+0000","last_change":"2024-02-12T17:04:40.522253+0000","last_active":"2024-02-12T17:07:18.413598+0000","last_peered":"2024-02-12T17:07:18.413598+0000","last_clean":"2024-02-12T17:07:18.413598+0000","last_became_active":"2024-02-12T17:04:39.882260+0000","last_became_peered":"2024-02-12T17:04:39.882260+0000","last_unstale":"2024-02-12T17:07:18.413598+0000","last_undegraded":"2024-02-12T17:07:18.413598+0000","last_fullsized":"2024-02-12T17:07:18.413598+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:02:43.665248+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:02:43.665248+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"443'18","reported_seq":746,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-12T17:07:21.383635+0000","last_change":"2024-02-12T17:06:35.066954+0000","last_active":"2024-02-12T17:07:21.383635+0000","last_peered":"2024-02-12T17:07:21.383635+0000","last_clean":"2024-02-12T17:07:21.383635+0000","last_became_active":"2024-02-12T17:05:55.356115+0000","last_became_peered":"2024-02-12T17:05:55.356115+0000","last_unstale":"2024-02-12T17:07:21.383635+0000","last_undegraded":"2024-02-12T17:07:21.383635+0000","last_fullsized":"2024-02-12T17:07:21.383635+0000","mapping_epoch":441,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-02-12T17:06:35.066866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.066866+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6119292928,"num_objects":93373,"num_object_clones":0,"num_object_copies":280119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93373,"num_write_kb":5975872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18483904512,"data_stored":18483904512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57673,"ondisk_log_size":57673,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":6},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":367,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1093,"num_read_kb":850,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":11090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":735,"ondisk_log_size":735,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234200,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4482116,"kb_used_data":3582460,"kb_used_omap":0,"kb_used_meta":899648,"kb_avail":89263036,"statfs":{"total":95995035648,"available":91405348864,"internally_reserved":0,"allocated":3668439040,"data_stored":3667500092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":921239552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":2.4670000000000001,"15min":1.994},"min":{"1min":0.23899999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.02,"5min":12.375,"15min":36.439},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.9570000000000001,"15min":2.044},"min":{"1min":0.29099999999999998,"5min":0.22,"15min":0.19600000000000001},"max":{"1min":2.581,"5min":8.5039999999999996,"15min":36.490000000000002},"last":0.45200000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":2.5539999999999998,"15min":1.806},"min":{"1min":0.27300000000000002,"5min":0.23400000000000001,"15min":0.20899999999999999},"max":{"1min":1.544,"5min":13.224,"15min":13.224},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":2.0110000000000001,"15min":1.698},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.22700000000000001},"max":{"1min":2.6139999999999999,"5min":10.194000000000001,"15min":17.324999999999999},"last":0.499}]},{"osd":2,"last update":"Mon Feb 12 17:07:00 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":2.1960000000000002,"15min":1.5860000000000001},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":3.5110000000000001,"5min":12.4,"15min":12.4},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.2330000000000001,"5min":2.794,"15min":2.024},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.23000000000000001},"max":{"1min":8.673,"5min":13.127000000000001,"15min":20.338000000000001},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":2.375,"15min":1.9059999999999999},"min":{"1min":0.34699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.9689999999999999,"5min":13.608000000000001,"15min":20.433},"last":1.554},{"interface":"front","average":{"1min":1.056,"5min":2.9940000000000002,"15min":2.2850000000000001},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.25600000000000001},"max":{"1min":3.4169999999999998,"5min":31.152000000000001,"15min":31.152000000000001},"last":0.81899999999999995}]},{"osd":4,"last update":"Mon Feb 12 17:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":2.9060000000000001,"15min":2.004},"min":{"1min":0.30199999999999999,"5min":0.23499999999999999,"15min":0.23100000000000001},"max":{"1min":3.484,"5min":19.030000000000001,"15min":50.406999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":3.206,"15min":2.4009999999999998},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.23400000000000001},"max":{"1min":2.0819999999999999,"5min":16.867000000000001,"15min":50.168999999999997},"last":0.78900000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":3.3980000000000001,"15min":2.085},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.24199999999999999},"max":{"1min":2.9169999999999998,"5min":19.702000000000002,"15min":19.702000000000002},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":3.0209999999999999,"15min":2.1779999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":1.73,"5min":17.585999999999999,"15min":18.651},"last":0.752}]},{"osd":6,"last update":"Mon Feb 12 17:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":2.411,"15min":1.8},"min":{"1min":0.33600000000000002,"5min":0.27800000000000002,"15min":0.27400000000000002},"max":{"1min":2.645,"5min":10.522,"15min":17.234999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":2.641,"15min":1.8029999999999999},"min":{"1min":0.34100000000000003,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.9449999999999998,"5min":13.676,"15min":13.676},"last":0.48699999999999999}]}]},{"osd":6,"up_from":0,"seq":193273528542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834575095,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738618,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1623884,"kb_used_data":715396,"kb_used_omap":0,"kb_used_meta":908480,"kb_avail":92121268,"statfs":{"total":95995035648,"available":94332178432,"internally_reserved":0,"allocated":732565504,"data_stored":731745656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":930283520},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.0760000000000001,"15min":1.3999999999999999},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":3.1850000000000001,"5min":6.2939999999999996,"15min":23.510999999999999},"last":0.44},{"interface":"front","average":{"1min":1.264,"5min":1.448,"15min":1.4450000000000001},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.193},"max":{"1min":10.009,"5min":16.16,"15min":16.16},"last":0.98199999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:07:15 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.147,"15min":1.5860000000000001},"min":{"1min":0.33700000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2270000000000003,"5min":6.2270000000000003,"15min":16.521000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.5780000000000001,"15min":1.581},"min":{"1min":0.254,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":8.6240000000000006,"5min":9.827,"15min":10.340999999999999},"last":1.0109999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:06:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.468,"15min":1.7370000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":9.8819999999999997,"5min":9.8819999999999997,"15min":21.100000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.5960000000000001,"15min":1.7669999999999999},"min":{"1min":0.32500000000000001,"5min":0.221,"15min":0.221},"max":{"1min":4.3730000000000002,"5min":16.759,"15min":16.759},"last":4.1849999999999996}]},{"osd":4,"last update":"Mon Feb 12 17:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.5760000000000001,"15min":1.6599999999999999},"min":{"1min":0.41099999999999998,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":3.851,"5min":7.8639999999999999,"15min":16.888999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.014,"5min":1.837,"15min":1.696},"min":{"1min":0.45700000000000002,"5min":0.28799999999999998,"15min":0.28399999999999997},"max":{"1min":3.5329999999999999,"5min":9.1020000000000003,"15min":16.899999999999999},"last":0.88}]},{"osd":5,"last update":"Mon Feb 12 17:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.9870000000000001,"15min":2.25},"min":{"1min":0.318,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":3.222,"5min":10.146000000000001,"15min":50.404000000000003},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.874,"5min":2.25,"15min":2.2360000000000002},"min":{"1min":0.36699999999999999,"5min":0.28599999999999998,"15min":0.28299999999999997},"max":{"1min":2.8050000000000002,"5min":19.751999999999999,"15min":49.979999999999997},"last":0.48499999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.714,"15min":1.744},"min":{"1min":0.42399999999999999,"5min":0.35199999999999998,"15min":0.29999999999999999},"max":{"1min":8.9570000000000007,"5min":8.9570000000000007,"15min":17.907},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.9079999999999999,"15min":1.996},"min":{"1min":0.376,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":3.9449999999999998,"5min":7.7510000000000003,"15min":21.062000000000001},"last":0.86699999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:07:15 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":2.4140000000000001,"15min":2.0409999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.32400000000000001},"max":{"1min":6.1909999999999998,"5min":19.373000000000001,"15min":19.373000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.222,"5min":2.1429999999999998,"15min":2.0259999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.33600000000000002},"max":{"1min":4.4009999999999998,"5min":8.923,"15min":34.722999999999999},"last":0.91300000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411568,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6574872,"kb_used_data":5686288,"kb_used_omap":0,"kb_used_meta":888576,"kb_avail":87170280,"statfs":{"total":95995035648,"available":89262366720,"internally_reserved":0,"allocated":5822758912,"data_stored":5821863538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":909901824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.4099999999999999,"15min":1.7130000000000001},"min":{"1min":0.20899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.7229999999999999,"5min":36.100999999999999,"15min":36.100999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.802,"5min":2.0259999999999998,"15min":1.788},"min":{"1min":0.23000000000000001,"5min":0.215,"15min":0.158},"max":{"1min":7.2469999999999999,"5min":36.299999999999997,"15min":36.299999999999997},"last":2.891}]},{"osd":1,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.0389999999999999,"15min":1.631},"min":{"1min":0.23300000000000001,"5min":0.22700000000000001,"15min":0.191},"max":{"1min":4.1310000000000002,"5min":9.609,"15min":17.739000000000001},"last":2.7610000000000001},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.1279999999999999,"15min":1.6950000000000001},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.223},"max":{"1min":5.4180000000000001,"5min":7.8609999999999998,"15min":15.766999999999999},"last":3.101}]},{"osd":3,"last update":"Mon Feb 12 17:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.2989999999999999,"15min":1.855},"min":{"1min":0.31,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":8.8209999999999997,"5min":8.8209999999999997,"15min":51.579999999999998},"last":3.1280000000000001},{"interface":"front","average":{"1min":1.01,"5min":1.2649999999999999,"15min":1.589},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.23200000000000001},"max":{"1min":3.1699999999999999,"5min":5.4180000000000001,"15min":48.988999999999997},"last":2.859}]},{"osd":4,"last update":"Mon Feb 12 17:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":1.6279999999999999,"15min":1.867},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.748,"5min":18.34,"15min":18.658999999999999},"last":0.502},{"interface":"front","average":{"1min":0.96799999999999997,"5min":2.0800000000000001,"15min":1.9430000000000001},"min":{"1min":0.41999999999999998,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":2.6000000000000001,"5min":18.355,"15min":18.355},"last":0.624}]},{"osd":5,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":2.0110000000000001,"15min":1.9990000000000001},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.27300000000000002},"max":{"1min":3.6059999999999999,"5min":32.57,"15min":32.57},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.2450000000000001,"5min":2.3570000000000002,"15min":2.278},"min":{"1min":0.314,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":4.226,"5min":31.905999999999999,"15min":31.905999999999999},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Feb 12 17:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.577,"15min":2.052},"min":{"1min":0.36399999999999999,"5min":0.33000000000000002,"15min":0.30199999999999999},"max":{"1min":8.9969999999999999,"5min":8.9969999999999999,"15min":17.782},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.651,"15min":1.804},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.29799999999999999},"max":{"1min":3.9199999999999999,"5min":10.6,"15min":13.446999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":3.5190000000000001,"15min":2.46},"min":{"1min":0.39800000000000002,"5min":0.33700000000000002,"15min":0.32600000000000001},"max":{"1min":4.0609999999999999,"5min":45.468000000000004,"15min":45.468000000000004},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.3440000000000001,"5min":3.1440000000000001,"15min":2.5089999999999999},"min":{"1min":0.39300000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.2910000000000004,"5min":46.656999999999996,"15min":46.656999999999996},"last":0.59899999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149932,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4457400,"kb_used_data":4160176,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":89287752,"statfs":{"total":95995035648,"available":91430658048,"internally_reserved":0,"allocated":4260020224,"data_stored":4259093459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.5009999999999999,"15min":1.675},"min":{"1min":0.26900000000000002,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.9079999999999999,"5min":58.451999999999998,"15min":58.451999999999998},"last":4.5970000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.9119999999999999,"15min":1.7729999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.452,"5min":57.497999999999998,"15min":57.497999999999998},"last":3.1219999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.296,"15min":1.508},"min":{"1min":0.30599999999999999,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":3.5,"5min":11.537000000000001,"15min":13.94},"last":2.3439999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.96099999999999997,"15min":1.3160000000000001},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":0.73199999999999998,"5min":10.278,"15min":14.962},"last":0.432}]},{"osd":2,"last update":"Mon Feb 12 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":1.171,"15min":1.6220000000000001},"min":{"1min":0.28899999999999998,"5min":0.24099999999999999,"15min":0.23400000000000001},"max":{"1min":2.008,"5min":10.302,"15min":49.112000000000002},"last":3.4510000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.96099999999999997,"15min":1.708},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.1539999999999999,"5min":6.4960000000000004,"15min":51.997999999999998},"last":4.5739999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.6040000000000001,"15min":1.893},"min":{"1min":0.30599999999999999,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":19.010000000000002},"last":2.2250000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":1.552,"15min":1.7290000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28799999999999998},"max":{"1min":1.536,"5min":10.250999999999999,"15min":19.033999999999999},"last":0.46100000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:07:20 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.919,"15min":2.0510000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.6429999999999998,"5min":14.236000000000001,"15min":19.882999999999999},"last":4.4630000000000001},{"interface":"front","average":{"1min":2.3559999999999999,"5min":2.3490000000000002,"15min":2.2160000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":7.6959999999999997,"5min":13.927,"15min":15.595000000000001},"last":2.198}]},{"osd":6,"last update":"Mon Feb 12 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.5860000000000001,"15min":1.5589999999999999},"min":{"1min":0.39000000000000001,"5min":0.38900000000000001,"15min":0.34100000000000003},"max":{"1min":3.246,"5min":10.212999999999999,"15min":12.856999999999999},"last":3.4169999999999998},{"interface":"front","average":{"1min":1.123,"5min":1.2889999999999999,"15min":1.7190000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.27500000000000002},"max":{"1min":8.1489999999999991,"5min":8.5039999999999996,"15min":19.925999999999998},"last":4.5380000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":2.5699999999999998,"15min":2.2629999999999999},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":6.7110000000000003,"5min":37.719999999999999,"15min":37.719999999999999},"last":2.2530000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":2.605,"15min":2.2189999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.1469999999999998,"5min":38.948,"15min":38.948},"last":4.5090000000000003}]}]},{"osd":4,"up_from":0,"seq":137438953704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757476,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4693112,"kb_used_data":3792432,"kb_used_omap":0,"kb_used_meta":900672,"kb_avail":89052040,"statfs":{"total":95995035648,"available":91189288960,"internally_reserved":0,"allocated":3883450368,"data_stored":3882522728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922288128},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.4239999999999999,"15min":1.8320000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.375,"5min":10.971,"15min":46.365000000000002},"last":8.5340000000000007},{"interface":"front","average":{"1min":0.86899999999999999,"5min":1.9179999999999999,"15min":2.2440000000000002},"min":{"1min":0.22500000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":3.1320000000000001,"5min":12.140000000000001,"15min":46.134},"last":7.3390000000000004}]},{"osd":1,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.482,"15min":1.6879999999999999},"min":{"1min":0.28299999999999997,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":3.5800000000000001,"5min":11.19,"15min":16.655000000000001},"last":2.0880000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.958,"15min":2.016},"min":{"1min":0.312,"5min":0.23799999999999999,"15min":0.21199999999999999},"max":{"1min":2.444,"5min":12.154,"15min":16.712},"last":2.1589999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.6259999999999999,"15min":1.9990000000000001},"min":{"1min":0.32200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.8899999999999997,"5min":11.007999999999999,"15min":18.218},"last":2.028},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.573,"15min":1.8580000000000001},"min":{"1min":0.30299999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.823,"5min":8.0470000000000006,"15min":18.327000000000002},"last":1.929}]},{"osd":3,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.5309999999999999,"15min":1.879},"min":{"1min":0.41499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.738,"5min":11.045,"15min":18.251000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.6399999999999999,"15min":1.794},"min":{"1min":0.34499999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3530000000000002,"5min":8.0790000000000006,"15min":20.263000000000002},"last":0.32200000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":2.181,"15min":1.8680000000000001},"min":{"1min":0.252,"5min":0.251,"15min":0.23699999999999999},"max":{"1min":1.835,"5min":12.656000000000001,"15min":12.656000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":1.8009999999999999,"15min":1.962},"min":{"1min":0.29399999999999998,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":1.4399999999999999,"5min":11.068,"15min":20.550000000000001},"last":0.432}]},{"osd":6,"last update":"Mon Feb 12 17:06:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":2.0169999999999999,"15min":1.79},"min":{"1min":0.33700000000000002,"5min":0.29199999999999998,"15min":0.26400000000000001},"max":{"1min":1.7969999999999999,"5min":12.121,"15min":12.281000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.8959999999999999,"15min":1.855},"min":{"1min":0.30499999999999999,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":4.0910000000000002,"5min":11.087,"15min":22.035},"last":0.38800000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":2.2759999999999998,"15min":1.9570000000000001},"min":{"1min":0.317,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.8560000000000001,"5min":13.647,"15min":19.184999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.625,"5min":2.7160000000000002,"15min":2.0430000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.26800000000000002},"max":{"1min":1.8660000000000001,"5min":20.274000000000001,"15min":20.274000000000001},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":731643904,"data_stored":731643904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5858918400,"data_stored":5858918400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292870144,"data_stored":4292870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3907846144,"data_stored":3907846144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3692625920,"data_stored":3692625920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:07:38.553 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T17:07:38.553 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-02-12T17:07:38.554 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:07:38.631 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:38.855 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:07:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:38.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:07:38.900 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 74 16 108152 108136 91.2657 76.1875 0.0112548 0.0109529 2024-02-12T17:07:38.965 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:39.038 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:37.769979+0000 osd.5 (osd.5) 235 : cluster [DBG] 4.d scrub starts 2024-02-12T17:07:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:37.774463+0000 osd.5 (osd.5) 236 : cluster [DBG] 4.d scrub ok 2024-02-12T17:07:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:37.884933+0000 osd.3 (osd.3) 161 : cluster [DBG] 2.1e scrub starts 2024-02-12T17:07:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:37.889950+0000 osd.3 (osd.3) 162 : cluster [DBG] 2.1e scrub ok 2024-02-12T17:07:39.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:37.769979+0000 osd.5 (osd.5) 235 : cluster [DBG] 4.d scrub starts 2024-02-12T17:07:39.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:37.774463+0000 osd.5 (osd.5) 236 : cluster [DBG] 4.d scrub ok 2024-02-12T17:07:39.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:37.884933+0000 osd.3 (osd.3) 161 : cluster [DBG] 2.1e scrub starts 2024-02-12T17:07:39.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:37.889950+0000 osd.3 (osd.3) 162 : cluster [DBG] 2.1e scrub ok 2024-02-12T17:07:39.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:37.769979+0000 osd.5 (osd.5) 235 : cluster [DBG] 4.d scrub starts 2024-02-12T17:07:39.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:37.774463+0000 osd.5 (osd.5) 236 : cluster [DBG] 4.d scrub ok 2024-02-12T17:07:39.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:37.884933+0000 osd.3 (osd.3) 161 : cluster [DBG] 2.1e scrub starts 2024-02-12T17:07:39.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:37.889950+0000 osd.3 (osd.3) 162 : cluster [DBG] 2.1e scrub ok 2024-02-12T17:07:39.606 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:39.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:39.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:39.748 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:39.772 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:39.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:39.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:39.900 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 75 16 109231 109215 90.9481 67.4375 0.013785 0.01099 2024-02-12T17:07:39.907 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:39.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:39.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:40.013 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:07:40.013 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:07:40.033 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:40.049 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:07:40.086 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:40.091 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:07:40.148 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:07:40.177 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:07:40.222 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:07:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:38.229590+0000 osd.7 (osd.7) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:38.229640+0000 osd.7 (osd.7) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:38.741598+0000 mgr.y (mgr.24481) 683 : cluster [DBG] pgmap v934: 128 pgs: 128 active+clean; 6.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-12T17:07:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:38.744778+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:07:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:38.751052+0000 osd.5 (osd.5) 238 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:07:40.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:38.871106+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e473: 8 total, 8 up, 5 in 2024-02-12T17:07:40.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:38.879811+0000 osd.3 (osd.3) 163 : cluster [DBG] 4.b scrub starts 2024-02-12T17:07:40.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:38.887697+0000 osd.3 (osd.3) 164 : cluster [DBG] 4.b scrub ok 2024-02-12T17:07:40.263 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:07:40.266 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:07:40.266 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:07:40.266 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:07:40.266 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:07:40.266 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:07:40.266 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:07:40.266 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:07:40.267 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:07:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:38.229590+0000 osd.7 (osd.7) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:38.229640+0000 osd.7 (osd.7) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:38.741598+0000 mgr.y (mgr.24481) 683 : cluster [DBG] pgmap v934: 128 pgs: 128 active+clean; 6.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-12T17:07:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:38.744778+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:07:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:38.751052+0000 osd.5 (osd.5) 238 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:07:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:38.871106+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e473: 8 total, 8 up, 5 in 2024-02-12T17:07:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:38.879811+0000 osd.3 (osd.3) 163 : cluster [DBG] 4.b scrub starts 2024-02-12T17:07:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:38.887697+0000 osd.3 (osd.3) 164 : cluster [DBG] 4.b scrub ok 2024-02-12T17:07:40.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:38.229590+0000 osd.7 (osd.7) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:40.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:38.229640+0000 osd.7 (osd.7) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:40.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:38.741598+0000 mgr.y (mgr.24481) 683 : cluster [DBG] pgmap v934: 128 pgs: 128 active+clean; 6.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-12T17:07:40.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:38.744778+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:07:40.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:38.751052+0000 osd.5 (osd.5) 238 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:07:40.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:38.871106+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e473: 8 total, 8 up, 5 in 2024-02-12T17:07:40.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:38.879811+0000 osd.3 (osd.3) 163 : cluster [DBG] 4.b scrub starts 2024-02-12T17:07:40.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:38.887697+0000 osd.3 (osd.3) 164 : cluster [DBG] 4.b scrub ok 2024-02-12T17:07:40.901 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 76 16 110265 110249 90.6019 64.625 0.0163721 0.0110338 2024-02-12T17:07:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:39.227757+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1c scrub starts 2024-02-12T17:07:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:39.235649+0000 osd.7 (osd.7) 235 : cluster [DBG] 2.1c scrub ok 2024-02-12T17:07:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:39.571273+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:41.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:39.571324+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:41.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:39.758813+0000 osd.5 (osd.5) 239 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:07:41.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:39.761885+0000 osd.5 (osd.5) 240 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:07:41.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:39.894024+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e474: 8 total, 8 up, 5 in 2024-02-12T17:07:41.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:39.914395+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.33:0/2876982589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:41.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:40.357359+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.33:0/3944571216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:41.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:40.357982+0000 mon.a (mon.0) 1647 : audit [INF] from='client.25678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:41.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:39.227757+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1c scrub starts 2024-02-12T17:07:41.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:39.235649+0000 osd.7 (osd.7) 235 : cluster [DBG] 2.1c scrub ok 2024-02-12T17:07:41.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:39.571273+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:41.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:39.571324+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:41.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:39.758813+0000 osd.5 (osd.5) 239 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:07:41.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:39.761885+0000 osd.5 (osd.5) 240 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:39.894024+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e474: 8 total, 8 up, 5 in 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:39.914395+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.33:0/2876982589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:40.357359+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.33:0/3944571216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:40.357982+0000 mon.a (mon.0) 1647 : audit [INF] from='client.25678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:39.227757+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1c scrub starts 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:39.235649+0000 osd.7 (osd.7) 235 : cluster [DBG] 2.1c scrub ok 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:39.571273+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:39.571324+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:41.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:39.758813+0000 osd.5 (osd.5) 239 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:07:41.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:39.761885+0000 osd.5 (osd.5) 240 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:07:41.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:39.894024+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e474: 8 total, 8 up, 5 in 2024-02-12T17:07:41.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:39.914395+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.33:0/2876982589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:41.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:40.357359+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.33:0/3944571216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:41.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:40.357982+0000 mon.a (mon.0) 1647 : audit [INF] from='client.25678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:07:41.283 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:07:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:40] "GET /metrics HTTP/1.1" 200 176506 "" "Prometheus/2.33.4" 2024-02-12T17:07:41.852 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:07:41.900 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 77 16 111432 111416 90.3725 72.9375 0.00969328 0.0110608 2024-02-12T17:07:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:40.741977+0000 mgr.y (mgr.24481) 684 : cluster [DBG] pgmap v937: 128 pgs: 128 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 866 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-12T17:07:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:40.891473+0000 mon.a (mon.0) 1648 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:07:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:40.891506+0000 mon.a (mon.0) 1649 : cluster [INF] Cluster is now healthy 2024-02-12T17:07:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:40.911100+0000 mon.a (mon.0) 1650 : audit [INF] from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:07:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:40.911166+0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e475: 8 total, 8 up, 5 in 2024-02-12T17:07:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:40.917287+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.f scrub starts 2024-02-12T17:07:42.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:40.934260+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.f scrub ok 2024-02-12T17:07:42.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:41.376373+0000 mon.a (mon.0) 1652 : cluster [DBG] osdmap e476: 8 total, 8 up, 5 in 2024-02-12T17:07:42.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:41.526905+0000 osd.4 (osd.4) 169 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:42.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:41.526963+0000 osd.4 (osd.4) 170 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:42.266 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:40.741977+0000 mgr.y (mgr.24481) 684 : cluster [DBG] pgmap v937: 128 pgs: 128 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 866 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:40.891473+0000 mon.a (mon.0) 1648 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:40.891506+0000 mon.a (mon.0) 1649 : cluster [INF] Cluster is now healthy 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:40.911100+0000 mon.a (mon.0) 1650 : audit [INF] from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:40.911166+0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e475: 8 total, 8 up, 5 in 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:40.917287+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.f scrub starts 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:40.934260+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.f scrub ok 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:41.376373+0000 mon.a (mon.0) 1652 : cluster [DBG] osdmap e476: 8 total, 8 up, 5 in 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:41.526905+0000 osd.4 (osd.4) 169 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:41.526963+0000 osd.4 (osd.4) 170 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:42.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:40.741977+0000 mgr.y (mgr.24481) 684 : cluster [DBG] pgmap v937: 128 pgs: 128 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 866 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:40.891473+0000 mon.a (mon.0) 1648 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:40.891506+0000 mon.a (mon.0) 1649 : cluster [INF] Cluster is now healthy 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:40.911100+0000 mon.a (mon.0) 1650 : audit [INF] from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:40.911166+0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e475: 8 total, 8 up, 5 in 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:40.917287+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.f scrub starts 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:40.934260+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.f scrub ok 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:41.376373+0000 mon.a (mon.0) 1652 : cluster [DBG] osdmap e476: 8 total, 8 up, 5 in 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:41.526905+0000 osd.4 (osd.4) 169 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:41.526963+0000 osd.4 (osd.4) 170 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:42.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:42.411 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:42.442 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:42.452 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:07:42.452 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:07:42.452 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112077 10.9 e477/451 10:93dd77a5:::benchmark_data_smithi033_160181_object112076:head v 477'7150, mlcod=477'7148)", 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.443605+0000", 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0071953410000000001, 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0072178809999999998, 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.443605+0000", 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.453 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.443605+0000", 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.443603+0000", 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2517000000000001e-05 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.443636+0000", 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2979999999999999e-05 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.443649+0000", 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7849999999999997e-06 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.443655+0000", 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012204200000000001 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.444875+0000", 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1498999999999999e-05 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.444897+0000", 2024-02-12T17:07:42.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001291834 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112080 10.1 e477/451 10:886b8132:::benchmark_data_smithi033_160181_object112079:head v 477'7126, mlcod=477'7123)", 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.445395+0000", 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0054046429999999998, 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054740170000000003, 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.445395+0000", 2024-02-12T17:07:42.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.445395+0000", 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.445394+0000", 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4329999999999998e-05 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.445428+0000", 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4499999999999993e-06 2024-02-12T17:07:42.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.445438+0000", 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9510000000000003e-06 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.445444+0000", 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001117547 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446561+0000", 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.107e-05 2024-02-12T17:07:42.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446582+0000", 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011866769999999999 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112078 10.a e477/451 10:524eb46d:::benchmark_data_smithi033_160181_object112077:head v 477'6966, mlcod=477'6964)", 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.446390+0000", 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0044098089999999998, 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045205410000000003, 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.459 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446390+0000", 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446390+0000", 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446389+0000", 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5259000000000002e-05 2024-02-12T17:07:42.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446434+0000", 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2940000000000006e-06 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446443+0000", 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3300000000000004e-06 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446449+0000", 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0666e-05 2024-02-12T17:07:42.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446490+0000", 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8448999999999999e-05 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.446508+0000", 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117723 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112084 10.7 e477/451 10:e8944b75:::benchmark_data_smithi033_160181_object112083:head v 477'7011, mlcod=476'7009)", 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.447497+0000", 2024-02-12T17:07:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0033030630000000002, 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003456279, 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.447497+0000", 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.447497+0000", 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.447496+0000", 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5610999999999999e-05 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.447521+0000", 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2765999999999999e-05 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.447534+0000", 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5699999999999998e-06 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.447540+0000", 2024-02-12T17:07:42.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7393999999999999e-05 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.447608+0000", 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4097999999999999e-05 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.447632+0000", 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000134936 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112079 10.5 e477/451 10:a5632cd3:::benchmark_data_smithi033_160181_object112078:head v 477'6962, mlcod=476'6960)", 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.448573+0000", 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0022271140000000001, 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024377299999999999, 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448573+0000", 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448573+0000", 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:07:42.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448571+0000", 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5848000000000001e-05 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448597+0000", 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4798e-05 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448612+0000", 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9610000000000002e-06 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448618+0000", 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2964e-05 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448711+0000", 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4547e-05 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448726+0000", 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000152645 2024-02-12T17:07:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112081 10.1 e477/451 10:8a6d1924:::benchmark_data_smithi033_160181_object112080:head v 477'7127, mlcod=477'7123)", 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.448640+0000", 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0021600389999999999, 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024103430000000001, 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448640+0000", 2024-02-12T17:07:42.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448640+0000", 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448639+0000", 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7056999999999999e-05 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448666+0000", 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4219999999999997e-06 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448673+0000", 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.318e-06 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.448677+0000", 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080787700000000003 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449485+0000", 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3356000000000001e-05 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449498+0000", 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00085837100000000003 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112085 10.b e477/451 10:d40cec82:::benchmark_data_smithi033_160181_object112084:head v 477'6987, mlcod=477'6985)", 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.449063+0000", 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.001737101, 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020272350000000001, 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449063+0000", 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449063+0000", 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449062+0000", 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1451e-05 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449103+0000", 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0839000000000001e-05 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449114+0000", 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449120+0000", 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2972000000000005e-05 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449193+0000", 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9769e-05 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449213+0000", 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000149534 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112086 10.c e477/451 10:363b40a7:::benchmark_data_smithi033_160181_object112085:head v 477'6970, mlcod=477'6968)", 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.449135+0000", 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.001665183, 2024-02-12T17:07:42.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020018929999999998, 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449135+0000", 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449135+0000", 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449134+0000", 2024-02-12T17:07:42.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4503999999999999e-05 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449189+0000", 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9000000000000006e-06 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449196+0000", 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2039999999999996e-06 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449202+0000", 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001831446 2024-02-12T17:07:42.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.451033+0000", 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8804000000000002e-05 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.451052+0000", 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019169980000000001 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.25603.0:112088 10.d e477/451 10:bf5161b1:::benchmark_data_smithi033_160181_object112087:head v 477'7075, mlcod=477'7073)", 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:42.449708+0000", 2024-02-12T17:07:42.478 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.001092272, 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014759300000000001, 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449708+0000", 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449708+0000", 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:07:42.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449706+0000", 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4967e-05 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449731+0000", 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0428000000000001e-05 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449742+0000", 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0410000000000003e-06 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:07:42.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449748+0000", 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1692e-05 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449789+0000", 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7808e-05 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:42.449807+0000", 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9436000000000001e-05 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 9 2024-02-12T17:07:42.481 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:07:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:42.482 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:42.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:42.583 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:42.669 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:42.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:07:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:07:42.900 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 78 16 112615 112599 90.1619 73.9375 0.00736519 0.0110869 2024-02-12T17:07:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:41.234725+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.d scrub starts 2024-02-12T17:07:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:41.239344+0000 osd.7 (osd.7) 237 : cluster [DBG] 5.d scrub ok 2024-02-12T17:07:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:41.747359+0000 osd.5 (osd.5) 241 : cluster [DBG] 5.e scrub starts 2024-02-12T17:07:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:41.785014+0000 osd.5 (osd.5) 242 : cluster [DBG] 5.e scrub ok 2024-02-12T17:07:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:42.389399+0000 mon.a (mon.0) 1653 : cluster [DBG] osdmap e477: 8 total, 8 up, 5 in 2024-02-12T17:07:43.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:41.234725+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.d scrub starts 2024-02-12T17:07:43.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:41.239344+0000 osd.7 (osd.7) 237 : cluster [DBG] 5.d scrub ok 2024-02-12T17:07:43.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:41.747359+0000 osd.5 (osd.5) 241 : cluster [DBG] 5.e scrub starts 2024-02-12T17:07:43.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:41.785014+0000 osd.5 (osd.5) 242 : cluster [DBG] 5.e scrub ok 2024-02-12T17:07:43.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:42.389399+0000 mon.a (mon.0) 1653 : cluster [DBG] osdmap e477: 8 total, 8 up, 5 in 2024-02-12T17:07:43.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:41.234725+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.d scrub starts 2024-02-12T17:07:43.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:41.239344+0000 osd.7 (osd.7) 237 : cluster [DBG] 5.d scrub ok 2024-02-12T17:07:43.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:41.747359+0000 osd.5 (osd.5) 241 : cluster [DBG] 5.e scrub starts 2024-02-12T17:07:43.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:41.785014+0000 osd.5 (osd.5) 242 : cluster [DBG] 5.e scrub ok 2024-02-12T17:07:43.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:42.389399+0000 mon.a (mon.0) 1653 : cluster [DBG] osdmap e477: 8 total, 8 up, 5 in 2024-02-12T17:07:43.905 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 79 16 113772 113756 89.9314 72.3125 0.0165782 0.0111153 2024-02-12T17:07:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:42.262021+0000 osd.7 (osd.7) 238 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:07:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:42.272803+0000 osd.7 (osd.7) 239 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:07:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:42.742283+0000 mgr.y (mgr.24481) 685 : cluster [DBG] pgmap v941: 128 pgs: 128 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail 2024-02-12T17:07:44.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:42.262021+0000 osd.7 (osd.7) 238 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:07:44.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:42.272803+0000 osd.7 (osd.7) 239 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:07:44.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:42.742283+0000 mgr.y (mgr.24481) 685 : cluster [DBG] pgmap v941: 128 pgs: 128 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail 2024-02-12T17:07:44.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:42.262021+0000 osd.7 (osd.7) 238 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:07:44.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:42.272803+0000 osd.7 (osd.7) 239 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:07:44.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:42.742283+0000 mgr.y (mgr.24481) 685 : cluster [DBG] pgmap v941: 128 pgs: 128 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail 2024-02-12T17:07:44.907 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:07:44.907031+0000 min lat: 0.00118871 max lat: 0.167728 avg lat: 0.0111353 2024-02-12T17:07:44.908 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:07:44.908 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 80 16 115005 114989 89.7677 77.0625 0.0232389 0.0111353 2024-02-12T17:07:44.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:07:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:07:45.456 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:45.466 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:45.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:45.495 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T17:07:45.568 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:45.633 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:45.807 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:45.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:45.908 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 81 16 116132 116116 89.5291 70.4375 0.0134403 0.0111646 2024-02-12T17:07:45.911 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:45.970 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:45.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:46.074 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:46.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:44.252344+0000 osd.7 (osd.7) 240 : cluster [DBG] 2.1f scrub starts 2024-02-12T17:07:46.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:44.261295+0000 osd.7 (osd.7) 241 : cluster [DBG] 2.1f scrub ok 2024-02-12T17:07:46.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:44.505075+0000 mgr.y (mgr.24481) 686 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:46.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:44.743055+0000 mgr.y (mgr.24481) 687 : cluster [DBG] pgmap v942: 128 pgs: 128 active+clean; 6.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:07:46.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:45.564414+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.c scrub starts 2024-02-12T17:07:46.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:45.567335+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.c scrub ok 2024-02-12T17:07:46.118 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:44.252344+0000 osd.7 (osd.7) 240 : cluster [DBG] 2.1f scrub starts 2024-02-12T17:07:46.118 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:44.261295+0000 osd.7 (osd.7) 241 : cluster [DBG] 2.1f scrub ok 2024-02-12T17:07:46.118 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:44.505075+0000 mgr.y (mgr.24481) 686 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:46.118 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:44.743055+0000 mgr.y (mgr.24481) 687 : cluster [DBG] pgmap v942: 128 pgs: 128 active+clean; 6.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:07:46.118 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:45.564414+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.c scrub starts 2024-02-12T17:07:46.118 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:45.567335+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.c scrub ok 2024-02-12T17:07:46.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:44.252344+0000 osd.7 (osd.7) 240 : cluster [DBG] 2.1f scrub starts 2024-02-12T17:07:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:44.261295+0000 osd.7 (osd.7) 241 : cluster [DBG] 2.1f scrub ok 2024-02-12T17:07:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:44.505075+0000 mgr.y (mgr.24481) 686 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:44.743055+0000 mgr.y (mgr.24481) 687 : cluster [DBG] pgmap v942: 128 pgs: 128 active+clean; 6.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-12T17:07:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:45.564414+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.c scrub starts 2024-02-12T17:07:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:45.567335+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.c scrub ok 2024-02-12T17:07:46.437 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:07:46.899 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:46.905 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:07:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:46.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:46.913 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 82 16 117158 117142 89.2159 64.125 0.0202521 0.0112043 2024-02-12T17:07:46.982 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:47.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:47.042 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:47.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:47.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:47.204 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:45.269834+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:07:47.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:45.272931+0000 osd.7 (osd.7) 243 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:07:47.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:46.383733+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e478: 8 total, 8 up, 5 in 2024-02-12T17:07:47.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:46.523595+0000 osd.0 (osd.0) 213 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:07:47.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:46.526742+0000 osd.0 (osd.0) 214 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:07:47.206 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:47.228 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:47.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:47.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:45.269834+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:07:47.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:45.272931+0000 osd.7 (osd.7) 243 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:07:47.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:46.383733+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e478: 8 total, 8 up, 5 in 2024-02-12T17:07:47.246 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:46.523595+0000 osd.0 (osd.0) 213 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:07:47.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:46.526742+0000 osd.0 (osd.0) 214 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:07:47.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:45.269834+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:07:47.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:45.272931+0000 osd.7 (osd.7) 243 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:07:47.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:46.383733+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e478: 8 total, 8 up, 5 in 2024-02-12T17:07:47.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:46.523595+0000 osd.0 (osd.0) 213 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:07:47.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:46.526742+0000 osd.0 (osd.0) 214 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:07:47.332 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:47.365 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:47.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:47.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:47.554 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:47.914 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 83 16 118250 118234 88.9594 68.25 0.0100585 0.0112371 2024-02-12T17:07:47.948 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:46.743454+0000 mgr.y (mgr.24481) 688 : cluster [DBG] pgmap v944: 128 pgs: 128 active+clean; 6.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 878 B/s rd, 60 MiB/s wr, 965 op/s 2024-02-12T17:07:47.949 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:47.496412+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:46.743454+0000 mgr.y (mgr.24481) 688 : cluster [DBG] pgmap v944: 128 pgs: 128 active+clean; 6.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 878 B/s rd, 60 MiB/s wr, 965 op/s 2024-02-12T17:07:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:47.496412+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:48.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:46.743454+0000 mgr.y (mgr.24481) 688 : cluster [DBG] pgmap v944: 128 pgs: 128 active+clean; 6.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 878 B/s rd, 60 MiB/s wr, 965 op/s 2024-02-12T17:07:48.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:47.496412+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:07:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:48.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:48.915 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 84 16 119380 119364 88.741 70.625 0.0240521 0.0112649 2024-02-12T17:07:48.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:48.989 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:48.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:48.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:49.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:49.129 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:49.153 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:49.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:49.214 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:47.250965+0000 osd.7 (osd.7) 244 : cluster [DBG] 4.a scrub starts 2024-02-12T17:07:49.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:47.261484+0000 osd.7 (osd.7) 245 : cluster [DBG] 4.a scrub ok 2024-02-12T17:07:49.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:47.769169+0000 osd.5 (osd.5) 243 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:07:49.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:47.772963+0000 osd.5 (osd.5) 244 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:07:49.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:47.841527+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:07:49.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:47.845922+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:07:49.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:48.484126+0000 osd.0 (osd.0) 215 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:07:49.215 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:48.487294+0000 osd.0 (osd.0) 216 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:07:49.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:47.250965+0000 osd.7 (osd.7) 244 : cluster [DBG] 4.a scrub starts 2024-02-12T17:07:49.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:47.261484+0000 osd.7 (osd.7) 245 : cluster [DBG] 4.a scrub ok 2024-02-12T17:07:49.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:47.769169+0000 osd.5 (osd.5) 243 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:07:49.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:47.772963+0000 osd.5 (osd.5) 244 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:07:49.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:47.841527+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:07:49.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:47.845922+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:07:49.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:48.484126+0000 osd.0 (osd.0) 215 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:07:49.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:48.487294+0000 osd.0 (osd.0) 216 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:07:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:47.250965+0000 osd.7 (osd.7) 244 : cluster [DBG] 4.a scrub starts 2024-02-12T17:07:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:47.261484+0000 osd.7 (osd.7) 245 : cluster [DBG] 4.a scrub ok 2024-02-12T17:07:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:47.769169+0000 osd.5 (osd.5) 243 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:07:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:47.772963+0000 osd.5 (osd.5) 244 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:07:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:47.841527+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:07:49.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:47.845922+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:07:49.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:48.484126+0000 osd.0 (osd.0) 215 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:07:49.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:48.487294+0000 osd.0 (osd.0) 216 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:07:49.259 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:49.286 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:49.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:49.417 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:49.447 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:49.475 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:49.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:49.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:49.610 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:49.637 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:49.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:49.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:49.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:49.915 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 85 16 120456 120440 88.4882 67.25 0.0147474 0.0112966 2024-02-12T17:07:49.971 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:48.202714+0000 osd.7 (osd.7) 246 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:07:49.971 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:48.211439+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:07:49.971 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:48.744087+0000 mgr.y (mgr.24481) 689 : cluster [DBG] pgmap v945: 128 pgs: 128 active+clean; 7.1 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 695 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-02-12T17:07:49.971 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:49.432970+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.33:0/1232015566' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T17:07:49.971 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:49.434652+0000 mon.a (mon.0) 1656 : audit [INF] from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T17:07:50.052 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 10 pg_num to 26 2024-02-12T17:07:50.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:48.202714+0000 osd.7 (osd.7) 246 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:07:50.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:48.211439+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:07:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:48.744087+0000 mgr.y (mgr.24481) 689 : cluster [DBG] pgmap v945: 128 pgs: 128 active+clean; 7.1 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 695 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-02-12T17:07:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:49.432970+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.33:0/1232015566' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T17:07:50.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:49.434652+0000 mon.a (mon.0) 1656 : audit [INF] from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T17:07:50.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:48.202714+0000 osd.7 (osd.7) 246 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:07:50.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:48.211439+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:07:50.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:48.744087+0000 mgr.y (mgr.24481) 689 : cluster [DBG] pgmap v945: 128 pgs: 128 active+clean; 7.1 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 695 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-02-12T17:07:50.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:49.432970+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.33:0/1232015566' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T17:07:50.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:49.434652+0000 mon.a (mon.0) 1656 : audit [INF] from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T17:07:50.915 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 86 16 121600 121584 88.2908 71.5 0.0232529 0.011322 2024-02-12T17:07:51.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:07:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:50] "GET /metrics HTTP/1.1" 200 176505 "" "Prometheus/2.33.4" 2024-02-12T17:07:51.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:49.811412+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:49.813597+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:49.831659+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.a scrub starts 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:49.837318+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.a scrub ok 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:49.986278+0000 mon.a (mon.0) 1657 : audit [INF] from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:49.986370+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e479: 8 total, 8 up, 5 in 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:50.416947+0000 osd.0 (osd.0) 217 : cluster [DBG] 10.8 scrub starts 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:50.744834+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:50.746484+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:49.811412+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:49.813597+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:07:51.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:49.831659+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.a scrub starts 2024-02-12T17:07:51.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:49.837318+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.a scrub ok 2024-02-12T17:07:51.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:49.986278+0000 mon.a (mon.0) 1657 : audit [INF] from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-12T17:07:51.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:49.986370+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e479: 8 total, 8 up, 5 in 2024-02-12T17:07:51.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:50.416947+0000 osd.0 (osd.0) 217 : cluster [DBG] 10.8 scrub starts 2024-02-12T17:07:51.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:50.744834+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:51.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:50.746484+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:49.811412+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:07:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:49.813597+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:07:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:49.831659+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.a scrub starts 2024-02-12T17:07:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:49.837318+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.a scrub ok 2024-02-12T17:07:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:49.986278+0000 mon.a (mon.0) 1657 : audit [INF] from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-12T17:07:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:49.986370+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e479: 8 total, 8 up, 5 in 2024-02-12T17:07:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:50.416947+0000 osd.0 (osd.0) 217 : cluster [DBG] 10.8 scrub starts 2024-02-12T17:07:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:50.744834+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:51.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:50.746484+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:51.607 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:51.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:51.712 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:51.741 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:51.841 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:51.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:51.915 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 87 16 121736 121720 87.3743 8.5 0.00477957 0.011336 2024-02-12T17:07:51.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:07:51.975 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:52.028 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:07:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:52.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:07:52.082 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:52.085 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:52.133 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:52.185 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:52.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:50.744486+0000 mgr.y (mgr.24481) 690 : cluster [DBG] pgmap v947: 128 pgs: 128 active+clean; 7.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-12T17:07:52.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:51.003444+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:07:52.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:51.003493+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e480: 8 total, 8 up, 5 in 2024-02-12T17:07:52.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:50.744486+0000 mgr.y (mgr.24481) 690 : cluster [DBG] pgmap v947: 128 pgs: 128 active+clean; 7.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-12T17:07:52.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:51.003444+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:07:52.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:51.003493+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e480: 8 total, 8 up, 5 in 2024-02-12T17:07:52.394 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:52.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:50.744486+0000 mgr.y (mgr.24481) 690 : cluster [DBG] pgmap v947: 128 pgs: 128 active+clean; 7.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-12T17:07:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:51.003444+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:07:52.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:51.003493+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e480: 8 total, 8 up, 5 in 2024-02-12T17:07:52.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:52.541 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:52.568 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:52.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:52.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:52.729 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:52.825 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:52.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:52.916 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 88 16 122892 122876 87.2019 72.25 0.0048451 0.0114635 2024-02-12T17:07:52.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:52.966 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:07:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:07:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:07:52.970 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:52.996 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:53.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:53.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:53.129 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:53.148 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:53.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:52.029418+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e481: 8 total, 8 up, 5 in 2024-02-12T17:07:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:52.745438+0000 mon.c (mon.2) 248 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:52.748162+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:53.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:53.279 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:53.298 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:53.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:53.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:52.029418+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e481: 8 total, 8 up, 5 in 2024-02-12T17:07:53.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:52.745438+0000 mon.c (mon.2) 248 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:53.326 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:52.748162+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:53.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:52.029418+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e481: 8 total, 8 up, 5 in 2024-02-12T17:07:53.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:52.745438+0000 mon.c (mon.2) 248 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:53.326 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:52.748162+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T17:07:53.402 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:53.431 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:53.916 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 89 16 123149 123133 86.4031 16.0625 0.00237362 0.0114625 2024-02-12T17:07:54.077 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:07:54.261 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:52.241675+0000 osd.7 (osd.7) 248 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:07:54.261 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:52.341724+0000 osd.7 (osd.7) 249 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:52.744966+0000 mgr.y (mgr.24481) 691 : cluster [DBG] pgmap v950: 138 pgs: 10 unknown, 128 active+clean; 7.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 853 B/s rd, 58 MiB/s wr, 927 op/s 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:52.807412+0000 osd.5 (osd.5) 247 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:52.811086+0000 osd.5 (osd.5) 248 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:53.041281+0000 mon.a (mon.0) 1664 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:53.041349+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e482: 8 total, 8 up, 5 in 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:53.241199+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.33:0/3474171265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:53.241551+0000 mon.a (mon.0) 1666 : audit [INF] from='client.25690 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:53.492043+0000 osd.0 (osd.0) 218 : cluster [DBG] 10.5 scrub starts 2024-02-12T17:07:54.262 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:52.241675+0000 osd.7 (osd.7) 248 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:52.341724+0000 osd.7 (osd.7) 249 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:52.744966+0000 mgr.y (mgr.24481) 691 : cluster [DBG] pgmap v950: 138 pgs: 10 unknown, 128 active+clean; 7.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 853 B/s rd, 58 MiB/s wr, 927 op/s 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:52.807412+0000 osd.5 (osd.5) 247 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:52.811086+0000 osd.5 (osd.5) 248 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:53.041281+0000 mon.a (mon.0) 1664 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:53.041349+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e482: 8 total, 8 up, 5 in 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:53.241199+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.33:0/3474171265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:53.241551+0000 mon.a (mon.0) 1666 : audit [INF] from='client.25690 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:07:54.263 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:53.492043+0000 osd.0 (osd.0) 218 : cluster [DBG] 10.5 scrub starts 2024-02-12T17:07:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:52.241675+0000 osd.7 (osd.7) 248 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:07:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:52.341724+0000 osd.7 (osd.7) 249 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:07:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:52.744966+0000 mgr.y (mgr.24481) 691 : cluster [DBG] pgmap v950: 138 pgs: 10 unknown, 128 active+clean; 7.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 853 B/s rd, 58 MiB/s wr, 927 op/s 2024-02-12T17:07:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:52.807412+0000 osd.5 (osd.5) 247 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:07:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:52.811086+0000 osd.5 (osd.5) 248 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:07:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:53.041281+0000 mon.a (mon.0) 1664 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T17:07:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:53.041349+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e482: 8 total, 8 up, 5 in 2024-02-12T17:07:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:53.241199+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.33:0/3474171265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:07:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:53.241551+0000 mon.a (mon.0) 1666 : audit [INF] from='client.25690 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:07:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:53.492043+0000 osd.0 (osd.0) 218 : cluster [DBG] 10.5 scrub starts 2024-02-12T17:07:54.916 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 90 16 123169 123153 85.4576 1.25 0.00252182 0.0114848 2024-02-12T17:07:54.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:07:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:07:55.040 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:07:55.040 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:07:55.040 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:07:55.040 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:07:55.040 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:07:55.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:53.893347+0000 osd.0 (osd.0) 219 : cluster [DBG] 10.5 scrub ok 2024-02-12T17:07:55.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:54.051740+0000 mon.a (mon.0) 1667 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:55.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:54.066856+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25690 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:07:55.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:54.066937+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e483: 8 total, 8 up, 5 in 2024-02-12T17:07:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:53.893347+0000 osd.0 (osd.0) 219 : cluster [DBG] 10.5 scrub ok 2024-02-12T17:07:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:54.051740+0000 mon.a (mon.0) 1667 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:54.066856+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25690 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:07:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:54.066937+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e483: 8 total, 8 up, 5 in 2024-02-12T17:07:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:53.893347+0000 osd.0 (osd.0) 219 : cluster [DBG] 10.5 scrub ok 2024-02-12T17:07:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:54.051740+0000 mon.a (mon.0) 1667 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:07:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:54.066856+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25690 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:07:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:54.066937+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e483: 8 total, 8 up, 5 in 2024-02-12T17:07:55.716 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:55.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:55.826 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:07:55.883 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:55.916 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 91 3 123169 123166 84.528 0.8125 1.24831 0.0116786 2024-02-12T17:07:56.031 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:07:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:56.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:07:56.148 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:54.510044+0000 mgr.y (mgr.24481) 692 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:54.745389+0000 mgr.y (mgr.24481) 693 : cluster [DBG] pgmap v953: 138 pgs: 9 peering, 1 active+clean+scrubbing, 128 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:55.066706+0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e484: 8 total, 8 up, 5 in 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:55.067505+0000 osd.3 (osd.3) 171 : cluster [DBG] 10.14 starting backfill to osd.5 from (0'0,0'0] MAX to 481'7633 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:55.067559+0000 osd.5 (osd.5) 249 : cluster [DBG] 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 481'7584 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:55.067961+0000 osd.3 (osd.3) 172 : cluster [DBG] 10.17 starting backfill to osd.7 from (0'0,0'0] MAX to 481'7643 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:55.068472+0000 osd.2 (osd.2) 170 : cluster [DBG] 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 481'7806 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:55.068533+0000 osd.2 (osd.2) 171 : cluster [DBG] 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 481'7711 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:55.070119+0000 osd.5 (osd.5) 250 : cluster [DBG] 10.16 starting backfill to osd.3 from (0'0,0'0] MAX to 481'7742 2024-02-12T17:07:56.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:55.075288+0000 osd.2 (osd.2) 172 : cluster [DBG] 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 481'7806 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:54.510044+0000 mgr.y (mgr.24481) 692 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:54.745389+0000 mgr.y (mgr.24481) 693 : cluster [DBG] pgmap v953: 138 pgs: 9 peering, 1 active+clean+scrubbing, 128 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:55.066706+0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e484: 8 total, 8 up, 5 in 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:55.067505+0000 osd.3 (osd.3) 171 : cluster [DBG] 10.14 starting backfill to osd.5 from (0'0,0'0] MAX to 481'7633 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:55.067559+0000 osd.5 (osd.5) 249 : cluster [DBG] 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 481'7584 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:55.067961+0000 osd.3 (osd.3) 172 : cluster [DBG] 10.17 starting backfill to osd.7 from (0'0,0'0] MAX to 481'7643 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:55.068472+0000 osd.2 (osd.2) 170 : cluster [DBG] 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 481'7806 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:55.068533+0000 osd.2 (osd.2) 171 : cluster [DBG] 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 481'7711 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:55.070119+0000 osd.5 (osd.5) 250 : cluster [DBG] 10.16 starting backfill to osd.3 from (0'0,0'0] MAX to 481'7742 2024-02-12T17:07:56.207 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:55.075288+0000 osd.2 (osd.2) 172 : cluster [DBG] 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 481'7806 2024-02-12T17:07:56.213 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:56.268 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:56.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:56.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:56.402 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:54.510044+0000 mgr.y (mgr.24481) 692 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:07:56.402 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:54.745389+0000 mgr.y (mgr.24481) 693 : cluster [DBG] pgmap v953: 138 pgs: 9 peering, 1 active+clean+scrubbing, 128 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T17:07:56.402 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:55.066706+0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e484: 8 total, 8 up, 5 in 2024-02-12T17:07:56.402 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:55.067505+0000 osd.3 (osd.3) 171 : cluster [DBG] 10.14 starting backfill to osd.5 from (0'0,0'0] MAX to 481'7633 2024-02-12T17:07:56.402 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:55.067559+0000 osd.5 (osd.5) 249 : cluster [DBG] 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 481'7584 2024-02-12T17:07:56.403 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:55.067961+0000 osd.3 (osd.3) 172 : cluster [DBG] 10.17 starting backfill to osd.7 from (0'0,0'0] MAX to 481'7643 2024-02-12T17:07:56.403 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:55.068472+0000 osd.2 (osd.2) 170 : cluster [DBG] 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 481'7806 2024-02-12T17:07:56.403 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:55.068533+0000 osd.2 (osd.2) 171 : cluster [DBG] 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 481'7711 2024-02-12T17:07:56.403 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:55.070119+0000 osd.5 (osd.5) 250 : cluster [DBG] 10.16 starting backfill to osd.3 from (0'0,0'0] MAX to 481'7742 2024-02-12T17:07:56.403 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:55.075288+0000 osd.2 (osd.2) 172 : cluster [DBG] 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 481'7806 2024-02-12T17:07:56.404 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:56.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:56.536 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:56.563 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:56.585 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:56.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:56.622 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [1, 6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:07:56.622 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:07:56.622 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-02-12T17:07:56.622 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd in 1 2024-02-12T17:07:56.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:56.734 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:56.784 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T17:07:56.916 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 92 3 123169 123166 83.6098 0 - 0.0116786 2024-02-12T17:07:57.418 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:57.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:57.526 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:57.596 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:57.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:07:57.655 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:07:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:57.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:07:57.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:56.356581+0000 osd.0 (osd.0) 220 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:57.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:56.356625+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:57.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:56.750753+0000 mgr.y (mgr.24481) 694 : cluster [DBG] pgmap v955: 138 pgs: 9 peering, 1 active+clean+scrubbing, 128 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 27 KiB/s, 0 objects/s recovering 2024-02-12T17:07:57.666 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:57.102119+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e485: 8 total, 8 up, 5 in 2024-02-12T17:07:57.666 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:56.356581+0000 osd.0 (osd.0) 220 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:57.667 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:56.356625+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:57.667 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:56.750753+0000 mgr.y (mgr.24481) 694 : cluster [DBG] pgmap v955: 138 pgs: 9 peering, 1 active+clean+scrubbing, 128 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 27 KiB/s, 0 objects/s recovering 2024-02-12T17:07:57.667 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:57.102119+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e485: 8 total, 8 up, 5 in 2024-02-12T17:07:57.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:56.356581+0000 osd.0 (osd.0) 220 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:07:57.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:56.356625+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:07:57.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:56.750753+0000 mgr.y (mgr.24481) 694 : cluster [DBG] pgmap v955: 138 pgs: 9 peering, 1 active+clean+scrubbing, 128 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 27 KiB/s, 0 objects/s recovering 2024-02-12T17:07:57.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:57.102119+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e485: 8 total, 8 up, 5 in 2024-02-12T17:07:57.767 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:07:57.795 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:57.815 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:07:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:57.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:07:57.920 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:07:57.922 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 93 3 123169 123166 82.7066 0 - 0.0116786 2024-02-12T17:07:57.981 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:58.041 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:07:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:58.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:07:58.100 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:58.109 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:07:58.143 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:07:58.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:58.172 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:58.190 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:07:58.192 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:58.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:58.234 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:07:58.267 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:07:58.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:07:58.302 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:07:58.324 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:58.334 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:07:58.347 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:07:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:58.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:07:58.367 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:07:58.398 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:07:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:07:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:07:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:07:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:07:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:07:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:07:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:07:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:07:58.452 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:58.479 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:58.499 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:58.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:58.604 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:07:58.632 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:58.652 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:07:58.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:58.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:07:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:07:58.086754+0000 mon.a (mon.0) 1672 : audit [DBG] from='client.? v1:172.21.15.33:0/1362356924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:58.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:07:58.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:07:58.086754+0000 mon.a (mon.0) 1672 : audit [DBG] from='client.? v1:172.21.15.33:0/1362356924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:58.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:07:58.086754+0000 mon.a (mon.0) 1672 : audit [DBG] from='client.? v1:172.21.15.33:0/1362356924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:07:58.784 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:07:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:07:58.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:07:58.910 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:07:58.941 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 94 3 123169 123166 81.8111 0 - 0.0116786 2024-02-12T17:07:58.963 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:07:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:07:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:07:58.753926+0000 mgr.y (mgr.24481) 695 : cluster [DBG] pgmap v957: 138 pgs: 4 active+remapped+backfilling, 4 peering, 1 active+clean+scrubbing, 129 active+clean; 7.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 897 B/s rd, 4.2 MiB/s wr, 67 op/s; 16238/358632 objects misplaced (4.528%); 29 MiB/s, 467 objects/s recovering 2024-02-12T17:07:59.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:07:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:07:58.753926+0000 mgr.y (mgr.24481) 695 : cluster [DBG] pgmap v957: 138 pgs: 4 active+remapped+backfilling, 4 peering, 1 active+clean+scrubbing, 129 active+clean; 7.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 897 B/s rd, 4.2 MiB/s wr, 67 op/s; 16238/358632 objects misplaced (4.528%); 29 MiB/s, 467 objects/s recovering 2024-02-12T17:07:59.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:07:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:07:58.753926+0000 mgr.y (mgr.24481) 695 : cluster [DBG] pgmap v957: 138 pgs: 4 active+remapped+backfilling, 4 peering, 1 active+clean+scrubbing, 129 active+clean; 7.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 897 B/s rd, 4.2 MiB/s wr, 67 op/s; 16238/358632 objects misplaced (4.528%); 29 MiB/s, 467 objects/s recovering 2024-02-12T17:07:59.942 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 95 3 123169 123166 80.9499 0 - 0.0116786 2024-02-12T17:08:00.942 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 96 3 123169 123166 80.1074 0 - 0.0116786 2024-02-12T17:08:01.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:08:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:00] "GET /metrics HTTP/1.1" 200 176505 "" "Prometheus/2.33.4" 2024-02-12T17:08:01.944 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 97 3 123169 123166 79.2815 0 - 0.0116786 2024-02-12T17:08:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:00.754399+0000 mgr.y (mgr.24481) 696 : cluster [DBG] pgmap v958: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 765 B/s rd, 73 MiB/s wr, 1.17k op/s; 23872/370194 objects misplaced (6.449%); 25 MiB/s, 404 objects/s recovering 2024-02-12T17:08:02.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:01.388387+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e486: 8 total, 8 up, 5 in 2024-02-12T17:08:02.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:00.754399+0000 mgr.y (mgr.24481) 696 : cluster [DBG] pgmap v958: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 765 B/s rd, 73 MiB/s wr, 1.17k op/s; 23872/370194 objects misplaced (6.449%); 25 MiB/s, 404 objects/s recovering 2024-02-12T17:08:02.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:01.388387+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e486: 8 total, 8 up, 5 in 2024-02-12T17:08:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:00.754399+0000 mgr.y (mgr.24481) 696 : cluster [DBG] pgmap v958: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 765 B/s rd, 73 MiB/s wr, 1.17k op/s; 23872/370194 objects misplaced (6.449%); 25 MiB/s, 404 objects/s recovering 2024-02-12T17:08:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:01.388387+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e486: 8 total, 8 up, 5 in 2024-02-12T17:08:02.944 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 98 3 123169 123166 78.4731 0 - 0.0116786 2024-02-12T17:08:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:08:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:08:03.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:02.497929+0000 mon.a (mon.0) 1674 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:03.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:02.754727+0000 mgr.y (mgr.24481) 697 : cluster [DBG] pgmap v960: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 665 B/s rd, 64 MiB/s wr, 1.02k op/s; 23872/370194 objects misplaced (6.449%); 22 MiB/s, 351 objects/s recovering 2024-02-12T17:08:03.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:02.497929+0000 mon.a (mon.0) 1674 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:03.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:02.754727+0000 mgr.y (mgr.24481) 697 : cluster [DBG] pgmap v960: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 665 B/s rd, 64 MiB/s wr, 1.02k op/s; 23872/370194 objects misplaced (6.449%); 22 MiB/s, 351 objects/s recovering 2024-02-12T17:08:03.943 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 99 3 123169 123166 77.6807 0 - 0.0116786 2024-02-12T17:08:03.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:02.497929+0000 mon.a (mon.0) 1674 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:03.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:02.754727+0000 mgr.y (mgr.24481) 697 : cluster [DBG] pgmap v960: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 665 B/s rd, 64 MiB/s wr, 1.02k op/s; 23872/370194 objects misplaced (6.449%); 22 MiB/s, 351 objects/s recovering 2024-02-12T17:08:04.944 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:08:04.943368+0000 min lat: 0.00118871 max lat: 2.27549 avg lat: 0.0116786 2024-02-12T17:08:04.944 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:08:04.944 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 100 3 123169 123166 76.9046 0 - 0.0116786 2024-02-12T17:08:04.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:08:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:08:05.364 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:08:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:05.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:08:05.475 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:05.635 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:04.520522+0000 mgr.y (mgr.24481) 698 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:05.776 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:05.944 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 101 3 123169 123166 76.1438 0 - 0.0116786 2024-02-12T17:08:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:04.520522+0000 mgr.y (mgr.24481) 698 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:04.626266+0000 mon.a (mon.0) 1675 : cluster [DBG] osdmap e487: 8 total, 8 up, 5 in 2024-02-12T17:08:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:04.739317+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:04.739362+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:04.755136+0000 mgr.y (mgr.24481) 699 : cluster [DBG] pgmap v962: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 17086/370194 objects misplaced (4.615%); 67 MiB/s, 1.07k objects/s recovering 2024-02-12T17:08:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:04.626266+0000 mon.a (mon.0) 1675 : cluster [DBG] osdmap e487: 8 total, 8 up, 5 in 2024-02-12T17:08:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:04.739317+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:04.739362+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:04.755136+0000 mgr.y (mgr.24481) 699 : cluster [DBG] pgmap v962: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 17086/370194 objects misplaced (4.615%); 67 MiB/s, 1.07k objects/s recovering 2024-02-12T17:08:06.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:04.520522+0000 mgr.y (mgr.24481) 698 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:06.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:04.626266+0000 mon.a (mon.0) 1675 : cluster [DBG] osdmap e487: 8 total, 8 up, 5 in 2024-02-12T17:08:06.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:04.739317+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:06.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:04.739362+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:06.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:04.755136+0000 mgr.y (mgr.24481) 699 : cluster [DBG] pgmap v962: 138 pgs: 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 3 peering, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 17086/370194 objects misplaced (4.615%); 67 MiB/s, 1.07k objects/s recovering 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total time run: 101.385 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total writes made: 123169 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Write size: 65536 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object size: 65536 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Bandwidth (MB/sec): 75.929 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Bandwidth: 39.9172 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max bandwidth (MB/sec): 204.375 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min bandwidth (MB/sec): 0 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average IOPS: 1214 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev IOPS: 638.675 2024-02-12T17:08:06.945 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max IOPS: 3270 2024-02-12T17:08:06.946 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min IOPS: 0 2024-02-12T17:08:06.946 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average Latency(s): 0.0119971 2024-02-12T17:08:06.946 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Latency(s): 0.0688349 2024-02-12T17:08:06.946 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max latency(s): 13.1588 2024-02-12T17:08:06.946 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min latency(s): 0.00118871 2024-02-12T17:08:06.956 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T17:08:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:05.635657+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e488: 8 total, 8 up, 5 in 2024-02-12T17:08:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:06.389513+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e489: 8 total, 8 up, 5 in 2024-02-12T17:08:07.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:05.635657+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e488: 8 total, 8 up, 5 in 2024-02-12T17:08:07.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:06.389513+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e489: 8 total, 8 up, 5 in 2024-02-12T17:08:07.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:05.635657+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e488: 8 total, 8 up, 5 in 2024-02-12T17:08:07.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:06.389513+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e489: 8 total, 8 up, 5 in 2024-02-12T17:08:07.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:08:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:06.101940+0000 osd.2 (osd.2) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:06.101983+0000 osd.2 (osd.2) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:07.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:06.757983+0000 mgr.y (mgr.24481) 700 : cluster [DBG] pgmap v965: 138 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+remapped+backfilling, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 954 B/s rd, 84 KiB/s wr, 2 op/s; 3779/370215 objects degraded (1.021%); 24511/370215 objects misplaced (6.621%); 67 MiB/s, 1.07k objects/s recovering 2024-02-12T17:08:07.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:07.404884+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e490: 8 total, 8 up, 5 in 2024-02-12T17:08:08.005 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:08.018 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:06.101940+0000 osd.2 (osd.2) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:08.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:06.101983+0000 osd.2 (osd.2) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:08.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:06.757983+0000 mgr.y (mgr.24481) 700 : cluster [DBG] pgmap v965: 138 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+remapped+backfilling, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 954 B/s rd, 84 KiB/s wr, 2 op/s; 3779/370215 objects degraded (1.021%); 24511/370215 objects misplaced (6.621%); 67 MiB/s, 1.07k objects/s recovering 2024-02-12T17:08:08.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:07.404884+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e490: 8 total, 8 up, 5 in 2024-02-12T17:08:08.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:06.101940+0000 osd.2 (osd.2) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:08.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:06.101983+0000 osd.2 (osd.2) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:08.020 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:06.757983+0000 mgr.y (mgr.24481) 700 : cluster [DBG] pgmap v965: 138 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+remapped+backfilling, 130 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 954 B/s rd, 84 KiB/s wr, 2 op/s; 3779/370215 objects degraded (1.021%); 24511/370215 objects misplaced (6.621%); 67 MiB/s, 1.07k objects/s recovering 2024-02-12T17:08:08.020 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:07.404884+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e490: 8 total, 8 up, 5 in 2024-02-12T17:08:08.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:08.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:08:08.153 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:08.210 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:08:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:08.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:08:08.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:08:08.441 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:08.485 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:08:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:08.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:08:08.606 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:08.634 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:07.646867+0000 mon.a (mon.0) 1679 : cluster [WRN] Health check failed: Degraded data redundancy: 3779/370215 objects degraded (1.021%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:08:09.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:07.646867+0000 mon.a (mon.0) 1679 : cluster [WRN] Health check failed: Degraded data redundancy: 3779/370215 objects degraded (1.021%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:08:09.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:07.646867+0000 mon.a (mon.0) 1679 : cluster [WRN] Health check failed: Degraded data redundancy: 3779/370215 objects degraded (1.021%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:08:09.409 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:09.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:09.519 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:09.573 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:09.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:09.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:08:09.714 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:08.760956+0000 mgr.y (mgr.24481) 701 : cluster [DBG] pgmap v967: 138 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 108 KiB/s wr, 171 op/s; 3779/368112 objects degraded (1.027%); 20719/368112 objects misplaced (5.628%); 43 MiB/s, 692 objects/s recovering 2024-02-12T17:08:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:08.760956+0000 mgr.y (mgr.24481) 701 : cluster [DBG] pgmap v967: 138 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 108 KiB/s wr, 171 op/s; 3779/368112 objects degraded (1.027%); 20719/368112 objects misplaced (5.628%); 43 MiB/s, 692 objects/s recovering 2024-02-12T17:08:10.039 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:08.760956+0000 mgr.y (mgr.24481) 701 : cluster [DBG] pgmap v967: 138 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 108 KiB/s wr, 171 op/s; 3779/368112 objects degraded (1.027%); 20719/368112 objects misplaced (5.628%); 43 MiB/s, 692 objects/s recovering 2024-02-12T17:08:10.886 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:08:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:10] "GET /metrics HTTP/1.1" 200 176546 "" "Prometheus/2.33.4" 2024-02-12T17:08:11.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:08:10.921+0000 7fd96a9de700 -1 mon.a@0(leader).osd e490 definitely_dead 0 2024-02-12T17:08:11.866 INFO:teuthology.orchestra.run.smithi033.stderr:marked in osd.1. 2024-02-12T17:08:12.018 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:08:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:12.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:08:12.023 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:08:12.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:08:12.166 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:10.761428+0000 mgr.y (mgr.24481) 702 : cluster [DBG] pgmap v968: 138 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 852 B/s rd, 75 KiB/s wr, 138 op/s; 3779/367758 objects degraded (1.028%); 20603/367758 objects misplaced (5.602%); 30 MiB/s, 486 objects/s recovering 2024-02-12T17:08:12.166 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:10.919480+0000 mon.b (mon.1) 93 : audit [DBG] from='client.? v1:172.21.15.33:0/1818857738' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-12T17:08:12.166 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:10.923250+0000 mon.a (mon.0) 1680 : audit [DBG] from='client.25696 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-12T17:08:12.166 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:10.761428+0000 mgr.y (mgr.24481) 702 : cluster [DBG] pgmap v968: 138 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 852 B/s rd, 75 KiB/s wr, 138 op/s; 3779/367758 objects degraded (1.028%); 20603/367758 objects misplaced (5.602%); 30 MiB/s, 486 objects/s recovering 2024-02-12T17:08:12.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:10.919480+0000 mon.b (mon.1) 93 : audit [DBG] from='client.? v1:172.21.15.33:0/1818857738' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-12T17:08:12.167 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:10.923250+0000 mon.a (mon.0) 1680 : audit [DBG] from='client.25696 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-12T17:08:12.169 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:10.761428+0000 mgr.y (mgr.24481) 702 : cluster [DBG] pgmap v968: 138 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 852 B/s rd, 75 KiB/s wr, 138 op/s; 3779/367758 objects degraded (1.028%); 20603/367758 objects misplaced (5.602%); 30 MiB/s, 486 objects/s recovering 2024-02-12T17:08:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:10.919480+0000 mon.b (mon.1) 93 : audit [DBG] from='client.? v1:172.21.15.33:0/1818857738' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-12T17:08:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:10.923250+0000 mon.a (mon.0) 1680 : audit [DBG] from='client.25696 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-12T17:08:12.255 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:08:12.266 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:08:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:12.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:08:12.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:12.491 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:12.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:12.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:12.722 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:12.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:08:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:08:12.741 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:12.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:12.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:12.845 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:08:12.874 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:12.893 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:08:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:12.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:08:12.997 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:13.027 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:11.831090+0000 mon.a (mon.0) 1681 : audit [INF] from='client.25696 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-02-12T17:08:13.027 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:11.831172+0000 mon.a (mon.0) 1682 : cluster [DBG] osdmap e491: 8 total, 8 up, 6 in 2024-02-12T17:08:13.028 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.829133+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2024-02-12T17:08:13.034 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:13.060 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:13.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:13.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:08:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:08:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:13.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:08:13.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:11.831090+0000 mon.a (mon.0) 1681 : audit [INF] from='client.25696 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-02-12T17:08:13.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:11.831172+0000 mon.a (mon.0) 1682 : cluster [DBG] osdmap e491: 8 total, 8 up, 6 in 2024-02-12T17:08:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.829133+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2024-02-12T17:08:13.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:11.831090+0000 mon.a (mon.0) 1681 : audit [INF] from='client.25696 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-02-12T17:08:13.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:11.831172+0000 mon.a (mon.0) 1682 : cluster [DBG] osdmap e491: 8 total, 8 up, 6 in 2024-02-12T17:08:13.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.829133+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2024-02-12T17:08:13.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:13.344 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.762013+0000 mgr.y (mgr.24481) 703 : cluster [DBG] pgmap v970: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.4 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 964 B/s rd, 160 B/s wr, 286 op/s; 32/364767 objects degraded (0.009%); 23955/364767 objects misplaced (6.567%); 28 MiB/s, 445 objects/s recovering 2024-02-12T17:08:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.829482+0000 osd.3 (osd.3) 173 : cluster [DBG] 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7779 2024-02-12T17:08:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.830675+0000 osd.3 (osd.3) 174 : cluster [DBG] 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7789 2024-02-12T17:08:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.830954+0000 osd.3 (osd.3) 175 : cluster [DBG] 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7696 2024-02-12T17:08:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.840327+0000 mon.a (mon.0) 1684 : cluster [WRN] Health check update: Degraded data redundancy: 32/364767 objects degraded (0.009%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:08:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.845109+0000 osd.2 (osd.2) 175 : cluster [INF] 10.19 continuing backfill to osd.0 from (466'4500,490'7919] 10:9e5a0868:::benchmark_data_smithi033_160181_object71619:head to 490'7919 2024-02-12T17:08:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.845146+0000 osd.2 (osd.2) 176 : cluster [DBG] 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7919 2024-02-12T17:08:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:12.853083+0000 osd.2 (osd.2) 177 : cluster [INF] 10.19 continuing backfill to osd.3 from (466'4500,490'7919] 10:9e5a0868:::benchmark_data_smithi033_160181_object71619:head to 490'7919 2024-02-12T17:08:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.828255+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2024-02-12T17:08:14.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.762013+0000 mgr.y (mgr.24481) 703 : cluster [DBG] pgmap v970: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.4 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 964 B/s rd, 160 B/s wr, 286 op/s; 32/364767 objects degraded (0.009%); 23955/364767 objects misplaced (6.567%); 28 MiB/s, 445 objects/s recovering 2024-02-12T17:08:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.829482+0000 osd.3 (osd.3) 173 : cluster [DBG] 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7779 2024-02-12T17:08:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.830675+0000 osd.3 (osd.3) 174 : cluster [DBG] 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7789 2024-02-12T17:08:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.830954+0000 osd.3 (osd.3) 175 : cluster [DBG] 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7696 2024-02-12T17:08:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.840327+0000 mon.a (mon.0) 1684 : cluster [WRN] Health check update: Degraded data redundancy: 32/364767 objects degraded (0.009%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:08:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.845109+0000 osd.2 (osd.2) 175 : cluster [INF] 10.19 continuing backfill to osd.0 from (466'4500,490'7919] 10:9e5a0868:::benchmark_data_smithi033_160181_object71619:head to 490'7919 2024-02-12T17:08:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.845146+0000 osd.2 (osd.2) 176 : cluster [DBG] 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7919 2024-02-12T17:08:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:12.853083+0000 osd.2 (osd.2) 177 : cluster [INF] 10.19 continuing backfill to osd.3 from (466'4500,490'7919] 10:9e5a0868:::benchmark_data_smithi033_160181_object71619:head to 490'7919 2024-02-12T17:08:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.828255+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2024-02-12T17:08:14.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.762013+0000 mgr.y (mgr.24481) 703 : cluster [DBG] pgmap v970: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 132 active+clean; 7.4 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 964 B/s rd, 160 B/s wr, 286 op/s; 32/364767 objects degraded (0.009%); 23955/364767 objects misplaced (6.567%); 28 MiB/s, 445 objects/s recovering 2024-02-12T17:08:14.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.829482+0000 osd.3 (osd.3) 173 : cluster [DBG] 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7779 2024-02-12T17:08:14.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.830675+0000 osd.3 (osd.3) 174 : cluster [DBG] 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7789 2024-02-12T17:08:14.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.830954+0000 osd.3 (osd.3) 175 : cluster [DBG] 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7696 2024-02-12T17:08:14.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.840327+0000 mon.a (mon.0) 1684 : cluster [WRN] Health check update: Degraded data redundancy: 32/364767 objects degraded (0.009%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:08:14.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.845109+0000 osd.2 (osd.2) 175 : cluster [INF] 10.19 continuing backfill to osd.0 from (466'4500,490'7919] 10:9e5a0868:::benchmark_data_smithi033_160181_object71619:head to 490'7919 2024-02-12T17:08:14.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.845146+0000 osd.2 (osd.2) 176 : cluster [DBG] 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7919 2024-02-12T17:08:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:12.853083+0000 osd.2 (osd.2) 177 : cluster [INF] 10.19 continuing backfill to osd.3 from (466'4500,490'7919] 10:9e5a0868:::benchmark_data_smithi033_160181_object71619:head to 490'7919 2024-02-12T17:08:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.828255+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2024-02-12T17:08:14.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:08:15.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.827171+0000 osd.7 (osd.7) 250 : cluster [DBG] 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7923 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.827645+0000 osd.7 (osd.7) 251 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 443'18 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.827742+0000 osd.7 (osd.7) 252 : cluster [DBG] 10.c starting backfill to osd.1 from (0'0,0'0] MAX to 490'7948 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.829295+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.3 starting backfill to osd.1 from (0'0,0'0] MAX to 433'15 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.829338+0000 osd.5 (osd.5) 254 : cluster [DBG] 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7849 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.829344+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 starting backfill to osd.1 from (0'0,0'0] MAX to 438'17 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.829376+0000 osd.5 (osd.5) 256 : cluster [DBG] 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7871 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.829899+0000 osd.2 (osd.2) 178 : cluster [DBG] 10.0 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7640 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.833485+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.5 starting backfill to osd.1 from (0'0,0'0] MAX to 410'15 2024-02-12T17:08:15.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.833942+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.f starting backfill to osd.1 from (0'0,0'0] MAX to 443'21 2024-02-12T17:08:15.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.834038+0000 osd.2 (osd.2) 180 : cluster [DBG] 10.b starting backfill to osd.1 from (0'0,0'0] MAX to 490'7921 2024-02-12T17:08:15.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.846730+0000 osd.2 (osd.2) 181 : cluster [DBG] 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7831 2024-02-12T17:08:15.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.856189+0000 osd.3 (osd.3) 176 : cluster [DBG] 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7760 2024-02-12T17:08:15.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.856303+0000 osd.3 (osd.3) 177 : cluster [DBG] 10.2 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7689 2024-02-12T17:08:15.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.857833+0000 osd.3 (osd.3) 178 : cluster [DBG] 5.13 starting backfill to osd.1 from (0'0,0'0] MAX to 419'14 2024-02-12T17:08:15.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.861890+0000 osd.3 (osd.3) 179 : cluster [DBG] 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 490'7914 2024-02-12T17:08:15.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:13.866775+0000 osd.3 (osd.3) 180 : cluster [DBG] 5.0 starting backfill to osd.1 from (0'0,0'0] MAX to 433'19 2024-02-12T17:08:15.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.827171+0000 osd.7 (osd.7) 250 : cluster [DBG] 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7923 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.827645+0000 osd.7 (osd.7) 251 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 443'18 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.827742+0000 osd.7 (osd.7) 252 : cluster [DBG] 10.c starting backfill to osd.1 from (0'0,0'0] MAX to 490'7948 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.829295+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.3 starting backfill to osd.1 from (0'0,0'0] MAX to 433'15 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.829338+0000 osd.5 (osd.5) 254 : cluster [DBG] 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7849 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.829344+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 starting backfill to osd.1 from (0'0,0'0] MAX to 438'17 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.829376+0000 osd.5 (osd.5) 256 : cluster [DBG] 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7871 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.829899+0000 osd.2 (osd.2) 178 : cluster [DBG] 10.0 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7640 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.833485+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.5 starting backfill to osd.1 from (0'0,0'0] MAX to 410'15 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.833942+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.f starting backfill to osd.1 from (0'0,0'0] MAX to 443'21 2024-02-12T17:08:15.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.834038+0000 osd.2 (osd.2) 180 : cluster [DBG] 10.b starting backfill to osd.1 from (0'0,0'0] MAX to 490'7921 2024-02-12T17:08:15.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.846730+0000 osd.2 (osd.2) 181 : cluster [DBG] 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7831 2024-02-12T17:08:15.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.856189+0000 osd.3 (osd.3) 176 : cluster [DBG] 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7760 2024-02-12T17:08:15.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.856303+0000 osd.3 (osd.3) 177 : cluster [DBG] 10.2 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7689 2024-02-12T17:08:15.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.857833+0000 osd.3 (osd.3) 178 : cluster [DBG] 5.13 starting backfill to osd.1 from (0'0,0'0] MAX to 419'14 2024-02-12T17:08:15.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.861890+0000 osd.3 (osd.3) 179 : cluster [DBG] 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 490'7914 2024-02-12T17:08:15.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:13.866775+0000 osd.3 (osd.3) 180 : cluster [DBG] 5.0 starting backfill to osd.1 from (0'0,0'0] MAX to 433'19 2024-02-12T17:08:15.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.827171+0000 osd.7 (osd.7) 250 : cluster [DBG] 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7923 2024-02-12T17:08:15.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.827645+0000 osd.7 (osd.7) 251 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 443'18 2024-02-12T17:08:15.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.827742+0000 osd.7 (osd.7) 252 : cluster [DBG] 10.c starting backfill to osd.1 from (0'0,0'0] MAX to 490'7948 2024-02-12T17:08:15.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.829295+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.3 starting backfill to osd.1 from (0'0,0'0] MAX to 433'15 2024-02-12T17:08:15.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.829338+0000 osd.5 (osd.5) 254 : cluster [DBG] 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7849 2024-02-12T17:08:15.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.829344+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 starting backfill to osd.1 from (0'0,0'0] MAX to 438'17 2024-02-12T17:08:15.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.829376+0000 osd.5 (osd.5) 256 : cluster [DBG] 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7871 2024-02-12T17:08:15.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.829899+0000 osd.2 (osd.2) 178 : cluster [DBG] 10.0 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7640 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.833485+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.5 starting backfill to osd.1 from (0'0,0'0] MAX to 410'15 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.833942+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.f starting backfill to osd.1 from (0'0,0'0] MAX to 443'21 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.834038+0000 osd.2 (osd.2) 180 : cluster [DBG] 10.b starting backfill to osd.1 from (0'0,0'0] MAX to 490'7921 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.846730+0000 osd.2 (osd.2) 181 : cluster [DBG] 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7831 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.856189+0000 osd.3 (osd.3) 176 : cluster [DBG] 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7760 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.856303+0000 osd.3 (osd.3) 177 : cluster [DBG] 10.2 starting backfill to osd.1 from (0'0,0'0] MAX to 490'7689 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.857833+0000 osd.3 (osd.3) 178 : cluster [DBG] 5.13 starting backfill to osd.1 from (0'0,0'0] MAX to 419'14 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.861890+0000 osd.3 (osd.3) 179 : cluster [DBG] 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 490'7914 2024-02-12T17:08:15.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:13.866775+0000 osd.3 (osd.3) 180 : cluster [DBG] 5.0 starting backfill to osd.1 from (0'0,0'0] MAX to 433'19 2024-02-12T17:08:16.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:14.531116+0000 mgr.y (mgr.24481) 704 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:16.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:14.762554+0000 mgr.y (mgr.24481) 705 : cluster [DBG] pgmap v973: 138 pgs: 1 active+remapped+backfill_toofull, 12 remapped+peering, 12 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 3 active+recovering, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 104 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 487 op/s; 100/359358 objects degraded (0.028%); 27174/359358 objects misplaced (7.562%); 5.6 MiB/s, 78 objects/s recovering 2024-02-12T17:08:16.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:14.861678+0000 mon.a (mon.0) 1686 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T17:08:16.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:14.861710+0000 mon.a (mon.0) 1687 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:08:16.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:14.531116+0000 mgr.y (mgr.24481) 704 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:16.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:14.762554+0000 mgr.y (mgr.24481) 705 : cluster [DBG] pgmap v973: 138 pgs: 1 active+remapped+backfill_toofull, 12 remapped+peering, 12 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 3 active+recovering, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 104 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 487 op/s; 100/359358 objects degraded (0.028%); 27174/359358 objects misplaced (7.562%); 5.6 MiB/s, 78 objects/s recovering 2024-02-12T17:08:16.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:14.861678+0000 mon.a (mon.0) 1686 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T17:08:16.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:14.861710+0000 mon.a (mon.0) 1687 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:08:16.179 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-02-12T17:08:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:14.531116+0000 mgr.y (mgr.24481) 704 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:14.762554+0000 mgr.y (mgr.24481) 705 : cluster [DBG] pgmap v973: 138 pgs: 1 active+remapped+backfill_toofull, 12 remapped+peering, 12 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 3 active+recovering, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 104 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 487 op/s; 100/359358 objects degraded (0.028%); 27174/359358 objects misplaced (7.562%); 5.6 MiB/s, 78 objects/s recovering 2024-02-12T17:08:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:14.861678+0000 mon.a (mon.0) 1686 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T17:08:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:14.861710+0000 mon.a (mon.0) 1687 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:08:16.439 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:08:16.439 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:08:17.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:16.414683+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e494: 8 total, 8 up, 6 in 2024-02-12T17:08:17.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:16.763081+0000 mgr.y (mgr.24481) 706 : cluster [DBG] pgmap v975: 138 pgs: 2 active+remapped+backfill_toofull, 12 remapped+peering, 12 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 1 active+recovery_wait, 3 active+recovering, 1 active+recovering+undersized+degraded+remapped, 99 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 207 B/s rd, 0 B/s wr, 497 op/s; 3751/357420 objects degraded (1.049%); 42111/357420 objects misplaced (11.782%); 4.3 MiB/s, 61 objects/s recovering 2024-02-12T17:08:17.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:16.414683+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e494: 8 total, 8 up, 6 in 2024-02-12T17:08:17.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:16.763081+0000 mgr.y (mgr.24481) 706 : cluster [DBG] pgmap v975: 138 pgs: 2 active+remapped+backfill_toofull, 12 remapped+peering, 12 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 1 active+recovery_wait, 3 active+recovering, 1 active+recovering+undersized+degraded+remapped, 99 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 207 B/s rd, 0 B/s wr, 497 op/s; 3751/357420 objects degraded (1.049%); 42111/357420 objects misplaced (11.782%); 4.3 MiB/s, 61 objects/s recovering 2024-02-12T17:08:17.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:16.414683+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e494: 8 total, 8 up, 6 in 2024-02-12T17:08:17.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:16.763081+0000 mgr.y (mgr.24481) 706 : cluster [DBG] pgmap v975: 138 pgs: 2 active+remapped+backfill_toofull, 12 remapped+peering, 12 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 1 active+recovery_wait, 3 active+recovering, 1 active+recovering+undersized+degraded+remapped, 99 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 207 B/s rd, 0 B/s wr, 497 op/s; 3751/357420 objects degraded (1.049%); 42111/357420 objects misplaced (11.782%); 4.3 MiB/s, 61 objects/s recovering 2024-02-12T17:08:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:17.502637+0000 mon.a (mon.0) 1689 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:17.734522+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? v1:172.21.15.33:0/2161799043' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:08:18.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:17.502637+0000 mon.a (mon.0) 1689 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:18.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:17.734522+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? v1:172.21.15.33:0/2161799043' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:08:18.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:17.502637+0000 mon.a (mon.0) 1689 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:18.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:17.734522+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? v1:172.21.15.33:0/2161799043' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:08:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:18.500576+0000 mon.a (mon.0) 1691 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:08:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:18.567254+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? v1:172.21.15.33:0/2161799043' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:08:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:18.567307+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e495: 8 total, 8 up, 6 in 2024-02-12T17:08:19.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:18.763593+0000 mgr.y (mgr.24481) 707 : cluster [DBG] pgmap v977: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 115 active+clean; 7.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 748 op/s; 3683/351450 objects degraded (1.048%); 74574/351450 objects misplaced (21.219%); 4.2 MiB/s, 73 objects/s recovering 2024-02-12T17:08:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:18.500576+0000 mon.a (mon.0) 1691 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:08:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:18.567254+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? v1:172.21.15.33:0/2161799043' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:08:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:18.567307+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e495: 8 total, 8 up, 6 in 2024-02-12T17:08:19.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:18.763593+0000 mgr.y (mgr.24481) 707 : cluster [DBG] pgmap v977: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 115 active+clean; 7.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 748 op/s; 3683/351450 objects degraded (1.048%); 74574/351450 objects misplaced (21.219%); 4.2 MiB/s, 73 objects/s recovering 2024-02-12T17:08:19.929 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:08:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:18.500576+0000 mon.a (mon.0) 1691 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:08:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:18.567254+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? v1:172.21.15.33:0/2161799043' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:08:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:18.567307+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e495: 8 total, 8 up, 6 in 2024-02-12T17:08:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:18.763593+0000 mgr.y (mgr.24481) 707 : cluster [DBG] pgmap v977: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 115 active+clean; 7.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 748 op/s; 3683/351450 objects degraded (1.048%); 74574/351450 objects misplaced (21.219%); 4.2 MiB/s, 73 objects/s recovering 2024-02-12T17:08:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:19.564147+0000 mon.a (mon.0) 1694 : cluster [WRN] Health check update: Degraded data redundancy: 3683/351450 objects degraded (1.048%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:20.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:19.564184+0000 mon.a (mon.0) 1695 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T17:08:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:19.564147+0000 mon.a (mon.0) 1694 : cluster [WRN] Health check update: Degraded data redundancy: 3683/351450 objects degraded (1.048%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:19.564184+0000 mon.a (mon.0) 1695 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T17:08:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:19.564147+0000 mon.a (mon.0) 1694 : cluster [WRN] Health check update: Degraded data redundancy: 3683/351450 objects degraded (1.048%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:19.564184+0000 mon.a (mon.0) 1695 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T17:08:21.093 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:21.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:21.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T17:08:21.180 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:08:21.181 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:08:21.181 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-02-12T17:08:21.181 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:08:21.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:21.235 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:21.244 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:08:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:20] "GET /metrics HTTP/1.1" 200 176639 "" "Prometheus/2.33.4" 2024-02-12T17:08:21.257 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:21.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:21.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:21.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:21.513 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:21.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:20.764087+0000 mgr.y (mgr.24481) 708 : cluster [DBG] pgmap v978: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 7.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 424 op/s; 3769/350550 objects degraded (1.075%); 74462/350550 objects misplaced (21.241%); 1.3 MiB/s, 36 objects/s recovering 2024-02-12T17:08:21.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:21.397105+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:08:21.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:21.404341+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e496: 8 total, 8 up, 6 in 2024-02-12T17:08:21.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:21.502150+0000 mon.c (mon.2) 249 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:08:21.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:20.764087+0000 mgr.y (mgr.24481) 708 : cluster [DBG] pgmap v978: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 7.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 424 op/s; 3769/350550 objects degraded (1.075%); 74462/350550 objects misplaced (21.241%); 1.3 MiB/s, 36 objects/s recovering 2024-02-12T17:08:21.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:21.397105+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:08:21.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:21.404341+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e496: 8 total, 8 up, 6 in 2024-02-12T17:08:21.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:21.502150+0000 mon.c (mon.2) 249 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:08:21.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:20.764087+0000 mgr.y (mgr.24481) 708 : cluster [DBG] pgmap v978: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 7.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 424 op/s; 3769/350550 objects degraded (1.075%); 74462/350550 objects misplaced (21.241%); 1.3 MiB/s, 36 objects/s recovering 2024-02-12T17:08:21.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:21.397105+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:08:21.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:21.404341+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e496: 8 total, 8 up, 6 in 2024-02-12T17:08:21.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:21.502150+0000 mon.c (mon.2) 249 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:08:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:08:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:08:23.495 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:23.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:23.623 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:23.704 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:22.764596+0000 mgr.y (mgr.24481) 709 : cluster [DBG] pgmap v980: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 7.0 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 807 B/s rd, 0 B/s wr, 559 op/s; 184/346806 objects degraded (0.053%); 77123/346806 objects misplaced (22.238%); 1.6 MiB/s, 38 objects/s recovering 2024-02-12T17:08:24.282 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:22.764596+0000 mgr.y (mgr.24481) 709 : cluster [DBG] pgmap v980: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 7.0 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 807 B/s rd, 0 B/s wr, 559 op/s; 184/346806 objects degraded (0.053%); 77123/346806 objects misplaced (22.238%); 1.6 MiB/s, 38 objects/s recovering 2024-02-12T17:08:24.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:22.764596+0000 mgr.y (mgr.24481) 709 : cluster [DBG] pgmap v980: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 7.0 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 807 B/s rd, 0 B/s wr, 559 op/s; 184/346806 objects degraded (0.053%); 77123/346806 objects misplaced (22.238%); 1.6 MiB/s, 38 objects/s recovering 2024-02-12T17:08:24.591 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:24.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:24.695 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:24.712 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:08:24.723 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:08:24.796 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:24.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:08:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:08:25.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:24.798244+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:08:25.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:24.814911+0000 mon.c (mon.2) 251 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:08:25.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:24.821902+0000 mon.a (mon.0) 1698 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:25.228 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:24.798244+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:08:25.228 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:24.814911+0000 mon.c (mon.2) 251 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:08:25.228 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:24.821902+0000 mon.a (mon.0) 1698 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:25.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:24.798244+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:08:25.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:24.814911+0000 mon.c (mon.2) 251 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:08:25.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:24.821902+0000 mon.a (mon.0) 1698 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:25.464 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:25.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:25.575 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:25.640 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:25.684 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:08:25.715 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:08:25.742 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:08:25.790 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:08:25.834 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:08:25.882 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:08:25.915 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:08:26.005 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:24.542534+0000 mgr.y (mgr.24481) 710 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:26.005 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:24.765279+0000 mgr.y (mgr.24481) 711 : cluster [DBG] pgmap v981: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.9 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s; 209/338469 objects degraded (0.062%); 75111/338469 objects misplaced (22.191%); 5.5 MiB/s, 97 objects/s recovering 2024-02-12T17:08:26.005 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:24.829443+0000 mon.a (mon.0) 1699 : cluster [WRN] Health check update: Degraded data redundancy: 209/338469 objects degraded (0.062%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:26.005 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:25.634219+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? v1:172.21.15.33:0/4247681527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:08:26.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:24.542534+0000 mgr.y (mgr.24481) 710 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:26.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:24.765279+0000 mgr.y (mgr.24481) 711 : cluster [DBG] pgmap v981: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.9 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s; 209/338469 objects degraded (0.062%); 75111/338469 objects misplaced (22.191%); 5.5 MiB/s, 97 objects/s recovering 2024-02-12T17:08:26.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:24.829443+0000 mon.a (mon.0) 1699 : cluster [WRN] Health check update: Degraded data redundancy: 209/338469 objects degraded (0.062%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:26.006 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:25.634219+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? v1:172.21.15.33:0/4247681527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:08:26.040 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:08:26.040 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:08:26.040 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:08:26.040 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:08:26.040 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:08:26.040 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:08:26.041 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:08:26.041 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:08:26.041 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:08:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:24.542534+0000 mgr.y (mgr.24481) 710 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:24.765279+0000 mgr.y (mgr.24481) 711 : cluster [DBG] pgmap v981: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.9 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s; 209/338469 objects degraded (0.062%); 75111/338469 objects misplaced (22.191%); 5.5 MiB/s, 97 objects/s recovering 2024-02-12T17:08:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:24.829443+0000 mon.a (mon.0) 1699 : cluster [WRN] Health check update: Degraded data redundancy: 209/338469 objects degraded (0.062%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:25.634219+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? v1:172.21.15.33:0/4247681527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:08:26.337 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:26.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:26.445 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:08:26.492 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:26.573 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:08:27.550 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:08:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:27.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:08:27.656 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:27.726 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:26.769037+0000 mgr.y (mgr.24481) 712 : cluster [DBG] pgmap v982: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.8 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 624 B/s rd, 0 B/s wr, 727 op/s; 292/334893 objects degraded (0.087%); 74202/334893 objects misplaced (22.157%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:08:27.726 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:26.769037+0000 mgr.y (mgr.24481) 712 : cluster [DBG] pgmap v982: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.8 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 624 B/s rd, 0 B/s wr, 727 op/s; 292/334893 objects degraded (0.087%); 74202/334893 objects misplaced (22.157%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:08:27.729 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:26.769037+0000 mgr.y (mgr.24481) 712 : cluster [DBG] pgmap v982: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.8 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 624 B/s rd, 0 B/s wr, 727 op/s; 292/334893 objects degraded (0.087%); 74202/334893 objects misplaced (22.157%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:08:28.017 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:28.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:28.126 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:28.153 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:28.174 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:28.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:28.278 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:08:28.342 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:28.201307+0000 mon.a (mon.0) 1700 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:29.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:28.201307+0000 mon.a (mon.0) 1700 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:29.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:28.201307+0000 mon.a (mon.0) 1700 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:29.937 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:08:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:29.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:08:30.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:30.074 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:30.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:30.200 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:08:30.229 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:30.251 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:08:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:30.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:08:30.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:30.444 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:30.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:28.769863+0000 mgr.y (mgr.24481) 713 : cluster [DBG] pgmap v983: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 788 op/s; 292/327807 objects degraded (0.089%); 72383/327807 objects misplaced (22.081%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:30.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:28.769863+0000 mgr.y (mgr.24481) 713 : cluster [DBG] pgmap v983: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 788 op/s; 292/327807 objects degraded (0.089%); 72383/327807 objects misplaced (22.081%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:30.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:28.769863+0000 mgr.y (mgr.24481) 713 : cluster [DBG] pgmap v983: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 788 op/s; 292/327807 objects degraded (0.089%); 72383/327807 objects misplaced (22.081%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:30.724 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:30.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:30.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:30.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:08:30.859 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:30.881 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:08:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:30.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:08:30.985 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:31.033 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:08:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:30] "GET /metrics HTTP/1.1" 200 176639 "" "Prometheus/2.33.4" 2024-02-12T17:08:31.046 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:31.298 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:30.771809+0000 mgr.y (mgr.24481) 714 : cluster [DBG] pgmap v984: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 795 op/s; 363/326691 objects degraded (0.111%); 72196/326691 objects misplaced (22.099%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:31.299 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:31.223214+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: Degraded data redundancy: 363/326691 objects degraded (0.111%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:30.771809+0000 mgr.y (mgr.24481) 714 : cluster [DBG] pgmap v984: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 795 op/s; 363/326691 objects degraded (0.111%); 72196/326691 objects misplaced (22.099%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:31.223214+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: Degraded data redundancy: 363/326691 objects degraded (0.111%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:30.771809+0000 mgr.y (mgr.24481) 714 : cluster [DBG] pgmap v984: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 795 op/s; 363/326691 objects degraded (0.111%); 72196/326691 objects misplaced (22.099%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:31.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:31.223214+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: Degraded data redundancy: 363/326691 objects degraded (0.111%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:31.819 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:31.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:31.926 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:31.954 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:31.959 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:08:31.975 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:31.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:32.080 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:08:32.109 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:32.131 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:08:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:32.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:08:32.235 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:08:32.309 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:08:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:32.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:08:32.737 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:32.787 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:32.969 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:32.980 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:08:32.980 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:08:32.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:08:32.980 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:141804 10.15 10.18def8b5 (undecoded) ondisk+write+known_if_redirected e496)", 2024-02-12T17:08:32.980 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:08:32.968698+0000", 2024-02-12T17:08:32.980 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0064651750000000001, 2024-02-12T17:08:32.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0064878069999999999, 2024-02-12T17:08:32.980 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 141804 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:08:32.968698+0000", 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:08:32.981 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:08:32.968698+0000", 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:08:32.968696+0000", 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5299999999999996e-06 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:08:32.968705+0000", 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2614e-05 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:08:32.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:08:32.968717+0000", 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8689999999999998e-06 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:08:32.968724+0000", 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5715e-05 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:08:32.983 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:08:32.984 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 1 2024-02-12T17:08:32.984 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:08:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:32.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:32.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:08:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:08:33.085 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:08:33.181 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:33.328 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:08:33.339 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:08:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:33.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:08:33.381 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:08:33.417 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:08:33.443 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:33.502 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:33.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:32.517394+0000 mon.a (mon.0) 1702 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:33.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:32.773098+0000 mgr.y (mgr.24481) 715 : cluster [DBG] pgmap v985: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 900 B/s rd, 0 B/s wr, 700 op/s; 445/322959 objects degraded (0.138%); 71277/322959 objects misplaced (22.070%); 6.3 MiB/s, 101 objects/s recovering 2024-02-12T17:08:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:32.517394+0000 mon.a (mon.0) 1702 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:33.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:32.773098+0000 mgr.y (mgr.24481) 715 : cluster [DBG] pgmap v985: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 900 B/s rd, 0 B/s wr, 700 op/s; 445/322959 objects degraded (0.138%); 71277/322959 objects misplaced (22.070%); 6.3 MiB/s, 101 objects/s recovering 2024-02-12T17:08:33.969 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T17:08:33.989 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:33.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:32.517394+0000 mon.a (mon.0) 1702 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:08:33.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:32.773098+0000 mgr.y (mgr.24481) 715 : cluster [DBG] pgmap v985: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 900 B/s rd, 0 B/s wr, 700 op/s; 445/322959 objects degraded (0.138%); 71277/322959 objects misplaced (22.070%); 6.3 MiB/s, 101 objects/s recovering 2024-02-12T17:08:34.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:34.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:34.107 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:34.185 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:34.247 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:34.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:34.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:34.408 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:34.819 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:33.361637+0000 mgr.y (mgr.24481) 716 : audit [DBG] from='client.35825 v1:172.21.15.33:0/2372809379' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:08:34.819 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:33.361637+0000 mgr.y (mgr.24481) 716 : audit [DBG] from='client.35825 v1:172.21.15.33:0/2372809379' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:08:34.846 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":985,"stamp":"2024-02-12T17:08:32.772616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7042504380,"num_objects":107653,"num_object_clones":0,"num_object_copies":322959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":445,"num_objects_misplaced":71277,"num_objects_unfound":0,"num_objects_dirty":107653,"num_whiteouts":0,"num_read":1549,"num_read_kb":1301,"num_write":135817,"num_write_kb":7638378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13573,"num_bytes_recovered":849625159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114002,"ondisk_log_size":114002,"up":414,"acting":412,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":26225520,"kb_used_data":21740160,"kb_used_omap":0,"kb_used_meta":4484288,"kb_avail":536245392,"statfs":{"total":575970213888,"available":549115281408,"internally_reserved":0,"allocated":22261923840,"data_stored":22255777752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4591910912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-520945664,"num_objects":-7949,"num_object_clones":0,"num_object_copies":-23847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":261,"num_objects_misplaced":-5846,"num_objects_unfound":0,"num_objects_dirty":-7949,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7949,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1154,"num_bytes_recovered":75628544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.367086"},"pg_stats":[{"pgid":"10.17","version":"496'8164","reported_seq":8634,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:31.034066+0000","last_change":"2024-02-12T17:08:12.978416+0000","last_active":"2024-02-12T17:08:31.034066+0000","last_peered":"2024-02-12T17:08:31.034066+0000","last_clean":"2024-02-12T17:07:53.019897+0000","last_became_active":"2024-02-12T17:08:12.975112+0000","last_became_peered":"2024-02-12T17:08:12.975112+0000","last_unstale":"2024-02-12T17:08:31.034066+0000","last_undegraded":"2024-02-12T17:08:31.034066+0000","last_fullsized":"2024-02-12T17:08:31.034066+0000","mapping_epoch":491,"log_start":"466'4200","ondisk_log_start":"466'4200","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3964,"ondisk_log_size":3964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":217317376,"num_objects":3316,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3316,"num_objects_unfound":0,"num_objects_dirty":3316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4356,"num_write_kb":245504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":12713984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[{"shards":"2,3,5","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":198,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568575+0000","last_change":"2024-02-12T17:08:12.868282+0000","last_active":"2024-02-12T17:08:18.568575+0000","last_peered":"2024-02-12T17:08:18.568575+0000","last_clean":"2024-02-12T17:08:18.568575+0000","last_became_active":"2024-02-12T17:08:12.867921+0000","last_became_peered":"2024-02-12T17:08:12.867921+0000","last_unstale":"2024-02-12T17:08:18.568575+0000","last_undegraded":"2024-02-12T17:08:18.568575+0000","last_fullsized":"2024-02-12T17:08:18.568575+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:44.261302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:44.261302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":178,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573373+0000","last_change":"2024-02-12T17:07:35.884990+0000","last_active":"2024-02-12T17:08:18.573373+0000","last_peered":"2024-02-12T17:08:18.573373+0000","last_clean":"2024-02-12T17:08:18.573373+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:08:18.573373+0000","last_undegraded":"2024-02-12T17:08:18.573373+0000","last_fullsized":"2024-02-12T17:08:18.573373+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"496'8216","reported_seq":15954,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:28.269065+0000","last_change":"2024-02-12T17:08:13.979804+0000","last_active":"2024-02-12T17:08:28.269065+0000","last_peered":"2024-02-12T17:08:28.269065+0000","last_clean":"2024-02-12T17:08:11.807156+0000","last_became_active":"2024-02-12T17:08:13.976755+0000","last_became_peered":"2024-02-12T17:08:13.976755+0000","last_unstale":"2024-02-12T17:08:28.269065+0000","last_undegraded":"2024-02-12T17:08:28.269065+0000","last_fullsized":"2024-02-12T17:08:28.269065+0000","mapping_epoch":492,"log_start":"466'4400","ondisk_log_start":"466'4400","created":480,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3816,"ondisk_log_size":3816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":224133120,"num_objects":3420,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3420,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4364,"num_write_kb":249088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3821,"num_bytes_recovered":250413056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,2,3],"avail_no_missing":["5","2","3","7"],"object_location_counts":[{"shards":"2,3,5","objects":3420}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":162,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572561+0000","last_change":"2024-02-12T17:07:37.890052+0000","last_active":"2024-02-12T17:08:18.572561+0000","last_peered":"2024-02-12T17:08:18.572561+0000","last_clean":"2024-02-12T17:08:18.572561+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:08:18.572561+0000","last_undegraded":"2024-02-12T17:08:18.572561+0000","last_fullsized":"2024-02-12T17:08:18.572561+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:37.889956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.889956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":24,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.583635+0000","last_change":"2024-02-12T17:08:14.031508+0000","last_active":"2024-02-12T17:08:18.583635+0000","last_peered":"2024-02-12T17:08:18.583635+0000","last_clean":"2024-02-12T17:08:18.583635+0000","last_became_active":"2024-02-12T17:08:12.952088+0000","last_became_peered":"2024-02-12T17:08:12.952088+0000","last_unstale":"2024-02-12T17:08:18.583635+0000","last_undegraded":"2024-02-12T17:08:18.583635+0000","last_fullsized":"2024-02-12T17:08:18.583635+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:07:47.845930+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.845930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.15","version":"496'8127","reported_seq":8680,"reported_epoch":496,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-12T17:08:31.019950+0000","last_change":"2024-02-12T17:08:16.657336+0000","last_active":"2024-02-12T17:08:31.019950+0000","last_peered":"2024-02-12T17:08:31.019950+0000","last_clean":"2024-02-12T17:07:53.075624+0000","last_became_active":"2024-02-12T17:08:13.019808+0000","last_became_peered":"2024-02-12T17:08:13.019808+0000","last_unstale":"2024-02-12T17:08:31.019950+0000","last_undegraded":"2024-02-12T17:08:13.003591+0000","last_fullsized":"2024-02-12T17:08:12.830836+0000","mapping_epoch":491,"log_start":"458'1877","ondisk_log_start":"458'1877","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"470'5845","last_scrub_stamp":"2024-02-12T17:07:27.706391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:07:27.706391+0000","log_size":6250,"ondisk_log_size":6250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":215220224,"num_objects":3284,"num_object_clones":0,"num_object_copies":9852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":263,"num_objects_misplaced":6737,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4354,"num_write_kb":244416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":15400960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,2],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":3237},{"shards":"2,3","objects":-216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":495,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568697+0000","last_change":"2024-02-12T17:07:29.165899+0000","last_active":"2024-02-12T17:08:18.568697+0000","last_peered":"2024-02-12T17:08:18.568697+0000","last_clean":"2024-02-12T17:08:18.568697+0000","last_became_active":"2024-02-12T17:03:50.562938+0000","last_became_peered":"2024-02-12T17:03:50.562938+0000","last_unstale":"2024-02-12T17:08:18.568697+0000","last_undegraded":"2024-02-12T17:08:18.568697+0000","last_fullsized":"2024-02-12T17:08:18.568697+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:29.165802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:07:29.165802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":198,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.581381+0000","last_change":"2024-02-12T17:08:13.170514+0000","last_active":"2024-02-12T17:08:18.581381+0000","last_peered":"2024-02-12T17:08:18.581381+0000","last_clean":"2024-02-12T17:08:18.581381+0000","last_became_active":"2024-02-12T17:08:12.907023+0000","last_became_peered":"2024-02-12T17:08:12.907023+0000","last_unstale":"2024-02-12T17:08:18.581381+0000","last_undegraded":"2024-02-12T17:08:18.581381+0000","last_fullsized":"2024-02-12T17:08:18.581381+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7","last_scrub_stamp":"2024-02-12T17:05:56.388366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.388366+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"496'8190","reported_seq":8662,"reported_epoch":496,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:08:30.998693+0000","last_change":"2024-02-12T17:08:15.668531+0000","last_active":"2024-02-12T17:08:30.998693+0000","last_peered":"2024-02-12T17:08:30.998693+0000","last_clean":"2024-02-12T17:07:52.963448+0000","last_became_active":"2024-02-12T17:08:12.974291+0000","last_became_peered":"2024-02-12T17:08:12.974291+0000","last_unstale":"2024-02-12T17:08:30.998693+0000","last_undegraded":"2024-02-12T17:08:30.998693+0000","last_fullsized":"2024-02-12T17:08:30.998693+0000","mapping_epoch":491,"log_start":"466'4200","ondisk_log_start":"466'4200","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3990,"ondisk_log_size":3990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":215056384,"num_objects":3282,"num_object_clones":0,"num_object_copies":9846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3282,"num_objects_unfound":0,"num_objects_dirty":3282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":245664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":12713984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":469,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569493+0000","last_change":"2024-02-12T17:07:39.235715+0000","last_active":"2024-02-12T17:08:18.569493+0000","last_peered":"2024-02-12T17:08:18.569493+0000","last_clean":"2024-02-12T17:08:18.569493+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:08:18.569493+0000","last_undegraded":"2024-02-12T17:08:18.569493+0000","last_fullsized":"2024-02-12T17:08:18.569493+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.235654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.235654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":186,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572864+0000","last_change":"2024-02-12T17:04:40.923654+0000","last_active":"2024-02-12T17:08:18.572864+0000","last_peered":"2024-02-12T17:08:18.572864+0000","last_clean":"2024-02-12T17:08:18.572864+0000","last_became_active":"2024-02-12T17:04:39.865216+0000","last_became_peered":"2024-02-12T17:04:39.865216+0000","last_unstale":"2024-02-12T17:08:18.572864+0000","last_undegraded":"2024-02-12T17:08:18.572864+0000","last_fullsized":"2024-02-12T17:08:18.572864+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"496'8161","reported_seq":16038,"reported_epoch":496,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:08:28.348829+0000","last_change":"2024-02-12T17:08:13.990581+0000","last_active":"2024-02-12T17:08:28.348829+0000","last_peered":"2024-02-12T17:08:28.348829+0000","last_clean":"2024-02-12T17:08:11.822202+0000","last_became_active":"2024-02-12T17:08:13.963868+0000","last_became_peered":"2024-02-12T17:08:13.963868+0000","last_unstale":"2024-02-12T17:08:28.348829+0000","last_undegraded":"2024-02-12T17:08:28.348829+0000","last_fullsized":"2024-02-12T17:08:28.348829+0000","mapping_epoch":492,"log_start":"466'4400","ondisk_log_start":"466'4400","created":480,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3761,"ondisk_log_size":3761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":225050624,"num_objects":3434,"num_object_clones":0,"num_object_copies":10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3434,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4326,"num_write_kb":248320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3896,"num_bytes_recovered":255328256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[2,7,3],"avail_no_missing":["2","3","5","7"],"object_location_counts":[{"shards":"2,3,7","objects":3434}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":171,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.581201+0000","last_change":"2024-02-12T17:06:29.479686+0000","last_active":"2024-02-12T17:08:18.581201+0000","last_peered":"2024-02-12T17:08:18.581201+0000","last_clean":"2024-02-12T17:08:18.581201+0000","last_became_active":"2024-02-12T17:04:39.887235+0000","last_became_peered":"2024-02-12T17:04:39.887235+0000","last_unstale":"2024-02-12T17:08:18.581201+0000","last_undegraded":"2024-02-12T17:08:18.581201+0000","last_fullsized":"2024-02-12T17:08:18.581201+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:29.479610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T17:06:29.479610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":208,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579328+0000","last_change":"2024-02-12T17:08:15.499025+0000","last_active":"2024-02-12T17:08:18.579328+0000","last_peered":"2024-02-12T17:08:18.579328+0000","last_clean":"2024-02-12T17:08:18.579328+0000","last_became_active":"2024-02-12T17:08:13.009087+0000","last_became_peered":"2024-02-12T17:08:13.009087+0000","last_unstale":"2024-02-12T17:08:18.579328+0000","last_undegraded":"2024-02-12T17:08:18.579328+0000","last_fullsized":"2024-02-12T17:08:18.579328+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"496'8055","reported_seq":8543,"reported_epoch":496,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:08:28.312587+0000","last_change":"2024-02-12T17:08:12.910579+0000","last_active":"2024-02-12T17:08:28.312587+0000","last_peered":"2024-02-12T17:08:28.312587+0000","last_clean":"2024-02-12T17:07:53.037515+0000","last_became_active":"2024-02-12T17:07:55.105679+0000","last_became_peered":"2024-02-12T17:07:55.105679+0000","last_unstale":"2024-02-12T17:08:28.312587+0000","last_undegraded":"2024-02-12T17:08:28.312587+0000","last_fullsized":"2024-02-12T17:08:28.312587+0000","mapping_epoch":483,"log_start":"466'4100","ondisk_log_start":"466'4100","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3955,"ondisk_log_size":3955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":219316224,"num_objects":3347,"num_object_clones":0,"num_object_copies":10041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3173,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4285,"num_write_kb":244192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":13500416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,7,3],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":3347}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":146,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568311+0000","last_change":"2024-02-12T17:08:12.865098+0000","last_active":"2024-02-12T17:08:18.568311+0000","last_peered":"2024-02-12T17:08:18.568311+0000","last_clean":"2024-02-12T17:08:18.568311+0000","last_became_active":"2024-02-12T17:08:12.864728+0000","last_became_peered":"2024-02-12T17:08:12.864728+0000","last_unstale":"2024-02-12T17:08:18.568311+0000","last_undegraded":"2024-02-12T17:08:18.568311+0000","last_fullsized":"2024-02-12T17:08:18.568311+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:52.828449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:06:52.828449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":30,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.583512+0000","last_change":"2024-02-12T17:08:17.496274+0000","last_active":"2024-02-12T17:08:18.583512+0000","last_peered":"2024-02-12T17:08:18.583512+0000","last_clean":"2024-02-12T17:08:18.583512+0000","last_became_active":"2024-02-12T17:08:12.960006+0000","last_became_peered":"2024-02-12T17:08:12.960006+0000","last_unstale":"2024-02-12T17:08:18.583512+0000","last_undegraded":"2024-02-12T17:08:18.583512+0000","last_fullsized":"2024-02-12T17:08:18.583512+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"496'8286","reported_seq":8356,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:28.342381+0000","last_change":"2024-02-12T17:07:54.077898+0000","last_active":"2024-02-12T17:08:28.342381+0000","last_peered":"2024-02-12T17:08:28.342381+0000","last_clean":"2024-02-12T17:08:28.342381+0000","last_became_active":"2024-02-12T17:07:54.073001+0000","last_became_peered":"2024-02-12T17:07:54.073001+0000","last_unstale":"2024-02-12T17:08:28.342381+0000","last_undegraded":"2024-02-12T17:08:28.342381+0000","last_fullsized":"2024-02-12T17:08:28.342381+0000","mapping_epoch":482,"log_start":"466'4500","ondisk_log_start":"466'4500","created":480,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3786,"ondisk_log_size":3786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":225607680,"num_objects":3443,"num_object_clones":0,"num_object_copies":10329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":251360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":157,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573849+0000","last_change":"2024-02-12T17:03:50.533143+0000","last_active":"2024-02-12T17:08:18.573849+0000","last_peered":"2024-02-12T17:08:18.573849+0000","last_clean":"2024-02-12T17:08:18.573849+0000","last_became_active":"2024-02-12T17:03:50.532233+0000","last_became_peered":"2024-02-12T17:03:50.532233+0000","last_unstale":"2024-02-12T17:08:18.573849+0000","last_undegraded":"2024-02-12T17:08:18.573849+0000","last_fullsized":"2024-02-12T17:08:18.573849+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":378,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573121+0000","last_change":"2024-02-12T17:08:13.700872+0000","last_active":"2024-02-12T17:08:18.573121+0000","last_peered":"2024-02-12T17:08:18.573121+0000","last_clean":"2024-02-12T17:08:18.573121+0000","last_became_active":"2024-02-12T17:08:13.012586+0000","last_became_peered":"2024-02-12T17:08:13.012586+0000","last_unstale":"2024-02-12T17:08:18.573121+0000","last_undegraded":"2024-02-12T17:08:18.573121+0000","last_fullsized":"2024-02-12T17:08:18.573121+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.10","version":"496'7922","reported_seq":7983,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:28.688528+0000","last_change":"2024-02-12T17:07:52.066684+0000","last_active":"2024-02-12T17:08:28.688528+0000","last_peered":"2024-02-12T17:08:28.688528+0000","last_clean":"2024-02-12T17:08:28.688528+0000","last_became_active":"2024-02-12T17:07:52.062972+0000","last_became_peered":"2024-02-12T17:07:52.062972+0000","last_unstale":"2024-02-12T17:08:28.688528+0000","last_undegraded":"2024-02-12T17:08:28.688528+0000","last_fullsized":"2024-02-12T17:08:28.688528+0000","mapping_epoch":480,"log_start":"466'4200","ondisk_log_start":"466'4200","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3722,"ondisk_log_size":3722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":217776128,"num_objects":3323,"num_object_clones":0,"num_object_copies":9969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4201,"num_write_kb":240768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":485,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572211+0000","last_change":"2024-02-12T17:08:13.011770+0000","last_active":"2024-02-12T17:08:18.572211+0000","last_peered":"2024-02-12T17:08:18.572211+0000","last_clean":"2024-02-12T17:08:18.572211+0000","last_became_active":"2024-02-12T17:08:13.002767+0000","last_became_peered":"2024-02-12T17:08:13.002767+0000","last_unstale":"2024-02-12T17:08:18.572211+0000","last_undegraded":"2024-02-12T17:08:18.572211+0000","last_fullsized":"2024-02-12T17:08:18.572211+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:55.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.027189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":537,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569090+0000","last_change":"2024-02-12T17:08:13.337827+0000","last_active":"2024-02-12T17:08:18.569090+0000","last_peered":"2024-02-12T17:08:18.569090+0000","last_clean":"2024-02-12T17:08:18.569090+0000","last_became_active":"2024-02-12T17:08:12.867632+0000","last_became_peered":"2024-02-12T17:08:12.867632+0000","last_unstale":"2024-02-12T17:08:18.569090+0000","last_undegraded":"2024-02-12T17:08:18.569090+0000","last_fullsized":"2024-02-12T17:08:18.569090+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":153,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568337+0000","last_change":"2024-02-12T17:05:45.618872+0000","last_active":"2024-02-12T17:08:18.568337+0000","last_peered":"2024-02-12T17:08:18.568337+0000","last_clean":"2024-02-12T17:08:18.568337+0000","last_became_active":"2024-02-12T17:05:45.618551+0000","last_became_peered":"2024-02-12T17:05:45.618551+0000","last_unstale":"2024-02-12T17:08:18.568337+0000","last_undegraded":"2024-02-12T17:08:18.568337+0000","last_fullsized":"2024-02-12T17:08:18.568337+0000","mapping_epoch":436,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":171,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.581468+0000","last_change":"2024-02-12T17:06:03.551775+0000","last_active":"2024-02-12T17:08:18.581468+0000","last_peered":"2024-02-12T17:08:18.581468+0000","last_clean":"2024-02-12T17:08:18.581468+0000","last_became_active":"2024-02-12T17:04:39.877005+0000","last_became_peered":"2024-02-12T17:04:39.877005+0000","last_unstale":"2024-02-12T17:08:18.581468+0000","last_undegraded":"2024-02-12T17:08:18.581468+0000","last_fullsized":"2024-02-12T17:08:18.581468+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":282,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572909+0000","last_change":"2024-02-12T17:08:13.012881+0000","last_active":"2024-02-12T17:08:18.572909+0000","last_peered":"2024-02-12T17:08:18.572909+0000","last_clean":"2024-02-12T17:08:18.572909+0000","last_became_active":"2024-02-12T17:08:12.972014+0000","last_became_peered":"2024-02-12T17:08:12.972014+0000","last_unstale":"2024-02-12T17:08:18.572909+0000","last_undegraded":"2024-02-12T17:08:18.572909+0000","last_fullsized":"2024-02-12T17:08:18.572909+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:31.071549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:31.071549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":169,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573652+0000","last_change":"2024-02-12T17:03:59.035349+0000","last_active":"2024-02-12T17:08:18.573652+0000","last_peered":"2024-02-12T17:08:18.573652+0000","last_clean":"2024-02-12T17:08:18.573652+0000","last_became_active":"2024-02-12T17:03:50.639321+0000","last_became_peered":"2024-02-12T17:03:50.639321+0000","last_unstale":"2024-02-12T17:08:18.573652+0000","last_undegraded":"2024-02-12T17:08:18.573652+0000","last_fullsized":"2024-02-12T17:08:18.573652+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":504,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.567987+0000","last_change":"2024-02-12T17:08:12.868061+0000","last_active":"2024-02-12T17:08:18.567987+0000","last_peered":"2024-02-12T17:08:18.567987+0000","last_clean":"2024-02-12T17:08:18.567987+0000","last_became_active":"2024-02-12T17:08:12.867679+0000","last_became_peered":"2024-02-12T17:08:12.867679+0000","last_unstale":"2024-02-12T17:08:18.567987+0000","last_undegraded":"2024-02-12T17:08:18.567987+0000","last_fullsized":"2024-02-12T17:08:18.567987+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":467,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568444+0000","last_change":"2024-02-12T17:06:03.911480+0000","last_active":"2024-02-12T17:08:18.568444+0000","last_peered":"2024-02-12T17:08:18.568444+0000","last_clean":"2024-02-12T17:08:18.568444+0000","last_became_active":"2024-02-12T17:04:39.886041+0000","last_became_peered":"2024-02-12T17:04:39.886041+0000","last_unstale":"2024-02-12T17:08:18.568444+0000","last_undegraded":"2024-02-12T17:08:18.568444+0000","last_fullsized":"2024-02-12T17:08:18.568444+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.911375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.911375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":503,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572787+0000","last_change":"2024-02-12T17:08:12.953357+0000","last_active":"2024-02-12T17:08:18.572787+0000","last_peered":"2024-02-12T17:08:18.572787+0000","last_clean":"2024-02-12T17:08:18.572787+0000","last_became_active":"2024-02-12T17:08:12.953021+0000","last_became_peered":"2024-02-12T17:08:12.953021+0000","last_unstale":"2024-02-12T17:08:18.572787+0000","last_undegraded":"2024-02-12T17:08:18.572787+0000","last_fullsized":"2024-02-12T17:08:18.572787+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:34.981898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.981898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":198,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.576782+0000","last_change":"2024-02-12T17:08:15.064224+0000","last_active":"2024-02-12T17:08:18.576782+0000","last_peered":"2024-02-12T17:08:18.576782+0000","last_clean":"2024-02-12T17:08:18.576782+0000","last_became_active":"2024-02-12T17:08:13.005895+0000","last_became_peered":"2024-02-12T17:08:13.005895+0000","last_unstale":"2024-02-12T17:08:18.576782+0000","last_undegraded":"2024-02-12T17:08:18.576782+0000","last_fullsized":"2024-02-12T17:08:18.576782+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579284+0000","last_change":"2024-02-12T17:08:12.961296+0000","last_active":"2024-02-12T17:08:18.579284+0000","last_peered":"2024-02-12T17:08:18.579284+0000","last_clean":"2024-02-12T17:08:18.579284+0000","last_became_active":"2024-02-12T17:08:12.960364+0000","last_became_peered":"2024-02-12T17:08:12.960364+0000","last_unstale":"2024-02-12T17:08:18.579284+0000","last_undegraded":"2024-02-12T17:08:18.579284+0000","last_fullsized":"2024-02-12T17:08:18.579284+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":162,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573502+0000","last_change":"2024-02-12T17:07:36.839451+0000","last_active":"2024-02-12T17:08:18.573502+0000","last_peered":"2024-02-12T17:08:18.573502+0000","last_clean":"2024-02-12T17:08:18.573502+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:08:18.573502+0000","last_undegraded":"2024-02-12T17:08:18.573502+0000","last_fullsized":"2024-02-12T17:08:18.573502+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.586662+0000","last_change":"2024-02-12T17:08:12.956341+0000","last_active":"2024-02-12T17:08:18.586662+0000","last_peered":"2024-02-12T17:08:18.586662+0000","last_clean":"2024-02-12T17:08:18.586662+0000","last_became_active":"2024-02-12T17:08:12.955449+0000","last_became_peered":"2024-02-12T17:08:12.955449+0000","last_unstale":"2024-02-12T17:08:18.586662+0000","last_undegraded":"2024-02-12T17:08:18.586662+0000","last_fullsized":"2024-02-12T17:08:18.586662+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":26,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.584412+0000","last_change":"2024-02-12T17:08:13.844476+0000","last_active":"2024-02-12T17:08:18.584412+0000","last_peered":"2024-02-12T17:08:18.584412+0000","last_clean":"2024-02-12T17:08:18.584412+0000","last_became_active":"2024-02-12T17:08:12.907219+0000","last_became_peered":"2024-02-12T17:08:12.907219+0000","last_unstale":"2024-02-12T17:08:18.584412+0000","last_undegraded":"2024-02-12T17:08:18.584412+0000","last_fullsized":"2024-02-12T17:08:18.584412+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:38.751057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.751057+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":521,"reported_epoch":495,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:18.576693+0000","last_change":"2024-02-12T17:08:13.981566+0000","last_active":"2024-02-12T17:08:18.576693+0000","last_peered":"2024-02-12T17:08:18.576693+0000","last_clean":"2024-02-12T17:08:07.414375+0000","last_became_active":"2024-02-12T17:08:13.981281+0000","last_became_peered":"2024-02-12T17:08:13.981281+0000","last_unstale":"2024-02-12T17:08:18.576693+0000","last_undegraded":"2024-02-12T17:08:18.576693+0000","last_fullsized":"2024-02-12T17:08:18.576693+0000","mapping_epoch":492,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":419,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.571830+0000","last_change":"2024-02-12T17:08:13.012437+0000","last_active":"2024-02-12T17:08:18.571830+0000","last_peered":"2024-02-12T17:08:18.571830+0000","last_clean":"2024-02-12T17:08:18.571830+0000","last_became_active":"2024-02-12T17:08:13.002761+0000","last_became_peered":"2024-02-12T17:08:13.002761+0000","last_unstale":"2024-02-12T17:08:18.571830+0000","last_undegraded":"2024-02-12T17:08:18.571830+0000","last_fullsized":"2024-02-12T17:08:18.571830+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":157,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.575979+0000","last_change":"2024-02-12T17:03:50.559509+0000","last_active":"2024-02-12T17:08:18.575979+0000","last_peered":"2024-02-12T17:08:18.575979+0000","last_clean":"2024-02-12T17:08:18.575979+0000","last_became_active":"2024-02-12T17:03:50.558598+0000","last_became_peered":"2024-02-12T17:03:50.558598+0000","last_unstale":"2024-02-12T17:08:18.575979+0000","last_undegraded":"2024-02-12T17:08:18.575979+0000","last_fullsized":"2024-02-12T17:08:18.575979+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":24,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579545+0000","last_change":"2024-02-12T17:08:13.176526+0000","last_active":"2024-02-12T17:08:18.579545+0000","last_peered":"2024-02-12T17:08:18.579545+0000","last_clean":"2024-02-12T17:08:18.579545+0000","last_became_active":"2024-02-12T17:08:12.908811+0000","last_became_peered":"2024-02-12T17:08:12.908811+0000","last_unstale":"2024-02-12T17:08:18.579545+0000","last_undegraded":"2024-02-12T17:08:18.579545+0000","last_fullsized":"2024-02-12T17:08:18.579545+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":495,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:22.497556+0000","last_change":"2024-02-12T17:05:55.414790+0000","last_active":"2024-02-12T17:08:22.497556+0000","last_peered":"2024-02-12T17:08:22.497556+0000","last_clean":"2024-02-12T17:08:22.497556+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:08:22.497556+0000","last_undegraded":"2024-02-12T17:08:22.497556+0000","last_fullsized":"2024-02-12T17:08:22.497556+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":250,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.580984+0000","last_change":"2024-02-12T17:06:51.622488+0000","last_active":"2024-02-12T17:08:18.580984+0000","last_peered":"2024-02-12T17:08:18.580984+0000","last_clean":"2024-02-12T17:08:18.580984+0000","last_became_active":"2024-02-12T17:04:39.886343+0000","last_became_peered":"2024-02-12T17:04:39.886343+0000","last_unstale":"2024-02-12T17:08:18.580984+0000","last_undegraded":"2024-02-12T17:08:18.580984+0000","last_fullsized":"2024-02-12T17:08:18.580984+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.622401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.622401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":572,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572748+0000","last_change":"2024-02-12T17:07:33.734742+0000","last_active":"2024-02-12T17:08:18.572748+0000","last_peered":"2024-02-12T17:08:18.572748+0000","last_clean":"2024-02-12T17:08:18.572748+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:08:18.572748+0000","last_undegraded":"2024-02-12T17:08:18.572748+0000","last_fullsized":"2024-02-12T17:08:18.572748+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:33.734675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:33.734675+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"493'19","reported_seq":454,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:22.492953+0000","last_change":"2024-02-12T17:08:13.896853+0000","last_active":"2024-02-12T17:08:22.492953+0000","last_peered":"2024-02-12T17:08:22.492953+0000","last_clean":"2024-02-12T17:08:07.495708+0000","last_became_active":"2024-02-12T17:08:13.896577+0000","last_became_peered":"2024-02-12T17:08:13.896577+0000","last_unstale":"2024-02-12T17:08:22.492953+0000","last_undegraded":"2024-02-12T17:08:22.492953+0000","last_fullsized":"2024-02-12T17:08:22.492953+0000","mapping_epoch":492,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,3,7],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":485,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.571748+0000","last_change":"2024-02-12T17:07:35.798822+0000","last_active":"2024-02-12T17:08:18.571748+0000","last_peered":"2024-02-12T17:08:18.571748+0000","last_clean":"2024-02-12T17:08:18.571748+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:08:18.571748+0000","last_undegraded":"2024-02-12T17:08:18.571748+0000","last_fullsized":"2024-02-12T17:08:18.571748+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:35.798720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.798720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":195,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.575900+0000","last_change":"2024-02-12T17:08:14.038528+0000","last_active":"2024-02-12T17:08:18.575900+0000","last_peered":"2024-02-12T17:08:18.575900+0000","last_clean":"2024-02-12T17:08:18.575900+0000","last_became_active":"2024-02-12T17:08:12.934950+0000","last_became_peered":"2024-02-12T17:08:12.934950+0000","last_unstale":"2024-02-12T17:08:18.575900+0000","last_undegraded":"2024-02-12T17:08:18.575900+0000","last_fullsized":"2024-02-12T17:08:18.575900+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"496'8236","reported_seq":19497,"reported_epoch":496,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:08:28.382468+0000","last_change":"2024-02-12T17:08:17.766683+0000","last_active":"2024-02-12T17:08:28.382468+0000","last_peered":"2024-02-12T17:08:28.382468+0000","last_clean":"2024-02-12T17:07:53.019577+0000","last_became_active":"2024-02-12T17:08:12.945314+0000","last_became_peered":"2024-02-12T17:08:12.945314+0000","last_unstale":"2024-02-12T17:08:28.382468+0000","last_undegraded":"2024-02-12T17:08:28.382468+0000","last_fullsized":"2024-02-12T17:08:28.382468+0000","mapping_epoch":491,"log_start":"466'4500","ondisk_log_start":"466'4500","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3736,"ondisk_log_size":3736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":229212160,"num_objects":3498,"num_object_clones":0,"num_object_copies":10494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4071,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4356,"num_write_kb":251296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4053,"num_bytes_recovered":265617408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[2,5,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":480,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568622+0000","last_change":"2024-02-12T17:06:37.116695+0000","last_active":"2024-02-12T17:08:18.568622+0000","last_peered":"2024-02-12T17:08:18.568622+0000","last_clean":"2024-02-12T17:08:18.568622+0000","last_became_active":"2024-02-12T17:04:39.889055+0000","last_became_peered":"2024-02-12T17:04:39.889055+0000","last_unstale":"2024-02-12T17:08:18.568622+0000","last_undegraded":"2024-02-12T17:08:18.568622+0000","last_fullsized":"2024-02-12T17:08:18.568622+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:37.116609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:06:37.116609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579885+0000","last_change":"2024-02-12T17:08:12.965855+0000","last_active":"2024-02-12T17:08:18.579885+0000","last_peered":"2024-02-12T17:08:18.579885+0000","last_clean":"2024-02-12T17:08:18.579885+0000","last_became_active":"2024-02-12T17:08:12.961766+0000","last_became_peered":"2024-02-12T17:08:12.961766+0000","last_unstale":"2024-02-12T17:08:18.579885+0000","last_undegraded":"2024-02-12T17:08:18.579885+0000","last_fullsized":"2024-02-12T17:08:18.579885+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:34.780676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:34.780676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"460'15","reported_seq":190,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.576331+0000","last_change":"2024-02-12T17:08:14.414273+0000","last_active":"2024-02-12T17:08:18.576331+0000","last_peered":"2024-02-12T17:08:18.576331+0000","last_clean":"2024-02-12T17:08:18.576331+0000","last_became_active":"2024-02-12T17:08:12.953326+0000","last_became_peered":"2024-02-12T17:08:12.953326+0000","last_unstale":"2024-02-12T17:08:18.576331+0000","last_undegraded":"2024-02-12T17:08:18.576331+0000","last_fullsized":"2024-02-12T17:08:18.576331+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"496'8033","reported_seq":9208,"reported_epoch":496,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-12T17:08:28.792300+0000","last_change":"2024-02-12T17:08:14.089150+0000","last_active":"2024-02-12T17:08:28.792300+0000","last_peered":"2024-02-12T17:08:28.792300+0000","last_clean":"2024-02-12T17:07:53.055073+0000","last_became_active":"2024-02-12T17:08:13.093882+0000","last_became_peered":"2024-02-12T17:08:13.093882+0000","last_unstale":"2024-02-12T17:08:28.792300+0000","last_undegraded":"2024-02-12T17:08:14.033538+0000","last_fullsized":"2024-02-12T17:08:12.833048+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":8033,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":219381760,"num_objects":3348,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":182,"num_objects_misplaced":6976,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4268,"num_write_kb":243680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":23855104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[0,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2","objects":-232},{"shards":"0,2,3,5","objects":1},{"shards":"0,2,3,5,7","objects":1},{"shards":"0,2,7","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":137,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569625+0000","last_change":"2024-02-12T17:07:45.273035+0000","last_active":"2024-02-12T17:08:18.569625+0000","last_peered":"2024-02-12T17:08:18.569625+0000","last_clean":"2024-02-12T17:08:18.569625+0000","last_became_active":"2024-02-12T17:04:39.936538+0000","last_became_peered":"2024-02-12T17:04:39.936538+0000","last_unstale":"2024-02-12T17:08:18.569625+0000","last_undegraded":"2024-02-12T17:08:18.569625+0000","last_fullsized":"2024-02-12T17:08:18.569625+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.272939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.272939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.580119+0000","last_change":"2024-02-12T17:08:13.026852+0000","last_active":"2024-02-12T17:08:18.580119+0000","last_peered":"2024-02-12T17:08:18.580119+0000","last_clean":"2024-02-12T17:08:18.580119+0000","last_became_active":"2024-02-12T17:08:13.026370+0000","last_became_peered":"2024-02-12T17:08:13.026370+0000","last_unstale":"2024-02-12T17:08:18.580119+0000","last_undegraded":"2024-02-12T17:08:18.580119+0000","last_fullsized":"2024-02-12T17:08:18.580119+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":172,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572657+0000","last_change":"2024-02-12T17:06:45.502715+0000","last_active":"2024-02-12T17:08:18.572657+0000","last_peered":"2024-02-12T17:08:18.572657+0000","last_clean":"2024-02-12T17:08:18.572657+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:08:18.572657+0000","last_undegraded":"2024-02-12T17:08:18.572657+0000","last_fullsized":"2024-02-12T17:08:18.572657+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"496'8210","reported_seq":8273,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:30.916918+0000","last_change":"2024-02-12T17:07:52.073731+0000","last_active":"2024-02-12T17:08:30.916918+0000","last_peered":"2024-02-12T17:08:30.916918+0000","last_clean":"2024-02-12T17:08:30.916918+0000","last_became_active":"2024-02-12T17:07:52.071694+0000","last_became_peered":"2024-02-12T17:07:52.071694+0000","last_unstale":"2024-02-12T17:08:30.916918+0000","last_undegraded":"2024-02-12T17:08:30.916918+0000","last_fullsized":"2024-02-12T17:08:30.916918+0000","mapping_epoch":451,"log_start":"466'4200","ondisk_log_start":"466'4200","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4010,"ondisk_log_size":4010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":216924160,"num_objects":3310,"num_object_clones":0,"num_object_copies":9930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4402,"num_write_kb":246784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":198,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.581284+0000","last_change":"2024-02-12T17:08:12.916367+0000","last_active":"2024-02-12T17:08:18.581284+0000","last_peered":"2024-02-12T17:08:18.581284+0000","last_clean":"2024-02-12T17:08:18.581284+0000","last_became_active":"2024-02-12T17:08:12.906537+0000","last_became_peered":"2024-02-12T17:08:12.906537+0000","last_unstale":"2024-02-12T17:08:18.581284+0000","last_undegraded":"2024-02-12T17:08:18.581284+0000","last_fullsized":"2024-02-12T17:08:18.581284+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":199,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568671+0000","last_change":"2024-02-12T17:07:32.199573+0000","last_active":"2024-02-12T17:08:18.568671+0000","last_peered":"2024-02-12T17:08:18.568671+0000","last_clean":"2024-02-12T17:08:18.568671+0000","last_became_active":"2024-02-12T17:04:39.890440+0000","last_became_peered":"2024-02-12T17:04:39.890440+0000","last_unstale":"2024-02-12T17:08:18.568671+0000","last_undegraded":"2024-02-12T17:08:18.568671+0000","last_fullsized":"2024-02-12T17:08:18.568671+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:07:32.199469+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:07:32.199469+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":26,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579983+0000","last_change":"2024-02-12T17:08:17.580410+0000","last_active":"2024-02-12T17:08:18.579983+0000","last_peered":"2024-02-12T17:08:18.579983+0000","last_clean":"2024-02-12T17:08:18.579983+0000","last_became_active":"2024-02-12T17:08:12.962272+0000","last_became_peered":"2024-02-12T17:08:12.962272+0000","last_unstale":"2024-02-12T17:08:18.579983+0000","last_undegraded":"2024-02-12T17:08:18.579983+0000","last_fullsized":"2024-02-12T17:08:18.579983+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.583729+0000","last_change":"2024-02-12T17:08:12.899966+0000","last_active":"2024-02-12T17:08:18.583729+0000","last_peered":"2024-02-12T17:08:18.583729+0000","last_clean":"2024-02-12T17:08:18.583729+0000","last_became_active":"2024-02-12T17:08:12.899606+0000","last_became_peered":"2024-02-12T17:08:12.899606+0000","last_unstale":"2024-02-12T17:08:18.583729+0000","last_undegraded":"2024-02-12T17:08:18.583729+0000","last_fullsized":"2024-02-12T17:08:18.583729+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"496'8678","reported_seq":8743,"reported_epoch":496,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:08:28.727887+0000","last_change":"2024-02-12T17:08:13.962728+0000","last_active":"2024-02-12T17:08:28.727887+0000","last_peered":"2024-02-12T17:08:28.727887+0000","last_clean":"2024-02-12T17:08:11.822229+0000","last_became_active":"2024-02-12T17:08:13.954131+0000","last_became_peered":"2024-02-12T17:08:13.954131+0000","last_unstale":"2024-02-12T17:08:28.727887+0000","last_undegraded":"2024-02-12T17:08:28.727887+0000","last_fullsized":"2024-02-12T17:08:28.727887+0000","mapping_epoch":492,"log_start":"468'5010","ondisk_log_start":"468'5010","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3668,"ondisk_log_size":3668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":444071936,"num_objects":6776,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6776,"num_objects_unfound":0,"num_objects_dirty":6776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8678,"num_write_kb":494528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,2],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":6776}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":482,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.605206+0000","last_change":"2024-02-12T17:07:23.791961+0000","last_active":"2024-02-12T17:08:18.605206+0000","last_peered":"2024-02-12T17:08:18.605206+0000","last_clean":"2024-02-12T17:08:18.605206+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:08:18.605206+0000","last_undegraded":"2024-02-12T17:08:18.605206+0000","last_fullsized":"2024-02-12T17:08:18.605206+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:23.791870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:23.791870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.583333+0000","last_change":"2024-02-12T17:08:12.910933+0000","last_active":"2024-02-12T17:08:18.583333+0000","last_peered":"2024-02-12T17:08:18.583333+0000","last_clean":"2024-02-12T17:08:18.583333+0000","last_became_active":"2024-02-12T17:08:12.910205+0000","last_became_peered":"2024-02-12T17:08:12.910205+0000","last_unstale":"2024-02-12T17:08:18.583333+0000","last_undegraded":"2024-02-12T17:08:18.583333+0000","last_fullsized":"2024-02-12T17:08:18.583333+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":26,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579208+0000","last_change":"2024-02-12T17:08:12.983471+0000","last_active":"2024-02-12T17:08:18.579208+0000","last_peered":"2024-02-12T17:08:18.579208+0000","last_clean":"2024-02-12T17:08:18.579208+0000","last_became_active":"2024-02-12T17:08:12.904961+0000","last_became_peered":"2024-02-12T17:08:12.904961+0000","last_unstale":"2024-02-12T17:08:18.579208+0000","last_undegraded":"2024-02-12T17:08:18.579208+0000","last_fullsized":"2024-02-12T17:08:18.579208+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:04.040555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:04.040555+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"493'17","reported_seq":697,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:22.492975+0000","last_change":"2024-02-12T17:08:13.948188+0000","last_active":"2024-02-12T17:08:22.492975+0000","last_peered":"2024-02-12T17:08:22.492975+0000","last_clean":"2024-02-12T17:08:07.495872+0000","last_became_active":"2024-02-12T17:08:13.947046+0000","last_became_peered":"2024-02-12T17:08:13.947046+0000","last_unstale":"2024-02-12T17:08:22.492975+0000","last_undegraded":"2024-02-12T17:08:22.492975+0000","last_fullsized":"2024-02-12T17:08:22.492975+0000","mapping_epoch":492,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,7],"avail_no_missing":["5","1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"496'8578","reported_seq":8631,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:28.260173+0000","last_change":"2024-02-12T17:06:18.391956+0000","last_active":"2024-02-12T17:08:28.260173+0000","last_peered":"2024-02-12T17:08:28.260173+0000","last_clean":"2024-02-12T17:08:28.260173+0000","last_became_active":"2024-02-12T17:06:18.391610+0000","last_became_peered":"2024-02-12T17:06:18.391610+0000","last_unstale":"2024-02-12T17:08:28.260173+0000","last_undegraded":"2024-02-12T17:08:28.260173+0000","last_fullsized":"2024-02-12T17:08:28.260173+0000","mapping_epoch":451,"log_start":"468'5012","ondisk_log_start":"468'5012","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3566,"ondisk_log_size":3566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437125120,"num_objects":6670,"num_object_clones":0,"num_object_copies":20010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":487936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":184,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572241+0000","last_change":"2024-02-12T17:04:58.474896+0000","last_active":"2024-02-12T17:08:18.572241+0000","last_peered":"2024-02-12T17:08:18.572241+0000","last_clean":"2024-02-12T17:08:18.572241+0000","last_became_active":"2024-02-12T17:04:58.474504+0000","last_became_peered":"2024-02-12T17:04:58.474504+0000","last_unstale":"2024-02-12T17:08:18.572241+0000","last_undegraded":"2024-02-12T17:08:18.572241+0000","last_fullsized":"2024-02-12T17:08:18.572241+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":508,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.571854+0000","last_change":"2024-02-12T17:08:12.953289+0000","last_active":"2024-02-12T17:08:18.571854+0000","last_peered":"2024-02-12T17:08:18.571854+0000","last_clean":"2024-02-12T17:08:18.571854+0000","last_became_active":"2024-02-12T17:08:12.952951+0000","last_became_peered":"2024-02-12T17:08:12.952951+0000","last_unstale":"2024-02-12T17:08:18.571854+0000","last_undegraded":"2024-02-12T17:08:18.571854+0000","last_fullsized":"2024-02-12T17:08:18.571854+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"492'56","reported_seq":40,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:22.495392+0000","last_change":"2024-02-12T17:08:16.636920+0000","last_active":"2024-02-12T17:08:22.495392+0000","last_peered":"2024-02-12T17:08:22.495392+0000","last_clean":"2024-02-12T17:08:22.495392+0000","last_became_active":"2024-02-12T17:08:12.958778+0000","last_became_peered":"2024-02-12T17:08:12.958778+0000","last_unstale":"2024-02-12T17:08:22.495392+0000","last_undegraded":"2024-02-12T17:08:22.495392+0000","last_fullsized":"2024-02-12T17:08:22.495392+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":341,"reported_epoch":495,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:08:18.593564+0000","last_change":"2024-02-12T17:08:13.930375+0000","last_active":"2024-02-12T17:08:18.593564+0000","last_peered":"2024-02-12T17:08:18.593564+0000","last_clean":"2024-02-12T17:05:12.349531+0000","last_became_active":"2024-02-12T17:08:13.904741+0000","last_became_peered":"2024-02-12T17:08:13.904741+0000","last_unstale":"2024-02-12T17:08:18.593564+0000","last_undegraded":"2024-02-12T17:08:18.593564+0000","last_fullsized":"2024-02-12T17:08:18.593564+0000","mapping_epoch":492,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[2,7,3],"avail_no_missing":["2","1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"496'8278","reported_seq":8341,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:28.334707+0000","last_change":"2024-02-12T17:07:52.060549+0000","last_active":"2024-02-12T17:08:28.334707+0000","last_peered":"2024-02-12T17:08:28.334707+0000","last_clean":"2024-02-12T17:08:28.334707+0000","last_became_active":"2024-02-12T17:07:52.058737+0000","last_became_peered":"2024-02-12T17:07:52.058737+0000","last_unstale":"2024-02-12T17:08:28.334707+0000","last_undegraded":"2024-02-12T17:08:28.334707+0000","last_fullsized":"2024-02-12T17:08:28.334707+0000","mapping_epoch":451,"log_start":"466'4500","ondisk_log_start":"466'4500","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3778,"ondisk_log_size":3778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":225673216,"num_objects":3443,"num_object_clones":0,"num_object_copies":10329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4397,"num_write_kb":250912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":487,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568901+0000","last_change":"2024-02-12T17:06:46.001986+0000","last_active":"2024-02-12T17:08:18.568901+0000","last_peered":"2024-02-12T17:08:18.568901+0000","last_clean":"2024-02-12T17:08:18.568901+0000","last_became_active":"2024-02-12T17:03:50.564700+0000","last_became_peered":"2024-02-12T17:03:50.564700+0000","last_unstale":"2024-02-12T17:08:18.568901+0000","last_undegraded":"2024-02-12T17:08:18.568901+0000","last_fullsized":"2024-02-12T17:08:18.568901+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.001888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.001888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":34,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:27.087638+0000","last_change":"2024-02-12T17:08:17.769666+0000","last_active":"2024-02-12T17:08:27.087638+0000","last_peered":"2024-02-12T17:08:27.087638+0000","last_clean":"2024-02-12T17:08:27.087638+0000","last_became_active":"2024-02-12T17:08:13.025836+0000","last_became_peered":"2024-02-12T17:08:13.025836+0000","last_unstale":"2024-02-12T17:08:27.087638+0000","last_undegraded":"2024-02-12T17:08:27.087638+0000","last_fullsized":"2024-02-12T17:08:27.087638+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":431,"num_read_kb":426,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":162,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573777+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:08:18.573777+0000","last_peered":"2024-02-12T17:08:18.573777+0000","last_clean":"2024-02-12T17:08:18.573777+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:08:18.573777+0000","last_undegraded":"2024-02-12T17:08:18.573777+0000","last_fullsized":"2024-02-12T17:08:18.573777+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":26,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.587836+0000","last_change":"2024-02-12T17:08:14.640643+0000","last_active":"2024-02-12T17:08:18.587836+0000","last_peered":"2024-02-12T17:08:18.587836+0000","last_clean":"2024-02-12T17:08:18.587836+0000","last_became_active":"2024-02-12T17:08:12.925243+0000","last_became_peered":"2024-02-12T17:08:12.925243+0000","last_unstale":"2024-02-12T17:08:18.587836+0000","last_undegraded":"2024-02-12T17:08:18.587836+0000","last_fullsized":"2024-02-12T17:08:18.587836+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":415,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572909+0000","last_change":"2024-02-12T17:06:25.129256+0000","last_active":"2024-02-12T17:08:18.572909+0000","last_peered":"2024-02-12T17:08:18.572909+0000","last_clean":"2024-02-12T17:08:18.572909+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:08:18.572909+0000","last_undegraded":"2024-02-12T17:08:18.572909+0000","last_fullsized":"2024-02-12T17:08:18.572909+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:25.129190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:25.129190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"496'8612","reported_seq":8671,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:28.369203+0000","last_change":"2024-02-12T17:08:13.980483+0000","last_active":"2024-02-12T17:08:28.369203+0000","last_peered":"2024-02-12T17:08:28.369203+0000","last_clean":"2024-02-12T17:08:11.788334+0000","last_became_active":"2024-02-12T17:08:13.976648+0000","last_became_peered":"2024-02-12T17:08:13.976648+0000","last_unstale":"2024-02-12T17:08:28.369203+0000","last_undegraded":"2024-02-12T17:08:28.369203+0000","last_fullsized":"2024-02-12T17:08:28.369203+0000","mapping_epoch":492,"log_start":"468'5070","ondisk_log_start":"468'5070","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3542,"ondisk_log_size":3542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":443154432,"num_objects":6762,"num_object_clones":0,"num_object_copies":20286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6762,"num_objects_unfound":0,"num_objects_dirty":6762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":491968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[2,7,3],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,3,7","objects":6762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":524,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573221+0000","last_change":"2024-02-12T17:08:13.013398+0000","last_active":"2024-02-12T17:08:18.573221+0000","last_peered":"2024-02-12T17:08:18.573221+0000","last_clean":"2024-02-12T17:08:18.573221+0000","last_became_active":"2024-02-12T17:08:13.012162+0000","last_became_peered":"2024-02-12T17:08:13.012162+0000","last_unstale":"2024-02-12T17:08:18.573221+0000","last_undegraded":"2024-02-12T17:08:18.573221+0000","last_fullsized":"2024-02-12T17:08:18.573221+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.058086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.058086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":180,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572411+0000","last_change":"2024-02-12T17:08:12.918605+0000","last_active":"2024-02-12T17:08:18.572411+0000","last_peered":"2024-02-12T17:08:18.572411+0000","last_clean":"2024-02-12T17:08:18.572411+0000","last_became_active":"2024-02-12T17:08:12.918233+0000","last_became_peered":"2024-02-12T17:08:12.918233+0000","last_unstale":"2024-02-12T17:08:18.572411+0000","last_undegraded":"2024-02-12T17:08:18.572411+0000","last_fullsized":"2024-02-12T17:08:18.572411+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:54.311805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:05:54.311805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":467,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572319+0000","last_change":"2024-02-12T17:05:56.054036+0000","last_active":"2024-02-12T17:08:18.572319+0000","last_peered":"2024-02-12T17:08:18.572319+0000","last_clean":"2024-02-12T17:08:18.572319+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:08:18.572319+0000","last_undegraded":"2024-02-12T17:08:18.572319+0000","last_fullsized":"2024-02-12T17:08:18.572319+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:56.053971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.053971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":508,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568042+0000","last_change":"2024-02-12T17:08:12.864271+0000","last_active":"2024-02-12T17:08:18.568042+0000","last_peered":"2024-02-12T17:08:18.568042+0000","last_clean":"2024-02-12T17:08:18.568042+0000","last_became_active":"2024-02-12T17:08:12.863947+0000","last_became_peered":"2024-02-12T17:08:12.863947+0000","last_unstale":"2024-02-12T17:08:18.568042+0000","last_undegraded":"2024-02-12T17:08:18.568042+0000","last_fullsized":"2024-02-12T17:08:18.568042+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"462'33","reported_seq":255,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569135+0000","last_change":"2024-02-12T17:06:53.886435+0000","last_active":"2024-02-12T17:08:18.569135+0000","last_peered":"2024-02-12T17:08:18.569135+0000","last_clean":"2024-02-12T17:08:18.569135+0000","last_became_active":"2024-02-12T17:04:39.889039+0000","last_became_peered":"2024-02-12T17:04:39.889039+0000","last_unstale":"2024-02-12T17:08:18.569135+0000","last_undegraded":"2024-02-12T17:08:18.569135+0000","last_fullsized":"2024-02-12T17:08:18.569135+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'33","last_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.886342+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":492,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569717+0000","last_change":"2024-02-12T17:05:40.432925+0000","last_active":"2024-02-12T17:08:18.569717+0000","last_peered":"2024-02-12T17:08:18.569717+0000","last_clean":"2024-02-12T17:08:18.569717+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:08:18.569717+0000","last_undegraded":"2024-02-12T17:08:18.569717+0000","last_fullsized":"2024-02-12T17:08:18.569717+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"496'8020","reported_seq":8085,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:28.640489+0000","last_change":"2024-02-12T17:07:52.089095+0000","last_active":"2024-02-12T17:08:28.640489+0000","last_peered":"2024-02-12T17:08:28.640489+0000","last_clean":"2024-02-12T17:08:28.640489+0000","last_became_active":"2024-02-12T17:07:52.081482+0000","last_became_peered":"2024-02-12T17:07:52.081482+0000","last_unstale":"2024-02-12T17:08:28.640489+0000","last_undegraded":"2024-02-12T17:08:28.640489+0000","last_fullsized":"2024-02-12T17:08:28.640489+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":8020,"ondisk_log_size":8020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220758016,"num_objects":3368,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":243936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.583179+0000","last_change":"2024-02-12T17:08:12.961975+0000","last_active":"2024-02-12T17:08:18.583179+0000","last_peered":"2024-02-12T17:08:18.583179+0000","last_clean":"2024-02-12T17:08:18.583179+0000","last_became_active":"2024-02-12T17:08:12.959495+0000","last_became_peered":"2024-02-12T17:08:12.959495+0000","last_unstale":"2024-02-12T17:08:18.583179+0000","last_undegraded":"2024-02-12T17:08:18.583179+0000","last_fullsized":"2024-02-12T17:08:18.583179+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":250,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573099+0000","last_change":"2024-02-12T17:06:46.537733+0000","last_active":"2024-02-12T17:08:18.573099+0000","last_peered":"2024-02-12T17:08:18.573099+0000","last_clean":"2024-02-12T17:08:18.573099+0000","last_became_active":"2024-02-12T17:04:39.892122+0000","last_became_peered":"2024-02-12T17:04:39.892122+0000","last_unstale":"2024-02-12T17:08:18.573099+0000","last_undegraded":"2024-02-12T17:08:18.573099+0000","last_fullsized":"2024-02-12T17:08:18.573099+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.537630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.537630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":510,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569617+0000","last_change":"2024-02-12T17:08:13.787214+0000","last_active":"2024-02-12T17:08:18.569617+0000","last_peered":"2024-02-12T17:08:18.569617+0000","last_clean":"2024-02-12T17:08:18.569617+0000","last_became_active":"2024-02-12T17:08:13.020495+0000","last_became_peered":"2024-02-12T17:08:13.020495+0000","last_unstale":"2024-02-12T17:08:18.569617+0000","last_undegraded":"2024-02-12T17:08:18.569617+0000","last_fullsized":"2024-02-12T17:08:18.569617+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":28,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.590187+0000","last_change":"2024-02-12T17:08:14.030482+0000","last_active":"2024-02-12T17:08:18.590187+0000","last_peered":"2024-02-12T17:08:18.590187+0000","last_clean":"2024-02-12T17:08:18.590187+0000","last_became_active":"2024-02-12T17:08:12.896922+0000","last_became_peered":"2024-02-12T17:08:12.896922+0000","last_unstale":"2024-02-12T17:08:18.590187+0000","last_undegraded":"2024-02-12T17:08:18.590187+0000","last_fullsized":"2024-02-12T17:08:18.590187+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":503,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573166+0000","last_change":"2024-02-12T17:08:13.013232+0000","last_active":"2024-02-12T17:08:18.573166+0000","last_peered":"2024-02-12T17:08:18.573166+0000","last_clean":"2024-02-12T17:08:18.573166+0000","last_became_active":"2024-02-12T17:08:13.011766+0000","last_became_peered":"2024-02-12T17:08:13.011766+0000","last_unstale":"2024-02-12T17:08:18.573166+0000","last_undegraded":"2024-02-12T17:08:18.573166+0000","last_fullsized":"2024-02-12T17:08:18.573166+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"496'8846","reported_seq":8897,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:31.026581+0000","last_change":"2024-02-12T17:06:18.394587+0000","last_active":"2024-02-12T17:08:31.026581+0000","last_peered":"2024-02-12T17:08:31.026581+0000","last_clean":"2024-02-12T17:08:31.026581+0000","last_became_active":"2024-02-12T17:06:18.394337+0000","last_became_peered":"2024-02-12T17:06:18.394337+0000","last_unstale":"2024-02-12T17:08:31.026581+0000","last_undegraded":"2024-02-12T17:08:31.026581+0000","last_fullsized":"2024-02-12T17:08:31.026581+0000","mapping_epoch":451,"log_start":"469'5465","ondisk_log_start":"469'5465","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3381,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":441647128,"num_objects":6740,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8846,"num_write_kb":498689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579712+0000","last_change":"2024-02-12T17:08:12.901915+0000","last_active":"2024-02-12T17:08:18.579712+0000","last_peered":"2024-02-12T17:08:18.579712+0000","last_clean":"2024-02-12T17:08:18.579712+0000","last_became_active":"2024-02-12T17:08:12.901276+0000","last_became_peered":"2024-02-12T17:08:12.901276+0000","last_unstale":"2024-02-12T17:08:18.579712+0000","last_undegraded":"2024-02-12T17:08:18.579712+0000","last_fullsized":"2024-02-12T17:08:18.579712+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":496,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569045+0000","last_change":"2024-02-12T17:08:12.864371+0000","last_active":"2024-02-12T17:08:18.569045+0000","last_peered":"2024-02-12T17:08:18.569045+0000","last_clean":"2024-02-12T17:08:18.569045+0000","last_became_active":"2024-02-12T17:08:12.864047+0000","last_became_peered":"2024-02-12T17:08:12.864047+0000","last_unstale":"2024-02-12T17:08:18.569045+0000","last_undegraded":"2024-02-12T17:08:18.569045+0000","last_fullsized":"2024-02-12T17:08:18.569045+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.961097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.961097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":506,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572407+0000","last_change":"2024-02-12T17:08:14.718393+0000","last_active":"2024-02-12T17:08:18.572407+0000","last_peered":"2024-02-12T17:08:18.572407+0000","last_clean":"2024-02-12T17:08:18.572407+0000","last_became_active":"2024-02-12T17:08:12.963639+0000","last_became_peered":"2024-02-12T17:08:12.963639+0000","last_unstale":"2024-02-12T17:08:18.572407+0000","last_undegraded":"2024-02-12T17:08:18.572407+0000","last_fullsized":"2024-02-12T17:08:18.572407+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":292,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.605554+0000","last_change":"2024-02-12T17:07:46.526809+0000","last_active":"2024-02-12T17:08:18.605554+0000","last_peered":"2024-02-12T17:08:18.605554+0000","last_clean":"2024-02-12T17:08:18.605554+0000","last_became_active":"2024-02-12T17:05:40.448219+0000","last_became_peered":"2024-02-12T17:05:40.448219+0000","last_unstale":"2024-02-12T17:08:18.605554+0000","last_undegraded":"2024-02-12T17:08:18.605554+0000","last_fullsized":"2024-02-12T17:08:18.605554+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:46.526746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:46.526746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"496'8627","reported_seq":8680,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:28.366815+0000","last_change":"2024-02-12T17:06:18.395287+0000","last_active":"2024-02-12T17:08:28.366815+0000","last_peered":"2024-02-12T17:08:28.366815+0000","last_clean":"2024-02-12T17:08:28.366815+0000","last_became_active":"2024-02-12T17:06:18.395022+0000","last_became_peered":"2024-02-12T17:06:18.395022+0000","last_unstale":"2024-02-12T17:08:28.366815+0000","last_undegraded":"2024-02-12T17:08:28.366815+0000","last_fullsized":"2024-02-12T17:08:28.366815+0000","mapping_epoch":451,"log_start":"468'5107","ondisk_log_start":"468'5107","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3520,"ondisk_log_size":3520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":433258496,"num_objects":6611,"num_object_clones":0,"num_object_copies":19833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8627,"num_write_kb":487616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":171,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572500+0000","last_change":"2024-02-12T17:06:01.456051+0000","last_active":"2024-02-12T17:08:18.572500+0000","last_peered":"2024-02-12T17:08:18.572500+0000","last_clean":"2024-02-12T17:08:18.572500+0000","last_became_active":"2024-02-12T17:04:39.885677+0000","last_became_peered":"2024-02-12T17:04:39.885677+0000","last_unstale":"2024-02-12T17:08:18.572500+0000","last_undegraded":"2024-02-12T17:08:18.572500+0000","last_fullsized":"2024-02-12T17:08:18.572500+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:01.455969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.455969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.590029+0000","last_change":"2024-02-12T17:08:12.953062+0000","last_active":"2024-02-12T17:08:18.590029+0000","last_peered":"2024-02-12T17:08:18.590029+0000","last_clean":"2024-02-12T17:08:18.590029+0000","last_became_active":"2024-02-12T17:08:12.952169+0000","last_became_peered":"2024-02-12T17:08:12.952169+0000","last_unstale":"2024-02-12T17:08:18.590029+0000","last_undegraded":"2024-02-12T17:08:18.590029+0000","last_fullsized":"2024-02-12T17:08:18.590029+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":202,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.575430+0000","last_change":"2024-02-12T17:08:15.668029+0000","last_active":"2024-02-12T17:08:18.575430+0000","last_peered":"2024-02-12T17:08:18.575430+0000","last_clean":"2024-02-12T17:08:18.575430+0000","last_became_active":"2024-02-12T17:08:13.006137+0000","last_became_peered":"2024-02-12T17:08:13.006137+0000","last_unstale":"2024-02-12T17:08:18.575430+0000","last_undegraded":"2024-02-12T17:08:18.575430+0000","last_fullsized":"2024-02-12T17:08:18.575430+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":495,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573575+0000","last_change":"2024-02-12T17:05:40.428597+0000","last_active":"2024-02-12T17:08:18.573575+0000","last_peered":"2024-02-12T17:08:18.573575+0000","last_clean":"2024-02-12T17:08:18.573575+0000","last_became_active":"2024-02-12T17:05:40.427915+0000","last_became_peered":"2024-02-12T17:05:40.427915+0000","last_unstale":"2024-02-12T17:08:18.573575+0000","last_undegraded":"2024-02-12T17:08:18.573575+0000","last_fullsized":"2024-02-12T17:08:18.573575+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"496'8754","reported_seq":8821,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:31.026369+0000","last_change":"2024-02-12T17:08:14.647447+0000","last_active":"2024-02-12T17:08:31.026369+0000","last_peered":"2024-02-12T17:08:31.026369+0000","last_clean":"2024-02-12T17:08:11.824735+0000","last_became_active":"2024-02-12T17:08:14.000799+0000","last_became_peered":"2024-02-12T17:08:14.000799+0000","last_unstale":"2024-02-12T17:08:31.026369+0000","last_undegraded":"2024-02-12T17:08:31.026369+0000","last_fullsized":"2024-02-12T17:08:31.026369+0000","mapping_epoch":492,"log_start":"469'5348","ondisk_log_start":"469'5348","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3406,"ondisk_log_size":3406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":432537600,"num_objects":6600,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6600,"num_objects_unfound":0,"num_objects_dirty":6600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":491328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[{"shards":"2,3,5","objects":6600}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":516,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569611+0000","last_change":"2024-02-12T17:08:12.996523+0000","last_active":"2024-02-12T17:08:18.569611+0000","last_peered":"2024-02-12T17:08:18.569611+0000","last_clean":"2024-02-12T17:08:18.569611+0000","last_became_active":"2024-02-12T17:08:12.879630+0000","last_became_peered":"2024-02-12T17:08:12.879630+0000","last_unstale":"2024-02-12T17:08:18.569611+0000","last_undegraded":"2024-02-12T17:08:18.569611+0000","last_fullsized":"2024-02-12T17:08:18.569611+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:07:52.341731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.341731+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":268,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568752+0000","last_change":"2024-02-12T17:07:42.272875+0000","last_active":"2024-02-12T17:08:18.568752+0000","last_peered":"2024-02-12T17:08:18.568752+0000","last_clean":"2024-02-12T17:08:18.568752+0000","last_became_active":"2024-02-12T17:03:50.562556+0000","last_became_peered":"2024-02-12T17:03:50.562556+0000","last_unstale":"2024-02-12T17:08:18.568752+0000","last_undegraded":"2024-02-12T17:08:18.568752+0000","last_fullsized":"2024-02-12T17:08:18.568752+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:42.272807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:42.272807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":554,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.573264+0000","last_change":"2024-02-12T17:06:28.073224+0000","last_active":"2024-02-12T17:08:18.573264+0000","last_peered":"2024-02-12T17:08:18.573264+0000","last_clean":"2024-02-12T17:08:18.573264+0000","last_became_active":"2024-02-12T17:04:39.884095+0000","last_became_peered":"2024-02-12T17:04:39.884095+0000","last_unstale":"2024-02-12T17:08:18.573264+0000","last_undegraded":"2024-02-12T17:08:18.573264+0000","last_fullsized":"2024-02-12T17:08:18.573264+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'28","last_scrub_stamp":"2024-02-12T17:06:28.073131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:28.073131+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"493'21","reported_seq":272,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:22.498026+0000","last_change":"2024-02-12T17:08:13.973138+0000","last_active":"2024-02-12T17:08:22.498026+0000","last_peered":"2024-02-12T17:08:22.498026+0000","last_clean":"2024-02-12T17:08:07.490809+0000","last_became_active":"2024-02-12T17:08:13.970873+0000","last_became_peered":"2024-02-12T17:08:13.970873+0000","last_unstale":"2024-02-12T17:08:22.498026+0000","last_undegraded":"2024-02-12T17:08:22.498026+0000","last_fullsized":"2024-02-12T17:08:22.498026+0000","mapping_epoch":492,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2,5],"avail_no_missing":["3","1","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"496'8001","reported_seq":8070,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:28.249172+0000","last_change":"2024-02-12T17:08:13.956429+0000","last_active":"2024-02-12T17:08:28.249172+0000","last_peered":"2024-02-12T17:08:28.249172+0000","last_clean":"2024-02-12T17:08:11.791648+0000","last_became_active":"2024-02-12T17:08:13.954036+0000","last_became_peered":"2024-02-12T17:08:13.954036+0000","last_unstale":"2024-02-12T17:08:28.249172+0000","last_undegraded":"2024-02-12T17:08:28.249172+0000","last_fullsized":"2024-02-12T17:08:28.249172+0000","mapping_epoch":492,"log_start":"466'4200","ondisk_log_start":"466'4200","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3801,"ondisk_log_size":3801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":215089152,"num_objects":3282,"num_object_clones":0,"num_object_copies":9846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3282,"num_objects_unfound":0,"num_objects_dirty":3282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[2,7,5],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":180,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.575662+0000","last_change":"2024-02-12T17:07:20.769058+0000","last_active":"2024-02-12T17:08:18.575662+0000","last_peered":"2024-02-12T17:08:18.575662+0000","last_clean":"2024-02-12T17:08:18.575662+0000","last_became_active":"2024-02-12T17:04:33.794457+0000","last_became_peered":"2024-02-12T17:04:33.794457+0000","last_unstale":"2024-02-12T17:08:18.575662+0000","last_undegraded":"2024-02-12T17:08:18.575662+0000","last_fullsized":"2024-02-12T17:08:18.575662+0000","mapping_epoch":347,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"493'23","reported_seq":718,"reported_epoch":496,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:08:22.492939+0000","last_change":"2024-02-12T17:08:13.999597+0000","last_active":"2024-02-12T17:08:22.492939+0000","last_peered":"2024-02-12T17:08:22.492939+0000","last_clean":"2024-02-12T17:08:07.495669+0000","last_became_active":"2024-02-12T17:08:13.918461+0000","last_became_peered":"2024-02-12T17:08:13.918461+0000","last_unstale":"2024-02-12T17:08:22.492939+0000","last_undegraded":"2024-02-12T17:08:22.492939+0000","last_fullsized":"2024-02-12T17:08:22.492939+0000","mapping_epoch":492,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'21","last_scrub_stamp":"2024-02-12T17:06:32.977637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:06:32.977637+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":141,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.567807+0000","last_change":"2024-02-12T17:08:12.864376+0000","last_active":"2024-02-12T17:08:18.567807+0000","last_peered":"2024-02-12T17:08:18.567807+0000","last_clean":"2024-02-12T17:08:18.567807+0000","last_became_active":"2024-02-12T17:08:12.864037+0000","last_became_peered":"2024-02-12T17:08:12.864037+0000","last_unstale":"2024-02-12T17:08:18.567807+0000","last_undegraded":"2024-02-12T17:08:18.567807+0000","last_fullsized":"2024-02-12T17:08:18.567807+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:24.976048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:06:24.976048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":26,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.590113+0000","last_change":"2024-02-12T17:08:14.975705+0000","last_active":"2024-02-12T17:08:18.590113+0000","last_peered":"2024-02-12T17:08:18.590113+0000","last_clean":"2024-02-12T17:08:18.590113+0000","last_became_active":"2024-02-12T17:08:12.952663+0000","last_became_peered":"2024-02-12T17:08:12.952663+0000","last_unstale":"2024-02-12T17:08:18.590113+0000","last_undegraded":"2024-02-12T17:08:18.590113+0000","last_fullsized":"2024-02-12T17:08:18.590113+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:05:55.344999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.344999+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"496'8292","reported_seq":8364,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:28.749079+0000","last_change":"2024-02-12T17:08:13.930607+0000","last_active":"2024-02-12T17:08:28.749079+0000","last_peered":"2024-02-12T17:08:28.749079+0000","last_clean":"2024-02-12T17:08:11.814847+0000","last_became_active":"2024-02-12T17:08:13.926132+0000","last_became_peered":"2024-02-12T17:08:13.926132+0000","last_unstale":"2024-02-12T17:08:28.749079+0000","last_undegraded":"2024-02-12T17:08:28.749079+0000","last_fullsized":"2024-02-12T17:08:28.749079+0000","mapping_epoch":492,"log_start":"466'4500","ondisk_log_start":"466'4500","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3792,"ondisk_log_size":3792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":223510528,"num_objects":3410,"num_object_clones":0,"num_object_copies":10230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3410,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":250528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":3410}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":515,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572935+0000","last_change":"2024-02-12T17:08:13.011711+0000","last_active":"2024-02-12T17:08:18.572935+0000","last_peered":"2024-02-12T17:08:18.572935+0000","last_clean":"2024-02-12T17:08:18.572935+0000","last_became_active":"2024-02-12T17:08:12.972066+0000","last_became_peered":"2024-02-12T17:08:12.972066+0000","last_unstale":"2024-02-12T17:08:18.572935+0000","last_undegraded":"2024-02-12T17:08:18.572935+0000","last_fullsized":"2024-02-12T17:08:18.572935+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"435'21","reported_seq":770,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:22.492931+0000","last_change":"2024-02-12T17:07:41.785111+0000","last_active":"2024-02-12T17:08:22.492931+0000","last_peered":"2024-02-12T17:08:22.492931+0000","last_clean":"2024-02-12T17:08:22.492931+0000","last_became_active":"2024-02-12T17:05:43.540504+0000","last_became_peered":"2024-02-12T17:05:43.540504+0000","last_unstale":"2024-02-12T17:08:22.492931+0000","last_undegraded":"2024-02-12T17:08:22.492931+0000","last_fullsized":"2024-02-12T17:08:22.492931+0000","mapping_epoch":434,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'21","last_scrub_stamp":"2024-02-12T17:07:41.785018+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.785018+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579618+0000","last_change":"2024-02-12T17:08:12.966155+0000","last_active":"2024-02-12T17:08:18.579618+0000","last_peered":"2024-02-12T17:08:18.579618+0000","last_clean":"2024-02-12T17:08:18.579618+0000","last_became_active":"2024-02-12T17:08:12.965556+0000","last_became_peered":"2024-02-12T17:08:12.965556+0000","last_unstale":"2024-02-12T17:08:18.579618+0000","last_undegraded":"2024-02-12T17:08:18.579618+0000","last_fullsized":"2024-02-12T17:08:18.579618+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":30,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.587754+0000","last_change":"2024-02-12T17:08:17.847813+0000","last_active":"2024-02-12T17:08:18.587754+0000","last_peered":"2024-02-12T17:08:18.587754+0000","last_clean":"2024-02-12T17:08:18.587754+0000","last_became_active":"2024-02-12T17:08:12.964494+0000","last_became_peered":"2024-02-12T17:08:12.964494+0000","last_unstale":"2024-02-12T17:08:18.587754+0000","last_undegraded":"2024-02-12T17:08:18.587754+0000","last_fullsized":"2024-02-12T17:08:18.587754+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"496'8089","reported_seq":8160,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:31.001665+0000","last_change":"2024-02-12T17:08:13.969943+0000","last_active":"2024-02-12T17:08:31.001665+0000","last_peered":"2024-02-12T17:08:31.001665+0000","last_clean":"2024-02-12T17:08:11.820762+0000","last_became_active":"2024-02-12T17:08:13.964915+0000","last_became_peered":"2024-02-12T17:08:13.964915+0000","last_unstale":"2024-02-12T17:08:31.001665+0000","last_undegraded":"2024-02-12T17:08:31.001665+0000","last_fullsized":"2024-02-12T17:08:31.001665+0000","mapping_epoch":492,"log_start":"466'4100","ondisk_log_start":"466'4100","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3989,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":217088000,"num_objects":3312,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3312,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4318,"num_write_kb":244192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[3,7,5],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":3312}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.590270+0000","last_change":"2024-02-12T17:08:12.908568+0000","last_active":"2024-02-12T17:08:18.590270+0000","last_peered":"2024-02-12T17:08:18.590270+0000","last_clean":"2024-02-12T17:08:18.590270+0000","last_became_active":"2024-02-12T17:08:12.908230+0000","last_became_peered":"2024-02-12T17:08:12.908230+0000","last_unstale":"2024-02-12T17:08:18.590270+0000","last_undegraded":"2024-02-12T17:08:18.590270+0000","last_fullsized":"2024-02-12T17:08:18.590270+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.077971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":334,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:22.492901+0000","last_change":"2024-02-12T17:07:41.239456+0000","last_active":"2024-02-12T17:08:22.492901+0000","last_peered":"2024-02-12T17:08:22.492901+0000","last_clean":"2024-02-12T17:08:22.492901+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:08:22.492901+0000","last_undegraded":"2024-02-12T17:08:22.492901+0000","last_fullsized":"2024-02-12T17:08:22.492901+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:07:41.239350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.239350+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579374+0000","last_change":"2024-02-12T17:08:12.961477+0000","last_active":"2024-02-12T17:08:18.579374+0000","last_peered":"2024-02-12T17:08:18.579374+0000","last_clean":"2024-02-12T17:08:18.579374+0000","last_became_active":"2024-02-12T17:08:12.960827+0000","last_became_peered":"2024-02-12T17:08:12.960827+0000","last_unstale":"2024-02-12T17:08:18.579374+0000","last_undegraded":"2024-02-12T17:08:18.579374+0000","last_fullsized":"2024-02-12T17:08:18.579374+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":211,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.575756+0000","last_change":"2024-02-12T17:08:13.694682+0000","last_active":"2024-02-12T17:08:18.575756+0000","last_peered":"2024-02-12T17:08:18.575756+0000","last_clean":"2024-02-12T17:08:18.575756+0000","last_became_active":"2024-02-12T17:08:12.918281+0000","last_became_peered":"2024-02-12T17:08:12.918281+0000","last_unstale":"2024-02-12T17:08:18.575756+0000","last_undegraded":"2024-02-12T17:08:18.575756+0000","last_fullsized":"2024-02-12T17:08:18.575756+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"496'8184","reported_seq":8247,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:28.308575+0000","last_change":"2024-02-12T17:07:52.060932+0000","last_active":"2024-02-12T17:08:28.308575+0000","last_peered":"2024-02-12T17:08:28.308575+0000","last_clean":"2024-02-12T17:08:28.308575+0000","last_became_active":"2024-02-12T17:07:52.058701+0000","last_became_peered":"2024-02-12T17:07:52.058701+0000","last_unstale":"2024-02-12T17:08:28.308575+0000","last_undegraded":"2024-02-12T17:08:28.308575+0000","last_fullsized":"2024-02-12T17:08:28.308575+0000","mapping_epoch":451,"log_start":"466'4400","ondisk_log_start":"466'4400","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3784,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":223281152,"num_objects":3407,"num_object_clones":0,"num_object_copies":10221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4349,"num_write_kb":248192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":198,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.576244+0000","last_change":"2024-02-12T17:08:12.928805+0000","last_active":"2024-02-12T17:08:18.576244+0000","last_peered":"2024-02-12T17:08:18.576244+0000","last_clean":"2024-02-12T17:08:18.576244+0000","last_became_active":"2024-02-12T17:08:12.928455+0000","last_became_peered":"2024-02-12T17:08:12.928455+0000","last_unstale":"2024-02-12T17:08:18.576244+0000","last_undegraded":"2024-02-12T17:08:18.576244+0000","last_fullsized":"2024-02-12T17:08:18.576244+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.583423+0000","last_change":"2024-02-12T17:08:12.910622+0000","last_active":"2024-02-12T17:08:18.583423+0000","last_peered":"2024-02-12T17:08:18.583423+0000","last_clean":"2024-02-12T17:08:18.583423+0000","last_became_active":"2024-02-12T17:08:12.902648+0000","last_became_peered":"2024-02-12T17:08:12.902648+0000","last_unstale":"2024-02-12T17:08:18.583423+0000","last_undegraded":"2024-02-12T17:08:18.583423+0000","last_fullsized":"2024-02-12T17:08:18.583423+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.567340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.567340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":478,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.569307+0000","last_change":"2024-02-12T17:06:51.877944+0000","last_active":"2024-02-12T17:08:18.569307+0000","last_peered":"2024-02-12T17:08:18.569307+0000","last_clean":"2024-02-12T17:08:18.569307+0000","last_became_active":"2024-02-12T17:04:39.888466+0000","last_became_peered":"2024-02-12T17:04:39.888466+0000","last_unstale":"2024-02-12T17:08:18.569307+0000","last_undegraded":"2024-02-12T17:08:18.569307+0000","last_fullsized":"2024-02-12T17:08:18.569307+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.877830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.877830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":32,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.580195+0000","last_change":"2024-02-12T17:08:15.420729+0000","last_active":"2024-02-12T17:08:18.580195+0000","last_peered":"2024-02-12T17:08:18.580195+0000","last_clean":"2024-02-12T17:08:18.580195+0000","last_became_active":"2024-02-12T17:08:12.955942+0000","last_became_peered":"2024-02-12T17:08:12.955942+0000","last_unstale":"2024-02-12T17:08:18.580195+0000","last_undegraded":"2024-02-12T17:08:18.580195+0000","last_fullsized":"2024-02-12T17:08:18.580195+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:07:37.774472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.774472+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"496'8168","reported_seq":8239,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:30.867593+0000","last_change":"2024-02-12T17:08:14.004009+0000","last_active":"2024-02-12T17:08:30.867593+0000","last_peered":"2024-02-12T17:08:30.867593+0000","last_clean":"2024-02-12T17:08:11.807038+0000","last_became_active":"2024-02-12T17:08:14.000844+0000","last_became_peered":"2024-02-12T17:08:14.000844+0000","last_unstale":"2024-02-12T17:08:30.867593+0000","last_undegraded":"2024-02-12T17:08:30.867593+0000","last_fullsized":"2024-02-12T17:08:30.867593+0000","mapping_epoch":492,"log_start":"466'4200","ondisk_log_start":"466'4200","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3968,"ondisk_log_size":3968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":216236032,"num_objects":3299,"num_object_clones":0,"num_object_copies":9897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3299,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4373,"num_write_kb":245536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":3299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":514,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.567890+0000","last_change":"2024-02-12T17:05:57.890025+0000","last_active":"2024-02-12T17:08:18.567890+0000","last_peered":"2024-02-12T17:08:18.567890+0000","last_clean":"2024-02-12T17:08:18.567890+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:08:18.567890+0000","last_undegraded":"2024-02-12T17:08:18.567890+0000","last_fullsized":"2024-02-12T17:08:18.567890+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.889955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.889955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":411,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.605702+0000","last_change":"2024-02-12T17:07:20.629548+0000","last_active":"2024-02-12T17:08:18.605702+0000","last_peered":"2024-02-12T17:08:18.605702+0000","last_clean":"2024-02-12T17:08:18.605702+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:08:18.605702+0000","last_undegraded":"2024-02-12T17:08:18.605702+0000","last_fullsized":"2024-02-12T17:08:18.605702+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.629467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.629467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":232,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572180+0000","last_change":"2024-02-12T17:08:12.953085+0000","last_active":"2024-02-12T17:08:18.572180+0000","last_peered":"2024-02-12T17:08:18.572180+0000","last_clean":"2024-02-12T17:08:18.572180+0000","last_became_active":"2024-02-12T17:08:12.952757+0000","last_became_peered":"2024-02-12T17:08:12.952757+0000","last_unstale":"2024-02-12T17:08:18.572180+0000","last_undegraded":"2024-02-12T17:08:18.572180+0000","last_fullsized":"2024-02-12T17:08:18.572180+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":34,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.579120+0000","last_change":"2024-02-12T17:08:13.087482+0000","last_active":"2024-02-12T17:08:18.579120+0000","last_peered":"2024-02-12T17:08:18.579120+0000","last_clean":"2024-02-12T17:08:18.579120+0000","last_became_active":"2024-02-12T17:08:12.908226+0000","last_became_peered":"2024-02-12T17:08:12.908226+0000","last_unstale":"2024-02-12T17:08:18.579120+0000","last_undegraded":"2024-02-12T17:08:18.579120+0000","last_fullsized":"2024-02-12T17:08:18.579120+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"496'8108","reported_seq":8179,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-02-12T17:08:28.738985+0000","last_change":"2024-02-12T17:07:53.893435+0000","last_active":"2024-02-12T17:08:28.738985+0000","last_peered":"2024-02-12T17:08:28.738985+0000","last_clean":"2024-02-12T17:08:28.738985+0000","last_became_active":"2024-02-12T17:07:52.071379+0000","last_became_peered":"2024-02-12T17:07:52.071379+0000","last_unstale":"2024-02-12T17:08:28.738985+0000","last_undegraded":"2024-02-12T17:08:28.738985+0000","last_fullsized":"2024-02-12T17:08:28.738985+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'7593","last_scrub_stamp":"2024-02-12T17:07:53.893352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:07:53.893352+0000","log_size":8108,"ondisk_log_size":8108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":198,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.568877+0000","last_change":"2024-02-12T17:06:43.014800+0000","last_active":"2024-02-12T17:08:18.568877+0000","last_peered":"2024-02-12T17:08:18.568877+0000","last_clean":"2024-02-12T17:08:18.568877+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:08:18.568877+0000","last_undegraded":"2024-02-12T17:08:18.568877+0000","last_fullsized":"2024-02-12T17:08:18.568877+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:06:43.014737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":412,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572139+0000","last_change":"2024-02-12T17:08:12.919537+0000","last_active":"2024-02-12T17:08:18.572139+0000","last_peered":"2024-02-12T17:08:18.572139+0000","last_clean":"2024-02-12T17:08:18.572139+0000","last_became_active":"2024-02-12T17:08:12.919194+0000","last_became_peered":"2024-02-12T17:08:12.919194+0000","last_unstale":"2024-02-12T17:08:18.572139+0000","last_undegraded":"2024-02-12T17:08:18.572139+0000","last_fullsized":"2024-02-12T17:08:18.572139+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":18,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.580099+0000","last_change":"2024-02-12T17:08:17.757666+0000","last_active":"2024-02-12T17:08:18.580099+0000","last_peered":"2024-02-12T17:08:18.580099+0000","last_clean":"2024-02-12T17:08:18.580099+0000","last_became_active":"2024-02-12T17:08:13.013035+0000","last_became_peered":"2024-02-12T17:08:13.013035+0000","last_unstale":"2024-02-12T17:08:18.580099+0000","last_undegraded":"2024-02-12T17:08:18.580099+0000","last_fullsized":"2024-02-12T17:08:18.580099+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":24,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.584498+0000","last_change":"2024-02-12T17:08:15.679265+0000","last_active":"2024-02-12T17:08:18.584498+0000","last_peered":"2024-02-12T17:08:18.584498+0000","last_clean":"2024-02-12T17:08:18.584498+0000","last_became_active":"2024-02-12T17:08:12.961078+0000","last_became_peered":"2024-02-12T17:08:12.961078+0000","last_unstale":"2024-02-12T17:08:18.584498+0000","last_undegraded":"2024-02-12T17:08:18.584498+0000","last_fullsized":"2024-02-12T17:08:18.584498+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:38.887701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.887701+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"496'8235","reported_seq":8310,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:28.310383+0000","last_change":"2024-02-12T17:08:13.979870+0000","last_active":"2024-02-12T17:08:28.310383+0000","last_peered":"2024-02-12T17:08:28.310383+0000","last_clean":"2024-02-12T17:08:11.723508+0000","last_became_active":"2024-02-12T17:08:13.976743+0000","last_became_peered":"2024-02-12T17:08:13.976743+0000","last_unstale":"2024-02-12T17:08:28.310383+0000","last_undegraded":"2024-02-12T17:08:28.310383+0000","last_fullsized":"2024-02-12T17:08:28.310383+0000","mapping_epoch":492,"log_start":"466'4400","ondisk_log_start":"466'4400","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3835,"ondisk_log_size":3835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":224329728,"num_objects":3423,"num_object_clones":0,"num_object_copies":10269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3423,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4383,"num_write_kb":249792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,2,7],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":3423}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":488,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.571665+0000","last_change":"2024-02-12T17:06:47.945378+0000","last_active":"2024-02-12T17:08:18.571665+0000","last_peered":"2024-02-12T17:08:18.571665+0000","last_clean":"2024-02-12T17:08:18.571665+0000","last_became_active":"2024-02-12T17:04:41.899598+0000","last_became_peered":"2024-02-12T17:04:41.899598+0000","last_unstale":"2024-02-12T17:08:18.571665+0000","last_undegraded":"2024-02-12T17:08:18.571665+0000","last_fullsized":"2024-02-12T17:08:18.571665+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:47.945301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:47.945301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":162,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.572401+0000","last_change":"2024-02-12T17:06:53.648932+0000","last_active":"2024-02-12T17:08:18.572401+0000","last_peered":"2024-02-12T17:08:18.572401+0000","last_clean":"2024-02-12T17:08:18.572401+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:08:18.572401+0000","last_undegraded":"2024-02-12T17:08:18.572401+0000","last_fullsized":"2024-02-12T17:08:18.572401+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":570,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-02-12T17:08:18.571665+0000","last_change":"2024-02-12T17:07:47.773053+0000","last_active":"2024-02-12T17:08:18.571665+0000","last_peered":"2024-02-12T17:08:18.571665+0000","last_clean":"2024-02-12T17:08:18.571665+0000","last_became_active":"2024-02-12T17:04:39.882260+0000","last_became_peered":"2024-02-12T17:04:39.882260+0000","last_unstale":"2024-02-12T17:08:18.571665+0000","last_undegraded":"2024-02-12T17:08:18.571665+0000","last_fullsized":"2024-02-12T17:08:18.571665+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:07:47.772969+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.772969+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"493'20","reported_seq":796,"reported_epoch":496,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:08:22.492936+0000","last_change":"2024-02-12T17:08:13.897563+0000","last_active":"2024-02-12T17:08:22.492936+0000","last_peered":"2024-02-12T17:08:22.492936+0000","last_clean":"2024-02-12T17:08:07.495835+0000","last_became_active":"2024-02-12T17:08:13.896055+0000","last_became_peered":"2024-02-12T17:08:13.896055+0000","last_unstale":"2024-02-12T17:08:22.492936+0000","last_undegraded":"2024-02-12T17:08:22.492936+0000","last_fullsized":"2024-02-12T17:08:22.492936+0000","mapping_epoch":492,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-02-12T17:06:35.066866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.066866+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,3,2],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":7042498584,"num_objects":107461,"num_object_clones":0,"num_object_copies":322383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":445,"num_objects_misplaced":71273,"num_objects_unfound":0,"num_objects_dirty":107461,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135103,"num_write_kb":7638337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12965,"num_bytes_recovered":849608704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22146330624,"data_stored":22146318408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113181,"ondisk_log_size":113181,"up":78,"acting":76,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":431,"num_read_kb":426,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1095,"num_read_kb":852,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":11860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":736,"ondisk_log_size":736,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":421,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234213,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5521800,"kb_used_data":4626240,"kb_used_omap":0,"kb_used_meta":895552,"kb_avail":88223352,"statfs":{"total":95995035648,"available":90340712448,"internally_reserved":0,"allocated":4737269760,"data_stored":4736245052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917045248},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":2.3119999999999998,"15min":2.0230000000000001},"min":{"1min":0.27100000000000002,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.8879999999999999,"5min":12.375,"15min":36.439},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.7989999999999999,"15min":2.085},"min":{"1min":0.27300000000000002,"5min":0.22,"15min":0.19600000000000001},"max":{"1min":3.109,"5min":8.5039999999999996,"15min":36.490000000000002},"last":2.3889999999999998}]},{"osd":1,"last update":"Mon Feb 12 17:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":2.4060000000000001,"15min":1.819},"min":{"1min":0.29099999999999998,"5min":0.27300000000000002,"15min":0.20899999999999999},"max":{"1min":1.6180000000000001,"5min":13.224,"15min":13.224},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.7609999999999999,"15min":1.7230000000000001},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.22700000000000001},"max":{"1min":2.4780000000000002,"5min":10.194000000000001,"15min":17.324999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":2.222,"15min":1.6439999999999999},"min":{"1min":0.26200000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":4.0599999999999996,"5min":12.4,"15min":12.4},"last":1.075},{"interface":"front","average":{"1min":1.046,"5min":2.6920000000000002,"15min":2.0590000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.23000000000000001},"max":{"1min":3.085,"5min":13.127000000000001,"15min":20.338000000000001},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":2.3929999999999998,"15min":1.962},"min":{"1min":0.28100000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.2569999999999997,"5min":13.608000000000001,"15min":20.433},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.1890000000000001,"5min":2.8780000000000001,"15min":2.327},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.25600000000000001},"max":{"1min":3.2330000000000001,"5min":31.152000000000001,"15min":31.152000000000001},"last":0.92300000000000004}]},{"osd":4,"last update":"Mon Feb 12 17:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":2.7250000000000001,"15min":2.0310000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":3.9279999999999999,"5min":19.030000000000001,"15min":50.406999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":2.9969999999999999,"15min":2.4079999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":2.0619999999999998,"5min":16.867000000000001,"15min":50.168999999999997},"last":0.39400000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":3.0299999999999998,"15min":2.0939999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.9689999999999999,"5min":19.702000000000002,"15min":19.702000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.62,"5min":2.7200000000000002,"15min":2.1850000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.6519999999999999,"5min":17.585999999999999,"15min":18.651},"last":0.376}]},{"osd":6,"last update":"Mon Feb 12 17:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":2.048,"15min":1.8120000000000001},"min":{"1min":0.28999999999999998,"5min":0.27800000000000002,"15min":0.27400000000000002},"max":{"1min":2.4500000000000002,"5min":10.522,"15min":17.234999999999999},"last":0.496},{"interface":"front","average":{"1min":0.92600000000000005,"5min":2.4889999999999999,"15min":1.825},"min":{"1min":0.29799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.1709999999999998,"5min":13.676,"15min":13.676},"last":0.55000000000000004}]}]},{"osd":6,"up_from":0,"seq":193273528554,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575107,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357936,"kb_used_data":56616,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93387216,"statfs":{"total":95995035648,"available":95628509184,"internally_reserved":0,"allocated":57974784,"data_stored":56911087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:07:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.1639999999999999,"15min":1.522},"min":{"1min":0.23699999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.125,"5min":7.8899999999999997,"15min":25.585000000000001},"last":6.093},{"interface":"front","average":{"1min":1.05,"5min":1.046,"15min":1.6140000000000001},"min":{"1min":0.248,"5min":0.221,"15min":0.189},"max":{"1min":4.9500000000000002,"5min":4.9500000000000002,"15min":26.798999999999999},"last":4.0419999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:08:16 2024","interfaces":[{"interface":"back","average":{"1min":2.996,"5min":1.7529999999999999,"15min":1.78},"min":{"1min":0.26400000000000001,"5min":0.224,"15min":0.191},"max":{"1min":12.621,"5min":12.621,"15min":25.413},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.871,"5min":1.4950000000000001,"15min":1.839},"min":{"1min":0.311,"5min":0.26000000000000001,"15min":0.19},"max":{"1min":5.2050000000000001,"5min":9.2910000000000004,"15min":26.693999999999999},"last":3.3090000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:07:35 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.4770000000000001,"15min":1.708},"min":{"1min":0.33100000000000002,"5min":0.29499999999999998,"15min":0.23100000000000001},"max":{"1min":5.1040000000000001,"5min":11.686,"15min":26.649999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.5900000000000001,"5min":0.97099999999999997,"15min":1.4059999999999999},"min":{"1min":0.504,"5min":0.249,"15min":0.23999999999999999},"max":{"1min":6.9409999999999998,"5min":6.9409999999999998,"15min":17.821000000000002},"last":12.053000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:07:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":2.2130000000000001,"15min":1.865},"min":{"1min":0.36799999999999999,"5min":0.34399999999999997,"15min":0.26100000000000001},"max":{"1min":7.0270000000000001,"5min":12.91,"15min":12.91},"last":6.1840000000000002},{"interface":"front","average":{"1min":0.998,"5min":2.1509999999999998,"15min":2.0179999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.25900000000000001},"max":{"1min":3.488,"5min":16.074000000000002,"15min":26.748999999999999},"last":0.433}]},{"osd":5,"last update":"Mon Feb 12 17:08:24 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":2.1720000000000002,"15min":2.121},"min":{"1min":0.35599999999999998,"5min":0.33600000000000002,"15min":0.28299999999999997},"max":{"1min":6.9889999999999999,"5min":22.154,"15min":22.154},"last":6.133},{"interface":"front","average":{"1min":0.81799999999999995,"5min":1.956,"15min":2.0939999999999999},"min":{"1min":0.34300000000000003,"5min":0.31900000000000001,"15min":0.27300000000000002},"max":{"1min":5.1180000000000003,"5min":21.803000000000001,"15min":26.722999999999999},"last":0.38400000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:07:43 2024","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.804,"15min":1.925},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.29999999999999999},"max":{"1min":7.0919999999999996,"5min":9.8569999999999993,"15min":13.834},"last":6.1609999999999996},{"interface":"front","average":{"1min":1.363,"5min":1.901,"15min":1.9930000000000001},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.307},"max":{"1min":5.0970000000000004,"5min":16.172999999999998,"15min":26.774999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:08:00 2024","interfaces":[{"interface":"back","average":{"1min":1.8899999999999999,"5min":2.0990000000000002,"15min":2.0430000000000001},"min":{"1min":0.40300000000000002,"5min":0.33100000000000002,"15min":0.28799999999999998},"max":{"1min":8.7560000000000002,"5min":8.7560000000000002,"15min":9.5239999999999991},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2290000000000001,"15min":2.2050000000000001},"min":{"1min":0.46500000000000002,"5min":0.41999999999999998,"15min":0.31},"max":{"1min":7.0629999999999997,"5min":13.965999999999999,"15min":20.548999999999999},"last":9.3510000000000009}]}]},{"osd":0,"up_from":8,"seq":34359738631,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1963780,"kb_used_data":1063420,"kb_used_omap":0,"kb_used_meta":900288,"kb_avail":91781372,"statfs":{"total":95995035648,"available":93984124928,"internally_reserved":0,"allocated":1088942080,"data_stored":1087985421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":921894912},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0940000000000001,"15min":1.4450000000000001},"min":{"1min":0.22,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":3.4039999999999999,"5min":6.2939999999999996,"15min":23.510999999999999},"last":2.2949999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.5,"15min":1.504},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.193},"max":{"1min":4.3780000000000001,"5min":16.16,"15min":16.16},"last":10.741}]},{"osd":2,"last update":"Mon Feb 12 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":1.341,"15min":1.6379999999999999},"min":{"1min":0.24399999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":10.404,"5min":10.404,"15min":16.521000000000001},"last":2.657},{"interface":"front","average":{"1min":2.7280000000000002,"5min":1.851,"15min":1.72},"min":{"1min":0.311,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":10.58,"5min":10.58,"15min":10.58},"last":10.682}]},{"osd":3,"last update":"Mon Feb 12 17:07:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5229999999999999,"15min":1.821},"min":{"1min":0.222,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":4.2889999999999997,"5min":9.8819999999999997,"15min":21.100000000000001},"last":4.9059999999999997},{"interface":"front","average":{"1min":2.1800000000000002,"5min":1.7589999999999999,"15min":1.8879999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.221},"max":{"1min":4.718,"5min":16.759,"15min":16.759},"last":7.016}]},{"osd":4,"last update":"Mon Feb 12 17:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":1.448,"15min":1.665},"min":{"1min":0.30199999999999999,"5min":0.28599999999999998,"15min":0.23599999999999999},"max":{"1min":2.1890000000000001,"5min":7.8639999999999999,"15min":16.888999999999999},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.73,"15min":1.724},"min":{"1min":0.36899999999999999,"5min":0.28799999999999998,"15min":0.28399999999999997},"max":{"1min":4.444,"5min":9.1020000000000003,"15min":16.899999999999999},"last":10.638999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:08:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.869,"15min":2.2749999999999999},"min":{"1min":0.318,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":5.681,"5min":10.146000000000001,"15min":50.404000000000003},"last":1.319},{"interface":"front","average":{"1min":1.419,"5min":2.2120000000000002,"15min":2.2679999999999998},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.28299999999999997},"max":{"1min":10.35,"5min":19.751999999999999,"15min":49.979999999999997},"last":2.2109999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:07:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.669,"15min":1.7829999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29999999999999999},"max":{"1min":4.4169999999999998,"5min":8.9570000000000007,"15min":17.907},"last":10.595000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.7470000000000001,"15min":2.0339999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.27900000000000003},"max":{"1min":5.7089999999999996,"5min":7.7510000000000003,"15min":21.062000000000001},"last":1.645}]},{"osd":7,"last update":"Mon Feb 12 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":2.383,"15min":2.069},"min":{"1min":0.374,"5min":0.36899999999999999,"15min":0.32400000000000001},"max":{"1min":10.381,"5min":19.373000000000001,"15min":19.373000000000001},"last":2.2469999999999999},{"interface":"front","average":{"1min":2.032,"5min":2.1240000000000001,"15min":2.1040000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":10.505000000000001,"5min":10.505000000000001,"15min":34.722999999999999},"last":10.714}]}]},{"osd":2,"up_from":18,"seq":77309411580,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7432640,"kb_used_data":6543288,"kb_used_omap":0,"kb_used_meta":888576,"kb_avail":86312512,"statfs":{"total":95995035648,"available":88384012288,"internally_reserved":0,"allocated":6700326912,"data_stored":6699310585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":909901824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:08:16 2024","interfaces":[{"interface":"back","average":{"1min":3.7959999999999998,"5min":1.6100000000000001,"15min":1.9350000000000001},"min":{"1min":0.223,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":21.445,"5min":21.445,"15min":36.100999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":2.794,"5min":1.883,"15min":1.9410000000000001},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.158},"max":{"1min":19.879000000000001,"5min":20.359000000000002,"15min":36.299999999999997},"last":1.544}]},{"osd":1,"last update":"Mon Feb 12 17:08:16 2024","interfaces":[{"interface":"back","average":{"1min":2.706,"5min":1.3979999999999999,"15min":1.7709999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.191},"max":{"1min":20.763999999999999,"5min":20.763999999999999,"15min":20.763999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":3.2669999999999999,"5min":1.528,"15min":1.8779999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":21.361999999999998,"5min":21.361999999999998,"15min":21.361999999999998},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:07:47 2024","interfaces":[{"interface":"back","average":{"1min":2.7999999999999998,"5min":1.5069999999999999,"15min":2.0169999999999999},"min":{"1min":0.433,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":7.96,"5min":8.8209999999999997,"15min":51.579999999999998},"last":2.6259999999999999},{"interface":"front","average":{"1min":2.4140000000000001,"5min":1.4610000000000001,"15min":1.724},"min":{"1min":0.35699999999999998,"5min":0.23699999999999999,"15min":0.23200000000000001},"max":{"1min":5.1920000000000002,"5min":5.4180000000000001,"15min":48.988999999999997},"last":2.3530000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:08:04 2024","interfaces":[{"interface":"back","average":{"1min":2.7429999999999999,"5min":1.9199999999999999,"15min":2.0110000000000001},"min":{"1min":0.34300000000000003,"5min":0.25600000000000001,"15min":0.253},"max":{"1min":21.515000000000001,"5min":21.515000000000001,"15min":21.515000000000001},"last":0.378},{"interface":"front","average":{"1min":2.3010000000000002,"5min":2.0939999999999999,"15min":2.0590000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.26800000000000002},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":1.6859999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:08:16 2024","interfaces":[{"interface":"back","average":{"1min":2.919,"5min":2.3380000000000001,"15min":2.1259999999999999},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.27300000000000002},"max":{"1min":21.542999999999999,"5min":32.57,"15min":32.57},"last":0.40500000000000003},{"interface":"front","average":{"1min":2.1389999999999998,"5min":2.5699999999999998,"15min":2.371},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":20.937000000000001,"5min":31.905999999999999,"15min":31.905999999999999},"last":0.81699999999999995}]},{"osd":6,"last update":"Mon Feb 12 17:07:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.597,"15min":2.1120000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.30199999999999999},"max":{"1min":4.3170000000000002,"5min":8.9969999999999999,"15min":17.782},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.6000000000000001,"15min":1.8600000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.29799999999999999},"max":{"1min":3.9870000000000001,"5min":10.6,"15min":13.446999999999999},"last":1.6559999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:08:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3879999999999999,"5min":3.569,"15min":2.5800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":20.221,"5min":45.468000000000004,"15min":45.468000000000004},"last":1.611},{"interface":"front","average":{"1min":2.169,"5min":3.2050000000000001,"15min":2.6070000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":20.995999999999999,"5min":46.656999999999996,"15min":46.656999999999996},"last":0.84599999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149943,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5661216,"kb_used_data":5067224,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":88083936,"statfs":{"total":95995035648,"available":90197950464,"internally_reserved":0,"allocated":5188837376,"data_stored":5187789410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:07:36 2024","interfaces":[{"interface":"back","average":{"1min":2.589,"5min":1.873,"15min":1.8260000000000001},"min":{"1min":0.19700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.5340000000000007,"5min":58.451999999999998,"15min":58.451999999999998},"last":3.242},{"interface":"front","average":{"1min":4.2690000000000001,"5min":2.5790000000000002,"15min":2.036},"min":{"1min":0.375,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":12.734,"5min":57.497999999999998,"15min":57.497999999999998},"last":1.468}]},{"osd":1,"last update":"Mon Feb 12 17:07:36 2024","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":1.649,"15min":1.6599999999999999},"min":{"1min":0.27400000000000002,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":7.6379999999999999,"5min":11.537000000000001,"15min":13.94},"last":0.625},{"interface":"front","average":{"1min":1.113,"5min":0.996,"15min":1.367},"min":{"1min":0.223,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":5.5339999999999998,"5min":10.278,"15min":14.962},"last":0.60199999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:07:36 2024","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.2649999999999999,"15min":1.71},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":6.3079999999999998,"5min":10.302,"15min":49.112000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":2.8900000000000001,"5min":1.369,"15min":1.8759999999999999},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":11.532999999999999,"5min":11.532999999999999,"15min":51.997999999999998},"last":3.3889999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:07:57 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.738,"15min":1.9870000000000001},"min":{"1min":0.317,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":5.0720000000000001,"5min":6.7569999999999997,"15min":19.010000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.431,"15min":1.7509999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.1699999999999999,"5min":10.250999999999999,"15min":19.033999999999999},"last":0.36199999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:08:22 2024","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":1.954,"15min":2.1299999999999999},"min":{"1min":0.30299999999999999,"5min":0.253,"15min":0.253},"max":{"1min":7.548,"5min":14.117000000000001,"15min":19.882999999999999},"last":3.2829999999999999},{"interface":"front","average":{"1min":2.0129999999999999,"5min":2.3650000000000002,"15min":2.2919999999999998},"min":{"1min":0.42199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":7.1509999999999998,"5min":12.502000000000001,"15min":15.595000000000001},"last":0.41499999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:07:36 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.5249999999999999,"15min":1.591},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":5.2770000000000001,"5min":10.212999999999999,"15min":12.856999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.8500000000000001,"5min":1.4219999999999999,"15min":1.8},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.27500000000000002},"max":{"1min":7.3929999999999998,"5min":8.5039999999999996,"15min":19.925999999999998},"last":3.3559999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:07:57 2024","interfaces":[{"interface":"back","average":{"1min":2.3849999999999998,"5min":2.7559999999999998,"15min":2.3809999999999998},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":7.673,"5min":37.719999999999999,"15min":37.719999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":2.0779999999999998,"5min":2.7839999999999998,"15min":2.319},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":8.2360000000000007,"5min":38.948,"15min":38.948},"last":3.3239999999999998}]}]},{"osd":4,"up_from":0,"seq":137438953716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757488,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5288148,"kb_used_data":4383372,"kb_used_omap":0,"kb_used_meta":904768,"kb_avail":88457004,"statfs":{"total":95995035648,"available":90579972096,"internally_reserved":0,"allocated":4488572928,"data_stored":4487536197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":926482432},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3819999999999999,"15min":1.8799999999999999},"min":{"1min":0.20699999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":8.5340000000000007,"5min":10.971,"15min":46.365000000000002},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.903,"15min":2.3090000000000002},"min":{"1min":0.24299999999999999,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":7.3390000000000004,"5min":12.140000000000001,"15min":46.134},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 17:08:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.319,"15min":1.718},"min":{"1min":0.28199999999999997,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":2.7429999999999999,"5min":11.19,"15min":16.655000000000001},"last":2.5019999999999998},{"interface":"front","average":{"1min":1.585,"5min":1.984,"15min":2.0880000000000001},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.21199999999999999},"max":{"1min":5.5410000000000004,"5min":12.154,"15min":16.712},"last":0.22600000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.5600000000000001,"15min":2.0139999999999998},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1639999999999997,"5min":11.007999999999999,"15min":18.218},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.343,"15min":1.8939999999999999},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.3759999999999999,"5min":7.4850000000000003,"15min":18.327000000000002},"last":0.24099999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.5109999999999999,"15min":1.923},"min":{"1min":0.34699999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.0380000000000003,"5min":11.045,"15min":18.251000000000001},"last":1.53},{"interface":"front","average":{"1min":1.327,"5min":1.526,"15min":1.843},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":8.5459999999999994,"5min":8.5459999999999994,"15min":20.263000000000002},"last":0.27300000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:08:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.952,"15min":1.881},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.23699999999999999},"max":{"1min":4.5549999999999997,"5min":12.656000000000001,"15min":12.656000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":1.6259999999999999,"15min":1.9670000000000001},"min":{"1min":0.26500000000000001,"5min":0.26300000000000001,"15min":0.24299999999999999},"max":{"1min":2.149,"5min":11.068,"15min":20.550000000000001},"last":0.29599999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.8069999999999999,"15min":1.8069999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26400000000000001},"max":{"1min":3.5830000000000002,"5min":12.121,"15min":12.281000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.6639999999999999,"15min":1.8580000000000001},"min":{"1min":0.313,"5min":0.27300000000000002,"15min":0.26700000000000002},"max":{"1min":2.1829999999999998,"5min":11.087,"15min":22.035},"last":0.32600000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:08:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.9039999999999999,"15min":1.9570000000000001},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.8119999999999998,"5min":13.647,"15min":19.184999999999999},"last":0.308},{"interface":"front","average":{"1min":0.94199999999999995,"5min":2.3780000000000001,"15min":2.0670000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26800000000000002},"max":{"1min":5.1239999999999997,"5min":20.274000000000001,"15min":20.274000000000001},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1086128128,"data_stored":1086128128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61734912,"data_stored":61734912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6658985984,"data_stored":6658981912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5151330304,"data_stored":5151326232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4466999296,"data_stored":4466999296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4721152000,"data_stored":4721147928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:08:34.854 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T17:08:34.854 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 36 2024-02-12T17:08:34.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:33.361637+0000 mgr.y (mgr.24481) 716 : audit [DBG] from='client.35825 v1:172.21.15.33:0/2372809379' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:08:34.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:08:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:08:35.083 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:35.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:35.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:08:35.217 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:35.237 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:08:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:35.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:08:35.341 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:08:35.385 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:35.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:34.553460+0000 mgr.y (mgr.24481) 717 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:35.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:34.773887+0000 mgr.y (mgr.24481) 718 : cluster [DBG] pgmap v986: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s; 517/314850 objects degraded (0.164%); 69255/314850 objects misplaced (21.996%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T17:08:35.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:34.553460+0000 mgr.y (mgr.24481) 717 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:35.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:34.773887+0000 mgr.y (mgr.24481) 718 : cluster [DBG] pgmap v986: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s; 517/314850 objects degraded (0.164%); 69255/314850 objects misplaced (21.996%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T17:08:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:34.553460+0000 mgr.y (mgr.24481) 717 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:34.773887+0000 mgr.y (mgr.24481) 718 : cluster [DBG] pgmap v986: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s; 517/314850 objects degraded (0.164%); 69255/314850 objects misplaced (21.996%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T17:08:36.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:36.418306+0000 mon.a (mon.0) 1703 : cluster [WRN] Health check update: Degraded data redundancy: 517/314850 objects degraded (0.164%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:36.418306+0000 mon.a (mon.0) 1703 : cluster [WRN] Health check update: Degraded data redundancy: 517/314850 objects degraded (0.164%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:37.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:36.418306+0000 mon.a (mon.0) 1703 : cluster [WRN] Health check update: Degraded data redundancy: 517/314850 objects degraded (0.164%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:37.800 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:08:37.806 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:08:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:37.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:08:37.917 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:08:37.943 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:36.774980+0000 mgr.y (mgr.24481) 719 : cluster [DBG] pgmap v987: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 747 op/s; 560/311565 objects degraded (0.180%); 68443/311565 objects misplaced (21.967%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T17:08:37.945 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:37.967 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:08:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:37.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:08:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:36.774980+0000 mgr.y (mgr.24481) 719 : cluster [DBG] pgmap v987: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 747 op/s; 560/311565 objects degraded (0.180%); 68443/311565 objects misplaced (21.967%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T17:08:38.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:36.774980+0000 mgr.y (mgr.24481) 719 : cluster [DBG] pgmap v987: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 747 op/s; 560/311565 objects degraded (0.180%); 68443/311565 objects misplaced (21.967%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T17:08:38.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:38.103 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:38.124 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:38.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:38.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:38.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:38.383 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:08:38.413 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:38.435 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:08:38.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:38.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:08:38.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:38.621 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:38.776241+0000 mgr.y (mgr.24481) 720 : cluster [DBG] pgmap v988: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 849 op/s; 560/304347 objects degraded (0.184%); 66613/304347 objects misplaced (21.887%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T17:08:40.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:38.776241+0000 mgr.y (mgr.24481) 720 : cluster [DBG] pgmap v988: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 849 op/s; 560/304347 objects degraded (0.184%); 66613/304347 objects misplaced (21.887%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T17:08:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:38.776241+0000 mgr.y (mgr.24481) 720 : cluster [DBG] pgmap v988: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 849 op/s; 560/304347 objects degraded (0.184%); 66613/304347 objects misplaced (21.887%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T17:08:41.169 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:08:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:40] "GET /metrics HTTP/1.1" 200 176627 "" "Prometheus/2.33.4" 2024-02-12T17:08:41.308 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:41.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:08:41.322 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:08:41.417 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:08:41.588 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:41.885 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:40.776836+0000 mgr.y (mgr.24481) 721 : cluster [DBG] pgmap v989: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 681 op/s; 612/303282 objects degraded (0.202%); 66443/303282 objects misplaced (21.908%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T17:08:41.886 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:41.421302+0000 mon.a (mon.0) 1704 : cluster [WRN] Health check update: Degraded data redundancy: 612/303282 objects degraded (0.202%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:40.776836+0000 mgr.y (mgr.24481) 721 : cluster [DBG] pgmap v989: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 681 op/s; 612/303282 objects degraded (0.202%); 66443/303282 objects misplaced (21.908%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T17:08:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:41.421302+0000 mon.a (mon.0) 1704 : cluster [WRN] Health check update: Degraded data redundancy: 612/303282 objects degraded (0.202%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:42.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:40.776836+0000 mgr.y (mgr.24481) 721 : cluster [DBG] pgmap v989: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 681 op/s; 612/303282 objects degraded (0.202%); 66443/303282 objects misplaced (21.908%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T17:08:42.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:41.421302+0000 mon.a (mon.0) 1704 : cluster [WRN] Health check update: Degraded data redundancy: 612/303282 objects degraded (0.202%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:42.950 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:08:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:42.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:08:42.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:08:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:08:43.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:08:43.093 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:43.116 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:08:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:43.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:08:43.220 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:43.289 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:43.414 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:43.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:43.528 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:43.557 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:43.579 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:43.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:43.684 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:43.714 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:43.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:43.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:43.873 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:43.895 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:43.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:44.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:44.231 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:42.777500+0000 mgr.y (mgr.24481) 722 : cluster [DBG] pgmap v990: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.1 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s; 680/299634 objects degraded (0.227%); 65534/299634 objects misplaced (21.871%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T17:08:44.232 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:42.777500+0000 mgr.y (mgr.24481) 722 : cluster [DBG] pgmap v990: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.1 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s; 680/299634 objects degraded (0.227%); 65534/299634 objects misplaced (21.871%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T17:08:44.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:42.777500+0000 mgr.y (mgr.24481) 722 : cluster [DBG] pgmap v990: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 6.1 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s; 680/299634 objects degraded (0.227%); 65534/299634 objects misplaced (21.871%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T17:08:44.242 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:44.949 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:08:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:08:45.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:43.612223+0000 osd.1 (osd.1) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:45.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:43.612274+0000 osd.1 (osd.1) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:45.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:43.902350+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2024-02-12T17:08:45.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:43.612223+0000 osd.1 (osd.1) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:45.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:43.612274+0000 osd.1 (osd.1) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:45.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:43.902350+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2024-02-12T17:08:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:43.612223+0000 osd.1 (osd.1) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:08:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:43.612274+0000 osd.1 (osd.1) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:08:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:43.902350+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2024-02-12T17:08:45.907 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:45.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:46.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:46.104 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:46.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:46.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:44.564524+0000 mgr.y (mgr.24481) 723 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:46.182 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:44.778001+0000 mgr.y (mgr.24481) 724 : cluster [DBG] pgmap v992: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 773 op/s; 761/291675 objects degraded (0.261%); 63617/291675 objects misplaced (21.811%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:46.182 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:44.564524+0000 mgr.y (mgr.24481) 723 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:46.182 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:44.778001+0000 mgr.y (mgr.24481) 724 : cluster [DBG] pgmap v992: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 773 op/s; 761/291675 objects degraded (0.261%); 63617/291675 objects misplaced (21.811%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:46.231 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:08:46.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:44.564524+0000 mgr.y (mgr.24481) 723 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:46.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:44.778001+0000 mgr.y (mgr.24481) 724 : cluster [DBG] pgmap v992: 138 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 773 op/s; 761/291675 objects degraded (0.261%); 63617/291675 objects misplaced (21.811%); 7.2 MiB/s, 114 objects/s recovering 2024-02-12T17:08:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:46.284 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:08:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:46.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:08:46.388 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:46.495 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:46.557 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:46.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:46.663 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:46.818 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:47.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:46.424226+0000 mon.a (mon.0) 1706 : cluster [WRN] Health check update: Degraded data redundancy: 761/291675 objects degraded (0.261%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:47.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:46.454879+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e498: 8 total, 8 up, 6 in 2024-02-12T17:08:47.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:46.424226+0000 mon.a (mon.0) 1706 : cluster [WRN] Health check update: Degraded data redundancy: 761/291675 objects degraded (0.261%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:47.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:46.454879+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e498: 8 total, 8 up, 6 in 2024-02-12T17:08:47.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:46.424226+0000 mon.a (mon.0) 1706 : cluster [WRN] Health check update: Degraded data redundancy: 761/291675 objects degraded (0.261%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:47.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:46.454879+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e498: 8 total, 8 up, 6 in 2024-02-12T17:08:47.374 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:47.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:47.481 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:47.509 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:47.529 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:47.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:47.633 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:47.662 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:47.683 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:47.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:47.789 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:47.927 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:46.778403+0000 mgr.y (mgr.24481) 725 : cluster [DBG] pgmap v994: 138 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 678 op/s; 819/288066 objects degraded (0.284%); 62712/288066 objects misplaced (21.770%); 5.3 MiB/s, 84 objects/s recovering 2024-02-12T17:08:48.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:46.778403+0000 mgr.y (mgr.24481) 725 : cluster [DBG] pgmap v994: 138 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 678 op/s; 819/288066 objects degraded (0.284%); 62712/288066 objects misplaced (21.770%); 5.3 MiB/s, 84 objects/s recovering 2024-02-12T17:08:48.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:46.778403+0000 mgr.y (mgr.24481) 725 : cluster [DBG] pgmap v994: 138 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 678 op/s; 819/288066 objects degraded (0.284%); 62712/288066 objects misplaced (21.770%); 5.3 MiB/s, 84 objects/s recovering 2024-02-12T17:08:49.085 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:08:49.398 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:48.162682+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.33:0/1536985553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:08:49.399 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:48.163004+0000 mon.a (mon.0) 1708 : audit [INF] from='client.25717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:08:49.399 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:48.162682+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.33:0/1536985553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:08:49.399 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:48.163004+0000 mon.a (mon.0) 1708 : audit [INF] from='client.25717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:08:49.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:48.162682+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.33:0/1536985553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:08:49.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:48.163004+0000 mon.a (mon.0) 1708 : audit [INF] from='client.25717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:08:50.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:48.781803+0000 mgr.y (mgr.24481) 726 : cluster [DBG] pgmap v995: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 891 op/s; 819/281895 objects degraded (0.291%); 61016/281895 objects misplaced (21.645%); 7.8 MiB/s, 125 objects/s recovering 2024-02-12T17:08:50.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:48.978727+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:08:50.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:49.036145+0000 mon.a (mon.0) 1710 : audit [INF] from='client.25717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:08:50.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:49.036191+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2024-02-12T17:08:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:48.781803+0000 mgr.y (mgr.24481) 726 : cluster [DBG] pgmap v995: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 891 op/s; 819/281895 objects degraded (0.291%); 61016/281895 objects misplaced (21.645%); 7.8 MiB/s, 125 objects/s recovering 2024-02-12T17:08:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:48.978727+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:08:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:49.036145+0000 mon.a (mon.0) 1710 : audit [INF] from='client.25717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:08:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:49.036191+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2024-02-12T17:08:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:48.781803+0000 mgr.y (mgr.24481) 726 : cluster [DBG] pgmap v995: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 891 op/s; 819/281895 objects degraded (0.291%); 61016/281895 objects misplaced (21.645%); 7.8 MiB/s, 125 objects/s recovering 2024-02-12T17:08:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:48.978727+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:08:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:49.036145+0000 mon.a (mon.0) 1710 : audit [INF] from='client.25717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:08:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:49.036191+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2024-02-12T17:08:50.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:08:50.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:08:50.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:08:50.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:08:50.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:08:50.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:08:50.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:08:50.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:08:50.967 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:08:50.967 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:08:50.967 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:08:50.967 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:08:50.967 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:08:50.967 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:08:50.967 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:08:50.967 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:08:51.028 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:08:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:50] "GET /metrics HTTP/1.1" 200 176633 "" "Prometheus/2.33.4" 2024-02-12T17:08:51.419 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:50.295654+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? v1:172.21.15.33:0/2236074420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:08:51.420 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:50.295654+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? v1:172.21.15.33:0/2236074420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:08:51.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:50.295654+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? v1:172.21.15.33:0/2236074420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:08:51.560 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:51.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:51.668 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:51.743 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:51.856 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:51.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:51.961 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:08:52.045 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:50.783108+0000 mgr.y (mgr.24481) 727 : cluster [DBG] pgmap v997: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 744 B/s rd, 0 B/s wr, 525 op/s; 900/280860 objects degraded (0.320%); 60819/280860 objects misplaced (21.655%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T17:08:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:51.428106+0000 mon.a (mon.0) 1712 : cluster [WRN] Health check update: Degraded data redundancy: 900/280860 objects degraded (0.320%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:51.437108+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2024-02-12T17:08:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:50.783108+0000 mgr.y (mgr.24481) 727 : cluster [DBG] pgmap v997: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 744 B/s rd, 0 B/s wr, 525 op/s; 900/280860 objects degraded (0.320%); 60819/280860 objects misplaced (21.655%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T17:08:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:51.428106+0000 mon.a (mon.0) 1712 : cluster [WRN] Health check update: Degraded data redundancy: 900/280860 objects degraded (0.320%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:52.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:51.437108+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2024-02-12T17:08:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:50.783108+0000 mgr.y (mgr.24481) 727 : cluster [DBG] pgmap v997: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 744 B/s rd, 0 B/s wr, 525 op/s; 900/280860 objects degraded (0.320%); 60819/280860 objects misplaced (21.655%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T17:08:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:51.428106+0000 mon.a (mon.0) 1712 : cluster [WRN] Health check update: Degraded data redundancy: 900/280860 objects degraded (0.320%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:52.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:51.437108+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2024-02-12T17:08:52.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:08:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:08:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:08:53.835 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:08:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:53.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:08:53.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:08:53.971 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:53.994 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:08:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:53.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:08:54.098 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:08:54.204 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:54.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:52.784679+0000 mgr.y (mgr.24481) 728 : cluster [DBG] pgmap v999: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.6 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 807 B/s rd, 0 B/s wr, 544 op/s; 950/277722 objects degraded (0.342%); 60010/277722 objects misplaced (21.608%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T17:08:54.284 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:52.784679+0000 mgr.y (mgr.24481) 728 : cluster [DBG] pgmap v999: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.6 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 807 B/s rd, 0 B/s wr, 544 op/s; 950/277722 objects degraded (0.342%); 60010/277722 objects misplaced (21.608%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T17:08:54.284 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:52.784679+0000 mgr.y (mgr.24481) 728 : cluster [DBG] pgmap v999: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.6 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 807 B/s rd, 0 B/s wr, 544 op/s; 950/277722 objects degraded (0.342%); 60010/277722 objects misplaced (21.608%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T17:08:54.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:08:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:08:55.864 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:08:55.864 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:08:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:54.579304+0000 mgr.y (mgr.24481) 729 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:54.785535+0000 mgr.y (mgr.24481) 730 : cluster [DBG] pgmap v1000: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 759 op/s; 1014/269847 objects degraded (0.376%); 57864/269847 objects misplaced (21.443%); 6.3 MiB/s, 100 objects/s recovering 2024-02-12T17:08:56.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:54.579304+0000 mgr.y (mgr.24481) 729 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:56.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:54.785535+0000 mgr.y (mgr.24481) 730 : cluster [DBG] pgmap v1000: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 759 op/s; 1014/269847 objects degraded (0.376%); 57864/269847 objects misplaced (21.443%); 6.3 MiB/s, 100 objects/s recovering 2024-02-12T17:08:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:54.579304+0000 mgr.y (mgr.24481) 729 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:08:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:54.785535+0000 mgr.y (mgr.24481) 730 : cluster [DBG] pgmap v1000: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 759 op/s; 1014/269847 objects degraded (0.376%); 57864/269847 objects misplaced (21.443%); 6.3 MiB/s, 100 objects/s recovering 2024-02-12T17:08:57.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:56.430261+0000 mon.a (mon.0) 1714 : cluster [WRN] Health check update: Degraded data redundancy: 1014/269847 objects degraded (0.376%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:56.430261+0000 mon.a (mon.0) 1714 : cluster [WRN] Health check update: Degraded data redundancy: 1014/269847 objects degraded (0.376%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:57.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:56.430261+0000 mon.a (mon.0) 1714 : cluster [WRN] Health check update: Degraded data redundancy: 1014/269847 objects degraded (0.376%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:08:57.611 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:08:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:57.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:08:57.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:57.745 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:57.767 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:57.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:57.871 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:08:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:57.910 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:08:57.921 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:08:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:57.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:08:58.025 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:58.085 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:58.196 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:08:58.202 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:56.787702+0000 mgr.y (mgr.24481) 731 : cluster [DBG] pgmap v1001: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 649 op/s; 1086/266310 objects degraded (0.408%); 56994/266310 objects misplaced (21.401%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:08:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:58.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:08:58.203 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:56.787702+0000 mgr.y (mgr.24481) 731 : cluster [DBG] pgmap v1001: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 649 op/s; 1086/266310 objects degraded (0.408%); 56994/266310 objects misplaced (21.401%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:08:58.304 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:08:58.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:58.353 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:08:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:08:58.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:08:58.458 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:08:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:56.787702+0000 mgr.y (mgr.24481) 731 : cluster [DBG] pgmap v1001: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 649 op/s; 1086/266310 objects degraded (0.408%); 56994/266310 objects misplaced (21.401%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:08:58.554 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:08:59.478 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 10 pg_num to 36 2024-02-12T17:08:59.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:58.702664+0000 mon.c (mon.2) 252 : audit [INF] from='client.? v1:172.21.15.33:0/3097079244' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-02-12T17:08:59.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:08:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:58.703917+0000 mon.a (mon.0) 1715 : audit [INF] from='client.35837 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-02-12T17:08:59.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:58.702664+0000 mon.c (mon.2) 252 : audit [INF] from='client.? v1:172.21.15.33:0/3097079244' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-02-12T17:08:59.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:08:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:58.703917+0000 mon.a (mon.0) 1715 : audit [INF] from='client.35837 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-02-12T17:08:59.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:58.702664+0000 mon.c (mon.2) 252 : audit [INF] from='client.? v1:172.21.15.33:0/3097079244' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-02-12T17:08:59.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:08:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:58.703917+0000 mon.a (mon.0) 1715 : audit [INF] from='client.35837 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-02-12T17:08:59.620 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:09:00.327 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T17:09:00.363 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:00.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:00.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:00.471 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:58.788542+0000 mgr.y (mgr.24481) 732 : cluster [DBG] pgmap v1002: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 719 op/s; 1086/259860 objects degraded (0.418%); 55084/259860 objects misplaced (21.198%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:08:59.131168+0000 mon.a (mon.0) 1716 : audit [INF] from='client.35837 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:59.131251+0000 mon.a (mon.0) 1717 : cluster [DBG] osdmap e501: 8 total, 8 up, 6 in 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:59.575558+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:08:59.575597+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:58.788542+0000 mgr.y (mgr.24481) 732 : cluster [DBG] pgmap v1002: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 719 op/s; 1086/259860 objects degraded (0.418%); 55084/259860 objects misplaced (21.198%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:08:59.131168+0000 mon.a (mon.0) 1716 : audit [INF] from='client.35837 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:59.131251+0000 mon.a (mon.0) 1717 : cluster [DBG] osdmap e501: 8 total, 8 up, 6 in 2024-02-12T17:09:00.478 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:59.575558+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:00.479 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:08:59.575597+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:58.788542+0000 mgr.y (mgr.24481) 732 : cluster [DBG] pgmap v1002: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 719 op/s; 1086/259860 objects degraded (0.418%); 55084/259860 objects misplaced (21.198%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:09:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:08:59.131168+0000 mon.a (mon.0) 1716 : audit [INF] from='client.35837 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-02-12T17:09:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:59.131251+0000 mon.a (mon.0) 1717 : cluster [DBG] osdmap e501: 8 total, 8 up, 6 in 2024-02-12T17:09:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:59.575558+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:08:59.575597+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:00.565 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:00.925 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:00.933 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:00.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:00.950 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:00.998 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:01.038 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:01.115 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:09:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:00] "GET /metrics HTTP/1.1" 200 176633 "" "Prometheus/2.33.4" 2024-02-12T17:09:01.116 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:01.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:00.129654+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e502: 8 total, 8 up, 6 in 2024-02-12T17:09:01.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:00.789845+0000 mon.c (mon.2) 253 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:01.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:00.793652+0000 mon.a (mon.0) 1719 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:01.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:01.148080+0000 mon.a (mon.0) 1720 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-12T17:09:01.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.148137+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e503: 8 total, 8 up, 6 in 2024-02-12T17:09:01.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:00.129654+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e502: 8 total, 8 up, 6 in 2024-02-12T17:09:01.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:00.789845+0000 mon.c (mon.2) 253 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:01.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:00.793652+0000 mon.a (mon.0) 1719 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:01.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:01.148080+0000 mon.a (mon.0) 1720 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-12T17:09:01.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.148137+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e503: 8 total, 8 up, 6 in 2024-02-12T17:09:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:00.129654+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e502: 8 total, 8 up, 6 in 2024-02-12T17:09:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:00.789845+0000 mon.c (mon.2) 253 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:00.793652+0000 mon.a (mon.0) 1719 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:01.148080+0000 mon.a (mon.0) 1720 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-12T17:09:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.148137+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e503: 8 total, 8 up, 6 in 2024-02-12T17:09:01.765 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:01.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:01.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:01.870 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:01.981 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:02.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:00.333694+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:02.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:00.333750+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:02.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:00.789042+0000 mgr.y (mgr.24481) 733 : cluster [DBG] pgmap v1005: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 1137/258891 objects degraded (0.439%); 54917/258891 objects misplaced (21.212%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:09:02.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.432157+0000 mon.a (mon.0) 1722 : cluster [WRN] Health check update: Degraded data redundancy: 1137/258891 objects degraded (0.439%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:02.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.445703+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2024-02-12T17:09:02.248 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:00.333694+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:02.248 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:00.333750+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:02.248 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:00.789042+0000 mgr.y (mgr.24481) 733 : cluster [DBG] pgmap v1005: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 1137/258891 objects degraded (0.439%); 54917/258891 objects misplaced (21.212%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:09:02.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.432157+0000 mon.a (mon.0) 1722 : cluster [WRN] Health check update: Degraded data redundancy: 1137/258891 objects degraded (0.439%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:02.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.445703+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2024-02-12T17:09:02.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.447539+0000 osd.7 (osd.7) 255 : cluster [INF] 10.1 continuing backfill to osd.1 from (466'4500,502'9014] MIN to 502'9014 2024-02-12T17:09:02.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.448235+0000 osd.2 (osd.2) 182 : cluster [INF] 10.0 continuing backfill to osd.1 from (466'4200,502'8747] MIN to 502'8747 2024-02-12T17:09:02.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.449102+0000 osd.7 (osd.7) 256 : cluster [INF] 10.c continuing backfill to osd.1 from (471'6510,502'10198] MIN to 502'10198 2024-02-12T17:09:02.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.450054+0000 osd.2 (osd.2) 183 : cluster [INF] 10.b continuing backfill to osd.1 from (472'6570,502'10132] MIN to 502'10132 2024-02-12T17:09:02.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.460887+0000 osd.3 (osd.3) 181 : cluster [INF] 10.2 continuing backfill to osd.1 from (466'4100,502'8775] MIN to 502'8775 2024-02-12T17:09:02.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:01.465688+0000 osd.3 (osd.3) 182 : cluster [DBG] 10.f starting backfill to osd.1 from (472'6748,502'10157] MIN to 502'10158 2024-02-12T17:09:02.510 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:00.333694+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:02.510 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:00.333750+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:02.510 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:00.789042+0000 mgr.y (mgr.24481) 733 : cluster [DBG] pgmap v1005: 138 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 785 op/s; 1137/258891 objects degraded (0.439%); 54917/258891 objects misplaced (21.212%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:09:02.510 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.432157+0000 mon.a (mon.0) 1722 : cluster [WRN] Health check update: Degraded data redundancy: 1137/258891 objects degraded (0.439%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:02.510 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.445703+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2024-02-12T17:09:02.510 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.447539+0000 osd.7 (osd.7) 255 : cluster [INF] 10.1 continuing backfill to osd.1 from (466'4500,502'9014] MIN to 502'9014 2024-02-12T17:09:02.511 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.448235+0000 osd.2 (osd.2) 182 : cluster [INF] 10.0 continuing backfill to osd.1 from (466'4200,502'8747] MIN to 502'8747 2024-02-12T17:09:02.511 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.449102+0000 osd.7 (osd.7) 256 : cluster [INF] 10.c continuing backfill to osd.1 from (471'6510,502'10198] MIN to 502'10198 2024-02-12T17:09:02.511 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.450054+0000 osd.2 (osd.2) 183 : cluster [INF] 10.b continuing backfill to osd.1 from (472'6570,502'10132] MIN to 502'10132 2024-02-12T17:09:02.511 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.460887+0000 osd.3 (osd.3) 181 : cluster [INF] 10.2 continuing backfill to osd.1 from (466'4100,502'8775] MIN to 502'8775 2024-02-12T17:09:02.511 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:01.465688+0000 osd.3 (osd.3) 182 : cluster [DBG] 10.f starting backfill to osd.1 from (472'6748,502'10157] MIN to 502'10158 2024-02-12T17:09:02.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.447539+0000 osd.7 (osd.7) 255 : cluster [INF] 10.1 continuing backfill to osd.1 from (466'4500,502'9014] MIN to 502'9014 2024-02-12T17:09:02.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.448235+0000 osd.2 (osd.2) 182 : cluster [INF] 10.0 continuing backfill to osd.1 from (466'4200,502'8747] MIN to 502'8747 2024-02-12T17:09:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.449102+0000 osd.7 (osd.7) 256 : cluster [INF] 10.c continuing backfill to osd.1 from (471'6510,502'10198] MIN to 502'10198 2024-02-12T17:09:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.450054+0000 osd.2 (osd.2) 183 : cluster [INF] 10.b continuing backfill to osd.1 from (472'6570,502'10132] MIN to 502'10132 2024-02-12T17:09:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.460887+0000 osd.3 (osd.3) 181 : cluster [INF] 10.2 continuing backfill to osd.1 from (466'4100,502'8775] MIN to 502'8775 2024-02-12T17:09:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:01.465688+0000 osd.3 (osd.3) 182 : cluster [DBG] 10.f starting backfill to osd.1 from (472'6748,502'10157] MIN to 502'10158 2024-02-12T17:09:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:09:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:09:03.038 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:03.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:03.143 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:03.170 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:03.193 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:03.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:03.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:09:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:03.343 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:09:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:03.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:09:03.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:03.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:02.448050+0000 mon.a (mon.0) 1724 : cluster [DBG] osdmap e505: 8 total, 8 up, 6 in 2024-02-12T17:09:03.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:02.789562+0000 mgr.y (mgr.24481) 734 : cluster [DBG] pgmap v1009: 148 pgs: 10 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 279 op/s; 1174/255831 objects degraded (0.459%); 54159/255831 objects misplaced (21.170%); 1016 KiB/s, 15 objects/s recovering 2024-02-12T17:09:03.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:02.789991+0000 mon.c (mon.2) 254 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:03.458 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:02.791336+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:03.502 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:03.731 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:02.448050+0000 mon.a (mon.0) 1724 : cluster [DBG] osdmap e505: 8 total, 8 up, 6 in 2024-02-12T17:09:03.731 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:02.789562+0000 mgr.y (mgr.24481) 734 : cluster [DBG] pgmap v1009: 148 pgs: 10 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 279 op/s; 1174/255831 objects degraded (0.459%); 54159/255831 objects misplaced (21.170%); 1016 KiB/s, 15 objects/s recovering 2024-02-12T17:09:03.731 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:02.789991+0000 mon.c (mon.2) 254 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:03.731 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:02.791336+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:02.448050+0000 mon.a (mon.0) 1724 : cluster [DBG] osdmap e505: 8 total, 8 up, 6 in 2024-02-12T17:09:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:02.789562+0000 mgr.y (mgr.24481) 734 : cluster [DBG] pgmap v1009: 148 pgs: 10 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 14 active+remapped+backfill_wait, 115 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 279 op/s; 1174/255831 objects degraded (0.459%); 54159/255831 objects misplaced (21.170%); 1016 KiB/s, 15 objects/s recovering 2024-02-12T17:09:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:02.789991+0000 mon.c (mon.2) 254 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:02.791336+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-12T17:09:04.466 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:04.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:04.576 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:04.603 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:04.623 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:04.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:04.727 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:04.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:09:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:09:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:02.919492+0000 osd.4 (osd.4) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:02.919538+0000 osd.4 (osd.4) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:03.297483+0000 osd.3 (osd.3) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:03.297533+0000 osd.3 (osd.3) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:03.458019+0000 mon.a (mon.0) 1726 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-02-12T17:09:04.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:03.458074+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e506: 8 total, 8 up, 6 in 2024-02-12T17:09:04.754 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:04.775 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:02.919492+0000 osd.4 (osd.4) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:02.919538+0000 osd.4 (osd.4) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:03.297483+0000 osd.3 (osd.3) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:03.297533+0000 osd.3 (osd.3) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:03.458019+0000 mon.a (mon.0) 1726 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-02-12T17:09:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:03.458074+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e506: 8 total, 8 up, 6 in 2024-02-12T17:09:04.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:02.919492+0000 osd.4 (osd.4) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:04.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:02.919538+0000 osd.4 (osd.4) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:04.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:03.297483+0000 osd.3 (osd.3) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:04.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:03.297533+0000 osd.3 (osd.3) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:04.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:03.458019+0000 mon.a (mon.0) 1726 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-02-12T17:09:04.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:03.458074+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e506: 8 total, 8 up, 6 in 2024-02-12T17:09:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:04.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:04.880 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:04.914 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:09:05.921 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:09:05.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:09:05.921 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-02-12T17:09:05.921 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd in 6 2024-02-12T17:09:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:04.458019+0000 osd.2 (osd.2) 184 : cluster [INF] 10.20 continuing backfill to osd.1 from (466'4200,502'8747] MIN to 502'8747 2024-02-12T17:09:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:04.458051+0000 osd.2 (osd.2) 185 : cluster [DBG] 10.20 starting backfill to osd.3 from (0'0,0'0] MAX to 502'8747 2024-02-12T17:09:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:04.459984+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e507: 8 total, 8 up, 6 in 2024-02-12T17:09:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:04.467096+0000 osd.3 (osd.3) 185 : cluster [DBG] 10.1f starting backfill to osd.1 from (472'6748,502'10157] MIN to 502'10158 2024-02-12T17:09:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:04.477329+0000 osd.3 (osd.3) 186 : cluster [DBG] 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 502'10158 2024-02-12T17:09:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:04.589530+0000 mgr.y (mgr.24481) 735 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:04.790098+0000 mgr.y (mgr.24481) 736 : cluster [DBG] pgmap v1012: 148 pgs: 3 remapped+peering, 2 peering, 3 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 121 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 327 op/s; 1199/252270 objects degraded (0.475%); 44174/252270 objects misplaced (17.511%); 24 MiB/s, 380 objects/s recovering 2024-02-12T17:09:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:04.983682+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.33:0/2441100456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:04.984057+0000 mon.a (mon.0) 1729 : audit [INF] from='client.35843 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.455604+0000 mon.a (mon.0) 1730 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:09:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:05.461689+0000 mon.a (mon.0) 1731 : audit [INF] from='client.35843 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:09:05.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.461754+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e508: 8 total, 8 up, 6 in 2024-02-12T17:09:05.993 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:04.458019+0000 osd.2 (osd.2) 184 : cluster [INF] 10.20 continuing backfill to osd.1 from (466'4200,502'8747] MIN to 502'8747 2024-02-12T17:09:05.993 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:04.458051+0000 osd.2 (osd.2) 185 : cluster [DBG] 10.20 starting backfill to osd.3 from (0'0,0'0] MAX to 502'8747 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:04.459984+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e507: 8 total, 8 up, 6 in 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:04.467096+0000 osd.3 (osd.3) 185 : cluster [DBG] 10.1f starting backfill to osd.1 from (472'6748,502'10157] MIN to 502'10158 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:04.477329+0000 osd.3 (osd.3) 186 : cluster [DBG] 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 502'10158 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:04.589530+0000 mgr.y (mgr.24481) 735 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:04.790098+0000 mgr.y (mgr.24481) 736 : cluster [DBG] pgmap v1012: 148 pgs: 3 remapped+peering, 2 peering, 3 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 121 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 327 op/s; 1199/252270 objects degraded (0.475%); 44174/252270 objects misplaced (17.511%); 24 MiB/s, 380 objects/s recovering 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:04.983682+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.33:0/2441100456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:04.984057+0000 mon.a (mon.0) 1729 : audit [INF] from='client.35843 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.455604+0000 mon.a (mon.0) 1730 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:05.461689+0000 mon.a (mon.0) 1731 : audit [INF] from='client.35843 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:09:05.994 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.461754+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e508: 8 total, 8 up, 6 in 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:04.458019+0000 osd.2 (osd.2) 184 : cluster [INF] 10.20 continuing backfill to osd.1 from (466'4200,502'8747] MIN to 502'8747 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:04.458051+0000 osd.2 (osd.2) 185 : cluster [DBG] 10.20 starting backfill to osd.3 from (0'0,0'0] MAX to 502'8747 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:04.459984+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e507: 8 total, 8 up, 6 in 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:04.467096+0000 osd.3 (osd.3) 185 : cluster [DBG] 10.1f starting backfill to osd.1 from (472'6748,502'10157] MIN to 502'10158 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:04.477329+0000 osd.3 (osd.3) 186 : cluster [DBG] 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 502'10158 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:04.589530+0000 mgr.y (mgr.24481) 735 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:04.790098+0000 mgr.y (mgr.24481) 736 : cluster [DBG] pgmap v1012: 148 pgs: 3 remapped+peering, 2 peering, 3 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 121 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 327 op/s; 1199/252270 objects degraded (0.475%); 44174/252270 objects misplaced (17.511%); 24 MiB/s, 380 objects/s recovering 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:04.983682+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.33:0/2441100456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:04.984057+0000 mon.a (mon.0) 1729 : audit [INF] from='client.35843 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.455604+0000 mon.a (mon.0) 1730 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:09:05.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:05.461689+0000 mon.a (mon.0) 1731 : audit [INF] from='client.35843 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:09:05.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.461754+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e508: 8 total, 8 up, 6 in 2024-02-12T17:09:06.272 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:06.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:06.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:09:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:06.441 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:09:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:06.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:09:06.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:06.599 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:06.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:06.705 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.459151+0000 osd.7 (osd.7) 257 : cluster [INF] 10.1c continuing backfill to osd.1 from (471'6510,502'10198] MIN to 502'10198 2024-02-12T17:09:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.459241+0000 osd.7 (osd.7) 258 : cluster [DBG] 10.1c starting backfill to osd.5 from (0'0,0'0] MAX to 502'10198 2024-02-12T17:09:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.461432+0000 osd.5 (osd.5) 258 : cluster [INF] 10.22 continuing backfill to osd.1 from (466'4100,502'8775] MIN to 502'8775 2024-02-12T17:09:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.463057+0000 osd.2 (osd.2) 186 : cluster [DBG] 10.1a starting backfill to osd.0 from (0'0,0'0] MAX to 504'10078 2024-02-12T17:09:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.463917+0000 osd.5 (osd.5) 259 : cluster [DBG] 10.1e starting backfill to osd.1 from (0'0,0'0] MAX to 504'10140 2024-02-12T17:09:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.464684+0000 osd.2 (osd.2) 187 : cluster [DBG] 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 504'10225 2024-02-12T17:09:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.465818+0000 osd.2 (osd.2) 188 : cluster [DBG] 10.23 starting backfill to osd.1 from (0'0,0'0] MAX to 504'8951 2024-02-12T17:09:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.470957+0000 osd.3 (osd.3) 187 : cluster [DBG] 10.1b starting backfill to osd.5 from (0'0,0'0] MAX to 502'10132 2024-02-12T17:09:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.473684+0000 osd.2 (osd.2) 189 : cluster [DBG] 10.1d starting backfill to osd.5 from (0'0,0'0] MAX to 504'10225 2024-02-12T17:09:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:05.474643+0000 osd.2 (osd.2) 190 : cluster [DBG] 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 504'8951 2024-02-12T17:09:06.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:06.434241+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check update: Degraded data redundancy: 1199/252270 objects degraded (0.475%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:06.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.459151+0000 osd.7 (osd.7) 257 : cluster [INF] 10.1c continuing backfill to osd.1 from (471'6510,502'10198] MIN to 502'10198 2024-02-12T17:09:06.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.459241+0000 osd.7 (osd.7) 258 : cluster [DBG] 10.1c starting backfill to osd.5 from (0'0,0'0] MAX to 502'10198 2024-02-12T17:09:06.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.461432+0000 osd.5 (osd.5) 258 : cluster [INF] 10.22 continuing backfill to osd.1 from (466'4100,502'8775] MIN to 502'8775 2024-02-12T17:09:06.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.463057+0000 osd.2 (osd.2) 186 : cluster [DBG] 10.1a starting backfill to osd.0 from (0'0,0'0] MAX to 504'10078 2024-02-12T17:09:06.967 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.463917+0000 osd.5 (osd.5) 259 : cluster [DBG] 10.1e starting backfill to osd.1 from (0'0,0'0] MAX to 504'10140 2024-02-12T17:09:06.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.464684+0000 osd.2 (osd.2) 187 : cluster [DBG] 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 504'10225 2024-02-12T17:09:06.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.465818+0000 osd.2 (osd.2) 188 : cluster [DBG] 10.23 starting backfill to osd.1 from (0'0,0'0] MAX to 504'8951 2024-02-12T17:09:06.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.470957+0000 osd.3 (osd.3) 187 : cluster [DBG] 10.1b starting backfill to osd.5 from (0'0,0'0] MAX to 502'10132 2024-02-12T17:09:06.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.473684+0000 osd.2 (osd.2) 189 : cluster [DBG] 10.1d starting backfill to osd.5 from (0'0,0'0] MAX to 504'10225 2024-02-12T17:09:06.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:05.474643+0000 osd.2 (osd.2) 190 : cluster [DBG] 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 504'8951 2024-02-12T17:09:06.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:06.434241+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check update: Degraded data redundancy: 1199/252270 objects degraded (0.475%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:06.968 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.459151+0000 osd.7 (osd.7) 257 : cluster [INF] 10.1c continuing backfill to osd.1 from (471'6510,502'10198] MIN to 502'10198 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.459241+0000 osd.7 (osd.7) 258 : cluster [DBG] 10.1c starting backfill to osd.5 from (0'0,0'0] MAX to 502'10198 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.461432+0000 osd.5 (osd.5) 258 : cluster [INF] 10.22 continuing backfill to osd.1 from (466'4100,502'8775] MIN to 502'8775 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.463057+0000 osd.2 (osd.2) 186 : cluster [DBG] 10.1a starting backfill to osd.0 from (0'0,0'0] MAX to 504'10078 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.463917+0000 osd.5 (osd.5) 259 : cluster [DBG] 10.1e starting backfill to osd.1 from (0'0,0'0] MAX to 504'10140 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.464684+0000 osd.2 (osd.2) 187 : cluster [DBG] 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 504'10225 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.465818+0000 osd.2 (osd.2) 188 : cluster [DBG] 10.23 starting backfill to osd.1 from (0'0,0'0] MAX to 504'8951 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.470957+0000 osd.3 (osd.3) 187 : cluster [DBG] 10.1b starting backfill to osd.5 from (0'0,0'0] MAX to 502'10132 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.473684+0000 osd.2 (osd.2) 189 : cluster [DBG] 10.1d starting backfill to osd.5 from (0'0,0'0] MAX to 504'10225 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:05.474643+0000 osd.2 (osd.2) 190 : cluster [DBG] 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 504'8951 2024-02-12T17:09:06.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:06.434241+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check update: Degraded data redundancy: 1199/252270 objects degraded (0.475%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:06.982 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:06.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:07.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:07.146 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:07.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:07.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:07.251 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:07.282 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:07.303 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:07.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:07.381 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:09:07.381 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:09:07.407 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:07.478 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:07.565 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:06.790818+0000 mgr.y (mgr.24481) 737 : cluster [DBG] pgmap v1014: 148 pgs: 2 remapped+peering, 2 peering, 2 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 11 active+remapped+backfill_wait, 121 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 945 B/s rd, 0 op/s; 1199/239667 objects degraded (0.500%); 47810/239667 objects misplaced (19.949%); 21 MiB/s, 332 objects/s recovering 2024-02-12T17:09:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:06.790818+0000 mgr.y (mgr.24481) 737 : cluster [DBG] pgmap v1014: 148 pgs: 2 remapped+peering, 2 peering, 2 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 11 active+remapped+backfill_wait, 121 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 945 B/s rd, 0 op/s; 1199/239667 objects degraded (0.500%); 47810/239667 objects misplaced (19.949%); 21 MiB/s, 332 objects/s recovering 2024-02-12T17:09:08.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:06.790818+0000 mgr.y (mgr.24481) 737 : cluster [DBG] pgmap v1014: 148 pgs: 2 remapped+peering, 2 peering, 2 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 11 active+remapped+backfill_wait, 121 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 945 B/s rd, 0 op/s; 1199/239667 objects degraded (0.500%); 47810/239667 objects misplaced (19.949%); 21 MiB/s, 332 objects/s recovering 2024-02-12T17:09:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:07.550244+0000 mon.a (mon.0) 1734 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:07.916921+0000 mon.a (mon.0) 1735 : audit [DBG] from='client.? v1:172.21.15.33:0/2214356690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:08.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:07.550244+0000 mon.a (mon.0) 1734 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:08.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:07.916921+0000 mon.a (mon.0) 1735 : audit [DBG] from='client.? v1:172.21.15.33:0/2214356690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:07.550244+0000 mon.a (mon.0) 1734 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:07.916921+0000 mon.a (mon.0) 1735 : audit [DBG] from='client.? v1:172.21.15.33:0/2214356690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:08.991 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:09:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:08.791759+0000 mgr.y (mgr.24481) 738 : cluster [DBG] pgmap v1015: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 5.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 480 op/s; 1199/247206 objects degraded (0.485%); 67169/247206 objects misplaced (27.171%); 33 MiB/s, 536 objects/s recovering 2024-02-12T17:09:10.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:08.791759+0000 mgr.y (mgr.24481) 738 : cluster [DBG] pgmap v1015: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 5.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 480 op/s; 1199/247206 objects degraded (0.485%); 67169/247206 objects misplaced (27.171%); 33 MiB/s, 536 objects/s recovering 2024-02-12T17:09:10.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:08.791759+0000 mgr.y (mgr.24481) 738 : cluster [DBG] pgmap v1015: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 5.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 480 op/s; 1199/247206 objects degraded (0.485%); 67169/247206 objects misplaced (27.171%); 33 MiB/s, 536 objects/s recovering 2024-02-12T17:09:11.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:09:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:10] "GET /metrics HTTP/1.1" 200 176640 "" "Prometheus/2.33.4" 2024-02-12T17:09:11.677 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:11.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:11.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T17:09:11.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:11.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:11.855 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:09:11.855 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:09:11.856 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:09:11.856 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:09:11.856 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:09:11.856 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:09:11.856 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:09:11.856 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:09:11.856 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:09:11.856 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:09:11.857 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:09:11.857 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:09:11.857 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:09:11.857 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:09:11.857 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:09:11.857 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:09:11.948 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:10.792303+0000 mgr.y (mgr.24481) 739 : cluster [DBG] pgmap v1016: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 5.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 697 B/s rd, 0 B/s wr, 314 op/s; 1199/246435 objects degraded (0.487%); 67053/246435 objects misplaced (27.209%); 17 MiB/s, 268 objects/s recovering 2024-02-12T17:09:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:11.436295+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: Degraded data redundancy: 1199/246435 objects degraded (0.487%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:12.001 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:10.792303+0000 mgr.y (mgr.24481) 739 : cluster [DBG] pgmap v1016: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 5.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 697 B/s rd, 0 B/s wr, 314 op/s; 1199/246435 objects degraded (0.487%); 67053/246435 objects misplaced (27.209%); 17 MiB/s, 268 objects/s recovering 2024-02-12T17:09:12.001 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:11.436295+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: Degraded data redundancy: 1199/246435 objects degraded (0.487%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:12.001 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:10.792303+0000 mgr.y (mgr.24481) 739 : cluster [DBG] pgmap v1016: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 5.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 697 B/s rd, 0 B/s wr, 314 op/s; 1199/246435 objects degraded (0.487%); 67053/246435 objects misplaced (27.209%); 17 MiB/s, 268 objects/s recovering 2024-02-12T17:09:12.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:11.436295+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: Degraded data redundancy: 1199/246435 objects degraded (0.487%), 2 pgs degraded (PG_DEGRADED) 2024-02-12T17:09:12.006 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:12.072 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:12.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:12.081 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:12.085 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:12.179 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:12.249 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:09:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:09:13.019 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:13.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:13.126 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:13.617 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:14.108 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:14.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:14.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:14.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:12.794675+0000 mgr.y (mgr.24481) 740 : cluster [DBG] pgmap v1017: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 614 B/s rd, 0 B/s wr, 390 op/s; 1243/242535 objects degraded (0.513%); 65857/242535 objects misplaced (27.154%); 15 MiB/s, 234 objects/s recovering 2024-02-12T17:09:14.242 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:14.265 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:14.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:12.794675+0000 mgr.y (mgr.24481) 740 : cluster [DBG] pgmap v1017: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 614 B/s rd, 0 B/s wr, 390 op/s; 1243/242535 objects degraded (0.513%); 65857/242535 objects misplaced (27.154%); 15 MiB/s, 234 objects/s recovering 2024-02-12T17:09:14.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:12.794675+0000 mgr.y (mgr.24481) 740 : cluster [DBG] pgmap v1017: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 614 B/s rd, 0 B/s wr, 390 op/s; 1243/242535 objects degraded (0.513%); 65857/242535 objects misplaced (27.154%); 15 MiB/s, 234 objects/s recovering 2024-02-12T17:09:14.369 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:14.447 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:14.478 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:14.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:14.590 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:09:14.624 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:14.648 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:09:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:14.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:09:14.753 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:14.815 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:14.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:09:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:09:14.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:09:16.062 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:14.430223+0000 osd.2 (osd.2) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:16.062 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:14.430272+0000 osd.2 (osd.2) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:16.062 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:14.599472+0000 mgr.y (mgr.24481) 741 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:16.062 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:14.795593+0000 mgr.y (mgr.24481) 742 : cluster [DBG] pgmap v1018: 148 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 921 B/s rd, 0 B/s wr, 613 op/s; 1318/233868 objects degraded (0.564%); 62960/233868 objects misplaced (26.921%); 18 MiB/s, 286 objects/s recovering 2024-02-12T17:09:16.062 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:14.893697+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e509: 8 total, 8 up, 6 in 2024-02-12T17:09:16.196 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:16.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:16.205 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:14.430223+0000 osd.2 (osd.2) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:16.205 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:14.430272+0000 osd.2 (osd.2) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:16.205 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:14.599472+0000 mgr.y (mgr.24481) 741 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:16.205 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:14.795593+0000 mgr.y (mgr.24481) 742 : cluster [DBG] pgmap v1018: 148 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 921 B/s rd, 0 B/s wr, 613 op/s; 1318/233868 objects degraded (0.564%); 62960/233868 objects misplaced (26.921%); 18 MiB/s, 286 objects/s recovering 2024-02-12T17:09:16.205 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:14.893697+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e509: 8 total, 8 up, 6 in 2024-02-12T17:09:16.206 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:14.430223+0000 osd.2 (osd.2) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:16.206 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:14.430272+0000 osd.2 (osd.2) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:16.206 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:14.599472+0000 mgr.y (mgr.24481) 741 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:16.206 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:14.795593+0000 mgr.y (mgr.24481) 742 : cluster [DBG] pgmap v1018: 148 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 921 B/s rd, 0 B/s wr, 613 op/s; 1318/233868 objects degraded (0.564%); 62960/233868 objects misplaced (26.921%); 18 MiB/s, 286 objects/s recovering 2024-02-12T17:09:16.206 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:14.893697+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e509: 8 total, 8 up, 6 in 2024-02-12T17:09:16.306 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:16.354 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:09:16.354 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:09:16.354 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-02-12T17:09:16.354 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:09:16.361 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:16.438075+0000 mon.a (mon.0) 1738 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:16.438100+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: Degraded data redundancy: 1318/233868 objects degraded (0.564%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:16.457535+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e510: 8 total, 8 up, 6 in 2024-02-12T17:09:17.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:16.438075+0000 mon.a (mon.0) 1738 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:17.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:16.438100+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: Degraded data redundancy: 1318/233868 objects degraded (0.564%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:17.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:16.457535+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e510: 8 total, 8 up, 6 in 2024-02-12T17:09:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:16.438075+0000 mon.a (mon.0) 1738 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:16.438100+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: Degraded data redundancy: 1318/233868 objects degraded (0.564%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:16.457535+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e510: 8 total, 8 up, 6 in 2024-02-12T17:09:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:16.796165+0000 mgr.y (mgr.24481) 743 : cluster [DBG] pgmap v1021: 148 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 709 op/s; 1362/230190 objects degraded (0.592%); 61811/230190 objects misplaced (26.852%); 8.5 MiB/s, 136 objects/s recovering 2024-02-12T17:09:18.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:17.507530+0000 mon.a (mon.0) 1741 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:16.796165+0000 mgr.y (mgr.24481) 743 : cluster [DBG] pgmap v1021: 148 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 709 op/s; 1362/230190 objects degraded (0.592%); 61811/230190 objects misplaced (26.852%); 8.5 MiB/s, 136 objects/s recovering 2024-02-12T17:09:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:17.507530+0000 mon.a (mon.0) 1741 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:16.796165+0000 mgr.y (mgr.24481) 743 : cluster [DBG] pgmap v1021: 148 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 709 op/s; 1362/230190 objects degraded (0.592%); 61811/230190 objects misplaced (26.852%); 8.5 MiB/s, 136 objects/s recovering 2024-02-12T17:09:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:17.507530+0000 mon.a (mon.0) 1741 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:19.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:18.518365+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e511: 8 total, 8 up, 6 in 2024-02-12T17:09:19.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:18.796833+0000 mgr.y (mgr.24481) 744 : cluster [DBG] pgmap v1023: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.11k op/s; 1362/222543 objects degraded (0.612%); 59375/222543 objects misplaced (26.680%); 18 MiB/s, 280 objects/s recovering 2024-02-12T17:09:19.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:18.518365+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e511: 8 total, 8 up, 6 in 2024-02-12T17:09:19.788 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:18.796833+0000 mgr.y (mgr.24481) 744 : cluster [DBG] pgmap v1023: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.11k op/s; 1362/222543 objects degraded (0.612%); 59375/222543 objects misplaced (26.680%); 18 MiB/s, 280 objects/s recovering 2024-02-12T17:09:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:18.518365+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e511: 8 total, 8 up, 6 in 2024-02-12T17:09:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:18.796833+0000 mgr.y (mgr.24481) 744 : cluster [DBG] pgmap v1023: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.11k op/s; 1362/222543 objects degraded (0.612%); 59375/222543 objects misplaced (26.680%); 18 MiB/s, 280 objects/s recovering 2024-02-12T17:09:20.037 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:20.051 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:20.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:20.186 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:20.327 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:09:20.117+0000 7fd96a9de700 -1 mon.a@0(leader).osd e512 definitely_dead 0 2024-02-12T17:09:20.533 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:20.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:20.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:20.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T17:09:20.582 INFO:teuthology.orchestra.run.smithi033.stderr:marked in osd.6. 2024-02-12T17:09:20.602 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:09:20.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:19.250173+0000 osd.5 (osd.5) 260 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:20.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:19.250217+0000 osd.5 (osd.5) 261 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:20.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:19.530267+0000 mon.a (mon.0) 1743 : cluster [DBG] osdmap e512: 8 total, 8 up, 6 in 2024-02-12T17:09:20.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:20.118296+0000 mon.b (mon.1) 97 : audit [DBG] from='client.? v1:172.21.15.33:0/46620462' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T17:09:20.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:20.118851+0000 mon.a (mon.0) 1744 : audit [DBG] from='client.25735 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T17:09:20.615 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:20.262541+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? v1:172.21.15.33:0/4167253233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:09:20.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:19.250173+0000 osd.5 (osd.5) 260 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:20.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:19.250217+0000 osd.5 (osd.5) 261 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:20.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:19.530267+0000 mon.a (mon.0) 1743 : cluster [DBG] osdmap e512: 8 total, 8 up, 6 in 2024-02-12T17:09:20.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:20.118296+0000 mon.b (mon.1) 97 : audit [DBG] from='client.? v1:172.21.15.33:0/46620462' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T17:09:20.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:20.118851+0000 mon.a (mon.0) 1744 : audit [DBG] from='client.25735 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T17:09:20.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:20.262541+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? v1:172.21.15.33:0/4167253233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:09:20.642 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:20.716 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:20.915 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:19.250173+0000 osd.5 (osd.5) 260 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:19.250217+0000 osd.5 (osd.5) 261 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:19.530267+0000 mon.a (mon.0) 1743 : cluster [DBG] osdmap e512: 8 total, 8 up, 6 in 2024-02-12T17:09:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:20.118296+0000 mon.b (mon.1) 97 : audit [DBG] from='client.? v1:172.21.15.33:0/46620462' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T17:09:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:20.118851+0000 mon.a (mon.0) 1744 : audit [DBG] from='client.25735 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T17:09:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:20.262541+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? v1:172.21.15.33:0/4167253233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:09:21.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:09:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:20] "GET /metrics HTTP/1.1" 200 176662 "" "Prometheus/2.33.4" 2024-02-12T17:09:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:20.526438+0000 mon.a (mon.0) 1746 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:20.559010+0000 mon.a (mon.0) 1747 : audit [INF] from='client.25735 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-12T17:09:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:20.559087+0000 mon.a (mon.0) 1748 : audit [INF] from='client.? v1:172.21.15.33:0/4167253233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:09:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:20.559126+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-02-12T17:09:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:20.797354+0000 mgr.y (mgr.24481) 745 : cluster [DBG] pgmap v1026: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 670 op/s; 1410/221469 objects degraded (0.637%); 59211/221469 objects misplaced (26.736%); 11 MiB/s, 176 objects/s recovering 2024-02-12T17:09:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.439527+0000 mon.a (mon.0) 1750 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.439562+0000 mon.a (mon.0) 1751 : cluster [WRN] Health check update: Degraded data redundancy: 1362/222543 objects degraded (0.612%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:21.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.444888+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-02-12T17:09:22.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:20.526438+0000 mon.a (mon.0) 1746 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:20.559010+0000 mon.a (mon.0) 1747 : audit [INF] from='client.25735 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:20.559087+0000 mon.a (mon.0) 1748 : audit [INF] from='client.? v1:172.21.15.33:0/4167253233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:20.559126+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:20.797354+0000 mgr.y (mgr.24481) 745 : cluster [DBG] pgmap v1026: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 670 op/s; 1410/221469 objects degraded (0.637%); 59211/221469 objects misplaced (26.736%); 11 MiB/s, 176 objects/s recovering 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.439527+0000 mon.a (mon.0) 1750 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.439562+0000 mon.a (mon.0) 1751 : cluster [WRN] Health check update: Degraded data redundancy: 1362/222543 objects degraded (0.612%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.444888+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:20.526438+0000 mon.a (mon.0) 1746 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:20.559010+0000 mon.a (mon.0) 1747 : audit [INF] from='client.25735 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:20.559087+0000 mon.a (mon.0) 1748 : audit [INF] from='client.? v1:172.21.15.33:0/4167253233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:09:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:20.559126+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-02-12T17:09:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:20.797354+0000 mgr.y (mgr.24481) 745 : cluster [DBG] pgmap v1026: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 16 active+remapped+backfill_wait, 120 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 670 op/s; 1410/221469 objects degraded (0.637%); 59211/221469 objects misplaced (26.736%); 11 MiB/s, 176 objects/s recovering 2024-02-12T17:09:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.439527+0000 mon.a (mon.0) 1750 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.439562+0000 mon.a (mon.0) 1751 : cluster [WRN] Health check update: Degraded data redundancy: 1362/222543 objects degraded (0.612%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.444888+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-02-12T17:09:22.275 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:22.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:22.380 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:22.408 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:22.517 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:22.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:22.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:22.650 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:09:22.650 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.445926+0000 osd.7 (osd.7) 259 : cluster [INF] 10.1c continuing backfill to osd.1 from (471'6510,512'10569] MIN to 512'10569 2024-02-12T17:09:22.650 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.446011+0000 osd.7 (osd.7) 260 : cluster [DBG] 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 512'10569 2024-02-12T17:09:22.650 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.449866+0000 osd.2 (osd.2) 193 : cluster [INF] 10.0 continuing backfill to osd.1 from (466'4200,512'8941] MIN to 512'8941 2024-02-12T17:09:22.650 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.449937+0000 osd.2 (osd.2) 194 : cluster [DBG] 10.0 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8941 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.452243+0000 osd.2 (osd.2) 195 : cluster [INF] 5.5 continuing backfill to osd.1 from (406'13,410'15] MIN to 410'15 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.452302+0000 osd.2 (osd.2) 196 : cluster [DBG] 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 410'15 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.452867+0000 osd.2 (osd.2) 197 : cluster [INF] 10.1d continuing backfill to osd.1 from (472'6765,512'10622] MIN to 512'10622 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.452917+0000 osd.7 (osd.7) 261 : cluster [DBG] 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 512'10567 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.452932+0000 osd.2 (osd.2) 198 : cluster [DBG] 10.1d starting backfill to osd.5 from (472'6765,512'10621] MIN to 512'10622 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.452953+0000 osd.2 (osd.2) 199 : cluster [INF] 10.b continuing backfill to osd.1 from (472'6570,512'10488] MIN to 512'10488 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.453003+0000 osd.2 (osd.2) 200 : cluster [DBG] 10.b starting backfill to osd.6 from (0'0,0'0] MAX to 512'10488 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.454246+0000 osd.5 (osd.5) 262 : cluster [INF] 10.12 continuing backfill to osd.2 from (466'4574,512'9210] 10:4892c73c:::benchmark_data_smithi033_160181_object71352:head to 512'9210 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.454303+0000 osd.5 (osd.5) 263 : cluster [DBG] 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9210 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.454608+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.f starting backfill to osd.6 from (0'0,0'0] MAX to 493'23 2024-02-12T17:09:22.651 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.454895+0000 osd.2 (osd.2) 201 : cluster [INF] 10.1a continuing backfill to osd.0 from (471'6512,512'10466] MIN to 512'10466 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.454966+0000 osd.2 (osd.2) 202 : cluster [DBG] 10.1a starting backfill to osd.6 from (0'0,0'0] MAX to 512'10466 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.459216+0000 osd.3 (osd.3) 188 : cluster [INF] 10.2 continuing backfill to osd.1 from (466'4100,512'8984] MIN to 512'8984 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.459273+0000 osd.3 (osd.3) 189 : cluster [DBG] 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8984 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.466283+0000 osd.5 (osd.5) 265 : cluster [INF] 10.16 continuing backfill to osd.1 from (466'4608,512'9344] 10:6e51e11c:::benchmark_data_smithi033_160181_object108652:head to 512'9344 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.466363+0000 osd.5 (osd.5) 266 : cluster [DBG] 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9344 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.466605+0000 osd.2 (osd.2) 203 : cluster [DBG] 10.1d starting backfill to osd.6 from (0'0,0'0] MAX to 512'10622 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.474522+0000 osd.3 (osd.3) 190 : cluster [DBG] 10.f starting backfill to osd.6 from (0'0,0'0] MAX to 512'10536 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.500320+0000 osd.3 (osd.3) 191 : cluster [INF] 10.17 continuing backfill to osd.1 from (467'4735,512'9208] 10:e8174cb0:::benchmark_data_smithi033_160181_object91942:head to 512'9208 2024-02-12T17:09:22.652 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:21.506242+0000 osd.3 (osd.3) 192 : cluster [DBG] 10.17 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9208 2024-02-12T17:09:22.653 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.446055+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-02-12T17:09:22.653 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:22.728 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:22.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:22.840 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:22.919 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.445926+0000 osd.7 (osd.7) 259 : cluster [INF] 10.1c continuing backfill to osd.1 from (471'6510,512'10569] MIN to 512'10569 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.446011+0000 osd.7 (osd.7) 260 : cluster [DBG] 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 512'10569 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.449866+0000 osd.2 (osd.2) 193 : cluster [INF] 10.0 continuing backfill to osd.1 from (466'4200,512'8941] MIN to 512'8941 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.449937+0000 osd.2 (osd.2) 194 : cluster [DBG] 10.0 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8941 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.452243+0000 osd.2 (osd.2) 195 : cluster [INF] 5.5 continuing backfill to osd.1 from (406'13,410'15] MIN to 410'15 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.452302+0000 osd.2 (osd.2) 196 : cluster [DBG] 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 410'15 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.452867+0000 osd.2 (osd.2) 197 : cluster [INF] 10.1d continuing backfill to osd.1 from (472'6765,512'10622] MIN to 512'10622 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.452917+0000 osd.7 (osd.7) 261 : cluster [DBG] 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 512'10567 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.452932+0000 osd.2 (osd.2) 198 : cluster [DBG] 10.1d starting backfill to osd.5 from (472'6765,512'10621] MIN to 512'10622 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.452953+0000 osd.2 (osd.2) 199 : cluster [INF] 10.b continuing backfill to osd.1 from (472'6570,512'10488] MIN to 512'10488 2024-02-12T17:09:22.920 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.453003+0000 osd.2 (osd.2) 200 : cluster [DBG] 10.b starting backfill to osd.6 from (0'0,0'0] MAX to 512'10488 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.454246+0000 osd.5 (osd.5) 262 : cluster [INF] 10.12 continuing backfill to osd.2 from (466'4574,512'9210] 10:4892c73c:::benchmark_data_smithi033_160181_object71352:head to 512'9210 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.454303+0000 osd.5 (osd.5) 263 : cluster [DBG] 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9210 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.454608+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.f starting backfill to osd.6 from (0'0,0'0] MAX to 493'23 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.454895+0000 osd.2 (osd.2) 201 : cluster [INF] 10.1a continuing backfill to osd.0 from (471'6512,512'10466] MIN to 512'10466 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.454966+0000 osd.2 (osd.2) 202 : cluster [DBG] 10.1a starting backfill to osd.6 from (0'0,0'0] MAX to 512'10466 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.459216+0000 osd.3 (osd.3) 188 : cluster [INF] 10.2 continuing backfill to osd.1 from (466'4100,512'8984] MIN to 512'8984 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.459273+0000 osd.3 (osd.3) 189 : cluster [DBG] 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8984 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.466283+0000 osd.5 (osd.5) 265 : cluster [INF] 10.16 continuing backfill to osd.1 from (466'4608,512'9344] 10:6e51e11c:::benchmark_data_smithi033_160181_object108652:head to 512'9344 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.466363+0000 osd.5 (osd.5) 266 : cluster [DBG] 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9344 2024-02-12T17:09:22.921 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.466605+0000 osd.2 (osd.2) 203 : cluster [DBG] 10.1d starting backfill to osd.6 from (0'0,0'0] MAX to 512'10622 2024-02-12T17:09:22.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.474522+0000 osd.3 (osd.3) 190 : cluster [DBG] 10.f starting backfill to osd.6 from (0'0,0'0] MAX to 512'10536 2024-02-12T17:09:22.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.500320+0000 osd.3 (osd.3) 191 : cluster [INF] 10.17 continuing backfill to osd.1 from (467'4735,512'9208] 10:e8174cb0:::benchmark_data_smithi033_160181_object91942:head to 512'9208 2024-02-12T17:09:22.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:21.506242+0000 osd.3 (osd.3) 192 : cluster [DBG] 10.17 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9208 2024-02-12T17:09:22.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.446055+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-02-12T17:09:22.922 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.445926+0000 osd.7 (osd.7) 259 : cluster [INF] 10.1c continuing backfill to osd.1 from (471'6510,512'10569] MIN to 512'10569 2024-02-12T17:09:22.922 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.446011+0000 osd.7 (osd.7) 260 : cluster [DBG] 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 512'10569 2024-02-12T17:09:22.922 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.449866+0000 osd.2 (osd.2) 193 : cluster [INF] 10.0 continuing backfill to osd.1 from (466'4200,512'8941] MIN to 512'8941 2024-02-12T17:09:22.922 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.449937+0000 osd.2 (osd.2) 194 : cluster [DBG] 10.0 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8941 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.452243+0000 osd.2 (osd.2) 195 : cluster [INF] 5.5 continuing backfill to osd.1 from (406'13,410'15] MIN to 410'15 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.452302+0000 osd.2 (osd.2) 196 : cluster [DBG] 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 410'15 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.452867+0000 osd.2 (osd.2) 197 : cluster [INF] 10.1d continuing backfill to osd.1 from (472'6765,512'10622] MIN to 512'10622 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.452917+0000 osd.7 (osd.7) 261 : cluster [DBG] 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 512'10567 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.452932+0000 osd.2 (osd.2) 198 : cluster [DBG] 10.1d starting backfill to osd.5 from (472'6765,512'10621] MIN to 512'10622 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.452953+0000 osd.2 (osd.2) 199 : cluster [INF] 10.b continuing backfill to osd.1 from (472'6570,512'10488] MIN to 512'10488 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.453003+0000 osd.2 (osd.2) 200 : cluster [DBG] 10.b starting backfill to osd.6 from (0'0,0'0] MAX to 512'10488 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.454246+0000 osd.5 (osd.5) 262 : cluster [INF] 10.12 continuing backfill to osd.2 from (466'4574,512'9210] 10:4892c73c:::benchmark_data_smithi033_160181_object71352:head to 512'9210 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.454303+0000 osd.5 (osd.5) 263 : cluster [DBG] 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9210 2024-02-12T17:09:22.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.454608+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.f starting backfill to osd.6 from (0'0,0'0] MAX to 493'23 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.454895+0000 osd.2 (osd.2) 201 : cluster [INF] 10.1a continuing backfill to osd.0 from (471'6512,512'10466] MIN to 512'10466 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.454966+0000 osd.2 (osd.2) 202 : cluster [DBG] 10.1a starting backfill to osd.6 from (0'0,0'0] MAX to 512'10466 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.459216+0000 osd.3 (osd.3) 188 : cluster [INF] 10.2 continuing backfill to osd.1 from (466'4100,512'8984] MIN to 512'8984 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.459273+0000 osd.3 (osd.3) 189 : cluster [DBG] 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8984 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.466283+0000 osd.5 (osd.5) 265 : cluster [INF] 10.16 continuing backfill to osd.1 from (466'4608,512'9344] 10:6e51e11c:::benchmark_data_smithi033_160181_object108652:head to 512'9344 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.466363+0000 osd.5 (osd.5) 266 : cluster [DBG] 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9344 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.466605+0000 osd.2 (osd.2) 203 : cluster [DBG] 10.1d starting backfill to osd.6 from (0'0,0'0] MAX to 512'10622 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.474522+0000 osd.3 (osd.3) 190 : cluster [DBG] 10.f starting backfill to osd.6 from (0'0,0'0] MAX to 512'10536 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.500320+0000 osd.3 (osd.3) 191 : cluster [INF] 10.17 continuing backfill to osd.1 from (467'4735,512'9208] 10:e8174cb0:::benchmark_data_smithi033_160181_object91942:head to 512'9208 2024-02-12T17:09:22.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:21.506242+0000 osd.3 (osd.3) 192 : cluster [DBG] 10.17 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9208 2024-02-12T17:09:22.925 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.446055+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-02-12T17:09:22.932 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:23.355 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:09:23.356 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:09:23.356 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:09:23.356 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:09:23.356 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:09:23.709 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:23.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.444475+0000 osd.7 (osd.7) 262 : cluster [DBG] 10.10 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8999 2024-02-12T17:09:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.445306+0000 osd.7 (osd.7) 263 : cluster [DBG] 5.10 starting backfill to osd.6 from (0'0,0'0] MAX to 430'18 2024-02-12T17:09:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.445801+0000 osd.1 (osd.1) 164 : cluster [DBG] 5.3 starting backfill to osd.6 from (0'0,0'0] MAX to 506'18 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.446065+0000 osd.5 (osd.5) 267 : cluster [DBG] 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9460 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.446563+0000 osd.5 (osd.5) 268 : cluster [DBG] 10.6 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9393 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.448475+0000 osd.7 (osd.7) 264 : cluster [DBG] 5.9 starting backfill to osd.6 from (0'0,0'0] MAX to 505'21 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.449647+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.c starting backfill to osd.6 from (0'0,0'0] MAX to 87'2 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.449715+0000 osd.2 (osd.2) 204 : cluster [DBG] 10.d starting backfill to osd.6 from (0'0,0'0] MAX to 512'10598 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.451750+0000 osd.2 (osd.2) 205 : cluster [DBG] 10.7 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9258 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.451872+0000 osd.2 (osd.2) 206 : cluster [DBG] 10.e starting backfill to osd.6 from (0'0,0'0] MAX to 512'10506 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.461420+0000 osd.2 (osd.2) 207 : cluster [DBG] 10.a starting backfill to osd.6 from (0'0,0'0] MAX to 512'10450 2024-02-12T17:09:23.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:22.808863+0000 mgr.y (mgr.24481) 746 : cluster [DBG] pgmap v1029: 148 pgs: 3 remapped+peering, 13 peering, 1 active+remapped+backfilling, 2 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 267 op/s; 1442/219324 objects degraded (0.657%); 52740/219324 objects misplaced (24.047%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T17:09:23.819 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:23.890 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.444475+0000 osd.7 (osd.7) 262 : cluster [DBG] 10.10 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8999 2024-02-12T17:09:23.890 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.445306+0000 osd.7 (osd.7) 263 : cluster [DBG] 5.10 starting backfill to osd.6 from (0'0,0'0] MAX to 430'18 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.445801+0000 osd.1 (osd.1) 164 : cluster [DBG] 5.3 starting backfill to osd.6 from (0'0,0'0] MAX to 506'18 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.446065+0000 osd.5 (osd.5) 267 : cluster [DBG] 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9460 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.446563+0000 osd.5 (osd.5) 268 : cluster [DBG] 10.6 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9393 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.448475+0000 osd.7 (osd.7) 264 : cluster [DBG] 5.9 starting backfill to osd.6 from (0'0,0'0] MAX to 505'21 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.449647+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.c starting backfill to osd.6 from (0'0,0'0] MAX to 87'2 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.449715+0000 osd.2 (osd.2) 204 : cluster [DBG] 10.d starting backfill to osd.6 from (0'0,0'0] MAX to 512'10598 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.451750+0000 osd.2 (osd.2) 205 : cluster [DBG] 10.7 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9258 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.451872+0000 osd.2 (osd.2) 206 : cluster [DBG] 10.e starting backfill to osd.6 from (0'0,0'0] MAX to 512'10506 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.461420+0000 osd.2 (osd.2) 207 : cluster [DBG] 10.a starting backfill to osd.6 from (0'0,0'0] MAX to 512'10450 2024-02-12T17:09:23.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:22.808863+0000 mgr.y (mgr.24481) 746 : cluster [DBG] pgmap v1029: 148 pgs: 3 remapped+peering, 13 peering, 1 active+remapped+backfilling, 2 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 267 op/s; 1442/219324 objects degraded (0.657%); 52740/219324 objects misplaced (24.047%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T17:09:23.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.444475+0000 osd.7 (osd.7) 262 : cluster [DBG] 10.10 starting backfill to osd.6 from (0'0,0'0] MAX to 512'8999 2024-02-12T17:09:23.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.445306+0000 osd.7 (osd.7) 263 : cluster [DBG] 5.10 starting backfill to osd.6 from (0'0,0'0] MAX to 430'18 2024-02-12T17:09:23.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.445801+0000 osd.1 (osd.1) 164 : cluster [DBG] 5.3 starting backfill to osd.6 from (0'0,0'0] MAX to 506'18 2024-02-12T17:09:23.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.446065+0000 osd.5 (osd.5) 267 : cluster [DBG] 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9460 2024-02-12T17:09:23.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.446563+0000 osd.5 (osd.5) 268 : cluster [DBG] 10.6 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9393 2024-02-12T17:09:23.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.448475+0000 osd.7 (osd.7) 264 : cluster [DBG] 5.9 starting backfill to osd.6 from (0'0,0'0] MAX to 505'21 2024-02-12T17:09:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.449647+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.c starting backfill to osd.6 from (0'0,0'0] MAX to 87'2 2024-02-12T17:09:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.449715+0000 osd.2 (osd.2) 204 : cluster [DBG] 10.d starting backfill to osd.6 from (0'0,0'0] MAX to 512'10598 2024-02-12T17:09:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.451750+0000 osd.2 (osd.2) 205 : cluster [DBG] 10.7 starting backfill to osd.6 from (0'0,0'0] MAX to 512'9258 2024-02-12T17:09:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.451872+0000 osd.2 (osd.2) 206 : cluster [DBG] 10.e starting backfill to osd.6 from (0'0,0'0] MAX to 512'10506 2024-02-12T17:09:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.461420+0000 osd.2 (osd.2) 207 : cluster [DBG] 10.a starting backfill to osd.6 from (0'0,0'0] MAX to 512'10450 2024-02-12T17:09:23.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:22.808863+0000 mgr.y (mgr.24481) 746 : cluster [DBG] pgmap v1029: 148 pgs: 3 remapped+peering, 13 peering, 1 active+remapped+backfilling, 2 active+clean+remapped, 4 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 267 op/s; 1442/219324 objects degraded (0.657%); 52740/219324 objects misplaced (24.047%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T17:09:23.900 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:24.101 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T17:09:24.260 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-02-12T17:09:24.262 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:24.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:24.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:24.431 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:24.433 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:24.599 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:24.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:24.707 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:24.771 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:24.822 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:24.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:24.934 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:09:24.963 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:23.625026+0000 mon.a (mon.0) 1754 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-02-12T17:09:24.963 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:09:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:09:24.966 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:24.997 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:09:25.021 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:09:25.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:23.625026+0000 mon.a (mon.0) 1754 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-02-12T17:09:25.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:23.625026+0000 mon.a (mon.0) 1754 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-02-12T17:09:25.064 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:09:25.105 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:09:25.135 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:09:25.169 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:09:25.217 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:09:25.252 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:09:25.275 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:09:25.275 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:09:25.276 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:09:25.276 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:09:25.276 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:09:25.276 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:09:25.276 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:09:25.276 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:09:25.824 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:09:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:25.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:09:25.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:09:25.933 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:25.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:24.609618+0000 mgr.y (mgr.24481) 747 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:25.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:24.633519+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-02-12T17:09:25.968 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:24.814792+0000 mgr.y (mgr.24481) 748 : cluster [DBG] pgmap v1032: 148 pgs: 3 remapped+peering, 14 peering, 1 active+remapped+backfilling, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 11 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 2 active+remapped, 98 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1019 B/s rd, 0 B/s wr, 569 op/s; 1452/214620 objects degraded (0.677%); 79544/214620 objects misplaced (37.063%); 2.9 MiB/s, 62 objects/s recovering 2024-02-12T17:09:25.969 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:24.831727+0000 mon.c (mon.2) 256 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:09:25.969 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:24.953721+0000 mon.b (mon.1) 98 : audit [DBG] from='client.? v1:172.21.15.33:0/3903404105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:25.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:24.609618+0000 mgr.y (mgr.24481) 747 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:25.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:24.633519+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-02-12T17:09:25.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:24.814792+0000 mgr.y (mgr.24481) 748 : cluster [DBG] pgmap v1032: 148 pgs: 3 remapped+peering, 14 peering, 1 active+remapped+backfilling, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 11 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 2 active+remapped, 98 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1019 B/s rd, 0 B/s wr, 569 op/s; 1452/214620 objects degraded (0.677%); 79544/214620 objects misplaced (37.063%); 2.9 MiB/s, 62 objects/s recovering 2024-02-12T17:09:25.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:24.831727+0000 mon.c (mon.2) 256 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:09:25.969 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:24.953721+0000 mon.b (mon.1) 98 : audit [DBG] from='client.? v1:172.21.15.33:0/3903404105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:24.609618+0000 mgr.y (mgr.24481) 747 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:24.633519+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-02-12T17:09:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:24.814792+0000 mgr.y (mgr.24481) 748 : cluster [DBG] pgmap v1032: 148 pgs: 3 remapped+peering, 14 peering, 1 active+remapped+backfilling, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 11 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 2 active+remapped, 98 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1019 B/s rd, 0 B/s wr, 569 op/s; 1452/214620 objects degraded (0.677%); 79544/214620 objects misplaced (37.063%); 2.9 MiB/s, 62 objects/s recovering 2024-02-12T17:09:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:24.831727+0000 mon.c (mon.2) 256 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:09:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:24.953721+0000 mon.b (mon.1) 98 : audit [DBG] from='client.? v1:172.21.15.33:0/3903404105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:26.009 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:26.377 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:09:26.377 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:09:26.719 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:25.557338+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:26.719 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:25.557410+0000 osd.0 (osd.0) 223 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:26.719 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:25.692804+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-02-12T17:09:26.719 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:25.952367+0000 mon.c (mon.2) 257 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:09:26.719 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:25.954761+0000 mon.c (mon.2) 258 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:09:26.719 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:25.962584+0000 mon.a (mon.0) 1757 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:26.720 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:26.441538+0000 mon.a (mon.0) 1758 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 12 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:26.720 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:26.441586+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: Degraded data redundancy: 1452/214620 objects degraded (0.677%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:27.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:25.557338+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:25.557410+0000 osd.0 (osd.0) 223 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:25.692804+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-02-12T17:09:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:25.952367+0000 mon.c (mon.2) 257 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:09:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:25.954761+0000 mon.c (mon.2) 258 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:09:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:25.962584+0000 mon.a (mon.0) 1757 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:26.441538+0000 mon.a (mon.0) 1758 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 12 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:26.441586+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: Degraded data redundancy: 1452/214620 objects degraded (0.677%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:25.557338+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:25.557410+0000 osd.0 (osd.0) 223 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:25.692804+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-02-12T17:09:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:25.952367+0000 mon.c (mon.2) 257 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:09:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:25.954761+0000 mon.c (mon.2) 258 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:09:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:25.962584+0000 mon.a (mon.0) 1757 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:26.441538+0000 mon.a (mon.0) 1758 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 12 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:26.441586+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: Degraded data redundancy: 1452/214620 objects degraded (0.677%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:27.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T17:09:27.532 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:27.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:26.702655+0000 mon.a (mon.0) 1760 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-02-12T17:09:27.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:26.815368+0000 mgr.y (mgr.24481) 749 : cluster [DBG] pgmap v1035: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 3 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 2 active+remapped, 111 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 519 op/s; 1457/213096 objects degraded (0.684%); 87725/213096 objects misplaced (41.167%); 2.9 MiB/s, 68 objects/s recovering 2024-02-12T17:09:27.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:26.702655+0000 mon.a (mon.0) 1760 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-02-12T17:09:27.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:26.815368+0000 mgr.y (mgr.24481) 749 : cluster [DBG] pgmap v1035: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 3 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 2 active+remapped, 111 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 519 op/s; 1457/213096 objects degraded (0.684%); 87725/213096 objects misplaced (41.167%); 2.9 MiB/s, 68 objects/s recovering 2024-02-12T17:09:27.778 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:09:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw-foo-a[65833]: debug 2024-02-12T17:09:27.514+0000 7efdb04dc700 -1 RGWWatcher::handle_error cookie 94508044033920 err (110) Connection timed out 2024-02-12T17:09:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:26.702655+0000 mon.a (mon.0) 1760 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-02-12T17:09:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:26.815368+0000 mgr.y (mgr.24481) 749 : cluster [DBG] pgmap v1035: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 peering, 3 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 2 active+remapped, 111 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 519 op/s; 1457/213096 objects degraded (0.684%); 87725/213096 objects misplaced (41.167%); 2.9 MiB/s, 68 objects/s recovering 2024-02-12T17:09:28.925 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:28.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:29.030 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:29.129 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:29.180 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:29.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:29.260 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:09:29.260 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:09:29.261 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2024-02-12T17:09:29.261 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:09:29.291 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:29.329 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:09:29.329 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:09:29.330 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:29.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:29.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:29.490 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:29.512 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:29.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:29.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:29.727 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:29.856 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:29.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:29.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:09:29.998 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:28.816128+0000 mgr.y (mgr.24481) 750 : cluster [DBG] pgmap v1036: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 113 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 575 op/s; 1457/206883 objects degraded (0.704%); 84511/206883 objects misplaced (40.850%); 10 MiB/s, 169 objects/s recovering 2024-02-12T17:09:30.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:28.816128+0000 mgr.y (mgr.24481) 750 : cluster [DBG] pgmap v1036: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 113 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 575 op/s; 1457/206883 objects degraded (0.704%); 84511/206883 objects misplaced (40.850%); 10 MiB/s, 169 objects/s recovering 2024-02-12T17:09:30.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:28.816128+0000 mgr.y (mgr.24481) 750 : cluster [DBG] pgmap v1036: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 113 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 575 op/s; 1457/206883 objects degraded (0.704%); 84511/206883 objects misplaced (40.850%); 10 MiB/s, 169 objects/s recovering 2024-02-12T17:09:30.308 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:30.926 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:09:31.150 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:09:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:31.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:09:31.257 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:31.287 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:09:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:30] "GET /metrics HTTP/1.1" 200 176662 "" "Prometheus/2.33.4" 2024-02-12T17:09:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:31.924 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T17:09:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:30.816719+0000 mgr.y (mgr.24481) 751 : cluster [DBG] pgmap v1037: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 113 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 828 B/s rd, 0 B/s wr, 465 op/s; 1477/206010 objects degraded (0.717%); 84375/206010 objects misplaced (40.957%); 8.4 MiB/s, 138 objects/s recovering 2024-02-12T17:09:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:31.443497+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:31.443535+0000 mon.a (mon.0) 1762 : cluster [WRN] Health check update: Degraded data redundancy: 1477/206010 objects degraded (0.717%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:31.456580+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-02-12T17:09:32.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:30.816719+0000 mgr.y (mgr.24481) 751 : cluster [DBG] pgmap v1037: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 113 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 828 B/s rd, 0 B/s wr, 465 op/s; 1477/206010 objects degraded (0.717%); 84375/206010 objects misplaced (40.957%); 8.4 MiB/s, 138 objects/s recovering 2024-02-12T17:09:32.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:31.443497+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:32.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:31.443535+0000 mon.a (mon.0) 1762 : cluster [WRN] Health check update: Degraded data redundancy: 1477/206010 objects degraded (0.717%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:32.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:31.456580+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-02-12T17:09:32.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:30.816719+0000 mgr.y (mgr.24481) 751 : cluster [DBG] pgmap v1037: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 113 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 828 B/s rd, 0 B/s wr, 465 op/s; 1477/206010 objects degraded (0.717%); 84375/206010 objects misplaced (40.957%); 8.4 MiB/s, 138 objects/s recovering 2024-02-12T17:09:32.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:31.443497+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:32.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:31.443535+0000 mon.a (mon.0) 1762 : cluster [WRN] Health check update: Degraded data redundancy: 1477/206010 objects degraded (0.717%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:32.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:31.456580+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-02-12T17:09:32.286 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:32.769 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:32.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:32.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:32.790 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:09:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:09:32.878 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:09:32.916 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:33.130 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:09:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:33.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:09:33.235 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:33.333 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:33.917 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:33.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:32.530552+0000 mon.a (mon.0) 1764 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:33.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:32.818808+0000 mgr.y (mgr.24481) 752 : cluster [DBG] pgmap v1039: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 719 B/s rd, 0 B/s wr, 503 op/s; 1500/203886 objects degraded (0.736%); 83238/203886 objects misplaced (40.826%); 8.8 MiB/s, 144 objects/s recovering 2024-02-12T17:09:33.929 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:32.530552+0000 mon.a (mon.0) 1764 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:33.929 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:32.818808+0000 mgr.y (mgr.24481) 752 : cluster [DBG] pgmap v1039: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 719 B/s rd, 0 B/s wr, 503 op/s; 1500/203886 objects degraded (0.736%); 83238/203886 objects misplaced (40.826%); 8.8 MiB/s, 144 objects/s recovering 2024-02-12T17:09:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:33.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:33.940 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:32.530552+0000 mon.a (mon.0) 1764 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:33.940 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:32.818808+0000 mgr.y (mgr.24481) 752 : cluster [DBG] pgmap v1039: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 719 B/s rd, 0 B/s wr, 503 op/s; 1500/203886 objects degraded (0.736%); 83238/203886 objects misplaced (40.826%); 8.8 MiB/s, 144 objects/s recovering 2024-02-12T17:09:34.031 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:34.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:34.252 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:09:34.252 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:09:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:34.712 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:34.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:34.813 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:34.935 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:34.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:09:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:09:35.093 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:35.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:35.199 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:35.286 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:35.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T17:09:35.671 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:35.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:35.781 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:35.887 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:35.917 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:34.630633+0000 mgr.y (mgr.24481) 753 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:35.917 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:34.829245+0000 mgr.y (mgr.24481) 754 : cluster [DBG] pgmap v1040: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 655 op/s; 1550/197148 objects degraded (0.786%); 80014/197148 objects misplaced (40.586%); 13 MiB/s, 208 objects/s recovering 2024-02-12T17:09:35.917 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:35.075562+0000 mon.c (mon.2) 259 : audit [INF] from='client.? v1:172.21.15.33:0/472208813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:35.917 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:35.076245+0000 mon.a (mon.0) 1765 : audit [INF] from='client.35864 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:35.929 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:09:35.943 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:35.958 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:34.630633+0000 mgr.y (mgr.24481) 753 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:35.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:34.829245+0000 mgr.y (mgr.24481) 754 : cluster [DBG] pgmap v1040: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 655 op/s; 1550/197148 objects degraded (0.786%); 80014/197148 objects misplaced (40.586%); 13 MiB/s, 208 objects/s recovering 2024-02-12T17:09:35.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:35.075562+0000 mon.c (mon.2) 259 : audit [INF] from='client.? v1:172.21.15.33:0/472208813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:35.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:35.076245+0000 mon.a (mon.0) 1765 : audit [INF] from='client.35864 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:35.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:34.630633+0000 mgr.y (mgr.24481) 753 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:35.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:34.829245+0000 mgr.y (mgr.24481) 754 : cluster [DBG] pgmap v1040: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 655 op/s; 1550/197148 objects degraded (0.786%); 80014/197148 objects misplaced (40.586%); 13 MiB/s, 208 objects/s recovering 2024-02-12T17:09:35.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:35.075562+0000 mon.c (mon.2) 259 : audit [INF] from='client.? v1:172.21.15.33:0/472208813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:35.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:35.076245+0000 mon.a (mon.0) 1765 : audit [INF] from='client.35864 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:09:36.494 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:36.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:36.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:36.639 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:36.841 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:36.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:36.883 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:09:36.896 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:09:36.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:35.885635+0000 mon.a (mon.0) 1766 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:35.894321+0000 mon.a (mon.0) 1767 : audit [INF] from='client.35864 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:09:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:35.894388+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-02-12T17:09:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:36.446663+0000 mon.a (mon.0) 1769 : cluster [WRN] Health check update: Degraded data redundancy: 1550/197148 objects degraded (0.786%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:36.466245+0000 mon.a (mon.0) 1770 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-02-12T17:09:37.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:35.885635+0000 mon.a (mon.0) 1766 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:37.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:35.894321+0000 mon.a (mon.0) 1767 : audit [INF] from='client.35864 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:09:37.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:35.894388+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-02-12T17:09:37.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:36.446663+0000 mon.a (mon.0) 1769 : cluster [WRN] Health check update: Degraded data redundancy: 1550/197148 objects degraded (0.786%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:37.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:36.466245+0000 mon.a (mon.0) 1770 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-02-12T17:09:37.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:35.885635+0000 mon.a (mon.0) 1766 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:37.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:35.894321+0000 mon.a (mon.0) 1767 : audit [INF] from='client.35864 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:09:37.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:35.894388+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-02-12T17:09:37.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:36.446663+0000 mon.a (mon.0) 1769 : cluster [WRN] Health check update: Degraded data redundancy: 1550/197148 objects degraded (0.786%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:37.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:36.466245+0000 mon.a (mon.0) 1770 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-02-12T17:09:37.636 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:37.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:37.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:09:37.780 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:37.862 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:09:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:37.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:09:37.903 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:09:37.903 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:09:37.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:38.033 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:36.829826+0000 mgr.y (mgr.24481) 755 : cluster [DBG] pgmap v1043: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 604 op/s; 1585/195123 objects degraded (0.812%); 78973/195123 objects misplaced (40.473%); 12 MiB/s, 185 objects/s recovering 2024-02-12T17:09:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:36.859982+0000 mgr.y (mgr.24481) 756 : audit [DBG] from='client.15918 v1:172.21.15.33:0/2600654560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:09:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:36.829826+0000 mgr.y (mgr.24481) 755 : cluster [DBG] pgmap v1043: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 604 op/s; 1585/195123 objects degraded (0.812%); 78973/195123 objects misplaced (40.473%); 12 MiB/s, 185 objects/s recovering 2024-02-12T17:09:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:36.859982+0000 mgr.y (mgr.24481) 756 : audit [DBG] from='client.15918 v1:172.21.15.33:0/2600654560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:09:38.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:36.829826+0000 mgr.y (mgr.24481) 755 : cluster [DBG] pgmap v1043: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 4.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 604 op/s; 1585/195123 objects degraded (0.812%); 78973/195123 objects misplaced (40.473%); 12 MiB/s, 185 objects/s recovering 2024-02-12T17:09:38.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:36.859982+0000 mgr.y (mgr.24481) 756 : audit [DBG] from='client.15918 v1:172.21.15.33:0/2600654560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:09:39.309 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T17:09:39.318 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:39.319 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:39.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:39.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:39.453 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:39.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:39.586 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:39.663 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:38.276688+0000 mon.a (mon.0) 1771 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:39.664 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:38.276688+0000 mon.a (mon.0) 1771 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:39.666 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:39.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:38.276688+0000 mon.a (mon.0) 1771 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:09:39.892 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:39.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:39.999 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:40.053 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:40.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:40.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:40.190 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:38.830743+0000 mgr.y (mgr.24481) 757 : cluster [DBG] pgmap v1044: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 663 op/s; 1585/189237 objects degraded (0.838%); 75892/189237 objects misplaced (40.104%); 14 MiB/s, 227 objects/s recovering 2024-02-12T17:09:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:39.290779+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-02-12T17:09:40.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:38.830743+0000 mgr.y (mgr.24481) 757 : cluster [DBG] pgmap v1044: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 663 op/s; 1585/189237 objects degraded (0.838%); 75892/189237 objects misplaced (40.104%); 14 MiB/s, 227 objects/s recovering 2024-02-12T17:09:40.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:39.290779+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-02-12T17:09:40.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:38.830743+0000 mgr.y (mgr.24481) 757 : cluster [DBG] pgmap v1044: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 663 op/s; 1585/189237 objects degraded (0.838%); 75892/189237 objects misplaced (40.104%); 14 MiB/s, 227 objects/s recovering 2024-02-12T17:09:40.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:39.290779+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-02-12T17:09:41.141 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:41.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:41.246 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:41.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:09:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:40] "GET /metrics HTTP/1.1" 200 176670 "" "Prometheus/2.33.4" 2024-02-12T17:09:41.347 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:09:41.400 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:41.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:40.291420+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-02-12T17:09:41.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:40.291420+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-02-12T17:09:41.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:40.291420+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-02-12T17:09:42.422 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:40.836621+0000 mgr.y (mgr.24481) 758 : cluster [DBG] pgmap v1047: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 590 op/s; 1621/188424 objects degraded (0.860%); 75740/188424 objects misplaced (40.197%); 12 MiB/s, 195 objects/s recovering 2024-02-12T17:09:42.422 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:41.451653+0000 mon.a (mon.0) 1774 : cluster [WRN] Health check update: Degraded data redundancy: 1621/188424 objects degraded (0.860%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:42.423 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:40.836621+0000 mgr.y (mgr.24481) 758 : cluster [DBG] pgmap v1047: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 590 op/s; 1621/188424 objects degraded (0.860%); 75740/188424 objects misplaced (40.197%); 12 MiB/s, 195 objects/s recovering 2024-02-12T17:09:42.423 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:41.451653+0000 mon.a (mon.0) 1774 : cluster [WRN] Health check update: Degraded data redundancy: 1621/188424 objects degraded (0.860%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:42.439 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1043,"stamp":"2024-02-12T17:09:36.829421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4249884348,"num_objects":65041,"num_object_clones":0,"num_object_copies":195123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1585,"num_objects_misplaced":78973,"num_objects_unfound":0,"num_objects_dirty":65041,"num_whiteouts":0,"num_read":1612,"num_read_kb":1364,"num_write":178429,"num_write_kb":7638378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23778,"num_bytes_recovered":1512065494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180089,"ondisk_log_size":180089,"up":444,"acting":442,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":508,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19345476,"kb_used_data":13955404,"kb_used_omap":0,"kb_used_meta":5389824,"kb_avail":636870588,"statfs":{"total":671965249536,"available":652155482112,"internally_reserved":0,"allocated":14290333696,"data_stored":14282478872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5519179776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-237830144,"num_objects":-3629,"num_object_clones":0,"num_object_copies":-10887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":108,"num_objects_misplaced":-5402,"num_objects_unfound":0,"num_objects_dirty":-3629,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":73007104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.013140"},"pg_stats":[{"pgid":"10.23","version":"520'9251","reported_seq":9351,"reported_epoch":520,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:33.293105+0000","last_change":"2024-02-12T17:09:05.525399+0000","last_active":"2024-02-12T17:09:33.293105+0000","last_peered":"2024-02-12T17:09:33.293105+0000","last_clean":"2024-02-12T17:09:01.563324+0000","last_became_active":"2024-02-12T17:09:05.519654+0000","last_became_peered":"2024-02-12T17:09:05.519654+0000","last_unstale":"2024-02-12T17:09:33.293105+0000","last_undegraded":"2024-02-12T17:09:33.293105+0000","last_fullsized":"2024-02-12T17:09:33.293105+0000","mapping_epoch":507,"log_start":"466'4400","ondisk_log_start":"466'4400","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4851,"ondisk_log_size":4851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66748416,"num_objects":1019,"num_object_clones":0,"num_object_copies":3057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2038,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":124096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":1019}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.22","version":"520'9045","reported_seq":9144,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.061300+0000","last_change":"2024-02-12T17:09:05.503764+0000","last_active":"2024-02-12T17:09:33.061300+0000","last_peered":"2024-02-12T17:09:33.061300+0000","last_clean":"2024-02-12T17:08:11.820762+0000","last_became_active":"2024-02-12T17:09:05.493219+0000","last_became_peered":"2024-02-12T17:09:05.493219+0000","last_unstale":"2024-02-12T17:09:33.061300+0000","last_undegraded":"2024-02-12T17:09:33.061300+0000","last_fullsized":"2024-02-12T17:09:33.061300+0000","mapping_epoch":507,"log_start":"466'4100","ondisk_log_start":"466'4100","created":503,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4945,"ondisk_log_size":4945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68370432,"num_objects":1043,"num_object_clones":0,"num_object_copies":3129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1043,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2772,"num_write_kb":122096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,3],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":1043}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"520'9332","reported_seq":9483,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:33.613559+0000","last_change":"2024-02-12T17:09:04.471058+0000","last_active":"2024-02-12T17:09:33.613559+0000","last_peered":"2024-02-12T17:09:33.613559+0000","last_clean":"2024-02-12T17:09:33.613559+0000","last_became_active":"2024-02-12T17:09:04.466720+0000","last_became_peered":"2024-02-12T17:09:04.466720+0000","last_unstale":"2024-02-12T17:09:33.613559+0000","last_undegraded":"2024-02-12T17:09:33.613559+0000","last_fullsized":"2024-02-12T17:09:33.613559+0000","mapping_epoch":506,"log_start":"466'4500","ondisk_log_start":"466'4500","created":503,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4832,"ondisk_log_size":4832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67256320,"num_objects":1026,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2888,"num_write_kb":125264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":688128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.20","version":"520'9046","reported_seq":9153,"reported_epoch":520,"state":"active+remapped","last_fresh":"2024-02-12T17:09:33.220539+0000","last_change":"2024-02-12T17:09:21.517594+0000","last_active":"2024-02-12T17:09:33.220539+0000","last_peered":"2024-02-12T17:09:33.220539+0000","last_clean":"2024-02-12T17:08:11.791648+0000","last_became_active":"2024-02-12T17:09:04.504467+0000","last_became_peered":"2024-02-12T17:09:04.504467+0000","last_unstale":"2024-02-12T17:09:33.220539+0000","last_undegraded":"2024-02-12T17:09:33.220539+0000","last_fullsized":"2024-02-12T17:09:33.220539+0000","mapping_epoch":506,"log_start":"466'4200","ondisk_log_start":"466'4200","created":503,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4846,"ondisk_log_size":4846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63504384,"num_objects":969,"num_object_clones":0,"num_object_copies":2907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1938,"num_objects_unfound":0,"num_objects_dirty":969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2812,"num_write_kb":120992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[2,7,5],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":969}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.17","version":"521'9456","reported_seq":10019,"reported_epoch":521,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:36.177662+0000","last_change":"2024-02-12T17:09:24.023618+0000","last_active":"2024-02-12T17:09:36.177662+0000","last_peered":"2024-02-12T17:09:36.177662+0000","last_clean":"2024-02-12T17:07:53.019897+0000","last_became_active":"2024-02-12T17:09:21.615745+0000","last_became_peered":"2024-02-12T17:09:21.615745+0000","last_unstale":"2024-02-12T17:09:36.177662+0000","last_undegraded":"2024-02-12T17:09:36.177662+0000","last_fullsized":"2024-02-12T17:09:36.177662+0000","mapping_epoch":513,"log_start":"468'4792","ondisk_log_start":"468'4792","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4664,"ondisk_log_size":4664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":132644864,"num_objects":2024,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4027,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":245504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":14483456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,2,5],"avail_no_missing":["3","2","5","7"],"object_location_counts":[{"shards":"2,3,5","objects":2024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":231,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699317+0000","last_change":"2024-02-12T17:09:21.483703+0000","last_active":"2024-02-12T17:09:26.699317+0000","last_peered":"2024-02-12T17:09:26.699317+0000","last_clean":"2024-02-12T17:09:26.699317+0000","last_became_active":"2024-02-12T17:09:21.482442+0000","last_became_peered":"2024-02-12T17:09:21.482442+0000","last_unstale":"2024-02-12T17:09:26.699317+0000","last_undegraded":"2024-02-12T17:09:26.699317+0000","last_fullsized":"2024-02-12T17:09:26.699317+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:44.261302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:44.261302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'27","reported_seq":203,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522495+0000","last_change":"2024-02-12T17:07:35.884990+0000","last_active":"2024-02-12T17:09:31.522495+0000","last_peered":"2024-02-12T17:09:31.522495+0000","last_clean":"2024-02-12T17:09:31.522495+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:09:31.522495+0000","last_undegraded":"2024-02-12T17:09:31.522495+0000","last_fullsized":"2024-02-12T17:09:31.522495+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"520'9555","reported_seq":21393,"reported_epoch":520,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:33.279909+0000","last_change":"2024-02-12T17:09:21.617022+0000","last_active":"2024-02-12T17:09:33.279909+0000","last_peered":"2024-02-12T17:09:33.279909+0000","last_clean":"2024-02-12T17:08:11.807156+0000","last_became_active":"2024-02-12T17:09:21.599750+0000","last_became_peered":"2024-02-12T17:09:21.599750+0000","last_unstale":"2024-02-12T17:09:33.279909+0000","last_undegraded":"2024-02-12T17:09:33.279909+0000","last_fullsized":"2024-02-12T17:09:33.279909+0000","mapping_epoch":513,"log_start":"466'4608","ondisk_log_start":"466'4608","created":480,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4947,"ondisk_log_size":4947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":136380416,"num_objects":2081,"num_object_clones":0,"num_object_copies":6243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2601,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5703,"num_write_kb":249088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5853,"num_bytes_recovered":383582208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[5,2,3],"avail_no_missing":["5","2","3","7"],"object_location_counts":[{"shards":"2,3,5","objects":2081}],"blocked_by":[],"up_primary":6,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":187,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523259+0000","last_change":"2024-02-12T17:07:37.890052+0000","last_active":"2024-02-12T17:09:31.523259+0000","last_peered":"2024-02-12T17:09:31.523259+0000","last_clean":"2024-02-12T17:09:31.523259+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:09:31.523259+0000","last_undegraded":"2024-02-12T17:09:31.523259+0000","last_fullsized":"2024-02-12T17:09:31.523259+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:37.889956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.889956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"315'19","reported_seq":26,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.701235+0000","last_change":"2024-02-12T17:09:22.989811+0000","last_active":"2024-02-12T17:09:26.701235+0000","last_peered":"2024-02-12T17:09:26.701235+0000","last_clean":"2024-02-12T17:09:26.701235+0000","last_became_active":"2024-02-12T17:09:21.527626+0000","last_became_peered":"2024-02-12T17:09:21.527626+0000","last_unstale":"2024-02-12T17:09:26.701235+0000","last_undegraded":"2024-02-12T17:09:26.701235+0000","last_fullsized":"2024-02-12T17:09:26.701235+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:07:47.845930+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.845930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"521'9388","reported_seq":11466,"reported_epoch":521,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-12T17:09:36.204529+0000","last_change":"2024-02-12T17:08:16.657336+0000","last_active":"2024-02-12T17:09:36.204529+0000","last_peered":"2024-02-12T17:09:36.204529+0000","last_clean":"2024-02-12T17:07:53.075624+0000","last_became_active":"2024-02-12T17:08:13.019808+0000","last_became_peered":"2024-02-12T17:08:13.019808+0000","last_unstale":"2024-02-12T17:09:36.204529+0000","last_undegraded":"2024-02-12T17:08:13.003591+0000","last_fullsized":"2024-02-12T17:08:12.830836+0000","mapping_epoch":491,"log_start":"467'4613","ondisk_log_start":"467'4613","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"470'5845","last_scrub_stamp":"2024-02-12T17:07:27.706391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:07:27.706391+0000","log_size":4775,"ondisk_log_size":4775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":132579328,"num_objects":2023,"num_object_clones":0,"num_object_copies":6069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":774,"num_objects_misplaced":4726,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":244416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":64552960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,2],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":1976},{"shards":"2,3","objects":-727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":528,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.700944+0000","last_change":"2024-02-12T17:09:21.483158+0000","last_active":"2024-02-12T17:09:26.700944+0000","last_peered":"2024-02-12T17:09:26.700944+0000","last_clean":"2024-02-12T17:09:26.700944+0000","last_became_active":"2024-02-12T17:09:21.482825+0000","last_became_peered":"2024-02-12T17:09:21.482825+0000","last_unstale":"2024-02-12T17:09:26.700944+0000","last_undegraded":"2024-02-12T17:09:26.700944+0000","last_fullsized":"2024-02-12T17:09:26.700944+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:29.165802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:07:29.165802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"278'7","reported_seq":223,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.527529+0000","last_change":"2024-02-12T17:08:13.170514+0000","last_active":"2024-02-12T17:09:31.527529+0000","last_peered":"2024-02-12T17:09:31.527529+0000","last_clean":"2024-02-12T17:09:31.527529+0000","last_became_active":"2024-02-12T17:08:12.907023+0000","last_became_peered":"2024-02-12T17:08:12.907023+0000","last_unstale":"2024-02-12T17:09:31.527529+0000","last_undegraded":"2024-02-12T17:09:31.527529+0000","last_fullsized":"2024-02-12T17:09:31.527529+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7","last_scrub_stamp":"2024-02-12T17:05:56.388366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.388366+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"521'9462","reported_seq":9961,"reported_epoch":521,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:09:36.236644+0000","last_change":"2024-02-12T17:08:45.670190+0000","last_active":"2024-02-12T17:09:36.236644+0000","last_peered":"2024-02-12T17:09:36.236644+0000","last_clean":"2024-02-12T17:07:52.963448+0000","last_became_active":"2024-02-12T17:08:12.974291+0000","last_became_peered":"2024-02-12T17:08:12.974291+0000","last_unstale":"2024-02-12T17:09:36.236644+0000","last_undegraded":"2024-02-12T17:09:36.236644+0000","last_fullsized":"2024-02-12T17:09:36.236644+0000","mapping_epoch":491,"log_start":"468'4730","ondisk_log_start":"468'4730","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4732,"ondisk_log_size":4732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":131694592,"num_objects":2010,"num_object_clones":0,"num_object_copies":6030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2010,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5668,"num_write_kb":245664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":12713984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":2010}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":493,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.702265+0000","last_change":"2024-02-12T17:07:39.235715+0000","last_active":"2024-02-12T17:09:26.702265+0000","last_peered":"2024-02-12T17:09:26.702265+0000","last_clean":"2024-02-12T17:09:26.702265+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:09:26.702265+0000","last_undegraded":"2024-02-12T17:09:26.702265+0000","last_fullsized":"2024-02-12T17:09:26.702265+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.235654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.235654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"278'15","reported_seq":230,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.526168+0000","last_change":"2024-02-12T17:09:23.653654+0000","last_active":"2024-02-12T17:09:31.526168+0000","last_peered":"2024-02-12T17:09:31.526168+0000","last_clean":"2024-02-12T17:09:31.526168+0000","last_became_active":"2024-02-12T17:09:21.602320+0000","last_became_peered":"2024-02-12T17:09:21.602320+0000","last_unstale":"2024-02-12T17:09:31.526168+0000","last_undegraded":"2024-02-12T17:09:31.526168+0000","last_fullsized":"2024-02-12T17:09:31.526168+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"520'9530","reported_seq":17440,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.281462+0000","last_change":"2024-02-12T17:09:20.609353+0000","last_active":"2024-02-12T17:09:33.281462+0000","last_peered":"2024-02-12T17:09:33.281462+0000","last_clean":"2024-02-12T17:08:11.822202+0000","last_became_active":"2024-02-12T17:08:13.963868+0000","last_became_peered":"2024-02-12T17:08:13.963868+0000","last_unstale":"2024-02-12T17:09:33.281462+0000","last_undegraded":"2024-02-12T17:09:33.281462+0000","last_fullsized":"2024-02-12T17:09:33.281462+0000","mapping_epoch":492,"log_start":"467'4680","ondisk_log_start":"467'4680","created":480,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4850,"ondisk_log_size":4850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":135331840,"num_objects":2065,"num_object_clones":0,"num_object_copies":6195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2065,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":248320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3896,"num_bytes_recovered":255328256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[2,7,3],"avail_no_missing":["2","3","5","7"],"object_location_counts":[{"shards":"2,3,7","objects":2065}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":205,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.527383+0000","last_change":"2024-02-12T17:09:21.507961+0000","last_active":"2024-02-12T17:09:31.527383+0000","last_peered":"2024-02-12T17:09:31.527383+0000","last_clean":"2024-02-12T17:09:31.527383+0000","last_became_active":"2024-02-12T17:09:21.507651+0000","last_became_peered":"2024-02-12T17:09:21.507651+0000","last_unstale":"2024-02-12T17:09:31.527383+0000","last_undegraded":"2024-02-12T17:09:31.527383+0000","last_fullsized":"2024-02-12T17:09:31.527383+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:29.479610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T17:06:29.479610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"278'18","reported_seq":254,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.530191+0000","last_change":"2024-02-12T17:09:22.957944+0000","last_active":"2024-02-12T17:09:31.530191+0000","last_peered":"2024-02-12T17:09:31.530191+0000","last_clean":"2024-02-12T17:09:31.530191+0000","last_became_active":"2024-02-12T17:09:21.492826+0000","last_became_peered":"2024-02-12T17:09:21.492826+0000","last_unstale":"2024-02-12T17:09:31.530191+0000","last_undegraded":"2024-02-12T17:09:31.530191+0000","last_fullsized":"2024-02-12T17:09:31.530191+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"520'9441","reported_seq":10030,"reported_epoch":520,"state":"active+remapped","last_fresh":"2024-02-12T17:09:33.310952+0000","last_change":"2024-02-12T17:09:23.304670+0000","last_active":"2024-02-12T17:09:33.310952+0000","last_peered":"2024-02-12T17:09:33.310952+0000","last_clean":"2024-02-12T17:07:53.037515+0000","last_became_active":"2024-02-12T17:09:21.605641+0000","last_became_peered":"2024-02-12T17:09:21.605641+0000","last_unstale":"2024-02-12T17:09:33.310952+0000","last_undegraded":"2024-02-12T17:09:33.310952+0000","last_fullsized":"2024-02-12T17:09:33.310952+0000","mapping_epoch":513,"log_start":"466'4574","ondisk_log_start":"466'4574","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4867,"ondisk_log_size":4867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128483328,"num_objects":1961,"num_object_clones":0,"num_object_copies":5883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3794,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":244192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":15466496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[5,7,3],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":1961}],"blocked_by":[],"up_primary":6,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":16,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.706330+0000","last_change":"2024-02-12T17:09:21.560647+0000","last_active":"2024-02-12T17:09:26.706330+0000","last_peered":"2024-02-12T17:09:26.706330+0000","last_clean":"2024-02-12T17:09:26.706330+0000","last_became_active":"2024-02-12T17:09:21.559769+0000","last_became_peered":"2024-02-12T17:09:21.559769+0000","last_unstale":"2024-02-12T17:09:26.706330+0000","last_undegraded":"2024-02-12T17:09:26.706330+0000","last_fullsized":"2024-02-12T17:09:26.706330+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:52.828449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:06:52.828449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"278'27","reported_seq":54,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.712886+0000","last_change":"2024-02-12T17:08:17.496274+0000","last_active":"2024-02-12T17:09:26.712886+0000","last_peered":"2024-02-12T17:09:26.712886+0000","last_clean":"2024-02-12T17:09:26.712886+0000","last_became_active":"2024-02-12T17:08:12.960006+0000","last_became_peered":"2024-02-12T17:08:12.960006+0000","last_unstale":"2024-02-12T17:09:26.712886+0000","last_undegraded":"2024-02-12T17:09:26.712886+0000","last_fullsized":"2024-02-12T17:09:26.712886+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"520'9649","reported_seq":9988,"reported_epoch":520,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:09:33.352476+0000","last_change":"2024-02-12T17:09:23.296534+0000","last_active":"2024-02-12T17:09:33.352476+0000","last_peered":"2024-02-12T17:09:33.352476+0000","last_clean":"2024-02-12T17:09:20.515844+0000","last_became_active":"2024-02-12T17:09:22.563520+0000","last_became_peered":"2024-02-12T17:09:22.563520+0000","last_unstale":"2024-02-12T17:09:33.352476+0000","last_undegraded":"2024-02-12T17:09:33.352476+0000","last_fullsized":"2024-02-12T17:09:33.352476+0000","mapping_epoch":514,"log_start":"468'4902","ondisk_log_start":"468'4902","created":480,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4747,"ondisk_log_size":4747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":136282112,"num_objects":2080,"num_object_clones":0,"num_object_copies":6240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1966,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5776,"num_write_kb":251360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":7667712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":2080}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":191,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522810+0000","last_change":"2024-02-12T17:09:21.496479+0000","last_active":"2024-02-12T17:09:31.522810+0000","last_peered":"2024-02-12T17:09:31.522810+0000","last_clean":"2024-02-12T17:09:31.522810+0000","last_became_active":"2024-02-12T17:09:21.493389+0000","last_became_peered":"2024-02-12T17:09:21.493389+0000","last_unstale":"2024-02-12T17:09:31.522810+0000","last_undegraded":"2024-02-12T17:09:31.522810+0000","last_fullsized":"2024-02-12T17:09:31.522810+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"278'19","reported_seq":28,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.710952+0000","last_change":"2024-02-12T17:09:21.677331+0000","last_active":"2024-02-12T17:09:26.710952+0000","last_peered":"2024-02-12T17:09:26.710952+0000","last_clean":"2024-02-12T17:09:26.710952+0000","last_became_active":"2024-02-12T17:09:21.494312+0000","last_became_peered":"2024-02-12T17:09:21.494312+0000","last_unstale":"2024-02-12T17:09:26.710952+0000","last_undegraded":"2024-02-12T17:09:26.710952+0000","last_fullsized":"2024-02-12T17:09:26.710952+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"520'9208","reported_seq":9312,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.626667+0000","last_change":"2024-02-12T17:09:23.308524+0000","last_active":"2024-02-12T17:09:33.626667+0000","last_peered":"2024-02-12T17:09:33.626667+0000","last_clean":"2024-02-12T17:09:20.542792+0000","last_became_active":"2024-02-12T17:09:22.566916+0000","last_became_peered":"2024-02-12T17:09:22.566916+0000","last_unstale":"2024-02-12T17:09:33.626667+0000","last_undegraded":"2024-02-12T17:09:33.626667+0000","last_fullsized":"2024-02-12T17:09:33.626667+0000","mapping_epoch":514,"log_start":"466'4200","ondisk_log_start":"466'4200","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5008,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133496832,"num_objects":2037,"num_object_clones":0,"num_object_copies":6111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2036,"num_objects_unfound":0,"num_objects_dirty":2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5487,"num_write_kb":240768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,5,2],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":2037}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":518,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699834+0000","last_change":"2024-02-12T17:09:21.507645+0000","last_active":"2024-02-12T17:09:26.699834+0000","last_peered":"2024-02-12T17:09:26.699834+0000","last_clean":"2024-02-12T17:09:26.699834+0000","last_became_active":"2024-02-12T17:09:21.507203+0000","last_became_peered":"2024-02-12T17:09:21.507203+0000","last_unstale":"2024-02-12T17:09:26.699834+0000","last_undegraded":"2024-02-12T17:09:26.699834+0000","last_fullsized":"2024-02-12T17:09:26.699834+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:55.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.027189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"278'20","reported_seq":561,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.698874+0000","last_change":"2024-02-12T17:08:13.337827+0000","last_active":"2024-02-12T17:09:26.698874+0000","last_peered":"2024-02-12T17:09:26.698874+0000","last_clean":"2024-02-12T17:09:26.698874+0000","last_became_active":"2024-02-12T17:08:12.867632+0000","last_became_peered":"2024-02-12T17:08:12.867632+0000","last_unstale":"2024-02-12T17:09:26.698874+0000","last_undegraded":"2024-02-12T17:09:26.698874+0000","last_fullsized":"2024-02-12T17:09:26.698874+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"521'10824","reported_seq":10933,"reported_epoch":521,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:36.213347+0000","last_change":"2024-02-12T17:09:04.639028+0000","last_active":"2024-02-12T17:09:36.213347+0000","last_peered":"2024-02-12T17:09:36.213347+0000","last_clean":"2024-02-12T17:08:11.824735+0000","last_became_active":"2024-02-12T17:09:04.521899+0000","last_became_peered":"2024-02-12T17:09:04.521899+0000","last_unstale":"2024-02-12T17:09:36.213347+0000","last_undegraded":"2024-02-12T17:09:36.213347+0000","last_fullsized":"2024-02-12T17:09:36.213347+0000","mapping_epoch":506,"log_start":"472'6748","ondisk_log_start":"472'6748","created":503,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4076,"ondisk_log_size":4076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":126615552,"num_objects":1932,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3864,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5745,"num_write_kb":245664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[{"shards":"2,3,5","objects":1932}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":189,"reported_epoch":519,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:26.696638+0000","last_change":"2024-02-12T17:09:22.566214+0000","last_active":"2024-02-12T17:09:26.696638+0000","last_peered":"2024-02-12T17:09:26.696638+0000","last_clean":"2024-02-12T17:09:19.578664+0000","last_became_active":"2024-02-12T17:09:22.563146+0000","last_became_peered":"2024-02-12T17:09:22.563146+0000","last_unstale":"2024-02-12T17:09:26.696638+0000","last_undegraded":"2024-02-12T17:09:26.696638+0000","last_fullsized":"2024-02-12T17:09:26.696638+0000","mapping_epoch":514,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,5,2],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":205,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.527602+0000","last_change":"2024-02-12T17:09:21.547851+0000","last_active":"2024-02-12T17:09:31.527602+0000","last_peered":"2024-02-12T17:09:31.527602+0000","last_clean":"2024-02-12T17:09:31.527602+0000","last_became_active":"2024-02-12T17:09:21.547628+0000","last_became_peered":"2024-02-12T17:09:21.547628+0000","last_unstale":"2024-02-12T17:09:31.527602+0000","last_undegraded":"2024-02-12T17:09:31.527602+0000","last_fullsized":"2024-02-12T17:09:31.527602+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.711027+0000","last_change":"2024-02-12T17:09:21.594631+0000","last_active":"2024-02-12T17:09:26.711027+0000","last_peered":"2024-02-12T17:09:26.711027+0000","last_clean":"2024-02-12T17:09:26.711027+0000","last_became_active":"2024-02-12T17:09:21.537322+0000","last_became_peered":"2024-02-12T17:09:21.537322+0000","last_unstale":"2024-02-12T17:09:26.711027+0000","last_undegraded":"2024-02-12T17:09:26.711027+0000","last_fullsized":"2024-02-12T17:09:26.711027+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:31.071549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:31.071549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"278'19","reported_seq":215,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522643+0000","last_change":"2024-02-12T17:09:21.993424+0000","last_active":"2024-02-12T17:09:31.522643+0000","last_peered":"2024-02-12T17:09:31.522643+0000","last_clean":"2024-02-12T17:09:31.522643+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:09:31.522643+0000","last_undegraded":"2024-02-12T17:09:31.522643+0000","last_fullsized":"2024-02-12T17:09:31.522643+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"520'10704","reported_seq":10800,"reported_epoch":520,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:33.264917+0000","last_change":"2024-02-12T17:09:05.538087+0000","last_active":"2024-02-12T17:09:33.264917+0000","last_peered":"2024-02-12T17:09:33.264917+0000","last_clean":"2024-02-12T17:09:02.447395+0000","last_became_active":"2024-02-12T17:09:05.527486+0000","last_became_peered":"2024-02-12T17:09:05.527486+0000","last_unstale":"2024-02-12T17:09:33.264917+0000","last_undegraded":"2024-02-12T17:09:33.264917+0000","last_fullsized":"2024-02-12T17:09:33.264917+0000","mapping_epoch":507,"log_start":"472'6607","ondisk_log_start":"472'6607","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4097,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":130056192,"num_objects":1985,"num_object_clones":0,"num_object_copies":5955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1985,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5635,"num_write_kb":243808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":1985}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":16,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.702188+0000","last_change":"2024-02-12T17:09:21.526918+0000","last_active":"2024-02-12T17:09:26.702188+0000","last_peered":"2024-02-12T17:09:26.702188+0000","last_clean":"2024-02-12T17:09:26.702188+0000","last_became_active":"2024-02-12T17:09:21.526617+0000","last_became_peered":"2024-02-12T17:09:21.526617+0000","last_unstale":"2024-02-12T17:09:26.702188+0000","last_undegraded":"2024-02-12T17:09:26.702188+0000","last_fullsized":"2024-02-12T17:09:26.702188+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":491,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.696555+0000","last_change":"2024-02-12T17:06:03.911480+0000","last_active":"2024-02-12T17:09:26.696555+0000","last_peered":"2024-02-12T17:09:26.696555+0000","last_clean":"2024-02-12T17:09:26.696555+0000","last_became_active":"2024-02-12T17:04:39.886041+0000","last_became_peered":"2024-02-12T17:04:39.886041+0000","last_unstale":"2024-02-12T17:09:26.696555+0000","last_undegraded":"2024-02-12T17:09:26.696555+0000","last_fullsized":"2024-02-12T17:09:26.696555+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.911375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.911375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":536,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.700445+0000","last_change":"2024-02-12T17:09:21.508557+0000","last_active":"2024-02-12T17:09:26.700445+0000","last_peered":"2024-02-12T17:09:26.700445+0000","last_clean":"2024-02-12T17:09:26.700445+0000","last_became_active":"2024-02-12T17:09:21.508243+0000","last_became_peered":"2024-02-12T17:09:21.508243+0000","last_unstale":"2024-02-12T17:09:26.700445+0000","last_undegraded":"2024-02-12T17:09:26.700445+0000","last_fullsized":"2024-02-12T17:09:26.700445+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:34.981898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.981898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"301'10","reported_seq":238,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523107+0000","last_change":"2024-02-12T17:09:23.293065+0000","last_active":"2024-02-12T17:09:31.523107+0000","last_peered":"2024-02-12T17:09:31.523107+0000","last_clean":"2024-02-12T17:09:31.523107+0000","last_became_active":"2024-02-12T17:09:21.514886+0000","last_became_peered":"2024-02-12T17:09:21.514886+0000","last_unstale":"2024-02-12T17:09:31.523107+0000","last_undegraded":"2024-02-12T17:09:31.523107+0000","last_fullsized":"2024-02-12T17:09:31.523107+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1d","version":"520'10851","reported_seq":10951,"reported_epoch":520,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:33.367875+0000","last_change":"2024-02-12T17:09:21.609474+0000","last_active":"2024-02-12T17:09:33.367875+0000","last_peered":"2024-02-12T17:09:33.367875+0000","last_clean":"2024-02-12T17:09:01.601556+0000","last_became_active":"2024-02-12T17:09:21.587026+0000","last_became_peered":"2024-02-12T17:09:21.587026+0000","last_unstale":"2024-02-12T17:09:33.367875+0000","last_undegraded":"2024-02-12T17:09:33.367875+0000","last_fullsized":"2024-02-12T17:09:33.367875+0000","mapping_epoch":513,"log_start":"472'6765","ondisk_log_start":"472'6765","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4086,"ondisk_log_size":4086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":134512652,"num_objects":2053,"num_object_clones":0,"num_object_copies":6159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6159,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5740,"num_write_kb":249345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[2,3,7],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,3,7","objects":2053}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.713989+0000","last_change":"2024-02-12T17:08:12.961296+0000","last_active":"2024-02-12T17:09:26.713989+0000","last_peered":"2024-02-12T17:09:26.713989+0000","last_clean":"2024-02-12T17:09:26.713989+0000","last_became_active":"2024-02-12T17:08:12.960364+0000","last_became_peered":"2024-02-12T17:08:12.960364+0000","last_unstale":"2024-02-12T17:09:26.713989+0000","last_undegraded":"2024-02-12T17:09:26.713989+0000","last_fullsized":"2024-02-12T17:09:26.713989+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":187,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522542+0000","last_change":"2024-02-12T17:07:36.839451+0000","last_active":"2024-02-12T17:09:31.522542+0000","last_peered":"2024-02-12T17:09:31.522542+0000","last_clean":"2024-02-12T17:09:31.522542+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:09:31.522542+0000","last_undegraded":"2024-02-12T17:09:31.522542+0000","last_fullsized":"2024-02-12T17:09:31.522542+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":47,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.712181+0000","last_change":"2024-02-12T17:09:21.475619+0000","last_active":"2024-02-12T17:09:26.712181+0000","last_peered":"2024-02-12T17:09:26.712181+0000","last_clean":"2024-02-12T17:09:26.712181+0000","last_became_active":"2024-02-12T17:09:21.475307+0000","last_became_peered":"2024-02-12T17:09:21.475307+0000","last_unstale":"2024-02-12T17:09:26.712181+0000","last_undegraded":"2024-02-12T17:09:26.712181+0000","last_fullsized":"2024-02-12T17:09:26.712181+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"278'19","reported_seq":28,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.705698+0000","last_change":"2024-02-12T17:09:21.628556+0000","last_active":"2024-02-12T17:09:26.705698+0000","last_peered":"2024-02-12T17:09:26.705698+0000","last_clean":"2024-02-12T17:09:26.705698+0000","last_became_active":"2024-02-12T17:09:21.504986+0000","last_became_peered":"2024-02-12T17:09:21.504986+0000","last_unstale":"2024-02-12T17:09:26.705698+0000","last_undegraded":"2024-02-12T17:09:26.705698+0000","last_fullsized":"2024-02-12T17:09:26.705698+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:38.751057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.751057+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"520'10776","reported_seq":10895,"reported_epoch":520,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:33.408737+0000","last_change":"2024-02-12T17:09:21.561496+0000","last_active":"2024-02-12T17:09:33.408737+0000","last_peered":"2024-02-12T17:09:33.408737+0000","last_clean":"2024-02-12T17:08:11.822229+0000","last_became_active":"2024-02-12T17:09:21.527381+0000","last_became_peered":"2024-02-12T17:09:21.527381+0000","last_unstale":"2024-02-12T17:09:33.408737+0000","last_undegraded":"2024-02-12T17:09:33.408737+0000","last_fullsized":"2024-02-12T17:09:33.408737+0000","mapping_epoch":513,"log_start":"471'6510","ondisk_log_start":"471'6510","created":503,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4266,"ondisk_log_size":4266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4100,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5677,"num_write_kb":247264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[7,2,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":2050}],"blocked_by":[],"up_primary":6,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":550,"reported_epoch":520,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:09:34.651447+0000","last_change":"2024-02-12T17:09:34.651447+0000","last_active":"2024-02-12T17:09:34.651447+0000","last_peered":"2024-02-12T17:09:34.651447+0000","last_clean":"2024-02-12T17:08:07.414375+0000","last_became_active":"2024-02-12T17:08:13.981281+0000","last_became_peered":"2024-02-12T17:08:13.981281+0000","last_unstale":"2024-02-12T17:09:34.651447+0000","last_undegraded":"2024-02-12T17:09:34.651447+0000","last_fullsized":"2024-02-12T17:09:34.651447+0000","mapping_epoch":492,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":452,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699899+0000","last_change":"2024-02-12T17:09:21.471748+0000","last_active":"2024-02-12T17:09:26.699899+0000","last_peered":"2024-02-12T17:09:26.699899+0000","last_clean":"2024-02-12T17:09:26.699899+0000","last_became_active":"2024-02-12T17:09:21.471472+0000","last_became_peered":"2024-02-12T17:09:21.471472+0000","last_unstale":"2024-02-12T17:09:26.699899+0000","last_undegraded":"2024-02-12T17:09:26.699899+0000","last_fullsized":"2024-02-12T17:09:26.699899+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":16,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.709895+0000","last_change":"2024-02-12T17:09:21.525331+0000","last_active":"2024-02-12T17:09:26.709895+0000","last_peered":"2024-02-12T17:09:26.709895+0000","last_clean":"2024-02-12T17:09:26.709895+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:09:26.709895+0000","last_undegraded":"2024-02-12T17:09:26.709895+0000","last_fullsized":"2024-02-12T17:09:26.709895+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"278'19","reported_seq":67,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.714212+0000","last_change":"2024-02-12T17:09:21.997990+0000","last_active":"2024-02-12T17:09:26.714212+0000","last_peered":"2024-02-12T17:09:26.714212+0000","last_clean":"2024-02-12T17:09:26.714212+0000","last_became_active":"2024-02-12T17:09:21.494080+0000","last_became_peered":"2024-02-12T17:09:21.494080+0000","last_unstale":"2024-02-12T17:09:26.714212+0000","last_undegraded":"2024-02-12T17:09:26.714212+0000","last_fullsized":"2024-02-12T17:09:26.714212+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1b","version":"521'10764","reported_seq":10870,"reported_epoch":521,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:36.114067+0000","last_change":"2024-02-12T17:09:24.676436+0000","last_active":"2024-02-12T17:09:36.114067+0000","last_peered":"2024-02-12T17:09:36.114067+0000","last_clean":"2024-02-12T17:08:11.788334+0000","last_became_active":"2024-02-12T17:09:05.499570+0000","last_became_peered":"2024-02-12T17:09:05.499570+0000","last_unstale":"2024-02-12T17:09:36.114067+0000","last_undegraded":"2024-02-12T17:09:36.114067+0000","last_fullsized":"2024-02-12T17:09:36.114067+0000","mapping_epoch":507,"log_start":"472'6570","ondisk_log_start":"472'6570","created":503,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4194,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":130351104,"num_objects":1989,"num_object_clones":0,"num_object_copies":5967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1983,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5698,"num_write_kb":245984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":533,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:32.521173+0000","last_change":"2024-02-12T17:05:55.414790+0000","last_active":"2024-02-12T17:09:32.521173+0000","last_peered":"2024-02-12T17:09:32.521173+0000","last_clean":"2024-02-12T17:09:32.521173+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:09:32.521173+0000","last_undegraded":"2024-02-12T17:09:32.521173+0000","last_fullsized":"2024-02-12T17:09:32.521173+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":275,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.527204+0000","last_change":"2024-02-12T17:06:51.622488+0000","last_active":"2024-02-12T17:09:31.527204+0000","last_peered":"2024-02-12T17:09:31.527204+0000","last_clean":"2024-02-12T17:09:31.527204+0000","last_became_active":"2024-02-12T17:04:39.886343+0000","last_became_peered":"2024-02-12T17:04:39.886343+0000","last_unstale":"2024-02-12T17:09:31.527204+0000","last_undegraded":"2024-02-12T17:09:31.527204+0000","last_fullsized":"2024-02-12T17:09:31.527204+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.622401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.622401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"278'15","reported_seq":596,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.701452+0000","last_change":"2024-02-12T17:07:33.734742+0000","last_active":"2024-02-12T17:09:26.701452+0000","last_peered":"2024-02-12T17:09:26.701452+0000","last_clean":"2024-02-12T17:09:26.701452+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:09:26.701452+0000","last_undegraded":"2024-02-12T17:09:26.701452+0000","last_fullsized":"2024-02-12T17:09:26.701452+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:33.734675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:33.734675+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1a","version":"520'10665","reported_seq":10768,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.399987+0000","last_change":"2024-02-12T17:09:21.611712+0000","last_active":"2024-02-12T17:09:33.399987+0000","last_peered":"2024-02-12T17:09:33.399987+0000","last_clean":"2024-02-12T17:09:01.597818+0000","last_became_active":"2024-02-12T17:09:21.562757+0000","last_became_peered":"2024-02-12T17:09:21.562757+0000","last_unstale":"2024-02-12T17:09:33.399987+0000","last_undegraded":"2024-02-12T17:09:33.399987+0000","last_fullsized":"2024-02-12T17:09:33.399987+0000","mapping_epoch":513,"log_start":"471'6512","ondisk_log_start":"471'6512","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4153,"ondisk_log_size":4153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":130842624,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3994,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":1997}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"505'20","reported_seq":506,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.481537+0000","last_change":"2024-02-12T17:09:01.357341+0000","last_active":"2024-02-12T17:09:31.481537+0000","last_peered":"2024-02-12T17:09:31.481537+0000","last_clean":"2024-02-12T17:09:31.481537+0000","last_became_active":"2024-02-12T17:09:01.356238+0000","last_became_peered":"2024-02-12T17:09:01.356238+0000","last_unstale":"2024-02-12T17:09:31.481537+0000","last_undegraded":"2024-02-12T17:09:31.481537+0000","last_fullsized":"2024-02-12T17:09:31.481537+0000","mapping_epoch":502,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":509,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699809+0000","last_change":"2024-02-12T17:07:35.798822+0000","last_active":"2024-02-12T17:09:26.699809+0000","last_peered":"2024-02-12T17:09:26.699809+0000","last_clean":"2024-02-12T17:09:26.699809+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:09:26.699809+0000","last_undegraded":"2024-02-12T17:09:26.699809+0000","last_fullsized":"2024-02-12T17:09:26.699809+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:35.798720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.798720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"278'20","reported_seq":220,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523015+0000","last_change":"2024-02-12T17:08:14.038528+0000","last_active":"2024-02-12T17:09:31.523015+0000","last_peered":"2024-02-12T17:09:31.523015+0000","last_clean":"2024-02-12T17:09:31.523015+0000","last_became_active":"2024-02-12T17:08:12.934950+0000","last_became_peered":"2024-02-12T17:08:12.934950+0000","last_unstale":"2024-02-12T17:09:31.523015+0000","last_undegraded":"2024-02-12T17:09:31.523015+0000","last_fullsized":"2024-02-12T17:09:31.523015+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"520'9639","reported_seq":26607,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:33.279983+0000","last_change":"2024-02-12T17:09:00.165252+0000","last_active":"2024-02-12T17:09:33.279983+0000","last_peered":"2024-02-12T17:09:33.279983+0000","last_clean":"2024-02-12T17:09:33.279983+0000","last_became_active":"2024-02-12T17:09:00.148036+0000","last_became_peered":"2024-02-12T17:09:00.148036+0000","last_unstale":"2024-02-12T17:09:33.279983+0000","last_undegraded":"2024-02-12T17:09:33.279983+0000","last_fullsized":"2024-02-12T17:09:33.279983+0000","mapping_epoch":501,"log_start":"466'4500","ondisk_log_start":"466'4500","created":480,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5139,"ondisk_log_size":5139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":137265152,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":251296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6295,"num_bytes_recovered":412549120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":513,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.700867+0000","last_change":"2024-02-12T17:09:21.479123+0000","last_active":"2024-02-12T17:09:26.700867+0000","last_peered":"2024-02-12T17:09:26.700867+0000","last_clean":"2024-02-12T17:09:26.700867+0000","last_became_active":"2024-02-12T17:09:21.478824+0000","last_became_peered":"2024-02-12T17:09:21.478824+0000","last_unstale":"2024-02-12T17:09:26.700867+0000","last_undegraded":"2024-02-12T17:09:26.700867+0000","last_fullsized":"2024-02-12T17:09:26.700867+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:37.116609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:06:37.116609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.702055+0000","last_change":"2024-02-12T17:09:21.481965+0000","last_active":"2024-02-12T17:09:26.702055+0000","last_peered":"2024-02-12T17:09:26.702055+0000","last_clean":"2024-02-12T17:09:26.702055+0000","last_became_active":"2024-02-12T17:09:21.481590+0000","last_became_peered":"2024-02-12T17:09:21.481590+0000","last_unstale":"2024-02-12T17:09:26.702055+0000","last_undegraded":"2024-02-12T17:09:26.702055+0000","last_fullsized":"2024-02-12T17:09:26.702055+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:34.780676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:34.780676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"460'15","reported_seq":215,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.528787+0000","last_change":"2024-02-12T17:08:14.414273+0000","last_active":"2024-02-12T17:09:31.528787+0000","last_peered":"2024-02-12T17:09:31.528787+0000","last_clean":"2024-02-12T17:09:31.528787+0000","last_became_active":"2024-02-12T17:08:12.953326+0000","last_became_peered":"2024-02-12T17:08:12.953326+0000","last_unstale":"2024-02-12T17:09:31.528787+0000","last_undegraded":"2024-02-12T17:09:31.528787+0000","last_fullsized":"2024-02-12T17:09:31.528787+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"520'9415","reported_seq":12873,"reported_epoch":520,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-12T17:09:33.801308+0000","last_change":"2024-02-12T17:08:14.089150+0000","last_active":"2024-02-12T17:09:33.801308+0000","last_peered":"2024-02-12T17:09:33.801308+0000","last_clean":"2024-02-12T17:07:53.055073+0000","last_became_active":"2024-02-12T17:08:13.093882+0000","last_became_peered":"2024-02-12T17:08:13.093882+0000","last_unstale":"2024-02-12T17:09:33.801308+0000","last_undegraded":"2024-02-12T17:08:14.033538+0000","last_fullsized":"2024-02-12T17:08:12.833048+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":9415,"ondisk_log_size":9415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128811008,"num_objects":1966,"num_object_clones":0,"num_object_copies":5898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":4841,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":243680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1117,"num_bytes_recovered":73203712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[0,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2","objects":-861},{"shards":"0,2,3,5","objects":1},{"shards":"0,2,3,5,7","objects":1},{"shards":"0,2,7","objects":2014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":161,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.702373+0000","last_change":"2024-02-12T17:07:45.273035+0000","last_active":"2024-02-12T17:09:26.702373+0000","last_peered":"2024-02-12T17:09:26.702373+0000","last_clean":"2024-02-12T17:09:26.702373+0000","last_became_active":"2024-02-12T17:04:39.936538+0000","last_became_peered":"2024-02-12T17:04:39.936538+0000","last_unstale":"2024-02-12T17:09:26.702373+0000","last_undegraded":"2024-02-12T17:09:26.702373+0000","last_fullsized":"2024-02-12T17:09:26.702373+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.272939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.272939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":38,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.709967+0000","last_change":"2024-02-12T17:08:13.026852+0000","last_active":"2024-02-12T17:09:26.709967+0000","last_peered":"2024-02-12T17:09:26.709967+0000","last_clean":"2024-02-12T17:09:26.709967+0000","last_became_active":"2024-02-12T17:08:13.026370+0000","last_became_peered":"2024-02-12T17:08:13.026370+0000","last_unstale":"2024-02-12T17:09:26.709967+0000","last_undegraded":"2024-02-12T17:09:26.709967+0000","last_fullsized":"2024-02-12T17:09:26.709967+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"277'19","reported_seq":197,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523353+0000","last_change":"2024-02-12T17:06:45.502715+0000","last_active":"2024-02-12T17:09:31.523353+0000","last_peered":"2024-02-12T17:09:31.523353+0000","last_clean":"2024-02-12T17:09:31.523353+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:09:31.523353+0000","last_undegraded":"2024-02-12T17:09:31.523353+0000","last_fullsized":"2024-02-12T17:09:31.523353+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"520'9497","reported_seq":9604,"reported_epoch":520,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:33.388707+0000","last_change":"2024-02-12T17:09:23.704948+0000","last_active":"2024-02-12T17:09:33.388707+0000","last_peered":"2024-02-12T17:09:33.388707+0000","last_clean":"2024-02-12T17:09:20.406672+0000","last_became_active":"2024-02-12T17:09:22.575591+0000","last_became_peered":"2024-02-12T17:09:22.575591+0000","last_unstale":"2024-02-12T17:09:33.388707+0000","last_undegraded":"2024-02-12T17:09:33.388707+0000","last_fullsized":"2024-02-12T17:09:33.388707+0000","mapping_epoch":514,"log_start":"467'4721","ondisk_log_start":"467'4721","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4776,"ondisk_log_size":4776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":132579328,"num_objects":2023,"num_object_clones":0,"num_object_copies":6069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2019,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5689,"num_write_kb":246784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[2,3,5],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":2023}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":223,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.527453+0000","last_change":"2024-02-12T17:08:12.916367+0000","last_active":"2024-02-12T17:09:31.527453+0000","last_peered":"2024-02-12T17:09:31.527453+0000","last_clean":"2024-02-12T17:09:31.527453+0000","last_became_active":"2024-02-12T17:08:12.906537+0000","last_became_peered":"2024-02-12T17:08:12.906537+0000","last_unstale":"2024-02-12T17:09:31.527453+0000","last_undegraded":"2024-02-12T17:09:31.527453+0000","last_fullsized":"2024-02-12T17:09:31.527453+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":238,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699412+0000","last_change":"2024-02-12T17:09:21.773533+0000","last_active":"2024-02-12T17:09:26.699412+0000","last_peered":"2024-02-12T17:09:26.699412+0000","last_clean":"2024-02-12T17:09:26.699412+0000","last_became_active":"2024-02-12T17:09:21.482826+0000","last_became_peered":"2024-02-12T17:09:21.482826+0000","last_unstale":"2024-02-12T17:09:26.699412+0000","last_undegraded":"2024-02-12T17:09:26.699412+0000","last_fullsized":"2024-02-12T17:09:26.699412+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:07:32.199469+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:07:32.199469+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"277'18","reported_seq":71,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.714041+0000","last_change":"2024-02-12T17:09:22.002951+0000","last_active":"2024-02-12T17:09:26.714041+0000","last_peered":"2024-02-12T17:09:26.714041+0000","last_clean":"2024-02-12T17:09:26.714041+0000","last_became_active":"2024-02-12T17:09:21.494474+0000","last_became_peered":"2024-02-12T17:09:21.494474+0000","last_unstale":"2024-02-12T17:09:26.714041+0000","last_undegraded":"2024-02-12T17:09:26.714041+0000","last_fullsized":"2024-02-12T17:09:26.714041+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.710234+0000","last_change":"2024-02-12T17:08:12.899966+0000","last_active":"2024-02-12T17:09:26.710234+0000","last_peered":"2024-02-12T17:09:26.710234+0000","last_clean":"2024-02-12T17:09:26.710234+0000","last_became_active":"2024-02-12T17:08:12.899606+0000","last_became_peered":"2024-02-12T17:08:12.899606+0000","last_unstale":"2024-02-12T17:09:26.710234+0000","last_undegraded":"2024-02-12T17:09:26.710234+0000","last_fullsized":"2024-02-12T17:09:26.710234+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"520'10771","reported_seq":10917,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.567245+0000","last_change":"2024-02-12T17:09:24.026264+0000","last_active":"2024-02-12T17:09:33.567245+0000","last_peered":"2024-02-12T17:09:33.567245+0000","last_clean":"2024-02-12T17:08:11.822229+0000","last_became_active":"2024-02-12T17:09:21.550588+0000","last_became_peered":"2024-02-12T17:09:21.550588+0000","last_unstale":"2024-02-12T17:09:33.567245+0000","last_undegraded":"2024-02-12T17:09:33.567245+0000","last_fullsized":"2024-02-12T17:09:33.567245+0000","mapping_epoch":513,"log_start":"471'6510","ondisk_log_start":"471'6510","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4261,"ondisk_log_size":4261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":134676480,"num_objects":2055,"num_object_clones":0,"num_object_copies":6165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2046,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5672,"num_write_kb":247264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,2],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":2055}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":506,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.743046+0000","last_change":"2024-02-12T17:07:23.791961+0000","last_active":"2024-02-12T17:09:26.743046+0000","last_peered":"2024-02-12T17:09:26.743046+0000","last_clean":"2024-02-12T17:09:26.743046+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:09:26.743046+0000","last_undegraded":"2024-02-12T17:09:26.743046+0000","last_fullsized":"2024-02-12T17:09:26.743046+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:23.791870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:23.791870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.712739+0000","last_change":"2024-02-12T17:08:12.910933+0000","last_active":"2024-02-12T17:09:26.712739+0000","last_peered":"2024-02-12T17:09:26.712739+0000","last_clean":"2024-02-12T17:09:26.712739+0000","last_became_active":"2024-02-12T17:08:12.910205+0000","last_became_peered":"2024-02-12T17:08:12.910205+0000","last_unstale":"2024-02-12T17:09:26.712739+0000","last_undegraded":"2024-02-12T17:09:26.712739+0000","last_fullsized":"2024-02-12T17:09:26.712739+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"278'20","reported_seq":50,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.713911+0000","last_change":"2024-02-12T17:08:12.983471+0000","last_active":"2024-02-12T17:09:26.713911+0000","last_peered":"2024-02-12T17:09:26.713911+0000","last_clean":"2024-02-12T17:09:26.713911+0000","last_became_active":"2024-02-12T17:08:12.904961+0000","last_became_peered":"2024-02-12T17:08:12.904961+0000","last_unstale":"2024-02-12T17:09:26.713911+0000","last_undegraded":"2024-02-12T17:09:26.713911+0000","last_fullsized":"2024-02-12T17:09:26.713911+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:04.040555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:04.040555+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"519'21","reported_seq":748,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.482286+0000","last_change":"2024-02-12T17:09:25.723429+0000","last_active":"2024-02-12T17:09:31.482286+0000","last_peered":"2024-02-12T17:09:31.482286+0000","last_clean":"2024-02-12T17:09:31.482286+0000","last_became_active":"2024-02-12T17:09:25.723083+0000","last_became_peered":"2024-02-12T17:09:25.723083+0000","last_unstale":"2024-02-12T17:09:31.482286+0000","last_undegraded":"2024-02-12T17:09:31.482286+0000","last_fullsized":"2024-02-12T17:09:31.482286+0000","mapping_epoch":517,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"520'10636","reported_seq":10732,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.200831+0000","last_change":"2024-02-12T17:09:22.993970+0000","last_active":"2024-02-12T17:09:33.200831+0000","last_peered":"2024-02-12T17:09:33.200831+0000","last_clean":"2024-02-12T17:09:20.471514+0000","last_became_active":"2024-02-12T17:09:22.553336+0000","last_became_peered":"2024-02-12T17:09:22.553336+0000","last_unstale":"2024-02-12T17:09:33.200831+0000","last_undegraded":"2024-02-12T17:09:33.200831+0000","last_fullsized":"2024-02-12T17:09:33.200831+0000","mapping_epoch":514,"log_start":"471'6512","ondisk_log_start":"471'6512","created":451,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4124,"ondisk_log_size":4124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":132743168,"num_objects":2025,"num_object_clones":0,"num_object_copies":6075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":2025}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":218,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522946+0000","last_change":"2024-02-12T17:09:21.515567+0000","last_active":"2024-02-12T17:09:31.522946+0000","last_peered":"2024-02-12T17:09:31.522946+0000","last_clean":"2024-02-12T17:09:31.522946+0000","last_became_active":"2024-02-12T17:09:21.514947+0000","last_became_peered":"2024-02-12T17:09:21.514947+0000","last_unstale":"2024-02-12T17:09:31.522946+0000","last_undegraded":"2024-02-12T17:09:31.522946+0000","last_fullsized":"2024-02-12T17:09:31.522946+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":541,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699984+0000","last_change":"2024-02-12T17:09:21.486130+0000","last_active":"2024-02-12T17:09:26.699984+0000","last_peered":"2024-02-12T17:09:26.699984+0000","last_clean":"2024-02-12T17:09:26.699984+0000","last_became_active":"2024-02-12T17:09:21.485925+0000","last_became_peered":"2024-02-12T17:09:21.485925+0000","last_unstale":"2024-02-12T17:09:26.699984+0000","last_undegraded":"2024-02-12T17:09:26.699984+0000","last_fullsized":"2024-02-12T17:09:26.699984+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"492'56","reported_seq":77,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.476064+0000","last_change":"2024-02-12T17:08:16.636920+0000","last_active":"2024-02-12T17:09:31.476064+0000","last_peered":"2024-02-12T17:09:31.476064+0000","last_clean":"2024-02-12T17:09:31.476064+0000","last_became_active":"2024-02-12T17:08:12.958778+0000","last_became_peered":"2024-02-12T17:08:12.958778+0000","last_unstale":"2024-02-12T17:09:31.476064+0000","last_undegraded":"2024-02-12T17:09:31.476064+0000","last_fullsized":"2024-02-12T17:09:31.476064+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":380,"reported_epoch":519,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:26.722082+0000","last_change":"2024-02-12T17:09:21.506653+0000","last_active":"2024-02-12T17:09:26.722082+0000","last_peered":"2024-02-12T17:09:26.722082+0000","last_clean":"2024-02-12T17:05:12.349531+0000","last_became_active":"2024-02-12T17:09:21.506192+0000","last_became_peered":"2024-02-12T17:09:21.506192+0000","last_unstale":"2024-02-12T17:09:26.722082+0000","last_undegraded":"2024-02-12T17:09:26.722082+0000","last_fullsized":"2024-02-12T17:09:26.722082+0000","mapping_epoch":513,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[2,7,3],"avail_no_missing":["2","1","3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"520'9687","reported_seq":9774,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:33.090634+0000","last_change":"2024-02-12T17:07:52.060549+0000","last_active":"2024-02-12T17:09:33.090634+0000","last_peered":"2024-02-12T17:09:33.090634+0000","last_clean":"2024-02-12T17:09:33.090634+0000","last_became_active":"2024-02-12T17:07:52.058737+0000","last_became_peered":"2024-02-12T17:07:52.058737+0000","last_unstale":"2024-02-12T17:09:33.090634+0000","last_undegraded":"2024-02-12T17:09:33.090634+0000","last_fullsized":"2024-02-12T17:09:33.090634+0000","mapping_epoch":451,"log_start":"468'4914","ondisk_log_start":"468'4914","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4773,"ondisk_log_size":4773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133332992,"num_objects":2034,"num_object_clones":0,"num_object_copies":6102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":250912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":520,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.701145+0000","last_change":"2024-02-12T17:09:21.502830+0000","last_active":"2024-02-12T17:09:26.701145+0000","last_peered":"2024-02-12T17:09:26.701145+0000","last_clean":"2024-02-12T17:09:26.701145+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:09:26.701145+0000","last_undegraded":"2024-02-12T17:09:26.701145+0000","last_fullsized":"2024-02-12T17:09:26.701145+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.001888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.001888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":135,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.535698+0000","last_change":"2024-02-12T17:09:21.558037+0000","last_active":"2024-02-12T17:09:31.535698+0000","last_peered":"2024-02-12T17:09:31.535698+0000","last_clean":"2024-02-12T17:09:31.535698+0000","last_became_active":"2024-02-12T17:09:21.473712+0000","last_became_peered":"2024-02-12T17:09:21.473712+0000","last_unstale":"2024-02-12T17:09:31.535698+0000","last_undegraded":"2024-02-12T17:09:31.535698+0000","last_fullsized":"2024-02-12T17:09:31.535698+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":494,"num_read_kb":489,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":187,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522734+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:09:31.522734+0000","last_peered":"2024-02-12T17:09:31.522734+0000","last_clean":"2024-02-12T17:09:31.522734+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:09:31.522734+0000","last_undegraded":"2024-02-12T17:09:31.522734+0000","last_fullsized":"2024-02-12T17:09:31.522734+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"278'22","reported_seq":50,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.714163+0000","last_change":"2024-02-12T17:08:14.640643+0000","last_active":"2024-02-12T17:09:26.714163+0000","last_peered":"2024-02-12T17:09:26.714163+0000","last_clean":"2024-02-12T17:09:26.714163+0000","last_became_active":"2024-02-12T17:08:12.925243+0000","last_became_peered":"2024-02-12T17:08:12.925243+0000","last_unstale":"2024-02-12T17:09:26.714163+0000","last_undegraded":"2024-02-12T17:09:26.714163+0000","last_fullsized":"2024-02-12T17:09:26.714163+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":439,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.700514+0000","last_change":"2024-02-12T17:06:25.129256+0000","last_active":"2024-02-12T17:09:26.700514+0000","last_peered":"2024-02-12T17:09:26.700514+0000","last_clean":"2024-02-12T17:09:26.700514+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:09:26.700514+0000","last_undegraded":"2024-02-12T17:09:26.700514+0000","last_fullsized":"2024-02-12T17:09:26.700514+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:25.129190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:25.129190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"520'10695","reported_seq":10798,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.220871+0000","last_change":"2024-02-12T17:09:21.629273+0000","last_active":"2024-02-12T17:09:33.220871+0000","last_peered":"2024-02-12T17:09:33.220871+0000","last_clean":"2024-02-12T17:08:11.788334+0000","last_became_active":"2024-02-12T17:09:21.608772+0000","last_became_peered":"2024-02-12T17:09:21.608772+0000","last_unstale":"2024-02-12T17:09:33.220871+0000","last_undegraded":"2024-02-12T17:09:33.220871+0000","last_fullsized":"2024-02-12T17:09:33.220871+0000","mapping_epoch":513,"log_start":"472'6570","ondisk_log_start":"472'6570","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4125,"ondisk_log_size":4125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":134873088,"num_objects":2058,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4116,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5629,"num_write_kb":245984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[2,7,3],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,3,7","objects":2058}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":548,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.702619+0000","last_change":"2024-02-12T17:08:13.013398+0000","last_active":"2024-02-12T17:09:26.702619+0000","last_peered":"2024-02-12T17:09:26.702619+0000","last_clean":"2024-02-12T17:09:26.702619+0000","last_became_active":"2024-02-12T17:08:13.012162+0000","last_became_peered":"2024-02-12T17:08:13.012162+0000","last_unstale":"2024-02-12T17:09:26.702619+0000","last_undegraded":"2024-02-12T17:09:26.702619+0000","last_fullsized":"2024-02-12T17:09:26.702619+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.058086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.058086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":214,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523687+0000","last_change":"2024-02-12T17:09:21.602904+0000","last_active":"2024-02-12T17:09:31.523687+0000","last_peered":"2024-02-12T17:09:31.523687+0000","last_clean":"2024-02-12T17:09:31.523687+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:09:31.523687+0000","last_undegraded":"2024-02-12T17:09:31.523687+0000","last_fullsized":"2024-02-12T17:09:31.523687+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:54.311805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:05:54.311805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":491,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.700077+0000","last_change":"2024-02-12T17:05:56.054036+0000","last_active":"2024-02-12T17:09:26.700077+0000","last_peered":"2024-02-12T17:09:26.700077+0000","last_clean":"2024-02-12T17:09:26.700077+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:09:26.700077+0000","last_undegraded":"2024-02-12T17:09:26.700077+0000","last_fullsized":"2024-02-12T17:09:26.700077+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:56.053971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.053971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":541,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.696326+0000","last_change":"2024-02-12T17:09:21.470407+0000","last_active":"2024-02-12T17:09:26.696326+0000","last_peered":"2024-02-12T17:09:26.696326+0000","last_clean":"2024-02-12T17:09:26.696326+0000","last_became_active":"2024-02-12T17:09:21.470133+0000","last_became_peered":"2024-02-12T17:09:21.470133+0000","last_unstale":"2024-02-12T17:09:26.696326+0000","last_undegraded":"2024-02-12T17:09:26.696326+0000","last_fullsized":"2024-02-12T17:09:26.696326+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"462'33","reported_seq":32,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.710470+0000","last_change":"2024-02-12T17:09:21.709538+0000","last_active":"2024-02-12T17:09:26.710470+0000","last_peered":"2024-02-12T17:09:26.710470+0000","last_clean":"2024-02-12T17:09:26.710470+0000","last_became_active":"2024-02-12T17:09:21.585762+0000","last_became_peered":"2024-02-12T17:09:21.585762+0000","last_unstale":"2024-02-12T17:09:26.710470+0000","last_undegraded":"2024-02-12T17:09:26.710470+0000","last_fullsized":"2024-02-12T17:09:26.710470+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'33","last_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.886342+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":516,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.702419+0000","last_change":"2024-02-12T17:05:40.432925+0000","last_active":"2024-02-12T17:09:26.702419+0000","last_peered":"2024-02-12T17:09:26.702419+0000","last_clean":"2024-02-12T17:09:26.702419+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:09:26.702419+0000","last_undegraded":"2024-02-12T17:09:26.702419+0000","last_fullsized":"2024-02-12T17:09:26.702419+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"520'9324","reported_seq":9413,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:33.850812+0000","last_change":"2024-02-12T17:07:52.089095+0000","last_active":"2024-02-12T17:09:33.850812+0000","last_peered":"2024-02-12T17:09:33.850812+0000","last_clean":"2024-02-12T17:09:33.850812+0000","last_became_active":"2024-02-12T17:07:52.081482+0000","last_became_peered":"2024-02-12T17:07:52.081482+0000","last_unstale":"2024-02-12T17:09:33.850812+0000","last_undegraded":"2024-02-12T17:09:33.850812+0000","last_fullsized":"2024-02-12T17:09:33.850812+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":9324,"ondisk_log_size":9324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":135299072,"num_objects":2064,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":243936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":38,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.712508+0000","last_change":"2024-02-12T17:08:12.961975+0000","last_active":"2024-02-12T17:09:26.712508+0000","last_peered":"2024-02-12T17:09:26.712508+0000","last_clean":"2024-02-12T17:09:26.712508+0000","last_became_active":"2024-02-12T17:08:12.959495+0000","last_became_peered":"2024-02-12T17:08:12.959495+0000","last_unstale":"2024-02-12T17:09:26.712508+0000","last_undegraded":"2024-02-12T17:09:26.712508+0000","last_fullsized":"2024-02-12T17:09:26.712508+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":275,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522345+0000","last_change":"2024-02-12T17:06:46.537733+0000","last_active":"2024-02-12T17:09:31.522345+0000","last_peered":"2024-02-12T17:09:31.522345+0000","last_clean":"2024-02-12T17:09:31.522345+0000","last_became_active":"2024-02-12T17:04:39.892122+0000","last_became_peered":"2024-02-12T17:04:39.892122+0000","last_unstale":"2024-02-12T17:09:31.522345+0000","last_undegraded":"2024-02-12T17:09:31.522345+0000","last_fullsized":"2024-02-12T17:09:31.522345+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.537630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.537630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":534,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.696158+0000","last_change":"2024-02-12T17:08:13.787214+0000","last_active":"2024-02-12T17:09:26.696158+0000","last_peered":"2024-02-12T17:09:26.696158+0000","last_clean":"2024-02-12T17:09:26.696158+0000","last_became_active":"2024-02-12T17:08:13.020495+0000","last_became_peered":"2024-02-12T17:08:13.020495+0000","last_unstale":"2024-02-12T17:09:26.696158+0000","last_undegraded":"2024-02-12T17:09:26.696158+0000","last_fullsized":"2024-02-12T17:09:26.696158+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"278'24","reported_seq":52,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.713261+0000","last_change":"2024-02-12T17:08:14.030482+0000","last_active":"2024-02-12T17:09:26.713261+0000","last_peered":"2024-02-12T17:09:26.713261+0000","last_clean":"2024-02-12T17:09:26.713261+0000","last_became_active":"2024-02-12T17:08:12.896922+0000","last_became_peered":"2024-02-12T17:08:12.896922+0000","last_unstale":"2024-02-12T17:09:26.713261+0000","last_undegraded":"2024-02-12T17:09:26.713261+0000","last_fullsized":"2024-02-12T17:09:26.713261+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":527,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.702550+0000","last_change":"2024-02-12T17:08:13.013232+0000","last_active":"2024-02-12T17:09:26.702550+0000","last_peered":"2024-02-12T17:09:26.702550+0000","last_clean":"2024-02-12T17:09:26.702550+0000","last_became_active":"2024-02-12T17:08:13.011766+0000","last_became_peered":"2024-02-12T17:08:13.011766+0000","last_unstale":"2024-02-12T17:09:26.702550+0000","last_undegraded":"2024-02-12T17:09:26.702550+0000","last_fullsized":"2024-02-12T17:09:26.702550+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"520'10799","reported_seq":10889,"reported_epoch":520,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:33.307171+0000","last_change":"2024-02-12T17:09:22.617634+0000","last_active":"2024-02-12T17:09:33.307171+0000","last_peered":"2024-02-12T17:09:33.307171+0000","last_clean":"2024-02-12T17:09:20.440256+0000","last_became_active":"2024-02-12T17:09:22.587759+0000","last_became_peered":"2024-02-12T17:09:22.587759+0000","last_unstale":"2024-02-12T17:09:33.307171+0000","last_undegraded":"2024-02-12T17:09:33.307171+0000","last_fullsized":"2024-02-12T17:09:33.307171+0000","mapping_epoch":514,"log_start":"472'6765","ondisk_log_start":"472'6765","created":451,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4034,"ondisk_log_size":4034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":137920524,"num_objects":2105,"num_object_clones":0,"num_object_copies":6315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2105,"num_objects_unfound":0,"num_objects_dirty":2105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":249344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[2,3,7],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,3,7","objects":2105}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":47,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.713804+0000","last_change":"2024-02-12T17:09:21.470376+0000","last_active":"2024-02-12T17:09:26.713804+0000","last_peered":"2024-02-12T17:09:26.713804+0000","last_clean":"2024-02-12T17:09:26.713804+0000","last_became_active":"2024-02-12T17:09:21.470073+0000","last_became_peered":"2024-02-12T17:09:21.470073+0000","last_unstale":"2024-02-12T17:09:26.713804+0000","last_undegraded":"2024-02-12T17:09:26.713804+0000","last_fullsized":"2024-02-12T17:09:26.713804+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":529,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.701304+0000","last_change":"2024-02-12T17:09:21.483837+0000","last_active":"2024-02-12T17:09:26.701304+0000","last_peered":"2024-02-12T17:09:26.701304+0000","last_clean":"2024-02-12T17:09:26.701304+0000","last_became_active":"2024-02-12T17:09:21.483493+0000","last_became_peered":"2024-02-12T17:09:21.483493+0000","last_unstale":"2024-02-12T17:09:26.701304+0000","last_undegraded":"2024-02-12T17:09:26.701304+0000","last_fullsized":"2024-02-12T17:09:26.701304+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.961097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.961097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"278'29","reported_seq":530,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.700239+0000","last_change":"2024-02-12T17:08:14.718393+0000","last_active":"2024-02-12T17:09:26.700239+0000","last_peered":"2024-02-12T17:09:26.700239+0000","last_clean":"2024-02-12T17:09:26.700239+0000","last_became_active":"2024-02-12T17:08:12.963639+0000","last_became_peered":"2024-02-12T17:08:12.963639+0000","last_unstale":"2024-02-12T17:09:26.700239+0000","last_undegraded":"2024-02-12T17:09:26.700239+0000","last_fullsized":"2024-02-12T17:09:26.700239+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.706407+0000","last_change":"2024-02-12T17:09:21.560337+0000","last_active":"2024-02-12T17:09:26.706407+0000","last_peered":"2024-02-12T17:09:26.706407+0000","last_clean":"2024-02-12T17:09:26.706407+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:09:26.706407+0000","last_undegraded":"2024-02-12T17:09:26.706407+0000","last_fullsized":"2024-02-12T17:09:26.706407+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:46.526746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:46.526746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.e","version":"520'10692","reported_seq":13347,"reported_epoch":520,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:09:33.442815+0000","last_change":"2024-02-12T17:09:23.893083+0000","last_active":"2024-02-12T17:09:33.442815+0000","last_peered":"2024-02-12T17:09:33.442815+0000","last_clean":"2024-02-12T17:09:20.465860+0000","last_became_active":"2024-02-12T17:09:22.563857+0000","last_became_peered":"2024-02-12T17:09:22.563857+0000","last_unstale":"2024-02-12T17:09:33.442815+0000","last_undegraded":"2024-02-12T17:09:33.442815+0000","last_fullsized":"2024-02-12T17:09:33.442815+0000","mapping_epoch":514,"log_start":"472'6607","ondisk_log_start":"472'6607","created":451,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4085,"ondisk_log_size":4085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":130842624,"num_objects":1996,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":778,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5622,"num_write_kb":243808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1280,"num_bytes_recovered":83886080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":1996}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":205,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523746+0000","last_change":"2024-02-12T17:09:21.602676+0000","last_active":"2024-02-12T17:09:31.523746+0000","last_peered":"2024-02-12T17:09:31.523746+0000","last_clean":"2024-02-12T17:09:31.523746+0000","last_became_active":"2024-02-12T17:09:21.601469+0000","last_became_peered":"2024-02-12T17:09:21.601469+0000","last_unstale":"2024-02-12T17:09:31.523746+0000","last_undegraded":"2024-02-12T17:09:31.523746+0000","last_fullsized":"2024-02-12T17:09:31.523746+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:01.455969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.455969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.713139+0000","last_change":"2024-02-12T17:09:21.495350+0000","last_active":"2024-02-12T17:09:26.713139+0000","last_peered":"2024-02-12T17:09:26.713139+0000","last_clean":"2024-02-12T17:09:26.713139+0000","last_became_active":"2024-02-12T17:09:21.495024+0000","last_became_peered":"2024-02-12T17:09:21.495024+0000","last_unstale":"2024-02-12T17:09:26.713139+0000","last_undegraded":"2024-02-12T17:09:26.713139+0000","last_fullsized":"2024-02-12T17:09:26.713139+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"278'30","reported_seq":227,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523196+0000","last_change":"2024-02-12T17:08:15.668029+0000","last_active":"2024-02-12T17:09:31.523196+0000","last_peered":"2024-02-12T17:09:31.523196+0000","last_clean":"2024-02-12T17:09:31.523196+0000","last_became_active":"2024-02-12T17:08:13.006137+0000","last_became_peered":"2024-02-12T17:08:13.006137+0000","last_unstale":"2024-02-12T17:09:31.523196+0000","last_undegraded":"2024-02-12T17:09:31.523196+0000","last_fullsized":"2024-02-12T17:09:31.523196+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":520,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523391+0000","last_change":"2024-02-12T17:05:40.428597+0000","last_active":"2024-02-12T17:09:31.523391+0000","last_peered":"2024-02-12T17:09:31.523391+0000","last_clean":"2024-02-12T17:09:31.523391+0000","last_became_active":"2024-02-12T17:05:40.427915+0000","last_became_peered":"2024-02-12T17:05:40.427915+0000","last_unstale":"2024-02-12T17:09:31.523391+0000","last_undegraded":"2024-02-12T17:09:31.523391+0000","last_fullsized":"2024-02-12T17:09:31.523391+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"521'10789","reported_seq":11139,"reported_epoch":521,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:09:36.204970+0000","last_change":"2024-02-12T17:09:25.372811+0000","last_active":"2024-02-12T17:09:36.204970+0000","last_peered":"2024-02-12T17:09:36.204970+0000","last_clean":"2024-02-12T17:08:11.824735+0000","last_became_active":"2024-02-12T17:09:21.596133+0000","last_became_peered":"2024-02-12T17:09:21.596133+0000","last_unstale":"2024-02-12T17:09:36.204970+0000","last_undegraded":"2024-02-12T17:09:36.204970+0000","last_fullsized":"2024-02-12T17:09:36.204970+0000","mapping_epoch":513,"log_start":"472'6748","ondisk_log_start":"472'6748","created":451,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4041,"ondisk_log_size":4041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128909312,"num_objects":1967,"num_object_clones":0,"num_object_copies":5901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1853,"num_objects_unfound":0,"num_objects_dirty":1967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5710,"num_write_kb":245664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":7733248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[3,2,5],"avail_no_missing":["3","1","2","5"],"object_location_counts":[{"shards":"2,3,5","objects":1967}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":553,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.697936+0000","last_change":"2024-02-12T17:09:22.252402+0000","last_active":"2024-02-12T17:09:26.697936+0000","last_peered":"2024-02-12T17:09:26.697936+0000","last_clean":"2024-02-12T17:09:26.697936+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:09:26.697936+0000","last_undegraded":"2024-02-12T17:09:26.697936+0000","last_fullsized":"2024-02-12T17:09:26.697936+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:07:52.341731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.341731+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.710715+0000","last_change":"2024-02-12T17:09:21.594780+0000","last_active":"2024-02-12T17:09:26.710715+0000","last_peered":"2024-02-12T17:09:26.710715+0000","last_clean":"2024-02-12T17:09:26.710715+0000","last_became_active":"2024-02-12T17:09:21.539652+0000","last_became_peered":"2024-02-12T17:09:21.539652+0000","last_unstale":"2024-02-12T17:09:26.710715+0000","last_undegraded":"2024-02-12T17:09:26.710715+0000","last_fullsized":"2024-02-12T17:09:26.710715+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:42.272807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:42.272807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"278'28","reported_seq":603,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699951+0000","last_change":"2024-02-12T17:09:22.165941+0000","last_active":"2024-02-12T17:09:26.699951+0000","last_peered":"2024-02-12T17:09:26.699951+0000","last_clean":"2024-02-12T17:09:26.699951+0000","last_became_active":"2024-02-12T17:09:21.531831+0000","last_became_peered":"2024-02-12T17:09:21.531831+0000","last_unstale":"2024-02-12T17:09:26.699951+0000","last_undegraded":"2024-02-12T17:09:26.699951+0000","last_fullsized":"2024-02-12T17:09:26.699951+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'28","last_scrub_stamp":"2024-02-12T17:06:28.073131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:28.073131+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"515'22","reported_seq":326,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:32.528029+0000","last_change":"2024-02-12T17:09:22.480614+0000","last_active":"2024-02-12T17:09:32.528029+0000","last_peered":"2024-02-12T17:09:32.528029+0000","last_clean":"2024-02-12T17:09:32.528029+0000","last_became_active":"2024-02-12T17:09:22.480266+0000","last_became_peered":"2024-02-12T17:09:22.480266+0000","last_unstale":"2024-02-12T17:09:32.528029+0000","last_undegraded":"2024-02-12T17:09:32.528029+0000","last_fullsized":"2024-02-12T17:09:32.528029+0000","mapping_epoch":514,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"520'9052","reported_seq":9165,"reported_epoch":520,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:09:33.170765+0000","last_change":"2024-02-12T17:09:21.551876+0000","last_active":"2024-02-12T17:09:33.170765+0000","last_peered":"2024-02-12T17:09:33.170765+0000","last_clean":"2024-02-12T17:08:11.791648+0000","last_became_active":"2024-02-12T17:09:21.541609+0000","last_became_peered":"2024-02-12T17:09:21.541609+0000","last_unstale":"2024-02-12T17:09:33.170765+0000","last_undegraded":"2024-02-12T17:09:33.170765+0000","last_fullsized":"2024-02-12T17:09:33.170765+0000","mapping_epoch":513,"log_start":"466'4200","ondisk_log_start":"466'4200","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4852,"ondisk_log_size":4852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63111168,"num_objects":963,"num_object_clones":0,"num_object_copies":2889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1926,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2818,"num_write_kb":120992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[2,7,5],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":963}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":230,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522829+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:09:31.522829+0000","last_peered":"2024-02-12T17:09:31.522829+0000","last_clean":"2024-02-12T17:09:31.522829+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:09:31.522829+0000","last_undegraded":"2024-02-12T17:09:31.522829+0000","last_fullsized":"2024-02-12T17:09:31.522829+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"515'24","reported_seq":768,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:31.480578+0000","last_change":"2024-02-12T17:09:21.599483+0000","last_active":"2024-02-12T17:09:31.480578+0000","last_peered":"2024-02-12T17:09:31.480578+0000","last_clean":"2024-02-12T17:08:07.495669+0000","last_became_active":"2024-02-12T17:09:21.546257+0000","last_became_peered":"2024-02-12T17:09:21.546257+0000","last_unstale":"2024-02-12T17:09:31.480578+0000","last_undegraded":"2024-02-12T17:09:31.480578+0000","last_fullsized":"2024-02-12T17:09:31.480578+0000","mapping_epoch":513,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'21","last_scrub_stamp":"2024-02-12T17:06:32.977637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:06:32.977637+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":165,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.696011+0000","last_change":"2024-02-12T17:08:12.864376+0000","last_active":"2024-02-12T17:09:26.696011+0000","last_peered":"2024-02-12T17:09:26.696011+0000","last_clean":"2024-02-12T17:09:26.696011+0000","last_became_active":"2024-02-12T17:08:12.864037+0000","last_became_peered":"2024-02-12T17:08:12.864037+0000","last_unstale":"2024-02-12T17:09:26.696011+0000","last_undegraded":"2024-02-12T17:09:26.696011+0000","last_fullsized":"2024-02-12T17:09:26.696011+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:24.976048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:06:24.976048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"278'19","reported_seq":28,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.710085+0000","last_change":"2024-02-12T17:09:21.666423+0000","last_active":"2024-02-12T17:09:26.710085+0000","last_peered":"2024-02-12T17:09:26.710085+0000","last_clean":"2024-02-12T17:09:26.710085+0000","last_became_active":"2024-02-12T17:09:21.498516+0000","last_became_peered":"2024-02-12T17:09:21.498516+0000","last_unstale":"2024-02-12T17:09:26.710085+0000","last_undegraded":"2024-02-12T17:09:26.710085+0000","last_fullsized":"2024-02-12T17:09:26.710085+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:05:55.344999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.344999+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"520'9312","reported_seq":9483,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.506170+0000","last_change":"2024-02-12T17:09:21.509246+0000","last_active":"2024-02-12T17:09:33.506170+0000","last_peered":"2024-02-12T17:09:33.506170+0000","last_clean":"2024-02-12T17:08:11.814847+0000","last_became_active":"2024-02-12T17:09:01.458677+0000","last_became_peered":"2024-02-12T17:09:01.458677+0000","last_unstale":"2024-02-12T17:09:33.506170+0000","last_undegraded":"2024-02-12T17:09:33.506170+0000","last_fullsized":"2024-02-12T17:09:33.506170+0000","mapping_epoch":492,"log_start":"466'4500","ondisk_log_start":"466'4500","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4812,"ondisk_log_size":4812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68567040,"num_objects":1046,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1012,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":125264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":1046}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":539,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.701695+0000","last_change":"2024-02-12T17:08:13.011711+0000","last_active":"2024-02-12T17:09:26.701695+0000","last_peered":"2024-02-12T17:09:26.701695+0000","last_clean":"2024-02-12T17:09:26.701695+0000","last_became_active":"2024-02-12T17:08:12.972066+0000","last_became_peered":"2024-02-12T17:08:12.972066+0000","last_unstale":"2024-02-12T17:09:26.701695+0000","last_undegraded":"2024-02-12T17:09:26.701695+0000","last_fullsized":"2024-02-12T17:09:26.701695+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"435'21","reported_seq":807,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.484702+0000","last_change":"2024-02-12T17:07:41.785111+0000","last_active":"2024-02-12T17:09:31.484702+0000","last_peered":"2024-02-12T17:09:31.484702+0000","last_clean":"2024-02-12T17:09:31.484702+0000","last_became_active":"2024-02-12T17:05:43.540504+0000","last_became_peered":"2024-02-12T17:05:43.540504+0000","last_unstale":"2024-02-12T17:09:31.484702+0000","last_undegraded":"2024-02-12T17:09:31.484702+0000","last_fullsized":"2024-02-12T17:09:31.484702+0000","mapping_epoch":434,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'21","last_scrub_stamp":"2024-02-12T17:07:41.785018+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.785018+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":38,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.713711+0000","last_change":"2024-02-12T17:08:12.966155+0000","last_active":"2024-02-12T17:09:26.713711+0000","last_peered":"2024-02-12T17:09:26.713711+0000","last_clean":"2024-02-12T17:09:26.713711+0000","last_became_active":"2024-02-12T17:08:12.965556+0000","last_became_peered":"2024-02-12T17:08:12.965556+0000","last_unstale":"2024-02-12T17:09:26.713711+0000","last_undegraded":"2024-02-12T17:09:26.713711+0000","last_fullsized":"2024-02-12T17:09:26.713711+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"279'34","reported_seq":54,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.714083+0000","last_change":"2024-02-12T17:08:17.847813+0000","last_active":"2024-02-12T17:09:26.714083+0000","last_peered":"2024-02-12T17:09:26.714083+0000","last_clean":"2024-02-12T17:09:26.714083+0000","last_became_active":"2024-02-12T17:08:12.964494+0000","last_became_peered":"2024-02-12T17:08:12.964494+0000","last_unstale":"2024-02-12T17:09:26.714083+0000","last_undegraded":"2024-02-12T17:09:26.714083+0000","last_fullsized":"2024-02-12T17:09:26.714083+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"521'9099","reported_seq":9216,"reported_epoch":521,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:35.938818+0000","last_change":"2024-02-12T17:09:24.683729+0000","last_active":"2024-02-12T17:09:35.938818+0000","last_peered":"2024-02-12T17:09:35.938818+0000","last_clean":"2024-02-12T17:08:11.820762+0000","last_became_active":"2024-02-12T17:09:21.602168+0000","last_became_peered":"2024-02-12T17:09:21.602168+0000","last_unstale":"2024-02-12T17:09:35.938818+0000","last_undegraded":"2024-02-12T17:09:35.938818+0000","last_fullsized":"2024-02-12T17:09:35.938818+0000","mapping_epoch":513,"log_start":"466'4100","ondisk_log_start":"466'4100","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4999,"ondisk_log_size":4999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64831488,"num_objects":989,"num_object_clones":0,"num_object_copies":2967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1978,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2826,"num_write_kb":122096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[3,7,5],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":38,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.713333+0000","last_change":"2024-02-12T17:08:12.908568+0000","last_active":"2024-02-12T17:09:26.713333+0000","last_peered":"2024-02-12T17:09:26.713333+0000","last_clean":"2024-02-12T17:09:26.713333+0000","last_became_active":"2024-02-12T17:08:12.908230+0000","last_became_peered":"2024-02-12T17:08:12.908230+0000","last_unstale":"2024-02-12T17:09:26.713333+0000","last_undegraded":"2024-02-12T17:09:26.713333+0000","last_fullsized":"2024-02-12T17:09:26.713333+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.077971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":371,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.478575+0000","last_change":"2024-02-12T17:07:41.239456+0000","last_active":"2024-02-12T17:09:31.478575+0000","last_peered":"2024-02-12T17:09:31.478575+0000","last_clean":"2024-02-12T17:09:31.478575+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:09:31.478575+0000","last_undegraded":"2024-02-12T17:09:31.478575+0000","last_fullsized":"2024-02-12T17:09:31.478575+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:07:41.239350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.239350+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":38,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.714063+0000","last_change":"2024-02-12T17:08:12.961477+0000","last_active":"2024-02-12T17:09:26.714063+0000","last_peered":"2024-02-12T17:09:26.714063+0000","last_clean":"2024-02-12T17:09:26.714063+0000","last_became_active":"2024-02-12T17:08:12.960827+0000","last_became_peered":"2024-02-12T17:08:12.960827+0000","last_unstale":"2024-02-12T17:09:26.714063+0000","last_undegraded":"2024-02-12T17:09:26.714063+0000","last_fullsized":"2024-02-12T17:09:26.714063+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"278'20","reported_seq":257,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.522923+0000","last_change":"2024-02-12T17:09:23.995231+0000","last_active":"2024-02-12T17:09:31.522923+0000","last_peered":"2024-02-12T17:09:31.522923+0000","last_clean":"2024-02-12T17:09:31.522923+0000","last_became_active":"2024-02-12T17:09:21.533801+0000","last_became_peered":"2024-02-12T17:09:21.533801+0000","last_unstale":"2024-02-12T17:09:31.522923+0000","last_undegraded":"2024-02-12T17:09:31.522923+0000","last_fullsized":"2024-02-12T17:09:31.522923+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"520'9265","reported_seq":9363,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:33.257438+0000","last_change":"2024-02-12T17:09:01.540513+0000","last_active":"2024-02-12T17:09:33.257438+0000","last_peered":"2024-02-12T17:09:33.257438+0000","last_clean":"2024-02-12T17:09:33.257438+0000","last_became_active":"2024-02-12T17:09:01.473480+0000","last_became_peered":"2024-02-12T17:09:01.473480+0000","last_unstale":"2024-02-12T17:09:33.257438+0000","last_undegraded":"2024-02-12T17:09:33.257438+0000","last_fullsized":"2024-02-12T17:09:33.257438+0000","mapping_epoch":451,"log_start":"466'4400","ondisk_log_start":"466'4400","created":451,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4865,"ondisk_log_size":4865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65830912,"num_objects":1004,"num_object_clones":0,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":124096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":232,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.528784+0000","last_change":"2024-02-12T17:09:21.618330+0000","last_active":"2024-02-12T17:09:31.528784+0000","last_peered":"2024-02-12T17:09:31.528784+0000","last_clean":"2024-02-12T17:09:31.528784+0000","last_became_active":"2024-02-12T17:09:21.617713+0000","last_became_peered":"2024-02-12T17:09:21.617713+0000","last_unstale":"2024-02-12T17:09:31.528784+0000","last_undegraded":"2024-02-12T17:09:31.528784+0000","last_fullsized":"2024-02-12T17:09:31.528784+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":47,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.712818+0000","last_change":"2024-02-12T17:09:21.496278+0000","last_active":"2024-02-12T17:09:26.712818+0000","last_peered":"2024-02-12T17:09:26.712818+0000","last_clean":"2024-02-12T17:09:26.712818+0000","last_became_active":"2024-02-12T17:09:21.496031+0000","last_became_peered":"2024-02-12T17:09:21.496031+0000","last_unstale":"2024-02-12T17:09:26.712818+0000","last_undegraded":"2024-02-12T17:09:26.712818+0000","last_fullsized":"2024-02-12T17:09:26.712818+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.567340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.567340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":502,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.696290+0000","last_change":"2024-02-12T17:06:51.877944+0000","last_active":"2024-02-12T17:09:26.696290+0000","last_peered":"2024-02-12T17:09:26.696290+0000","last_clean":"2024-02-12T17:09:26.696290+0000","last_became_active":"2024-02-12T17:04:39.888466+0000","last_became_peered":"2024-02-12T17:04:39.888466+0000","last_unstale":"2024-02-12T17:09:26.696290+0000","last_undegraded":"2024-02-12T17:09:26.696290+0000","last_fullsized":"2024-02-12T17:09:26.696290+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.877830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.877830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"278'31","reported_seq":83,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.712045+0000","last_change":"2024-02-12T17:09:22.940804+0000","last_active":"2024-02-12T17:09:26.712045+0000","last_peered":"2024-02-12T17:09:26.712045+0000","last_clean":"2024-02-12T17:09:26.712045+0000","last_became_active":"2024-02-12T17:09:21.494303+0000","last_became_peered":"2024-02-12T17:09:21.494303+0000","last_unstale":"2024-02-12T17:09:26.712045+0000","last_undegraded":"2024-02-12T17:09:26.712045+0000","last_fullsized":"2024-02-12T17:09:26.712045+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:07:37.774472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.774472+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"521'9479","reported_seq":9841,"reported_epoch":521,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T17:09:36.204707+0000","last_change":"2024-02-12T17:09:24.032468+0000","last_active":"2024-02-12T17:09:36.204707+0000","last_peered":"2024-02-12T17:09:36.204707+0000","last_clean":"2024-02-12T17:08:11.807038+0000","last_became_active":"2024-02-12T17:08:14.000844+0000","last_became_peered":"2024-02-12T17:08:14.000844+0000","last_unstale":"2024-02-12T17:09:36.204707+0000","last_undegraded":"2024-02-12T17:09:36.204707+0000","last_fullsized":"2024-02-12T17:09:36.204707+0000","mapping_epoch":492,"log_start":"468'4845","ondisk_log_start":"468'4845","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4634,"ondisk_log_size":4634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":130318336,"num_objects":1988,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1865,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":245536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":8650752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":538,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.696057+0000","last_change":"2024-02-12T17:05:57.890025+0000","last_active":"2024-02-12T17:09:26.696057+0000","last_peered":"2024-02-12T17:09:26.696057+0000","last_clean":"2024-02-12T17:09:26.696057+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:09:26.696057+0000","last_undegraded":"2024-02-12T17:09:26.696057+0000","last_fullsized":"2024-02-12T17:09:26.696057+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.889955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.889955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":435,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.744455+0000","last_change":"2024-02-12T17:07:20.629548+0000","last_active":"2024-02-12T17:09:26.744455+0000","last_peered":"2024-02-12T17:09:26.744455+0000","last_clean":"2024-02-12T17:09:26.744455+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:09:26.744455+0000","last_undegraded":"2024-02-12T17:09:26.744455+0000","last_fullsized":"2024-02-12T17:09:26.744455+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.629467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.629467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":256,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.700868+0000","last_change":"2024-02-12T17:08:12.953085+0000","last_active":"2024-02-12T17:09:26.700868+0000","last_peered":"2024-02-12T17:09:26.700868+0000","last_clean":"2024-02-12T17:09:26.700868+0000","last_became_active":"2024-02-12T17:08:12.952757+0000","last_became_peered":"2024-02-12T17:08:12.952757+0000","last_unstale":"2024-02-12T17:09:26.700868+0000","last_undegraded":"2024-02-12T17:09:26.700868+0000","last_fullsized":"2024-02-12T17:09:26.700868+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"279'35","reported_seq":36,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.706493+0000","last_change":"2024-02-12T17:09:23.047646+0000","last_active":"2024-02-12T17:09:26.706493+0000","last_peered":"2024-02-12T17:09:26.706493+0000","last_clean":"2024-02-12T17:09:26.706493+0000","last_became_active":"2024-02-12T17:09:21.559285+0000","last_became_peered":"2024-02-12T17:09:21.559285+0000","last_unstale":"2024-02-12T17:09:26.706493+0000","last_undegraded":"2024-02-12T17:09:26.706493+0000","last_fullsized":"2024-02-12T17:09:26.706493+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"520'9411","reported_seq":9506,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:33.726819+0000","last_change":"2024-02-12T17:07:53.893435+0000","last_active":"2024-02-12T17:09:33.726819+0000","last_peered":"2024-02-12T17:09:33.726819+0000","last_clean":"2024-02-12T17:09:33.726819+0000","last_became_active":"2024-02-12T17:07:52.071379+0000","last_became_peered":"2024-02-12T17:07:52.071379+0000","last_unstale":"2024-02-12T17:09:33.726819+0000","last_undegraded":"2024-02-12T17:09:33.726819+0000","last_fullsized":"2024-02-12T17:09:33.726819+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'7593","last_scrub_stamp":"2024-02-12T17:07:53.893352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:07:53.893352+0000","log_size":9411,"ondisk_log_size":9411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1818,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":222,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.696585+0000","last_change":"2024-02-12T17:06:43.014800+0000","last_active":"2024-02-12T17:09:26.696585+0000","last_peered":"2024-02-12T17:09:26.696585+0000","last_clean":"2024-02-12T17:09:26.696585+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:09:26.696585+0000","last_undegraded":"2024-02-12T17:09:26.696585+0000","last_fullsized":"2024-02-12T17:09:26.696585+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:06:43.014737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":437,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523413+0000","last_change":"2024-02-12T17:08:12.919537+0000","last_active":"2024-02-12T17:09:31.523413+0000","last_peered":"2024-02-12T17:09:31.523413+0000","last_clean":"2024-02-12T17:09:31.523413+0000","last_became_active":"2024-02-12T17:08:12.919194+0000","last_became_peered":"2024-02-12T17:08:12.919194+0000","last_unstale":"2024-02-12T17:09:31.523413+0000","last_undegraded":"2024-02-12T17:09:31.523413+0000","last_fullsized":"2024-02-12T17:09:31.523413+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":42,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.711955+0000","last_change":"2024-02-12T17:08:17.757666+0000","last_active":"2024-02-12T17:09:26.711955+0000","last_peered":"2024-02-12T17:09:26.711955+0000","last_clean":"2024-02-12T17:09:26.711955+0000","last_became_active":"2024-02-12T17:08:13.013035+0000","last_became_peered":"2024-02-12T17:08:13.013035+0000","last_unstale":"2024-02-12T17:09:26.711955+0000","last_undegraded":"2024-02-12T17:09:26.711955+0000","last_fullsized":"2024-02-12T17:09:26.711955+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"278'15","reported_seq":48,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.711258+0000","last_change":"2024-02-12T17:08:15.679265+0000","last_active":"2024-02-12T17:09:26.711258+0000","last_peered":"2024-02-12T17:09:26.711258+0000","last_clean":"2024-02-12T17:09:26.711258+0000","last_became_active":"2024-02-12T17:08:12.961078+0000","last_became_peered":"2024-02-12T17:08:12.961078+0000","last_unstale":"2024-02-12T17:09:26.711258+0000","last_undegraded":"2024-02-12T17:09:26.711258+0000","last_fullsized":"2024-02-12T17:09:26.711258+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:38.887701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.887701+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"520'9581","reported_seq":14849,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:09:33.292363+0000","last_change":"2024-02-12T17:09:22.585673+0000","last_active":"2024-02-12T17:09:33.292363+0000","last_peered":"2024-02-12T17:09:33.292363+0000","last_clean":"2024-02-12T17:09:20.548450+0000","last_became_active":"2024-02-12T17:09:22.566995+0000","last_became_peered":"2024-02-12T17:09:22.566995+0000","last_unstale":"2024-02-12T17:09:33.292363+0000","last_undegraded":"2024-02-12T17:09:33.292363+0000","last_fullsized":"2024-02-12T17:09:33.292363+0000","mapping_epoch":514,"log_start":"468'4931","ondisk_log_start":"468'4931","created":451,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4650,"ondisk_log_size":4650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":136118272,"num_objects":2077,"num_object_clones":0,"num_object_copies":6231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2077,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5729,"num_write_kb":249792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2573,"num_bytes_recovered":168624128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,2],"avail_no_missing":["5","1","2","7"],"object_location_counts":[{"shards":"1,2,5","objects":2077}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":521,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699655+0000","last_change":"2024-02-12T17:09:21.531721+0000","last_active":"2024-02-12T17:09:26.699655+0000","last_peered":"2024-02-12T17:09:26.699655+0000","last_clean":"2024-02-12T17:09:26.699655+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:09:26.699655+0000","last_undegraded":"2024-02-12T17:09:26.699655+0000","last_fullsized":"2024-02-12T17:09:26.699655+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:47.945301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:47.945301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":187,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.523090+0000","last_change":"2024-02-12T17:06:53.648932+0000","last_active":"2024-02-12T17:09:31.523090+0000","last_peered":"2024-02-12T17:09:31.523090+0000","last_clean":"2024-02-12T17:09:31.523090+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:09:31.523090+0000","last_undegraded":"2024-02-12T17:09:31.523090+0000","last_fullsized":"2024-02-12T17:09:31.523090+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"278'34","reported_seq":619,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-02-12T17:09:26.699683+0000","last_change":"2024-02-12T17:09:22.375867+0000","last_active":"2024-02-12T17:09:26.699683+0000","last_peered":"2024-02-12T17:09:26.699683+0000","last_clean":"2024-02-12T17:09:26.699683+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:09:26.699683+0000","last_undegraded":"2024-02-12T17:09:26.699683+0000","last_fullsized":"2024-02-12T17:09:26.699683+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:07:47.772969+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.772969+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"519'25","reported_seq":875,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-12T17:09:31.478831+0000","last_change":"2024-02-12T17:09:24.656843+0000","last_active":"2024-02-12T17:09:31.478831+0000","last_peered":"2024-02-12T17:09:31.478831+0000","last_clean":"2024-02-12T17:09:31.478831+0000","last_became_active":"2024-02-12T17:09:24.656509+0000","last_became_peered":"2024-02-12T17:09:24.656509+0000","last_unstale":"2024-02-12T17:09:31.478831+0000","last_undegraded":"2024-02-12T17:09:31.478831+0000","last_fullsized":"2024-02-12T17:09:31.478831+0000","mapping_epoch":516,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-02-12T17:06:35.066866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.066866+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":36,"stat_sum":{"num_bytes":4249878552,"num_objects":64849,"num_object_clones":0,"num_object_copies":194547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1585,"num_objects_misplaced":78970,"num_objects_unfound":0,"num_objects_dirty":64849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177715,"num_write_kb":7638337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23077,"num_bytes_recovered":1512046592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14180757504,"data_stored":14180745288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179256,"ondisk_log_size":179256,"up":108,"acting":106,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":494,"num_read_kb":489,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1095,"num_read_kb":852,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":656,"num_bytes_recovered":13444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":736,"ondisk_log_size":736,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234226,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3873572,"kb_used_data":2978012,"kb_used_omap":0,"kb_used_meta":895552,"kb_avail":89871580,"statfs":{"total":95995035648,"available":92028497920,"internally_reserved":0,"allocated":3049484288,"data_stored":3048353665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917045248},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":2.048,"15min":2.0499999999999998},"min":{"1min":0.218,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":6.2880000000000003,"5min":9.6600000000000001,"15min":36.439},"last":0.316},{"interface":"front","average":{"1min":1.393,"5min":1.7410000000000001,"15min":2.0859999999999999},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.19600000000000001},"max":{"1min":4.8540000000000001,"5min":8.5039999999999996,"15min":36.490000000000002},"last":0.22900000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":2.23,"15min":1.8560000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":6.4539999999999997,"5min":13.224,"15min":13.224},"last":0.25800000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.6679999999999999,"15min":1.7549999999999999},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.22700000000000001},"max":{"1min":6.4660000000000002,"5min":10.194000000000001,"15min":17.324999999999999},"last":0.30499999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":1.877,"15min":1.6919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":13.303000000000001,"5min":13.303000000000001,"15min":13.303000000000001},"last":24.824000000000002},{"interface":"front","average":{"1min":1.9450000000000001,"5min":2.6219999999999999,"15min":2.1280000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.23000000000000001},"max":{"1min":6.5099999999999998,"5min":13.127000000000001,"15min":20.338000000000001},"last":28.753}]},{"osd":3,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.8300000000000001,"15min":1.8799999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.4700000000000002,"5min":8.6489999999999991,"15min":20.433},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.454,"5min":2.4420000000000002,"15min":2.298},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.25600000000000001},"max":{"1min":6.0999999999999996,"5min":13.196,"15min":31.152000000000001},"last":0.28399999999999997}]},{"osd":4,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":1.806,"15min":2.012},"min":{"1min":0.26500000000000001,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":1.258,"5min":12.108000000000001,"15min":50.406999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":2.214,"15min":2.4089999999999998},"min":{"1min":0.27000000000000002,"5min":0.249,"15min":0.23999999999999999},"max":{"1min":0.51200000000000001,"5min":16.574999999999999,"15min":50.168999999999997},"last":0.51900000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":2.056,"15min":2.0899999999999999},"min":{"1min":0.25800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.4019999999999999,"5min":14.074,"15min":19.702000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":2.2170000000000001,"15min":2.1829999999999998},"min":{"1min":0.29499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53800000000000003,"5min":17.585999999999999,"15min":18.651},"last":0.622}]},{"osd":6,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":1.5620000000000001,"15min":1.8049999999999999},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27400000000000002},"max":{"1min":0.54600000000000004,"5min":8.2420000000000009,"15min":17.234999999999999},"last":0.378},{"interface":"front","average":{"1min":0.49099999999999999,"5min":1.794,"15min":1.8109999999999999},"min":{"1min":0.32900000000000001,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":1.4330000000000001,"5min":9.7799999999999994,"15min":13.676},"last":0.65600000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528567,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":403684,"kb_used_data":94172,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93341468,"statfs":{"total":95995035648,"available":95581663232,"internally_reserved":0,"allocated":96432128,"data_stored":95301881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:08:37 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.8280000000000001,"15min":1.851},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":4.375,"5min":14.949,"15min":22.817},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.6539999999999999,"15min":1.754},"min":{"1min":0.251,"5min":0.23000000000000001,"15min":0.22800000000000001},"max":{"1min":2.6640000000000001,"5min":17.059999999999999,"15min":17.059999999999999},"last":0.88900000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:08:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.5629999999999999,"15min":1.5229999999999999},"min":{"1min":0.29299999999999998,"5min":0.27200000000000002,"15min":0.23699999999999999},"max":{"1min":3.9159999999999999,"5min":8.0489999999999995,"15min":10.865},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.865,"15min":1.7250000000000001},"min":{"1min":0.34399999999999997,"5min":0.26900000000000002,"15min":0.23200000000000001},"max":{"1min":3.1970000000000001,"5min":17.106999999999999,"15min":17.106999999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:08:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":2.1960000000000002,"15min":2.0939999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.22900000000000001},"max":{"1min":7.1719999999999997,"5min":24.300999999999998,"15min":56.798000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.637,"5min":2.0019999999999998,"15min":2.1339999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":14.131,"5min":24.143999999999998,"15min":56.780999999999999},"last":3.093}]},{"osd":3,"last update":"Mon Feb 12 17:08:37 2024","interfaces":[{"interface":"back","average":{"1min":2.5529999999999999,"5min":2.2069999999999999,"15min":1.7769999999999999},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":18.616,"5min":24.626000000000001,"15min":24.626000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.8700000000000001,"5min":2.6320000000000001,"15min":2.1499999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26900000000000002},"max":{"1min":7.1559999999999997,"5min":26.686,"15min":26.686},"last":2.5779999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":1.4019999999999999,"15min":1.694},"min":{"1min":0.255,"5min":0.248,"15min":0.246},"max":{"1min":0.65000000000000002,"5min":11.526999999999999,"15min":24.956},"last":1.5449999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.7310000000000001,"15min":1.593},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.23799999999999999},"max":{"1min":3.8690000000000002,"5min":13.728,"15min":13.728},"last":0.42499999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.6899999999999999,"15min":1.9650000000000001},"min":{"1min":0.307,"5min":0.29599999999999999,"15min":0.245},"max":{"1min":3.8929999999999998,"5min":11.571,"15min":21.411000000000001},"last":0.5},{"interface":"front","average":{"1min":0.41699999999999998,"5min":1.859,"15min":2.246},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.73899999999999999,"5min":10.826000000000001,"15min":27.472000000000001},"last":1.621}]},{"osd":7,"last update":"Mon Feb 12 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":2.0310000000000001,"15min":2.0950000000000002},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.5329999999999999,"5min":18.951000000000001,"15min":31.033000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":1.3460000000000001,"15min":2.0390000000000001},"min":{"1min":0.317,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.499,"5min":10.468999999999999,"15min":29.34},"last":1.5720000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575120,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":973388,"kb_used_data":383620,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":92771764,"statfs":{"total":95995035648,"available":94998286336,"internally_reserved":0,"allocated":392826880,"data_stored":391657624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.17499999999999999},"max":{"1min":6.4260000000000002,"5min":7.8899999999999997,"15min":25.585000000000001},"last":65.328999999999994},{"interface":"front","average":{"1min":2.3639999999999999,"5min":1.399,"15min":1.575},"min":{"1min":0.193,"5min":0.193,"15min":0.189},"max":{"1min":13.144,"5min":13.144,"15min":26.798999999999999},"last":65.372}]},{"osd":2,"last update":"Mon Feb 12 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":5.9429999999999996,"5min":2.6989999999999998,"15min":2.0019999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.191},"max":{"1min":51.820999999999998,"5min":51.820999999999998,"15min":51.820999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.395,"15min":1.847},"min":{"1min":0.223,"5min":0.223,"15min":0.19},"max":{"1min":53.802999999999997,"5min":53.802999999999997,"15min":53.802999999999997},"last":0.376}]},{"osd":3,"last update":"Mon Feb 12 17:08:35 2024","interfaces":[{"interface":"back","average":{"1min":6.2690000000000001,"5min":2.528,"15min":1.9890000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.23100000000000001},"max":{"1min":64.528000000000006,"5min":64.528000000000006,"15min":64.528000000000006},"last":0.33900000000000002},{"interface":"front","average":{"1min":5.6740000000000004,"5min":1.978,"15min":1.6579999999999999},"min":{"1min":0.35899999999999999,"5min":0.249,"15min":0.23999999999999999},"max":{"1min":63.753999999999998,"5min":63.753999999999998,"15min":63.753999999999998},"last":2.2570000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":2.222,"15min":1.853},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.28000000000000003},"max":{"1min":6.3810000000000002,"5min":12.91,"15min":12.91},"last":2.2269999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":2.1579999999999999,"15min":1.968},"min":{"1min":0.374,"5min":0.33000000000000002,"15min":0.25900000000000001},"max":{"1min":8.5350000000000001,"5min":16.074000000000002,"15min":26.748999999999999},"last":0.40899999999999997}]},{"osd":5,"last update":"Mon Feb 12 17:09:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.8819999999999999,"15min":2.0739999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.306,"5min":22.154,"15min":22.154},"last":2.2050000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.754,"15min":1.913},"min":{"1min":0.317,"5min":0.317,"15min":0.27300000000000002},"max":{"1min":8.5820000000000007,"5min":21.803000000000001,"15min":21.803000000000001},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:08:44 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.6539999999999999,"15min":1.9059999999999999},"min":{"1min":0.45400000000000001,"5min":0.36499999999999999,"15min":0.30599999999999999},"max":{"1min":6.4039999999999999,"5min":9.1460000000000008,"15min":13.834},"last":2.1659999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.772,"15min":1.986},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.307},"max":{"1min":8.6140000000000008,"5min":16.172999999999998,"15min":26.774999999999999},"last":0.438}]},{"osd":7,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.9239999999999999,"15min":2.0289999999999999},"min":{"1min":0.377,"5min":0.33100000000000002,"15min":0.28799999999999998},"max":{"1min":7.3099999999999996,"5min":8.7560000000000002,"15min":9.5239999999999991},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.8200000000000001,"5min":2.0790000000000002,"15min":2.1669999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":9.3510000000000009,"5min":13.965999999999999,"15min":20.548999999999999},"last":2.242}]}]},{"osd":0,"up_from":8,"seq":34359738644,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662360,"kb_used_data":770192,"kb_used_omap":0,"kb_used_meta":892096,"kb_avail":92082792,"statfs":{"total":95995035648,"available":94292779008,"internally_reserved":0,"allocated":788676608,"data_stored":787621986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913506304},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":1.3049999999999999,"15min":1.429},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.17399999999999999},"max":{"1min":10.807,"5min":10.807,"15min":23.510999999999999},"last":0.438},{"interface":"front","average":{"1min":2.9940000000000002,"5min":1.9690000000000001,"15min":1.573},"min":{"1min":0.246,"5min":0.20399999999999999,"15min":0.193},"max":{"1min":10.741,"5min":16.16,"15min":16.16},"last":5.843}]},{"osd":2,"last update":"Mon Feb 12 17:09:16 2024","interfaces":[{"interface":"back","average":{"1min":4.3410000000000002,"5min":2.056,"15min":1.722},"min":{"1min":0.35399999999999998,"5min":0.24399999999999999,"15min":0.20200000000000001},"max":{"1min":15.472,"5min":15.472,"15min":16.521000000000001},"last":5.7229999999999999},{"interface":"front","average":{"1min":4.4889999999999999,"5min":2.5409999999999999,"15min":1.839},"min":{"1min":0.45200000000000001,"5min":0.254,"15min":0.20300000000000001},"max":{"1min":13.726000000000001,"5min":13.726000000000001,"15min":13.726000000000001},"last":6.2590000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":3.3359999999999999,"5min":1.8879999999999999,"15min":1.9650000000000001},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.214},"max":{"1min":13.362,"5min":13.362,"15min":21.100000000000001},"last":9.1690000000000005},{"interface":"front","average":{"1min":3.0920000000000001,"5min":2.0670000000000002,"15min":1.954},"min":{"1min":0.27800000000000002,"5min":0.221,"15min":0.221},"max":{"1min":13.005000000000001,"5min":16.759,"15min":16.759},"last":9.3239999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.375,"15min":1.6399999999999999},"min":{"1min":0.34399999999999997,"5min":0.28599999999999998,"15min":0.23599999999999999},"max":{"1min":10.300000000000001,"5min":10.300000000000001,"15min":16.888999999999999},"last":0.498},{"interface":"front","average":{"1min":2.597,"5min":1.877,"15min":1.8100000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28399999999999997},"max":{"1min":10.638999999999999,"5min":10.638999999999999,"15min":16.899999999999999},"last":5.819}]},{"osd":5,"last update":"Mon Feb 12 17:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.742,"15min":2.2450000000000001},"min":{"1min":0.34000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.843,"5min":10.146000000000001,"15min":50.404000000000003},"last":1.6180000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":1.875,"15min":2.1840000000000002},"min":{"1min":0.38200000000000001,"5min":0.28599999999999998,"15min":0.28299999999999997},"max":{"1min":2.2109999999999999,"5min":19.751999999999999,"15min":49.979999999999997},"last":2.1440000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:08:49 2024","interfaces":[{"interface":"back","average":{"1min":2.7810000000000001,"5min":1.9179999999999999,"15min":1.8819999999999999},"min":{"1min":0.36299999999999999,"5min":0.33100000000000002,"15min":0.29999999999999999},"max":{"1min":10.595000000000001,"5min":10.595000000000001,"15min":17.907},"last":5.798},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.6160000000000001,"15min":2.024},"min":{"1min":0.38600000000000001,"5min":0.372,"15min":0.27900000000000003},"max":{"1min":6.798,"5min":7.7510000000000003,"15min":21.062000000000001},"last":1.262}]},{"osd":7,"last update":"Mon Feb 12 17:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.9410000000000001,"15min":1.9930000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.2469999999999999,"5min":12.836,"15min":19.373000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.698,"5min":2.0990000000000002,"15min":2.105},"min":{"1min":0.39800000000000002,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":10.714,"5min":10.714,"15min":34.722999999999999},"last":5.766}]}]},{"osd":2,"up_from":18,"seq":77309411593,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4827344,"kb_used_data":3934600,"kb_used_omap":0,"kb_used_meta":892672,"kb_avail":88917808,"statfs":{"total":95995035648,"available":91051835392,"internally_reserved":0,"allocated":4029030400,"data_stored":4027920503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914096128},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":3.056,"5min":2.0950000000000002,"15min":1.958},"min":{"1min":0.191,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.2829999999999995,"5min":21.445,"15min":36.100999999999999},"last":0.311},{"interface":"front","average":{"1min":3.0089999999999999,"5min":2.3969999999999998,"15min":1.9890000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.158},"max":{"1min":9.6910000000000007,"5min":20.359000000000002,"15min":36.299999999999997},"last":0.223}]},{"osd":1,"last update":"Mon Feb 12 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":3.282,"5min":1.948,"15min":1.776},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.19600000000000001},"max":{"1min":10.922000000000001,"5min":20.763999999999999,"15min":20.763999999999999},"last":5.2400000000000002},{"interface":"front","average":{"1min":4.3310000000000004,"5min":2.3029999999999999,"15min":1.984},"min":{"1min":0.31900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":12.552,"5min":21.361999999999998,"15min":21.361999999999998},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:08:49 2024","interfaces":[{"interface":"back","average":{"1min":4.6900000000000004,"5min":2.2290000000000001,"15min":2.206},"min":{"1min":0.39900000000000002,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":21.474,"5min":21.474,"15min":51.579999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":4.9850000000000003,"5min":2.2189999999999999,"15min":1.9990000000000001},"min":{"1min":0.42499999999999999,"5min":0.26400000000000001,"15min":0.23200000000000001},"max":{"1min":21.702999999999999,"5min":21.702999999999999,"15min":48.988999999999997},"last":0.70899999999999996}]},{"osd":4,"last update":"Mon Feb 12 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":2.7639999999999998,"5min":1.9970000000000001,"15min":2.097},"min":{"1min":0.378,"5min":0.25600000000000001,"15min":0.253},"max":{"1min":9.2579999999999991,"5min":21.515000000000001,"15min":21.515000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.8799999999999999,"5min":1.9670000000000001,"15min":2.1179999999999999},"min":{"1min":0.38500000000000001,"5min":0.29999999999999999,"15min":0.26900000000000002},"max":{"1min":6.0670000000000002,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.63400000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":2.1640000000000001,"15min":2.1579999999999999},"min":{"1min":0.33900000000000002,"5min":0.29299999999999998,"15min":0.27500000000000002},"max":{"1min":9.2219999999999995,"5min":21.542999999999999,"15min":32.57},"last":0.36499999999999999},{"interface":"front","average":{"1min":2.3769999999999998,"5min":2.1379999999999999,"15min":2.3279999999999998},"min":{"1min":0.35299999999999998,"5min":0.255,"15min":0.255},"max":{"1min":7.4089999999999998,"5min":20.937000000000001,"15min":31.905999999999999},"last":5.3920000000000003}]},{"osd":6,"last update":"Mon Feb 12 17:08:49 2024","interfaces":[{"interface":"back","average":{"1min":2.323,"5min":1.8140000000000001,"15min":2.1909999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.30199999999999999},"max":{"1min":20.966999999999999,"5min":20.966999999999999,"15min":20.966999999999999},"last":5.3029999999999999},{"interface":"front","average":{"1min":2.3639999999999999,"5min":1.732,"15min":1.96},"min":{"1min":0.45000000000000001,"5min":0.34899999999999998,"15min":0.29799999999999999},"max":{"1min":20.193000000000001,"5min":20.193000000000001,"15min":20.193000000000001},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Feb 12 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.6160000000000001,"15min":2.597},"min":{"1min":0.33100000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":7.2309999999999999,"5min":20.221,"15min":45.468000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.4710000000000001,"5min":2.1779999999999999,"15min":2.5790000000000002},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.4539999999999997,"5min":20.995999999999999,"15min":46.656999999999996},"last":5.3540000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149956,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3981816,"kb_used_data":3076272,"kb_used_omap":0,"kb_used_meta":905536,"kb_avail":89763336,"statfs":{"total":95995035648,"available":91917656064,"internally_reserved":0,"allocated":3150102528,"data_stored":3148952892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":927268864},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:08:36 2024","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":1.823,"15min":1.8939999999999999},"min":{"1min":0.20899999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":18.324999999999999,"5min":18.324999999999999,"15min":58.451999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":2.6269999999999998,"5min":2.4390000000000001,"15min":2.1269999999999998},"min":{"1min":0.255,"5min":0.214,"15min":0.19900000000000001},"max":{"1min":11.957000000000001,"5min":19.748999999999999,"15min":57.497999999999998},"last":5.4550000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:08:36 2024","interfaces":[{"interface":"back","average":{"1min":3.0800000000000001,"5min":2.0649999999999999,"15min":1.7909999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":12.371,"5min":12.371,"15min":13.94},"last":5.3849999999999998},{"interface":"front","average":{"1min":2.4169999999999998,"5min":1.3,"15min":1.401},"min":{"1min":0.32500000000000001,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":8.0739999999999998,"5min":10.278,"15min":14.962},"last":2.4260000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:08:36 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.462,"15min":1.7390000000000001},"min":{"1min":0.33400000000000002,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":8.0530000000000008,"5min":10.302,"15min":49.112000000000002},"last":2.3220000000000001},{"interface":"front","average":{"1min":3.1269999999999998,"5min":1.8240000000000001,"15min":1.976},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":51.997999999999998},"last":0.65000000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":2.9790000000000001,"5min":2.0510000000000002,"15min":2.1110000000000002},"min":{"1min":0.32100000000000001,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":18.401,"5min":18.401,"15min":19.010000000000002},"last":5.3390000000000004},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4890000000000001,"15min":1.738},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.9880000000000004,"5min":10.250999999999999,"15min":19.033999999999999},"last":2.3839999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:09:26 2024","interfaces":[{"interface":"back","average":{"1min":2.8300000000000001,"5min":1.958,"15min":2.1099999999999999},"min":{"1min":0.51700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":17.940999999999999,"5min":17.940999999999999,"15min":17.940999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":2.3730000000000002,"5min":2.3740000000000001,"15min":2.3290000000000002},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.2590000000000003,"5min":12.502000000000001,"15min":15.595000000000001},"last":5.4329999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:08:36 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.4810000000000001,"15min":1.5660000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.0259999999999998,"5min":10.212999999999999,"15min":12.856999999999999},"last":2.3580000000000001},{"interface":"front","average":{"1min":2.6200000000000001,"5min":1.657,"15min":1.8580000000000001},"min":{"1min":0.45800000000000002,"5min":0.32600000000000001,"15min":0.27500000000000002},"max":{"1min":18.265000000000001,"5min":18.265000000000001,"15min":19.925999999999998},"last":0.49199999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":3.0379999999999998,"5min":2.7909999999999999,"15min":2.4980000000000002},"min":{"1min":0.42199999999999999,"5min":0.315,"15min":0.315},"max":{"1min":18.484000000000002,"5min":37.719999999999999,"15min":37.719999999999999},"last":5.4089999999999998},{"interface":"front","average":{"1min":2.9340000000000002,"5min":2.6619999999999999,"15min":2.367},"min":{"1min":0.51600000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":17.992999999999999,"5min":38.948,"15min":38.948},"last":0.51200000000000001}]}]},{"osd":4,"up_from":0,"seq":137438953729,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757501,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3623312,"kb_used_data":2718536,"kb_used_omap":0,"kb_used_meta":904768,"kb_avail":90121840,"statfs":{"total":95995035648,"available":92284764160,"internally_reserved":0,"allocated":2783780864,"data_stored":2782670321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":926482432},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.4379999999999999,"15min":1.73},"min":{"1min":0.218,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.9480000000000004,"5min":10.971,"15min":46.365000000000002},"last":8.1780000000000008},{"interface":"front","average":{"1min":1.579,"5min":1.7829999999999999,"15min":2.1520000000000001},"min":{"1min":0.26400000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":5.2300000000000004,"5min":12.140000000000001,"15min":46.134},"last":6.9320000000000004}]},{"osd":1,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.387,"15min":1.7450000000000001},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.23200000000000001},"max":{"1min":8.3620000000000001,"5min":11.19,"15min":16.655000000000001},"last":1.244},{"interface":"front","average":{"1min":2.3149999999999999,"5min":2.02,"15min":2.097},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.21199999999999999},"max":{"1min":10.973000000000001,"5min":12.154,"15min":16.712},"last":1.159}]},{"osd":2,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":1.76,"15min":1.9770000000000001},"min":{"1min":0.246,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":8.5510000000000002,"5min":11.007999999999999,"15min":16.940999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.641,"5min":1.4099999999999999,"15min":1.823},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":9.0500000000000007,"5min":9.0500000000000007,"15min":17.103999999999999},"last":0.311}]},{"osd":3,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":3.6459999999999999,"5min":2.0640000000000001,"15min":1.919},"min":{"1min":0.26100000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":34.072000000000003,"5min":34.072000000000003,"15min":34.072000000000003},"last":1.407},{"interface":"front","average":{"1min":4.3440000000000003,"5min":2.1240000000000001,"15min":1.9319999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":34.177999999999997,"5min":34.177999999999997,"15min":34.177999999999997},"last":1.214}]},{"osd":4,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.4419999999999999,"15min":1.849},"min":{"1min":0.27000000000000002,"5min":0.251,"15min":0.23699999999999999},"max":{"1min":0.78300000000000003,"5min":12.656000000000001,"15min":12.656000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":1.2330000000000001,"15min":1.952},"min":{"1min":0.254,"5min":0.254,"15min":0.249},"max":{"1min":0.54600000000000004,"5min":11.068,"15min":20.550000000000001},"last":0.41799999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":1.573,"15min":1.806},"min":{"1min":0.29799999999999999,"5min":0.27500000000000002,"15min":0.26400000000000001},"max":{"1min":0.92200000000000004,"5min":12.121,"15min":12.281000000000001},"last":0.376},{"interface":"front","average":{"1min":0.48299999999999998,"5min":1.423,"15min":1.8600000000000001},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.29,"5min":11.087,"15min":22.035},"last":0.441}]},{"osd":7,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":1.429,"15min":1.9430000000000001},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.98699999999999999,"5min":13.647,"15min":19.184999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":1.7370000000000001,"15min":2.0409999999999999},"min":{"1min":0.34599999999999997,"5min":0.27800000000000002,"15min":0.26800000000000002},"max":{"1min":1.155,"5min":20.274000000000001,"15min":20.274000000000001},"last":2.149}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":780271616,"data_stored":780271616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":387645440,"data_stored":387645440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3994095616,"data_stored":3994091544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3118993408,"data_stored":3118989336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2765684736,"data_stored":2765684736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":95420416,"data_stored":95420416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3038646272,"data_stored":3038642200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:09:42.446 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set iscsi pgp_num 3 2024-02-12T17:09:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:40.836621+0000 mgr.y (mgr.24481) 758 : cluster [DBG] pgmap v1047: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 12 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 12 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 590 op/s; 1621/188424 objects degraded (0.860%); 75740/188424 objects misplaced (40.197%); 12 MiB/s, 195 objects/s recovering 2024-02-12T17:09:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:41.451653+0000 mon.a (mon.0) 1774 : cluster [WRN] Health check update: Degraded data redundancy: 1621/188424 objects degraded (0.860%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:42.493 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:09:42.493 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:09:42.844 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:09:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:09:43.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T17:09:43.265 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:43.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:43.376 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:43.378 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:43.478 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:43.523 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:42.838560+0000 mgr.y (mgr.24481) 759 : cluster [DBG] pgmap v1048: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 11 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 456 op/s; 1674/186414 objects degraded (0.898%); 74705/186414 objects misplaced (40.075%); 9.4 MiB/s, 150 objects/s recovering 2024-02-12T17:09:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:43.342631+0000 mon.a (mon.0) 1775 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 15 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:43.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:42.838560+0000 mgr.y (mgr.24481) 759 : cluster [DBG] pgmap v1048: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 11 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 456 op/s; 1674/186414 objects degraded (0.898%); 74705/186414 objects misplaced (40.075%); 9.4 MiB/s, 150 objects/s recovering 2024-02-12T17:09:43.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:43.342631+0000 mon.a (mon.0) 1775 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 15 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:43.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:43.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:43.669 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:42.838560+0000 mgr.y (mgr.24481) 759 : cluster [DBG] pgmap v1048: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 11 active+remapped+backfill_wait, 2 active+remapped, 114 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 456 op/s; 1674/186414 objects degraded (0.898%); 74705/186414 objects misplaced (40.075%); 9.4 MiB/s, 150 objects/s recovering 2024-02-12T17:09:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:43.342631+0000 mon.a (mon.0) 1775 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 15 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:43.801 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:43.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:43.907 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:43.940 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:44.894 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:44.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:44.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:09:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:09:45.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:45.590 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:45.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:45.695 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:45.774 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:45.897 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:09:45.897 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:44.642461+0000 mgr.y (mgr.24481) 760 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:44.771102+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:44.771153+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:44.842732+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:44.843236+0000 mgr.y (mgr.24481) 761 : cluster [DBG] pgmap v1049: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 614 op/s; 1701/180378 objects degraded (0.943%); 71695/180378 objects misplaced (39.747%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:44.642461+0000 mgr.y (mgr.24481) 760 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:44.771102+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:44.771153+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:46.036 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:44.842732+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-02-12T17:09:46.037 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:44.843236+0000 mgr.y (mgr.24481) 761 : cluster [DBG] pgmap v1049: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 614 op/s; 1701/180378 objects degraded (0.943%); 71695/180378 objects misplaced (39.747%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:09:46.063 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:46.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:46.075 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:09:46.104 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:09:46.151 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:09:46.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:46.199 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:09:46.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:44.642461+0000 mgr.y (mgr.24481) 760 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:46.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:44.771102+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:09:46.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:44.771153+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:09:46.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:44.842732+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-02-12T17:09:46.205 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:44.843236+0000 mgr.y (mgr.24481) 761 : cluster [DBG] pgmap v1049: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 614 op/s; 1701/180378 objects degraded (0.943%); 71695/180378 objects misplaced (39.747%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:09:46.208 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:09:46.296 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:09:46.340 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:09:46.387 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:09:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:09:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:09:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:09:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:09:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:09:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:09:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:09:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:09:46.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T17:09:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:46.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:46.776 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:46.809 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:46.836 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:46.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:46.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:46.976 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:46.997 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:47.002 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:47.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:47.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:47.140 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:46.021715+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? v1:172.21.15.33:0/2620680434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:47.140 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:46.459116+0000 mon.a (mon.0) 1777 : cluster [WRN] Health check update: Degraded data redundancy: 1701/180378 objects degraded (0.943%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:47.140 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:46.491251+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-02-12T17:09:47.142 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:47.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:47.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:46.021715+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? v1:172.21.15.33:0/2620680434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:47.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:46.459116+0000 mon.a (mon.0) 1777 : cluster [WRN] Health check update: Degraded data redundancy: 1701/180378 objects degraded (0.943%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:47.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:46.491251+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-02-12T17:09:47.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:46.021715+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? v1:172.21.15.33:0/2620680434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:09:47.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:46.459116+0000 mon.a (mon.0) 1777 : cluster [WRN] Health check update: Degraded data redundancy: 1701/180378 objects degraded (0.943%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:47.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:46.491251+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-02-12T17:09:47.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:09:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:48.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:46.844344+0000 mgr.y (mgr.24481) 762 : cluster [DBG] pgmap v1052: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 782 B/s rd, 0 B/s wr, 508 op/s; 1743/178449 objects degraded (0.977%); 70685/178449 objects misplaced (39.611%); 12 MiB/s, 187 objects/s recovering 2024-02-12T17:09:48.263 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:09:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:48.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:09:48.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:46.844344+0000 mgr.y (mgr.24481) 762 : cluster [DBG] pgmap v1052: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 782 B/s rd, 0 B/s wr, 508 op/s; 1743/178449 objects degraded (0.977%); 70685/178449 objects misplaced (39.611%); 12 MiB/s, 187 objects/s recovering 2024-02-12T17:09:48.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:46.844344+0000 mgr.y (mgr.24481) 762 : cluster [DBG] pgmap v1052: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 782 B/s rd, 0 B/s wr, 508 op/s; 1743/178449 objects degraded (0.977%); 70685/178449 objects misplaced (39.611%); 12 MiB/s, 187 objects/s recovering 2024-02-12T17:09:48.367 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:48.395 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:48.497 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:48.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:48.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:09:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:174508 10.a 10.52a07b4a (undecoded) ondisk+write+known_if_redirected e513)", 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:20.635232+0000", 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.143857681, 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83036299300000005, 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.791 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 174508 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.635232+0000", 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.635232+0000", 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.635230+0000", 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7209999999999994e-06 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.635237+0000", 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1199000000000001e-05 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.635249+0000", 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0770000000000008e-06 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.635257+0000", 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2894e-05 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.635300+0000", 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7050000000000002e-06 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.635306+0000", 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83027806299999996 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.465584+0000", 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0550999999999999e-05 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.465595+0000", 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83036299300000005 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(4.13 514/513 [PushOp(4:c8333aab:gc::gc.11:head, version: 65'7, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:c8333aab:gc::gc.11:head@65'7, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:21.550294+0000", 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.228795118000001, 2024-02-12T17:09:48.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046455146000000003, 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.550294+0000", 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.550294+0000", 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.550293+0000", 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5255999999999999e-05 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.550309+0000", 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5109000000000001e-05 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.550324+0000", 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1377e-05 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.550335+0000", 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046133418000000002 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.596468+0000", 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0809999999999999e-06 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.596475+0000", 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00027503199999999998 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.596750+0000", 2024-02-12T17:09:48.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046455146000000003 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "pg_update_log_missing(5.f epoch 515/513 rep_tid 214282 entries 515'24 (0'0) error 5:f95f44c2:::notify.0:head by client.24355.0:2749 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:22.521698+0000", 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.257391048999999, 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046571142000000003, 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.521698+0000", 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.521698+0000", 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.521698+0000", 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3400000000000004e-06 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.521707+0000", 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4817999999999999e-05 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.521722+0000", 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5959999999999993e-06 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.521732+0000", 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043972360000000002 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.565704+0000", 2024-02-12T17:09:48.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025654670000000001 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.568270+0000", 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046571142000000003 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188406 10.e 10:75e8eb83:::benchmark_data_smithi033_160181_object65234:head [delete] snapc 0=[] ondisk+write+known_if_redirected e525)", 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:46.355702+0000", 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 2.4233875770000002, 2024-02-12T17:09:48.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031680985000000002, 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188406 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.355702+0000", 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.355702+0000", 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.355700+0000", 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6449999999999999e-06 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.355707+0000", 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7839999999999992e-06 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.355715+0000", 2024-02-12T17:09:48.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3590000000000001e-06 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.355720+0000", 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8595999999999999e-05 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.355748+0000", 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7854000000000004e-05 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.355826+0000", 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024154100000000001 2024-02-12T17:09:48.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.356068+0000", 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011711759999999999 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.357239+0000", 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027950739999999998 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.360034+0000", 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.027258032000000001 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.387292+0000", 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1740999999999998e-05 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.387344+0000", 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9141999999999999e-05 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.387383+0000", 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031680985000000002 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.807 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188438 10.e 10:77354c7c:::benchmark_data_smithi033_160181_object65266:head [delete] snapc 0=[] ondisk+write+known_if_redirected e525)", 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:46.414489+0000", 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 2.3646009499999998, 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014366413, 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.808 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188438 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414489+0000", 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414489+0000", 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414486+0000", 2024-02-12T17:09:48.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439999999999999e-06 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414495+0000", 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.323e-05 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414508+0000", 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8979999999999996e-06 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414516+0000", 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9839999999999997e-05 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414566+0000", 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103836 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414670+0000", 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018472599999999999 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.414854+0000", 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011620949999999999 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.416017+0000", 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071467999999999998 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.416731+0000", 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012024491 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.428756+0000", 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5599000000000001e-05 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.428811+0000", 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3708000000000002e-05 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.428855+0000", 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014366413 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.813 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188447 10.e 10:7787800b:::benchmark_data_smithi033_160181_object65275:head [delete] snapc 0=[] ondisk+write+known_if_redirected e525)", 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:46.425887+0000", 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 2.353202574, 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015272699000000001, 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188447 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.425887+0000", 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.425887+0000", 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.425885+0000", 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7009999999999996e-06 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.425892+0000", 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4708e-05 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.425907+0000", 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7790000000000001e-06 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.425913+0000", 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9018999999999999e-05 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.425962+0000", 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012567700000000001 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.426088+0000", 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024991599999999998 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.426337+0000", 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010234109999999999 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.427361+0000", 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012178950000000001 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.428579+0000", 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012527464 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.441106+0000", 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.934e-05 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.441136+0000", 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4097e-05 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.441160+0000", 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015272699000000001 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188504 10.e 10:70d5c3a2:::benchmark_data_smithi033_160181_object65332:head [delete] snapc 0=[] ondisk+write+known_if_redirected e525)", 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:46.519412+0000", 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 2.259677103, 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015046123, 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188504 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519412+0000", 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519412+0000", 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519410+0000", 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5459999999999992e-06 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519418+0000", 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.376e-05 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519432+0000", 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7499999999999997e-06 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519439+0000", 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4503999999999999e-05 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519483+0000", 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000150216 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519634+0000", 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025192200000000002 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.519885+0000", 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033655619999999999 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.523251+0000", 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2570000000000001e-06 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.523258+0000", 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011117303 2024-02-12T17:09:48.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.534376+0000", 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6391999999999999e-05 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.534422+0000", 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6551000000000003e-05 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.534459+0000", 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015046123 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.824 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188691 10.e 10:74fffa7b:::benchmark_data_smithi033_160181_object65519:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:46.887687+0000", 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.8914026100000001, 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016065573999999999, 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188691 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.825 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.887687+0000", 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.887687+0000", 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.887684+0000", 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0818999999999999e-05 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.887695+0000", 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8150000000000001e-05 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.887713+0000", 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7150000000000004e-06 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.887722+0000", 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9016e-05 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.887771+0000", 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013943199999999999 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.887910+0000", 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00028462399999999998 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.888195+0000", 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001328816 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.889524+0000", 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052652799999999996 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.890050+0000", 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013622853000000001 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.903673+0000", 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6557999999999998e-05 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.903709+0000", 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2997999999999997e-05 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.903752+0000", 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016065573999999999 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188741 10.e 10:701bfa5f:::benchmark_data_smithi033_160181_object65569:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:46.980218+0000", 2024-02-12T17:09:48.830 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.798871823, 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020079222000000001, 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188741 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980218+0000", 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980218+0000", 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999962 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980214+0000", 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3935e-05 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980228+0000", 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9979999999999998e-05 2024-02-12T17:09:48.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980248+0000", 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6239999999999997e-06 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980257+0000", 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4283000000000002e-05 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980312+0000", 2024-02-12T17:09:48.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014204300000000001 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980454+0000", 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000285641 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.980739+0000", 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044837540000000004 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.985223+0000", 2024-02-12T17:09:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044489519999999999 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:46.989672+0000", 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010534905000000001 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.000207+0000", 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1560000000000001e-05 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.000259+0000", 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8389999999999997e-05 2024-02-12T17:09:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.000297+0000", 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020079222000000001 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188792 10.e 10:7122d4b5:::benchmark_data_smithi033_160181_object65620:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:47.049746+0000", 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.729343788, 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016876417000000001, 2024-02-12T17:09:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188792 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.049746+0000", 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.049746+0000", 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.049743+0000", 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5650000000000004e-06 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.049751+0000", 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0105000000000001e-05 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.049761+0000", 2024-02-12T17:09:48.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3459999999999999e-06 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.049767+0000", 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6288999999999997e-05 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.049814+0000", 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012491999999999999 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.049939+0000", 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026141800000000002 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.050200+0000", 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012545690000000001 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.051455+0000", 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030958500000000003 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.051764+0000", 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014789218 2024-02-12T17:09:48.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.066553+0000", 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1943999999999998e-05 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.066585+0000", 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6863000000000003e-05 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.066622+0000", 2024-02-12T17:09:48.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016876417000000001 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188844 10.e 10:70a13465:::benchmark_data_smithi033_160181_object65672:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:47.132612+0000", 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.646477295, 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020931745000000002, 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.842 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188844 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.132612+0000", 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.132612+0000", 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.132610+0000", 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0450000000000003e-06 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.132617+0000", 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7949999999999996e-06 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.132626+0000", 2024-02-12T17:09:48.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.656e-06 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.132630+0000", 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7840000000000003e-05 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.132678+0000", 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138407 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.132817+0000", 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00028514799999999999 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.133102+0000", 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0032579110000000001 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.136360+0000", 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00077181899999999998 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.137132+0000", 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016311211999999999 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.153443+0000", 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3721000000000003e-05 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.153497+0000", 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7411999999999999e-05 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.153544+0000", 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020931745000000002 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188873 10.e 10:721e9298:::benchmark_data_smithi033_160181_object65701:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:47.177273+0000", 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.6018161420000001, 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017018390000000001, 2024-02-12T17:09:48.848 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188873 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177273+0000", 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177273+0000", 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177272+0000", 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0719999999999996e-06 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177278+0000", 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0183999999999999e-05 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177288+0000", 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0759999999999999e-06 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177294+0000", 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6335999999999998e-05 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177341+0000", 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0001138 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177454+0000", 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023298599999999999 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.177687+0000", 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025260690000000001 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.180213+0000", 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001091306 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.181305+0000", 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012930476999999999 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.194235+0000", 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3016999999999999e-05 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.194268+0000", 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3592e-05 2024-02-12T17:09:48.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.194292+0000", 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017018390000000001 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24397.0:906 4.5 4:a7f90313:reshard::reshard.0000000012:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:47.248870+0000", 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.530219204, 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020958178000000001, 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.854 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24397", 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 906 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.248870+0000", 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.248870+0000", 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.248868+0000", 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4050000000000003e-06 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.248876+0000", 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9089999999999995e-06 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.248886+0000", 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.605e-06 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.248890+0000", 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2863e-05 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.248923+0000", 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5891999999999999e-05 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.248989+0000", 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011515600000000001 2024-02-12T17:09:48.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 3,7", 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.249104+0000", 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00027904699999999999 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.249383+0000", 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00031707399999999998 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.249700+0000", 2024-02-12T17:09:48.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020033127000000001 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.269733+0000", 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1291000000000001e-05 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.269785+0000", 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.384e-05 2024-02-12T17:09:48.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.269828+0000", 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020958178000000001 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:188947 10.e 10:71cbdedf:::benchmark_data_smithi033_160181_object65775:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.861 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:47.283625+0000", 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.495464455, 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016264246999999999, 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 188947 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.283625+0000", 2024-02-12T17:09:48.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.283625+0000", 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.283624+0000", 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2049999999999998e-06 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.283627+0000", 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4620000000000004e-06 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.283633+0000", 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999997e-06 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.283637+0000", 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025957509999999999 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.286233+0000", 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7683999999999998e-05 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.286300+0000", 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114749 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.286415+0000", 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00124799 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.287663+0000", 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002116794 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.289780+0000", 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010023914 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.299804+0000", 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6699000000000001e-05 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.299851+0000", 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8751999999999998e-05 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.299889+0000", 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016264246999999999 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:189065 10.e 10:71aded4d:::benchmark_data_smithi033_160181_object65893:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:47.514775+0000", 2024-02-12T17:09:48.867 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.264314186, 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015148327, 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 189065 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.514775+0000", 2024-02-12T17:09:48.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.514775+0000", 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.514772+0000", 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1329999999999999e-06 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.514781+0000", 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2968e-05 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.514794+0000", 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0759999999999999e-06 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.514800+0000", 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1239000000000003e-05 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.514841+0000", 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6192000000000005e-05 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.514917+0000", 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020191 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.515119+0000", 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018018730000000001 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.516921+0000", 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024280800000000001 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.517164+0000", 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012676936 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.529841+0000", 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.634e-05 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.529887+0000", 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6702999999999997e-05 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:47.529924+0000", 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015148327 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:189495 10.e 10:74b36cb0:::benchmark_data_smithi033_160181_object66323:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:48.217936+0000", 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.56115357899999996, 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20467887000000001, 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 189495 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.217936+0000", 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.217936+0000", 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.217934+0000", 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8980000000000001e-06 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.217940+0000", 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.704e-06 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.217949+0000", 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.195e-06 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.217955+0000", 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7153e-05 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.218012+0000", 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012355499999999999 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.218136+0000", 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025502000000000001 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.218391+0000", 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077878 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.219469+0000", 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00055754600000000002 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.220026+0000", 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20251651100000001 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.422543+0000", 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3241000000000002e-05 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.422586+0000", 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8965000000000001e-05 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.422615+0000", 2024-02-12T17:09:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20467887000000001 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:189631 10.e 10:7052c5aa:::benchmark_data_smithi033_160181_object66459:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:48.586459+0000", 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.192630788, 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019735144, 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.880 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 189631 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586459+0000", 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586459+0000", 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586457+0000", 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7119999999999999e-06 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586462+0000", 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2530000000000004e-06 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586470+0000", 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8200000000000002e-06 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586476+0000", 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3525000000000002e-05 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586510+0000", 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7713999999999995e-05 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586587+0000", 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016958500000000001 2024-02-12T17:09:48.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.586757+0000", 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010660520000000001 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.587823+0000", 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00063078500000000003 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.588454+0000", 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017657020999999998 2024-02-12T17:09:48.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.606111+0000", 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3340999999999998e-05 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.606144+0000", 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9858000000000003e-05 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.606194+0000", 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019735144 2024-02-12T17:09:48.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:189708 10.e 10:72c188d8:::benchmark_data_smithi033_160181_object66536:head [delete] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:48.700734+0000", 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.078355431000000003, 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020986687, 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 189708 2024-02-12T17:09:48.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.700734+0000", 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.700734+0000", 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.700732+0000", 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7519999999999996e-06 2024-02-12T17:09:48.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.700739+0000", 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3905e-05 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.700753+0000", 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3730000000000004e-06 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.700761+0000", 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3852e-05 2024-02-12T17:09:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.700795+0000", 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2227999999999999e-05 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.700877+0000", 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017874199999999999 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.701056+0000", 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029787210000000001 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.704034+0000", 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023057949999999998 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.706340+0000", 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015284192 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.721624+0000", 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1696000000000001e-05 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.721676+0000", 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4790999999999999e-05 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.721721+0000", 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020986687 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24397.0:1003 4.1f 4:f93251ce:::obj_delete_at_hint.0000000042:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:48.729669+0000", 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.049420556999999997, 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016162422999999999, 2024-02-12T17:09:48.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24397", 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1003 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729669+0000", 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729669+0000", 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729667+0000", 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2670000000000006e-06 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729675+0000", 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1603000000000001e-05 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729687+0000", 2024-02-12T17:09:48.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9339999999999997e-06 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729692+0000", 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8355999999999997e-05 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729730+0000", 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0506e-05 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729791+0000", 2024-02-12T17:09:48.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000126709 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,5", 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.729917+0000", 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00029999099999999999 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.730217+0000", 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025319699999999998 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.730470+0000", 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015272841000000001 2024-02-12T17:09:48.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.745743+0000", 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2163000000000002e-05 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.745795+0000", 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5951999999999997e-05 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.745831+0000", 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016162422999999999 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.896 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24397.0:1004 4.5 4:a77dac5b:::obj_delete_at_hint.0000000043:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:48.746340+0000", 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.032749256999999997, 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014765429, 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24397", 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/1589842856", 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1004 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746340+0000", 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746340+0000", 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746338+0000", 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2819999999999996e-06 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746347+0000", 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2092e-05 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746359+0000", 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2219999999999999e-06 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746365+0000", 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6702000000000001e-05 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:48.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746412+0000", 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2855000000000005e-05 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746494+0000", 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018280400000000001 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 3,7", 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.746677+0000", 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000480691 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.747158+0000", 2024-02-12T17:09:48.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00040358400000000002 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.747561+0000", 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013453708999999999 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.761015+0000", 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7376999999999997e-05 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.761063+0000", 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3121999999999998e-05 2024-02-12T17:09:48.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:48.761106+0000", 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014765429 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:48.902 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:09:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:48.903 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:09:48.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:09:49.004 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:49.084 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:49.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:49.770 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:49.801 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:49.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:48.845276+0000 mgr.y (mgr.24481) 763 : cluster [DBG] pgmap v1053: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 678 op/s; 1743/172689 objects degraded (1.009%); 67796/172689 objects misplaced (39.259%); 15 MiB/s, 244 objects/s recovering 2024-02-12T17:09:50.004 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:50.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:50.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:50.182 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:48.845276+0000 mgr.y (mgr.24481) 763 : cluster [DBG] pgmap v1053: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 678 op/s; 1743/172689 objects degraded (1.009%); 67796/172689 objects misplaced (39.259%); 15 MiB/s, 244 objects/s recovering 2024-02-12T17:09:50.188 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:48.845276+0000 mgr.y (mgr.24481) 763 : cluster [DBG] pgmap v1053: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 678 op/s; 1743/172689 objects degraded (1.009%); 67796/172689 objects misplaced (39.259%); 15 MiB/s, 244 objects/s recovering 2024-02-12T17:09:50.191 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:50.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:50.438 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:50.527 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T17:09:50.553 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:50.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:09:50.864 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:51.279 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:09:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:50] "GET /metrics HTTP/1.1" 200 176676 "" "Prometheus/2.33.4" 2024-02-12T17:09:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:50.851712+0000 mgr.y (mgr.24481) 764 : cluster [DBG] pgmap v1054: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 630 op/s; 1778/171804 objects degraded (1.035%); 67645/171804 objects misplaced (39.373%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:09:52.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:51.460678+0000 mon.a (mon.0) 1779 : cluster [WRN] Health check update: Degraded data redundancy: 1778/171804 objects degraded (1.035%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:52.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:50.851712+0000 mgr.y (mgr.24481) 764 : cluster [DBG] pgmap v1054: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 630 op/s; 1778/171804 objects degraded (1.035%); 67645/171804 objects misplaced (39.373%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:09:52.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:51.460678+0000 mon.a (mon.0) 1779 : cluster [WRN] Health check update: Degraded data redundancy: 1778/171804 objects degraded (1.035%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:52.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:50.851712+0000 mgr.y (mgr.24481) 764 : cluster [DBG] pgmap v1054: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 630 op/s; 1778/171804 objects degraded (1.035%); 67645/171804 objects misplaced (39.373%); 14 MiB/s, 230 objects/s recovering 2024-02-12T17:09:52.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:51.460678+0000 mon.a (mon.0) 1779 : cluster [WRN] Health check update: Degraded data redundancy: 1778/171804 objects degraded (1.035%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:52.513 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 3 pgp_num to 3 2024-02-12T17:09:52.665 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:52.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:52.696 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:09:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:09:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:09:52.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:52.861 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:52.200094+0000 mon.c (mon.2) 260 : audit [INF] from='client.? v1:172.21.15.33:0/3062213181' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-12T17:09:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:52.200601+0000 mon.a (mon.0) 1780 : audit [INF] from='client.35879 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-12T17:09:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:52.484446+0000 mon.a (mon.0) 1781 : audit [INF] from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-12T17:09:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:52.484511+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-02-12T17:09:53.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:52.200094+0000 mon.c (mon.2) 260 : audit [INF] from='client.? v1:172.21.15.33:0/3062213181' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-12T17:09:53.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:52.200601+0000 mon.a (mon.0) 1780 : audit [INF] from='client.35879 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-12T17:09:53.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:52.484446+0000 mon.a (mon.0) 1781 : audit [INF] from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-12T17:09:53.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:52.484511+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-02-12T17:09:53.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:52.200094+0000 mon.c (mon.2) 260 : audit [INF] from='client.? v1:172.21.15.33:0/3062213181' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-12T17:09:53.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:52.200601+0000 mon.a (mon.0) 1780 : audit [INF] from='client.35879 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-12T17:09:53.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:52.484446+0000 mon.a (mon.0) 1781 : audit [INF] from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-12T17:09:53.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:52.484511+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-02-12T17:09:53.540 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:09:53.690 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:09:53.690 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:09:53.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.690 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:192455 10.1c 10:38312923:::benchmark_data_smithi033_160181_object69283:head [delete] snapc 0=[] ondisk+write+known_if_redirected e528)", 2024-02-12T17:09:53.690 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:53.667443+0000", 2024-02-12T17:09:53.690 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.020544217, 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020573866999999999, 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 192455 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667443+0000", 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667443+0000", 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667441+0000", 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8390000000000007e-06 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667448+0000", 2024-02-12T17:09:53.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3358e-05 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667462+0000", 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3769999999999998e-06 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667469+0000", 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3113000000000004e-05 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667532+0000", 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000154537 2024-02-12T17:09:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667687+0000", 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00032902099999999998 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,3,6", 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.668016+0000", 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002313875 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.670330+0000", 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0032200369999999998 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.673550+0000", 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0050664960000000002 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.678616+0000", 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030762580000000001 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.681692+0000", 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014249632999999999 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:53.695 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:192458 10.1c 10:3f22b29d:::benchmark_data_smithi033_160181_object69286:head [delete] snapc 0=[] ondisk+write+known_if_redirected e528)", 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:53.667485+0000", 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.020501658999999998, 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020664088000000001, 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 192458 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:53.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667485+0000", 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667485+0000", 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667485+0000", 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7929999999999999e-06 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667488+0000", 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7869999999999998e-06 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667494+0000", 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.284e-06 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.667498+0000", 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001653271 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:53.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.669152+0000", 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015202099999999999 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.669304+0000", 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023037000000000001 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,3,6", 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.669534+0000", 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000896132 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.670430+0000", 2024-02-12T17:09:53.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003145629 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.673576+0000", 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0050711879999999999 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.678647+0000", 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011161684999999999 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.700 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:192460 10.1c 10:3ae3ed76:::benchmark_data_smithi033_160181_object69288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e528)", 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:53.683491+0000", 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0044962309999999998, 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047792609999999999, 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 192460 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683491+0000", 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683491+0000", 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683489+0000", 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4549999999999998e-06 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683496+0000", 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2615e-05 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683509+0000", 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8190000000000003e-06 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683515+0000", 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3993999999999999e-05 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683559+0000", 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7349999999999995e-05 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683647+0000", 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030426599999999998 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,3,6", 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683951+0000", 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046032400000000002 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:192462 10.1 10:800b3283:::benchmark_data_smithi033_160181_object69290:head [delete] snapc 0=[] ondisk+write+known_if_redirected e528)", 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:53.683532+0000", 2024-02-12T17:09:53.704 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0044550129999999999, 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0048378630000000004, 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 192462 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683532+0000", 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683532+0000", 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683531+0000", 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6509999999999999e-06 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683535+0000", 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6189999999999999e-06 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:53.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683541+0000", 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1500000000000001e-06 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683545+0000", 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6783000000000003e-05 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683591+0000", 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013085399999999999 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.683722+0000", 2024-02-12T17:09:53.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000312458 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.684035+0000", 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037979699999999999 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.687833+0000", 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004300768 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.25603.0:192463 10.9 e528/480 10:94aa8b1f:::benchmark_data_smithi033_160181_object69291:head v 528'10044, mlcod=527'10042)", 2024-02-12T17:09:53.708 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:53.686829+0000", 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.001157621, 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016321280000000001, 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.686829+0000", 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.686829+0000", 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:09:53.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.686828+0000", 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3690000000000001e-06 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.686835+0000", 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.382e-05 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.686849+0000", 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1630000000000001e-06 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.686855+0000", 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7326000000000001e-05 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.686903+0000", 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9772999999999999e-05 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.686922+0000", 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3010999999999994e-05 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:09:53.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:09:53.712 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T17:09:53.712 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 5 2024-02-12T17:09:53.712 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:09:53.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:52.852376+0000 mgr.y (mgr.24481) 765 : cluster [DBG] pgmap v1056: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 480 op/s; 1795/169926 objects degraded (1.056%); 66646/169926 objects misplaced (39.221%); 8.8 MiB/s, 141 objects/s recovering 2024-02-12T17:09:53.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:53.193771+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.33:0/2444241037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:09:53.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:53.196012+0000 mon.a (mon.0) 1783 : audit [INF] from='client.25768 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:09:53.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:53.481841+0000 mon.a (mon.0) 1784 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:53.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:53.498297+0000 mon.a (mon.0) 1785 : audit [INF] from='client.25768 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:09:53.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:53.498361+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-02-12T17:09:53.983 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:53.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:54.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:09:54.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:52.852376+0000 mgr.y (mgr.24481) 765 : cluster [DBG] pgmap v1056: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 480 op/s; 1795/169926 objects degraded (1.056%); 66646/169926 objects misplaced (39.221%); 8.8 MiB/s, 141 objects/s recovering 2024-02-12T17:09:54.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:53.193771+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.33:0/2444241037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:09:54.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:53.196012+0000 mon.a (mon.0) 1783 : audit [INF] from='client.25768 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:09:54.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:53.481841+0000 mon.a (mon.0) 1784 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:54.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:53.498297+0000 mon.a (mon.0) 1785 : audit [INF] from='client.25768 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:09:54.119 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:53.498361+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-02-12T17:09:54.121 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:54.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:52.852376+0000 mgr.y (mgr.24481) 765 : cluster [DBG] pgmap v1056: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 13 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 2 active+remapped, 115 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 480 op/s; 1795/169926 objects degraded (1.056%); 66646/169926 objects misplaced (39.221%); 8.8 MiB/s, 141 objects/s recovering 2024-02-12T17:09:54.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:53.193771+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.33:0/2444241037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:09:54.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:53.196012+0000 mon.a (mon.0) 1783 : audit [INF] from='client.25768 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:09:54.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:53.481841+0000 mon.a (mon.0) 1784 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:09:54.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:53.498297+0000 mon.a (mon.0) 1785 : audit [INF] from='client.25768 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:09:54.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:53.498361+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-02-12T17:09:54.486 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:09:54.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:54.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:09:54.591 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:54.676 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:54.707 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:54.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:09:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:09:55.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T17:09:55.307 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:55.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:55.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:55.462 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:09:55.537 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:55.577 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:55.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:55.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:09:55.750 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:55.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:54.509943+0000 mon.a (mon.0) 1787 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-02-12T17:09:55.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:54.653477+0000 mgr.y (mgr.24481) 766 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:55.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:54.854983+0000 mgr.y (mgr.24481) 767 : cluster [DBG] pgmap v1059: 148 pgs: 1 peering, 6 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 52 KiB/s rd, 0 B/s wr, 604 op/s; 1837/163347 objects degraded (1.125%); 63342/163347 objects misplaced (38.778%); 13 MiB/s, 208 objects/s recovering 2024-02-12T17:09:55.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:54.509943+0000 mon.a (mon.0) 1787 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-02-12T17:09:55.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:54.653477+0000 mgr.y (mgr.24481) 766 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:55.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:54.854983+0000 mgr.y (mgr.24481) 767 : cluster [DBG] pgmap v1059: 148 pgs: 1 peering, 6 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 52 KiB/s rd, 0 B/s wr, 604 op/s; 1837/163347 objects degraded (1.125%); 63342/163347 objects misplaced (38.778%); 13 MiB/s, 208 objects/s recovering 2024-02-12T17:09:55.863 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:09:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:55.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:09:55.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:55.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:54.509943+0000 mon.a (mon.0) 1787 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-02-12T17:09:55.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:54.653477+0000 mgr.y (mgr.24481) 766 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:09:55.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:54.854983+0000 mgr.y (mgr.24481) 767 : cluster [DBG] pgmap v1059: 148 pgs: 1 peering, 6 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 52 KiB/s rd, 0 B/s wr, 604 op/s; 1837/163347 objects degraded (1.125%); 63342/163347 objects misplaced (38.778%); 13 MiB/s, 208 objects/s recovering 2024-02-12T17:09:56.014 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:56.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:09:56.772 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:56.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:56.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:56.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:55.510593+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:56.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:56.463530+0000 mon.a (mon.0) 1789 : cluster [WRN] Health check update: Degraded data redundancy: 1837/163347 objects degraded (1.125%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:56.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:56.493688+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-02-12T17:09:56.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:55.510593+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:56.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:56.463530+0000 mon.a (mon.0) 1789 : cluster [WRN] Health check update: Degraded data redundancy: 1837/163347 objects degraded (1.125%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:56.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:56.493688+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-02-12T17:09:56.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:56.916 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:55.510593+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:09:56.916 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:56.463530+0000 mon.a (mon.0) 1789 : cluster [WRN] Health check update: Degraded data redundancy: 1837/163347 objects degraded (1.125%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:09:56.916 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:56.493688+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-02-12T17:09:56.919 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:57.779 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:09:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:57.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:09:57.884 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:09:57.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:56.857657+0000 mgr.y (mgr.24481) 768 : cluster [DBG] pgmap v1061: 148 pgs: 1 peering, 7 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 50 KiB/s rd, 0 B/s wr, 737 op/s; 1866/161346 objects degraded (1.157%); 62292/161346 objects misplaced (38.608%); 17 MiB/s, 271 objects/s recovering 2024-02-12T17:09:57.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:56.857657+0000 mgr.y (mgr.24481) 768 : cluster [DBG] pgmap v1061: 148 pgs: 1 peering, 7 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 50 KiB/s rd, 0 B/s wr, 737 op/s; 1866/161346 objects degraded (1.157%); 62292/161346 objects misplaced (38.608%); 17 MiB/s, 271 objects/s recovering 2024-02-12T17:09:57.926 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:57.969 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:56.857657+0000 mgr.y (mgr.24481) 768 : cluster [DBG] pgmap v1061: 148 pgs: 1 peering, 7 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 10 active+remapped+backfill_wait, 116 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 50 KiB/s rd, 0 B/s wr, 737 op/s; 1866/161346 objects degraded (1.157%); 62292/161346 objects misplaced (38.608%); 17 MiB/s, 271 objects/s recovering 2024-02-12T17:09:58.588 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:09:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:58.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:09:58.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:09:58.782 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:58.836 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:09:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:09:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:09:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:09:58.837 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:09:58.861 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:57.596573+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-02-12T17:09:58.873 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:09:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:58.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:09:58.887 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:57.596573+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-02-12T17:09:58.887 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:57.596573+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-02-12T17:09:58.983 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:09:59.064 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:59.215 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:09:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:09:59.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:09:59.324 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:09:59.361 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:09:59.435 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:09:59.448 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:09:59.473 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:09:59.518 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:09:59.601 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:09:59.602 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:09:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:09:59.673 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:09:59.738 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:09:59.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:09:59.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:09:59.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:09:59.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:09:59.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:09:59.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:09:59.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:09:59.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:09:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:58.602282+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-02-12T17:09:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:09:58.858370+0000 mgr.y (mgr.24481) 769 : cluster [DBG] pgmap v1064: 148 pgs: 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 117 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 4.4 KiB/s rd, 0 B/s wr, 591 op/s; 1866/155724 objects degraded (1.198%); 58868/155724 objects misplaced (37.803%); 24 MiB/s, 380 objects/s recovering 2024-02-12T17:09:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:09:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:09:59.393984+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? v1:172.21.15.33:0/2924068227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:00.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:58.602282+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-02-12T17:10:00.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:09:58.858370+0000 mgr.y (mgr.24481) 769 : cluster [DBG] pgmap v1064: 148 pgs: 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 117 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 4.4 KiB/s rd, 0 B/s wr, 591 op/s; 1866/155724 objects degraded (1.198%); 58868/155724 objects misplaced (37.803%); 24 MiB/s, 380 objects/s recovering 2024-02-12T17:10:00.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:09:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:09:59.393984+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? v1:172.21.15.33:0/2924068227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:00.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:58.602282+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-02-12T17:10:00.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:09:58.858370+0000 mgr.y (mgr.24481) 769 : cluster [DBG] pgmap v1064: 148 pgs: 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 117 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 4.4 KiB/s rd, 0 B/s wr, 591 op/s; 1866/155724 objects degraded (1.198%); 58868/155724 objects misplaced (37.803%); 24 MiB/s, 380 objects/s recovering 2024-02-12T17:10:00.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:09:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:09:59.393984+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? v1:172.21.15.33:0/2924068227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:00.068 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T17:10:00.133 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:00.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:00.186 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:10:00.186 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:10:00.186 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-12T17:10:00.186 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-12T17:10:00.186 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.1 2024-02-12T17:10:00.238 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:00.359 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:00.494 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:10:00.531 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:10:00 smithi033 systemd[1]: Stopping Ceph osd.1 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:10:00.531 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:10:00 smithi033 bash[174371]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:10:00.633 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:00.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:00.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:00.738 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:00.808 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:00.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000126+0000 mon.a (mon.0) 1793 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull; Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized 2024-02-12T17:10:00.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000158+0000 mon.a (mon.0) 1794 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T17:10:00.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000169+0000 mon.a (mon.0) 1795 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull 2024-02-12T17:10:00.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000179+0000 mon.a (mon.0) 1796 : cluster [WRN] pg 5.f is active+remapped+backfill_toofull, acting [5,2,3] 2024-02-12T17:10:00.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000188+0000 mon.a (mon.0) 1797 : cluster [WRN] pg 5.13 is active+remapped+backfill_wait+backfill_toofull, acting [3,5,7] 2024-02-12T17:10:00.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000198+0000 mon.a (mon.0) 1798 : cluster [WRN] pg 10.1 is active+remapped+backfill_toofull, acting [7,2,5] 2024-02-12T17:10:00.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000207+0000 mon.a (mon.0) 1799 : cluster [WRN] pg 10.2 is active+remapped+backfill_wait+backfill_toofull, acting [3,7,5] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000217+0000 mon.a (mon.0) 1800 : cluster [WRN] pg 10.7 is active+remapped+backfill_toofull, acting [2,3,5] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000226+0000 mon.a (mon.0) 1801 : cluster [WRN] pg 10.a is active+remapped+backfill_toofull, acting [2,5,3] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000236+0000 mon.a (mon.0) 1802 : cluster [WRN] pg 10.b is active+remapped+backfill_toofull, acting [2,7,3] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000246+0000 mon.a (mon.0) 1803 : cluster [WRN] pg 10.12 is active+remapped+backfill_toofull, acting [5,7,3] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000256+0000 mon.a (mon.0) 1804 : cluster [WRN] pg 10.13 is active+remapped+backfill_wait+backfill_toofull, acting [2,7,3] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000265+0000 mon.a (mon.0) 1805 : cluster [WRN] pg 10.14 is active+remapped+backfill_wait+backfill_toofull, acting [3,2,7] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000275+0000 mon.a (mon.0) 1806 : cluster [WRN] pg 10.17 is active+remapped+backfill_toofull, acting [3,2,5] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000285+0000 mon.a (mon.0) 1807 : cluster [WRN] pg 10.1a is active+remapped+backfill_toofull, acting [2,5,3] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000294+0000 mon.a (mon.0) 1808 : cluster [WRN] pg 10.20 is active+remapped+backfill_toofull, acting [2,7,5] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000313+0000 mon.a (mon.0) 1809 : cluster [WRN] pg 10.22 is active+remapped+backfill_toofull, acting [5,7,3] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000322+0000 mon.a (mon.0) 1810 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000331+0000 mon.a (mon.0) 1811 : cluster [WRN] pg 10.15 is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [3,2] 2024-02-12T17:10:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.000340+0000 mon.a (mon.0) 1812 : cluster [WRN] pg 10.18 is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [0,2] 2024-02-12T17:10:00.884 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000126+0000 mon.a (mon.0) 1793 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull; Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized 2024-02-12T17:10:00.884 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000158+0000 mon.a (mon.0) 1794 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T17:10:00.884 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000169+0000 mon.a (mon.0) 1795 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull 2024-02-12T17:10:00.884 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000179+0000 mon.a (mon.0) 1796 : cluster [WRN] pg 5.f is active+remapped+backfill_toofull, acting [5,2,3] 2024-02-12T17:10:00.884 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000188+0000 mon.a (mon.0) 1797 : cluster [WRN] pg 5.13 is active+remapped+backfill_wait+backfill_toofull, acting [3,5,7] 2024-02-12T17:10:00.884 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000198+0000 mon.a (mon.0) 1798 : cluster [WRN] pg 10.1 is active+remapped+backfill_toofull, acting [7,2,5] 2024-02-12T17:10:00.884 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000207+0000 mon.a (mon.0) 1799 : cluster [WRN] pg 10.2 is active+remapped+backfill_wait+backfill_toofull, acting [3,7,5] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000217+0000 mon.a (mon.0) 1800 : cluster [WRN] pg 10.7 is active+remapped+backfill_toofull, acting [2,3,5] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000226+0000 mon.a (mon.0) 1801 : cluster [WRN] pg 10.a is active+remapped+backfill_toofull, acting [2,5,3] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000236+0000 mon.a (mon.0) 1802 : cluster [WRN] pg 10.b is active+remapped+backfill_toofull, acting [2,7,3] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000246+0000 mon.a (mon.0) 1803 : cluster [WRN] pg 10.12 is active+remapped+backfill_toofull, acting [5,7,3] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000256+0000 mon.a (mon.0) 1804 : cluster [WRN] pg 10.13 is active+remapped+backfill_wait+backfill_toofull, acting [2,7,3] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000265+0000 mon.a (mon.0) 1805 : cluster [WRN] pg 10.14 is active+remapped+backfill_wait+backfill_toofull, acting [3,2,7] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000275+0000 mon.a (mon.0) 1806 : cluster [WRN] pg 10.17 is active+remapped+backfill_toofull, acting [3,2,5] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000285+0000 mon.a (mon.0) 1807 : cluster [WRN] pg 10.1a is active+remapped+backfill_toofull, acting [2,5,3] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000294+0000 mon.a (mon.0) 1808 : cluster [WRN] pg 10.20 is active+remapped+backfill_toofull, acting [2,7,5] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000313+0000 mon.a (mon.0) 1809 : cluster [WRN] pg 10.22 is active+remapped+backfill_toofull, acting [5,7,3] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000322+0000 mon.a (mon.0) 1810 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000331+0000 mon.a (mon.0) 1811 : cluster [WRN] pg 10.15 is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [3,2] 2024-02-12T17:10:00.885 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.000340+0000 mon.a (mon.0) 1812 : cluster [WRN] pg 10.18 is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [0,2] 2024-02-12T17:10:00.886 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:10:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:00] "GET /metrics HTTP/1.1" 200 176676 "" "Prometheus/2.33.4" 2024-02-12T17:10:00.965 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:00.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000126+0000 mon.a (mon.0) 1793 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull; Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000158+0000 mon.a (mon.0) 1794 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000169+0000 mon.a (mon.0) 1795 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 14 pgs backfill_toofull 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000179+0000 mon.a (mon.0) 1796 : cluster [WRN] pg 5.f is active+remapped+backfill_toofull, acting [5,2,3] 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000188+0000 mon.a (mon.0) 1797 : cluster [WRN] pg 5.13 is active+remapped+backfill_wait+backfill_toofull, acting [3,5,7] 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000198+0000 mon.a (mon.0) 1798 : cluster [WRN] pg 10.1 is active+remapped+backfill_toofull, acting [7,2,5] 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000207+0000 mon.a (mon.0) 1799 : cluster [WRN] pg 10.2 is active+remapped+backfill_wait+backfill_toofull, acting [3,7,5] 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000217+0000 mon.a (mon.0) 1800 : cluster [WRN] pg 10.7 is active+remapped+backfill_toofull, acting [2,3,5] 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000226+0000 mon.a (mon.0) 1801 : cluster [WRN] pg 10.a is active+remapped+backfill_toofull, acting [2,5,3] 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000236+0000 mon.a (mon.0) 1802 : cluster [WRN] pg 10.b is active+remapped+backfill_toofull, acting [2,7,3] 2024-02-12T17:10:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000246+0000 mon.a (mon.0) 1803 : cluster [WRN] pg 10.12 is active+remapped+backfill_toofull, acting [5,7,3] 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000256+0000 mon.a (mon.0) 1804 : cluster [WRN] pg 10.13 is active+remapped+backfill_wait+backfill_toofull, acting [2,7,3] 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000265+0000 mon.a (mon.0) 1805 : cluster [WRN] pg 10.14 is active+remapped+backfill_wait+backfill_toofull, acting [3,2,7] 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000275+0000 mon.a (mon.0) 1806 : cluster [WRN] pg 10.17 is active+remapped+backfill_toofull, acting [3,2,5] 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000285+0000 mon.a (mon.0) 1807 : cluster [WRN] pg 10.1a is active+remapped+backfill_toofull, acting [2,5,3] 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000294+0000 mon.a (mon.0) 1808 : cluster [WRN] pg 10.20 is active+remapped+backfill_toofull, acting [2,7,5] 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000313+0000 mon.a (mon.0) 1809 : cluster [WRN] pg 10.22 is active+remapped+backfill_toofull, acting [5,7,3] 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000322+0000 mon.a (mon.0) 1810 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000331+0000 mon.a (mon.0) 1811 : cluster [WRN] pg 10.15 is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [3,2] 2024-02-12T17:10:00.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.000340+0000 mon.a (mon.0) 1812 : cluster [WRN] pg 10.18 is stuck undersized for 106s, current state active+recovering+undersized+degraded+remapped, last acting [0,2] 2024-02-12T17:10:01.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:01.122 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:01.338 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:01.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:01.446 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:01.563 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:01.632 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:00.858957+0000 mgr.y (mgr.24481) 770 : cluster [DBG] pgmap v1065: 148 pgs: 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 117 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 469 op/s; 1898/154977 objects degraded (1.225%); 58720/154977 objects misplaced (37.889%); 18 MiB/s, 284 objects/s recovering 2024-02-12T17:10:01.632 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:01.484247+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check update: Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:10:01.786 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:00.858957+0000 mgr.y (mgr.24481) 770 : cluster [DBG] pgmap v1065: 148 pgs: 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 117 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 469 op/s; 1898/154977 objects degraded (1.225%); 58720/154977 objects misplaced (37.889%); 18 MiB/s, 284 objects/s recovering 2024-02-12T17:10:01.786 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:01.484247+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check update: Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:10:01.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:00.858957+0000 mgr.y (mgr.24481) 770 : cluster [DBG] pgmap v1065: 148 pgs: 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 117 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 469 op/s; 1898/154977 objects degraded (1.225%); 58720/154977 objects misplaced (37.889%); 18 MiB/s, 284 objects/s recovering 2024-02-12T17:10:01.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:01.484247+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check update: Degraded data redundancy: 1866/155724 objects degraded (1.198%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:10:01.977 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:01.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:01.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:02.086 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:02.183 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:02.658 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:02.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:02.764 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:02.807 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:10:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:10:02.807 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:03.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:02.761484+0000 mon.a (mon.0) 1814 : cluster [INF] osd.1 marked itself down and dead 2024-02-12T17:10:03.027 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:10:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[53989]: debug 2024-02-12T17:10:02.748+0000 7f751f7c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:10:03.028 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:10:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[53989]: debug 2024-02-12T17:10:02.749+0000 7f751f7c2700 -1 osd.1 532 *** Got signal Terminated *** 2024-02-12T17:10:03.028 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:10:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[53989]: debug 2024-02-12T17:10:02.749+0000 7f751f7c2700 -1 osd.1 532 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:10:03.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:02.761484+0000 mon.a (mon.0) 1814 : cluster [INF] osd.1 marked itself down and dead 2024-02-12T17:10:03.165 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:10:03.165 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:10:03.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:02.761484+0000 mon.a (mon.0) 1814 : cluster [INF] osd.1 marked itself down and dead 2024-02-12T17:10:03.383 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:03.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:03.487 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:03.502 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:10:03.516 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:03.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T17:10:03.974 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:03.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:04.085 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:04.135 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:02.815476+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T17:10:04.135 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:02.831101+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2024-02-12T17:10:04.135 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:02.859392+0000 mgr.y (mgr.24481) 771 : cluster [DBG] pgmap v1067: 148 pgs: 25 stale+active+clean, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 92 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 457 op/s; 1938/153120 objects degraded (1.266%); 57756/153120 objects misplaced (37.719%); 18 MiB/s, 285 objects/s recovering 2024-02-12T17:10:04.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:02.815476+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T17:10:04.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:02.831101+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2024-02-12T17:10:04.136 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:02.859392+0000 mgr.y (mgr.24481) 771 : cluster [DBG] pgmap v1067: 148 pgs: 25 stale+active+clean, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 92 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 457 op/s; 1938/153120 objects degraded (1.266%); 57756/153120 objects misplaced (37.719%); 18 MiB/s, 285 objects/s recovering 2024-02-12T17:10:04.148 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:02.815476+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T17:10:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:02.831101+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2024-02-12T17:10:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:02.859392+0000 mgr.y (mgr.24481) 771 : cluster [DBG] pgmap v1067: 148 pgs: 25 stale+active+clean, 1 peering, 4 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 92 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 457 op/s; 1938/153120 objects degraded (1.266%); 57756/153120 objects misplaced (37.719%); 18 MiB/s, 285 objects/s recovering 2024-02-12T17:10:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:04.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:04.369 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:04.774 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:10:04 smithi033 bash[174533]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1 2024-02-12T17:10:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:04.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:04.821 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:04.824 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:04.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:04.955 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:10:04.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.852604+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2024-02-12T17:10:04.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.855922+0000 osd.7 (osd.7) 265 : cluster [INF] 10.1c continuing backfill to osd.6 from (471'6510,532'11265] MIN to 532'11265 2024-02-12T17:10:04.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.856436+0000 osd.3 (osd.3) 194 : cluster [INF] 10.2 continuing backfill to osd.6 from (466'4100,532'9329] MIN to 532'9329 2024-02-12T17:10:04.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.859544+0000 osd.2 (osd.2) 208 : cluster [INF] 10.b continuing backfill to osd.6 from (472'6570,532'11203] MIN to 532'11203 2024-02-12T17:10:04.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.859580+0000 osd.2 (osd.2) 209 : cluster [INF] 10.20 continuing backfill to osd.3 from (466'4200,532'9290] MIN to 532'9290 2024-02-12T17:10:04.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.860196+0000 osd.2 (osd.2) 210 : cluster [INF] 10.0 continuing backfill to osd.6 from (466'4200,532'9297] MIN to 532'9297 2024-02-12T17:10:04.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.860212+0000 osd.2 (osd.2) 211 : cluster [INF] 10.1d continuing backfill to osd.5 from (472'6765,532'11394] MIN to 532'11394 2024-02-12T17:10:04.956 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.860243+0000 osd.2 (osd.2) 212 : cluster [INF] 10.1d continuing backfill to osd.6 from (472'6765,532'11394] MIN to 532'11394 2024-02-12T17:10:04.957 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.862141+0000 osd.5 (osd.5) 269 : cluster [INF] 10.16 continuing backfill to osd.6 from (468'5133,532'10077] MIN to 532'10077 2024-02-12T17:10:04.957 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.862684+0000 osd.2 (osd.2) 213 : cluster [INF] 5.5 continuing backfill to osd.6 from (406'13,410'15] MIN to 410'15 2024-02-12T17:10:04.957 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.864230+0000 osd.2 (osd.2) 214 : cluster [INF] 10.23 continuing backfill to osd.7 from (466'4400,532'9504] MIN to 532'9504 2024-02-12T17:10:04.957 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:03.867758+0000 osd.3 (osd.3) 195 : cluster [INF] 10.1f continuing backfill to osd.7 from (472'6748,532'11275] MIN to 532'11275 2024-02-12T17:10:04.958 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:04.985 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:04.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:05.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.852604+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2024-02-12T17:10:05.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.855922+0000 osd.7 (osd.7) 265 : cluster [INF] 10.1c continuing backfill to osd.6 from (471'6510,532'11265] MIN to 532'11265 2024-02-12T17:10:05.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.856436+0000 osd.3 (osd.3) 194 : cluster [INF] 10.2 continuing backfill to osd.6 from (466'4100,532'9329] MIN to 532'9329 2024-02-12T17:10:05.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.859544+0000 osd.2 (osd.2) 208 : cluster [INF] 10.b continuing backfill to osd.6 from (472'6570,532'11203] MIN to 532'11203 2024-02-12T17:10:05.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.859580+0000 osd.2 (osd.2) 209 : cluster [INF] 10.20 continuing backfill to osd.3 from (466'4200,532'9290] MIN to 532'9290 2024-02-12T17:10:05.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.860196+0000 osd.2 (osd.2) 210 : cluster [INF] 10.0 continuing backfill to osd.6 from (466'4200,532'9297] MIN to 532'9297 2024-02-12T17:10:05.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.860212+0000 osd.2 (osd.2) 211 : cluster [INF] 10.1d continuing backfill to osd.5 from (472'6765,532'11394] MIN to 532'11394 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.860243+0000 osd.2 (osd.2) 212 : cluster [INF] 10.1d continuing backfill to osd.6 from (472'6765,532'11394] MIN to 532'11394 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.862141+0000 osd.5 (osd.5) 269 : cluster [INF] 10.16 continuing backfill to osd.6 from (468'5133,532'10077] MIN to 532'10077 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.862684+0000 osd.2 (osd.2) 213 : cluster [INF] 5.5 continuing backfill to osd.6 from (406'13,410'15] MIN to 410'15 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.864230+0000 osd.2 (osd.2) 214 : cluster [INF] 10.23 continuing backfill to osd.7 from (466'4400,532'9504] MIN to 532'9504 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:03.867758+0000 osd.3 (osd.3) 195 : cluster [INF] 10.1f continuing backfill to osd.7 from (472'6748,532'11275] MIN to 532'11275 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.852604+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.855922+0000 osd.7 (osd.7) 265 : cluster [INF] 10.1c continuing backfill to osd.6 from (471'6510,532'11265] MIN to 532'11265 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.856436+0000 osd.3 (osd.3) 194 : cluster [INF] 10.2 continuing backfill to osd.6 from (466'4100,532'9329] MIN to 532'9329 2024-02-12T17:10:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.859544+0000 osd.2 (osd.2) 208 : cluster [INF] 10.b continuing backfill to osd.6 from (472'6570,532'11203] MIN to 532'11203 2024-02-12T17:10:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.859580+0000 osd.2 (osd.2) 209 : cluster [INF] 10.20 continuing backfill to osd.3 from (466'4200,532'9290] MIN to 532'9290 2024-02-12T17:10:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.860196+0000 osd.2 (osd.2) 210 : cluster [INF] 10.0 continuing backfill to osd.6 from (466'4200,532'9297] MIN to 532'9297 2024-02-12T17:10:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.860212+0000 osd.2 (osd.2) 211 : cluster [INF] 10.1d continuing backfill to osd.5 from (472'6765,532'11394] MIN to 532'11394 2024-02-12T17:10:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.860243+0000 osd.2 (osd.2) 212 : cluster [INF] 10.1d continuing backfill to osd.6 from (472'6765,532'11394] MIN to 532'11394 2024-02-12T17:10:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.862141+0000 osd.5 (osd.5) 269 : cluster [INF] 10.16 continuing backfill to osd.6 from (468'5133,532'10077] MIN to 532'10077 2024-02-12T17:10:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.862684+0000 osd.2 (osd.2) 213 : cluster [INF] 5.5 continuing backfill to osd.6 from (406'13,410'15] MIN to 410'15 2024-02-12T17:10:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.864230+0000 osd.2 (osd.2) 214 : cluster [INF] 10.23 continuing backfill to osd.7 from (466'4400,532'9504] MIN to 532'9504 2024-02-12T17:10:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:03.867758+0000 osd.3 (osd.3) 195 : cluster [INF] 10.1f continuing backfill to osd.7 from (472'6748,532'11275] MIN to 532'11275 2024-02-12T17:10:05.090 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:05.165 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:05.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:05.271 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:05.364 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:05.750 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:05.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:05.856 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:05.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:04.663449+0000 mgr.y (mgr.24481) 772 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:05.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:04.858055+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2024-02-12T17:10:05.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:04.863764+0000 mgr.y (mgr.24481) 773 : cluster [DBG] pgmap v1070: 148 pgs: 3 active+undersized, 12 remapped+peering, 8 stale+active+clean, 37 peering, 3 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 70 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 450 op/s; 2379/147609 objects degraded (1.612%); 32454/147609 objects misplaced (21.986%); 4.7 MiB/s, 74 objects/s recovering 2024-02-12T17:10:05.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:04.881790+0000 osd.3 (osd.3) 196 : cluster [INF] 10.17 continuing backfill to osd.6 from (470'5527,532'9948] MIN to 532'9948 2024-02-12T17:10:05.946 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:04.663449+0000 mgr.y (mgr.24481) 772 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:05.946 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:04.858055+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2024-02-12T17:10:05.946 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:04.863764+0000 mgr.y (mgr.24481) 773 : cluster [DBG] pgmap v1070: 148 pgs: 3 active+undersized, 12 remapped+peering, 8 stale+active+clean, 37 peering, 3 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 70 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 450 op/s; 2379/147609 objects degraded (1.612%); 32454/147609 objects misplaced (21.986%); 4.7 MiB/s, 74 objects/s recovering 2024-02-12T17:10:05.946 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:04.881790+0000 osd.3 (osd.3) 196 : cluster [INF] 10.17 continuing backfill to osd.6 from (470'5527,532'9948] MIN to 532'9948 2024-02-12T17:10:05.948 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:06.140 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:06.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:04.663449+0000 mgr.y (mgr.24481) 772 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:04.858055+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2024-02-12T17:10:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:04.863764+0000 mgr.y (mgr.24481) 773 : cluster [DBG] pgmap v1070: 148 pgs: 3 active+undersized, 12 remapped+peering, 8 stale+active+clean, 37 peering, 3 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 70 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 450 op/s; 2379/147609 objects degraded (1.612%); 32454/147609 objects misplaced (21.986%); 4.7 MiB/s, 74 objects/s recovering 2024-02-12T17:10:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:04.881790+0000 osd.3 (osd.3) 196 : cluster [INF] 10.17 continuing backfill to osd.6 from (470'5527,532'9948] MIN to 532'9948 2024-02-12T17:10:06.246 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:06.318 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:06.617 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:06.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:06.725 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:06.804 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:05.869567+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-12T17:10:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:05.869605+0000 mon.a (mon.0) 1820 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:06.485621+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: Degraded data redundancy: 2379/147609 objects degraded (1.612%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-12T17:10:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:06.506664+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2024-02-12T17:10:07.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:05.869567+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-12T17:10:07.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:05.869605+0000 mon.a (mon.0) 1820 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:07.264 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:06.485621+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: Degraded data redundancy: 2379/147609 objects degraded (1.612%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-12T17:10:07.265 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:06.506664+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2024-02-12T17:10:07.265 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:05.869567+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-12T17:10:07.265 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:05.869605+0000 mon.a (mon.0) 1820 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:07.265 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:06.485621+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: Degraded data redundancy: 2379/147609 objects degraded (1.612%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-12T17:10:07.265 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:06.506664+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2024-02-12T17:10:07.290 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:07.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:07.396 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:07.493 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:07.808 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:07.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:07.916 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:07.948 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:10:07.977 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:07.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:08.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:08.123 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:08.146 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:08.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:06.864353+0000 mgr.y (mgr.24481) 774 : cluster [DBG] pgmap v1072: 148 pgs: 3 active+clean+remapped, 7 active+undersized, 12 remapped+peering, 37 peering, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 14 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 60 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 4519/145818 objects degraded (3.099%); 29662/145818 objects misplaced (20.342%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:10:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:07.929478+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? v1:172.21.15.33:0/2456339745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:10:08.251 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:08.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:06.864353+0000 mgr.y (mgr.24481) 774 : cluster [DBG] pgmap v1072: 148 pgs: 3 active+clean+remapped, 7 active+undersized, 12 remapped+peering, 37 peering, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 14 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 60 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 4519/145818 objects degraded (3.099%); 29662/145818 objects misplaced (20.342%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:10:08.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:07.929478+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? v1:172.21.15.33:0/2456339745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:10:08.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:06.864353+0000 mgr.y (mgr.24481) 774 : cluster [DBG] pgmap v1072: 148 pgs: 3 active+clean+remapped, 7 active+undersized, 12 remapped+peering, 37 peering, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 14 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 60 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 4519/145818 objects degraded (3.099%); 29662/145818 objects misplaced (20.342%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:10:08.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:07.929478+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? v1:172.21.15.33:0/2456339745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:10:08.303 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:08.679 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:08.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:08.790 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:08.829 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:08.858 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:08.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:08.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:08.985 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:07.932179+0000 mon.a (mon.0) 1824 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:10:08.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:07.936800+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? v1:172.21.15.33:0/2456339745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:10:08.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:07.936848+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2024-02-12T17:10:08.986 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:07.932179+0000 mon.a (mon.0) 1824 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:10:08.986 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:07.936800+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? v1:172.21.15.33:0/2456339745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:10:08.986 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:07.936848+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2024-02-12T17:10:09.020 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:09.077 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:09.082 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:07.932179+0000 mon.a (mon.0) 1824 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:10:09.082 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:07.936800+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? v1:172.21.15.33:0/2456339745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:10:09.082 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:07.936848+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2024-02-12T17:10:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:09.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:09.184 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:09.222 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:09.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:09.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:09.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:09.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:09.611 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:09.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:09.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:09.841 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:09.848 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:10:09.848 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:10:10.212 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:10.218 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:08.866906+0000 mgr.y (mgr.24481) 775 : cluster [DBG] pgmap v1074: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 60 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 6064/141207 objects degraded (4.294%); 49184/141207 objects misplaced (34.831%); 13 MiB/s, 202 objects/s recovering 2024-02-12T17:10:10.219 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:08.944927+0000 mon.a (mon.0) 1827 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-02-12T17:10:10.219 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:08.980814+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e538: 8 total, 7 up, 7 in 2024-02-12T17:10:10.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:10.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:08.866906+0000 mgr.y (mgr.24481) 775 : cluster [DBG] pgmap v1074: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 60 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 6064/141207 objects degraded (4.294%); 49184/141207 objects misplaced (34.831%); 13 MiB/s, 202 objects/s recovering 2024-02-12T17:10:10.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:08.944927+0000 mon.a (mon.0) 1827 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-02-12T17:10:10.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:08.980814+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e538: 8 total, 7 up, 7 in 2024-02-12T17:10:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:08.866906+0000 mgr.y (mgr.24481) 775 : cluster [DBG] pgmap v1074: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait, 60 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 6064/141207 objects degraded (4.294%); 49184/141207 objects misplaced (34.831%); 13 MiB/s, 202 objects/s recovering 2024-02-12T17:10:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:08.944927+0000 mon.a (mon.0) 1827 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-02-12T17:10:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:08.980814+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e538: 8 total, 7 up, 7 in 2024-02-12T17:10:10.320 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:10.434 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:10.459 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:10:10.506 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:10:10.550 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:10:10.604 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:10:10.637 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:10:10.676 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:10:10.695 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:10.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:10.710 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:10:10.710 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:10:10.711 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:10:10.711 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:10:10.711 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:10:10.711 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:10:10.711 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:10:10.711 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:10:10.711 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:10:10.805 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:10.887 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:10.998 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:10:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:10] "GET /metrics HTTP/1.1" 200 176702 "" "Prometheus/2.33.4" 2024-02-12T17:10:10.999 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:10.388551+0000 mon.b (mon.1) 101 : audit [DBG] from='client.? v1:172.21.15.33:0/2175628151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:11.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:10.388551+0000 mon.b (mon.1) 101 : audit [DBG] from='client.? v1:172.21.15.33:0/2175628151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:11.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:10.388551+0000 mon.b (mon.1) 101 : audit [DBG] from='client.? v1:172.21.15.33:0/2175628151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:11.508 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T17:10:11.584 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:10:12.064 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:12.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:12.177 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:12.219 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:12.261 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:12.263 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:10.868017+0000 mgr.y (mgr.24481) 776 : cluster [DBG] pgmap v1076: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 394 op/s; 5994/140508 objects degraded (4.266%); 48200/140508 objects misplaced (34.304%); 11 MiB/s, 170 objects/s recovering 2024-02-12T17:10:12.264 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:11.491839+0000 mon.a (mon.0) 1829 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:12.264 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:11.491886+0000 mon.a (mon.0) 1830 : cluster [WRN] Health check update: Degraded data redundancy: 5994/140508 objects degraded (4.266%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:12.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:12.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:10.868017+0000 mgr.y (mgr.24481) 776 : cluster [DBG] pgmap v1076: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 394 op/s; 5994/140508 objects degraded (4.266%); 48200/140508 objects misplaced (34.304%); 11 MiB/s, 170 objects/s recovering 2024-02-12T17:10:12.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:11.491839+0000 mon.a (mon.0) 1829 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:12.287 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:11.491886+0000 mon.a (mon.0) 1830 : cluster [WRN] Health check update: Degraded data redundancy: 5994/140508 objects degraded (4.266%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:12.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:10.868017+0000 mgr.y (mgr.24481) 776 : cluster [DBG] pgmap v1076: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 394 op/s; 5994/140508 objects degraded (4.266%); 48200/140508 objects misplaced (34.304%); 11 MiB/s, 170 objects/s recovering 2024-02-12T17:10:12.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:11.491839+0000 mon.a (mon.0) 1829 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:12.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:11.491886+0000 mon.a (mon.0) 1830 : cluster [WRN] Health check update: Degraded data redundancy: 5994/140508 objects degraded (4.266%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:12.366 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:12.411 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:12.436 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:12.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:12.542 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:12.584 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.1.service' 2024-02-12T17:10:12.612 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:12.727 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:12.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:10:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:10:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:12.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:12.753 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:10:12.753 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-12T17:10:12.753 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-02-12T17:10:12.753 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi033.front.sepia.ceph.com 2024-02-12T17:10:12.753 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.1 --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-02-12T17:10:12.848 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:12.912 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:13.064 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:13.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:10:13.104 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:10:13.156 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:13.170 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:13.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:13.200 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/osd.1/config 2024-02-12T17:10:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:13.237 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:13.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:13.351 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:13.426 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:13.491 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:13.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:13.603 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:13.638 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:13.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:13.768 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:13.813 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:13.850 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:13.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:13.959 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:14.034 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:14.171 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:14.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:12.868854+0000 mgr.y (mgr.24481) 777 : cluster [DBG] pgmap v1077: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 418 op/s; 5768/137856 objects degraded (4.184%); 47155/137856 objects misplaced (34.206%); 10 MiB/s, 164 objects/s recovering 2024-02-12T17:10:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:14.193 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:12.868854+0000 mgr.y (mgr.24481) 777 : cluster [DBG] pgmap v1077: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 418 op/s; 5768/137856 objects degraded (4.184%); 47155/137856 objects misplaced (34.206%); 10 MiB/s, 164 objects/s recovering 2024-02-12T17:10:14.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:14.294 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:14.325 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:12.868854+0000 mgr.y (mgr.24481) 777 : cluster [DBG] pgmap v1077: 148 pgs: 1 remapped+peering, 7 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 418 op/s; 5768/137856 objects degraded (4.184%); 47155/137856 objects misplaced (34.206%); 10 MiB/s, 164 objects/s recovering 2024-02-12T17:10:14.328 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:14.360 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:14.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:14.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:14.508 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:14.537 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:14.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:14.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:14.675 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:10:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:10:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:14.736 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:14.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:14.840 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:14.901 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:14.934 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:14.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:15.042 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:15.104 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:10:15.136 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:15.520 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:15.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:15.624 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:15.661 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:15.690 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:15.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:15.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:15.875 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:15.956 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:15.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:16.070 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:16.148 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:14.673479+0000 mgr.y (mgr.24481) 778 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:16.149 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:14.870257+0000 mgr.y (mgr.24481) 779 : cluster [DBG] pgmap v1078: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 647 op/s; 5533/130299 objects degraded (4.246%); 45252/130299 objects misplaced (34.729%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T17:10:16.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:14.673479+0000 mgr.y (mgr.24481) 778 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:16.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:14.870257+0000 mgr.y (mgr.24481) 779 : cluster [DBG] pgmap v1078: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 647 op/s; 5533/130299 objects degraded (4.246%); 45252/130299 objects misplaced (34.729%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T17:10:16.155 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:16.238 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:16.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:16.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:16.343 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:16.385 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:14.673479+0000 mgr.y (mgr.24481) 778 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:16.385 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:14.870257+0000 mgr.y (mgr.24481) 779 : cluster [DBG] pgmap v1078: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 647 op/s; 5533/130299 objects degraded (4.246%); 45252/130299 objects misplaced (34.729%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T17:10:16.389 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:16.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:16.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:16.619 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:16.753 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:16.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:16.865 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:16.925 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:17.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:16.493940+0000 mon.a (mon.0) 1831 : cluster [WRN] Health check update: Degraded data redundancy: 5533/130299 objects degraded (4.246%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:17.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:16.781445+0000 mon.c (mon.2) 262 : audit [INF] from='client.? v1:172.21.15.33:0/4255490471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:10:17.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:16.782645+0000 mon.a (mon.0) 1832 : audit [INF] from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:10:17.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:16.493940+0000 mon.a (mon.0) 1831 : cluster [WRN] Health check update: Degraded data redundancy: 5533/130299 objects degraded (4.246%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:17.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:16.781445+0000 mon.c (mon.2) 262 : audit [INF] from='client.? v1:172.21.15.33:0/4255490471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:10:17.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:16.782645+0000 mon.a (mon.0) 1832 : audit [INF] from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:10:17.325 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:17.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:17.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:16.493940+0000 mon.a (mon.0) 1831 : cluster [WRN] Health check update: Degraded data redundancy: 5533/130299 objects degraded (4.246%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:16.781445+0000 mon.c (mon.2) 262 : audit [INF] from='client.? v1:172.21.15.33:0/4255490471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:10:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:16.782645+0000 mon.a (mon.0) 1832 : audit [INF] from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:10:17.523 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:17.589 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:10:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:16.871242+0000 mgr.y (mgr.24481) 780 : cluster [DBG] pgmap v1079: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 685 op/s; 5323/127488 objects degraded (4.175%); 44112/127488 objects misplaced (34.601%); 9.7 MiB/s, 155 objects/s recovering 2024-02-12T17:10:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:17.514568+0000 mon.a (mon.0) 1833 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:17.528066+0000 mon.a (mon.0) 1834 : audit [INF] from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:10:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:17.528130+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e539: 8 total, 7 up, 7 in 2024-02-12T17:10:18.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:17.528197+0000 mon.a (mon.0) 1836 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:16.871242+0000 mgr.y (mgr.24481) 780 : cluster [DBG] pgmap v1079: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 685 op/s; 5323/127488 objects degraded (4.175%); 44112/127488 objects misplaced (34.601%); 9.7 MiB/s, 155 objects/s recovering 2024-02-12T17:10:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:17.514568+0000 mon.a (mon.0) 1833 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:17.528066+0000 mon.a (mon.0) 1834 : audit [INF] from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:10:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:17.528130+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e539: 8 total, 7 up, 7 in 2024-02-12T17:10:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:17.528197+0000 mon.a (mon.0) 1836 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:18.345 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:18.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:18.451 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:16.871242+0000 mgr.y (mgr.24481) 780 : cluster [DBG] pgmap v1079: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 685 op/s; 5323/127488 objects degraded (4.175%); 44112/127488 objects misplaced (34.601%); 9.7 MiB/s, 155 objects/s recovering 2024-02-12T17:10:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:17.514568+0000 mon.a (mon.0) 1833 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:17.528066+0000 mon.a (mon.0) 1834 : audit [INF] from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:10:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:17.528130+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e539: 8 total, 7 up, 7 in 2024-02-12T17:10:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:17.528197+0000 mon.a (mon.0) 1836 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:18.555 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:18.585 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:18.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:18.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:18.735 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:18.759 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:18.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:18.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:18.898 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:18.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:19.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:19.069 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:19.093 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:19.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:19.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:19.234 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:19.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:19.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:19.428 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:19.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:19.533 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:19.568 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:19.587 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:19.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:19.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:19.777 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:19.957 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:10:19.957 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:10:19.957 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:10:19.957 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:10:19.957 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:10:20.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:18.876965+0000 mgr.y (mgr.24481) 781 : cluster [DBG] pgmap v1081: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 693 op/s; 5218/120642 objects degraded (4.325%); 41492/120642 objects misplaced (34.393%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:10:20.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:18.876965+0000 mgr.y (mgr.24481) 781 : cluster [DBG] pgmap v1081: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 693 op/s; 5218/120642 objects degraded (4.325%); 41492/120642 objects misplaced (34.393%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:10:20.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:18.876965+0000 mgr.y (mgr.24481) 781 : cluster [DBG] pgmap v1081: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 693 op/s; 5218/120642 objects degraded (4.325%); 41492/120642 objects misplaced (34.393%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:10:21.143 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:10:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:20] "GET /metrics HTTP/1.1" 200 176761 "" "Prometheus/2.33.4" 2024-02-12T17:10:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:20.878783+0000 mgr.y (mgr.24481) 782 : cluster [DBG] pgmap v1082: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 693 op/s; 5115/119703 objects degraded (4.273%); 41435/119703 objects misplaced (34.615%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T17:10:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:21.495774+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check update: Degraded data redundancy: 5115/119703 objects degraded (4.273%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:21.518635+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e540: 8 total, 7 up, 7 in 2024-02-12T17:10:22.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:20.878783+0000 mgr.y (mgr.24481) 782 : cluster [DBG] pgmap v1082: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 693 op/s; 5115/119703 objects degraded (4.273%); 41435/119703 objects misplaced (34.615%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T17:10:22.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:21.495774+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check update: Degraded data redundancy: 5115/119703 objects degraded (4.273%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:22.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:21.518635+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e540: 8 total, 7 up, 7 in 2024-02-12T17:10:22.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:20.878783+0000 mgr.y (mgr.24481) 782 : cluster [DBG] pgmap v1082: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 693 op/s; 5115/119703 objects degraded (4.273%); 41435/119703 objects misplaced (34.615%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T17:10:22.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:21.495774+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check update: Degraded data redundancy: 5115/119703 objects degraded (4.273%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:22.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:21.518635+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e540: 8 total, 7 up, 7 in 2024-02-12T17:10:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:10:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:10:23.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:22.549220+0000 osd.7 (osd.7) 266 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:23.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:22.549268+0000 osd.7 (osd.7) 267 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:23.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:22.549220+0000 osd.7 (osd.7) 266 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:23.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:22.549268+0000 osd.7 (osd.7) 267 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:23.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:22.549220+0000 osd.7 (osd.7) 266 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:23.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:22.549268+0000 osd.7 (osd.7) 267 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:24.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:22.879880+0000 mgr.y (mgr.24481) 783 : cluster [DBG] pgmap v1084: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 551 op/s; 4921/117063 objects degraded (4.204%); 40398/117063 objects misplaced (34.510%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T17:10:24.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:23.140678+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:24.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:23.140720+0000 osd.6 (osd.6) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:24.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:23.191175+0000 mon.a (mon.0) 1839 : cluster [DBG] osdmap e541: 8 total, 7 up, 7 in 2024-02-12T17:10:24.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:22.879880+0000 mgr.y (mgr.24481) 783 : cluster [DBG] pgmap v1084: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 551 op/s; 4921/117063 objects degraded (4.204%); 40398/117063 objects misplaced (34.510%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T17:10:24.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:23.140678+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:24.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:23.140720+0000 osd.6 (osd.6) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:24.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:23.191175+0000 mon.a (mon.0) 1839 : cluster [DBG] osdmap e541: 8 total, 7 up, 7 in 2024-02-12T17:10:24.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:22.879880+0000 mgr.y (mgr.24481) 783 : cluster [DBG] pgmap v1084: 148 pgs: 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 551 op/s; 4921/117063 objects degraded (4.204%); 40398/117063 objects misplaced (34.510%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T17:10:24.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:23.140678+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:24.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:23.140720+0000 osd.6 (osd.6) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:24.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:23.191175+0000 mon.a (mon.0) 1839 : cluster [DBG] osdmap e541: 8 total, 7 up, 7 in 2024-02-12T17:10:24.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:10:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:10:25.652 INFO:teuthology.orchestra.run.smithi033.stdout:6.b 2024-02-12T17:10:25.652 INFO:teuthology.orchestra.run.smithi033.stdout:2.a 2024-02-12T17:10:25.652 INFO:teuthology.orchestra.run.smithi033.stdout:10.c 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:10.1c 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:2.9 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:2.6 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:5.a 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:2.1f 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:6.e 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:10.6 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:10.23 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:10.0 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:4.a 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:10.20 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:10.22 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:10.16 2024-02-12T17:10:25.653 INFO:teuthology.orchestra.run.smithi033.stdout:10.15 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:4.1d 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:6.4 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:10.17 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:2.8 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:5.5 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:10.f 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:2.0 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:10.1b 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:6.7 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:10.1d 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:10.b 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:4.14 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:4.17 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:10.19 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:1.0 2024-02-12T17:10:25.654 INFO:teuthology.orchestra.run.smithi033.stdout:2.15 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:10.13 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:10.2 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:10.18 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:4.e 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:5.8 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:10.1 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:10.14 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:6.1 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:10.1f 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:2.11 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:4.b 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:10.1e 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:2.10 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:2.c 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:4.f 2024-02-12T17:10:25.655 INFO:teuthology.orchestra.run.smithi033.stdout:2.d 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:5.c 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:3.0 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.3 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.18 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.10 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:2.4 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.6 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.12 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.13 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.1b 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.0 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.1c 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.1e 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:4.d 2024-02-12T17:10:25.656 INFO:teuthology.orchestra.run.smithi033.stdout:2.5 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:4.1f 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:5.7 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:5.0 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:4.4 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:10.4 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:4.7 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:4.9 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:2.2 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:4.c 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:4.2 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:5.11 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:6.d 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:5.12 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:5.13 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:5.15 2024-02-12T17:10:25.657 INFO:teuthology.orchestra.run.smithi033.stdout:5.9 2024-02-12T17:10:25.658 INFO:teuthology.orchestra.run.smithi033.stdout:5.f 2024-02-12T17:10:25.658 INFO:teuthology.orchestra.run.smithi033.stdout:6.10 2024-02-12T17:10:25.886 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:25.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:25.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:25.993 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:26.091 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:26.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:24.688461+0000 mgr.y (mgr.24481) 784 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:26.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:24.885655+0000 mgr.y (mgr.24481) 785 : cluster [DBG] pgmap v1086: 148 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 789 op/s; 4714/110091 objects degraded (4.282%); 37972/110091 objects misplaced (34.491%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:10:26.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:25.045006+0000 osd.4 (osd.4) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:25.045058+0000 osd.4 (osd.4) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:25.263781+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e542: 8 total, 7 up, 7 in 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:25.797520+0000 osd.3 (osd.3) 197 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:25.797569+0000 osd.3 (osd.3) 198 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:25.981538+0000 mon.c (mon.2) 263 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:24.688461+0000 mgr.y (mgr.24481) 784 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:24.885655+0000 mgr.y (mgr.24481) 785 : cluster [DBG] pgmap v1086: 148 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 789 op/s; 4714/110091 objects degraded (4.282%); 37972/110091 objects misplaced (34.491%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:25.045006+0000 osd.4 (osd.4) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:25.045058+0000 osd.4 (osd.4) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:25.263781+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e542: 8 total, 7 up, 7 in 2024-02-12T17:10:26.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:25.797520+0000 osd.3 (osd.3) 197 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:26.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:25.797569+0000 osd.3 (osd.3) 198 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:26.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:25.981538+0000 mon.c (mon.2) 263 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:10:26.581 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:26.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:26.620 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T17:10:26.687 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:26.721 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:24.688461+0000 mgr.y (mgr.24481) 784 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:26.721 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:24.885655+0000 mgr.y (mgr.24481) 785 : cluster [DBG] pgmap v1086: 148 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 789 op/s; 4714/110091 objects degraded (4.282%); 37972/110091 objects misplaced (34.491%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:10:26.721 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:25.045006+0000 osd.4 (osd.4) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:26.721 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:25.045058+0000 osd.4 (osd.4) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:26.721 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:25.263781+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e542: 8 total, 7 up, 7 in 2024-02-12T17:10:26.721 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:25.797520+0000 osd.3 (osd.3) 197 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:26.722 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:25.797569+0000 osd.3 (osd.3) 198 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:26.722 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:25.981538+0000 mon.c (mon.2) 263 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:10:26.722 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:26.745 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:26.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:26.852 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:26.885 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:26.899 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:26.905 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:26.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:27.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:27.039 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:27.068 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:27.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:27.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:27.207 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:27.236 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:27.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:27.308 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:26.299157+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e543: 8 total, 7 up, 7 in 2024-02-12T17:10:27.309 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:26.500435+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 4714/110091 objects degraded (4.282%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:27.309 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:26.518381+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e544: 8 total, 7 up, 7 in 2024-02-12T17:10:27.309 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:26.829977+0000 mon.c (mon.2) 264 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:10:27.309 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:26.830888+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:10:27.309 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:26.856379+0000 mon.a (mon.0) 1844 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:27.347 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:27.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.1 --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 4.6 --file /var/log/ceph/exp.4.6.1 2024-02-12T17:10:27.448 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:27.710 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:27.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:27.716 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:26.299157+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e543: 8 total, 7 up, 7 in 2024-02-12T17:10:27.716 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:26.500435+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 4714/110091 objects degraded (4.282%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:27.716 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:26.518381+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e544: 8 total, 7 up, 7 in 2024-02-12T17:10:27.716 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:26.829977+0000 mon.c (mon.2) 264 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:10:27.717 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:26.830888+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:10:27.717 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:26.856379+0000 mon.a (mon.0) 1844 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:27.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:26.299157+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e543: 8 total, 7 up, 7 in 2024-02-12T17:10:27.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:26.500435+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 4714/110091 objects degraded (4.282%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:27.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:26.518381+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e544: 8 total, 7 up, 7 in 2024-02-12T17:10:27.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:26.829977+0000 mon.c (mon.2) 264 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:10:27.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:26.830888+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:10:27.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:26.856379+0000 mon.a (mon.0) 1844 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:27.817 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:27.845 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:27.857 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/osd.1/config 2024-02-12T17:10:27.874 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:27.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:27.980 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:28.025 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:28.049 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:28.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:28.063 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:10:28.154 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:28.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:28.327 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:28.367 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:28.392 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:28.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:28.489 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:26.886369+0000 mgr.y (mgr.24481) 786 : cluster [DBG] pgmap v1090: 148 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 783 op/s; 4534/107655 objects degraded (4.212%); 37006/107655 objects misplaced (34.375%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T17:10:28.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:26.886369+0000 mgr.y (mgr.24481) 786 : cluster [DBG] pgmap v1090: 148 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 783 op/s; 4534/107655 objects degraded (4.212%); 37006/107655 objects misplaced (34.375%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T17:10:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:26.886369+0000 mgr.y (mgr.24481) 786 : cluster [DBG] pgmap v1090: 148 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 783 op/s; 4534/107655 objects degraded (4.212%); 37006/107655 objects misplaced (34.375%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T17:10:28.548 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:28.662 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:28.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:28.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:28.810 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:28.837 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:28.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:28.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:29.032 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:29.177 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:29.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:29.286 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:29.352 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:29.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:29.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:29.505 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:29.534 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:29.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:29.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:29.690 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:29.716 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:29.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:29.753 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:29.819 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:29.866 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:29.896 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:29.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:30.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:30.043 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:30.077 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:30.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:30.181 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:30.217 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:30.243 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:30.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:30.263 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:30.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:30.403 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:30.482 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:30.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:30.539 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:28.887166+0000 mgr.y (mgr.24481) 787 : cluster [DBG] pgmap v1091: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 903 op/s; 4423/101673 objects degraded (4.350%); 34695/101673 objects misplaced (34.124%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T17:10:30.539 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:29.344414+0000 osd.5 (osd.5) 270 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:30.539 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:29.344465+0000 osd.5 (osd.5) 271 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:30.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:28.887166+0000 mgr.y (mgr.24481) 787 : cluster [DBG] pgmap v1091: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 903 op/s; 4423/101673 objects degraded (4.350%); 34695/101673 objects misplaced (34.124%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T17:10:30.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:29.344414+0000 osd.5 (osd.5) 270 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:30.539 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:29.344465+0000 osd.5 (osd.5) 271 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:30.585 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:30.713 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:30.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:28.887166+0000 mgr.y (mgr.24481) 787 : cluster [DBG] pgmap v1091: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 903 op/s; 4423/101673 objects degraded (4.350%); 34695/101673 objects misplaced (34.124%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T17:10:30.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:29.344414+0000 osd.5 (osd.5) 270 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:30.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:29.344465+0000 osd.5 (osd.5) 271 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:30.773 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:30.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:30.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:30.988 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:31.159 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:10:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:30] "GET /metrics HTTP/1.1" 200 176761 "" "Prometheus/2.33.4" 2024-02-12T17:10:31.243 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:10:31.262 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:10:31.323 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:10:31.353 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:10:31.393 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:10:31.437 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:10:31.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:10:31.500 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:30.355485+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e545: 8 total, 7 up, 7 in 2024-02-12T17:10:31.501 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:31.184827+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.33:0/3121046148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:31.501 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:30.355485+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e545: 8 total, 7 up, 7 in 2024-02-12T17:10:31.501 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:31.184827+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.33:0/3121046148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:30.355485+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e545: 8 total, 7 up, 7 in 2024-02-12T17:10:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:31.184827+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.33:0/3121046148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:32.505 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:32.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:32.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:32.650 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:32.680 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:32.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:32.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:32.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:10:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:10:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:30.888900+0000 mgr.y (mgr.24481) 788 : cluster [DBG] pgmap v1093: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 910 B/s rd, 0 B/s wr, 551 op/s; 4329/100803 objects degraded (4.295%); 34640/100803 objects misplaced (34.364%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:10:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:31.510548+0000 mon.a (mon.0) 1846 : cluster [WRN] Health check update: Degraded data redundancy: 4329/100803 objects degraded (4.295%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:31.537412+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e546: 8 total, 7 up, 7 in 2024-02-12T17:10:32.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:30.888900+0000 mgr.y (mgr.24481) 788 : cluster [DBG] pgmap v1093: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 910 B/s rd, 0 B/s wr, 551 op/s; 4329/100803 objects degraded (4.295%); 34640/100803 objects misplaced (34.364%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:10:32.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:31.510548+0000 mon.a (mon.0) 1846 : cluster [WRN] Health check update: Degraded data redundancy: 4329/100803 objects degraded (4.295%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:32.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:31.537412+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e546: 8 total, 7 up, 7 in 2024-02-12T17:10:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:30.888900+0000 mgr.y (mgr.24481) 788 : cluster [DBG] pgmap v1093: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 910 B/s rd, 0 B/s wr, 551 op/s; 4329/100803 objects degraded (4.295%); 34640/100803 objects misplaced (34.364%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:10:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:31.510548+0000 mon.a (mon.0) 1846 : cluster [WRN] Health check update: Degraded data redundancy: 4329/100803 objects degraded (4.295%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:31.537412+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e546: 8 total, 7 up, 7 in 2024-02-12T17:10:32.790 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:32.839 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:32.861 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:32.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:32.969 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:32.998 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:33.024 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:33.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:33.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:33.162 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:33.187 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:33.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:33.292 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:33.366 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:10:33.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:10:33.770 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:33.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:33.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:33.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:32.525310+0000 mon.a (mon.0) 1848 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:33.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:32.889551+0000 mgr.y (mgr.24481) 789 : cluster [DBG] pgmap v1095: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 502 op/s; 4120/98076 objects degraded (4.201%); 33604/98076 objects misplaced (34.263%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T17:10:33.960 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:33.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:32.525310+0000 mon.a (mon.0) 1848 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:33.961 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:32.889551+0000 mgr.y (mgr.24481) 789 : cluster [DBG] pgmap v1095: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 502 op/s; 4120/98076 objects degraded (4.201%); 33604/98076 objects misplaced (34.263%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T17:10:33.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:32.525310+0000 mon.a (mon.0) 1848 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:10:33.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:32.889551+0000 mgr.y (mgr.24481) 789 : cluster [DBG] pgmap v1095: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 2.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 502 op/s; 4120/98076 objects degraded (4.201%); 33604/98076 objects misplaced (34.263%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T17:10:34.102 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:34.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:34.208 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:10:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:10:34.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:34.277 INFO:teuthology.orchestra.run.smithi033.stderr:Exporting 4.6 info 4.6( v 526'36 (0'0,526'36] local-lis/les=491/492 n=6 ec=59/59 lis/c=491/491 les/c/f=492/493/0 sis=491) 2024-02-12T17:10:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:34.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:34.281 INFO:teuthology.orchestra.run.smithi033.stderr:Read #4:6192870f:::obj_delete_at_hint.0000000053:head# 2024-02-12T17:10:34.281 INFO:teuthology.orchestra.run.smithi033.stderr:Read #4:624175ac:::obj_delete_at_hint.0000000091:head# 2024-02-12T17:10:34.282 INFO:teuthology.orchestra.run.smithi033.stderr:Read #4:63d917ed:::obj_delete_at_hint.0000000065:head# 2024-02-12T17:10:34.282 INFO:teuthology.orchestra.run.smithi033.stderr:Read #4:64735f45:::obj_delete_at_hint.0000000083:head# 2024-02-12T17:10:34.282 INFO:teuthology.orchestra.run.smithi033.stderr:Read #4:64760185:::obj_delete_at_hint.0000000010:head# 2024-02-12T17:10:34.282 INFO:teuthology.orchestra.run.smithi033.stderr:Read #4:66924383:reshard::reshard.0000000005:head# 2024-02-12T17:10:34.282 INFO:teuthology.orchestra.run.smithi033.stderr:Export successful 2024-02-12T17:10:34.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:34.413 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:34.446 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:34.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:34.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:34.610 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:34.633 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:34.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:34.737 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:34.778 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:34.808 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:34.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:34.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:34.992 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:10:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:10:34.992 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:35.169 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:35.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:35.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:35.184 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T17:10:35.275 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:35.313 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:35.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:35.448 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:35.486 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:35.506 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:35.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:35.611 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:35.692 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:35.767 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:10:35.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:34.709497+0000 mgr.y (mgr.24481) 790 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:35.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:34.890532+0000 mgr.y (mgr.24481) 791 : cluster [DBG] pgmap v1096: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 703 op/s; 3917/90801 objects degraded (4.314%); 31114/90801 objects misplaced (34.266%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:10:36.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:34.709497+0000 mgr.y (mgr.24481) 790 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:36.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:34.890532+0000 mgr.y (mgr.24481) 791 : cluster [DBG] pgmap v1096: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 703 op/s; 3917/90801 objects degraded (4.314%); 31114/90801 objects misplaced (34.266%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:10:36.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:34.709497+0000 mgr.y (mgr.24481) 790 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:36.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:34.890532+0000 mgr.y (mgr.24481) 791 : cluster [DBG] pgmap v1096: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 703 op/s; 3917/90801 objects degraded (4.314%); 31114/90801 objects misplaced (34.266%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:10:36.872 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:36.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:36.894 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.1 --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 4.6 2024-02-12T17:10:36.981 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:37.022 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:37.049 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:37.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:37.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:37.186 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:37.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:37.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:36.518133+0000 mon.a (mon.0) 1849 : cluster [WRN] Health check update: Degraded data redundancy: 3917/90801 objects degraded (4.314%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:37.218 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:36.518133+0000 mon.a (mon.0) 1849 : cluster [WRN] Health check update: Degraded data redundancy: 3917/90801 objects degraded (4.314%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:36.518133+0000 mon.a (mon.0) 1849 : cluster [WRN] Health check update: Degraded data redundancy: 3917/90801 objects degraded (4.314%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:37.312 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:37.348 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:37.374 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:37.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:37.478 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:37.639 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:37.639 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:37.647 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:37.668 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/osd.1/config 2024-02-12T17:10:37.673 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:37.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:37.780 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:37.877 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:37.979 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:37.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:38.087 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:38.137 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:36.891378+0000 mgr.y (mgr.24481) 792 : cluster [DBG] pgmap v1097: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 566 op/s; 3696/88092 objects degraded (4.196%); 30073/88092 objects misplaced (34.138%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:10:38.137 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:37.557679+0000 mon.a (mon.0) 1850 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:38.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:36.891378+0000 mgr.y (mgr.24481) 792 : cluster [DBG] pgmap v1097: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 566 op/s; 3696/88092 objects degraded (4.196%); 30073/88092 objects misplaced (34.138%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:10:38.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:37.557679+0000 mon.a (mon.0) 1850 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:38.149 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:38.202 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:38.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:38.308 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:38.377 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:38.460 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:38.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:38.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:36.891378+0000 mgr.y (mgr.24481) 792 : cluster [DBG] pgmap v1097: 148 pgs: 5 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 566 op/s; 3696/88092 objects degraded (4.196%); 30073/88092 objects misplaced (34.138%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T17:10:38.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:37.557679+0000 mon.a (mon.0) 1850 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:10:38.567 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:38.940 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:38.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:39.050 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:39.055 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:10:39.118 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:39.158 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:39.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:39.239 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:38.033374+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? v1:172.21.15.33:0/2239476948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:10:39.240 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:38.033374+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? v1:172.21.15.33:0/2239476948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:10:39.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:39.299 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:38.033374+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? v1:172.21.15.33:0/2239476948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:10:39.300 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:39.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:39.429 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:39.486 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:39.952 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:39.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:40.057 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:40.095 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:40.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:40.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:40.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:38.892304+0000 mgr.y (mgr.24481) 793 : cluster [DBG] pgmap v1098: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 742 op/s; 3578/81825 objects degraded (4.373%); 27711/81825 objects misplaced (33.866%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T17:10:40.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:38.995109+0000 mon.a (mon.0) 1852 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:40.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:39.022838+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? v1:172.21.15.33:0/2239476948' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:10:40.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:39.022904+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e547: 8 total, 7 up, 7 in 2024-02-12T17:10:40.245 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:38.892304+0000 mgr.y (mgr.24481) 793 : cluster [DBG] pgmap v1098: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 742 op/s; 3578/81825 objects degraded (4.373%); 27711/81825 objects misplaced (33.866%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T17:10:40.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:38.995109+0000 mon.a (mon.0) 1852 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:40.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:39.022838+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? v1:172.21.15.33:0/2239476948' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:10:40.246 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:39.022904+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e547: 8 total, 7 up, 7 in 2024-02-12T17:10:40.268 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:38.892304+0000 mgr.y (mgr.24481) 793 : cluster [DBG] pgmap v1098: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 742 op/s; 3578/81825 objects degraded (4.373%); 27711/81825 objects misplaced (33.866%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T17:10:40.268 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:38.995109+0000 mon.a (mon.0) 1852 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:40.268 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:39.022838+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? v1:172.21.15.33:0/2239476948' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:10:40.268 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:39.022904+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e547: 8 total, 7 up, 7 in 2024-02-12T17:10:40.272 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:40.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:40.406 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:40.455 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:40.491 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:40.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:40.598 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:40.692 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:41.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:10:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:40] "GET /metrics HTTP/1.1" 200 176771 "" "Prometheus/2.33.4" 2024-02-12T17:10:41.876 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:41.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:41.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:42.026 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:42.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:42.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:42.223 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:42.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:42.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:40.893010+0000 mgr.y (mgr.24481) 794 : cluster [DBG] pgmap v1100: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 610 op/s; 3474/80973 objects degraded (4.290%); 27654/80973 objects misplaced (34.152%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:10:42.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:41.524298+0000 mon.a (mon.0) 1855 : cluster [WRN] Health check update: Degraded data redundancy: 3474/80973 objects degraded (4.290%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:42.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:41.530097+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e548: 8 total, 7 up, 7 in 2024-02-12T17:10:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:40.893010+0000 mgr.y (mgr.24481) 794 : cluster [DBG] pgmap v1100: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 610 op/s; 3474/80973 objects degraded (4.290%); 27654/80973 objects misplaced (34.152%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:10:42.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:41.524298+0000 mon.a (mon.0) 1855 : cluster [WRN] Health check update: Degraded data redundancy: 3474/80973 objects degraded (4.290%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:42.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:41.530097+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e548: 8 total, 7 up, 7 in 2024-02-12T17:10:42.327 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:42.417 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:40.893010+0000 mgr.y (mgr.24481) 794 : cluster [DBG] pgmap v1100: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 610 op/s; 3474/80973 objects degraded (4.290%); 27654/80973 objects misplaced (34.152%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:10:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:41.524298+0000 mon.a (mon.0) 1855 : cluster [WRN] Health check update: Degraded data redundancy: 3474/80973 objects degraded (4.290%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:41.530097+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e548: 8 total, 7 up, 7 in 2024-02-12T17:10:42.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:10:42.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:10:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:10:43.837 INFO:teuthology.orchestra.run.smithi033.stdout: marking collection for removal 2024-02-12T17:10:43.837 INFO:teuthology.orchestra.run.smithi033.stdout:setting '_remove' omap key 2024-02-12T17:10:43.837 INFO:teuthology.orchestra.run.smithi033.stdout:finish_remove_pgs 4.6_head removing 4.6 2024-02-12T17:10:43.837 INFO:teuthology.orchestra.run.smithi033.stdout:Remove successful 2024-02-12T17:10:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:42.893672+0000 mgr.y (mgr.24481) 795 : cluster [DBG] pgmap v1102: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 525 op/s; 3255/78189 objects degraded (4.163%); 26616/78189 objects misplaced (34.041%); 3.9 MiB/s, 63 objects/s recovering 2024-02-12T17:10:44.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:42.893672+0000 mgr.y (mgr.24481) 795 : cluster [DBG] pgmap v1102: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 525 op/s; 3255/78189 objects degraded (4.163%); 26616/78189 objects misplaced (34.041%); 3.9 MiB/s, 63 objects/s recovering 2024-02-12T17:10:44.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:42.893672+0000 mgr.y (mgr.24481) 795 : cluster [DBG] pgmap v1102: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 525 op/s; 3255/78189 objects degraded (4.163%); 26616/78189 objects misplaced (34.041%); 3.9 MiB/s, 63 objects/s recovering 2024-02-12T17:10:44.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:10:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:10:45.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:43.648179+0000 osd.2 (osd.2) 215 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:45.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:43.648230+0000 osd.2 (osd.2) 216 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:45.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:44.085146+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e549: 8 total, 7 up, 7 in 2024-02-12T17:10:45.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:43.648179+0000 osd.2 (osd.2) 215 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:45.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:43.648230+0000 osd.2 (osd.2) 216 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:45.366 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:44.085146+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e549: 8 total, 7 up, 7 in 2024-02-12T17:10:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:43.648179+0000 osd.2 (osd.2) 215 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:43.648230+0000 osd.2 (osd.2) 216 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:44.085146+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e549: 8 total, 7 up, 7 in 2024-02-12T17:10:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:44.720711+0000 mgr.y (mgr.24481) 796 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:44.896809+0000 mgr.y (mgr.24481) 797 : cluster [DBG] pgmap v1104: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 615 op/s; 3076/70755 objects degraded (4.347%); 24019/70755 objects misplaced (33.947%); 4.1 MiB/s, 64 objects/s recovering 2024-02-12T17:10:46.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:44.720711+0000 mgr.y (mgr.24481) 796 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:46.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:44.896809+0000 mgr.y (mgr.24481) 797 : cluster [DBG] pgmap v1104: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 615 op/s; 3076/70755 objects degraded (4.347%); 24019/70755 objects misplaced (33.947%); 4.1 MiB/s, 64 objects/s recovering 2024-02-12T17:10:46.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:44.720711+0000 mgr.y (mgr.24481) 796 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:46.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:44.896809+0000 mgr.y (mgr.24481) 797 : cluster [DBG] pgmap v1104: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 615 op/s; 3076/70755 objects degraded (4.347%); 24019/70755 objects misplaced (33.947%); 4.1 MiB/s, 64 objects/s recovering 2024-02-12T17:10:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:46.525745+0000 mon.a (mon.0) 1858 : cluster [WRN] Health check update: Degraded data redundancy: 3076/70755 objects degraded (4.347%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:46.548337+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e550: 8 total, 7 up, 7 in 2024-02-12T17:10:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:46.525745+0000 mon.a (mon.0) 1858 : cluster [WRN] Health check update: Degraded data redundancy: 3076/70755 objects degraded (4.347%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:46.548337+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e550: 8 total, 7 up, 7 in 2024-02-12T17:10:47.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:46.525745+0000 mon.a (mon.0) 1858 : cluster [WRN] Health check update: Degraded data redundancy: 3076/70755 objects degraded (4.347%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:47.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:46.548337+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e550: 8 total, 7 up, 7 in 2024-02-12T17:10:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:46.899767+0000 mgr.y (mgr.24481) 798 : cluster [DBG] pgmap v1106: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 717 op/s; 2861/68052 objects degraded (4.204%); 23001/68052 objects misplaced (33.799%); 5.1 MiB/s, 81 objects/s recovering 2024-02-12T17:10:48.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:46.899767+0000 mgr.y (mgr.24481) 798 : cluster [DBG] pgmap v1106: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 717 op/s; 2861/68052 objects degraded (4.204%); 23001/68052 objects misplaced (33.799%); 5.1 MiB/s, 81 objects/s recovering 2024-02-12T17:10:48.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:46.899767+0000 mgr.y (mgr.24481) 798 : cluster [DBG] pgmap v1106: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 717 op/s; 2861/68052 objects degraded (4.204%); 23001/68052 objects misplaced (33.799%); 5.1 MiB/s, 81 objects/s recovering 2024-02-12T17:10:50.196 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:10:50.197 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:10:50.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:48.901003+0000 mgr.y (mgr.24481) 799 : cluster [DBG] pgmap v1107: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 755 op/s; 2749/61533 objects degraded (4.468%); 20551/61533 objects misplaced (33.398%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:10:50.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:48.901003+0000 mgr.y (mgr.24481) 799 : cluster [DBG] pgmap v1107: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 755 op/s; 2749/61533 objects degraded (4.468%); 20551/61533 objects misplaced (33.398%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:10:50.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:48.901003+0000 mgr.y (mgr.24481) 799 : cluster [DBG] pgmap v1107: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 755 op/s; 2749/61533 objects degraded (4.468%); 20551/61533 objects misplaced (33.398%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:10:50.727 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:50.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:50.839 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:50.952 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:51.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:10:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:50] "GET /metrics HTTP/1.1" 200 176771 "" "Prometheus/2.33.4" 2024-02-12T17:10:51.098 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:51.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:51.206 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:51.262 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:51.615 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:10:51.616 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:10:51.616 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:10:51.616 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:10:51.616 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:10:51.616 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:10:51.624 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:51.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:51.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:10:51.648 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:10:51.729 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:51.813 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:52.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:50.901719+0000 mgr.y (mgr.24481) 800 : cluster [DBG] pgmap v1108: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 733 op/s; 2653/60609 objects degraded (4.377%); 20496/60609 objects misplaced (33.817%); 3.8 MiB/s, 61 objects/s recovering 2024-02-12T17:10:52.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:51.291944+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? v1:172.21.15.33:0/3661860476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:52.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:51.532739+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check update: Degraded data redundancy: 2653/60609 objects degraded (4.377%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:52.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:50.901719+0000 mgr.y (mgr.24481) 800 : cluster [DBG] pgmap v1108: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 733 op/s; 2653/60609 objects degraded (4.377%); 20496/60609 objects misplaced (33.817%); 3.8 MiB/s, 61 objects/s recovering 2024-02-12T17:10:52.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:51.291944+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? v1:172.21.15.33:0/3661860476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:52.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:51.532739+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check update: Degraded data redundancy: 2653/60609 objects degraded (4.377%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:52.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:50.901719+0000 mgr.y (mgr.24481) 800 : cluster [DBG] pgmap v1108: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 733 op/s; 2653/60609 objects degraded (4.377%); 20496/60609 objects misplaced (33.817%); 3.8 MiB/s, 61 objects/s recovering 2024-02-12T17:10:52.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:51.291944+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? v1:172.21.15.33:0/3661860476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:10:52.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:51.532739+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check update: Degraded data redundancy: 2653/60609 objects degraded (4.377%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:10:52.942 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:10:52.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:10:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:10:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:10:53.408 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:53.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:53.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T17:10:53.513 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:10:53.553 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:53.581 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:10:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:53.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:10:53.609 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.1 --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.6.1 2024-02-12T17:10:53.629 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:53.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:53.822 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:53.960 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:53.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:53.977 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/osd.1/config 2024-02-12T17:10:54.068 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:54.124 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:54.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:52.902533+0000 mgr.y (mgr.24481) 801 : cluster [DBG] pgmap v1109: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 524 op/s; 2426/57762 objects degraded (4.200%); 19445/57762 objects misplaced (33.664%); 3.9 MiB/s, 61 objects/s recovering 2024-02-12T17:10:54.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:52.902533+0000 mgr.y (mgr.24481) 801 : cluster [DBG] pgmap v1109: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 524 op/s; 2426/57762 objects degraded (4.200%); 19445/57762 objects misplaced (33.664%); 3.9 MiB/s, 61 objects/s recovering 2024-02-12T17:10:54.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:52.902533+0000 mgr.y (mgr.24481) 801 : cluster [DBG] pgmap v1109: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 524 op/s; 2426/57762 objects degraded (4.200%); 19445/57762 objects misplaced (33.664%); 3.9 MiB/s, 61 objects/s recovering 2024-02-12T17:10:54.817 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:54.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:54.924 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:54.944 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:10:54.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:10:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:10:55.006 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:55.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:54.353550+0000 osd.0 (osd.0) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:55.247 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:54.353614+0000 osd.0 (osd.0) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:55.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:54.353550+0000 osd.0 (osd.0) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:55.247 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:54.353614+0000 osd.0 (osd.0) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:55.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:54.353550+0000 osd.0 (osd.0) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:10:55.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:54.353614+0000 osd.0 (osd.0) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:10:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:54.730494+0000 mgr.y (mgr.24481) 802 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:54.903310+0000 mgr.y (mgr.24481) 803 : cluster [DBG] pgmap v1110: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 682 op/s; 2204/50298 objects degraded (4.382%); 16944/50298 objects misplaced (33.687%); 4.1 MiB/s, 66 objects/s recovering 2024-02-12T17:10:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:55.240415+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e551: 8 total, 7 up, 7 in 2024-02-12T17:10:56.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:54.730494+0000 mgr.y (mgr.24481) 802 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:56.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:54.903310+0000 mgr.y (mgr.24481) 803 : cluster [DBG] pgmap v1110: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 682 op/s; 2204/50298 objects degraded (4.382%); 16944/50298 objects misplaced (33.687%); 4.1 MiB/s, 66 objects/s recovering 2024-02-12T17:10:56.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:55.240415+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e551: 8 total, 7 up, 7 in 2024-02-12T17:10:56.696 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:10:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:56.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:10:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:54.730494+0000 mgr.y (mgr.24481) 802 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:10:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:54.903310+0000 mgr.y (mgr.24481) 803 : cluster [DBG] pgmap v1110: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 1.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 682 op/s; 2204/50298 objects degraded (4.382%); 16944/50298 objects misplaced (33.687%); 4.1 MiB/s, 66 objects/s recovering 2024-02-12T17:10:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:55.240415+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e551: 8 total, 7 up, 7 in 2024-02-12T17:10:56.805 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:56.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:56.981 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:10:57.056 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:57.329 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:10:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:57.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:10:57.346 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:56.534219+0000 mon.a (mon.0) 1862 : cluster [WRN] Health check update: Degraded data redundancy: 2204/50298 objects degraded (4.382%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:57.347 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:56.556165+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e552: 8 total, 7 up, 7 in 2024-02-12T17:10:57.351 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:56.534219+0000 mon.a (mon.0) 1862 : cluster [WRN] Health check update: Degraded data redundancy: 2204/50298 objects degraded (4.382%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:57.351 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:56.556165+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e552: 8 total, 7 up, 7 in 2024-02-12T17:10:57.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:10:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:57.516 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:10:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:57.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:10:57.521 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:56.534219+0000 mon.a (mon.0) 1862 : cluster [WRN] Health check update: Degraded data redundancy: 2204/50298 objects degraded (4.382%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:10:57.521 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:56.556165+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e552: 8 total, 7 up, 7 in 2024-02-12T17:10:57.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:10:57.659 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:57.686 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:10:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:57.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:10:57.790 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:10:57.837 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:57.869 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:10:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:57.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:10:57.973 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:10:58.079 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:58.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:56.903936+0000 mgr.y (mgr.24481) 804 : cluster [DBG] pgmap v1113: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 982 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 575 op/s; 2015/47730 objects degraded (4.222%); 15974/47730 objects misplaced (33.467%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T17:10:58.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:58.219871+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.33:0/3208672727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:10:58.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:58.220507+0000 mon.a (mon.0) 1864 : audit [INF] from='client.25801 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:10:58.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:56.903936+0000 mgr.y (mgr.24481) 804 : cluster [DBG] pgmap v1113: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 982 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 575 op/s; 2015/47730 objects degraded (4.222%); 15974/47730 objects misplaced (33.467%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T17:10:58.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:58.219871+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.33:0/3208672727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:10:58.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:58.220507+0000 mon.a (mon.0) 1864 : audit [INF] from='client.25801 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:10:58.542 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:10:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:10:58.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:10:58.566 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:58.572 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:10:58.595 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:10:58.646 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:10:58.712 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:10:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:56.903936+0000 mgr.y (mgr.24481) 804 : cluster [DBG] pgmap v1113: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 982 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 575 op/s; 2015/47730 objects degraded (4.222%); 15974/47730 objects misplaced (33.467%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T17:10:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:58.219871+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.33:0/3208672727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:10:58.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:58.220507+0000 mon.a (mon.0) 1864 : audit [INF] from='client.25801 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:10:59.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:58.565969+0000 mon.a (mon.0) 1865 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:59.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:10:58.580968+0000 mon.a (mon.0) 1866 : audit [INF] from='client.25801 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:10:59.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:10:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:58.581043+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e553: 8 total, 7 up, 7 in 2024-02-12T17:10:59.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:58.565969+0000 mon.a (mon.0) 1865 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:59.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:10:58.580968+0000 mon.a (mon.0) 1866 : audit [INF] from='client.25801 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:10:59.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:10:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:58.581043+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e553: 8 total, 7 up, 7 in 2024-02-12T17:10:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:58.565969+0000 mon.a (mon.0) 1865 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:10:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:10:58.580968+0000 mon.a (mon.0) 1866 : audit [INF] from='client.25801 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:10:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:10:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:58.581043+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e553: 8 total, 7 up, 7 in 2024-02-12T17:11:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:10:58.904720+0000 mgr.y (mgr.24481) 805 : cluster [DBG] pgmap v1115: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 858 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 889 op/s; 1903/41775 objects degraded (4.555%); 13791/41775 objects misplaced (33.013%); 4.7 MiB/s, 75 objects/s recovering 2024-02-12T17:11:00.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:10:58.904720+0000 mgr.y (mgr.24481) 805 : cluster [DBG] pgmap v1115: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 858 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 889 op/s; 1903/41775 objects degraded (4.555%); 13791/41775 objects misplaced (33.013%); 4.7 MiB/s, 75 objects/s recovering 2024-02-12T17:11:00.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:10:58.904720+0000 mgr.y (mgr.24481) 805 : cluster [DBG] pgmap v1115: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 858 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 889 op/s; 1903/41775 objects degraded (4.555%); 13791/41775 objects misplaced (33.013%); 4.7 MiB/s, 75 objects/s recovering 2024-02-12T17:11:01.179 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:11:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:00] "GET /metrics HTTP/1.1" 200 176771 "" "Prometheus/2.33.4" 2024-02-12T17:11:01.488 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:00.302583+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e554: 8 total, 7 up, 7 in 2024-02-12T17:11:01.489 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:00.302583+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e554: 8 total, 7 up, 7 in 2024-02-12T17:11:01.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:00.302583+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e554: 8 total, 7 up, 7 in 2024-02-12T17:11:01.912 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:01.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:02.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:02.053 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:02.078 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:02.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:02.182 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:02.242 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:02.358 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:00.905383+0000 mgr.y (mgr.24481) 806 : cluster [DBG] pgmap v1117: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 840 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 904 B/s rd, 0 B/s wr, 555 op/s; 1813/40878 objects degraded (4.435%); 13736/40878 objects misplaced (33.602%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:11:02.358 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:01.324234+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e555: 8 total, 7 up, 7 in 2024-02-12T17:11:02.500 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:11:02.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:00.905383+0000 mgr.y (mgr.24481) 806 : cluster [DBG] pgmap v1117: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 840 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 904 B/s rd, 0 B/s wr, 555 op/s; 1813/40878 objects degraded (4.435%); 13736/40878 objects misplaced (33.602%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:11:02.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:01.324234+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e555: 8 total, 7 up, 7 in 2024-02-12T17:11:02.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:01.544834+0000 mon.a (mon.0) 1870 : cluster [WRN] Health check update: Degraded data redundancy: 1813/40878 objects degraded (4.435%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:11:02.533 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:01.585225+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e556: 8 total, 7 up, 7 in 2024-02-12T17:11:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:00.905383+0000 mgr.y (mgr.24481) 806 : cluster [DBG] pgmap v1117: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 9 active+remapped+backfill_wait, 60 active+clean; 840 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 904 B/s rd, 0 B/s wr, 555 op/s; 1813/40878 objects degraded (4.435%); 13736/40878 objects misplaced (33.602%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:11:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:01.324234+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e555: 8 total, 7 up, 7 in 2024-02-12T17:11:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:01.544834+0000 mon.a (mon.0) 1870 : cluster [WRN] Health check update: Degraded data redundancy: 1813/40878 objects degraded (4.435%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:11:02.533 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:01.585225+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e556: 8 total, 7 up, 7 in 2024-02-12T17:11:02.717 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:02.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:11:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:11:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:01.544834+0000 mon.a (mon.0) 1870 : cluster [WRN] Health check update: Degraded data redundancy: 1813/40878 objects degraded (4.435%), 33 pgs degraded (PG_DEGRADED) 2024-02-12T17:11:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:01.585225+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e556: 8 total, 7 up, 7 in 2024-02-12T17:11:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:02.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:02.850 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:02.905 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:02.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:03.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:03.041 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:03.065 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:03.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:03.171 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:03.299 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:03.487 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:03.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:03.614 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:03.747 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:04.002 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:04.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:04.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:04.135 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:04.154 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:04.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:04.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:04.288 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:04.311 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:04.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:04.405 INFO:teuthology.orchestra.run.smithi033.stdout:get_pg_num_history pg_num_history pg_num_history(e532 pg_nums {1={19=1},2={57=32},3={57=3},4={59=32},5={238=22,409=21,413=20,417=19,421=18,425=17,429=16,431=22},6={63=32,360=31,364=30,368=29,371=28,375=27,379=26,383=25,388=24,392=23,396=22},9={304=16},10={451=16,480=26,503=36}} deleted_pools 446,9) 2024-02-12T17:11:04.405 INFO:teuthology.orchestra.run.smithi033.stdout:Importing pgid 4.6 2024-02-12T17:11:04.414 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:04.418 INFO:teuthology.orchestra.run.smithi033.stdout:Write #4:6192870f:::obj_delete_at_hint.0000000053:head# 2024-02-12T17:11:04.418 INFO:teuthology.orchestra.run.smithi033.stdout:snapset 0=[]:{} 2024-02-12T17:11:04.418 INFO:teuthology.orchestra.run.smithi033.stdout:Write #4:624175ac:::obj_delete_at_hint.0000000091:head# 2024-02-12T17:11:04.418 INFO:teuthology.orchestra.run.smithi033.stdout:snapset 0=[]:{} 2024-02-12T17:11:04.424 INFO:teuthology.orchestra.run.smithi033.stdout:Write #4:63d917ed:::obj_delete_at_hint.0000000065:head# 2024-02-12T17:11:04.425 INFO:teuthology.orchestra.run.smithi033.stdout:snapset 0=[]:{} 2024-02-12T17:11:04.425 INFO:teuthology.orchestra.run.smithi033.stdout:Write #4:64735f45:::obj_delete_at_hint.0000000083:head# 2024-02-12T17:11:04.425 INFO:teuthology.orchestra.run.smithi033.stdout:snapset 0=[]:{} 2024-02-12T17:11:04.427 INFO:teuthology.orchestra.run.smithi033.stdout:Write #4:64760185:::obj_delete_at_hint.0000000010:head# 2024-02-12T17:11:04.427 INFO:teuthology.orchestra.run.smithi033.stdout:snapset 0=[]:{} 2024-02-12T17:11:04.451 INFO:teuthology.orchestra.run.smithi033.stdout:Write #4:66924383:reshard::reshard.0000000005:head# 2024-02-12T17:11:04.451 INFO:teuthology.orchestra.run.smithi033.stdout:snapset 0=[]:{} 2024-02-12T17:11:04.451 INFO:teuthology.orchestra.run.smithi033.stdout:write_pg epoch 526 info 4.6( v 526'36 (0'0,526'36] local-lis/les=491/492 n=6 ec=59/59 lis/c=491/491 les/c/f=492/493/0 sis=491) 2024-02-12T17:11:04.451 INFO:teuthology.orchestra.run.smithi033.stdout:Import successful 2024-02-12T17:11:04.476 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:04.617 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:04.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:04.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:02.906010+0000 mgr.y (mgr.24481) 807 : cluster [DBG] pgmap v1120: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 6 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 8 active+remapped+backfill_wait, 60 active+clean; 792 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 367 op/s; 1629/38604 objects degraded (4.220%); 12938/38604 objects misplaced (33.515%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:11:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:03.417591+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e557: 8 total, 7 up, 7 in 2024-02-12T17:11:04.758 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:04.777 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:04.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:02.906010+0000 mgr.y (mgr.24481) 807 : cluster [DBG] pgmap v1120: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 6 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 8 active+remapped+backfill_wait, 60 active+clean; 792 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 367 op/s; 1629/38604 objects degraded (4.220%); 12938/38604 objects misplaced (33.515%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:11:04.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:03.417591+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e557: 8 total, 7 up, 7 in 2024-02-12T17:11:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:04.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:04.785 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:02.906010+0000 mgr.y (mgr.24481) 807 : cluster [DBG] pgmap v1120: 148 pgs: 6 active+remapped+backfill_wait+backfill_toofull, 8 active+clean+remapped, 27 active+undersized, 6 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 8 active+remapped+backfill_wait, 60 active+clean; 792 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 367 op/s; 1629/38604 objects degraded (4.220%); 12938/38604 objects misplaced (33.515%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:11:04.785 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:03.417591+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e557: 8 total, 7 up, 7 in 2024-02-12T17:11:04.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:04.918 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:04.943 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:04.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:05.049 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:05.081 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:11:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:11:05.086 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:05.113 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:05.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:05.221 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:05.265 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:05.456 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:04.431602+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e558: 8 total, 7 up, 7 in 2024-02-12T17:11:05.504 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:05.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:04.431602+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e558: 8 total, 7 up, 7 in 2024-02-12T17:11:05.510 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:04.431602+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e558: 8 total, 7 up, 7 in 2024-02-12T17:11:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:05.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:05.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:05.691 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:06.042 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:11:06.042 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:11:06.042 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:11:06.042 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:11:06.042 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:11:06.043 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:11:06.046 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:11:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:11:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:11:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:11:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:11:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:11:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:11:06.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:11:06.047 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:11:06.057 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:06.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:06.166 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:06.253 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:06.527 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:06.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:06.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:04.744762+0000 mgr.y (mgr.24481) 808 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:06.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:04.911505+0000 mgr.y (mgr.24481) 809 : cluster [DBG] pgmap v1123: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 6 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 61 active+clean; 666 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s; 1436/32565 objects degraded (4.410%); 10835/32565 objects misplaced (33.272%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T17:11:06.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:05.426418+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:06.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:05.461441+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e559: 8 total, 7 up, 7 in 2024-02-12T17:11:06.534 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:05.730628+0000 mon.b (mon.1) 104 : audit [DBG] from='client.? v1:172.21.15.33:0/2313672614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:06.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:04.744762+0000 mgr.y (mgr.24481) 808 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:06.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:04.911505+0000 mgr.y (mgr.24481) 809 : cluster [DBG] pgmap v1123: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 6 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 61 active+clean; 666 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s; 1436/32565 objects degraded (4.410%); 10835/32565 objects misplaced (33.272%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T17:11:06.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:05.426418+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:06.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:05.461441+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e559: 8 total, 7 up, 7 in 2024-02-12T17:11:06.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:05.730628+0000 mon.b (mon.1) 104 : audit [DBG] from='client.? v1:172.21.15.33:0/2313672614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:06.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:06.662 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:06.688 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:06.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:04.744762+0000 mgr.y (mgr.24481) 808 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:04.911505+0000 mgr.y (mgr.24481) 809 : cluster [DBG] pgmap v1123: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 6 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 61 active+clean; 666 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s; 1436/32565 objects degraded (4.410%); 10835/32565 objects misplaced (33.272%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T17:11:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:05.426418+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:05.461441+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e559: 8 total, 7 up, 7 in 2024-02-12T17:11:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:05.730628+0000 mon.b (mon.1) 104 : audit [DBG] from='client.? v1:172.21.15.33:0/2313672614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:06.792 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:06.841 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:06.878 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:06.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:06.982 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:07.068 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:07.441 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:07.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:07.550 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:07.600 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:07.647 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:07.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:07.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:06.470089+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e560: 8 total, 7 up, 7 in 2024-02-12T17:11:07.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:06.552285+0000 mon.a (mon.0) 1877 : cluster [WRN] Health check update: Degraded data redundancy: 1436/32565 objects degraded (4.410%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:07.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:07.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:06.470089+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e560: 8 total, 7 up, 7 in 2024-02-12T17:11:07.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:06.552285+0000 mon.a (mon.0) 1877 : cluster [WRN] Health check update: Degraded data redundancy: 1436/32565 objects degraded (4.410%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:07.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:06.470089+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e560: 8 total, 7 up, 7 in 2024-02-12T17:11:07.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:06.552285+0000 mon.a (mon.0) 1877 : cluster [WRN] Health check update: Degraded data redundancy: 1436/32565 objects degraded (4.410%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:07.834 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:07.969 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:07.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:07.982 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/exp.4.6.1 2024-02-12T17:11:08.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:08.100 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:08.125 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:08.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:08.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:08.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:08.259 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:08.289 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:08.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:08.394 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:08.489 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:08.526 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:11:08.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:11:08.599 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:11:08.599 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:11:08.619 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:08.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:08.725 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:08.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:06.912412+0000 mgr.y (mgr.24481) 810 : cluster [DBG] pgmap v1126: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 6 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 61 active+clean; 629 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s; 1302/30789 objects degraded (4.229%); 10149/30789 objects misplaced (32.963%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T17:11:08.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:07.581852+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e561: 8 total, 7 up, 7 in 2024-02-12T17:11:08.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:06.912412+0000 mgr.y (mgr.24481) 810 : cluster [DBG] pgmap v1126: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 6 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 61 active+clean; 629 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s; 1302/30789 objects degraded (4.229%); 10149/30789 objects misplaced (32.963%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T17:11:08.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:07.581852+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e561: 8 total, 7 up, 7 in 2024-02-12T17:11:08.807 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:08.923 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:08.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:06.912412+0000 mgr.y (mgr.24481) 810 : cluster [DBG] pgmap v1126: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 6 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 61 active+clean; 629 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s; 1302/30789 objects degraded (4.229%); 10149/30789 objects misplaced (32.963%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T17:11:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:07.581852+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e561: 8 total, 7 up, 7 in 2024-02-12T17:11:09.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:09.071 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:09.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:09.201 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:09.306 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:09.389 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:09.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:09.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:09.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:09.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:09.703 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:09.729 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:09.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:09.807 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T17:11:09.833 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:09.858 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:08.599767+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e562: 8 total, 7 up, 7 in 2024-02-12T17:11:09.859 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:08.914159+0000 mgr.y (mgr.24481) 811 : cluster [DBG] pgmap v1129: 148 pgs: 2 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 62 active+clean; 523 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 572 op/s; 1219/25701 objects degraded (4.743%); 8412/25701 objects misplaced (32.730%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T17:11:09.861 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:09.881 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:08.599767+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e562: 8 total, 7 up, 7 in 2024-02-12T17:11:09.881 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:08.914159+0000 mgr.y (mgr.24481) 811 : cluster [DBG] pgmap v1129: 148 pgs: 2 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 62 active+clean; 523 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 572 op/s; 1219/25701 objects degraded (4.743%); 8412/25701 objects misplaced (32.730%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T17:11:09.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:08.599767+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e562: 8 total, 7 up, 7 in 2024-02-12T17:11:09.882 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:08.914159+0000 mgr.y (mgr.24481) 811 : cluster [DBG] pgmap v1129: 148 pgs: 2 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 62 active+clean; 523 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 572 op/s; 1219/25701 objects degraded (4.743%); 8412/25701 objects misplaced (32.730%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T17:11:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:09.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:09.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:10.023 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:10.045 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:10.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:10.150 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:10.215 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:10.253 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:10.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:10.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:10.392 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:10.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:10.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:10.527 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:11:10.553 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:10.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:10.687 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:10.748 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:10.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:10.855 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:10.931 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:10.981 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:10.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:10.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:10.993 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:11.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:11.121 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:11.146 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:11.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:11.250 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:11.279 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:11:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:10] "GET /metrics HTTP/1.1" 200 176767 "" "Prometheus/2.33.4" 2024-02-12T17:11:11.300 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:11.380 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:11.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:11.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:11.566 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:11.682 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:11.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:11.790 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:11.863 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:12.107 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:12.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:10.917199+0000 mgr.y (mgr.24481) 812 : cluster [DBG] pgmap v1130: 148 pgs: 2 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 62 active+clean; 508 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 938 B/s rd, 0 B/s wr, 464 op/s; 1145/24972 objects degraded (4.585%); 8360/24972 objects misplaced (33.477%); 3.1 MiB/s, 50 objects/s recovering 2024-02-12T17:11:12.289 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:11.554557+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 1145/24972 objects degraded (4.585%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:12.289 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:10.917199+0000 mgr.y (mgr.24481) 812 : cluster [DBG] pgmap v1130: 148 pgs: 2 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 62 active+clean; 508 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 938 B/s rd, 0 B/s wr, 464 op/s; 1145/24972 objects degraded (4.585%); 8360/24972 objects misplaced (33.477%); 3.1 MiB/s, 50 objects/s recovering 2024-02-12T17:11:12.289 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:11.554557+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 1145/24972 objects degraded (4.585%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:12.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:10.917199+0000 mgr.y (mgr.24481) 812 : cluster [DBG] pgmap v1130: 148 pgs: 2 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfill_wait+backfill_toofull, 9 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 62 active+clean; 508 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 938 B/s rd, 0 B/s wr, 464 op/s; 1145/24972 objects degraded (4.585%); 8360/24972 objects misplaced (33.477%); 3.1 MiB/s, 50 objects/s recovering 2024-02-12T17:11:12.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:11.554557+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 1145/24972 objects degraded (4.585%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:12.731 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:12.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:12.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:12.936 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:12.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:11:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:11:13.007 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:13.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:13.042 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:11:13.113 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:13.127 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1, 6] out_osds: [4] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-12T17:11:13.127 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:11:13.127 INFO:tasks.thrashosds.thrasher:Shrinking pool .rgw.root 2024-02-12T17:11:13.127 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .rgw.root by 10 min 8 2024-02-12T17:11:13.127 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:11:13.225 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:13.283 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:12.790512+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.33:0/2497357475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:13.283 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:12.790512+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.33:0/2497357475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:13.402 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:13.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:12.790512+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.33:0/2497357475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:13.515 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:13.549 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:13.576 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:13.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:13.680 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:13.756 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:14.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:12.918421+0000 mgr.y (mgr.24481) 813 : cluster [DBG] pgmap v1131: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 10 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 63 active+clean; 451 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 794 B/s rd, 0 B/s wr, 442 op/s; 965/22248 objects degraded (4.337%); 7391/22248 objects misplaced (33.221%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T17:11:14.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:13.013168+0000 mon.a (mon.0) 1882 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:14.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:13.013513+0000 mon.a (mon.0) 1883 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:11:14.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:13.028985+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? v1:172.21.15.33:0/2497357475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:11:14.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:13.029053+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e563: 8 total, 7 up, 7 in 2024-02-12T17:11:14.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:12.918421+0000 mgr.y (mgr.24481) 813 : cluster [DBG] pgmap v1131: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 10 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 63 active+clean; 451 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 794 B/s rd, 0 B/s wr, 442 op/s; 965/22248 objects degraded (4.337%); 7391/22248 objects misplaced (33.221%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T17:11:14.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:13.013168+0000 mon.a (mon.0) 1882 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:14.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:13.013513+0000 mon.a (mon.0) 1883 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:11:14.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:13.028985+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? v1:172.21.15.33:0/2497357475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:11:14.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:13.029053+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e563: 8 total, 7 up, 7 in 2024-02-12T17:11:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:12.918421+0000 mgr.y (mgr.24481) 813 : cluster [DBG] pgmap v1131: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 10 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 5 active+remapped+backfill_wait, 63 active+clean; 451 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 794 B/s rd, 0 B/s wr, 442 op/s; 965/22248 objects degraded (4.337%); 7391/22248 objects misplaced (33.221%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T17:11:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:13.013168+0000 mon.a (mon.0) 1882 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:13.013513+0000 mon.a (mon.0) 1883 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:11:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:13.028985+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? v1:172.21.15.33:0/2497357475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:11:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:13.029053+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e563: 8 total, 7 up, 7 in 2024-02-12T17:11:15.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:11:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:11:15.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:14.034495+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e564: 8 total, 7 up, 7 in 2024-02-12T17:11:15.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:14.034495+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e564: 8 total, 7 up, 7 in 2024-02-12T17:11:15.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:14.034495+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e564: 8 total, 7 up, 7 in 2024-02-12T17:11:16.392 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:14.759924+0000 mgr.y (mgr.24481) 814 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:16.392 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:14.919052+0000 mgr.y (mgr.24481) 815 : cluster [DBG] pgmap v1134: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 10 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 4 active+remapped+backfill_wait, 63 active+clean; 306 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 810 B/s rd, 0 B/s wr, 625 op/s; 742/15279 objects degraded (4.856%); 5169/15279 objects misplaced (33.831%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T17:11:16.392 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:15.044822+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e565: 8 total, 7 up, 7 in 2024-02-12T17:11:16.392 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:14.759924+0000 mgr.y (mgr.24481) 814 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:16.393 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:14.919052+0000 mgr.y (mgr.24481) 815 : cluster [DBG] pgmap v1134: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 10 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 4 active+remapped+backfill_wait, 63 active+clean; 306 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 810 B/s rd, 0 B/s wr, 625 op/s; 742/15279 objects degraded (4.856%); 5169/15279 objects misplaced (33.831%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T17:11:16.393 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:15.044822+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e565: 8 total, 7 up, 7 in 2024-02-12T17:11:16.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:14.759924+0000 mgr.y (mgr.24481) 814 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:16.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:14.919052+0000 mgr.y (mgr.24481) 815 : cluster [DBG] pgmap v1134: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 10 active+clean+remapped, 27 active+undersized, 5 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 4 active+remapped+backfill_wait, 63 active+clean; 306 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 810 B/s rd, 0 B/s wr, 625 op/s; 742/15279 objects degraded (4.856%); 5169/15279 objects misplaced (33.831%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T17:11:16.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:15.044822+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e565: 8 total, 7 up, 7 in 2024-02-12T17:11:16.657 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:16.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:16.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:16.816 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:17.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:16.559446+0000 mon.a (mon.0) 1888 : cluster [WRN] Health check update: Degraded data redundancy: 742/15279 objects degraded (4.856%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:17.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:16.593769+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e566: 8 total, 7 up, 7 in 2024-02-12T17:11:17.489 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:16.559446+0000 mon.a (mon.0) 1888 : cluster [WRN] Health check update: Degraded data redundancy: 742/15279 objects degraded (4.856%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:17.489 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:16.593769+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e566: 8 total, 7 up, 7 in 2024-02-12T17:11:17.489 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:16.559446+0000 mon.a (mon.0) 1888 : cluster [WRN] Health check update: Degraded data redundancy: 742/15279 objects degraded (4.856%), 33 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:16.593769+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e566: 8 total, 7 up, 7 in 2024-02-12T17:11:17.934 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:11:17.934 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:11:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:16.921710+0000 mgr.y (mgr.24481) 816 : cluster [DBG] pgmap v1137: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 11 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 4 active+remapped+backfill_wait, 63 active+clean; 251 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s; 548/12642 objects degraded (4.335%); 4414/12642 objects misplaced (34.915%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:11:18.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:17.522494+0000 mon.a (mon.0) 1890 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:18.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:16.921710+0000 mgr.y (mgr.24481) 816 : cluster [DBG] pgmap v1137: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 11 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 4 active+remapped+backfill_wait, 63 active+clean; 251 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s; 548/12642 objects degraded (4.335%); 4414/12642 objects misplaced (34.915%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:11:18.282 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:17.522494+0000 mon.a (mon.0) 1890 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:16.921710+0000 mgr.y (mgr.24481) 816 : cluster [DBG] pgmap v1137: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 11 active+clean+remapped, 27 active+undersized, 4 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 31 active+undersized+degraded, 4 active+remapped+backfill_wait, 63 active+clean; 251 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s; 548/12642 objects degraded (4.335%); 4414/12642 objects misplaced (34.915%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T17:11:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:17.522494+0000 mon.a (mon.0) 1890 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:18.795 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:11:18.817 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:11:18.848 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:11:18.879 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:11:18.900 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:11:18.921 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:11:18.942 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:11:18.942 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:11:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:11:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:11:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:11:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:11:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:11:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:11:18.943 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:11:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:18.072351+0000 mon.a (mon.0) 1891 : cluster [DBG] osdmap e567: 8 total, 7 up, 7 in 2024-02-12T17:11:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:18.762598+0000 mon.a (mon.0) 1892 : audit [DBG] from='client.? v1:172.21.15.33:0/650791317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:19.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:18.072351+0000 mon.a (mon.0) 1891 : cluster [DBG] osdmap e567: 8 total, 7 up, 7 in 2024-02-12T17:11:19.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:18.762598+0000 mon.a (mon.0) 1892 : audit [DBG] from='client.? v1:172.21.15.33:0/650791317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:19.367 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:19.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:19.474 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:19.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:18.072351+0000 mon.a (mon.0) 1891 : cluster [DBG] osdmap e567: 8 total, 7 up, 7 in 2024-02-12T17:11:19.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:18.762598+0000 mon.a (mon.0) 1892 : audit [DBG] from='client.? v1:172.21.15.33:0/650791317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:19.512 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:19.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:19.644 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:19.693 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:20.126 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T17:11:20.177 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:20.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:20.198 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:20.288 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:20.342 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:20.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:20.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:20.476 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:18.922861+0000 mgr.y (mgr.24481) 817 : cluster [DBG] pgmap v1139: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 28 active+undersized, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 2 active+remapped+backfill_wait, 63 active+clean; 123 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 680 op/s; 455/6468 objects degraded (7.035%); 2394/6468 objects misplaced (37.013%); 5.2 MiB/s, 82 objects/s recovering 2024-02-12T17:11:20.476 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:19.097627+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e568: 8 total, 7 up, 7 in 2024-02-12T17:11:20.478 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:20.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:18.922861+0000 mgr.y (mgr.24481) 817 : cluster [DBG] pgmap v1139: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 28 active+undersized, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 2 active+remapped+backfill_wait, 63 active+clean; 123 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 680 op/s; 455/6468 objects degraded (7.035%); 2394/6468 objects misplaced (37.013%); 5.2 MiB/s, 82 objects/s recovering 2024-02-12T17:11:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:19.097627+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e568: 8 total, 7 up, 7 in 2024-02-12T17:11:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:18.922861+0000 mgr.y (mgr.24481) 817 : cluster [DBG] pgmap v1139: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 28 active+undersized, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 2 active+remapped+backfill_wait, 63 active+clean; 123 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 680 op/s; 455/6468 objects degraded (7.035%); 2394/6468 objects misplaced (37.013%); 5.2 MiB/s, 82 objects/s recovering 2024-02-12T17:11:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:19.097627+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e568: 8 total, 7 up, 7 in 2024-02-12T17:11:20.602 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:20.630 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:20.762 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:20.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:11:20.775 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:11:20.867 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:20.894 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:20.913 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:20.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:21.018 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:21.061 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:21.083 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:21.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:21.140 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:20.134580+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e569: 8 total, 7 up, 7 in 2024-02-12T17:11:21.140 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:11:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:20] "GET /metrics HTTP/1.1" 200 176765 "" "Prometheus/2.33.4" 2024-02-12T17:11:21.188 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:21.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:20.134580+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e569: 8 total, 7 up, 7 in 2024-02-12T17:11:21.280 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:21.316 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:21.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:21.415 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:11:21.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:21.441 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:20.134580+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e569: 8 total, 7 up, 7 in 2024-02-12T17:11:21.447 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:21.465 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:21.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:21.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:21.614 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:21.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:21.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:21.723 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:21.744 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:21.763 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:21.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:21.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:21.892 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:21.912 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:21.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:21.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:22.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:22.042 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:22.060 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:22.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:22.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:22.224 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:22.225 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:20.923742+0000 mgr.y (mgr.24481) 818 : cluster [DBG] pgmap v1142: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 3 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 2 active+remapped+backfill_wait, 63 active+clean; 104 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 560 op/s; 386/5586 objects degraded (6.910%); 2340/5586 objects misplaced (41.890%); 4.1 MiB/s, 64 objects/s recovering 2024-02-12T17:11:22.225 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:21.141409+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e570: 8 total, 7 up, 7 in 2024-02-12T17:11:22.225 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:21.561181+0000 mon.a (mon.0) 1896 : cluster [WRN] Health check update: Degraded data redundancy: 386/5586 objects degraded (6.910%), 31 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:22.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:20.923742+0000 mgr.y (mgr.24481) 818 : cluster [DBG] pgmap v1142: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 3 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 2 active+remapped+backfill_wait, 63 active+clean; 104 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 560 op/s; 386/5586 objects degraded (6.910%); 2340/5586 objects misplaced (41.890%); 4.1 MiB/s, 64 objects/s recovering 2024-02-12T17:11:22.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:21.141409+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e570: 8 total, 7 up, 7 in 2024-02-12T17:11:22.232 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:21.561181+0000 mon.a (mon.0) 1896 : cluster [WRN] Health check update: Degraded data redundancy: 386/5586 objects degraded (6.910%), 31 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:22.251 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Removed 123169 objects 2024-02-12T17:11:22.251 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Clean up completed and total clean up time :195.295 2024-02-12T17:11:22.263 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:22.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:22.369 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:22.400 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:20.923742+0000 mgr.y (mgr.24481) 818 : cluster [DBG] pgmap v1142: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 3 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 2 active+remapped+backfill_wait, 63 active+clean; 104 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 560 op/s; 386/5586 objects degraded (6.910%); 2340/5586 objects misplaced (41.890%); 4.1 MiB/s, 64 objects/s recovering 2024-02-12T17:11:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:21.141409+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e570: 8 total, 7 up, 7 in 2024-02-12T17:11:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:21.561181+0000 mon.a (mon.0) 1896 : cluster [WRN] Health check update: Degraded data redundancy: 386/5586 objects degraded (6.910%), 31 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:22.607 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:22.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:22.612 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:22.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:22.738 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:22.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:22.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:11:22.861 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:22.885 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:11:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:11:22.887 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:22.906 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:22.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:23.009 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:23.038 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:23.059 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:23.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:23.165 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:23.194 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:23.350 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:22.147035+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e571: 8 total, 7 up, 7 in 2024-02-12T17:11:23.350 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:22.147035+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e571: 8 total, 7 up, 7 in 2024-02-12T17:11:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:23.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:23.452 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:23.478 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:22.147035+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e571: 8 total, 7 up, 7 in 2024-02-12T17:11:23.480 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:23.500 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:23.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:23.604 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:23.633 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:23.652 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:23.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:23.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:23.782 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:23.800 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:23.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:23.903 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:23.938 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:24.067 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:24.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:24.170 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:22.924354+0000 mgr.y (mgr.24481) 819 : cluster [DBG] pgmap v1145: 148 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 3 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 1 active+remapped+backfill_wait, 63 active+clean; 73 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 227 op/s; 214/4092 objects degraded (5.230%); 2000/4092 objects misplaced (48.876%); 1.6 MiB/s, 28 objects/s recovering 2024-02-12T17:11:24.171 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:22.924354+0000 mgr.y (mgr.24481) 819 : cluster [DBG] pgmap v1145: 148 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 3 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 1 active+remapped+backfill_wait, 63 active+clean; 73 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 227 op/s; 214/4092 objects degraded (5.230%); 2000/4092 objects misplaced (48.876%); 1.6 MiB/s, 28 objects/s recovering 2024-02-12T17:11:24.173 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:24.218 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:24.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:24.322 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:24.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:24.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:24.489 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:22.924354+0000 mgr.y (mgr.24481) 819 : cluster [DBG] pgmap v1145: 148 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait+backfill_toofull, 13 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 3 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 1 active+remapped+backfill_wait, 63 active+clean; 73 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 227 op/s; 214/4092 objects degraded (5.230%); 2000/4092 objects misplaced (48.876%); 1.6 MiB/s, 28 objects/s recovering 2024-02-12T17:11:24.497 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:24.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:24.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:24.651 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:25.069 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:11:25.072 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:11:25.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:11:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:11:25.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:25.151412+0000 mon.a (mon.0) 1898 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:25.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:25.151412+0000 mon.a (mon.0) 1898 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:25.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:25.151412+0000 mon.a (mon.0) 1898 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:25.995 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:26.002 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:26.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:26.017 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1146,"stamp":"2024-02-12T17:11:24.924545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37263024,"num_objects":760,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":214,"num_objects_misplaced":1544,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":2099,"num_read_kb":1851,"num_write":243519,"num_write_kb":7638378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36020,"num_bytes_recovered":2310556626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190573,"ondisk_log_size":190573,"up":366,"acting":384,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7507140,"kb_used_data":1528204,"kb_used_omap":0,"kb_used_meta":5978880,"kb_avail":648708924,"statfs":{"total":671965249536,"available":664277938176,"internally_reserved":0,"allocated":1564880896,"data_stored":1559003234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6122373120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-91406348,"num_objects":-1396,"num_object_clones":0,"num_object_copies":-4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-241,"num_objects_misplaced":-850,"num_objects_unfound":0,"num_objects_dirty":-1396,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1853,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":11730968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.829964"},"pg_stats":[{"pgid":"10.23","version":"570'10235","reported_seq":11851,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.245007+0000","last_change":"2024-02-12T17:10:09.044643+0000","last_active":"2024-02-12T17:11:21.245007+0000","last_peered":"2024-02-12T17:11:21.245007+0000","last_clean":"2024-02-12T17:11:21.245007+0000","last_became_active":"2024-02-12T17:10:09.022343+0000","last_became_peered":"2024-02-12T17:10:09.022343+0000","last_unstale":"2024-02-12T17:11:21.245007+0000","last_undegraded":"2024-02-12T17:11:21.245007+0000","last_fullsized":"2024-02-12T17:11:21.245007+0000","mapping_epoch":537,"log_start":"466'4400","ondisk_log_start":"466'4400","created":503,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5835,"ondisk_log_size":5835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2260992,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":35,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":124096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":47251456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,2],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":35}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"569'9993","reported_seq":10156,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.149624+0000","last_change":"2024-02-12T17:10:03.954723+0000","last_active":"2024-02-12T17:11:21.149624+0000","last_peered":"2024-02-12T17:11:21.149624+0000","last_clean":"2024-02-12T17:11:21.149624+0000","last_became_active":"2024-02-12T17:10:03.943788+0000","last_became_peered":"2024-02-12T17:10:03.943788+0000","last_unstale":"2024-02-12T17:11:21.149624+0000","last_undegraded":"2024-02-12T17:11:21.149624+0000","last_fullsized":"2024-02-12T17:11:21.149624+0000","mapping_epoch":533,"log_start":"466'4100","ondisk_log_start":"466'4100","created":503,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5893,"ondisk_log_size":5893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6242304,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":95,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":122096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,3],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":95}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"569'10365","reported_seq":10567,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.154856+0000","last_change":"2024-02-12T17:09:04.471058+0000","last_active":"2024-02-12T17:11:21.154856+0000","last_peered":"2024-02-12T17:11:21.154856+0000","last_clean":"2024-02-12T17:11:21.154856+0000","last_became_active":"2024-02-12T17:09:04.466720+0000","last_became_peered":"2024-02-12T17:09:04.466720+0000","last_unstale":"2024-02-12T17:11:21.154856+0000","last_undegraded":"2024-02-12T17:11:21.154856+0000","last_fullsized":"2024-02-12T17:11:21.154856+0000","mapping_epoch":506,"log_start":"466'4500","ondisk_log_start":"466'4500","created":503,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5865,"ondisk_log_size":5865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3921,"num_write_kb":125264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":688128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.20","version":"569'10089","reported_seq":10589,"reported_epoch":569,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:20.358780+0000","last_change":"2024-02-12T17:11:08.670209+0000","last_active":"2024-02-12T17:11:20.358780+0000","last_peered":"2024-02-12T17:11:20.358780+0000","last_clean":"2024-02-12T17:11:20.358780+0000","last_became_active":"2024-02-12T17:11:08.635120+0000","last_became_peered":"2024-02-12T17:11:08.635120+0000","last_unstale":"2024-02-12T17:11:20.358780+0000","last_undegraded":"2024-02-12T17:11:20.358780+0000","last_fullsized":"2024-02-12T17:11:20.358780+0000","mapping_epoch":561,"log_start":"466'4200","ondisk_log_start":"466'4200","created":503,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5889,"ondisk_log_size":5889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":120992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":10223616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.17","version":"569'11481","reported_seq":12141,"reported_epoch":570,"state":"remapped+peering","last_fresh":"2024-02-12T17:11:21.157990+0000","last_change":"2024-02-12T17:11:21.159198+0000","last_active":"2024-02-12T17:11:21.157990+0000","last_peered":"2024-02-12T17:11:20.273293+0000","last_clean":"2024-02-12T17:11:20.273293+0000","last_became_active":"2024-02-12T17:10:04.939799+0000","last_became_peered":"2024-02-12T17:10:04.939799+0000","last_unstale":"2024-02-12T17:11:21.157990+0000","last_undegraded":"2024-02-12T17:11:21.157990+0000","last_fullsized":"2024-02-12T17:11:21.157990+0000","mapping_epoch":570,"log_start":"495'7918","ondisk_log_start":"495'7918","created":480,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3563,"ondisk_log_size":3563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7673,"num_write_kb":245504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":15335424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":288,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.169068+0000","last_change":"2024-02-12T17:10:03.887377+0000","last_active":"2024-02-12T17:11:21.169068+0000","last_peered":"2024-02-12T17:11:21.169068+0000","last_clean":"2024-02-12T17:09:58.613399+0000","last_became_active":"2024-02-12T17:10:03.887377+0000","last_became_peered":"2024-02-12T17:10:03.887377+0000","last_unstale":"2024-02-12T17:11:21.169068+0000","last_undegraded":"2024-02-12T17:11:21.169068+0000","last_fullsized":"2024-02-12T17:10:03.858724+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:44.261302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:44.261302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"526'39","reported_seq":270,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.151063+0000","last_change":"2024-02-12T17:07:35.884990+0000","last_active":"2024-02-12T17:11:20.151063+0000","last_peered":"2024-02-12T17:11:20.151063+0000","last_clean":"2024-02-12T17:11:20.151063+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:11:20.151063+0000","last_undegraded":"2024-02-12T17:11:20.151063+0000","last_fullsized":"2024-02-12T17:11:20.151063+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"569'11564","reported_seq":23746,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.143850+0000","last_change":"2024-02-12T17:11:15.130747+0000","last_active":"2024-02-12T17:11:21.143850+0000","last_peered":"2024-02-12T17:11:21.143850+0000","last_clean":"2024-02-12T17:11:21.143850+0000","last_became_active":"2024-02-12T17:11:15.082511+0000","last_became_peered":"2024-02-12T17:11:15.082511+0000","last_unstale":"2024-02-12T17:11:21.143850+0000","last_undegraded":"2024-02-12T17:11:21.143850+0000","last_fullsized":"2024-02-12T17:11:21.143850+0000","mapping_epoch":564,"log_start":"479'7624","ondisk_log_start":"479'7624","created":480,"last_epoch_clean":565,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3940,"ondisk_log_size":3940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4718592,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":72,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7712,"num_write_kb":249088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5988,"num_bytes_recovered":392429568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5,2],"avail_no_missing":["6","2","5"],"object_location_counts":[{"shards":"2,5,6","objects":72}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":236,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.237586+0000","last_change":"2024-02-12T17:07:37.890052+0000","last_active":"2024-02-12T17:11:20.237586+0000","last_peered":"2024-02-12T17:11:20.237586+0000","last_clean":"2024-02-12T17:11:20.237586+0000","last_became_active":"2024-02-12T17:03:50.545485+0000","last_became_peered":"2024-02-12T17:03:50.545485+0000","last_unstale":"2024-02-12T17:11:20.237586+0000","last_undegraded":"2024-02-12T17:11:20.237586+0000","last_fullsized":"2024-02-12T17:11:20.237586+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:37.889956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:01:39.176431+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.889956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":92,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.142944+0000","last_change":"2024-02-12T17:10:03.866489+0000","last_active":"2024-02-12T17:11:21.142944+0000","last_peered":"2024-02-12T17:11:21.142944+0000","last_clean":"2024-02-12T17:09:58.609245+0000","last_became_active":"2024-02-12T17:10:03.866489+0000","last_became_peered":"2024-02-12T17:10:03.866489+0000","last_unstale":"2024-02-12T17:11:21.142944+0000","last_undegraded":"2024-02-12T17:10:03.856093+0000","last_fullsized":"2024-02-12T17:10:03.855974+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:07:47.845930+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.845930+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"569'11367","reported_seq":15767,"reported_epoch":570,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-12T17:11:21.411981+0000","last_change":"2024-02-12T17:10:03.999909+0000","last_active":"2024-02-12T17:11:21.411981+0000","last_peered":"2024-02-12T17:11:21.411981+0000","last_clean":"2024-02-12T17:07:53.075624+0000","last_became_active":"2024-02-12T17:10:03.953116+0000","last_became_peered":"2024-02-12T17:10:03.953116+0000","last_unstale":"2024-02-12T17:11:21.411981+0000","last_undegraded":"2024-02-12T17:10:03.861176+0000","last_fullsized":"2024-02-12T17:10:03.856341+0000","mapping_epoch":533,"log_start":"495'7795","ondisk_log_start":"495'7795","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"470'5845","last_scrub_stamp":"2024-02-12T17:07:27.706391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:07:27.706391+0000","log_size":3572,"ondisk_log_size":3572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":44,"num_objects_misplaced":659,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7594,"num_write_kb":244416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2117,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,2],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"2,3","objects":-615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":579,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.166859+0000","last_change":"2024-02-12T17:09:21.483158+0000","last_active":"2024-02-12T17:11:21.166859+0000","last_peered":"2024-02-12T17:11:21.166859+0000","last_clean":"2024-02-12T17:11:21.166859+0000","last_became_active":"2024-02-12T17:09:21.482825+0000","last_became_peered":"2024-02-12T17:09:21.482825+0000","last_unstale":"2024-02-12T17:11:21.166859+0000","last_undegraded":"2024-02-12T17:11:21.166859+0000","last_fullsized":"2024-02-12T17:11:21.166859+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:29.165802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:04.051345+0000","last_clean_scrub_stamp":"2024-02-12T17:07:29.165802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":281,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.157491+0000","last_change":"2024-02-12T17:10:03.942545+0000","last_active":"2024-02-12T17:11:20.157491+0000","last_peered":"2024-02-12T17:11:20.157491+0000","last_clean":"2024-02-12T17:09:58.614625+0000","last_became_active":"2024-02-12T17:10:03.942545+0000","last_became_peered":"2024-02-12T17:10:03.942545+0000","last_unstale":"2024-02-12T17:11:20.157491+0000","last_undegraded":"2024-02-12T17:10:03.861855+0000","last_fullsized":"2024-02-12T17:10:03.861756+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7","last_scrub_stamp":"2024-02-12T17:05:56.388366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.388366+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"568'11473","reported_seq":12034,"reported_epoch":569,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:20.157499+0000","last_change":"2024-02-12T17:10:04.066512+0000","last_active":"2024-02-12T17:11:20.157499+0000","last_peered":"2024-02-12T17:11:20.157499+0000","last_clean":"2024-02-12T17:11:20.157499+0000","last_became_active":"2024-02-12T17:10:03.937886+0000","last_became_peered":"2024-02-12T17:10:03.937886+0000","last_unstale":"2024-02-12T17:11:20.157499+0000","last_undegraded":"2024-02-12T17:11:20.157499+0000","last_fullsized":"2024-02-12T17:11:20.157499+0000","mapping_epoch":533,"log_start":"496'7969","ondisk_log_start":"496'7969","created":480,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3504,"ondisk_log_size":3504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7679,"num_write_kb":245664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":12713984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":544,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.166762+0000","last_change":"2024-02-12T17:07:39.235715+0000","last_active":"2024-02-12T17:11:21.166762+0000","last_peered":"2024-02-12T17:11:21.166762+0000","last_clean":"2024-02-12T17:11:21.166762+0000","last_became_active":"2024-02-12T16:49:39.103227+0000","last_became_peered":"2024-02-12T16:49:39.103227+0000","last_unstale":"2024-02-12T17:11:21.166762+0000","last_undegraded":"2024-02-12T17:11:21.166762+0000","last_fullsized":"2024-02-12T17:11:21.166762+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.235654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.235654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":288,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.154266+0000","last_change":"2024-02-12T17:09:23.653654+0000","last_active":"2024-02-12T17:11:20.154266+0000","last_peered":"2024-02-12T17:11:20.154266+0000","last_clean":"2024-02-12T17:11:20.154266+0000","last_became_active":"2024-02-12T17:09:21.602320+0000","last_became_peered":"2024-02-12T17:09:21.602320+0000","last_unstale":"2024-02-12T17:11:20.154266+0000","last_undegraded":"2024-02-12T17:11:20.154266+0000","last_fullsized":"2024-02-12T17:11:20.154266+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"569'11611","reported_seq":19580,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.155388+0000","last_change":"2024-02-12T17:10:04.924672+0000","last_active":"2024-02-12T17:11:21.155388+0000","last_peered":"2024-02-12T17:11:21.155388+0000","last_clean":"2024-02-12T17:11:21.155388+0000","last_became_active":"2024-02-12T17:10:04.891552+0000","last_became_peered":"2024-02-12T17:10:04.891552+0000","last_unstale":"2024-02-12T17:11:21.155388+0000","last_undegraded":"2024-02-12T17:11:21.155388+0000","last_fullsized":"2024-02-12T17:11:21.155388+0000","mapping_epoch":534,"log_start":"495'7939","ondisk_log_start":"495'7939","created":480,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3672,"ondisk_log_size":3672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7776,"num_write_kb":248320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3896,"num_bytes_recovered":255328256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,3],"avail_no_missing":["7","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":254,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.157253+0000","last_change":"2024-02-12T17:09:21.507961+0000","last_active":"2024-02-12T17:11:20.157253+0000","last_peered":"2024-02-12T17:11:20.157253+0000","last_clean":"2024-02-12T17:11:20.157253+0000","last_became_active":"2024-02-12T17:09:21.507651+0000","last_became_peered":"2024-02-12T17:09:21.507651+0000","last_unstale":"2024-02-12T17:11:20.157253+0000","last_undegraded":"2024-02-12T17:11:20.157253+0000","last_fullsized":"2024-02-12T17:11:20.157253+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:29.479610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T17:06:29.479610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":318,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.186473+0000","last_change":"2024-02-12T17:10:03.890949+0000","last_active":"2024-02-12T17:11:20.186473+0000","last_peered":"2024-02-12T17:11:20.186473+0000","last_clean":"2024-02-12T17:09:58.620042+0000","last_became_active":"2024-02-12T17:10:03.890949+0000","last_became_peered":"2024-02-12T17:10:03.890949+0000","last_unstale":"2024-02-12T17:11:20.186473+0000","last_undegraded":"2024-02-12T17:10:03.864727+0000","last_fullsized":"2024-02-12T17:10:03.863335+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'18","last_scrub_stamp":"2024-02-12T17:02:40.949850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.949850+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"569'11455","reported_seq":12129,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-02-12T17:11:22.248312+0000","last_change":"2024-02-12T17:11:22.248312+0000","last_active":"2024-02-12T17:11:22.248312+0000","last_peered":"2024-02-12T17:11:22.248312+0000","last_clean":"2024-02-12T17:11:22.248312+0000","last_became_active":"2024-02-12T17:11:22.157938+0000","last_became_peered":"2024-02-12T17:11:22.157938+0000","last_unstale":"2024-02-12T17:11:22.248312+0000","last_undegraded":"2024-02-12T17:11:22.248312+0000","last_fullsized":"2024-02-12T17:11:22.248312+0000","mapping_epoch":570,"log_start":"490'7613","ondisk_log_start":"490'7613","created":480,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3842,"ondisk_log_size":3842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7685,"num_write_kb":244192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":15859712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":67,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.144802+0000","last_change":"2024-02-12T17:09:21.560647+0000","last_active":"2024-02-12T17:11:21.144802+0000","last_peered":"2024-02-12T17:11:21.144802+0000","last_clean":"2024-02-12T17:11:21.144802+0000","last_became_active":"2024-02-12T17:09:21.559769+0000","last_became_peered":"2024-02-12T17:09:21.559769+0000","last_unstale":"2024-02-12T17:11:21.144802+0000","last_undegraded":"2024-02-12T17:11:21.144802+0000","last_fullsized":"2024-02-12T17:11:21.144802+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:52.828449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:06:52.828449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":121,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.158210+0000","last_change":"2024-02-12T17:10:03.934455+0000","last_active":"2024-02-12T17:11:20.158210+0000","last_peered":"2024-02-12T17:11:20.158210+0000","last_clean":"2024-02-12T17:09:50.594583+0000","last_became_active":"2024-02-12T17:10:03.934455+0000","last_became_peered":"2024-02-12T17:10:03.934455+0000","last_unstale":"2024-02-12T17:11:20.158210+0000","last_undegraded":"2024-02-12T17:10:03.866782+0000","last_fullsized":"2024-02-12T17:10:03.866683+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.11","version":"569'11722","reported_seq":14126,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.146230+0000","last_change":"2024-02-12T17:11:01.395814+0000","last_active":"2024-02-12T17:11:21.146230+0000","last_peered":"2024-02-12T17:11:21.146230+0000","last_clean":"2024-02-12T17:11:21.146230+0000","last_became_active":"2024-02-12T17:11:01.341878+0000","last_became_peered":"2024-02-12T17:11:01.341878+0000","last_unstale":"2024-02-12T17:11:21.146230+0000","last_undegraded":"2024-02-12T17:11:21.146230+0000","last_fullsized":"2024-02-12T17:11:21.146230+0000","mapping_epoch":554,"log_start":"490'7884","ondisk_log_start":"490'7884","created":480,"last_epoch_clean":555,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3838,"ondisk_log_size":3838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":425984,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":251360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":73269248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":240,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.151487+0000","last_change":"2024-02-12T17:09:21.496479+0000","last_active":"2024-02-12T17:11:20.151487+0000","last_peered":"2024-02-12T17:11:20.151487+0000","last_clean":"2024-02-12T17:11:20.151487+0000","last_became_active":"2024-02-12T17:09:21.493389+0000","last_became_peered":"2024-02-12T17:09:21.493389+0000","last_unstale":"2024-02-12T17:11:20.151487+0000","last_undegraded":"2024-02-12T17:11:20.151487+0000","last_fullsized":"2024-02-12T17:11:20.151487+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:24.819667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:24.819667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":97,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.144392+0000","last_change":"2024-02-12T17:10:03.869766+0000","last_active":"2024-02-12T17:11:21.144392+0000","last_peered":"2024-02-12T17:11:21.144392+0000","last_clean":"2024-02-12T17:09:58.609101+0000","last_became_active":"2024-02-12T17:10:03.869766+0000","last_became_peered":"2024-02-12T17:10:03.869766+0000","last_unstale":"2024-02-12T17:11:21.144392+0000","last_undegraded":"2024-02-12T17:10:03.860263+0000","last_fullsized":"2024-02-12T17:10:03.860160+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"569'11147","reported_seq":12809,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.155060+0000","last_change":"2024-02-12T17:11:04.495312+0000","last_active":"2024-02-12T17:11:21.155060+0000","last_peered":"2024-02-12T17:11:21.155060+0000","last_clean":"2024-02-12T17:11:21.155060+0000","last_became_active":"2024-02-12T17:11:04.453866+0000","last_became_peered":"2024-02-12T17:11:04.453866+0000","last_unstale":"2024-02-12T17:11:21.155060+0000","last_undegraded":"2024-02-12T17:11:21.155060+0000","last_fullsized":"2024-02-12T17:11:21.155060+0000","mapping_epoch":557,"log_start":"481'7452","ondisk_log_start":"481'7452","created":480,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3695,"ondisk_log_size":3695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7426,"num_write_kb":240768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":748,"num_bytes_recovered":49020928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":569,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.146501+0000","last_change":"2024-02-12T17:09:21.507645+0000","last_active":"2024-02-12T17:11:21.146501+0000","last_peered":"2024-02-12T17:11:21.146501+0000","last_clean":"2024-02-12T17:11:21.146501+0000","last_became_active":"2024-02-12T17:09:21.507203+0000","last_became_peered":"2024-02-12T17:09:21.507203+0000","last_unstale":"2024-02-12T17:11:21.146501+0000","last_undegraded":"2024-02-12T17:11:21.146501+0000","last_fullsized":"2024-02-12T17:11:21.146501+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:55.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.027189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":633,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.168119+0000","last_change":"2024-02-12T17:10:03.919345+0000","last_active":"2024-02-12T17:11:21.168119+0000","last_peered":"2024-02-12T17:11:21.168119+0000","last_clean":"2024-02-12T17:09:58.612919+0000","last_became_active":"2024-02-12T17:10:03.919345+0000","last_became_peered":"2024-02-12T17:10:03.919345+0000","last_unstale":"2024-02-12T17:11:21.168119+0000","last_undegraded":"2024-02-12T17:10:03.858535+0000","last_fullsized":"2024-02-12T17:10:03.858425+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"570'12766","reported_seq":13419,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.256263+0000","last_change":"2024-02-12T17:11:21.256056+0000","last_active":"2024-02-12T17:11:21.256263+0000","last_peered":"2024-02-12T17:11:21.256263+0000","last_clean":"2024-02-12T17:11:21.256263+0000","last_became_active":"2024-02-12T17:11:21.185104+0000","last_became_peered":"2024-02-12T17:11:21.185104+0000","last_unstale":"2024-02-12T17:11:21.256263+0000","last_undegraded":"2024-02-12T17:11:21.256263+0000","last_fullsized":"2024-02-12T17:11:21.256263+0000","mapping_epoch":569,"log_start":"496'8415","ondisk_log_start":"496'8415","created":503,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4351,"ondisk_log_size":4351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7687,"num_write_kb":245664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":15204352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,3],"avail_no_missing":["7","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":255,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.155232+0000","last_change":"2024-02-12T17:09:54.524875+0000","last_active":"2024-02-12T17:11:21.155232+0000","last_peered":"2024-02-12T17:11:21.155232+0000","last_clean":"2024-02-12T17:11:21.155232+0000","last_became_active":"2024-02-12T17:09:54.524490+0000","last_became_peered":"2024-02-12T17:09:54.524490+0000","last_unstale":"2024-02-12T17:11:21.155232+0000","last_undegraded":"2024-02-12T17:11:21.155232+0000","last_fullsized":"2024-02-12T17:11:21.155232+0000","mapping_epoch":528,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":254,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.157627+0000","last_change":"2024-02-12T17:09:21.547851+0000","last_active":"2024-02-12T17:11:20.157627+0000","last_peered":"2024-02-12T17:11:20.157627+0000","last_clean":"2024-02-12T17:11:20.157627+0000","last_became_active":"2024-02-12T17:09:21.547628+0000","last_became_peered":"2024-02-12T17:09:21.547628+0000","last_unstale":"2024-02-12T17:11:20.157627+0000","last_undegraded":"2024-02-12T17:11:20.157627+0000","last_fullsized":"2024-02-12T17:11:20.157627+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":67,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.144522+0000","last_change":"2024-02-12T17:09:21.594631+0000","last_active":"2024-02-12T17:11:21.144522+0000","last_peered":"2024-02-12T17:11:21.144522+0000","last_clean":"2024-02-12T17:11:21.144522+0000","last_became_active":"2024-02-12T17:09:21.537322+0000","last_became_peered":"2024-02-12T17:09:21.537322+0000","last_unstale":"2024-02-12T17:11:21.144522+0000","last_undegraded":"2024-02-12T17:11:21.144522+0000","last_fullsized":"2024-02-12T17:11:21.144522+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:31.071549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:31.071549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"526'27","reported_seq":276,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.151338+0000","last_change":"2024-02-12T17:09:21.993424+0000","last_active":"2024-02-12T17:11:20.151338+0000","last_peered":"2024-02-12T17:11:20.151338+0000","last_clean":"2024-02-12T17:11:20.151338+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:11:20.151338+0000","last_undegraded":"2024-02-12T17:11:20.151338+0000","last_fullsized":"2024-02-12T17:11:20.151338+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"570'12708","reported_seq":12868,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.260818+0000","last_change":"2024-02-12T17:10:04.006396+0000","last_active":"2024-02-12T17:11:21.260818+0000","last_peered":"2024-02-12T17:11:21.260818+0000","last_clean":"2024-02-12T17:11:21.260818+0000","last_became_active":"2024-02-12T17:10:03.947606+0000","last_became_peered":"2024-02-12T17:10:03.947606+0000","last_unstale":"2024-02-12T17:11:21.260818+0000","last_undegraded":"2024-02-12T17:11:21.260818+0000","last_fullsized":"2024-02-12T17:11:21.260818+0000","mapping_epoch":533,"log_start":"481'7596","ondisk_log_start":"481'7596","created":503,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5112,"ondisk_log_size":5112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7639,"num_write_kb":243808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":73,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.143404+0000","last_change":"2024-02-12T17:10:03.891053+0000","last_active":"2024-02-12T17:11:21.143404+0000","last_peered":"2024-02-12T17:11:21.143404+0000","last_clean":"2024-02-12T17:09:58.609024+0000","last_became_active":"2024-02-12T17:10:03.891053+0000","last_became_peered":"2024-02-12T17:10:03.891053+0000","last_unstale":"2024-02-12T17:11:21.143404+0000","last_undegraded":"2024-02-12T17:11:21.143404+0000","last_fullsized":"2024-02-12T17:10:03.855518+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":542,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.154773+0000","last_change":"2024-02-12T17:06:03.911480+0000","last_active":"2024-02-12T17:11:21.154773+0000","last_peered":"2024-02-12T17:11:21.154773+0000","last_clean":"2024-02-12T17:11:21.154773+0000","last_became_active":"2024-02-12T17:04:39.886041+0000","last_became_peered":"2024-02-12T17:04:39.886041+0000","last_unstale":"2024-02-12T17:11:21.154773+0000","last_undegraded":"2024-02-12T17:11:21.154773+0000","last_fullsized":"2024-02-12T17:11:21.154773+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.911375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.911375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":587,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.147825+0000","last_change":"2024-02-12T17:09:21.508557+0000","last_active":"2024-02-12T17:11:21.147825+0000","last_peered":"2024-02-12T17:11:21.147825+0000","last_clean":"2024-02-12T17:11:21.147825+0000","last_became_active":"2024-02-12T17:09:21.508243+0000","last_became_peered":"2024-02-12T17:09:21.508243+0000","last_unstale":"2024-02-12T17:11:21.147825+0000","last_undegraded":"2024-02-12T17:11:21.147825+0000","last_fullsized":"2024-02-12T17:11:21.147825+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:34.981898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.981898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":299,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.146110+0000","last_change":"2024-02-12T17:10:03.867728+0000","last_active":"2024-02-12T17:11:20.146110+0000","last_peered":"2024-02-12T17:11:20.146110+0000","last_clean":"2024-02-12T17:09:58.609134+0000","last_became_active":"2024-02-12T17:10:03.867728+0000","last_became_peered":"2024-02-12T17:10:03.867728+0000","last_unstale":"2024-02-12T17:11:20.146110+0000","last_undegraded":"2024-02-12T17:10:03.854546+0000","last_fullsized":"2024-02-12T17:10:03.854479+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1d","version":"571'12938","reported_seq":13169,"reported_epoch":571,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:22.151187+0000","last_change":"2024-02-12T17:11:21.212001+0000","last_active":"2024-02-12T17:11:22.151187+0000","last_peered":"2024-02-12T17:11:22.151187+0000","last_clean":"2024-02-12T17:11:22.151187+0000","last_became_active":"2024-02-12T17:11:21.151081+0000","last_became_peered":"2024-02-12T17:11:21.151081+0000","last_unstale":"2024-02-12T17:11:22.151187+0000","last_undegraded":"2024-02-12T17:11:22.151187+0000","last_fullsized":"2024-02-12T17:11:22.151187+0000","mapping_epoch":569,"log_start":"490'7927","ondisk_log_start":"490'7927","created":503,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5011,"ondisk_log_size":5011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7827,"num_write_kb":249345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1179672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5,2],"avail_no_missing":["6","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":321,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.149666+0000","last_change":"2024-02-12T17:10:03.943360+0000","last_active":"2024-02-12T17:11:21.149666+0000","last_peered":"2024-02-12T17:11:21.149666+0000","last_clean":"2024-02-12T17:05:38.389408+0000","last_became_active":"2024-02-12T17:10:03.943360+0000","last_became_peered":"2024-02-12T17:10:03.943360+0000","last_unstale":"2024-02-12T17:11:21.149666+0000","last_undegraded":"2024-02-12T17:11:21.149666+0000","last_fullsized":"2024-02-12T17:10:03.861151+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":236,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.151190+0000","last_change":"2024-02-12T17:07:36.839451+0000","last_active":"2024-02-12T17:11:20.151190+0000","last_peered":"2024-02-12T17:11:20.151190+0000","last_clean":"2024-02-12T17:11:20.151190+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:11:20.151190+0000","last_undegraded":"2024-02-12T17:11:20.151190+0000","last_fullsized":"2024-02-12T17:11:20.151190+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":367,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.150115+0000","last_change":"2024-02-12T17:10:03.880787+0000","last_active":"2024-02-12T17:11:21.150115+0000","last_peered":"2024-02-12T17:11:21.150115+0000","last_clean":"2024-02-12T17:09:19.537752+0000","last_became_active":"2024-02-12T17:10:03.880787+0000","last_became_peered":"2024-02-12T17:10:03.880787+0000","last_unstale":"2024-02-12T17:11:21.150115+0000","last_undegraded":"2024-02-12T17:11:21.150115+0000","last_fullsized":"2024-02-12T17:10:03.861487+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":97,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.143530+0000","last_change":"2024-02-12T17:10:03.875817+0000","last_active":"2024-02-12T17:11:21.143530+0000","last_peered":"2024-02-12T17:11:21.143530+0000","last_clean":"2024-02-12T17:09:58.607780+0000","last_became_active":"2024-02-12T17:10:03.875817+0000","last_became_peered":"2024-02-12T17:10:03.875817+0000","last_unstale":"2024-02-12T17:11:21.143530+0000","last_undegraded":"2024-02-12T17:10:03.859527+0000","last_fullsized":"2024-02-12T17:10:03.859419+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:38.751057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.751057+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"570'12790","reported_seq":13289,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.236669+0000","last_change":"2024-02-12T17:11:21.209989+0000","last_active":"2024-02-12T17:11:21.236669+0000","last_peered":"2024-02-12T17:11:21.236669+0000","last_clean":"2024-02-12T17:11:21.236669+0000","last_became_active":"2024-02-12T17:11:21.153569+0000","last_became_peered":"2024-02-12T17:11:21.153569+0000","last_unstale":"2024-02-12T17:11:21.236669+0000","last_undegraded":"2024-02-12T17:11:21.236669+0000","last_fullsized":"2024-02-12T17:11:21.236669+0000","mapping_epoch":569,"log_start":"494'8091","ondisk_log_start":"494'8091","created":503,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4699,"ondisk_log_size":4699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":247264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":10027008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2,7],"avail_no_missing":["6","2","7"],"object_location_counts":[{"shards":"2,6,7","objects":36}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":608,"reported_epoch":569,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:20.151764+0000","last_change":"2024-02-12T17:10:03.941991+0000","last_active":"2024-02-12T17:11:20.151764+0000","last_peered":"2024-02-12T17:11:20.151764+0000","last_clean":"2024-02-12T17:11:20.151764+0000","last_became_active":"2024-02-12T17:10:03.935291+0000","last_became_peered":"2024-02-12T17:10:03.935291+0000","last_unstale":"2024-02-12T17:11:20.151764+0000","last_undegraded":"2024-02-12T17:11:20.151764+0000","last_fullsized":"2024-02-12T17:11:20.151764+0000","mapping_epoch":533,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":509,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.146604+0000","last_change":"2024-02-12T17:10:03.879412+0000","last_active":"2024-02-12T17:11:21.146604+0000","last_peered":"2024-02-12T17:11:21.146604+0000","last_clean":"2024-02-12T17:09:58.605818+0000","last_became_active":"2024-02-12T17:10:03.879412+0000","last_became_peered":"2024-02-12T17:10:03.879412+0000","last_unstale":"2024-02-12T17:11:21.146604+0000","last_undegraded":"2024-02-12T17:11:21.146604+0000","last_fullsized":"2024-02-12T17:10:03.862546+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":67,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.143948+0000","last_change":"2024-02-12T17:09:21.525331+0000","last_active":"2024-02-12T17:11:21.143948+0000","last_peered":"2024-02-12T17:11:21.143948+0000","last_clean":"2024-02-12T17:11:21.143948+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:11:21.143948+0000","last_undegraded":"2024-02-12T17:11:21.143948+0000","last_fullsized":"2024-02-12T17:11:21.143948+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:40.656513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:02:40.656513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":126,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.144663+0000","last_change":"2024-02-12T17:10:03.869410+0000","last_active":"2024-02-12T17:11:21.144663+0000","last_peered":"2024-02-12T17:11:21.144663+0000","last_clean":"2024-02-12T17:09:49.876861+0000","last_became_active":"2024-02-12T17:10:03.869410+0000","last_became_peered":"2024-02-12T17:10:03.869410+0000","last_unstale":"2024-02-12T17:11:21.144663+0000","last_undegraded":"2024-02-12T17:10:03.856610+0000","last_fullsized":"2024-02-12T17:10:03.856493+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1b","version":"569'12740","reported_seq":14710,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.487792+0000","last_change":"2024-02-12T17:11:06.605771+0000","last_active":"2024-02-12T17:11:20.487792+0000","last_peered":"2024-02-12T17:11:20.487792+0000","last_clean":"2024-02-12T17:11:20.487792+0000","last_became_active":"2024-02-12T17:11:06.511831+0000","last_became_peered":"2024-02-12T17:11:06.511831+0000","last_unstale":"2024-02-12T17:11:20.487792+0000","last_undegraded":"2024-02-12T17:11:20.487792+0000","last_fullsized":"2024-02-12T17:11:20.487792+0000","mapping_epoch":559,"log_start":"496'8370","ondisk_log_start":"496'8370","created":503,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4370,"ondisk_log_size":4370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7674,"num_write_kb":245984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":908,"num_bytes_recovered":59506688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":603,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.159071+0000","last_change":"2024-02-12T17:05:55.414790+0000","last_active":"2024-02-12T17:11:20.159071+0000","last_peered":"2024-02-12T17:11:20.159071+0000","last_clean":"2024-02-12T17:11:20.159071+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:11:20.159071+0000","last_undegraded":"2024-02-12T17:11:20.159071+0000","last_fullsized":"2024-02-12T17:11:20.159071+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":324,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.157065+0000","last_change":"2024-02-12T17:06:51.622488+0000","last_active":"2024-02-12T17:11:20.157065+0000","last_peered":"2024-02-12T17:11:20.157065+0000","last_clean":"2024-02-12T17:11:20.157065+0000","last_became_active":"2024-02-12T17:04:39.886343+0000","last_became_peered":"2024-02-12T17:04:39.886343+0000","last_unstale":"2024-02-12T17:11:20.157065+0000","last_undegraded":"2024-02-12T17:11:20.157065+0000","last_fullsized":"2024-02-12T17:11:20.157065+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.622401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.622401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":656,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.150009+0000","last_change":"2024-02-12T17:07:33.734742+0000","last_active":"2024-02-12T17:11:21.150009+0000","last_peered":"2024-02-12T17:11:21.150009+0000","last_clean":"2024-02-12T17:11:21.150009+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:11:21.150009+0000","last_undegraded":"2024-02-12T17:11:21.150009+0000","last_fullsized":"2024-02-12T17:11:21.150009+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:33.734675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:33.734675+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1a","version":"569'12681","reported_seq":12859,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-02-12T17:11:22.248366+0000","last_change":"2024-02-12T17:11:22.248366+0000","last_active":"2024-02-12T17:11:22.248366+0000","last_peered":"2024-02-12T17:11:22.248366+0000","last_clean":"2024-02-12T17:11:22.248366+0000","last_became_active":"2024-02-12T17:11:22.202377+0000","last_became_peered":"2024-02-12T17:11:22.202377+0000","last_unstale":"2024-02-12T17:11:22.248366+0000","last_undegraded":"2024-02-12T17:11:22.248366+0000","last_fullsized":"2024-02-12T17:11:22.248366+0000","mapping_epoch":570,"log_start":"481'7581","ondisk_log_start":"481'7581","created":503,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5100,"ondisk_log_size":5100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7644,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.15","version":"534'21","reported_seq":586,"reported_epoch":571,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:22.153419+0000","last_change":"2024-02-12T17:10:03.995272+0000","last_active":"2024-02-12T17:11:22.153419+0000","last_peered":"2024-02-12T17:11:22.153419+0000","last_clean":"2024-02-12T17:10:02.526826+0000","last_became_active":"2024-02-12T17:10:03.995272+0000","last_became_peered":"2024-02-12T17:10:03.995272+0000","last_unstale":"2024-02-12T17:11:22.153419+0000","last_undegraded":"2024-02-12T17:10:03.873726+0000","last_fullsized":"2024-02-12T17:10:03.873602+0000","mapping_epoch":533,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":560,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.147404+0000","last_change":"2024-02-12T17:07:35.798822+0000","last_active":"2024-02-12T17:11:21.147404+0000","last_peered":"2024-02-12T17:11:21.147404+0000","last_clean":"2024-02-12T17:11:21.147404+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:11:21.147404+0000","last_undegraded":"2024-02-12T17:11:21.147404+0000","last_fullsized":"2024-02-12T17:11:21.147404+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:35.798720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.798720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":290,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.158351+0000","last_change":"2024-02-12T17:10:03.901247+0000","last_active":"2024-02-12T17:11:20.158351+0000","last_peered":"2024-02-12T17:11:20.158351+0000","last_clean":"2024-02-12T17:09:58.611807+0000","last_became_active":"2024-02-12T17:10:03.901247+0000","last_became_peered":"2024-02-12T17:10:03.901247+0000","last_unstale":"2024-02-12T17:11:20.158351+0000","last_undegraded":"2024-02-12T17:10:03.867068+0000","last_fullsized":"2024-02-12T17:10:03.866968+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"569'11701","reported_seq":28724,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.410715+0000","last_change":"2024-02-12T17:10:03.872945+0000","last_active":"2024-02-12T17:11:20.410715+0000","last_peered":"2024-02-12T17:11:20.410715+0000","last_clean":"2024-02-12T17:10:02.747096+0000","last_became_active":"2024-02-12T17:10:03.872945+0000","last_became_peered":"2024-02-12T17:10:03.872945+0000","last_unstale":"2024-02-12T17:11:20.410715+0000","last_undegraded":"2024-02-12T17:10:03.854455+0000","last_fullsized":"2024-02-12T17:10:03.854326+0000","mapping_epoch":533,"log_start":"496'8147","ondisk_log_start":"496'8147","created":480,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3554,"ondisk_log_size":3554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2129920,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7821,"num_write_kb":251296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6295,"num_bytes_recovered":412549120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0","2","5","7"],"object_location_counts":[{"shards":"0,3","objects":33}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":564,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.166778+0000","last_change":"2024-02-12T17:09:21.479123+0000","last_active":"2024-02-12T17:11:21.166778+0000","last_peered":"2024-02-12T17:11:21.166778+0000","last_clean":"2024-02-12T17:11:21.166778+0000","last_became_active":"2024-02-12T17:09:21.478824+0000","last_became_peered":"2024-02-12T17:09:21.478824+0000","last_unstale":"2024-02-12T17:11:21.166778+0000","last_undegraded":"2024-02-12T17:11:21.166778+0000","last_fullsized":"2024-02-12T17:11:21.166778+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:37.116609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:06:37.116609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":73,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.143467+0000","last_change":"2024-02-12T17:10:03.868003+0000","last_active":"2024-02-12T17:11:21.143467+0000","last_peered":"2024-02-12T17:11:21.143467+0000","last_clean":"2024-02-12T17:09:58.608897+0000","last_became_active":"2024-02-12T17:10:03.868003+0000","last_became_peered":"2024-02-12T17:10:03.868003+0000","last_unstale":"2024-02-12T17:11:21.143467+0000","last_undegraded":"2024-02-12T17:11:21.143467+0000","last_fullsized":"2024-02-12T17:10:03.855249+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:34.780676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:34.780676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":279,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.229364+0000","last_change":"2024-02-12T17:10:03.919074+0000","last_active":"2024-02-12T17:11:20.229364+0000","last_peered":"2024-02-12T17:11:20.229364+0000","last_clean":"2024-02-12T17:09:58.621311+0000","last_became_active":"2024-02-12T17:10:03.919074+0000","last_became_peered":"2024-02-12T17:10:03.919074+0000","last_unstale":"2024-02-12T17:11:20.229364+0000","last_undegraded":"2024-02-12T17:10:03.875183+0000","last_fullsized":"2024-02-12T17:10:03.875072+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"569'11431","reported_seq":17737,"reported_epoch":571,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-12T17:11:24.060418+0000","last_change":"2024-02-12T17:11:17.700639+0000","last_active":"2024-02-12T17:11:24.060418+0000","last_peered":"2024-02-12T17:11:24.060418+0000","last_clean":"2024-02-12T17:07:53.055073+0000","last_became_active":"2024-02-12T17:10:03.870817+0000","last_became_peered":"2024-02-12T17:10:03.870817+0000","last_unstale":"2024-02-12T17:11:24.060418+0000","last_undegraded":"2024-02-12T17:11:24.060418+0000","last_fullsized":"2024-02-12T17:10:03.853902+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":480,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":11431,"ondisk_log_size":11431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":626,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":243680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2342,"num_bytes_recovered":150732800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[0,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2","objects":-626}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":212,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.166903+0000","last_change":"2024-02-12T17:07:45.273035+0000","last_active":"2024-02-12T17:11:21.166903+0000","last_peered":"2024-02-12T17:11:21.166903+0000","last_clean":"2024-02-12T17:11:21.166903+0000","last_became_active":"2024-02-12T17:04:39.936538+0000","last_became_peered":"2024-02-12T17:04:39.936538+0000","last_unstale":"2024-02-12T17:11:21.166903+0000","last_undegraded":"2024-02-12T17:11:21.166903+0000","last_fullsized":"2024-02-12T17:11:21.166903+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.272939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.272939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":432,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.141387+0000","last_change":"2024-02-12T17:10:03.867952+0000","last_active":"2024-02-12T17:11:21.141387+0000","last_peered":"2024-02-12T17:11:21.141387+0000","last_clean":"2024-02-12T17:08:07.412687+0000","last_became_active":"2024-02-12T17:10:03.867952+0000","last_became_peered":"2024-02-12T17:10:03.867952+0000","last_unstale":"2024-02-12T17:11:21.141387+0000","last_undegraded":"2024-02-12T17:11:21.141387+0000","last_fullsized":"2024-02-12T17:10:03.854012+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":255,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.237721+0000","last_change":"2024-02-12T17:06:45.502715+0000","last_active":"2024-02-12T17:11:20.237721+0000","last_peered":"2024-02-12T17:11:20.237721+0000","last_clean":"2024-02-12T17:11:20.237721+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:11:20.237721+0000","last_undegraded":"2024-02-12T17:11:20.237721+0000","last_fullsized":"2024-02-12T17:11:20.237721+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"569'11518","reported_seq":11709,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-02-12T17:11:22.226411+0000","last_change":"2024-02-12T17:11:22.226411+0000","last_active":"2024-02-12T17:11:22.226411+0000","last_peered":"2024-02-12T17:11:22.226411+0000","last_clean":"2024-02-12T17:11:22.226411+0000","last_became_active":"2024-02-12T17:11:22.172004+0000","last_became_peered":"2024-02-12T17:11:22.172004+0000","last_unstale":"2024-02-12T17:11:22.226411+0000","last_undegraded":"2024-02-12T17:11:22.226411+0000","last_fullsized":"2024-02-12T17:11:22.226411+0000","mapping_epoch":570,"log_start":"479'7563","ondisk_log_start":"479'7563","created":451,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3955,"ondisk_log_size":3955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":246784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":278,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-12T17:11:20.157340+0000","last_change":"2024-02-12T17:10:03.893996+0000","last_active":"2024-02-12T17:11:20.157340+0000","last_peered":"2024-02-12T17:11:20.157340+0000","last_clean":"2024-02-12T17:09:58.614534+0000","last_became_active":"2024-02-12T17:10:03.893996+0000","last_became_peered":"2024-02-12T17:10:03.893996+0000","last_unstale":"2024-02-12T17:11:20.157340+0000","last_undegraded":"2024-02-12T17:11:20.157340+0000","last_fullsized":"2024-02-12T17:10:03.859995+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":289,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.169211+0000","last_change":"2024-02-12T17:09:21.773533+0000","last_active":"2024-02-12T17:11:21.169211+0000","last_peered":"2024-02-12T17:11:21.169211+0000","last_clean":"2024-02-12T17:11:21.169211+0000","last_became_active":"2024-02-12T17:09:21.482826+0000","last_became_peered":"2024-02-12T17:09:21.482826+0000","last_unstale":"2024-02-12T17:11:21.169211+0000","last_undegraded":"2024-02-12T17:11:21.169211+0000","last_fullsized":"2024-02-12T17:11:21.169211+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:07:32.199469+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:07:32.199469+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":129,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.151581+0000","last_change":"2024-02-12T17:10:03.865554+0000","last_active":"2024-02-12T17:11:20.151581+0000","last_peered":"2024-02-12T17:11:20.151581+0000","last_clean":"2024-02-12T17:09:48.576268+0000","last_became_active":"2024-02-12T17:10:03.865554+0000","last_became_peered":"2024-02-12T17:10:03.865554+0000","last_unstale":"2024-02-12T17:11:20.151581+0000","last_undegraded":"2024-02-12T17:10:03.855688+0000","last_fullsized":"2024-02-12T17:10:03.855574+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":434,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.146211+0000","last_change":"2024-02-12T17:10:03.891860+0000","last_active":"2024-02-12T17:11:21.146211+0000","last_peered":"2024-02-12T17:11:21.146211+0000","last_clean":"2024-02-12T17:04:37.847517+0000","last_became_active":"2024-02-12T17:10:03.891860+0000","last_became_peered":"2024-02-12T17:10:03.891860+0000","last_unstale":"2024-02-12T17:11:21.146211+0000","last_undegraded":"2024-02-12T17:11:21.146211+0000","last_fullsized":"2024-02-12T17:10:03.873207+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"569'12862","reported_seq":13339,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-02-12T17:11:22.229653+0000","last_change":"2024-02-12T17:11:22.229653+0000","last_active":"2024-02-12T17:11:22.229653+0000","last_peered":"2024-02-12T17:11:22.229653+0000","last_clean":"2024-02-12T17:11:22.229653+0000","last_became_active":"2024-02-12T17:11:22.196198+0000","last_became_peered":"2024-02-12T17:11:22.196198+0000","last_unstale":"2024-02-12T17:11:22.229653+0000","last_undegraded":"2024-02-12T17:11:22.229653+0000","last_fullsized":"2024-02-12T17:11:22.229653+0000","mapping_epoch":570,"log_start":"495'8181","ondisk_log_start":"495'8181","created":451,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4681,"ondisk_log_size":4681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":247264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":9306112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":557,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.140386+0000","last_change":"2024-02-12T17:07:23.791961+0000","last_active":"2024-02-12T17:11:21.140386+0000","last_peered":"2024-02-12T17:11:21.140386+0000","last_clean":"2024-02-12T17:11:21.140386+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:11:21.140386+0000","last_undegraded":"2024-02-12T17:11:21.140386+0000","last_fullsized":"2024-02-12T17:11:21.140386+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:23.791870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:23.791870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":354,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.140903+0000","last_change":"2024-02-12T17:10:03.864945+0000","last_active":"2024-02-12T17:11:21.140903+0000","last_peered":"2024-02-12T17:11:21.140903+0000","last_clean":"2024-02-12T17:08:07.411291+0000","last_became_active":"2024-02-12T17:10:03.864945+0000","last_became_peered":"2024-02-12T17:10:03.864945+0000","last_unstale":"2024-02-12T17:11:21.140903+0000","last_undegraded":"2024-02-12T17:11:21.140903+0000","last_fullsized":"2024-02-12T17:10:03.855363+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":115,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.149764+0000","last_change":"2024-02-12T17:10:03.871713+0000","last_active":"2024-02-12T17:11:21.149764+0000","last_peered":"2024-02-12T17:11:21.149764+0000","last_clean":"2024-02-12T17:09:51.456745+0000","last_became_active":"2024-02-12T17:10:03.871713+0000","last_became_peered":"2024-02-12T17:10:03.871713+0000","last_unstale":"2024-02-12T17:11:21.149764+0000","last_undegraded":"2024-02-12T17:10:03.862325+0000","last_fullsized":"2024-02-12T17:10:03.862210+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:04.040555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:04.040555+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"519'21","reported_seq":821,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-02-12T17:11:22.148719+0000","last_change":"2024-02-12T17:09:25.723429+0000","last_active":"2024-02-12T17:11:22.148719+0000","last_peered":"2024-02-12T17:11:22.148719+0000","last_clean":"2024-02-12T17:11:22.148719+0000","last_became_active":"2024-02-12T17:09:25.723083+0000","last_became_peered":"2024-02-12T17:09:25.723083+0000","last_unstale":"2024-02-12T17:11:22.148719+0000","last_undegraded":"2024-02-12T17:11:22.148719+0000","last_fullsized":"2024-02-12T17:11:22.148719+0000","mapping_epoch":517,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"569'12642","reported_seq":12797,"reported_epoch":570,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:11:21.148270+0000","last_change":"2024-02-12T17:11:04.518418+0000","last_active":"2024-02-12T17:11:21.148270+0000","last_peered":"2024-02-12T17:11:21.148270+0000","last_clean":"2024-02-12T17:09:20.471514+0000","last_became_active":"2024-02-12T17:09:22.553336+0000","last_became_peered":"2024-02-12T17:09:22.553336+0000","last_unstale":"2024-02-12T17:11:21.148270+0000","last_undegraded":"2024-02-12T17:11:21.148270+0000","last_fullsized":"2024-02-12T17:11:21.148270+0000","mapping_epoch":514,"log_start":"478'7472","ondisk_log_start":"478'7472","created":451,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5170,"ondisk_log_size":5170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1277952,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":19}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":267,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.178587+0000","last_change":"2024-02-12T17:09:21.515567+0000","last_active":"2024-02-12T17:11:20.178587+0000","last_peered":"2024-02-12T17:11:20.178587+0000","last_clean":"2024-02-12T17:11:20.178587+0000","last_became_active":"2024-02-12T17:09:21.514947+0000","last_became_peered":"2024-02-12T17:09:21.514947+0000","last_unstale":"2024-02-12T17:11:20.178587+0000","last_undegraded":"2024-02-12T17:11:20.178587+0000","last_fullsized":"2024-02-12T17:11:20.178587+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":598,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.147508+0000","last_change":"2024-02-12T17:10:03.879478+0000","last_active":"2024-02-12T17:11:21.147508+0000","last_peered":"2024-02-12T17:11:21.147508+0000","last_clean":"2024-02-12T17:09:58.604742+0000","last_became_active":"2024-02-12T17:10:03.879478+0000","last_became_peered":"2024-02-12T17:10:03.879478+0000","last_unstale":"2024-02-12T17:11:21.147508+0000","last_undegraded":"2024-02-12T17:11:21.147508+0000","last_fullsized":"2024-02-12T17:10:03.861800+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"534'69","reported_seq":163,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.159158+0000","last_change":"2024-02-12T17:10:03.934831+0000","last_active":"2024-02-12T17:11:20.159158+0000","last_peered":"2024-02-12T17:11:20.159158+0000","last_clean":"2024-02-12T17:09:51.482838+0000","last_became_active":"2024-02-12T17:10:03.934831+0000","last_became_peered":"2024-02-12T17:10:03.934831+0000","last_unstale":"2024-02-12T17:11:20.159158+0000","last_undegraded":"2024-02-12T17:10:03.873010+0000","last_fullsized":"2024-02-12T17:10:03.872891+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":442,"reported_epoch":570,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:11:21.145712+0000","last_change":"2024-02-12T17:11:00.875666+0000","last_active":"2024-02-12T17:11:21.145712+0000","last_peered":"2024-02-12T17:11:21.145712+0000","last_clean":"2024-02-12T17:05:12.349531+0000","last_became_active":"2024-02-12T17:10:03.997213+0000","last_became_peered":"2024-02-12T17:10:03.997213+0000","last_unstale":"2024-02-12T17:11:21.145712+0000","last_undegraded":"2024-02-12T17:11:21.145712+0000","last_fullsized":"2024-02-12T17:11:21.145712+0000","mapping_epoch":533,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[2,7,3],"avail_no_missing":["2","3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"570'11754","reported_seq":11892,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.260849+0000","last_change":"2024-02-12T17:07:52.060549+0000","last_active":"2024-02-12T17:11:21.260849+0000","last_peered":"2024-02-12T17:11:21.260849+0000","last_clean":"2024-02-12T17:11:21.260849+0000","last_became_active":"2024-02-12T17:07:52.058737+0000","last_became_peered":"2024-02-12T17:07:52.058737+0000","last_unstale":"2024-02-12T17:11:21.260849+0000","last_undegraded":"2024-02-12T17:11:21.260849+0000","last_fullsized":"2024-02-12T17:11:21.260849+0000","mapping_epoch":451,"log_start":"490'7879","ondisk_log_start":"490'7879","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3875,"ondisk_log_size":3875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7873,"num_write_kb":250912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":571,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.167072+0000","last_change":"2024-02-12T17:09:21.502830+0000","last_active":"2024-02-12T17:11:21.167072+0000","last_peered":"2024-02-12T17:11:21.167072+0000","last_clean":"2024-02-12T17:11:21.167072+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:11:21.167072+0000","last_undegraded":"2024-02-12T17:11:21.167072+0000","last_fullsized":"2024-02-12T17:11:21.167072+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.001888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.001888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":230,"reported_epoch":571,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:22.148341+0000","last_change":"2024-02-12T17:10:03.874040+0000","last_active":"2024-02-12T17:11:22.148341+0000","last_peered":"2024-02-12T17:11:22.148341+0000","last_clean":"2024-02-12T17:09:20.333789+0000","last_became_active":"2024-02-12T17:10:03.874040+0000","last_became_peered":"2024-02-12T17:10:03.874040+0000","last_unstale":"2024-02-12T17:11:22.148341+0000","last_undegraded":"2024-02-12T17:10:03.860815+0000","last_fullsized":"2024-02-12T17:10:03.856356+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":562,"num_read_kb":557,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":236,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.151417+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:11:20.151417+0000","last_peered":"2024-02-12T17:11:20.151417+0000","last_clean":"2024-02-12T17:11:20.151417+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:11:20.151417+0000","last_undegraded":"2024-02-12T17:11:20.151417+0000","last_fullsized":"2024-02-12T17:11:20.151417+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":106,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.151054+0000","last_change":"2024-02-12T17:10:03.886514+0000","last_active":"2024-02-12T17:11:21.151054+0000","last_peered":"2024-02-12T17:11:21.151054+0000","last_clean":"2024-02-12T17:09:50.311156+0000","last_became_active":"2024-02-12T17:10:03.886514+0000","last_became_peered":"2024-02-12T17:10:03.886514+0000","last_unstale":"2024-02-12T17:11:21.151054+0000","last_undegraded":"2024-02-12T17:10:03.872376+0000","last_fullsized":"2024-02-12T17:10:03.872265+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":490,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.147868+0000","last_change":"2024-02-12T17:06:25.129256+0000","last_active":"2024-02-12T17:11:21.147868+0000","last_peered":"2024-02-12T17:11:21.147868+0000","last_clean":"2024-02-12T17:11:21.147868+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:11:21.147868+0000","last_undegraded":"2024-02-12T17:11:21.147868+0000","last_fullsized":"2024-02-12T17:11:21.147868+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:25.129190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:25.129190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"569'12766","reported_seq":13215,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.143114+0000","last_change":"2024-02-12T17:11:15.094934+0000","last_active":"2024-02-12T17:11:21.143114+0000","last_peered":"2024-02-12T17:11:21.143114+0000","last_clean":"2024-02-12T17:11:21.143114+0000","last_became_active":"2024-02-12T17:11:15.052791+0000","last_became_peered":"2024-02-12T17:11:15.052791+0000","last_unstale":"2024-02-12T17:11:21.143114+0000","last_undegraded":"2024-02-12T17:11:21.143114+0000","last_fullsized":"2024-02-12T17:11:21.143114+0000","mapping_epoch":564,"log_start":"481'7617","ondisk_log_start":"481'7617","created":451,"last_epoch_clean":565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5149,"ondisk_log_size":5149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":245984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":8781824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7,2],"avail_no_missing":["6","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":605,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.151407+0000","last_change":"2024-02-12T17:10:03.943122+0000","last_active":"2024-02-12T17:11:21.151407+0000","last_peered":"2024-02-12T17:11:21.151407+0000","last_clean":"2024-02-12T17:09:58.606717+0000","last_became_active":"2024-02-12T17:10:03.943122+0000","last_became_peered":"2024-02-12T17:10:03.943122+0000","last_unstale":"2024-02-12T17:11:21.151407+0000","last_undegraded":"2024-02-12T17:11:21.151407+0000","last_fullsized":"2024-02-12T17:10:03.872803+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.058086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.058086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":263,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.153781+0000","last_change":"2024-02-12T17:09:21.602904+0000","last_active":"2024-02-12T17:11:20.153781+0000","last_peered":"2024-02-12T17:11:20.153781+0000","last_clean":"2024-02-12T17:11:20.153781+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:11:20.153781+0000","last_undegraded":"2024-02-12T17:11:20.153781+0000","last_fullsized":"2024-02-12T17:11:20.153781+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:54.311805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:05:54.311805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":542,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.146904+0000","last_change":"2024-02-12T17:05:56.054036+0000","last_active":"2024-02-12T17:11:21.146904+0000","last_peered":"2024-02-12T17:11:21.146904+0000","last_clean":"2024-02-12T17:11:21.146904+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:11:21.146904+0000","last_undegraded":"2024-02-12T17:11:21.146904+0000","last_fullsized":"2024-02-12T17:11:21.146904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:56.053971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.053971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":598,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.154931+0000","last_change":"2024-02-12T17:10:03.877565+0000","last_active":"2024-02-12T17:11:21.154931+0000","last_peered":"2024-02-12T17:11:21.154931+0000","last_clean":"2024-02-12T17:09:58.611920+0000","last_became_active":"2024-02-12T17:10:03.877565+0000","last_became_peered":"2024-02-12T17:10:03.877565+0000","last_unstale":"2024-02-12T17:11:21.154931+0000","last_undegraded":"2024-02-12T17:11:21.154931+0000","last_fullsized":"2024-02-12T17:10:03.854437+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":101,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.144012+0000","last_change":"2024-02-12T17:09:21.709538+0000","last_active":"2024-02-12T17:11:21.144012+0000","last_peered":"2024-02-12T17:11:21.144012+0000","last_clean":"2024-02-12T17:11:21.144012+0000","last_became_active":"2024-02-12T17:09:21.585762+0000","last_became_peered":"2024-02-12T17:09:21.585762+0000","last_unstale":"2024-02-12T17:11:21.144012+0000","last_undegraded":"2024-02-12T17:11:21.144012+0000","last_fullsized":"2024-02-12T17:11:21.144012+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'33","last_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.886342+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":571,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.632080+0000","last_change":"2024-02-12T17:11:21.632080+0000","last_active":"2024-02-12T17:11:21.632080+0000","last_peered":"2024-02-12T17:11:21.632080+0000","last_clean":"2024-02-12T17:11:21.632080+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:11:21.632080+0000","last_undegraded":"2024-02-12T17:11:21.632080+0000","last_fullsized":"2024-02-12T17:11:21.632080+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"569'11338","reported_seq":11478,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.140486+0000","last_change":"2024-02-12T17:07:52.089095+0000","last_active":"2024-02-12T17:11:21.140486+0000","last_peered":"2024-02-12T17:11:21.140486+0000","last_clean":"2024-02-12T17:11:21.140486+0000","last_became_active":"2024-02-12T17:07:52.081482+0000","last_became_peered":"2024-02-12T17:07:52.081482+0000","last_unstale":"2024-02-12T17:11:21.140486+0000","last_undegraded":"2024-02-12T17:11:21.140486+0000","last_fullsized":"2024-02-12T17:11:21.140486+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":11338,"ondisk_log_size":11338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3309568,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7572,"num_write_kb":243936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":464,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.145911+0000","last_change":"2024-02-12T17:10:03.942690+0000","last_active":"2024-02-12T17:11:21.145911+0000","last_peered":"2024-02-12T17:11:21.145911+0000","last_clean":"2024-02-12T17:04:37.845034+0000","last_became_active":"2024-02-12T17:10:03.942690+0000","last_became_peered":"2024-02-12T17:10:03.942690+0000","last_unstale":"2024-02-12T17:11:21.145911+0000","last_undegraded":"2024-02-12T17:11:21.145911+0000","last_fullsized":"2024-02-12T17:10:03.872324+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":324,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.150916+0000","last_change":"2024-02-12T17:06:46.537733+0000","last_active":"2024-02-12T17:11:20.150916+0000","last_peered":"2024-02-12T17:11:20.150916+0000","last_clean":"2024-02-12T17:11:20.150916+0000","last_became_active":"2024-02-12T17:04:39.892122+0000","last_became_peered":"2024-02-12T17:04:39.892122+0000","last_unstale":"2024-02-12T17:11:20.150916+0000","last_undegraded":"2024-02-12T17:11:20.150916+0000","last_fullsized":"2024-02-12T17:11:20.150916+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.537630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.537630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":591,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.154717+0000","last_change":"2024-02-12T17:10:03.878327+0000","last_active":"2024-02-12T17:11:21.154717+0000","last_peered":"2024-02-12T17:11:21.154717+0000","last_clean":"2024-02-12T17:09:58.611363+0000","last_became_active":"2024-02-12T17:10:03.878327+0000","last_became_peered":"2024-02-12T17:10:03.878327+0000","last_unstale":"2024-02-12T17:11:21.154717+0000","last_undegraded":"2024-02-12T17:10:03.854352+0000","last_fullsized":"2024-02-12T17:10:03.854244+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":120,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.167661+0000","last_change":"2024-02-12T17:10:03.875711+0000","last_active":"2024-02-12T17:11:21.167661+0000","last_peered":"2024-02-12T17:11:21.167661+0000","last_clean":"2024-02-12T17:09:50.556595+0000","last_became_active":"2024-02-12T17:10:03.875711+0000","last_became_peered":"2024-02-12T17:10:03.875711+0000","last_unstale":"2024-02-12T17:11:21.167661+0000","last_undegraded":"2024-02-12T17:10:03.857938+0000","last_fullsized":"2024-02-12T17:10:03.857823+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":584,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.151280+0000","last_change":"2024-02-12T17:10:03.873160+0000","last_active":"2024-02-12T17:11:21.151280+0000","last_peered":"2024-02-12T17:11:21.151280+0000","last_clean":"2024-02-12T17:09:58.606674+0000","last_became_active":"2024-02-12T17:10:03.873160+0000","last_became_peered":"2024-02-12T17:10:03.873160+0000","last_unstale":"2024-02-12T17:11:21.151280+0000","last_undegraded":"2024-02-12T17:11:21.151280+0000","last_fullsized":"2024-02-12T17:10:03.861159+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"570'12870","reported_seq":16720,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.260916+0000","last_change":"2024-02-12T17:09:53.553034+0000","last_active":"2024-02-12T17:11:21.260916+0000","last_peered":"2024-02-12T17:11:21.260916+0000","last_clean":"2024-02-12T17:11:21.260916+0000","last_became_active":"2024-02-12T17:09:53.526468+0000","last_became_peered":"2024-02-12T17:09:53.526468+0000","last_unstale":"2024-02-12T17:11:21.260916+0000","last_undegraded":"2024-02-12T17:11:21.260916+0000","last_fullsized":"2024-02-12T17:11:21.260916+0000","mapping_epoch":527,"log_start":"477'7119","ondisk_log_start":"477'7119","created":451,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5751,"ondisk_log_size":5751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2195468,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7759,"num_write_kb":249344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1848,"num_bytes_recovered":121110528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":78,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.143170+0000","last_change":"2024-02-12T17:10:03.868467+0000","last_active":"2024-02-12T17:11:21.143170+0000","last_peered":"2024-02-12T17:11:21.143170+0000","last_clean":"2024-02-12T17:09:19.536649+0000","last_became_active":"2024-02-12T17:10:03.868467+0000","last_became_peered":"2024-02-12T17:10:03.868467+0000","last_unstale":"2024-02-12T17:11:21.143170+0000","last_undegraded":"2024-02-12T17:11:21.143170+0000","last_fullsized":"2024-02-12T17:10:03.854942+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":586,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.167239+0000","last_change":"2024-02-12T17:10:03.886899+0000","last_active":"2024-02-12T17:11:21.167239+0000","last_peered":"2024-02-12T17:11:21.167239+0000","last_clean":"2024-02-12T17:09:58.612998+0000","last_became_active":"2024-02-12T17:10:03.886899+0000","last_became_peered":"2024-02-12T17:10:03.886899+0000","last_unstale":"2024-02-12T17:11:21.167239+0000","last_undegraded":"2024-02-12T17:11:21.167239+0000","last_fullsized":"2024-02-12T17:10:03.857864+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.961097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.961097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":602,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.147019+0000","last_change":"2024-02-12T17:10:03.886500+0000","last_active":"2024-02-12T17:11:21.147019+0000","last_peered":"2024-02-12T17:11:21.147019+0000","last_clean":"2024-02-12T17:09:58.605538+0000","last_became_active":"2024-02-12T17:10:03.886500+0000","last_became_peered":"2024-02-12T17:10:03.886500+0000","last_unstale":"2024-02-12T17:11:21.147019+0000","last_undegraded":"2024-02-12T17:10:03.861901+0000","last_fullsized":"2024-02-12T17:10:03.861792+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":67,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.144880+0000","last_change":"2024-02-12T17:09:21.560337+0000","last_active":"2024-02-12T17:11:21.144880+0000","last_peered":"2024-02-12T17:11:21.144880+0000","last_clean":"2024-02-12T17:11:21.144880+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:11:21.144880+0000","last_undegraded":"2024-02-12T17:11:21.144880+0000","last_fullsized":"2024-02-12T17:11:21.144880+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:46.526746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:46.526746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.e","version":"569'12669","reported_seq":16870,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.145246+0000","last_change":"2024-02-12T17:09:40.334809+0000","last_active":"2024-02-12T17:11:21.145246+0000","last_peered":"2024-02-12T17:11:21.145246+0000","last_clean":"2024-02-12T17:11:21.145246+0000","last_became_active":"2024-02-12T17:09:40.312504+0000","last_became_peered":"2024-02-12T17:09:40.312504+0000","last_unstale":"2024-02-12T17:11:21.145246+0000","last_undegraded":"2024-02-12T17:11:21.145246+0000","last_fullsized":"2024-02-12T17:11:21.145246+0000","mapping_epoch":523,"log_start":"476'6920","ondisk_log_start":"476'6920","created":451,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5749,"ondisk_log_size":5749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1277952,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7599,"num_write_kb":243808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2022,"num_bytes_recovered":132513792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":254,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.153867+0000","last_change":"2024-02-12T17:09:21.602676+0000","last_active":"2024-02-12T17:11:20.153867+0000","last_peered":"2024-02-12T17:11:20.153867+0000","last_clean":"2024-02-12T17:11:20.153867+0000","last_became_active":"2024-02-12T17:09:21.601469+0000","last_became_peered":"2024-02-12T17:09:21.601469+0000","last_unstale":"2024-02-12T17:11:20.153867+0000","last_undegraded":"2024-02-12T17:11:20.153867+0000","last_fullsized":"2024-02-12T17:11:20.153867+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:01.455969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.455969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":78,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.145080+0000","last_change":"2024-02-12T17:10:03.867829+0000","last_active":"2024-02-12T17:11:21.145080+0000","last_peered":"2024-02-12T17:11:21.145080+0000","last_clean":"2024-02-12T17:09:19.537467+0000","last_became_active":"2024-02-12T17:10:03.867829+0000","last_became_peered":"2024-02-12T17:10:03.867829+0000","last_unstale":"2024-02-12T17:11:21.145080+0000","last_undegraded":"2024-02-12T17:11:21.145080+0000","last_fullsized":"2024-02-12T17:10:03.860640+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:18.084477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.084477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":300,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.146288+0000","last_change":"2024-02-12T17:10:03.894531+0000","last_active":"2024-02-12T17:11:20.146288+0000","last_peered":"2024-02-12T17:11:20.146288+0000","last_clean":"2024-02-12T17:09:58.609394+0000","last_became_active":"2024-02-12T17:10:03.894531+0000","last_became_peered":"2024-02-12T17:10:03.894531+0000","last_unstale":"2024-02-12T17:11:20.146288+0000","last_undegraded":"2024-02-12T17:10:03.855132+0000","last_fullsized":"2024-02-12T17:10:03.855063+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":569,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.151261+0000","last_change":"2024-02-12T17:05:40.428597+0000","last_active":"2024-02-12T17:11:20.151261+0000","last_peered":"2024-02-12T17:11:20.151261+0000","last_clean":"2024-02-12T17:11:20.151261+0000","last_became_active":"2024-02-12T17:05:40.427915+0000","last_became_peered":"2024-02-12T17:05:40.427915+0000","last_unstale":"2024-02-12T17:11:20.151261+0000","last_undegraded":"2024-02-12T17:11:20.151261+0000","last_fullsized":"2024-02-12T17:11:20.151261+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"569'12746","reported_seq":14275,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.143682+0000","last_change":"2024-02-12T17:11:19.141506+0000","last_active":"2024-02-12T17:11:21.143682+0000","last_peered":"2024-02-12T17:11:21.143682+0000","last_clean":"2024-02-12T17:11:21.143682+0000","last_became_active":"2024-02-12T17:11:19.107579+0000","last_became_peered":"2024-02-12T17:11:19.107579+0000","last_unstale":"2024-02-12T17:11:21.143682+0000","last_undegraded":"2024-02-12T17:11:21.143682+0000","last_fullsized":"2024-02-12T17:11:21.143682+0000","mapping_epoch":567,"log_start":"495'8212","ondisk_log_start":"495'8212","created":451,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":4534,"ondisk_log_size":4534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":245664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":44236800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":604,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.155927+0000","last_change":"2024-02-12T17:09:22.252402+0000","last_active":"2024-02-12T17:11:21.155927+0000","last_peered":"2024-02-12T17:11:21.155927+0000","last_clean":"2024-02-12T17:11:21.155927+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:11:21.155927+0000","last_undegraded":"2024-02-12T17:11:21.155927+0000","last_fullsized":"2024-02-12T17:11:21.155927+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:07:52.341731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.341731+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.143765+0000","last_change":"2024-02-12T17:09:21.594780+0000","last_active":"2024-02-12T17:11:21.143765+0000","last_peered":"2024-02-12T17:11:21.143765+0000","last_clean":"2024-02-12T17:11:21.143765+0000","last_became_active":"2024-02-12T17:09:21.539652+0000","last_became_peered":"2024-02-12T17:09:21.539652+0000","last_unstale":"2024-02-12T17:11:21.143765+0000","last_undegraded":"2024-02-12T17:11:21.143765+0000","last_fullsized":"2024-02-12T17:11:21.143765+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:42.272807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:42.272807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":675,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.146792+0000","last_change":"2024-02-12T17:09:22.165941+0000","last_active":"2024-02-12T17:11:21.146792+0000","last_peered":"2024-02-12T17:11:21.146792+0000","last_clean":"2024-02-12T17:11:21.146792+0000","last_became_active":"2024-02-12T17:09:21.531831+0000","last_became_peered":"2024-02-12T17:09:21.531831+0000","last_unstale":"2024-02-12T17:11:21.146792+0000","last_undegraded":"2024-02-12T17:11:21.146792+0000","last_fullsized":"2024-02-12T17:11:21.146792+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'28","last_scrub_stamp":"2024-02-12T17:06:28.073131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:28.073131+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"534'23","reported_seq":403,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-12T17:11:20.146271+0000","last_change":"2024-02-12T17:10:03.891067+0000","last_active":"2024-02-12T17:11:20.146271+0000","last_peered":"2024-02-12T17:11:20.146271+0000","last_clean":"2024-02-12T17:10:02.532802+0000","last_became_active":"2024-02-12T17:10:03.891067+0000","last_became_peered":"2024-02-12T17:10:03.891067+0000","last_unstale":"2024-02-12T17:11:20.146271+0000","last_undegraded":"2024-02-12T17:11:20.146271+0000","last_fullsized":"2024-02-12T17:10:03.854819+0000","mapping_epoch":533,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"569'10039","reported_seq":10590,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.142896+0000","last_change":"2024-02-12T17:11:01.633914+0000","last_active":"2024-02-12T17:11:21.142896+0000","last_peered":"2024-02-12T17:11:21.142896+0000","last_clean":"2024-02-12T17:11:21.142896+0000","last_became_active":"2024-02-12T17:11:01.607413+0000","last_became_peered":"2024-02-12T17:11:01.607413+0000","last_unstale":"2024-02-12T17:11:21.142896+0000","last_undegraded":"2024-02-12T17:11:21.142896+0000","last_fullsized":"2024-02-12T17:11:21.142896+0000","mapping_epoch":555,"log_start":"466'4200","ondisk_log_start":"466'4200","created":451,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5839,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3805,"num_write_kb":120992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7,2],"avail_no_missing":["6","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":279,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.157959+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:11:20.157959+0000","last_peered":"2024-02-12T17:11:20.157959+0000","last_clean":"2024-02-12T17:11:20.157959+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:11:20.157959+0000","last_undegraded":"2024-02-12T17:11:20.157959+0000","last_fullsized":"2024-02-12T17:11:20.157959+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"571'25","reported_seq":862,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-02-12T17:11:22.155928+0000","last_change":"2024-02-12T17:11:21.154509+0000","last_active":"2024-02-12T17:11:22.155928+0000","last_peered":"2024-02-12T17:11:22.155928+0000","last_clean":"2024-02-12T17:11:22.155928+0000","last_became_active":"2024-02-12T17:11:21.154083+0000","last_became_peered":"2024-02-12T17:11:21.154083+0000","last_unstale":"2024-02-12T17:11:22.155928+0000","last_undegraded":"2024-02-12T17:11:22.155928+0000","last_fullsized":"2024-02-12T17:11:22.155928+0000","mapping_epoch":569,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'21","last_scrub_stamp":"2024-02-12T17:06:32.977637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:06:32.977637+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":222,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.154431+0000","last_change":"2024-02-12T17:10:03.877238+0000","last_active":"2024-02-12T17:11:21.154431+0000","last_peered":"2024-02-12T17:11:21.154431+0000","last_clean":"2024-02-12T17:09:58.611234+0000","last_became_active":"2024-02-12T17:10:03.877238+0000","last_became_peered":"2024-02-12T17:10:03.877238+0000","last_unstale":"2024-02-12T17:11:21.154431+0000","last_undegraded":"2024-02-12T17:11:21.154431+0000","last_fullsized":"2024-02-12T17:10:03.853847+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:24.976048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:06:24.976048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":97,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.144966+0000","last_change":"2024-02-12T17:10:03.868111+0000","last_active":"2024-02-12T17:11:21.144966+0000","last_peered":"2024-02-12T17:11:21.144966+0000","last_clean":"2024-02-12T17:09:58.607923+0000","last_became_active":"2024-02-12T17:10:03.868111+0000","last_became_peered":"2024-02-12T17:10:03.868111+0000","last_unstale":"2024-02-12T17:11:21.144966+0000","last_undegraded":"2024-02-12T17:10:03.860743+0000","last_fullsized":"2024-02-12T17:10:03.860640+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:05:55.344999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.344999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"569'10358","reported_seq":10593,"reported_epoch":570,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:21.155132+0000","last_change":"2024-02-12T17:10:03.887338+0000","last_active":"2024-02-12T17:11:21.155132+0000","last_peered":"2024-02-12T17:11:21.155132+0000","last_clean":"2024-02-12T17:11:21.155132+0000","last_became_active":"2024-02-12T17:10:03.876797+0000","last_became_peered":"2024-02-12T17:10:03.876797+0000","last_unstale":"2024-02-12T17:11:21.155132+0000","last_undegraded":"2024-02-12T17:11:21.155132+0000","last_fullsized":"2024-02-12T17:11:21.155132+0000","mapping_epoch":533,"log_start":"466'4500","ondisk_log_start":"466'4500","created":451,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5858,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":125264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,5],"avail_no_missing":["7","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":596,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.150320+0000","last_change":"2024-02-12T17:10:03.874296+0000","last_active":"2024-02-12T17:11:21.150320+0000","last_peered":"2024-02-12T17:11:21.150320+0000","last_clean":"2024-02-12T17:09:58.605834+0000","last_became_active":"2024-02-12T17:10:03.874296+0000","last_became_peered":"2024-02-12T17:10:03.874296+0000","last_unstale":"2024-02-12T17:11:21.150320+0000","last_undegraded":"2024-02-12T17:11:21.150320+0000","last_fullsized":"2024-02-12T17:10:03.864117+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"435'21","reported_seq":880,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-02-12T17:11:22.155701+0000","last_change":"2024-02-12T17:07:41.785111+0000","last_active":"2024-02-12T17:11:22.155701+0000","last_peered":"2024-02-12T17:11:22.155701+0000","last_clean":"2024-02-12T17:11:22.155701+0000","last_became_active":"2024-02-12T17:05:43.540504+0000","last_became_peered":"2024-02-12T17:05:43.540504+0000","last_unstale":"2024-02-12T17:11:22.155701+0000","last_undegraded":"2024-02-12T17:11:22.155701+0000","last_fullsized":"2024-02-12T17:11:22.155701+0000","mapping_epoch":434,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'21","last_scrub_stamp":"2024-02-12T17:07:41.785018+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.785018+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":171,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.155312+0000","last_change":"2024-02-12T17:10:03.879456+0000","last_active":"2024-02-12T17:11:21.155312+0000","last_peered":"2024-02-12T17:11:21.155312+0000","last_clean":"2024-02-12T16:59:07.210188+0000","last_became_active":"2024-02-12T17:10:03.879456+0000","last_became_peered":"2024-02-12T17:10:03.879456+0000","last_unstale":"2024-02-12T17:11:21.155312+0000","last_undegraded":"2024-02-12T17:11:21.155312+0000","last_fullsized":"2024-02-12T17:10:03.854791+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":121,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.229125+0000","last_change":"2024-02-12T17:10:03.918869+0000","last_active":"2024-02-12T17:11:20.229125+0000","last_peered":"2024-02-12T17:11:20.229125+0000","last_clean":"2024-02-12T17:09:51.940971+0000","last_became_active":"2024-02-12T17:10:03.918869+0000","last_became_peered":"2024-02-12T17:10:03.918869+0000","last_unstale":"2024-02-12T17:11:20.229125+0000","last_undegraded":"2024-02-12T17:10:03.874864+0000","last_fullsized":"2024-02-12T17:10:03.874752+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"569'10130","reported_seq":10563,"reported_epoch":569,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:20.436370+0000","last_change":"2024-02-12T17:11:14.104173+0000","last_active":"2024-02-12T17:11:20.436370+0000","last_peered":"2024-02-12T17:11:20.436370+0000","last_clean":"2024-02-12T17:11:20.436370+0000","last_became_active":"2024-02-12T17:11:14.063805+0000","last_became_peered":"2024-02-12T17:11:14.063805+0000","last_unstale":"2024-02-12T17:11:20.436370+0000","last_undegraded":"2024-02-12T17:11:20.436370+0000","last_fullsized":"2024-02-12T17:11:20.436370+0000","mapping_epoch":563,"log_start":"466'4100","ondisk_log_start":"466'4100","created":451,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":6030,"ondisk_log_size":6030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3857,"num_write_kb":122096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":277,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.147693+0000","last_change":"2024-02-12T17:10:03.877621+0000","last_active":"2024-02-12T17:11:21.147693+0000","last_peered":"2024-02-12T17:11:21.147693+0000","last_clean":"2024-02-12T17:04:37.844929+0000","last_became_active":"2024-02-12T17:10:03.877621+0000","last_became_peered":"2024-02-12T17:10:03.877621+0000","last_unstale":"2024-02-12T17:11:21.147693+0000","last_undegraded":"2024-02-12T17:11:21.147693+0000","last_fullsized":"2024-02-12T17:10:03.867715+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.077971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":444,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-02-12T17:11:22.149677+0000","last_change":"2024-02-12T17:07:41.239456+0000","last_active":"2024-02-12T17:11:22.149677+0000","last_peered":"2024-02-12T17:11:22.149677+0000","last_clean":"2024-02-12T17:11:22.149677+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:11:22.149677+0000","last_undegraded":"2024-02-12T17:11:22.149677+0000","last_fullsized":"2024-02-12T17:11:22.149677+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:07:41.239350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.239350+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-12T17:11:20.154127+0000","last_change":"2024-02-12T17:10:03.893191+0000","last_active":"2024-02-12T17:11:20.154127+0000","last_peered":"2024-02-12T17:11:20.154127+0000","last_clean":"2024-02-12T17:08:07.415377+0000","last_became_active":"2024-02-12T17:10:03.893191+0000","last_became_peered":"2024-02-12T17:10:03.893191+0000","last_unstale":"2024-02-12T17:11:20.154127+0000","last_undegraded":"2024-02-12T17:11:20.154127+0000","last_fullsized":"2024-02-12T17:10:03.856411+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":327,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.158064+0000","last_change":"2024-02-12T17:10:03.890980+0000","last_active":"2024-02-12T17:11:20.158064+0000","last_peered":"2024-02-12T17:11:20.158064+0000","last_clean":"2024-02-12T17:09:58.611650+0000","last_became_active":"2024-02-12T17:10:03.890980+0000","last_became_peered":"2024-02-12T17:10:03.890980+0000","last_unstale":"2024-02-12T17:11:20.158064+0000","last_undegraded":"2024-02-12T17:10:03.866190+0000","last_fullsized":"2024-02-12T17:10:03.866089+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:02:16.750400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:16.750400+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"569'10288","reported_seq":10437,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.146800+0000","last_change":"2024-02-12T17:09:01.540513+0000","last_active":"2024-02-12T17:11:21.146800+0000","last_peered":"2024-02-12T17:11:21.146800+0000","last_clean":"2024-02-12T17:11:21.146800+0000","last_became_active":"2024-02-12T17:09:01.473480+0000","last_became_peered":"2024-02-12T17:09:01.473480+0000","last_unstale":"2024-02-12T17:11:21.146800+0000","last_undegraded":"2024-02-12T17:11:21.146800+0000","last_fullsized":"2024-02-12T17:11:21.146800+0000","mapping_epoch":451,"log_start":"466'4400","ondisk_log_start":"466'4400","created":451,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":5888,"ondisk_log_size":5888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":124096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":281,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.229267+0000","last_change":"2024-02-12T17:09:21.618330+0000","last_active":"2024-02-12T17:11:20.229267+0000","last_peered":"2024-02-12T17:11:20.229267+0000","last_clean":"2024-02-12T17:11:20.229267+0000","last_became_active":"2024-02-12T17:09:21.617713+0000","last_became_peered":"2024-02-12T17:09:21.617713+0000","last_unstale":"2024-02-12T17:11:20.229267+0000","last_undegraded":"2024-02-12T17:11:20.229267+0000","last_fullsized":"2024-02-12T17:11:20.229267+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":123,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.140770+0000","last_change":"2024-02-12T17:10:03.867805+0000","last_active":"2024-02-12T17:11:21.140770+0000","last_peered":"2024-02-12T17:11:21.140770+0000","last_clean":"2024-02-12T17:09:19.538702+0000","last_became_active":"2024-02-12T17:10:03.867805+0000","last_became_peered":"2024-02-12T17:10:03.867805+0000","last_unstale":"2024-02-12T17:11:21.140770+0000","last_undegraded":"2024-02-12T17:11:21.140770+0000","last_fullsized":"2024-02-12T17:10:03.855027+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.567340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.567340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":553,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.155256+0000","last_change":"2024-02-12T17:06:51.877944+0000","last_active":"2024-02-12T17:11:21.155256+0000","last_peered":"2024-02-12T17:11:21.155256+0000","last_clean":"2024-02-12T17:11:21.155256+0000","last_became_active":"2024-02-12T17:04:39.888466+0000","last_became_peered":"2024-02-12T17:04:39.888466+0000","last_unstale":"2024-02-12T17:11:21.155256+0000","last_undegraded":"2024-02-12T17:11:21.155256+0000","last_fullsized":"2024-02-12T17:11:21.155256+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.877830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.877830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":154,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.143573+0000","last_change":"2024-02-12T17:10:03.868805+0000","last_active":"2024-02-12T17:11:21.143573+0000","last_peered":"2024-02-12T17:11:21.143573+0000","last_clean":"2024-02-12T17:09:51.524928+0000","last_became_active":"2024-02-12T17:10:03.868805+0000","last_became_peered":"2024-02-12T17:10:03.868805+0000","last_unstale":"2024-02-12T17:11:21.143573+0000","last_undegraded":"2024-02-12T17:10:03.856452+0000","last_fullsized":"2024-02-12T17:10:03.856333+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:07:37.774472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.774472+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"569'11466","reported_seq":12482,"reported_epoch":569,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:11:20.435032+0000","last_change":"2024-02-12T17:10:03.932156+0000","last_active":"2024-02-12T17:11:20.435032+0000","last_peered":"2024-02-12T17:11:20.435032+0000","last_clean":"2024-02-12T17:11:20.435032+0000","last_became_active":"2024-02-12T17:10:03.901208+0000","last_became_peered":"2024-02-12T17:10:03.901208+0000","last_unstale":"2024-02-12T17:11:20.435032+0000","last_undegraded":"2024-02-12T17:11:20.435032+0000","last_fullsized":"2024-02-12T17:11:20.435032+0000","mapping_epoch":533,"log_start":"495'7926","ondisk_log_start":"495'7926","created":451,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3540,"ondisk_log_size":3540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":245536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":589,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.154562+0000","last_change":"2024-02-12T17:05:57.890025+0000","last_active":"2024-02-12T17:11:21.154562+0000","last_peered":"2024-02-12T17:11:21.154562+0000","last_clean":"2024-02-12T17:11:21.154562+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:11:21.154562+0000","last_undegraded":"2024-02-12T17:11:21.154562+0000","last_fullsized":"2024-02-12T17:11:21.154562+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.889955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.889955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":486,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.140682+0000","last_change":"2024-02-12T17:07:20.629548+0000","last_active":"2024-02-12T17:11:21.140682+0000","last_peered":"2024-02-12T17:11:21.140682+0000","last_clean":"2024-02-12T17:11:21.140682+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:11:21.140682+0000","last_undegraded":"2024-02-12T17:11:21.140682+0000","last_fullsized":"2024-02-12T17:11:21.140682+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.629467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.629467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":313,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.146095+0000","last_change":"2024-02-12T17:10:03.873123+0000","last_active":"2024-02-12T17:11:21.146095+0000","last_peered":"2024-02-12T17:11:21.146095+0000","last_clean":"2024-02-12T17:09:58.605059+0000","last_became_active":"2024-02-12T17:10:03.873123+0000","last_became_peered":"2024-02-12T17:10:03.873123+0000","last_unstale":"2024-02-12T17:11:21.146095+0000","last_undegraded":"2024-02-12T17:11:21.146095+0000","last_fullsized":"2024-02-12T17:10:03.863800+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":117,"reported_epoch":570,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:21.144980+0000","last_change":"2024-02-12T17:10:03.883757+0000","last_active":"2024-02-12T17:11:21.144980+0000","last_peered":"2024-02-12T17:11:21.144980+0000","last_clean":"2024-02-12T17:09:58.608650+0000","last_became_active":"2024-02-12T17:10:03.883757+0000","last_became_peered":"2024-02-12T17:10:03.883757+0000","last_unstale":"2024-02-12T17:11:21.144980+0000","last_undegraded":"2024-02-12T17:10:03.857329+0000","last_fullsized":"2024-02-12T17:10:03.857219+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"570'11461","reported_seq":11607,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.269515+0000","last_change":"2024-02-12T17:07:53.893435+0000","last_active":"2024-02-12T17:11:21.269515+0000","last_peered":"2024-02-12T17:11:21.269515+0000","last_clean":"2024-02-12T17:11:21.269515+0000","last_became_active":"2024-02-12T17:07:52.071379+0000","last_became_peered":"2024-02-12T17:07:52.071379+0000","last_unstale":"2024-02-12T17:11:21.269515+0000","last_undegraded":"2024-02-12T17:11:21.269515+0000","last_fullsized":"2024-02-12T17:11:21.269515+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":451,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'7593","last_scrub_stamp":"2024-02-12T17:07:53.893352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:07:53.893352+0000","log_size":11461,"ondisk_log_size":11461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3868,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":273,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.155518+0000","last_change":"2024-02-12T17:06:43.014800+0000","last_active":"2024-02-12T17:11:21.155518+0000","last_peered":"2024-02-12T17:11:21.155518+0000","last_clean":"2024-02-12T17:11:21.155518+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:11:21.155518+0000","last_undegraded":"2024-02-12T17:11:21.155518+0000","last_fullsized":"2024-02-12T17:11:21.155518+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:06:43.014737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":492,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-12T17:11:20.153386+0000","last_change":"2024-02-12T17:10:03.870578+0000","last_active":"2024-02-12T17:11:20.153386+0000","last_peered":"2024-02-12T17:11:20.153386+0000","last_clean":"2024-02-12T17:09:58.609952+0000","last_became_active":"2024-02-12T17:10:03.870578+0000","last_became_peered":"2024-02-12T17:10:03.870578+0000","last_unstale":"2024-02-12T17:11:20.153386+0000","last_undegraded":"2024-02-12T17:11:20.153386+0000","last_fullsized":"2024-02-12T17:10:03.855937+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:17.122185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.122185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":214,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.189663+0000","last_change":"2024-02-12T17:10:03.951224+0000","last_active":"2024-02-12T17:11:20.189663+0000","last_peered":"2024-02-12T17:11:20.189663+0000","last_clean":"2024-02-12T17:08:07.422291+0000","last_became_active":"2024-02-12T17:10:03.951224+0000","last_became_peered":"2024-02-12T17:10:03.951224+0000","last_unstale":"2024-02-12T17:11:20.189663+0000","last_undegraded":"2024-02-12T17:10:03.865614+0000","last_fullsized":"2024-02-12T17:10:03.865499+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":106,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:20.156844+0000","last_change":"2024-02-12T17:10:03.935729+0000","last_active":"2024-02-12T17:11:20.156844+0000","last_peered":"2024-02-12T17:11:20.156844+0000","last_clean":"2024-02-12T17:09:51.251658+0000","last_became_active":"2024-02-12T17:10:03.935729+0000","last_became_peered":"2024-02-12T17:10:03.935729+0000","last_unstale":"2024-02-12T17:11:20.156844+0000","last_undegraded":"2024-02-12T17:10:03.860763+0000","last_fullsized":"2024-02-12T17:10:03.860648+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:38.887701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.887701+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"569'11730","reported_seq":20611,"reported_epoch":570,"state":"active+undersized","last_fresh":"2024-02-12T17:11:21.145849+0000","last_change":"2024-02-12T17:11:17.349571+0000","last_active":"2024-02-12T17:11:21.145849+0000","last_peered":"2024-02-12T17:11:21.145849+0000","last_clean":"2024-02-12T17:10:02.652574+0000","last_became_active":"2024-02-12T17:10:03.881164+0000","last_became_peered":"2024-02-12T17:10:03.881164+0000","last_unstale":"2024-02-12T17:11:21.145849+0000","last_undegraded":"2024-02-12T17:11:21.145849+0000","last_fullsized":"2024-02-12T17:10:03.862533+0000","mapping_epoch":533,"log_start":"490'7835","ondisk_log_start":"490'7835","created":451,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:17.342707+0000","last_clean_scrub_stamp":"2024-02-12T17:06:17.342707+0000","log_size":3895,"ondisk_log_size":3895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":249792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4344,"num_bytes_recovered":284688384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","2","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":572,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.147350+0000","last_change":"2024-02-12T17:09:21.531721+0000","last_active":"2024-02-12T17:11:21.147350+0000","last_peered":"2024-02-12T17:11:21.147350+0000","last_clean":"2024-02-12T17:11:21.147350+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:11:21.147350+0000","last_undegraded":"2024-02-12T17:11:21.147350+0000","last_fullsized":"2024-02-12T17:11:21.147350+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:47.945301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:47.945301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":236,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-12T17:11:20.178726+0000","last_change":"2024-02-12T17:06:53.648932+0000","last_active":"2024-02-12T17:11:20.178726+0000","last_peered":"2024-02-12T17:11:20.178726+0000","last_clean":"2024-02-12T17:11:20.178726+0000","last_became_active":"2024-02-12T17:03:50.557346+0000","last_became_peered":"2024-02-12T17:03:50.557346+0000","last_unstale":"2024-02-12T17:11:20.178726+0000","last_undegraded":"2024-02-12T17:11:20.178726+0000","last_fullsized":"2024-02-12T17:11:20.178726+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":688,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-02-12T17:11:21.146351+0000","last_change":"2024-02-12T17:09:22.375867+0000","last_active":"2024-02-12T17:11:21.146351+0000","last_peered":"2024-02-12T17:11:21.146351+0000","last_clean":"2024-02-12T17:11:21.146351+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:11:21.146351+0000","last_undegraded":"2024-02-12T17:11:21.146351+0000","last_fullsized":"2024-02-12T17:11:21.146351+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:07:47.772969+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.772969+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"534'26","reported_seq":955,"reported_epoch":571,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:11:22.149754+0000","last_change":"2024-02-12T17:10:03.878642+0000","last_active":"2024-02-12T17:11:22.149754+0000","last_peered":"2024-02-12T17:11:22.149754+0000","last_clean":"2024-02-12T17:10:02.527484+0000","last_became_active":"2024-02-12T17:10:03.878642+0000","last_became_peered":"2024-02-12T17:10:03.878642+0000","last_unstale":"2024-02-12T17:11:22.149754+0000","last_undegraded":"2024-02-12T17:10:03.855313+0000","last_fullsized":"2024-02-12T17:10:03.855201+0000","mapping_epoch":533,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-02-12T17:06:35.066866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.066866+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":36,"stat_sum":{"num_bytes":37257228,"num_objects":568,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":1543,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242519,"num_write_kb":7638337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35318,"num_bytes_recovered":2310537240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1547567104,"data_stored":1547567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189449,"ondisk_log_size":189449,"up":88,"acting":104,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":82,"acting":82,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":562,"num_read_kb":557,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1514,"num_read_kb":1271,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":656,"num_bytes_recovered":13444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1023,"ondisk_log_size":1023,"up":72,"acting":72,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":55,"acting":57,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":59,"acting":59,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234248,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1553336,"kb_used_data":657776,"kb_used_omap":0,"kb_used_meta":895552,"kb_avail":92191816,"statfs":{"total":95995035648,"available":94404419584,"internally_reserved":0,"allocated":673562624,"data_stored":672771740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917045248},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.9300000000000002,"5min":1.6459999999999999,"15min":1.9430000000000001},"min":{"1min":0.221,"5min":0.218,"15min":0.16700000000000001},"max":{"1min":8.8510000000000009,"5min":8.8510000000000009,"15min":16.759},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.8540000000000001,"5min":1.3480000000000001,"15min":1.95},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.19600000000000001},"max":{"1min":6.633,"5min":7.8440000000000003,"15min":17.844000000000001},"last":0.31}]},{"osd":1,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":2.23,"15min":1.8560000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":6.4539999999999997,"5min":13.224,"15min":13.224},"last":1.234},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.6679999999999999,"15min":1.7549999999999999},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.22700000000000001},"max":{"1min":6.4660000000000002,"5min":10.194000000000001,"15min":17.324999999999999},"last":1.165}]},{"osd":2,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.1299999999999999,"5min":4.4089999999999998,"15min":2.6579999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":10.401,"5min":218.35300000000001,"15min":218.35300000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":1.706,"5min":4.0010000000000003,"15min":2.8969999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.23000000000000001},"max":{"1min":7.8600000000000003,"5min":218.42099999999999,"15min":218.42099999999999},"last":0.438}]},{"osd":3,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":3.226,"5min":1.651,"15min":1.8779999999999999},"min":{"1min":0.317,"5min":0.255,"15min":0.255},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":13.608000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":2.7879999999999998,"5min":1.8899999999999999,"15min":2.3039999999999998},"min":{"1min":0.29899999999999999,"5min":0.28399999999999997,"15min":0.25600000000000001},"max":{"1min":8.1600000000000001,"5min":13.699,"15min":31.152000000000001},"last":0.40600000000000003}]},{"osd":4,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.67500000000000004,"15min":2.012},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8439999999999999,"5min":3.9279999999999999,"15min":50.406999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.70499999999999996,"15min":2.431},"min":{"1min":0.29999999999999999,"5min":0.249,"15min":0.249},"max":{"1min":3.988,"5min":4.5609999999999999,"15min":50.168999999999997},"last":0.46500000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.2090000000000001,"15min":2.2970000000000002},"min":{"1min":0.26000000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.8660000000000001,"5min":56.560000000000002,"15min":56.560000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.149,"15min":2.3620000000000001},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.117,"5min":53.963999999999999,"15min":53.963999999999999},"last":0.47999999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.621,"15min":1.786},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.1800000000000002,"5min":2.645,"15min":17.234999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.78700000000000003,"15min":1.8149999999999999},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":2.9079999999999999,"5min":3.5329999999999999,"15min":13.676},"last":0.44900000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528589,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311292,"kb_used_data":1780,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433860,"statfs":{"total":95995035648,"available":95676272640,"internally_reserved":0,"allocated":1822720,"data_stored":1028140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:10:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.853,"15min":1.944},"min":{"1min":0.20899999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":11.9,"5min":27.744,"15min":27.744},"last":0.40500000000000003},{"interface":"front","average":{"1min":2.3900000000000001,"5min":1.8049999999999999,"15min":1.897},"min":{"1min":0.23699999999999999,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":7.8040000000000003,"5min":27.614000000000001,"15min":27.614000000000001},"last":0.30099999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:09:40 2024","interfaces":[{"interface":"back","average":{"1min":2.9870000000000001,"5min":1.74,"15min":1.629},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":14.884,"5min":14.884,"15min":14.884},"last":1.77},{"interface":"front","average":{"1min":3.0430000000000001,"5min":1.8220000000000001,"15min":1.8340000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":17.106999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:10:44 2024","interfaces":[{"interface":"back","average":{"1min":2.9929999999999999,"5min":2.6960000000000002,"15min":2.3700000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.22900000000000001},"max":{"1min":19.951000000000001,"5min":24.300999999999998,"15min":56.798000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.8919999999999999,"15min":2.21},"min":{"1min":0.28999999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.0609999999999999,"5min":24.143999999999998,"15min":56.780999999999999},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:10:44 2024","interfaces":[{"interface":"back","average":{"1min":3.2269999999999999,"5min":2.472,"15min":1.9450000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":15.355,"5min":24.626000000000001,"15min":24.626000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.5979999999999999,"15min":2.2240000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.8949999999999996,"5min":26.686,"15min":26.686},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:10:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.55500000000000005,"15min":1.6819999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.456,"5min":5.2220000000000004,"15min":24.956},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.749,"15min":1.599},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":2.6360000000000001,"5min":13.728,"15min":13.728},"last":0.36799999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:10:44 2024","interfaces":[{"interface":"back","average":{"1min":2.4420000000000002,"5min":0.995,"15min":2.0739999999999998},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.245},"max":{"1min":34.869,"5min":34.869,"15min":34.869},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.2679999999999998,"5min":1.071,"15min":2.3210000000000002},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":33.591000000000001,"5min":33.591000000000001,"15min":33.591000000000001},"last":0.34300000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:11:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.71699999999999997,"15min":2.0859999999999999},"min":{"1min":0.28299999999999997,"5min":0.255,"15min":0.255},"max":{"1min":6.9279999999999999,"5min":6.9279999999999999,"15min":31.033000000000001},"last":0.378},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.72799999999999998,"15min":2.0110000000000001},"min":{"1min":0.34300000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.7090000000000001,"5min":5.2389999999999999,"15min":29.34},"last":0.42099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575125,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917072,"kb_used_data":319176,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":92828080,"statfs":{"total":95995035648,"available":95055953920,"internally_reserved":0,"allocated":326836224,"data_stored":325617050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2009999999999996,"5min":2.5369999999999999,"15min":1.7230000000000001},"min":{"1min":0.23499999999999999,"5min":0.20699999999999999,"15min":0.17499999999999999},"max":{"1min":65.328999999999994,"5min":65.328999999999994,"15min":65.328999999999994},"last":3.5190000000000001},{"interface":"front","average":{"1min":6.6280000000000001,"5min":2.516,"15min":1.7050000000000001},"min":{"1min":0.22800000000000001,"5min":0.193,"15min":0.189},"max":{"1min":65.372,"5min":65.372,"15min":65.372},"last":10.164}]},{"osd":2,"last update":"Mon Feb 12 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":5.9429999999999996,"5min":2.6989999999999998,"15min":2.0019999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.191},"max":{"1min":51.820999999999998,"5min":51.820999999999998,"15min":51.820999999999998},"last":2.2229999999999999},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.395,"15min":1.847},"min":{"1min":0.223,"5min":0.223,"15min":0.19},"max":{"1min":53.802999999999997,"5min":53.802999999999997,"15min":53.802999999999997},"last":9.8190000000000008}]},{"osd":3,"last update":"Mon Feb 12 17:09:37 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":2.7789999999999999,"15min":1.823},"min":{"1min":0.308,"5min":0.28999999999999998,"15min":0.23100000000000001},"max":{"1min":10.977,"5min":64.528000000000006,"15min":64.528000000000006},"last":12.565},{"interface":"front","average":{"1min":2.9289999999999998,"5min":2.4449999999999998,"15min":1.647},"min":{"1min":0.34200000000000003,"5min":0.312,"15min":0.246},"max":{"1min":11.124000000000001,"5min":63.753999999999998,"15min":63.753999999999998},"last":3.452}]},{"osd":4,"last update":"Mon Feb 12 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.6859999999999999,"15min":1.8879999999999999},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":7.556,"5min":12.02,"15min":12.91},"last":3.419},{"interface":"front","average":{"1min":1.849,"5min":1.5449999999999999,"15min":1.8260000000000001},"min":{"1min":0.374,"5min":0.33000000000000002,"15min":0.25900000000000001},"max":{"1min":9.6159999999999997,"5min":12.193,"15min":16.074000000000002},"last":4.04}]},{"osd":5,"last update":"Mon Feb 12 17:09:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.8819999999999999,"15min":2.0739999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.306,"5min":22.154,"15min":22.154},"last":3.4980000000000002},{"interface":"front","average":{"1min":1.423,"5min":1.754,"15min":1.913},"min":{"1min":0.317,"5min":0.317,"15min":0.27300000000000002},"max":{"1min":8.5820000000000007,"5min":21.803000000000001,"15min":21.803000000000001},"last":4.0060000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.7170000000000001,"15min":1.9430000000000001},"min":{"1min":0.38,"5min":0.36499999999999999,"15min":0.30599999999999999},"max":{"1min":7.6210000000000004,"5min":7.6210000000000004,"15min":13.834},"last":3.4750000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.7470000000000001,"15min":1.7789999999999999},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.307},"max":{"1min":4.3700000000000001,"5min":16.172999999999998,"15min":16.172999999999998},"last":4.0640000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.9239999999999999,"15min":2.0289999999999999},"min":{"1min":0.377,"5min":0.33100000000000002,"15min":0.28799999999999998},"max":{"1min":7.3099999999999996,"5min":8.7560000000000002,"15min":9.5239999999999991},"last":2.1880000000000002},{"interface":"front","average":{"1min":1.8200000000000001,"5min":2.0790000000000002,"15min":2.1669999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":9.3510000000000009,"5min":13.965999999999999,"15min":20.548999999999999},"last":5.835}]}]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139504,"kb_used_data":243304,"kb_used_omap":0,"kb_used_meta":896192,"kb_avail":92605648,"statfs":{"total":95995035648,"available":94828183552,"internally_reserved":0,"allocated":249143296,"data_stored":248424341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917700608},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:09:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8809999999999998,"5min":1.694,"15min":1.484},"min":{"1min":0.253,"5min":0.22,"15min":0.17399999999999999},"max":{"1min":9.7420000000000009,"5min":10.807,"15min":23.510999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":4.3070000000000004,"5min":2.5259999999999998,"15min":1.6970000000000001},"min":{"1min":0.29699999999999999,"5min":0.20399999999999999,"15min":0.193},"max":{"1min":10.092000000000001,"5min":10.741,"15min":16.16},"last":6.2830000000000004}]},{"osd":2,"last update":"Mon Feb 12 17:11:20 2024","interfaces":[{"interface":"back","average":{"1min":4.5679999999999996,"5min":3.2109999999999999,"15min":1.8160000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.20200000000000001},"max":{"1min":13.791,"5min":15.472,"15min":15.472},"last":0.313},{"interface":"front","average":{"1min":4.4429999999999996,"5min":4.1059999999999999,"15min":2.2290000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":11.875999999999999,"5min":26.039999999999999,"15min":26.039999999999999},"last":0.26400000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:10:38 2024","interfaces":[{"interface":"back","average":{"1min":6.5259999999999998,"5min":4.4290000000000003,"15min":2.4470000000000001},"min":{"1min":0.23000000000000001,"5min":0.214,"15min":0.214},"max":{"1min":18.757999999999999,"5min":87.311000000000007,"15min":87.311000000000007},"last":0.30199999999999999},{"interface":"front","average":{"1min":4.8719999999999999,"5min":4.2169999999999996,"15min":2.3929999999999998},"min":{"1min":0.40100000000000002,"5min":0.27800000000000002,"15min":0.221},"max":{"1min":12.51,"5min":90.991,"15min":90.991},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":1.3240000000000001,"15min":1.5640000000000001},"min":{"1min":0.33400000000000002,"5min":0.30199999999999999,"15min":0.23599999999999999},"max":{"1min":11.43,"5min":11.43,"15min":16.888999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":7.0090000000000003,"5min":2.6619999999999999,"15min":2.1429999999999998},"min":{"1min":0.34699999999999998,"5min":0.32500000000000001,"15min":0.28399999999999997},"max":{"1min":26.082999999999998,"5min":26.082999999999998,"15min":26.082999999999998},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:10:27 2024","interfaces":[{"interface":"back","average":{"1min":5.1840000000000002,"5min":2.234,"15min":2.4340000000000002},"min":{"1min":0.32500000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":17.754000000000001,"5min":17.754000000000001,"15min":50.404000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":2.919,"5min":1.8500000000000001,"15min":2.1560000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28299999999999997},"max":{"1min":11.398,"5min":19.751999999999999,"15min":49.979999999999997},"last":0.46000000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":7.2750000000000004,"5min":2.9929999999999999,"15min":2.23},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.29999999999999999},"max":{"1min":26.109999999999999,"5min":26.109999999999999,"15min":26.109999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":4.8319999999999999,"5min":2.294,"15min":2.1989999999999998},"min":{"1min":0.38600000000000001,"5min":0.372,"15min":0.33600000000000002},"max":{"1min":15.847,"5min":15.847,"15min":16.294},"last":0.66800000000000004}]},{"osd":7,"last update":"Mon Feb 12 17:11:20 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":1.7849999999999999,"15min":2.0430000000000001},"min":{"1min":0.44600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":11.459,"5min":11.459,"15min":19.373000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":3.3210000000000002,"5min":2.8180000000000001,"15min":2.4940000000000002},"min":{"1min":0.44600000000000001,"5min":0.38200000000000001,"15min":0.35599999999999998},"max":{"1min":8.0250000000000004,"5min":26.135000000000002,"15min":34.722999999999999},"last":0.65700000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411615,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335012,"kb_used_data":153948,"kb_used_omap":0,"kb_used_meta":1181056,"kb_avail":92410140,"statfs":{"total":95995035648,"available":94627983360,"internally_reserved":0,"allocated":157642752,"data_stored":156872594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1209401344},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:19 2024","interfaces":[{"interface":"back","average":{"1min":2.5270000000000001,"5min":2.6459999999999999,"15min":1.7310000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17000000000000001},"max":{"1min":11.253,"5min":21.445,"15min":36.100999999999999},"last":3.169},{"interface":"front","average":{"1min":3.3570000000000002,"5min":2.7559999999999998,"15min":1.9750000000000001},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.193},"max":{"1min":7.617,"5min":19.879000000000001,"15min":36.299999999999997},"last":4.1040000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":3.282,"5min":1.948,"15min":1.776},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.19600000000000001},"max":{"1min":10.922000000000001,"5min":20.763999999999999,"15min":20.763999999999999},"last":6.2999999999999998},{"interface":"front","average":{"1min":4.3310000000000004,"5min":2.3029999999999999,"15min":1.984},"min":{"1min":0.31900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":12.552,"5min":21.361999999999998,"15min":21.361999999999998},"last":6.657}]},{"osd":3,"last update":"Mon Feb 12 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":6.3239999999999998,"5min":3.9279999999999999,"15min":2.1560000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.24199999999999999},"max":{"1min":40.759999999999998,"5min":40.759999999999998,"15min":40.759999999999998},"last":2.3279999999999998},{"interface":"front","average":{"1min":3.1160000000000001,"5min":3.1099999999999999,"15min":1.877},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23200000000000001},"max":{"1min":7.9749999999999996,"5min":21.702999999999999,"15min":21.702999999999999},"last":3.2789999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:11:11 2024","interfaces":[{"interface":"back","average":{"1min":2.6440000000000001,"5min":2.2109999999999999,"15min":2.0169999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":11.204000000000001,"5min":21.515000000000001,"15min":21.515000000000001},"last":5.5739999999999998},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.998,"15min":2.1440000000000001},"min":{"1min":0.307,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6900000000000004,"5min":20.149000000000001,"15min":20.149000000000001},"last":5.5640000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:11:19 2024","interfaces":[{"interface":"back","average":{"1min":2.597,"5min":2.2000000000000002,"15min":2.0339999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":11.351000000000001,"5min":21.542999999999999,"15min":32.57},"last":3.2280000000000002},{"interface":"front","average":{"1min":2.2850000000000001,"5min":2.121,"15min":2.2759999999999998},"min":{"1min":0.35099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":10.896000000000001,"5min":20.937000000000001,"15min":31.905999999999999},"last":0.35099999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":3.2120000000000002,"5min":2.214,"15min":2.0499999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.30199999999999999},"max":{"1min":8.5850000000000009,"5min":20.966999999999999,"15min":20.966999999999999},"last":2.1749999999999998},{"interface":"front","average":{"1min":2.0550000000000002,"5min":1.7569999999999999,"15min":1.889},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.29799999999999999},"max":{"1min":6.3099999999999996,"5min":20.193000000000001,"15min":20.193000000000001},"last":3.306}]},{"osd":7,"last update":"Mon Feb 12 17:11:19 2024","interfaces":[{"interface":"back","average":{"1min":3.1299999999999999,"5min":2.1139999999999999,"15min":2.5640000000000001},"min":{"1min":0.34899999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":7.5919999999999996,"5min":20.221,"15min":45.468000000000004},"last":4.9550000000000001},{"interface":"front","average":{"1min":2.2309999999999999,"5min":2.254,"15min":2.5489999999999999},"min":{"1min":0.40600000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":10.484999999999999,"5min":20.995999999999999,"15min":46.656999999999996},"last":0.80200000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149977,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908484,"kb_used_data":2940,"kb_used_omap":0,"kb_used_meta":905536,"kb_avail":92836668,"statfs":{"total":95995035648,"available":95064748032,"internally_reserved":0,"allocated":3010560,"data_stored":2201525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":927268864},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:10:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4640000000000004,"5min":3.3889999999999998,"15min":1.9239999999999999},"min":{"1min":0.48199999999999998,"5min":0.19700000000000001,"15min":0.17100000000000001},"max":{"1min":12.334,"5min":18.497,"15min":58.451999999999998},"last":6.9320000000000004},{"interface":"front","average":{"1min":4.835,"5min":3.5790000000000002,"15min":2.3239999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":10.167,"5min":18.815999999999999,"15min":57.497999999999998},"last":1.361}]},{"osd":1,"last update":"Mon Feb 12 17:09:40 2024","interfaces":[{"interface":"back","average":{"1min":5.2539999999999996,"5min":2.9020000000000001,"15min":1.8779999999999999},"min":{"1min":0.28799999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.859999999999999,"5min":18.859999999999999,"15min":18.859999999999999},"last":8.9280000000000008},{"interface":"front","average":{"1min":3.4009999999999998,"5min":1.7889999999999999,"15min":1.3759999999999999},"min":{"1min":0.48499999999999999,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":14.654,"5min":14.654,"15min":14.962},"last":5.1630000000000003}]},{"osd":2,"last update":"Mon Feb 12 17:10:41 2024","interfaces":[{"interface":"back","average":{"1min":5.9139999999999997,"5min":3.0409999999999999,"15min":1.873},"min":{"1min":0.52300000000000002,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":28.033999999999999,"5min":37.892000000000003,"15min":49.112000000000002},"last":3.4830000000000001},{"interface":"front","average":{"1min":7.9320000000000004,"5min":3.8650000000000002,"15min":2.0419999999999998},"min":{"1min":0.80600000000000005,"5min":0.22,"15min":0.22},"max":{"1min":15.606,"5min":30.949999999999999,"15min":51.997999999999998},"last":6.8869999999999996}]},{"osd":4,"last update":"Mon Feb 12 17:11:05 2024","interfaces":[{"interface":"back","average":{"1min":3.286,"5min":2.6779999999999999,"15min":2.246},"min":{"1min":0.40999999999999998,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":7.9169999999999998,"5min":18.401,"15min":19.010000000000002},"last":1.3879999999999999},{"interface":"front","average":{"1min":2.831,"5min":1.5329999999999999,"15min":1.77},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.8470000000000004,"5min":7.9880000000000004,"15min":19.033999999999999},"last":2.1099999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:10:28 2024","interfaces":[{"interface":"back","average":{"1min":5.1429999999999998,"5min":2.544,"15min":2.2349999999999999},"min":{"1min":0.45200000000000001,"5min":0.253,"15min":0.253},"max":{"1min":14.692,"5min":17.940999999999999,"15min":17.940999999999999},"last":6.8300000000000001},{"interface":"front","average":{"1min":5.069,"5min":2.6960000000000002,"15min":2.4900000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":16.564,"5min":16.564,"15min":16.564},"last":1.4119999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:10:41 2024","interfaces":[{"interface":"back","average":{"1min":3.9089999999999998,"5min":1.8440000000000001,"15min":1.716},"min":{"1min":0.32400000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.8010000000000002,"5min":14.618,"15min":14.618},"last":2.1349999999999998},{"interface":"front","average":{"1min":5.1710000000000003,"5min":2.6720000000000002,"15min":1.9550000000000001},"min":{"1min":0.40999999999999998,"5min":0.32600000000000001,"15min":0.27500000000000002},"max":{"1min":12.257999999999999,"5min":18.265000000000001,"15min":18.265000000000001},"last":6.8570000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:11:05 2024","interfaces":[{"interface":"back","average":{"1min":4.8719999999999999,"5min":3.1389999999999998,"15min":2.7269999999999999},"min":{"1min":0.45200000000000001,"5min":0.35399999999999998,"15min":0.315},"max":{"1min":13.159000000000001,"5min":18.484000000000002,"15min":37.719999999999999},"last":1.431},{"interface":"front","average":{"1min":5.1920000000000002,"5min":3.0019999999999998,"15min":2.5950000000000002},"min":{"1min":0.40799999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":17.015000000000001,"5min":17.992999999999999,"15min":38.948},"last":6.8730000000000002}]}]},{"osd":4,"up_from":0,"seq":137438953751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757523,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1342440,"kb_used_data":149280,"kb_used_omap":0,"kb_used_meta":1193152,"kb_avail":92402712,"statfs":{"total":95995035648,"available":94620377088,"internally_reserved":0,"allocated":152862720,"data_stored":152087844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1221787648},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.4139999999999999,"15min":1.7729999999999999},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":7.4059999999999997,"5min":10.858000000000001,"15min":46.365000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.9510000000000001,"5min":1.536,"15min":2.1509999999999998},"min":{"1min":0.29899999999999999,"5min":0.22500000000000001,"15min":0.19900000000000001},"max":{"1min":15.368,"5min":15.368,"15min":46.134},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.387,"15min":1.7450000000000001},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.23200000000000001},"max":{"1min":8.3620000000000001,"5min":11.19,"15min":16.655000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":2.3149999999999999,"5min":2.02,"15min":2.097},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.21199999999999999},"max":{"1min":10.973000000000001,"5min":12.154,"15min":16.712},"last":1.319}]},{"osd":2,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":2.6560000000000001,"5min":1.7270000000000001,"15min":2.012},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":15.612,"5min":15.612,"15min":16.940999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.8570000000000002,"5min":1.538,"15min":1.821},"min":{"1min":0.32000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":13.845000000000001,"5min":13.845000000000001,"15min":17.103999999999999},"last":0.38400000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":9.0950000000000006,"5min":3.6949999999999998,"15min":2.4910000000000001},"min":{"1min":0.30299999999999999,"5min":0.26100000000000001,"15min":0.22600000000000001},"max":{"1min":43.259,"5min":43.259,"15min":43.259},"last":0.36399999999999999},{"interface":"front","average":{"1min":5.7750000000000004,"5min":3.1459999999999999,"15min":2.2719999999999998},"min":{"1min":0.34799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":43.198,"5min":43.198,"15min":43.198},"last":0.443}]},{"osd":4,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.55500000000000005,"15min":1.833},"min":{"1min":0.246,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":4.625,"5min":4.625,"15min":12.656000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.53700000000000003,"15min":1.9259999999999999},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":4.3250000000000002,"5min":4.3250000000000002,"15min":20.550000000000001},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Feb 12 17:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57299999999999995,"15min":1.7430000000000001},"min":{"1min":0.27700000000000002,"5min":0.27500000000000002,"15min":0.26400000000000001},"max":{"1min":2.161,"5min":3.5830000000000002,"15min":12.281000000000001},"last":0.375},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.63500000000000001,"15min":1.855},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":4.3659999999999997,"5min":4.3659999999999997,"15min":22.035},"last":0.436}]},{"osd":7,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.60599999999999998,"15min":1.905},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":5.3230000000000004,"5min":5.3230000000000004,"15min":19.184999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.72399999999999998,"15min":2.0299999999999998},"min":{"1min":0.35799999999999998,"5min":0.27800000000000002,"15min":0.26900000000000002},"max":{"1min":4.8159999999999998,"5min":5.1239999999999997,"15min":20.274000000000001},"last":0.42599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247398400,"data_stored":247398400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323420160,"data_stored":323420160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155844608,"data_stored":155844608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":149094400,"data_stored":149094400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":671744000,"data_stored":671744000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:11:26.024 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-12T17:11:26.024 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set .rgw.root pg_num 22 2024-02-12T17:11:26.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:26.141 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:26.162 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:26.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:26.266 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:26.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:24.770326+0000 mgr.y (mgr.24481) 820 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:26.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:24.925251+0000 mgr.y (mgr.24481) 821 : cluster [DBG] pgmap v1146: 148 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 16 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 318 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 1.9 MiB/s, 40 objects/s recovering 2024-02-12T17:11:26.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:25.051069+0000 mgr.y (mgr.24481) 822 : audit [DBG] from='client.15987 v1:172.21.15.33:0/3960717454' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:11:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:24.770326+0000 mgr.y (mgr.24481) 820 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:24.925251+0000 mgr.y (mgr.24481) 821 : cluster [DBG] pgmap v1146: 148 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 16 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 318 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 1.9 MiB/s, 40 objects/s recovering 2024-02-12T17:11:26.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:25.051069+0000 mgr.y (mgr.24481) 822 : audit [DBG] from='client.15987 v1:172.21.15.33:0/3960717454' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:11:26.301 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:26.334 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:26.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:26.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:26.466 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:24.770326+0000 mgr.y (mgr.24481) 820 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:26.467 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:24.925251+0000 mgr.y (mgr.24481) 821 : cluster [DBG] pgmap v1146: 148 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 16 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 318 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 1.9 MiB/s, 40 objects/s recovering 2024-02-12T17:11:26.467 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:25.051069+0000 mgr.y (mgr.24481) 822 : audit [DBG] from='client.15987 v1:172.21.15.33:0/3960717454' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:11:26.468 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:26.489 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:26.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:26.593 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:26.718 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:26.828 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:26.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:26.934 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:26.963 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:27.043 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:27.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:27.147 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:27.174 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:27.192 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:27.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:26.562592+0000 mon.a (mon.0) 1899 : cluster [WRN] Health check update: Degraded data redundancy: 214/2280 objects degraded (9.386%), 31 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:26.572492+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e572: 8 total, 7 up, 7 in 2024-02-12T17:11:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:26.842676+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? v1:172.21.15.33:0/1746843916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:11:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:26.869878+0000 mon.c (mon.2) 267 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:11:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:26.562592+0000 mon.a (mon.0) 1899 : cluster [WRN] Health check update: Degraded data redundancy: 214/2280 objects degraded (9.386%), 31 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:26.572492+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e572: 8 total, 7 up, 7 in 2024-02-12T17:11:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:26.842676+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? v1:172.21.15.33:0/1746843916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:11:27.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:26.869878+0000 mon.c (mon.2) 267 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:11:27.297 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:27.325 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:26.562592+0000 mon.a (mon.0) 1899 : cluster [WRN] Health check update: Degraded data redundancy: 214/2280 objects degraded (9.386%), 31 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:26.572492+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e572: 8 total, 7 up, 7 in 2024-02-12T17:11:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:26.842676+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? v1:172.21.15.33:0/1746843916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:11:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:26.869878+0000 mon.c (mon.2) 267 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:11:27.593 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:11:28.111 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:11:28.111 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:11:28.111 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-02-12T17:11:28.111 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:11:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:26.925840+0000 mgr.y (mgr.24481) 823 : cluster [DBG] pgmap v1148: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 257 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 1.3 MiB/s, 39 objects/s recovering 2024-02-12T17:11:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:27.581093+0000 mon.a (mon.0) 1902 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:27.588868+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? v1:172.21.15.33:0/1746843916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:11:28.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:27.588944+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e573: 8 total, 7 up, 7 in 2024-02-12T17:11:28.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:26.925840+0000 mgr.y (mgr.24481) 823 : cluster [DBG] pgmap v1148: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 257 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 1.3 MiB/s, 39 objects/s recovering 2024-02-12T17:11:28.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:27.581093+0000 mon.a (mon.0) 1902 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:28.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:27.588868+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? v1:172.21.15.33:0/1746843916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:11:28.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:27.588944+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e573: 8 total, 7 up, 7 in 2024-02-12T17:11:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:26.925840+0000 mgr.y (mgr.24481) 823 : cluster [DBG] pgmap v1148: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 257 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 1.3 MiB/s, 39 objects/s recovering 2024-02-12T17:11:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:27.581093+0000 mon.a (mon.0) 1902 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:27.588868+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? v1:172.21.15.33:0/1746843916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:11:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:27.588944+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e573: 8 total, 7 up, 7 in 2024-02-12T17:11:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:29.125839+0000 mon.c (mon.2) 268 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:11:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:29.126709+0000 mon.c (mon.2) 269 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:11:29.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:29.131736+0000 mon.a (mon.0) 1905 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:29.125839+0000 mon.c (mon.2) 268 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:11:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:29.126709+0000 mon.c (mon.2) 269 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:11:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:29.131736+0000 mon.a (mon.0) 1905 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:29.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:29.125839+0000 mon.c (mon.2) 268 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:11:29.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:29.126709+0000 mon.c (mon.2) 269 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:11:29.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:29.131736+0000 mon.a (mon.0) 1905 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:30.017 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:30.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:30.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:30.149 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:30.168 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:30.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:30.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:30.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:28.926624+0000 mgr.y (mgr.24481) 824 : cluster [DBG] pgmap v1150: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 140 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 698 KiB/s, 25 objects/s recovering 2024-02-12T17:11:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:28.926624+0000 mgr.y (mgr.24481) 824 : cluster [DBG] pgmap v1150: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 140 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 698 KiB/s, 25 objects/s recovering 2024-02-12T17:11:30.299 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:30.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:30.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:30.452 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:30.458 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:30.481 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:30.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:30.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:28.926624+0000 mgr.y (mgr.24481) 824 : cluster [DBG] pgmap v1150: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 140 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 698 KiB/s, 25 objects/s recovering 2024-02-12T17:11:30.589 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:30.634 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:30.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:30.738 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:30.764 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:30.782 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:30.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:30.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:30.920 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:31.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:11:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:30] "GET /metrics HTTP/1.1" 200 176765 "" "Prometheus/2.33.4" 2024-02-12T17:11:31.404 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T17:11:32.021 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:11:32.276 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:32.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:30.927600+0000 mgr.y (mgr.24481) 825 : cluster [DBG] pgmap v1151: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 119 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 592 KiB/s, 28 objects/s recovering 2024-02-12T17:11:32.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:31.575551+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e574: 8 total, 7 up, 7 in 2024-02-12T17:11:32.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:30.927600+0000 mgr.y (mgr.24481) 825 : cluster [DBG] pgmap v1151: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 119 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 592 KiB/s, 28 objects/s recovering 2024-02-12T17:11:32.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:31.575551+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e574: 8 total, 7 up, 7 in 2024-02-12T17:11:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:32.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:32.383 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:32.411 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:32.430 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:32.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:32.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:30.927600+0000 mgr.y (mgr.24481) 825 : cluster [DBG] pgmap v1151: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 119 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 592 KiB/s, 28 objects/s recovering 2024-02-12T17:11:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:31.575551+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e574: 8 total, 7 up, 7 in 2024-02-12T17:11:32.533 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:32.563 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:32.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:11:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:11:33.223 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:11:33.231 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:33.260 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:11:33.289 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:11:33.311 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:11:33.329 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:11:33.348 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:11:33.367 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:11:33.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:11:33.368 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:11:33.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:11:33.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:11:33.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:11:33.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:11:33.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:11:33.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:11:33.628 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:32.520287+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:33.628 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:32.928200+0000 mgr.y (mgr.24481) 826 : cluster [DBG] pgmap v1153: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 806 B/s rd, 0 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 17 objects/s recovering 2024-02-12T17:11:33.628 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:33.204099+0000 mon.a (mon.0) 1908 : audit [DBG] from='client.? v1:172.21.15.33:0/4246202267' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:33.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:32.520287+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:33.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:32.928200+0000 mgr.y (mgr.24481) 826 : cluster [DBG] pgmap v1153: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 806 B/s rd, 0 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 17 objects/s recovering 2024-02-12T17:11:33.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:33.204099+0000 mon.a (mon.0) 1908 : audit [DBG] from='client.? v1:172.21.15.33:0/4246202267' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:33.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:32.520287+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:11:33.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:32.928200+0000 mgr.y (mgr.24481) 826 : cluster [DBG] pgmap v1153: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 806 B/s rd, 0 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 17 objects/s recovering 2024-02-12T17:11:33.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:33.204099+0000 mon.a (mon.0) 1908 : audit [DBG] from='client.? v1:172.21.15.33:0/4246202267' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:34.623 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 2 pg_num to 22 2024-02-12T17:11:34.642 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:33.499689+0000 osd.3 (osd.3) 199 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:34.642 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:33.499741+0000 osd.3 (osd.3) 200 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:34.642 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:33.529155+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e575: 8 total, 7 up, 7 in 2024-02-12T17:11:34.642 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:34.072177+0000 mon.c (mon.2) 270 : audit [INF] from='client.? v1:172.21.15.33:0/3890144912' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:11:34.642 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:34.072535+0000 mon.a (mon.0) 1910 : audit [INF] from='client.35942 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:11:34.642 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:33.499689+0000 osd.3 (osd.3) 199 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:34.643 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:33.499741+0000 osd.3 (osd.3) 200 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:34.643 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:33.529155+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e575: 8 total, 7 up, 7 in 2024-02-12T17:11:34.643 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:34.072177+0000 mon.c (mon.2) 270 : audit [INF] from='client.? v1:172.21.15.33:0/3890144912' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:11:34.643 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:34.072535+0000 mon.a (mon.0) 1910 : audit [INF] from='client.35942 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:11:34.782 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:33.499689+0000 osd.3 (osd.3) 199 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:34.782 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:33.499741+0000 osd.3 (osd.3) 200 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:34.782 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:33.529155+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e575: 8 total, 7 up, 7 in 2024-02-12T17:11:34.782 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:34.072177+0000 mon.c (mon.2) 270 : audit [INF] from='client.? v1:172.21.15.33:0/3890144912' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:11:34.782 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:34.072535+0000 mon.a (mon.0) 1910 : audit [INF] from='client.35942 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2024-02-12T17:11:34.815 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:34.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:34.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:34.924 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:34.950 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:34.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:35.073 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:35.103 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:35.229 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:35.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:35.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:11:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:11:35.335 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:35.363 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:35.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:34.533188+0000 mon.a (mon.0) 1911 : audit [INF] from='client.35942 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:11:35.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:34.533261+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e576: 8 total, 7 up, 7 in 2024-02-12T17:11:35.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:34.781071+0000 mgr.y (mgr.24481) 827 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:34.928786+0000 mgr.y (mgr.24481) 828 : cluster [DBG] pgmap v1156: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 27 objects/s recovering 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:34.929111+0000 mon.c (mon.2) 271 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:34.930034+0000 mon.a (mon.0) 1913 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:34.533188+0000 mon.a (mon.0) 1911 : audit [INF] from='client.35942 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:34.533261+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e576: 8 total, 7 up, 7 in 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:34.781071+0000 mgr.y (mgr.24481) 827 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:34.928786+0000 mgr.y (mgr.24481) 828 : cluster [DBG] pgmap v1156: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 27 objects/s recovering 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:34.929111+0000 mon.c (mon.2) 271 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:11:35.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:34.930034+0000 mon.a (mon.0) 1913 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:11:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:11:35.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:11:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:34.533188+0000 mon.a (mon.0) 1911 : audit [INF] from='client.35942 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]': finished 2024-02-12T17:11:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:34.533261+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e576: 8 total, 7 up, 7 in 2024-02-12T17:11:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:34.781071+0000 mgr.y (mgr.24481) 827 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:34.928786+0000 mgr.y (mgr.24481) 828 : cluster [DBG] pgmap v1156: 148 pgs: 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 69 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 27 objects/s recovering 2024-02-12T17:11:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:34.929111+0000 mon.c (mon.2) 271 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:11:35.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:34.930034+0000 mon.a (mon.0) 1913 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T17:11:36.277 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:36.305 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-02-12T17:11:36.305 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-02-12T17:11:36.322 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:36.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:36.427 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:36.476 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:36.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:36.580 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:36.691 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:35.184979+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:35.185052+0000 osd.5 (osd.5) 273 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:35.536422+0000 mon.a (mon.0) 1914 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:11:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:35.536495+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e577: 8 total, 7 up, 7 in 2024-02-12T17:11:36.793 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:36.798 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:35.184979+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:36.798 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:35.185052+0000 osd.5 (osd.5) 273 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:36.798 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:35.536422+0000 mon.a (mon.0) 1914 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:11:36.798 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:35.536495+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e577: 8 total, 7 up, 7 in 2024-02-12T17:11:36.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:35.184979+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:36.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:35.185052+0000 osd.5 (osd.5) 273 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:36.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:35.536422+0000 mon.a (mon.0) 1914 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T17:11:36.798 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:35.536495+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e577: 8 total, 7 up, 7 in 2024-02-12T17:11:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:36.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:36.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:36.944 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:36.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:37.048 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:37.078 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:37.531 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:37.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:37.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:37.689 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:37.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:36.543769+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e578: 8 total, 7 up, 7 in 2024-02-12T17:11:37.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:36.929417+0000 mgr.y (mgr.24481) 829 : cluster [DBG] pgmap v1159: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 68 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 26 objects/s recovering 2024-02-12T17:11:37.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:37.539868+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e579: 8 total, 7 up, 7 in 2024-02-12T17:11:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:36.543769+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e578: 8 total, 7 up, 7 in 2024-02-12T17:11:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:36.929417+0000 mgr.y (mgr.24481) 829 : cluster [DBG] pgmap v1159: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 68 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 26 objects/s recovering 2024-02-12T17:11:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:37.539868+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e579: 8 total, 7 up, 7 in 2024-02-12T17:11:37.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:11:37.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:36.543769+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e578: 8 total, 7 up, 7 in 2024-02-12T17:11:37.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:36.929417+0000 mgr.y (mgr.24481) 829 : cluster [DBG] pgmap v1159: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 17 active+clean+remapped, 1 active+recovering+undersized+remapped, 28 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 68 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1544/2280 objects misplaced (67.719%); 0 B/s, 26 objects/s recovering 2024-02-12T17:11:37.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:37.539868+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e579: 8 total, 7 up, 7 in 2024-02-12T17:11:38.083 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:11:38.083 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:11:38.688 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:38.729 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T17:11:38.955 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:38.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:39.061 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:39.160 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:39.236 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:38.930919+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:11:39.236 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:38.931998+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:11:39.236 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:38.930919+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:11:39.237 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:38.931998+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:11:39.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:38.930919+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:11:39.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:38.931998+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T17:11:40.009 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:11:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:38.930539+0000 mgr.y (mgr.24481) 830 : cluster [DBG] pgmap v1161: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 21 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 66 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 11 objects/s recovering 2024-02-12T17:11:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:38.983516+0000 mon.a (mon.0) 1919 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:38.983544+0000 mon.a (mon.0) 1920 : cluster [WRN] Health check update: Degraded data redundancy: 214/2280 objects degraded (9.386%), 31 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:38.997760+0000 mon.a (mon.0) 1921 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T17:11:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:38.997804+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e580: 8 total, 7 up, 7 in 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:39.820406+0000 mon.c (mon.2) 273 : audit [INF] from='client.? v1:172.21.15.33:0/3155112797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:39.820747+0000 mon.a (mon.0) 1923 : audit [INF] from='client.35948 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:38.930539+0000 mgr.y (mgr.24481) 830 : cluster [DBG] pgmap v1161: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 21 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 66 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 11 objects/s recovering 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:38.983516+0000 mon.a (mon.0) 1919 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:38.983544+0000 mon.a (mon.0) 1920 : cluster [WRN] Health check update: Degraded data redundancy: 214/2280 objects degraded (9.386%), 31 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:38.997760+0000 mon.a (mon.0) 1921 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:38.997804+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e580: 8 total, 7 up, 7 in 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:39.820406+0000 mon.c (mon.2) 273 : audit [INF] from='client.? v1:172.21.15.33:0/3155112797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:40.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:39.820747+0000 mon.a (mon.0) 1923 : audit [INF] from='client.35948 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:38.930539+0000 mgr.y (mgr.24481) 830 : cluster [DBG] pgmap v1161: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 21 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 66 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 11 objects/s recovering 2024-02-12T17:11:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:38.983516+0000 mon.a (mon.0) 1919 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:11:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:38.983544+0000 mon.a (mon.0) 1920 : cluster [WRN] Health check update: Degraded data redundancy: 214/2280 objects degraded (9.386%), 31 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:38.997760+0000 mon.a (mon.0) 1921 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T17:11:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:38.997804+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e580: 8 total, 7 up, 7 in 2024-02-12T17:11:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:39.820406+0000 mon.c (mon.2) 273 : audit [INF] from='client.? v1:172.21.15.33:0/3155112797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:39.820747+0000 mon.a (mon.0) 1923 : audit [INF] from='client.35948 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:11:40.945 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:40.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:41.049 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:41.137 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:11:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:40] "GET /metrics HTTP/1.1" 200 176741 "" "Prometheus/2.33.4" 2024-02-12T17:11:41.137 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:39.992715+0000 mon.a (mon.0) 1924 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:41.137 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:40.002345+0000 mon.a (mon.0) 1925 : audit [INF] from='client.35948 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:11:41.137 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:40.002428+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e581: 8 total, 7 up, 7 in 2024-02-12T17:11:41.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:39.992715+0000 mon.a (mon.0) 1924 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:41.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:40.002345+0000 mon.a (mon.0) 1925 : audit [INF] from='client.35948 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:11:41.138 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:40.002428+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e581: 8 total, 7 up, 7 in 2024-02-12T17:11:41.140 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:41.306 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1, 6] out_osds: [4] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-12T17:11:41.306 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:11:41.306 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-02-12T17:11:41.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd in 4 2024-02-12T17:11:41.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:39.992715+0000 mon.a (mon.0) 1924 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:41.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:40.002345+0000 mon.a (mon.0) 1925 : audit [INF] from='client.35948 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:11:41.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:40.002428+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e581: 8 total, 7 up, 7 in 2024-02-12T17:11:41.649 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:41.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:41.755 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:41.784 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:42.042 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:40.931217+0000 mgr.y (mgr.24481) 831 : cluster [DBG] pgmap v1164: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 21 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 66 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 12 objects/s recovering 2024-02-12T17:11:42.042 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:41.006781+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e582: 8 total, 7 up, 7 in 2024-02-12T17:11:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:40.931217+0000 mgr.y (mgr.24481) 831 : cluster [DBG] pgmap v1164: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 21 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 66 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 12 objects/s recovering 2024-02-12T17:11:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:41.006781+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e582: 8 total, 7 up, 7 in 2024-02-12T17:11:42.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:40.931217+0000 mgr.y (mgr.24481) 831 : cluster [DBG] pgmap v1164: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 21 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 66 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 12 objects/s recovering 2024-02-12T17:11:42.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:41.006781+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e582: 8 total, 7 up, 7 in 2024-02-12T17:11:42.741 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:42.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:42.751 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:42.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:42.875 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:11:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:11:42.877 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:42.897 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:42.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:43.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:43.034 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:43.163 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:43.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:42.009037+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e583: 8 total, 7 up, 7 in 2024-02-12T17:11:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:42.932156+0000 mon.c (mon.2) 274 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:11:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:42.933051+0000 mon.a (mon.0) 1929 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:11:43.280 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:43.307 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:42.009037+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e583: 8 total, 7 up, 7 in 2024-02-12T17:11:43.307 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:42.932156+0000 mon.c (mon.2) 274 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:11:43.307 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:42.933051+0000 mon.a (mon.0) 1929 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:11:43.307 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:42.009037+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e583: 8 total, 7 up, 7 in 2024-02-12T17:11:43.307 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:42.932156+0000 mon.c (mon.2) 274 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:11:43.307 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:42.933051+0000 mon.a (mon.0) 1929 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-12T17:11:43.309 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:44.139 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:42.931819+0000 mgr.y (mgr.24481) 832 : cluster [DBG] pgmap v1167: 147 pgs: 1 active+remapped+backfill_toofull, 23 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 64 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 25 objects/s recovering 2024-02-12T17:11:44.139 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:43.037362+0000 mon.a (mon.0) 1930 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T17:11:44.139 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:43.037555+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e584: 8 total, 7 up, 7 in 2024-02-12T17:11:44.140 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:42.931819+0000 mgr.y (mgr.24481) 832 : cluster [DBG] pgmap v1167: 147 pgs: 1 active+remapped+backfill_toofull, 23 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 64 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 25 objects/s recovering 2024-02-12T17:11:44.140 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:43.037362+0000 mon.a (mon.0) 1930 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T17:11:44.140 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:43.037555+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e584: 8 total, 7 up, 7 in 2024-02-12T17:11:44.365 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:44.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:44.472 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:42.931819+0000 mgr.y (mgr.24481) 832 : cluster [DBG] pgmap v1167: 147 pgs: 1 active+remapped+backfill_toofull, 23 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 64 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 25 objects/s recovering 2024-02-12T17:11:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:43.037362+0000 mon.a (mon.0) 1930 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-12T17:11:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:43.037555+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e584: 8 total, 7 up, 7 in 2024-02-12T17:11:44.499 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:44.518 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:44.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:44.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:44.648 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:44.666 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:44.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:44.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:44.774 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:11:44.787 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:11:44.787 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:11:44.787 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:11:44.787 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:11:44.787 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:11:44.788 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:11:44.788 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:11:44.788 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:11:44.788 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:11:44.788 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:11:44.789 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:11:44.789 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:11:44.789 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:11:44.789 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:11:44.789 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:11:44.802 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:45.057 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:11:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:11:45.066 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_3' removed 2024-02-12T17:11:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:44.036675+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e585: 8 total, 7 up, 7 in 2024-02-12T17:11:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:44.378694+0000 mon.b (mon.1) 105 : audit [DBG] from='client.? v1:172.21.15.33:0/3040709673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:44.540209+0000 mon.c (mon.2) 275 : audit [INF] from='client.? v1:172.21.15.33:0/3236843778' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:11:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:44.540600+0000 mon.a (mon.0) 1933 : audit [INF] from='client.35954 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:11:45.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:44.036675+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e585: 8 total, 7 up, 7 in 2024-02-12T17:11:45.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:44.378694+0000 mon.b (mon.1) 105 : audit [DBG] from='client.? v1:172.21.15.33:0/3040709673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:45.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:44.540209+0000 mon.c (mon.2) 275 : audit [INF] from='client.? v1:172.21.15.33:0/3236843778' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:11:45.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:44.540600+0000 mon.a (mon.0) 1933 : audit [INF] from='client.35954 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:11:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:44.036675+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e585: 8 total, 7 up, 7 in 2024-02-12T17:11:45.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:44.378694+0000 mon.b (mon.1) 105 : audit [DBG] from='client.? v1:172.21.15.33:0/3040709673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:45.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:44.540209+0000 mon.c (mon.2) 275 : audit [INF] from='client.? v1:172.21.15.33:0/3236843778' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:11:45.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:44.540600+0000 mon.a (mon.0) 1933 : audit [INF] from='client.35954 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:11:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:11:46.214 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:11:46.256 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:46.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:46.360 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:46.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:44.782615+0000 mgr.y (mgr.24481) 833 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:46.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:44.932750+0000 mgr.y (mgr.24481) 834 : cluster [DBG] pgmap v1170: 147 pgs: 1 active+remapped+backfill_toofull, 23 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 64 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 24 objects/s recovering 2024-02-12T17:11:46.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:45.059320+0000 mon.a (mon.0) 1934 : audit [INF] from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:11:46.385 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:45.059398+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e586: 8 total, 7 up, 7 in 2024-02-12T17:11:46.386 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:44.782615+0000 mgr.y (mgr.24481) 833 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:46.386 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:44.932750+0000 mgr.y (mgr.24481) 834 : cluster [DBG] pgmap v1170: 147 pgs: 1 active+remapped+backfill_toofull, 23 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 64 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 24 objects/s recovering 2024-02-12T17:11:46.386 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:45.059320+0000 mon.a (mon.0) 1934 : audit [INF] from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:11:46.386 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:45.059398+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e586: 8 total, 7 up, 7 in 2024-02-12T17:11:46.388 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:44.782615+0000 mgr.y (mgr.24481) 833 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:44.932750+0000 mgr.y (mgr.24481) 834 : cluster [DBG] pgmap v1170: 147 pgs: 1 active+remapped+backfill_toofull, 23 active+clean+remapped, 1 active+recovering+undersized+remapped, 27 active+undersized, 1 active+recovering+undersized+degraded+remapped, 30 active+undersized+degraded, 64 active+clean; 36 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2280 objects degraded (9.386%); 1525/2280 objects misplaced (66.886%); 0 B/s, 24 objects/s recovering 2024-02-12T17:11:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:45.059320+0000 mon.a (mon.0) 1934 : audit [INF] from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:11:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:45.059398+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e586: 8 total, 7 up, 7 in 2024-02-12T17:11:46.705 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:47.162 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:46.059657+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e587: 8 total, 7 up, 7 in 2024-02-12T17:11:47.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:46.576710+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e588: 8 total, 7 up, 7 in 2024-02-12T17:11:47.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:46.059657+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e587: 8 total, 7 up, 7 in 2024-02-12T17:11:47.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:46.576710+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e588: 8 total, 7 up, 7 in 2024-02-12T17:11:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:46.059657+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e587: 8 total, 7 up, 7 in 2024-02-12T17:11:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:46.576710+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e588: 8 total, 7 up, 7 in 2024-02-12T17:11:48.215 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:11:48.239 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:48.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:48.344 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:48.372 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:46.933649+0000 mgr.y (mgr.24481) 835 : cluster [DBG] pgmap v1174: 110 pgs: 1 activating, 1 active+remapped+backfill_toofull, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:48.373 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:47.598534+0000 mon.a (mon.0) 1938 : cluster [WRN] Health check update: Degraded data redundancy: 137/576 objects degraded (23.785%), 29 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:48.373 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:46.933649+0000 mgr.y (mgr.24481) 835 : cluster [DBG] pgmap v1174: 110 pgs: 1 activating, 1 active+remapped+backfill_toofull, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:48.373 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:47.598534+0000 mon.a (mon.0) 1938 : cluster [WRN] Health check update: Degraded data redundancy: 137/576 objects degraded (23.785%), 29 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:48.375 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:48.476 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:48.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:46.933649+0000 mgr.y (mgr.24481) 835 : cluster [DBG] pgmap v1174: 110 pgs: 1 activating, 1 active+remapped+backfill_toofull, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:48.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:47.598534+0000 mon.a (mon.0) 1938 : cluster [WRN] Health check update: Degraded data redundancy: 137/576 objects degraded (23.785%), 29 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:48.491 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:11:48.491 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T17:11:48.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T17:11:48.491 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:11:48.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.491 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:2079 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94508044027008 gen 4] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:49.519705+0000", 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 478.96721974399998, 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.07566921, 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 2079 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.492 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519705+0000", 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519705+0000", 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519703+0000", 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.149e-06 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519712+0000", 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5268e-05 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519727+0000", 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4039999999999998e-06 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519735+0000", 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082878199999999996 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.528022+0000", 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2620000000000001e-06 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.528030+0000", 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.023413739 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.551443+0000", 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013260600000000001 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for missing object", 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.551576+0000", 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030046614999999999 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.581623+0000", 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102934 2024-02-12T17:11:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.581726+0000", 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080166199999999997 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.582527+0000", 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4708000000000005e-05 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.582592+0000", 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012782679999999999 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.595375+0000", 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.07566921 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:2082 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94508044032768 gen 3] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:49.519751+0000", 2024-02-12T17:11:48.498 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 478.96717399599999, 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1089269939999999, 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 2082 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519751+0000", 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519751+0000", 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519750+0000", 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5159999999999999e-06 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519754+0000", 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1280000000000002e-06 2024-02-12T17:11:48.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519761+0000", 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9090000000000002e-06 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519765+0000", 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012885931999999999 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.532651+0000", 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8889999999999996e-06 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.532658+0000", 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.02618929 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.558847+0000", 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117404 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for missing object", 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.558964+0000", 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055259906999999997 2024-02-12T17:11:48.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.614224+0000", 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3797999999999999e-05 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.614318+0000", 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011640520000000001 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.615482+0000", 2024-02-12T17:11:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9492000000000001e-05 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.615522+0000", 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013156467 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.628678+0000", 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1089269939999999 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:2084 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94508044035072 gen 5] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:49.519777+0000", 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 478.967148533, 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1847904810000001, 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 2084 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519777+0000", 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519777+0000", 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519776+0000", 2024-02-12T17:11:48.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0199999999999999e-06 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519779+0000", 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.916e-06 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519784+0000", 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5540000000000002e-06 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.519788+0000", 2024-02-12T17:11:48.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013420333 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.533208+0000", 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7599999999999997e-06 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.533215+0000", 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0242792709999999 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.557494+0000", 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000138823 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for missing object", 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.557633+0000", 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.057228728 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.614862+0000", 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8821999999999997e-05 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.614960+0000", 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.086605242999999998 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.701566+0000", 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110369 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.701676+0000", 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028911449999999999 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.704567+0000", 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1847904810000001 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:183803 9.4 9.4a8ba9e4 (undecoded) ondisk+retry+write+known_if_redirected e347)", 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:49.534667+0000", 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 478.95225768, 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98432076599999996, 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:11:48.511 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 183803 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.534667+0000", 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.534667+0000", 2024-02-12T17:11:48.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.534665+0000", 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4470000000000003e-06 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.534671+0000", 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3770000000000002e-06 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.534681+0000", 2024-02-12T17:11:48.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1930000000000004e-06 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.534688+0000", 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5102e-05 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.534763+0000", 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6789999999999999e-06 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.534770+0000", 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98420558000000002 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.518975+0000", 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2805e-05 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.518988+0000", 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98432076599999996 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:183805 9.4 9.29a08024 (undecoded) ondisk+write+known_if_redirected e347)", 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:49.554446+0000", 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 478.93247889000003, 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96462812600000003, 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 183805 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.554446+0000", 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.554446+0000", 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.554444+0000", 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4649999999999994e-06 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.554452+0000", 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0499999999999999e-05 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.554463+0000", 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4669999999999998e-06 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.554470+0000", 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00057862899999999999 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.555049+0000", 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7879999999999997e-06 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:49.555054+0000", 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96401114499999996 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.519065+0000", 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.662e-06 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.519074+0000", 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96462812600000003 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:2302 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94508044038528] snapc 0=[] ondisk+write+known_if_redirected e423)", 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:37.446470+0000", 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 371.04045488600002, 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.95734354300000002, 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.520 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 2302 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446470+0000", 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446470+0000", 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446468+0000", 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7400000000000001e-06 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446474+0000", 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8810000000000002e-06 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446483+0000", 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3199999999999999e-06 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446488+0000", 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118849 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446607+0000", 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5308e-05 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446623+0000", 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.95696944500000003 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.403592+0000", 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014158700000000001 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.403734+0000", 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0162999999999998e-05 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.403814+0000", 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.95734354300000002 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:2310 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94508044038528 gen 1] snapc 0=[] ondisk+write+known_if_redirected e429)", 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:37.446506+0000", 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 371.04041925199999, 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.961184178, 2024-02-12T17:11:48.525 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 2310 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446506+0000", 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446506+0000", 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446505+0000", 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3019999999999999e-06 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446508+0000", 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3789999999999997e-06 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446514+0000", 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0879999999999997e-06 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446518+0000", 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023482 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446753+0000", 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9739999999999999e-06 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446759+0000", 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.95708279500000004 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.403842+0000", 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1978000000000003e-05 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.403893+0000", 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037965749999999999 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.407690+0000", 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.961184178 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.24355.0:2311 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94508044038528 gen 2] snapc 0=[] ondisk+write+known_if_redirected e429)", 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:05:37.446641+0000", 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 371.04028441000003, 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96112166099999996, 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.24355", 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 2311 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446641+0000", 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446641+0000", 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446639+0000", 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4750000000000001e-06 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446645+0000", 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1969999999999996e-06 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446653+0000", 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8440000000000003e-06 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446659+0000", 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011244200000000001 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446771+0000", 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3760000000000001e-06 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:37.446776+0000", 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.95738732400000004 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.404163+0000", 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5683000000000001e-05 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.404229+0000", 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035335560000000002 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:05:38.407762+0000", 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96112166099999996 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121716 10.d 10:bc90c8e2:::benchmark_data_smithi033_160181_object121715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e479)", 2024-02-12T17:11:48.535 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.012591+0000", 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 237.474333707, 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.230738953, 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121716 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012591+0000", 2024-02-12T17:11:48.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012591+0000", 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012590+0000", 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3519e-05 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012613+0000", 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6739999999999997e-06 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012621+0000", 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1070000000000004e-06 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012626+0000", 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2017000000000002e-05 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012658+0000", 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010315300000000001 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012761+0000", 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000140233 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:11:48.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012901+0000", 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011569759000000001 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.024471+0000", 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018617400999999999 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.043089+0000", 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.200149407 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.243238+0000", 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1875000000000001e-05 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.243280+0000", 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0528000000000001e-05 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.243330+0000", 2024-02-12T17:11:48.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.230738953 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121717 10.d 10:b4b57995:::benchmark_data_smithi033_160181_object121716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e479)", 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.012635+0000", 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 237.47428975, 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.23085219000000001, 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:11:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121717 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012635+0000", 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012635+0000", 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012635+0000", 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9944000000000001e-05 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012655+0000", 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8680000000000004e-06 2024-02-12T17:11:48.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012661+0000", 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1400000000000002e-06 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.012665+0000", 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021955759999999999 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.014860+0000", 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120882 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.014981+0000", 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014941499999999999 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.015131+0000", 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093958930000000006 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.024526+0000", 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018588624000000002 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.043115+0000", 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.200282232 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.243397+0000", 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2194999999999997e-05 2024-02-12T17:11:48.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.243449+0000", 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8090000000000003e-05 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.243488+0000", 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.23085219000000001 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121725 10.2 10:43ae1bd9:::benchmark_data_smithi033_160181_object121724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.059953+0000", 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 237.42697210099999, 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.016973425, 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121725 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.059953+0000", 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.059953+0000", 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.059950+0000", 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4181999999999999e-05 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.059985+0000", 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1177e-05 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.059996+0000", 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4339999999999997e-06 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060002+0000", 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.090021806999999995 2024-02-12T17:11:48.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.150024+0000", 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2578000000000001e-05 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.150037+0000", 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90859053700000003 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.058627+0000", 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000155477 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.058783+0000", 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032459600000000002 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.059107+0000", 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088344840000000001 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.067942+0000", 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001451403 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.069393+0000", 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074479419999999999 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.076841+0000", 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5083999999999997e-05 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.076886+0000", 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0357e-05 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.076926+0000", 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.016973425 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121729 10.12 10:4d9feca0:::benchmark_data_smithi033_160181_object121728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.193992+0000", 2024-02-12T17:11:48.555 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 237.29293353, 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88658243699999995, 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121729 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193992+0000", 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193992+0000", 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193989+0000", 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1131000000000001e-05 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.194010+0000", 2024-02-12T17:11:48.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.22e-05 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.194023+0000", 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9310000000000002e-06 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.194030+0000", 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8102e-05 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.194088+0000", 2024-02-12T17:11:48.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1187e-05 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.194099+0000", 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88205338899999997 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.076152+0000", 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000138641 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.076291+0000", 2024-02-12T17:11:48.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000224952 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,7", 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.076516+0000", 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0027352769999999999 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.079251+0000", 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00109056 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.080342+0000", 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000122967 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.080465+0000", 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3479999999999995e-05 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.080528+0000", 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5924999999999998e-05 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.080574+0000", 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88658243699999995 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123093 10.18 10.f24edf18 (undecoded) ondisk+retry+write+known_if_redirected e482)", 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:53.083404+0000", 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 235.40352123700001, 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98685763199999998, 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.562 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123093 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.083404+0000", 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.083404+0000", 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:11:48.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.083403+0000", 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4655999999999999e-05 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.083428+0000", 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5549999999999997e-06 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.083435+0000", 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6969999999999998e-06 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.083441+0000", 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1378999999999997e-05 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.083492+0000", 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9789999999999999e-06 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.083501+0000", 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.986748976 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070250+0000", 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1141000000000001e-05 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070262+0000", 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98685763199999998 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123110 10.19 10.15058639 (undecoded) ondisk+write+known_if_redirected e482)", 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:53.090991+0000", 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 235.39593378399999, 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.97772742999999995, 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123110 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.090991+0000", 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.090991+0000", 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.090989+0000", 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1894000000000001e-05 2024-02-12T17:11:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.091011+0000", 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9039999999999998e-06 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.091020+0000", 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3889999999999998e-06 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.091025+0000", 2024-02-12T17:11:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9549000000000003e-05 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.091074+0000", 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6369999999999999e-06 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:53.091082+0000", 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.97762597500000004 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.068708+0000", 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.098e-05 2024-02-12T17:11:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.068719+0000", 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.97772742999999995 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123169 10.17 10:ee1b3981:::benchmark_data_smithi033_160181_object123168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e483)", 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:54.110044+0000", 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 234.37688143599999, 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0217698900000001, 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123169 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.110044+0000", 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.110044+0000", 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.110042+0000", 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1782000000000001e-05 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.110084+0000", 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0917000000000001e-05 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.110095+0000", 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1789999999999999e-06 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.110100+0000", 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0688e-05 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.110151+0000", 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1790000000000008e-06 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.110159+0000", 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99183505500000002 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.101994+0000", 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015641 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.102150+0000", 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029521199999999999 2024-02-12T17:11:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5,7", 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.102445+0000", 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001440048 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.103885+0000", 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014135619999999999 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.105299+0000", 2024-02-12T17:11:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00078099899999999999 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.106080+0000", 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025544549999999999 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.131625+0000", 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4139000000000004e-05 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.131689+0000", 2024-02-12T17:11:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000124929 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.131814+0000", 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0217698900000001 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174492 10.17 10:eba84375:::benchmark_data_smithi033_160181_object51320:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.553546+0000", 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 147.933379016, 2024-02-12T17:11:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.077153547, 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174492 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.553546+0000", 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.553546+0000", 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.553544+0000", 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7549999999999997e-06 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.553551+0000", 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4520000000000003e-06 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.553561+0000", 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2020000000000003e-06 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.553566+0000", 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00270725 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.556273+0000", 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014530899999999999 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.556418+0000", 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000373674 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.556792+0000", 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010358993 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.567151+0000", 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00094039199999999996 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.568091+0000", 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024706293000000001 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.592798+0000", 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4609999999999996e-06 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.592804+0000", 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0378351720000001 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.630639+0000", 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0398e-05 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.630700+0000", 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.077153547 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174495 10.17 10:e832b6c7:::benchmark_data_smithi033_160181_object51323:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:11:48.584 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.558511+0000", 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 147.928414403, 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.099763354, 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174495 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558511+0000", 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558511+0000", 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558508+0000", 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4059999999999994e-06 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558518+0000", 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2097e-05 2024-02-12T17:11:48.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558530+0000", 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8669999999999996e-06 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558537+0000", 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064254500000000001 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.559179+0000", 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014760399999999999 2024-02-12T17:11:48.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.559327+0000", 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00045767900000000001 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.559784+0000", 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073777180000000001 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.567162+0000", 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039582649999999999 2024-02-12T17:11:48.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.571120+0000", 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021696961000000001 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.592817+0000", 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4039999999999997e-06 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.592822+0000", 2024-02-12T17:11:48.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.037906913 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.630729+0000", 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000107252 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for missing object", 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.630836+0000", 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027366266 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.658202+0000", 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1898999999999997e-05 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.658274+0000", 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.099763354 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174497 10.f 10:f6e0f1bb:::benchmark_data_smithi033_160181_object51325:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.558553+0000", 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 147.928371946, 2024-02-12T17:11:48.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.072822025, 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174497 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558553+0000", 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558553+0000", 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558553+0000", 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3129999999999999e-06 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558556+0000", 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5350000000000004e-06 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558561+0000", 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1459999999999998e-06 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.558566+0000", 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026891039999999999 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.561255+0000", 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039221680000000002 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.565177+0000", 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000408916 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,2,5", 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.565586+0000", 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016524210000000001 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.567238+0000", 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039257609999999998 2024-02-12T17:11:48.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.571164+0000", 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019744688 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.590909+0000", 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8009999999999998e-06 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.590915+0000", 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0226611269999999 2024-02-12T17:11:48.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.613577+0000", 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117324 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for missing object", 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.613694+0000", 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017633172999999999 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631327+0000", 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8201000000000001e-05 2024-02-12T17:11:48.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631375+0000", 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.072822025 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174511 10.f 10:f05aa85a:::benchmark_data_smithi033_160181_object51339:head [delete] snapc 0=[] ondisk+write+known_if_redirected e513)", 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.638711+0000", 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 147.84821434899999, 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99238542699999999, 2024-02-12T17:11:48.598 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174511 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.638711+0000", 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.638711+0000", 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.638710+0000", 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5640000000000001e-06 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.638714+0000", 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5459999999999999e-06 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.638719+0000", 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8590000000000004e-06 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.638724+0000", 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012293580000000001 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.639953+0000", 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2289999999999997e-06 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.639961+0000", 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.97375699699999996 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.613718+0000", 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000108687 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.613826+0000", 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00030087700000000001 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,5,6", 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.614127+0000", 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037817609999999998 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.617909+0000", 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029044100000000001 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.618199+0000", 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022144809999999999 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.620414+0000", 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010523088999999999 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.630937+0000", 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0607e-05 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.630968+0000", 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012864300000000001 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631096+0000", 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99238542699999999 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:246340 10.17 10:ee1b3981:::benchmark_data_smithi033_160181_object123168:head [delete] snapc 0=[] ondisk+write+known_if_redirected e570)", 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:11:21.249376+0000", 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.237548747000002, 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98791709900000002, 2024-02-12T17:11:48.605 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 246340 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:21.249376+0000", 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:11:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:21.249376+0000", 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:21.249374+0000", 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6479999999999992e-06 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:21.249383+0000", 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4791e-05 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:11:48.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:21.249397+0000", 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7569999999999998e-06 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:21.249404+0000", 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0319000000000003e-05 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:21.249464+0000", 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1133e-05 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:11:48.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:21.249476+0000", 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96334599600000004 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:22.212822+0000", 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012684 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:22.212948+0000", 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014475999999999999 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:22.213093+0000", 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018708061000000002 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:22.231801+0000", 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052751559999999996 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:22.237076+0000", 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8151000000000001e-05 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:22.237114+0000", 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3584000000000001e-05 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:22.237168+0000", 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125411 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:22.237293+0000", 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98791709900000002 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:48.612 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:48.612 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:11:48.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:48.640 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:48.789 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:48.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:48.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:48.922 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:48.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:49.046 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:49.075 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:49.095 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:49.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:49.198 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:49.228 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:49.257 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:49.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:49.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:49.391 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:49.412 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:49.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:49.515 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:49.546 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:49.600 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:49.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:49.614 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T17:11:49.615 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T17:11:49.615 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-02-12T17:11:49.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-02-12T17:11:49.635 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:49.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:49.731 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:50.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T17:11:50.053 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:50.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:48.934586+0000 mgr.y (mgr.24481) 836 : cluster [DBG] pgmap v1175: 110 pgs: 1 active+remapped+backfill_toofull, 1 activating, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 837 B/s rd, 0 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:50.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:48.934586+0000 mgr.y (mgr.24481) 836 : cluster [DBG] pgmap v1175: 110 pgs: 1 active+remapped+backfill_toofull, 1 activating, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 837 B/s rd, 0 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:50.327 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:11:50.489 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:48.934586+0000 mgr.y (mgr.24481) 836 : cluster [DBG] pgmap v1175: 110 pgs: 1 active+remapped+backfill_toofull, 1 activating, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 837 B/s rd, 0 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:51.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:11:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:50] "GET /metrics HTTP/1.1" 200 174330 "" "Prometheus/2.33.4" 2024-02-12T17:11:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:11:50.830+0000 7fd96a9de700 -1 mon.a@0(leader).osd e589 definitely_dead 0 2024-02-12T17:11:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:marked in osd.4. 2024-02-12T17:11:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:49.490021+0000 osd.7 (osd.7) 268 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:49.490104+0000 osd.7 (osd.7) 269 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:49.922479+0000 osd.6 (osd.6) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:49.922562+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:50.079042+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e589: 8 total, 7 up, 7 in 2024-02-12T17:11:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:50.830656+0000 mon.a (mon.0) 1940 : audit [DBG] from='client.? v1:172.21.15.33:0/1611837049' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:11:51.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:49.490021+0000 osd.7 (osd.7) 268 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:49.490104+0000 osd.7 (osd.7) 269 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:49.922479+0000 osd.6 (osd.6) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:49.922562+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:50.079042+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e589: 8 total, 7 up, 7 in 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:50.830656+0000 mon.a (mon.0) 1940 : audit [DBG] from='client.? v1:172.21.15.33:0/1611837049' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:49.490021+0000 osd.7 (osd.7) 268 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:49.490104+0000 osd.7 (osd.7) 269 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:49.922479+0000 osd.6 (osd.6) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:49.922562+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:50.079042+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e589: 8 total, 7 up, 7 in 2024-02-12T17:11:51.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:50.830656+0000 mon.a (mon.0) 1940 : audit [DBG] from='client.? v1:172.21.15.33:0/1611837049' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:11:52.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:50.935177+0000 mgr.y (mgr.24481) 837 : cluster [DBG] pgmap v1177: 110 pgs: 1 active+remapped+backfill_toofull, 1 activating, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 871 B/s rd, 0 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:52.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:51.087121+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.33:0/1611837049' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-12T17:11:52.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:51.087170+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e590: 8 total, 7 up, 8 in 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:51.353009+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:51.353048+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:51.578726+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e591: 8 total, 7 up, 8 in 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:50.935177+0000 mgr.y (mgr.24481) 837 : cluster [DBG] pgmap v1177: 110 pgs: 1 active+remapped+backfill_toofull, 1 activating, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 871 B/s rd, 0 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:51.087121+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.33:0/1611837049' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:51.087170+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e590: 8 total, 7 up, 8 in 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:51.353009+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:51.353048+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:51.578726+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e591: 8 total, 7 up, 8 in 2024-02-12T17:11:52.293 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:11:52.293 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:11:52.293 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-02-12T17:11:52.293 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:11:52.315 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:50.935177+0000 mgr.y (mgr.24481) 837 : cluster [DBG] pgmap v1177: 110 pgs: 1 active+remapped+backfill_toofull, 1 activating, 7 active+clean+remapped, 26 active+undersized, 29 active+undersized+degraded, 46 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 871 B/s rd, 0 op/s; 137/576 objects degraded (23.785%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:52.315 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:51.087121+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.33:0/1611837049' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-12T17:11:52.315 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:51.087170+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e590: 8 total, 7 up, 8 in 2024-02-12T17:11:52.315 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:51.353009+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:52.316 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:51.353048+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:52.316 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:51.578726+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e591: 8 total, 7 up, 8 in 2024-02-12T17:11:52.416 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-02-12T17:11:52.659 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:11:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:11:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:11:52.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T17:11:53.137 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:51.582900+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.5 starting backfill to osd.4 from (0'0,0'0] MAX to 410'15 2024-02-12T17:11:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:51.586122+0000 osd.2 (osd.2) 218 : cluster [INF] 5.5 continuing backfill to osd.6 from (406'13,410'15] MIN to 410'15 2024-02-12T17:11:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:52.592829+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e592: 8 total, 7 up, 8 in 2024-02-12T17:11:53.270 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:11:53.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:51.582900+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.5 starting backfill to osd.4 from (0'0,0'0] MAX to 410'15 2024-02-12T17:11:53.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:51.586122+0000 osd.2 (osd.2) 218 : cluster [INF] 5.5 continuing backfill to osd.6 from (406'13,410'15] MIN to 410'15 2024-02-12T17:11:53.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:52.592829+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e592: 8 total, 7 up, 8 in 2024-02-12T17:11:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:51.582900+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.5 starting backfill to osd.4 from (0'0,0'0] MAX to 410'15 2024-02-12T17:11:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:51.586122+0000 osd.2 (osd.2) 218 : cluster [INF] 5.5 continuing backfill to osd.6 from (406'13,410'15] MIN to 410'15 2024-02-12T17:11:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:52.592829+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e592: 8 total, 7 up, 8 in 2024-02-12T17:11:53.721 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:53.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:53.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:53.854 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:54.102 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:54.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:54.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:54.232 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:54.248 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:52.590739+0000 osd.5 (osd.5) 274 : cluster [DBG] 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'4 2024-02-12T17:11:54.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:52.592058+0000 osd.3 (osd.3) 201 : cluster [DBG] 5.0 starting backfill to osd.4 from (0'0,0'0] MAX to 534'23 2024-02-12T17:11:54.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:52.592611+0000 osd.5 (osd.5) 275 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 435'21 2024-02-12T17:11:54.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:52.593061+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 571'25 2024-02-12T17:11:54.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:52.594763+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 starting backfill to osd.4 from (0'0,0'0] MAX to 430'18 2024-02-12T17:11:54.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:52.598335+0000 osd.7 (osd.7) 271 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 534'26 2024-02-12T17:11:54.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:52.935964+0000 mgr.y (mgr.24481) 838 : cluster [DBG] pgmap v1181: 110 pgs: 1 remapped+peering, 11 peering, 1 active+remapped+backfill_toofull, 6 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 43 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 112/576 objects degraded (19.444%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:54.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:53.587284+0000 mon.a (mon.0) 1945 : cluster [WRN] Health check update: Degraded data redundancy: 112/576 objects degraded (19.444%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:54.249 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:53.593204+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e593: 8 total, 7 up, 8 in 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:52.590739+0000 osd.5 (osd.5) 274 : cluster [DBG] 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'4 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:52.592058+0000 osd.3 (osd.3) 201 : cluster [DBG] 5.0 starting backfill to osd.4 from (0'0,0'0] MAX to 534'23 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:52.592611+0000 osd.5 (osd.5) 275 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 435'21 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:52.593061+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 571'25 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:52.594763+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 starting backfill to osd.4 from (0'0,0'0] MAX to 430'18 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:52.598335+0000 osd.7 (osd.7) 271 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 534'26 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:52.935964+0000 mgr.y (mgr.24481) 838 : cluster [DBG] pgmap v1181: 110 pgs: 1 remapped+peering, 11 peering, 1 active+remapped+backfill_toofull, 6 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 43 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 112/576 objects degraded (19.444%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:53.587284+0000 mon.a (mon.0) 1945 : cluster [WRN] Health check update: Degraded data redundancy: 112/576 objects degraded (19.444%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:54.250 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:53.593204+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e593: 8 total, 7 up, 8 in 2024-02-12T17:11:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:52.590739+0000 osd.5 (osd.5) 274 : cluster [DBG] 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'4 2024-02-12T17:11:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:52.592058+0000 osd.3 (osd.3) 201 : cluster [DBG] 5.0 starting backfill to osd.4 from (0'0,0'0] MAX to 534'23 2024-02-12T17:11:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:52.592611+0000 osd.5 (osd.5) 275 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 435'21 2024-02-12T17:11:54.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:52.593061+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 571'25 2024-02-12T17:11:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:52.594763+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 starting backfill to osd.4 from (0'0,0'0] MAX to 430'18 2024-02-12T17:11:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:52.598335+0000 osd.7 (osd.7) 271 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 534'26 2024-02-12T17:11:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:52.935964+0000 mgr.y (mgr.24481) 838 : cluster [DBG] pgmap v1181: 110 pgs: 1 remapped+peering, 11 peering, 1 active+remapped+backfill_toofull, 6 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 43 active+clean; 5.7 KiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 112/576 objects degraded (19.444%); 1/576 objects misplaced (0.174%) 2024-02-12T17:11:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:53.587284+0000 mon.a (mon.0) 1945 : cluster [WRN] Health check update: Degraded data redundancy: 112/576 objects degraded (19.444%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-12T17:11:54.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:53.593204+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e593: 8 total, 7 up, 8 in 2024-02-12T17:11:54.659 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:54.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:54.697 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:11:54.728 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:11:54.745 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:11:54.762 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:11:54.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:54.780 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:11:54.795 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:54.799 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:11:54.824 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:11:54.824 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:11:54.824 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:11:54.825 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:11:54.825 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:11:54.825 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:11:54.825 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:11:54.825 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:11:54.825 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:11:54.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:11:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:11:55.158 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:11:55.159 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:11:55.248 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:55.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:55.354 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:55.384 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:55.457 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:55.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:55.562 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:11:55.590 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:55.610 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:11:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:54.538181+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:54.538228+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:54.595844+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e594: 8 total, 7 up, 8 in 2024-02-12T17:11:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:54.671054+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.33:0/4099695037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:11:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:54.671467+0000 mon.a (mon.0) 1948 : audit [INF] from='client.35972 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:11:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:54.676931+0000 mon.a (mon.0) 1949 : audit [DBG] from='client.? v1:172.21.15.33:0/2991999741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:54.792580+0000 mgr.y (mgr.24481) 839 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:54.936491+0000 mgr.y (mgr.24481) 840 : cluster [DBG] pgmap v1184: 110 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 11 peering, 14 active+clean+remapped, 16 active+undersized, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 16 active+undersized+degraded, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+degraded+remapped, 42 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 799 B/s rd, 0 op/s; 101/576 objects degraded (17.535%); 23/576 objects misplaced (3.993%); 106 B/s, 8 objects/s recovering 2024-02-12T17:11:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:11:55.746 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:11:55.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T17:11:55.947 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:11:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:55.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:11:56.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:54.538181+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:56.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:54.538228+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:54.595844+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e594: 8 total, 7 up, 8 in 2024-02-12T17:11:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:54.671054+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.33:0/4099695037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:11:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:54.671467+0000 mon.a (mon.0) 1948 : audit [INF] from='client.35972 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:11:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:54.676931+0000 mon.a (mon.0) 1949 : audit [DBG] from='client.? v1:172.21.15.33:0/2991999741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:54.792580+0000 mgr.y (mgr.24481) 839 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:54.936491+0000 mgr.y (mgr.24481) 840 : cluster [DBG] pgmap v1184: 110 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 11 peering, 14 active+clean+remapped, 16 active+undersized, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 16 active+undersized+degraded, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+degraded+remapped, 42 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 799 B/s rd, 0 op/s; 101/576 objects degraded (17.535%); 23/576 objects misplaced (3.993%); 106 B/s, 8 objects/s recovering 2024-02-12T17:11:56.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:54.538181+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:11:56.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:54.538228+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:11:56.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:54.595844+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e594: 8 total, 7 up, 8 in 2024-02-12T17:11:56.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:54.671054+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.33:0/4099695037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:11:56.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:54.671467+0000 mon.a (mon.0) 1948 : audit [INF] from='client.35972 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:11:56.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:54.676931+0000 mon.a (mon.0) 1949 : audit [DBG] from='client.? v1:172.21.15.33:0/2991999741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:11:56.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:54.792580+0000 mgr.y (mgr.24481) 839 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:11:56.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:54.936491+0000 mgr.y (mgr.24481) 840 : cluster [DBG] pgmap v1184: 110 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 11 peering, 14 active+clean+remapped, 16 active+undersized, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 16 active+undersized+degraded, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+degraded+remapped, 42 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 799 B/s rd, 0 op/s; 101/576 objects degraded (17.535%); 23/576 objects misplaced (3.993%); 106 B/s, 8 objects/s recovering 2024-02-12T17:11:56.051 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:11:56.082 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:56.207 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:11:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:55.592609+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:55.601995+0000 mon.a (mon.0) 1951 : audit [INF] from='client.35972 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:11:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:55.602067+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e595: 8 total, 7 up, 8 in 2024-02-12T17:11:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:55.825684+0000 osd.5 (osd.5) 277 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:11:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:55.829150+0000 osd.5 (osd.5) 278 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:11:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:56.580582+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e596: 8 total, 7 up, 8 in 2024-02-12T17:11:56.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:55.592609+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:56.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:55.601995+0000 mon.a (mon.0) 1951 : audit [INF] from='client.35972 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:11:56.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:55.602067+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e595: 8 total, 7 up, 8 in 2024-02-12T17:11:56.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:55.825684+0000 osd.5 (osd.5) 277 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:11:56.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:55.829150+0000 osd.5 (osd.5) 278 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:11:56.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:56.580582+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e596: 8 total, 7 up, 8 in 2024-02-12T17:11:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:55.592609+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:11:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:55.601995+0000 mon.a (mon.0) 1951 : audit [INF] from='client.35972 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:11:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:55.602067+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e595: 8 total, 7 up, 8 in 2024-02-12T17:11:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:55.825684+0000 osd.5 (osd.5) 277 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:11:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:55.829150+0000 osd.5 (osd.5) 278 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:11:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:56.580582+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e596: 8 total, 7 up, 8 in 2024-02-12T17:11:56.831 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:11:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:56.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:11:56.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:56.964 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:56.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:57.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:57.127 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:57.417 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-12T17:11:57.417 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:11:57.417 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-12T17:11:57.417 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T17:11:57.597 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:57.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:57.705 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:11:57.737 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:57.743 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:11:57.747 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:11:57.766 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:11:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:57.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:11:57.811 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:11:57.811 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:11:57.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:57.902 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:58.000 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_4' created 2024-02-12T17:11:58.027 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:58.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:58.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:58.156 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:56.937157+0000 mgr.y (mgr.24481) 841 : cluster [DBG] pgmap v1187: 110 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 21 active+clean+remapped, 16 active+undersized, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 16 active+undersized+degraded, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+degraded+remapped, 47 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 101/576 objects degraded (17.535%); 43/576 objects misplaced (7.465%); 513 B/s, 15 objects/s recovering 2024-02-12T17:11:58.156 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:56.937546+0000 mon.c (mon.2) 277 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:11:58.156 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:56.938451+0000 mon.a (mon.0) 1954 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:11:58.158 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:58.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:56.937157+0000 mgr.y (mgr.24481) 841 : cluster [DBG] pgmap v1187: 110 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 21 active+clean+remapped, 16 active+undersized, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 16 active+undersized+degraded, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+degraded+remapped, 47 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 101/576 objects degraded (17.535%); 43/576 objects misplaced (7.465%); 513 B/s, 15 objects/s recovering 2024-02-12T17:11:58.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:56.937546+0000 mon.c (mon.2) 277 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:11:58.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:56.938451+0000 mon.a (mon.0) 1954 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:11:58.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:56.937157+0000 mgr.y (mgr.24481) 841 : cluster [DBG] pgmap v1187: 110 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 21 active+clean+remapped, 16 active+undersized, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 16 active+undersized+degraded, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+degraded+remapped, 47 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 101/576 objects degraded (17.535%); 43/576 objects misplaced (7.465%); 513 B/s, 15 objects/s recovering 2024-02-12T17:11:58.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:56.937546+0000 mon.c (mon.2) 277 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:11:58.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:56.938451+0000 mon.a (mon.0) 1954 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-12T17:11:58.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T17:11:58.508 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:58.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:58.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:11:58.639 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:58.657 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:11:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:58.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:11:58.761 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:58.777 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:11:58.790 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:58.814 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:11:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:58.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:11:58.917 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:11:58.945 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:59.023 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:11:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:59.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:11:59.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:57.534769+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? v1:172.21.15.33:0/1078526643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-12T17:11:59.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:57.813572+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:11:59.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:57.816531+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:11:59.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:57.991728+0000 mon.a (mon.0) 1956 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T17:11:59.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:57.991816+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.33:0/1078526643' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-12T17:11:59.053 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:57.991870+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e597: 8 total, 7 up, 8 in 2024-02-12T17:11:59.099 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:57.534769+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? v1:172.21.15.33:0/1078526643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-12T17:11:59.100 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:57.813572+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:11:59.100 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:57.816531+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:11:59.100 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:57.991728+0000 mon.a (mon.0) 1956 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T17:11:59.100 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:57.991816+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.33:0/1078526643' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-12T17:11:59.100 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:57.991870+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e597: 8 total, 7 up, 8 in 2024-02-12T17:11:59.100 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:57.534769+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? v1:172.21.15.33:0/1078526643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-12T17:11:59.100 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:57.813572+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:11:59.101 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:57.816531+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:11:59.101 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:57.991728+0000 mon.a (mon.0) 1956 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-12T17:11:59.101 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:57.991816+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.33:0/1078526643' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-12T17:11:59.101 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:57.991870+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e597: 8 total, 7 up, 8 in 2024-02-12T17:11:59.138 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:11:59.169 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:11:59.626 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:11:59.818 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:11:59.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:11:59.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:11:59.861 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:11:59.861 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:11:59.861 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 598, 2024-02-12T17:11:59.861 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T17:11:59.861 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2024-02-12T16:43:50.282468+0000", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2024-02-12T17:11:58.988441+0000", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2024-02-12T17:10:02.814427+0000", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2024-02-12T17:11:51.074104+0000", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5804032, 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "nodeep-scrub", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 18, 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T17:11:59.862 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 11, 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 8, 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "jewel", 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "pacific", 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "device_health_metrics", 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:47:33.056945+0000", 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:11:59.863 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2024-02-12T17:11:59.864 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "71", 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:11:59.865 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:11:59.866 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:11:59.867 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_devicehealth": {} 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".rgw.root", 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:37.647343+0000", 2024-02-12T17:11:59.868 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 30, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:11:59.869 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 29, 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "2.1e", 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 586, 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 585, 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 585, 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "597", 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "597", 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "597", 2024-02-12T17:11:59.870 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:11:59.871 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:11:59.872 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "iscsi", 2024-02-12T17:11:59.873 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:38.076200+0000", 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 3, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 3, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 3, 2024-02-12T17:11:59.874 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 3, 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 3, 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "527", 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:11:59.875 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:11:59.876 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:11:59.877 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 4, 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.log", 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:39.589565+0000", 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:11:59.878 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:11:59.879 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "61", 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:11:59.880 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:11:59.881 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:11:59.882 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 5, 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.control", 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:41.623856+0000", 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:11:59.883 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "5.10", 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 429, 2024-02-12T17:11:59.884 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 428, 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 428, 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "380'16" 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "433", 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "429", 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "431", 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:11:59.885 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:11:59.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:11:59.887 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 6, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.meta", 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:43.693806+0000", 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:11:59.888 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "6.16", 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 396, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 395, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 395, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "396", 2024-02-12T17:11:59.889 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "396", 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "394", 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:11:59.890 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:11:59.891 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_bias": 4 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 11, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_4", 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T17:11:57.982074+0000", 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 32769, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool,creating", 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:11:59.893 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:11:59.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "597", 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:11:59.895 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:11:59.896 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": {} 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:11:59.897 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7", 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0.2548370361328125, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 8, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 533, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-02-12T17:11:59.898 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:11:59.899 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2024-02-12T17:11:59.900 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/2545411201", 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6802/2545411201", 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6804/2545411201", 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6803/2545411201", 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:11:59.901 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24", 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 0, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 13, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 516, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 533, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.902 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6805/1723231708", 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6806/1723231708", 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6808/1723231708", 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6807/1723231708", 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "exists" 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c", 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0, 2024-02-12T17:11:59.905 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 18, 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.907 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3126580659", 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6810/3126580659", 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6812/3126580659", 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6811/3126580659", 2024-02-12T17:11:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9", 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 26, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:11:59.909 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2024-02-12T17:11:59.910 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6813/3941314957", 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6814/3941314957", 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6816/3941314957", 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6815/3941314957", 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:11:59.912 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0498129c-861c-4f7e-b8ba-219949da3130", 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.54083251953125, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 32, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:11:59.913 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6800", 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.914 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6801", 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6803", 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6802", 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6800/544126779", 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6801/544126779", 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6803/544126779", 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6802/544126779", 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:11:59.916 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50", 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 38, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:11:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6804", 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6805", 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.918 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6807", 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6806", 2024-02-12T17:11:59.919 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6804/3229884005", 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6805/3229884005", 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6807/3229884005", 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6806/3229884005", 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:11:59.920 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821", 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 45, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.921 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6808", 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6809", 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:11:59.922 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6811", 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6810", 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.923 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6808/507305558", 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6809/507305558", 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6811/507305558", 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6810/507305558", 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66", 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:11:59.924 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 54, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6812", 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.925 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6813", 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6815", 2024-02-12T17:11:59.926 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6814", 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6812/3401977873", 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6813/3401977873", 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6815/3401977873", 2024-02-12T17:11:59.927 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6814/3401977873", 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:11:59.928 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:10:54.353618+0000", 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:09:44.771155+0000", 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 532 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:11:59.929 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:54.538230+0000", 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:11:59.930 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:33.499743+0000", 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:51.353050+0000", 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:11:59.931 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:35.185056+0000", 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:11:59.932 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:49.922567+0000", 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:49.490108+0000", 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:11:59.933 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [], 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [ 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.5", 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.6", 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.934 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: 3 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.8", 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.d", 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.935 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.11", 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.16", 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: 3 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.18", 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.19", 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.1a", 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:11:59.938 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.1c", 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: 7 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.1d", 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:11:59.939 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.2", 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.9", 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.940 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.b", 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.d", 2024-02-12T17:11:59.941 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.e", 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: 3 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.f", 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: 7 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.13", 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.18", 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: 3 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.1b", 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.944 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.1d", 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.0", 2024-02-12T17:11:59.945 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.5", 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.c", 2024-02-12T17:11:59.946 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: 0, 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.10", 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.947 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.13", 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: 7 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "6.e", 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.948 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": { 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/2829464018": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/1835128904": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2277321750": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2116602769": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3967118778": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2298361247": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/218227980": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2070582473": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/4291575956": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3157562110": "2024-02-13T16:44:22.727241+0000" 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.949 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "ruleset-failure-domain": "osd", 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [], 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2024-02-12T17:11:59.950 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:11:59.951 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:11:59.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:11:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:00.002 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:00.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:00.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:00.139 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:00.150 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:00.176 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:58.771255+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:58.772137+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:58.845270+0000 osd.4 (osd.4) 177 : cluster [DBG] 5.a scrub starts 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:58.848271+0000 osd.4 (osd.4) 178 : cluster [DBG] 5.a scrub ok 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:58.937964+0000 mgr.y (mgr.24481) 842 : cluster [DBG] pgmap v1189: 126 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 6 unknown, 26 active+clean+remapped, 16 active+undersized, 16 active+undersized+degraded, 51 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 424 B/s, 9 objects/s recovering 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:58.988084+0000 mon.a (mon.0) 1959 : cluster [WRN] Health check update: Degraded data redundancy: 80/576 objects degraded (13.889%), 16 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:58.995292+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e598: 8 total, 7 up, 8 in 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:59.129305+0000 osd.3 (osd.3) 202 : cluster [DBG] 4.c scrub starts 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:59.132912+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.c scrub ok 2024-02-12T17:12:00.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:11:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:11:59.858496+0000 mon.c (mon.2) 278 : audit [DBG] from='client.? v1:172.21.15.33:0/3259428072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:12:00.263 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:00.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:00.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:58.771255+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:12:00.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:58.772137+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:12:00.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:58.845270+0000 osd.4 (osd.4) 177 : cluster [DBG] 5.a scrub starts 2024-02-12T17:12:00.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:58.848271+0000 osd.4 (osd.4) 178 : cluster [DBG] 5.a scrub ok 2024-02-12T17:12:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:58.937964+0000 mgr.y (mgr.24481) 842 : cluster [DBG] pgmap v1189: 126 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 6 unknown, 26 active+clean+remapped, 16 active+undersized, 16 active+undersized+degraded, 51 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 424 B/s, 9 objects/s recovering 2024-02-12T17:12:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:58.988084+0000 mon.a (mon.0) 1959 : cluster [WRN] Health check update: Degraded data redundancy: 80/576 objects degraded (13.889%), 16 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:58.995292+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e598: 8 total, 7 up, 8 in 2024-02-12T17:12:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:59.129305+0000 osd.3 (osd.3) 202 : cluster [DBG] 4.c scrub starts 2024-02-12T17:12:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:59.132912+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.c scrub ok 2024-02-12T17:12:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:11:59.858496+0000 mon.c (mon.2) 278 : audit [DBG] from='client.? v1:172.21.15.33:0/3259428072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:12:00.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:58.771255+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:12:00.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:58.772137+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:12:00.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:58.845270+0000 osd.4 (osd.4) 177 : cluster [DBG] 5.a scrub starts 2024-02-12T17:12:00.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:58.848271+0000 osd.4 (osd.4) 178 : cluster [DBG] 5.a scrub ok 2024-02-12T17:12:00.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:58.937964+0000 mgr.y (mgr.24481) 842 : cluster [DBG] pgmap v1189: 126 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 6 unknown, 26 active+clean+remapped, 16 active+undersized, 16 active+undersized+degraded, 51 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 424 B/s, 9 objects/s recovering 2024-02-12T17:12:00.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:58.988084+0000 mon.a (mon.0) 1959 : cluster [WRN] Health check update: Degraded data redundancy: 80/576 objects degraded (13.889%), 16 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:00.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:58.995292+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e598: 8 total, 7 up, 8 in 2024-02-12T17:12:00.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:59.129305+0000 osd.3 (osd.3) 202 : cluster [DBG] 4.c scrub starts 2024-02-12T17:12:00.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:59.132912+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.c scrub ok 2024-02-12T17:12:00.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:11:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:11:59.858496+0000 mon.c (mon.2) 278 : audit [DBG] from='client.? v1:172.21.15.33:0/3259428072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:12:00.368 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:00.494 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:00.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:00.501 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:12:00.502 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:12:00.598 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:00.628 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:01.054 INFO:tasks.thrashosds.thrasher:j is {'epoch': 598, 'fsid': 'cf6ad642-c9c5-11ee-95b9-87774f69a715', 'created': '2024-02-12T16:43:50.282468+0000', 'modified': '2024-02-12T17:11:58.988441+0000', 'last_up_change': '2024-02-12T17:10:02.814427+0000', 'last_in_change': '2024-02-12T17:11:51.074104+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T16:47:33.056945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-12T16:49:37.647343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 30, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 29, 'last_pg_merge_meta': {'source_pgid': '2.1e', 'ready_epoch': 586, 'last_epoch_started': 585, 'last_epoch_clean': 585, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '597', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '597', 'last_force_op_resend_preluminous': '597', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-12T16:49:38.076200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '527', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-12T16:49:39.589565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-12T16:49:41.623856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.10', 'ready_epoch': 429, 'last_epoch_started': 428, 'last_epoch_clean': 428, 'source_version': "0'0", 'target_version': "380'16"}, 'last_change': '433', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '429', 'last_force_op_resend_preluminous': '431', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-12T16:49:43.693806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '6.16', 'ready_epoch': 396, 'last_epoch_started': 395, 'last_epoch_clean': 395, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '396', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '396', 'last_force_op_resend_preluminous': '394', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-02-12T17:11:57.982074+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '597', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '5dd172eb-ee9d-4700-9b80-60f91c5f2cb7', 'up': 1, 'in': 1, 'weight': 0.2548370361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 533, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 2545411201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6802', 'nonce': 2545411201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6804', 'nonce': 2545411201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 2545411201}]}, 'public_addr': '172.21.15.33:6801/2545411201', 'cluster_addr': '172.21.15.33:6802/2545411201', 'heartbeat_back_addr': '172.21.15.33:6804/2545411201', 'heartbeat_front_addr': '172.21.15.33:6803/2545411201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71231bc4-1716-431d-bd88-f6a3cad0bd24', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 516, 'down_at': 533, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1723231708}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6806', 'nonce': 1723231708}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6808', 'nonce': 1723231708}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1723231708}]}, 'public_addr': '172.21.15.33:6805/1723231708', 'cluster_addr': '172.21.15.33:6806/1723231708', 'heartbeat_back_addr': '172.21.15.33:6808/1723231708', 'heartbeat_front_addr': '172.21.15.33:6807/1723231708', 'state': ['exists']}, {'osd': 2, 'uuid': '8a9c7127-d17c-4c44-b666-4211006de47c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3126580659}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6810', 'nonce': 3126580659}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6812', 'nonce': 3126580659}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3126580659}]}, 'public_addr': '172.21.15.33:6809/3126580659', 'cluster_addr': '172.21.15.33:6810/3126580659', 'heartbeat_back_addr': '172.21.15.33:6812/3126580659', 'heartbeat_front_addr': '172.21.15.33:6811/3126580659', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3cc40671-c8f7-412c-a0f1-987f48a016b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3941314957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6814', 'nonce': 3941314957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6816', 'nonce': 3941314957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3941314957}]}, 'public_addr': '172.21.15.33:6813/3941314957', 'cluster_addr': '172.21.15.33:6814/3941314957', 'heartbeat_back_addr': '172.21.15.33:6816/3941314957', 'heartbeat_front_addr': '172.21.15.33:6815/3941314957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0498129c-861c-4f7e-b8ba-219949da3130', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.54083251953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6800', 'nonce': 544126779}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 544126779}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 544126779}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 544126779}]}, 'public_addr': '172.21.15.94:6800/544126779', 'cluster_addr': '172.21.15.94:6801/544126779', 'heartbeat_back_addr': '172.21.15.94:6803/544126779', 'heartbeat_front_addr': '172.21.15.94:6802/544126779', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '05718a1e-1164-4b54-a493-23c1cd3adb50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 3229884005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 3229884005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 3229884005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 3229884005}]}, 'public_addr': '172.21.15.94:6804/3229884005', 'cluster_addr': '172.21.15.94:6805/3229884005', 'heartbeat_back_addr': '172.21.15.94:6807/3229884005', 'heartbeat_front_addr': '172.21.15.94:6806/3229884005', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dc48640f-8f27-405c-a9a4-cf1e3e6c8821', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 507305558}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 507305558}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 507305558}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 507305558}]}, 'public_addr': '172.21.15.94:6808/507305558', 'cluster_addr': '172.21.15.94:6809/507305558', 'heartbeat_back_addr': '172.21.15.94:6811/507305558', 'heartbeat_front_addr': '172.21.15.94:6810/507305558', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7d55aa3f-06b6-4fc3-97fd-62b535f78b66', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3401977873}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 3401977873}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 3401977873}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 3401977873}]}, 'public_addr': '172.21.15.94:6812/3401977873', 'cluster_addr': '172.21.15.94:6813/3401977873', 'heartbeat_back_addr': '172.21.15.94:6815/3401977873', 'heartbeat_front_addr': '172.21.15.94:6814/3401977873', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:10:54.353618+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:09:44.771155+0000', 'dead_epoch': 532}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:54.538230+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:33.499743+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:51.353050+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:35.185056+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:49.922567+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:49.490108+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '2.5', 'osds': [7, 4, 6]}, {'pgid': '2.6', 'osds': [6, 4, 3]}, {'pgid': '2.8', 'osds': [7, 4, 5]}, {'pgid': '2.d', 'osds': [3, 4, 2]}, {'pgid': '2.11', 'osds': [6, 4, 5]}, {'pgid': '2.16', 'osds': [6, 4, 3]}, {'pgid': '2.18', 'osds': [7, 4, 5]}, {'pgid': '2.19', 'osds': [7, 3, 0]}, {'pgid': '2.1a', 'osds': [3, 7, 2]}, {'pgid': '2.1c', 'osds': [5, 2, 7]}, {'pgid': '2.1d', 'osds': [3, 4, 2]}, {'pgid': '4.2', 'osds': [5, 4, 2]}, {'pgid': '4.9', 'osds': [3, 4, 6]}, {'pgid': '4.b', 'osds': [4, 3, 5]}, {'pgid': '4.d', 'osds': [4, 2, 6]}, {'pgid': '4.e', 'osds': [6, 4, 3]}, {'pgid': '4.f', 'osds': [3, 4, 7]}, {'pgid': '4.13', 'osds': [4, 6, 5]}, {'pgid': '4.18', 'osds': [4, 6, 3]}, {'pgid': '4.1b', 'osds': [3, 4, 2]}, {'pgid': '4.1d', 'osds': [3, 4, 6]}, {'pgid': '5.0', 'osds': [3, 4, 2]}, {'pgid': '5.5', 'osds': [6, 4, 2]}, {'pgid': '5.c', 'osds': [4, 0, 6]}, {'pgid': '5.10', 'osds': [7, 6, 5]}, {'pgid': '5.13', 'osds': [3, 5, 7]}, {'pgid': '6.e', 'osds': [4, 2, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.33:6800/2829464018': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/1835128904': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2277321750': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2116602769': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:6800/3967118778': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/2298361247': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/218227980': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/2070582473': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/4291575956': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:6800/3157562110': '2024-02-13T16:44:22.727241+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T17:12:01.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:12:01.143 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:11:59.999823+0000 mon.a (mon.0) 1961 : cluster [DBG] osdmap e599: 8 total, 7 up, 8 in 2024-02-12T17:12:01.159 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:12:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:00] "GET /metrics HTTP/1.1" 200 174330 "" "Prometheus/2.33.4" 2024-02-12T17:12:01.160 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:11:59.999823+0000 mon.a (mon.0) 1961 : cluster [DBG] osdmap e599: 8 total, 7 up, 8 in 2024-02-12T17:12:01.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:11:59.999823+0000 mon.a (mon.0) 1961 : cluster [DBG] osdmap e599: 8 total, 7 up, 8 in 2024-02-12T17:12:01.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T17:12:01.466 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:12:01.477 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:01.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:01.544 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:01.552 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:01.583 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:01.609 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:01.715 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:01.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:01.819 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:01.848 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:02.197 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:02.200 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:00.841827+0000 osd.4 (osd.4) 179 : cluster [DBG] 6.e scrub starts 2024-02-12T17:12:02.200 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:00.901869+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.e scrub ok 2024-02-12T17:12:02.200 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:00.938681+0000 mgr.y (mgr.24481) 843 : cluster [DBG] pgmap v1192: 125 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 6 unknown, 25 active+clean+remapped, 16 active+undersized, 16 active+undersized+degraded, 51 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 46 B/s, 2 objects/s recovering 2024-02-12T17:12:02.200 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:01.005573+0000 mon.a (mon.0) 1962 : cluster [DBG] osdmap e600: 8 total, 7 up, 8 in 2024-02-12T17:12:02.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:00.841827+0000 osd.4 (osd.4) 179 : cluster [DBG] 6.e scrub starts 2024-02-12T17:12:02.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:00.901869+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.e scrub ok 2024-02-12T17:12:02.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:00.938681+0000 mgr.y (mgr.24481) 843 : cluster [DBG] pgmap v1192: 125 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 6 unknown, 25 active+clean+remapped, 16 active+undersized, 16 active+undersized+degraded, 51 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 46 B/s, 2 objects/s recovering 2024-02-12T17:12:02.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:01.005573+0000 mon.a (mon.0) 1962 : cluster [DBG] osdmap e600: 8 total, 7 up, 8 in 2024-02-12T17:12:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:02.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:00.841827+0000 osd.4 (osd.4) 179 : cluster [DBG] 6.e scrub starts 2024-02-12T17:12:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:00.901869+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.e scrub ok 2024-02-12T17:12:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:00.938681+0000 mgr.y (mgr.24481) 843 : cluster [DBG] pgmap v1192: 125 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 6 unknown, 25 active+clean+remapped, 16 active+undersized, 16 active+undersized+degraded, 51 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 46 B/s, 2 objects/s recovering 2024-02-12T17:12:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:01.005573+0000 mon.a (mon.0) 1962 : cluster [DBG] osdmap e600: 8 total, 7 up, 8 in 2024-02-12T17:12:02.302 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:02.331 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:02.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:12:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:12:03.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:01.858019+0000 osd.5 (osd.5) 281 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:12:03.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:01.858969+0000 osd.5 (osd.5) 282 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:12:03.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:01.993270+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.19 scrub starts 2024-02-12T17:12:03.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:01.996393+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.19 scrub ok 2024-02-12T17:12:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:01.858019+0000 osd.5 (osd.5) 281 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:12:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:01.858969+0000 osd.5 (osd.5) 282 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:12:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:01.993270+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.19 scrub starts 2024-02-12T17:12:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:01.996393+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.19 scrub ok 2024-02-12T17:12:03.311 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:12:03.311 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:12:03.336 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:01.858019+0000 osd.5 (osd.5) 281 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:12:03.336 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:01.858969+0000 osd.5 (osd.5) 282 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:12:03.336 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:01.993270+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.19 scrub starts 2024-02-12T17:12:03.336 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:01.996393+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.19 scrub ok 2024-02-12T17:12:03.369 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:03.381 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:03.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:03.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:03.521 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:04.067 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:12:04.068 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:12:04.084 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:12:04.102 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:12:04.120 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:12:04.137 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:12:04.155 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:12:04.155 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:12:04.156 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:12:04.156 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:12:04.156 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:12:04.156 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:12:04.156 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:12:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:12:04.217 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:12:04.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:02.751238+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:12:04.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:02.752276+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:12:04.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:02.939401+0000 mgr.y (mgr.24481) 844 : cluster [DBG] pgmap v1194: 125 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 25 active+clean+remapped, 19 active+undersized, 16 active+undersized+degraded, 54 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 40 B/s, 2 objects/s recovering 2024-02-12T17:12:04.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:02.751238+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:12:04.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:02.752276+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:12:04.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:02.939401+0000 mgr.y (mgr.24481) 844 : cluster [DBG] pgmap v1194: 125 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 25 active+clean+remapped, 19 active+undersized, 16 active+undersized+degraded, 54 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 40 B/s, 2 objects/s recovering 2024-02-12T17:12:04.418 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:02.751238+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:12:04.418 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:02.752276+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:12:04.418 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:02.939401+0000 mgr.y (mgr.24481) 844 : cluster [DBG] pgmap v1194: 125 pgs: 1 active+remapped+backfill_toofull, 10 creating+peering, 25 active+clean+remapped, 19 active+undersized, 16 active+undersized+degraded, 54 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%); 40 B/s, 2 objects/s recovering 2024-02-12T17:12:04.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T17:12:04.471 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:04.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:04.574 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:04.601 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:04.619 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:04.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:04.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:04.742 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:12:04.750 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:04.769 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:04.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:04.873 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:04.908 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:05.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:12:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:12:05.109 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:05.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:03.902005+0000 osd.4 (osd.4) 181 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:12:05.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:03.904971+0000 osd.4 (osd.4) 182 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:12:05.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:03.919966+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:12:05.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:03.922890+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:12:05.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:04.031312+0000 mon.b (mon.1) 106 : audit [DBG] from='client.? v1:172.21.15.33:0/4189493807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:05.111 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:04.657878+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.b scrub starts 2024-02-12T17:12:05.112 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:04.711329+0000 osd.0 (osd.0) 227 : cluster [DBG] 5.b scrub ok 2024-02-12T17:12:05.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:03.902005+0000 osd.4 (osd.4) 181 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:12:05.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:03.904971+0000 osd.4 (osd.4) 182 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:12:05.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:03.919966+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:12:05.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:03.922890+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:12:05.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:04.031312+0000 mon.b (mon.1) 106 : audit [DBG] from='client.? v1:172.21.15.33:0/4189493807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:05.112 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:04.657878+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.b scrub starts 2024-02-12T17:12:05.113 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:04.711329+0000 osd.0 (osd.0) 227 : cluster [DBG] 5.b scrub ok 2024-02-12T17:12:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:05.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:05.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:05.242 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:03.902005+0000 osd.4 (osd.4) 181 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:12:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:03.904971+0000 osd.4 (osd.4) 182 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:12:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:03.919966+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:12:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:03.922890+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:12:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:04.031312+0000 mon.b (mon.1) 106 : audit [DBG] from='client.? v1:172.21.15.33:0/4189493807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:04.657878+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.b scrub starts 2024-02-12T17:12:05.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:04.711329+0000 osd.0 (osd.0) 227 : cluster [DBG] 5.b scrub ok 2024-02-12T17:12:05.590 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:05.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:05.694 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:05.727 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:06.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:04.703877+0000 osd.6 (osd.6) 151 : cluster [DBG] 4.e scrub starts 2024-02-12T17:12:06.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:04.705642+0000 osd.6 (osd.6) 152 : cluster [DBG] 4.e scrub ok 2024-02-12T17:12:06.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:04.803653+0000 mgr.y (mgr.24481) 845 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:06.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:04.940210+0000 mgr.y (mgr.24481) 846 : cluster [DBG] pgmap v1195: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:06.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:05.112597+0000 osd.3 (osd.3) 204 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:12:06.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:05.116066+0000 osd.3 (osd.3) 205 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:12:06.159 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:04.703877+0000 osd.6 (osd.6) 151 : cluster [DBG] 4.e scrub starts 2024-02-12T17:12:06.159 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:04.705642+0000 osd.6 (osd.6) 152 : cluster [DBG] 4.e scrub ok 2024-02-12T17:12:06.159 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:04.803653+0000 mgr.y (mgr.24481) 845 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:06.160 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:04.940210+0000 mgr.y (mgr.24481) 846 : cluster [DBG] pgmap v1195: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:06.160 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:05.112597+0000 osd.3 (osd.3) 204 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:12:06.160 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:05.116066+0000 osd.3 (osd.3) 205 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:12:06.229 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:04.703877+0000 osd.6 (osd.6) 151 : cluster [DBG] 4.e scrub starts 2024-02-12T17:12:06.229 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:04.705642+0000 osd.6 (osd.6) 152 : cluster [DBG] 4.e scrub ok 2024-02-12T17:12:06.230 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:04.803653+0000 mgr.y (mgr.24481) 845 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:06.230 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:04.940210+0000 mgr.y (mgr.24481) 846 : cluster [DBG] pgmap v1195: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:06.230 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:05.112597+0000 osd.3 (osd.3) 204 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:12:06.230 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:05.116066+0000 osd.3 (osd.3) 205 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:12:06.523 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:12:06.523 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:12:06.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:12:06.523 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:12:06.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.523 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183785 9.7 9:e8886c22:::benchmark_data_smithi033_131576_object46195:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.502793+0000", 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 497.01649084100001, 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1740439839999999, 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183785 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502793+0000", 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502793+0000", 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502792+0000", 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2820000000000002e-06 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502798+0000", 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2590000000000006e-06 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502807+0000", 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6879999999999996e-06 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502813+0000", 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4116000000000005e-05 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.502907+0000", 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012017 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.503027+0000", 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013745900000000001 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.503165+0000", 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002240578 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.505405+0000", 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0058207989999999998 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511226+0000", 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.040592560999999999 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551819+0000", 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3529999999999997e-06 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551826+0000", 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99712210199999995 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548948+0000", 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2199999999999998e-06 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548957+0000", 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.127834097 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676791+0000", 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5977999999999999e-05 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676837+0000", 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1740439839999999 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183788 9.a 9:5036b012:::benchmark_data_smithi033_131576_object46198:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:06.530 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.511400+0000", 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 497.007883677, 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.167927138, 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183788 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511400+0000", 2024-02-12T17:12:06.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511400+0000", 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511398+0000", 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1103e-05 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511409+0000", 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6739e-05 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511426+0000", 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9079999999999995e-06 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511434+0000", 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5638999999999997e-05 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511529+0000", 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115553 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511645+0000", 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012956600000000001 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511774+0000", 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010709754 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522484+0000", 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4372000000000001e-05 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522499+0000", 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020223532999999998 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.542722+0000", 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1199999999999996e-06 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.542729+0000", 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.993438987 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.536168+0000", 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6004000000000001e-05 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.536184+0000", 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.143099299 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.679283+0000", 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4094000000000002e-05 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.679328+0000", 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.167927138 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183790 9.7 9:e6205a30:::benchmark_data_smithi033_131576_object46200:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.511449+0000", 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 497.00783548200002, 2024-02-12T17:12:06.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1655356800000001, 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183790 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511449+0000", 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511449+0000", 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511448+0000", 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0199999999999996e-06 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511452+0000", 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2700000000000001e-06 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511458+0000", 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0380000000000001e-06 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511462+0000", 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023379049999999999 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.513800+0000", 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-02-12T17:12:06.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.513903+0000", 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000119113 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.514022+0000", 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087794180000000006 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522802+0000", 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029046764999999999 2024-02-12T17:12:06.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551849+0000", 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.323e-06 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551856+0000", 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99712761900000002 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548984+0000", 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.678e-06 2024-02-12T17:12:06.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548991+0000", 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1279593509999999 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676951+0000", 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3662999999999999e-05 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676984+0000", 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1655356800000001 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183791 9.f 9:fdb45a54:::benchmark_data_smithi033_131576_object46201:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.511471+0000", 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 497.00781322099999, 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1655592750000001, 2024-02-12T17:12:06.544 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183791 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511471+0000", 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511471+0000", 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511470+0000", 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9380000000000001e-06 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511473+0000", 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0540000000000002e-06 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511478+0000", 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5880000000000002e-06 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.511482+0000", 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040833400000000004 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.515565+0000", 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7264999999999998e-05 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.515662+0000", 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011482800000000001 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.515777+0000", 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070314419999999997 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522809+0000", 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028937609999999999 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551746+0000", 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4230000000000005e-06 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551755+0000", 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99540099699999995 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.547156+0000", 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0518e-05 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.547166+0000", 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1298328820000001 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.676999+0000", 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.099e-05 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677030+0000", 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1655592750000001 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.550 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183795 9.f 9:fc4b72ca:::benchmark_data_smithi033_131576_object46205:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.517291+0000", 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 497.00199349600001, 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1597805719999998, 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183795 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517291+0000", 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517291+0000", 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517289+0000", 2024-02-12T17:12:06.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5079999999999999e-06 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517296+0000", 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9369999999999993e-06 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517305+0000", 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9850000000000003e-06 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517311+0000", 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00049205400000000002 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517803+0000", 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8632e-05 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.517902+0000", 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116575 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,4", 2024-02-12T17:12:06.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.518018+0000", 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047957549999999996 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522814+0000", 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028968608999999999 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551783+0000", 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2039999999999997e-06 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551791+0000", 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99689064500000002 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548681+0000", 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9699999999999994e-06 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548691+0000", 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.12835124 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677043+0000", 2024-02-12T17:12:06.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8580000000000001e-05 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677071+0000", 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1597805719999998 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24355.0:2081 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94508044031616 gen 3] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.519445+0000", 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 496.99983934699998, 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.144085775, 2024-02-12T17:12:06.557 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24355", 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2081 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519445+0000", 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519445+0000", 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:06.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519444+0000", 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5200000000000002e-06 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519448+0000", 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1379999999999998e-06 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519453+0000", 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0740000000000003e-06 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519457+0000", 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2222000000000002e-05 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519509+0000", 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3050000000000004e-06 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519516+0000", 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.033933252999999997 2024-02-12T17:12:06.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.553450+0000", 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8100000000000004e-06 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.553458+0000", 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0242327739999999 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.577691+0000", 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030178699999999998 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for degraded object", 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.577993+0000", 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042871052 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.620864+0000", 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5253000000000005e-05 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.620949+0000", 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042581320999999998 2024-02-12T17:12:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.663531+0000", 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.144085775 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183800 9.7 9:e9f894db:::benchmark_data_smithi033_131576_object46210:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.526801+0000", 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 496.99248271699997, 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1920783080000001, 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183800 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526801+0000", 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526801+0000", 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526799+0000", 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0271e-05 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526809+0000", 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1632e-05 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526821+0000", 2024-02-12T17:12:06.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3779999999999998e-06 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526827+0000", 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025060631999999999 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551888+0000", 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4320000000000002e-06 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.551895+0000", 2024-02-12T17:12:06.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99711852300000003 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.549014+0000", 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.593e-06 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.549021+0000", 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1279905619999999 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677012+0000", 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5311999999999998e-05 2024-02-12T17:12:06.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677107+0000", 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010268900000000001 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677210+0000", 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002630837 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.679841+0000", 2024-02-12T17:12:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00041140600000000001 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680252+0000", 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059192439999999997 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.686171+0000", 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032511917000000001 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718683+0000", 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6761999999999995e-05 2024-02-12T17:12:06.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718750+0000", 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000129562 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718880+0000", 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1920783080000001 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183807 9.a 9:57e153ef:::benchmark_data_smithi033_131576_object46217:head [delete] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:12:06.570 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.566277+0000", 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 496.95300668200002, 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.142295142, 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183807 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566277+0000", 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566277+0000", 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566275+0000", 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6699999999999997e-06 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566282+0000", 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7689999999999996e-06 2024-02-12T17:12:06.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566290+0000", 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.651e-06 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566294+0000", 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4171999999999997e-05 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566329+0000", 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6659999999999996e-06 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566334+0000", 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97030596000000002 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.536640+0000", 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5310000000000001e-06 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.536649+0000", 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.143557961 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680207+0000", 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3682000000000003e-05 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680290+0000", 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8075e-05 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680368+0000", 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022998889999999998 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.682668+0000", 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028804249999999998 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.685549+0000", 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060336299999999999 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.686152+0000", 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022233121000000002 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.708385+0000", 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7024000000000002e-05 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.708442+0000", 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013035600000000001 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.708573+0000", 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.142295142 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183809 9.f 9:fdff3c78:::benchmark_data_smithi033_131576_object46219:head [delete] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.579023+0000", 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 496.94026137399999, 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1398916479999999, 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183809 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579023+0000", 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579023+0000", 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579020+0000", 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0170000000000002e-06 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579030+0000", 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0334e-05 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579040+0000", 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3850000000000001e-06 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579045+0000", 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0598000000000001e-05 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579076+0000", 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1750000000000002e-06 2024-02-12T17:12:06.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.579082+0000", 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96971665299999998 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548799+0000", 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9719999999999992e-06 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.548807+0000", 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1282766399999999 2024-02-12T17:12:06.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677083+0000", 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9552999999999997e-05 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677143+0000", 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114326 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,4", 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.677257+0000", 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033515070000000001 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.680609+0000", 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012439339999999999 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.681853+0000", 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043585680000000002 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.686211+0000", 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032530057000000001 2024-02-12T17:12:06.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718741+0000", 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899999999999999e-05 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718789+0000", 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012528800000000001 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.718914+0000", 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1398916479999999 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.584 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:123088 10.12 10:49421f1b:::benchmark_data_smithi033_160181_object123087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e482)", 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:53.073854+0000", 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.445430227, 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0338758690000001, 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 123088 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.073854+0000", 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.073854+0000", 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.073852+0000", 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8109000000000002e-05 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.073911+0000", 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8479999999999996e-06 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.073920+0000", 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6029999999999998e-06 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.073925+0000", 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0281999999999998e-05 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.073965+0000", 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3080000000000004e-06 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.073972+0000", 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99884762400000005 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.072819+0000", 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1841000000000001e-05 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.072831+0000", 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.034812692 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.107644+0000", 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5991999999999999e-05 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.107730+0000", 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0338758690000001 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:123104 10.16 10:6fa4bed7:::benchmark_data_smithi033_160181_object123103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:53.086392+0000", 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.432892608, 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0551708309999999, 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 123104 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.086392+0000", 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.086392+0000", 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.086390+0000", 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7556000000000003e-05 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.086447+0000", 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1524e-05 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.086459+0000", 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2750000000000001e-06 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.086465+0000", 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4888999999999994e-05 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.086540+0000", 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7030000000000001e-05 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.086557+0000", 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98441137599999995 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.070969+0000", 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2849e-05 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.070981+0000", 2024-02-12T17:12:06.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0511737990000001 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.122155+0000", 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016093600000000001 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.122316+0000", 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026998400000000002 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.122586+0000", 2024-02-12T17:12:06.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021047319999999998 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.124691+0000", 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1838000000000002e-05 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.124733+0000", 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0098006729999999993 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.134533+0000", 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0068371439999999999 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.141370+0000", 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2514000000000005e-05 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.141433+0000", 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000129338 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.141562+0000", 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0551708309999999 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:123109 10.12 10:48d90940:::benchmark_data_smithi033_160181_object123108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:53.088698+0000", 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.43058602299999, 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0362632870000001, 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 123109 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.088698+0000", 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.088698+0000", 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.088696+0000", 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3810000000000001e-05 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.088760+0000", 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3530000000000001e-06 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.088770+0000", 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.088775+0000", 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1384000000000003e-05 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.088806+0000", 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7139999999999998e-06 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.088812+0000", 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98407077600000004 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.072883+0000", 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0451999999999999e-05 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.072893+0000", 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.034870365 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.107763+0000", 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013812899999999999 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.107902+0000", 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00028809400000000002 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.108190+0000", 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023862000000000002 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.110576+0000", 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7359999999999996e-06 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.110586+0000", 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095154009999999997 2024-02-12T17:12:06.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.120101+0000", 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0046671339999999999 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.124768+0000", 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8780000000000003e-05 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.124827+0000", 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013445800000000001 2024-02-12T17:12:06.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.124961+0000", 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0362632870000001 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:123111 10.16 10:6e4e0d40:::benchmark_data_smithi033_160181_object123110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:53.090533+0000", 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.42875084900001, 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0512361139999999, 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 123111 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.090533+0000", 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.090533+0000", 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.090532+0000", 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7490999999999995e-05 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.090599+0000", 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0841e-05 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.090610+0000", 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3089999999999998e-06 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.090617+0000", 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6838000000000002e-05 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.090654+0000", 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.090661+0000", 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98034590200000005 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.071007+0000", 2024-02-12T17:12:06.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0209999999999999e-06 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.071012+0000", 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0527791390000001 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.123791+0000", 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4920000000000006e-05 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.123886+0000", 2024-02-12T17:12:06.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015178500000000001 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.124038+0000", 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088855900000000003 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.124926+0000", 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00036891000000000002 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.125295+0000", 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011771405 2024-02-12T17:12:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.137067+0000", 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00454726 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.141614+0000", 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9580999999999998e-05 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.141664+0000", 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000105796 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.141769+0000", 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0512361139999999 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:123143 10.12 10:4a40e669:::benchmark_data_smithi033_160181_object123142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:53.124924+0000", 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 253.39436043399999, 2024-02-12T17:12:06.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0002312999999998, 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 123143 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.124924+0000", 2024-02-12T17:12:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.124924+0000", 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.124922+0000", 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6313000000000002e-05 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.124968+0000", 2024-02-12T17:12:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1986e-05 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.124980+0000", 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6469999999999997e-06 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.124985+0000", 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1410999999999999e-05 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.125046+0000", 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0859000000000001e-05 2024-02-12T17:12:06.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.125057+0000", 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94788953600000003 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.072947+0000", 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.147e-05 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.072958+0000", 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.036284974 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.109243+0000", 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120015 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.109363+0000", 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000255715 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.109619+0000", 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097561200000000005 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.110594+0000", 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060937700000000003 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.111204+0000", 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0089272940000000005 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.120131+0000", 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004883882 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.125015+0000", 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1665000000000001e-05 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.125067+0000", 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8447000000000002e-05 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:55.125155+0000", 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0002312999999998 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:173344 10.6 10:64087d5e:::benchmark_data_smithi033_160181_object50172:head [delete] snapc 0=[] ondisk+write+known_if_redirected e511)", 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:18.581197+0000", 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 167.938086903, 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98295730000000003, 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 173344 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:18.581197+0000", 2024-02-12T17:12:06.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:18.581197+0000", 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:18.581195+0000", 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9059999999999996e-06 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:18.581202+0000", 2024-02-12T17:12:06.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1267000000000001e-05 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:18.581213+0000", 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6690000000000003e-06 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:18.581219+0000", 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5899999999999998e-05 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:18.581255+0000", 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1109999999999996e-06 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:18.581261+0000", 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96943079200000004 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:19.550692+0000", 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1675999999999995e-05 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:19.550774+0000", 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000217383 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:19.550991+0000", 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033786559999999998 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:19.554370+0000", 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001084081 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:19.555454+0000", 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086515050000000003 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:19.564105+0000", 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6778999999999999e-05 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:19.564132+0000", 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2589999999999999e-05 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:19.564155+0000", 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98295730000000003 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:174482 10.12 10:4a99bc85:::benchmark_data_smithi033_160181_object51310:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:20.543090+0000", 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.97619389600001, 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0755434209999999, 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 174482 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543090+0000", 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543090+0000", 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543088+0000", 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9410000000000001e-06 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543095+0000", 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9569999999999997e-06 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543105+0000", 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0989999999999998e-06 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543110+0000", 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4688000000000003e-05 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543155+0000", 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012679100000000001 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543282+0000", 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00039060100000000002 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.543672+0000", 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030499350000000001 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.546722+0000", 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3939e-05 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.546776+0000", 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085237300000000002 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547628+0000", 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022551458999999999 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.570180+0000", 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1895999999999999e-05 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.570192+0000", 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0483808619999999 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.618573+0000", 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0968999999999999e-05 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.618634+0000", 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0755434209999999 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:174487 10.12 10:4a18fb73:::benchmark_data_smithi033_160181_object51315:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:20.544653+0000", 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.974631257, 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.074048519, 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.631 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 174487 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.544653+0000", 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.544653+0000", 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.544652+0000", 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.245e-06 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.544656+0000", 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0120000000000002e-06 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.544662+0000", 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3420000000000001e-06 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.544667+0000", 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00055196800000000001 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.545218+0000", 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116029 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.545335+0000", 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00027614399999999998 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3,7", 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.545611+0000", 2024-02-12T17:12:06.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017210789999999999 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547332+0000", 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0840999999999997e-05 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547373+0000", 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00029942499999999997 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547672+0000", 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023127581000000001 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.570800+0000", 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8990000000000007e-06 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.570808+0000", 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0478475229999999 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.618656+0000", 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5382e-05 2024-02-12T17:12:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.618701+0000", 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.074048519 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:174494 10.16 10:6ddc9720:::benchmark_data_smithi033_160181_object51322:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:20.556888+0000", 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.96239654300001, 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0600397210000001, 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 174494 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.556888+0000", 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.556888+0000", 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.556886+0000", 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1299999999999998e-06 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.556892+0000", 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8900000000000002e-06 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.556902+0000", 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7950000000000002e-06 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.556909+0000", 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7431999999999997e-05 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.556946+0000", 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000132488 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.557079+0000", 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00042638499999999999 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,3", 2024-02-12T17:12:06.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.557505+0000", 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012023867000000001 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.569529+0000", 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018735850000000001 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.571403+0000", 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6380000000000001e-06 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.571408+0000", 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.045438533 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.616847+0000", 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0586000000000004e-05 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.616927+0000", 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0600397210000001 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:194854 10.6 10:645a1580:::benchmark_data_smithi033_160181_object71682:head [delete] snapc 0=[] ondisk+write+known_if_redirected e530)", 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:57.591769+0000", 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.92751550899999, 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0422078640000001, 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 194854 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.591769+0000", 2024-02-12T17:12:06.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.591769+0000", 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.591768+0000", 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4750000000000004e-06 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.591772+0000", 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5130000000000002e-06 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.591782+0000", 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0850000000000003e-06 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.591791+0000", 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4914999999999999e-05 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.591836+0000", 2024-02-12T17:12:06.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000140424 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.591976+0000", 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038267900000000003 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,6", 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.592359+0000", 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022349150000000001 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.594594+0000", 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052549699999999999 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.595119+0000", 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051679880000000001 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.600287+0000", 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2734999999999999e-05 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.600300+0000", 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.033599822 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.633900+0000", 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6789e-05 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.633976+0000", 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0422078640000001 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:194874 10.6 10:62ee851d:::benchmark_data_smithi033_160181_object71702:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T17:12:06.648 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:57.642581+0000", 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.87670314600001, 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.004659583, 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 194874 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.642581+0000", 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.642581+0000", 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.642580+0000", 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2739999999999998e-06 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.642585+0000", 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8779999999999998e-06 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.642593+0000", 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2810000000000002e-06 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.642597+0000", 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9379000000000001e-05 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.642626+0000", 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5360000000000003e-06 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:06.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:57.642631+0000", 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99174468800000004 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.634376+0000", 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117902 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.634494+0000", 2024-02-12T17:12:06.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000339482 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,6", 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.634833+0000", 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030486390000000001 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.637882+0000", 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030209569999999999 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.640903+0000", 2024-02-12T17:12:06.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062276579999999996 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.647130+0000", 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5458999999999998e-05 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.647196+0000", 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4858999999999999e-05 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:06.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:58.647241+0000", 2024-02-12T17:12:06.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.004659583 2024-02-12T17:12:06.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.655 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:06.655 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:06.655 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:12:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:06.656 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:06.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:12:06.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:12:06.757 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:06.790 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:06.892 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:06.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:07.007 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:07.035 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:12:07.037 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:07.214 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:05.846985+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:12:07.214 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:05.850451+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:12:07.214 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:06.439410+0000 osd.0 (osd.0) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:07.214 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:06.439489+0000 osd.0 (osd.0) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:07.214 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:06.999726+0000 mon.c (mon.2) 279 : audit [INF] from='client.? v1:172.21.15.33:0/1761199025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:07.214 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:06.999984+0000 mon.a (mon.0) 1963 : audit [INF] from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:07.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T17:12:07.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:05.846985+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:12:07.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:05.850451+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:12:07.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:06.439410+0000 osd.0 (osd.0) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:07.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:06.439489+0000 osd.0 (osd.0) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:07.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:06.999726+0000 mon.c (mon.2) 279 : audit [INF] from='client.? v1:172.21.15.33:0/1761199025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:06.999984+0000 mon.a (mon.0) 1963 : audit [INF] from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:05.846985+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:12:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:05.850451+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:12:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:06.439410+0000 osd.0 (osd.0) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:06.439489+0000 osd.0 (osd.0) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:06.999726+0000 mon.c (mon.2) 279 : audit [INF] from='client.? v1:172.21.15.33:0/1761199025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:06.999984+0000 mon.a (mon.0) 1963 : audit [INF] from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:07.386 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:07.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:07.492 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:07.520 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:07.545 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:12:07.618 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:07.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:07.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:07.722 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:07.750 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:07.823 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:07.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:07.927 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:07.956 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:08.036 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:08.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:08.141 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:08.166 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:06.901457+0000 osd.4 (osd.4) 183 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:12:08.166 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:06.904349+0000 osd.4 (osd.4) 184 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:12:08.166 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:06.941054+0000 mgr.y (mgr.24481) 847 : cluster [DBG] pgmap v1196: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:08.166 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:07.019208+0000 mon.a (mon.0) 1964 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:12:08.166 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:07.031041+0000 mon.a (mon.0) 1965 : audit [INF] from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:12:08.167 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:07.031128+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e601: 8 total, 7 up, 8 in 2024-02-12T17:12:08.167 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:07.988406+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? v1:172.21.15.33:0/1448593585' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:12:08.169 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:08.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:06.901457+0000 osd.4 (osd.4) 183 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:12:08.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:06.904349+0000 osd.4 (osd.4) 184 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:12:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:06.941054+0000 mgr.y (mgr.24481) 847 : cluster [DBG] pgmap v1196: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:07.019208+0000 mon.a (mon.0) 1964 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:12:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:07.031041+0000 mon.a (mon.0) 1965 : audit [INF] from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:12:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:07.031128+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e601: 8 total, 7 up, 8 in 2024-02-12T17:12:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:07.988406+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? v1:172.21.15.33:0/1448593585' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:12:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:06.901457+0000 osd.4 (osd.4) 183 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:12:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:06.904349+0000 osd.4 (osd.4) 184 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:12:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:06.941054+0000 mgr.y (mgr.24481) 847 : cluster [DBG] pgmap v1196: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:08.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:07.019208+0000 mon.a (mon.0) 1964 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:12:08.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:07.031041+0000 mon.a (mon.0) 1965 : audit [INF] from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:12:08.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:07.031128+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e601: 8 total, 7 up, 8 in 2024-02-12T17:12:08.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:07.988406+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? v1:172.21.15.33:0/1448593585' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T17:12:08.317 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:08.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:08.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:08.448 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:08.597 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-02-12T17:12:08.658 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:12:08.695 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:08.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:08.753 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:12:08.756 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:12:08.800 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:08.829 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:08.933 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:08.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:09.037 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:09.065 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:09.178 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:12:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:12:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-02-12T17:12:09.178 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:12:09.189 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:09.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:09.285 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:07.812288+0000 osd.5 (osd.5) 285 : cluster [DBG] 5.f scrub starts 2024-02-12T17:12:09.285 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:07.826222+0000 osd.5 (osd.5) 286 : cluster [DBG] 5.f scrub ok 2024-02-12T17:12:09.285 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:08.010025+0000 osd.7 (osd.7) 276 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:12:09.285 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:08.013243+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:12:09.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:08.391120+0000 mon.a (mon.0) 1968 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:09.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:08.592896+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.33:0/1448593585' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:12:09.286 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:08.592954+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e602: 8 total, 7 up, 8 in 2024-02-12T17:12:09.286 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:07.812288+0000 osd.5 (osd.5) 285 : cluster [DBG] 5.f scrub starts 2024-02-12T17:12:09.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:07.826222+0000 osd.5 (osd.5) 286 : cluster [DBG] 5.f scrub ok 2024-02-12T17:12:09.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:08.010025+0000 osd.7 (osd.7) 276 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:12:09.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:08.013243+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:12:09.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:08.391120+0000 mon.a (mon.0) 1968 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:09.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:08.592896+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.33:0/1448593585' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:12:09.287 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:08.592954+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e602: 8 total, 7 up, 8 in 2024-02-12T17:12:09.293 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:09.323 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:07.812288+0000 osd.5 (osd.5) 285 : cluster [DBG] 5.f scrub starts 2024-02-12T17:12:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:07.826222+0000 osd.5 (osd.5) 286 : cluster [DBG] 5.f scrub ok 2024-02-12T17:12:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:08.010025+0000 osd.7 (osd.7) 276 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:12:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:08.013243+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:12:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:08.391120+0000 mon.a (mon.0) 1968 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:08.592896+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.33:0/1448593585' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T17:12:09.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:08.592954+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e602: 8 total, 7 up, 8 in 2024-02-12T17:12:09.697 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1197,"stamp":"2024-02-12T17:12:07.028800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":53,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":2136,"num_read_kb":1888,"num_write":1000,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":21641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1130,"ondisk_log_size":1130,"up":309,"acting":334,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":423,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8315656,"kb_used_data":1126152,"kb_used_omap":0,"kb_used_meta":7187520,"kb_avail":741645560,"statfs":{"total":767960285184,"available":759445053440,"internally_reserved":0,"allocated":1153179648,"data_stored":1145534366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7360020480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.029803"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":300,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.001955+0000","last_change":"2024-02-12T17:07:35.884990+0000","last_active":"2024-02-12T17:12:00.001955+0000","last_peered":"2024-02-12T17:12:00.001955+0000","last_clean":"2024-02-12T17:12:00.001955+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:12:00.001955+0000","last_undegraded":"2024-02-12T17:12:00.001955+0000","last_fullsized":"2024-02-12T17:12:00.001955+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":30,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.004629+0000","last_change":"2024-02-12T17:11:53.988743+0000","last_active":"2024-02-12T17:12:00.004629+0000","last_peered":"2024-02-12T17:12:00.004629+0000","last_clean":"2024-02-12T17:12:00.004629+0000","last_became_active":"2024-02-12T17:11:52.621153+0000","last_became_peered":"2024-02-12T17:11:52.621153+0000","last_unstale":"2024-02-12T17:12:00.004629+0000","last_undegraded":"2024-02-12T17:12:00.004629+0000","last_fullsized":"2024-02-12T17:12:00.004629+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:07:47.845930+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.845930+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,3],"avail_no_missing":["4","3","6"],"object_location_counts":[{"shards":"3,4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":327,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.002807+0000","last_change":"2024-02-12T17:11:52.849834+0000","last_active":"2024-02-12T17:12:00.002807+0000","last_peered":"2024-02-12T17:12:00.002807+0000","last_clean":"2024-02-12T17:12:00.002807+0000","last_became_active":"2024-02-12T17:11:52.621205+0000","last_became_peered":"2024-02-12T17:11:52.621205+0000","last_unstale":"2024-02-12T17:12:00.002807+0000","last_undegraded":"2024-02-12T17:12:00.002807+0000","last_fullsized":"2024-02-12T17:12:00.002807+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7","last_scrub_stamp":"2024-02-12T17:05:56.388366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.388366+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":35,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.000994+0000","last_change":"2024-02-12T17:11:37.549503+0000","last_active":"2024-02-12T17:12:00.000994+0000","last_peered":"2024-02-12T17:12:00.000994+0000","last_clean":"2024-02-12T17:12:00.000994+0000","last_became_active":"2024-02-12T17:11:37.549052+0000","last_became_peered":"2024-02-12T17:11:37.549052+0000","last_unstale":"2024-02-12T17:12:00.000994+0000","last_undegraded":"2024-02-12T17:12:00.000994+0000","last_fullsized":"2024-02-12T17:12:00.000994+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.235654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.235654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,7],"avail_no_missing":["5","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":337,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002399+0000","last_change":"2024-02-12T17:11:52.013271+0000","last_active":"2024-02-12T17:12:00.002399+0000","last_peered":"2024-02-12T17:12:00.002399+0000","last_clean":"2024-02-12T17:12:00.002399+0000","last_became_active":"2024-02-12T17:11:51.598092+0000","last_became_peered":"2024-02-12T17:11:51.598092+0000","last_unstale":"2024-02-12T17:12:00.002399+0000","last_undegraded":"2024-02-12T17:12:00.002399+0000","last_fullsized":"2024-02-12T17:12:00.002399+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":276,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.960896+0000","last_change":"2024-02-12T17:11:51.620071+0000","last_active":"2024-02-12T17:12:00.960896+0000","last_peered":"2024-02-12T17:12:00.960896+0000","last_clean":"2024-02-12T17:12:00.960896+0000","last_became_active":"2024-02-12T17:11:51.619753+0000","last_became_peered":"2024-02-12T17:11:51.619753+0000","last_unstale":"2024-02-12T17:12:00.960896+0000","last_undegraded":"2024-02-12T17:12:00.960896+0000","last_fullsized":"2024-02-12T17:12:00.960896+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:29.479610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T17:06:29.479610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":375,"reported_epoch":600,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:05.116184+0000","last_change":"2024-02-12T17:12:05.116184+0000","last_active":"2024-02-12T17:12:05.116184+0000","last_peered":"2024-02-12T17:12:05.116184+0000","last_clean":"2024-02-12T17:12:05.116184+0000","last_became_active":"2024-02-12T17:11:52.628218+0000","last_became_peered":"2024-02-12T17:11:52.628218+0000","last_unstale":"2024-02-12T17:12:05.116184+0000","last_undegraded":"2024-02-12T17:12:05.116184+0000","last_fullsized":"2024-02-12T17:12:05.116184+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":1540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":33,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.001932+0000","last_change":"2024-02-12T17:11:37.552612+0000","last_active":"2024-02-12T17:12:00.001932+0000","last_peered":"2024-02-12T17:12:00.001932+0000","last_clean":"2024-02-12T17:12:00.001932+0000","last_became_active":"2024-02-12T17:11:37.552178+0000","last_became_peered":"2024-02-12T17:11:37.552178+0000","last_unstale":"2024-02-12T17:12:00.001932+0000","last_undegraded":"2024-02-12T17:12:00.001932+0000","last_fullsized":"2024-02-12T17:12:00.001932+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:52.828449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:06:52.828449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,2],"avail_no_missing":["3","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":151,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.003294+0000","last_change":"2024-02-12T17:10:03.934455+0000","last_active":"2024-02-12T17:12:00.003294+0000","last_peered":"2024-02-12T17:12:00.003294+0000","last_clean":"2024-02-12T17:09:50.594583+0000","last_became_active":"2024-02-12T17:10:03.934455+0000","last_became_peered":"2024-02-12T17:10:03.934455+0000","last_unstale":"2024-02-12T17:12:00.003294+0000","last_undegraded":"2024-02-12T17:10:03.866782+0000","last_fullsized":"2024-02-12T17:10:03.866683+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":38,"reported_epoch":600,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:01.996493+0000","last_change":"2024-02-12T17:12:01.996493+0000","last_active":"2024-02-12T17:12:01.996493+0000","last_peered":"2024-02-12T17:12:01.996493+0000","last_clean":"2024-02-12T17:12:01.996493+0000","last_became_active":"2024-02-12T17:11:37.565444+0000","last_became_peered":"2024-02-12T17:11:37.565444+0000","last_unstale":"2024-02-12T17:12:01.996493+0000","last_undegraded":"2024-02-12T17:12:01.996493+0000","last_fullsized":"2024-02-12T17:12:01.996493+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:01.996397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:01.996397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":126,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.792230+0000","last_change":"2024-02-12T17:10:03.869766+0000","last_active":"2024-02-12T17:12:00.792230+0000","last_peered":"2024-02-12T17:12:00.792230+0000","last_clean":"2024-02-12T17:09:58.609101+0000","last_became_active":"2024-02-12T17:10:03.869766+0000","last_became_peered":"2024-02-12T17:10:03.869766+0000","last_unstale":"2024-02-12T17:12:00.792230+0000","last_undegraded":"2024-02-12T17:10:03.860263+0000","last_fullsized":"2024-02-12T17:10:03.860160+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":548,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.960414+0000","last_change":"2024-02-12T17:11:52.609554+0000","last_active":"2024-02-12T17:12:00.960414+0000","last_peered":"2024-02-12T17:12:00.960414+0000","last_clean":"2024-02-12T17:12:00.960414+0000","last_became_active":"2024-02-12T17:11:52.608576+0000","last_became_peered":"2024-02-12T17:11:52.608576+0000","last_unstale":"2024-02-12T17:12:00.960414+0000","last_undegraded":"2024-02-12T17:12:00.960414+0000","last_fullsized":"2024-02-12T17:12:00.960414+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:55.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.027189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,5],"avail_no_missing":["7","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":662,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.961456+0000","last_change":"2024-02-12T17:10:03.919345+0000","last_active":"2024-02-12T17:12:00.961456+0000","last_peered":"2024-02-12T17:12:00.961456+0000","last_clean":"2024-02-12T17:09:58.612919+0000","last_became_active":"2024-02-12T17:10:03.919345+0000","last_became_peered":"2024-02-12T17:10:03.919345+0000","last_unstale":"2024-02-12T17:12:00.961456+0000","last_undegraded":"2024-02-12T17:10:03.858535+0000","last_fullsized":"2024-02-12T17:10:03.858425+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":296,"reported_epoch":599,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:12:00.960383+0000","last_change":"2024-02-12T17:11:52.611937+0000","last_active":"2024-02-12T17:12:00.960383+0000","last_peered":"2024-02-12T17:12:00.960383+0000","last_clean":"2024-02-12T17:11:51.020252+0000","last_became_active":"2024-02-12T17:11:52.607640+0000","last_became_peered":"2024-02-12T17:11:52.607640+0000","last_unstale":"2024-02-12T17:12:00.960383+0000","last_undegraded":"2024-02-12T17:12:00.960383+0000","last_fullsized":"2024-02-12T17:12:00.960383+0000","mapping_epoch":591,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":284,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002819+0000","last_change":"2024-02-12T17:09:21.547851+0000","last_active":"2024-02-12T17:12:00.002819+0000","last_peered":"2024-02-12T17:12:00.002819+0000","last_clean":"2024-02-12T17:12:00.002819+0000","last_became_active":"2024-02-12T17:09:21.547628+0000","last_became_peered":"2024-02-12T17:09:21.547628+0000","last_unstale":"2024-02-12T17:12:00.002819+0000","last_undegraded":"2024-02-12T17:12:00.002819+0000","last_fullsized":"2024-02-12T17:12:00.002819+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":105,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.791358+0000","last_change":"2024-02-12T17:11:36.552870+0000","last_active":"2024-02-12T17:12:00.791358+0000","last_peered":"2024-02-12T17:12:00.791358+0000","last_clean":"2024-02-12T17:12:00.791358+0000","last_became_active":"2024-02-12T17:11:36.552546+0000","last_became_peered":"2024-02-12T17:11:36.552546+0000","last_unstale":"2024-02-12T17:12:00.791358+0000","last_undegraded":"2024-02-12T17:12:00.791358+0000","last_fullsized":"2024-02-12T17:12:00.791358+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:31.071549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:31.071549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"526'27","reported_seq":306,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002152+0000","last_change":"2024-02-12T17:09:21.993424+0000","last_active":"2024-02-12T17:12:00.002152+0000","last_peered":"2024-02-12T17:12:00.002152+0000","last_clean":"2024-02-12T17:12:00.002152+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:12:00.002152+0000","last_undegraded":"2024-02-12T17:12:00.002152+0000","last_fullsized":"2024-02-12T17:12:00.002152+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:02:39.569600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:39.569600+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":111,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.792006+0000","last_change":"2024-02-12T17:11:51.604630+0000","last_active":"2024-02-12T17:12:00.792006+0000","last_peered":"2024-02-12T17:12:00.792006+0000","last_clean":"2024-02-12T17:12:00.792006+0000","last_became_active":"2024-02-12T17:11:51.604349+0000","last_became_peered":"2024-02-12T17:11:51.604349+0000","last_unstale":"2024-02-12T17:12:00.792006+0000","last_undegraded":"2024-02-12T17:12:00.792006+0000","last_fullsized":"2024-02-12T17:12:00.792006+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":580,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.960781+0000","last_change":"2024-02-12T17:11:51.600102+0000","last_active":"2024-02-12T17:12:00.960781+0000","last_peered":"2024-02-12T17:12:00.960781+0000","last_clean":"2024-02-12T17:12:00.960781+0000","last_became_active":"2024-02-12T17:11:51.599882+0000","last_became_peered":"2024-02-12T17:11:51.599882+0000","last_unstale":"2024-02-12T17:12:00.960781+0000","last_undegraded":"2024-02-12T17:12:00.960781+0000","last_fullsized":"2024-02-12T17:12:00.960781+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.911375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:28.846760+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.911375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":566,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.791830+0000","last_change":"2024-02-12T17:11:52.603067+0000","last_active":"2024-02-12T17:12:00.791830+0000","last_peered":"2024-02-12T17:12:00.791830+0000","last_clean":"2024-02-12T17:12:00.791830+0000","last_became_active":"2024-02-12T17:11:52.602172+0000","last_became_peered":"2024-02-12T17:11:52.602172+0000","last_unstale":"2024-02-12T17:12:00.791830+0000","last_undegraded":"2024-02-12T17:12:00.791830+0000","last_fullsized":"2024-02-12T17:12:00.791830+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:34.981898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.981898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,3],"avail_no_missing":["6","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":329,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.001933+0000","last_change":"2024-02-12T17:10:03.867728+0000","last_active":"2024-02-12T17:12:00.001933+0000","last_peered":"2024-02-12T17:12:00.001933+0000","last_clean":"2024-02-12T17:09:58.609134+0000","last_became_active":"2024-02-12T17:10:03.867728+0000","last_became_peered":"2024-02-12T17:10:03.867728+0000","last_unstale":"2024-02-12T17:12:00.001933+0000","last_undegraded":"2024-02-12T17:10:03.854546+0000","last_fullsized":"2024-02-12T17:10:03.854479+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":350,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001491+0000","last_change":"2024-02-12T17:10:03.943360+0000","last_active":"2024-02-12T17:12:00.001491+0000","last_peered":"2024-02-12T17:12:00.001491+0000","last_clean":"2024-02-12T17:05:38.389408+0000","last_became_active":"2024-02-12T17:10:03.943360+0000","last_became_peered":"2024-02-12T17:10:03.943360+0000","last_unstale":"2024-02-12T17:12:00.001491+0000","last_undegraded":"2024-02-12T17:12:00.001491+0000","last_fullsized":"2024-02-12T17:10:03.861151+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":266,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002065+0000","last_change":"2024-02-12T17:07:36.839451+0000","last_active":"2024-02-12T17:12:00.002065+0000","last_peered":"2024-02-12T17:12:00.002065+0000","last_clean":"2024-02-12T17:12:00.002065+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:12:00.002065+0000","last_undegraded":"2024-02-12T17:12:00.002065+0000","last_fullsized":"2024-02-12T17:12:00.002065+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":396,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001376+0000","last_change":"2024-02-12T17:10:03.880787+0000","last_active":"2024-02-12T17:12:00.001376+0000","last_peered":"2024-02-12T17:12:00.001376+0000","last_clean":"2024-02-12T17:09:19.537752+0000","last_became_active":"2024-02-12T17:10:03.880787+0000","last_became_peered":"2024-02-12T17:10:03.880787+0000","last_unstale":"2024-02-12T17:12:00.001376+0000","last_undegraded":"2024-02-12T17:12:00.001376+0000","last_fullsized":"2024-02-12T17:10:03.861487+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":32,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.006956+0000","last_change":"2024-02-12T17:11:54.262603+0000","last_active":"2024-02-12T17:12:00.006956+0000","last_peered":"2024-02-12T17:12:00.006956+0000","last_clean":"2024-02-12T17:12:00.006956+0000","last_became_active":"2024-02-12T17:11:52.612653+0000","last_became_peered":"2024-02-12T17:11:52.612653+0000","last_unstale":"2024-02-12T17:12:00.006956+0000","last_undegraded":"2024-02-12T17:12:00.006956+0000","last_fullsized":"2024-02-12T17:12:00.006956+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:38.751057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.751057+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,5],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":638,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.002091+0000","last_change":"2024-02-12T17:10:03.941991+0000","last_active":"2024-02-12T17:12:00.002091+0000","last_peered":"2024-02-12T17:12:00.002091+0000","last_clean":"2024-02-12T17:12:00.002091+0000","last_became_active":"2024-02-12T17:10:03.935291+0000","last_became_peered":"2024-02-12T17:10:03.935291+0000","last_unstale":"2024-02-12T17:12:00.002091+0000","last_undegraded":"2024-02-12T17:12:00.002091+0000","last_fullsized":"2024-02-12T17:12:00.002091+0000","mapping_epoch":533,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":538,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.000729+0000","last_change":"2024-02-12T17:10:03.879412+0000","last_active":"2024-02-12T17:12:00.000729+0000","last_peered":"2024-02-12T17:12:00.000729+0000","last_clean":"2024-02-12T17:09:58.605818+0000","last_became_active":"2024-02-12T17:10:03.879412+0000","last_became_peered":"2024-02-12T17:10:03.879412+0000","last_unstale":"2024-02-12T17:12:00.000729+0000","last_undegraded":"2024-02-12T17:12:00.000729+0000","last_fullsized":"2024-02-12T17:10:03.862546+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":103,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-12T17:12:02.752352+0000","last_change":"2024-02-12T17:12:02.752352+0000","last_active":"2024-02-12T17:12:02.752352+0000","last_peered":"2024-02-12T17:12:02.752352+0000","last_clean":"2024-02-12T17:12:02.752352+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:12:02.752352+0000","last_undegraded":"2024-02-12T17:12:02.752352+0000","last_fullsized":"2024-02-12T17:12:02.752352+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:02.752280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:02.752280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":155,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.792070+0000","last_change":"2024-02-12T17:10:03.869410+0000","last_active":"2024-02-12T17:12:00.792070+0000","last_peered":"2024-02-12T17:12:00.792070+0000","last_clean":"2024-02-12T17:09:49.876861+0000","last_became_active":"2024-02-12T17:10:03.869410+0000","last_became_peered":"2024-02-12T17:10:03.869410+0000","last_unstale":"2024-02-12T17:12:00.792070+0000","last_undegraded":"2024-02-12T17:10:03.856610+0000","last_fullsized":"2024-02-12T17:10:03.856493+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":642,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-12T17:12:01.007614+0000","last_change":"2024-02-12T17:05:55.414790+0000","last_active":"2024-02-12T17:12:01.007614+0000","last_peered":"2024-02-12T17:12:01.007614+0000","last_clean":"2024-02-12T17:12:01.007614+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:12:01.007614+0000","last_undegraded":"2024-02-12T17:12:01.007614+0000","last_fullsized":"2024-02-12T17:12:01.007614+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":19,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.007071+0000","last_change":"2024-02-12T17:11:51.595383+0000","last_active":"2024-02-12T17:12:00.007071+0000","last_peered":"2024-02-12T17:12:00.007071+0000","last_clean":"2024-02-12T17:12:00.007071+0000","last_became_active":"2024-02-12T17:11:51.595076+0000","last_became_peered":"2024-02-12T17:11:51.595076+0000","last_unstale":"2024-02-12T17:12:00.007071+0000","last_undegraded":"2024-02-12T17:12:00.007071+0000","last_fullsized":"2024-02-12T17:12:00.007071+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.622401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.622401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":685,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.001271+0000","last_change":"2024-02-12T17:07:33.734742+0000","last_active":"2024-02-12T17:12:00.001271+0000","last_peered":"2024-02-12T17:12:00.001271+0000","last_clean":"2024-02-12T17:12:00.001271+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:12:00.001271+0000","last_undegraded":"2024-02-12T17:12:00.001271+0000","last_fullsized":"2024-02-12T17:12:00.001271+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:33.734675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:33.734675+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"534'21","reported_seq":623,"reported_epoch":600,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:01.621346+0000","last_change":"2024-02-12T17:10:03.995272+0000","last_active":"2024-02-12T17:12:01.621346+0000","last_peered":"2024-02-12T17:12:01.621346+0000","last_clean":"2024-02-12T17:10:02.526826+0000","last_became_active":"2024-02-12T17:10:03.995272+0000","last_became_peered":"2024-02-12T17:10:03.995272+0000","last_unstale":"2024-02-12T17:12:01.621346+0000","last_undegraded":"2024-02-12T17:10:03.873726+0000","last_fullsized":"2024-02-12T17:10:03.873602+0000","mapping_epoch":533,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":589,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.000965+0000","last_change":"2024-02-12T17:07:35.798822+0000","last_active":"2024-02-12T17:12:00.000965+0000","last_peered":"2024-02-12T17:12:00.000965+0000","last_clean":"2024-02-12T17:12:00.000965+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:12:00.000965+0000","last_undegraded":"2024-02-12T17:12:00.000965+0000","last_fullsized":"2024-02-12T17:12:00.000965+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:35.798720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.798720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":320,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.003419+0000","last_change":"2024-02-12T17:10:03.901247+0000","last_active":"2024-02-12T17:12:00.003419+0000","last_peered":"2024-02-12T17:12:00.003419+0000","last_clean":"2024-02-12T17:09:58.611807+0000","last_became_active":"2024-02-12T17:10:03.901247+0000","last_became_peered":"2024-02-12T17:10:03.901247+0000","last_unstale":"2024-02-12T17:12:00.003419+0000","last_undegraded":"2024-02-12T17:10:03.867068+0000","last_fullsized":"2024-02-12T17:10:03.866968+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":602,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.961162+0000","last_change":"2024-02-12T17:11:51.608975+0000","last_active":"2024-02-12T17:12:00.961162+0000","last_peered":"2024-02-12T17:12:00.961162+0000","last_clean":"2024-02-12T17:12:00.961162+0000","last_became_active":"2024-02-12T17:11:51.608628+0000","last_became_peered":"2024-02-12T17:11:51.608628+0000","last_unstale":"2024-02-12T17:12:00.961162+0000","last_undegraded":"2024-02-12T17:12:00.961162+0000","last_fullsized":"2024-02-12T17:12:00.961162+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:37.116609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:06:37.116609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":112,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.791306+0000","last_change":"2024-02-12T17:11:52.607460+0000","last_active":"2024-02-12T17:12:00.791306+0000","last_peered":"2024-02-12T17:12:00.791306+0000","last_clean":"2024-02-12T17:12:00.791306+0000","last_became_active":"2024-02-12T17:11:52.607042+0000","last_became_peered":"2024-02-12T17:11:52.607042+0000","last_unstale":"2024-02-12T17:12:00.791306+0000","last_undegraded":"2024-02-12T17:12:00.791306+0000","last_fullsized":"2024-02-12T17:12:00.791306+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:34.780676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:34.780676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,5],"avail_no_missing":["6","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":309,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.003078+0000","last_change":"2024-02-12T17:10:03.919074+0000","last_active":"2024-02-12T17:12:00.003078+0000","last_peered":"2024-02-12T17:12:00.003078+0000","last_clean":"2024-02-12T17:09:58.621311+0000","last_became_active":"2024-02-12T17:10:03.919074+0000","last_became_peered":"2024-02-12T17:10:03.919074+0000","last_unstale":"2024-02-12T17:12:00.003078+0000","last_undegraded":"2024-02-12T17:10:03.875183+0000","last_fullsized":"2024-02-12T17:10:03.875072+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":19,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.005477+0000","last_change":"2024-02-12T17:11:51.596024+0000","last_active":"2024-02-12T17:12:00.005477+0000","last_peered":"2024-02-12T17:12:00.005477+0000","last_clean":"2024-02-12T17:12:00.005477+0000","last_became_active":"2024-02-12T17:11:51.595762+0000","last_became_peered":"2024-02-12T17:11:51.595762+0000","last_unstale":"2024-02-12T17:12:00.005477+0000","last_undegraded":"2024-02-12T17:12:00.005477+0000","last_fullsized":"2024-02-12T17:12:00.005477+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.272939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.272939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":461,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:11:59.999987+0000","last_change":"2024-02-12T17:10:03.867952+0000","last_active":"2024-02-12T17:11:59.999987+0000","last_peered":"2024-02-12T17:11:59.999987+0000","last_clean":"2024-02-12T17:08:07.412687+0000","last_became_active":"2024-02-12T17:10:03.867952+0000","last_became_peered":"2024-02-12T17:10:03.867952+0000","last_unstale":"2024-02-12T17:11:59.999987+0000","last_undegraded":"2024-02-12T17:11:59.999987+0000","last_fullsized":"2024-02-12T17:10:03.854012+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":285,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002742+0000","last_change":"2024-02-12T17:06:45.502715+0000","last_active":"2024-02-12T17:12:00.002742+0000","last_peered":"2024-02-12T17:12:00.002742+0000","last_clean":"2024-02-12T17:12:00.002742+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:12:00.002742+0000","last_undegraded":"2024-02-12T17:12:00.002742+0000","last_fullsized":"2024-02-12T17:12:00.002742+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.006010+0000","last_change":"2024-02-12T17:11:59.000993+0000","last_active":"2024-02-12T17:12:00.006010+0000","last_peered":"2024-02-12T17:12:00.006010+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.000993+0000","last_became_peered":"2024-02-12T17:11:59.000993+0000","last_unstale":"2024-02-12T17:12:00.006010+0000","last_undegraded":"2024-02-12T17:12:00.006010+0000","last_fullsized":"2024-02-12T17:11:58.993807+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":308,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.002652+0000","last_change":"2024-02-12T17:10:03.893996+0000","last_active":"2024-02-12T17:12:00.002652+0000","last_peered":"2024-02-12T17:12:00.002652+0000","last_clean":"2024-02-12T17:09:58.614534+0000","last_became_active":"2024-02-12T17:10:03.893996+0000","last_became_peered":"2024-02-12T17:10:03.893996+0000","last_unstale":"2024-02-12T17:12:00.002652+0000","last_undegraded":"2024-02-12T17:12:00.002652+0000","last_fullsized":"2024-02-12T17:10:03.859995+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":371,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.006795+0000","last_change":"2024-02-12T17:11:55.608014+0000","last_active":"2024-02-12T17:12:00.006795+0000","last_peered":"2024-02-12T17:12:00.006795+0000","last_clean":"2024-02-12T17:12:00.006795+0000","last_became_active":"2024-02-12T17:11:55.607676+0000","last_became_peered":"2024-02-12T17:11:55.607676+0000","last_unstale":"2024-02-12T17:12:00.006795+0000","last_undegraded":"2024-02-12T17:12:00.006795+0000","last_fullsized":"2024-02-12T17:12:00.006795+0000","mapping_epoch":594,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:07:32.199469+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:07:32.199469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":181,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.002370+0000","last_change":"2024-02-12T17:11:53.100445+0000","last_active":"2024-02-12T17:12:00.002370+0000","last_peered":"2024-02-12T17:12:00.002370+0000","last_clean":"2024-02-12T17:12:00.002370+0000","last_became_active":"2024-02-12T17:11:52.623783+0000","last_became_peered":"2024-02-12T17:11:52.623783+0000","last_unstale":"2024-02-12T17:12:00.002370+0000","last_undegraded":"2024-02-12T17:12:00.002370+0000","last_fullsized":"2024-02-12T17:12:00.002370+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":463,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001079+0000","last_change":"2024-02-12T17:10:03.891860+0000","last_active":"2024-02-12T17:12:00.001079+0000","last_peered":"2024-02-12T17:12:00.001079+0000","last_clean":"2024-02-12T17:04:37.847517+0000","last_became_active":"2024-02-12T17:10:03.891860+0000","last_became_peered":"2024-02-12T17:10:03.891860+0000","last_unstale":"2024-02-12T17:12:00.001079+0000","last_undegraded":"2024-02-12T17:12:00.001079+0000","last_fullsized":"2024-02-12T17:10:03.873207+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.005310+0000","last_change":"2024-02-12T17:11:59.009932+0000","last_active":"2024-02-12T17:12:00.005310+0000","last_peered":"2024-02-12T17:12:00.005310+0000","last_clean":"2024-02-12T17:12:00.005310+0000","last_became_active":"2024-02-12T17:11:59.009600+0000","last_became_peered":"2024-02-12T17:11:59.009600+0000","last_unstale":"2024-02-12T17:12:00.005310+0000","last_undegraded":"2024-02-12T17:12:00.005310+0000","last_fullsized":"2024-02-12T17:12:00.005310+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":586,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:11:59.999883+0000","last_change":"2024-02-12T17:07:23.791961+0000","last_active":"2024-02-12T17:11:59.999883+0000","last_peered":"2024-02-12T17:11:59.999883+0000","last_clean":"2024-02-12T17:11:59.999883+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:11:59.999883+0000","last_undegraded":"2024-02-12T17:11:59.999883+0000","last_fullsized":"2024-02-12T17:11:59.999883+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:23.791870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:23.791870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":383,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.000293+0000","last_change":"2024-02-12T17:10:03.864945+0000","last_active":"2024-02-12T17:12:00.000293+0000","last_peered":"2024-02-12T17:12:00.000293+0000","last_clean":"2024-02-12T17:08:07.411291+0000","last_became_active":"2024-02-12T17:10:03.864945+0000","last_became_peered":"2024-02-12T17:10:03.864945+0000","last_unstale":"2024-02-12T17:12:00.000293+0000","last_undegraded":"2024-02-12T17:12:00.000293+0000","last_fullsized":"2024-02-12T17:10:03.855363+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":171,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.001509+0000","last_change":"2024-02-12T17:11:55.829278+0000","last_active":"2024-02-12T17:12:00.001509+0000","last_peered":"2024-02-12T17:12:00.001509+0000","last_clean":"2024-02-12T17:12:00.001509+0000","last_became_active":"2024-02-12T17:11:52.612854+0000","last_became_peered":"2024-02-12T17:11:52.612854+0000","last_unstale":"2024-02-12T17:12:00.001509+0000","last_undegraded":"2024-02-12T17:12:00.001509+0000","last_fullsized":"2024-02-12T17:12:00.001509+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:55.829155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:55.829155+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"519'21","reported_seq":858,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-12T17:12:01.619995+0000","last_change":"2024-02-12T17:09:25.723429+0000","last_active":"2024-02-12T17:12:01.619995+0000","last_peered":"2024-02-12T17:12:01.619995+0000","last_clean":"2024-02-12T17:12:01.619995+0000","last_became_active":"2024-02-12T17:09:25.723083+0000","last_became_peered":"2024-02-12T17:09:25.723083+0000","last_unstale":"2024-02-12T17:12:01.619995+0000","last_undegraded":"2024-02-12T17:12:01.619995+0000","last_fullsized":"2024-02-12T17:12:01.619995+0000","mapping_epoch":517,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.002377+0000","last_change":"2024-02-12T17:11:59.003539+0000","last_active":"2024-02-12T17:12:00.002377+0000","last_peered":"2024-02-12T17:12:00.002377+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.003539+0000","last_became_peered":"2024-02-12T17:11:59.003539+0000","last_unstale":"2024-02-12T17:12:00.002377+0000","last_undegraded":"2024-02-12T17:12:00.002377+0000","last_fullsized":"2024-02-12T17:11:58.996599+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":306,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002597+0000","last_change":"2024-02-12T17:11:51.607702+0000","last_active":"2024-02-12T17:12:00.002597+0000","last_peered":"2024-02-12T17:12:00.002597+0000","last_clean":"2024-02-12T17:12:00.002597+0000","last_became_active":"2024-02-12T17:11:51.607338+0000","last_became_peered":"2024-02-12T17:11:51.607338+0000","last_unstale":"2024-02-12T17:12:00.002597+0000","last_undegraded":"2024-02-12T17:12:00.002597+0000","last_fullsized":"2024-02-12T17:12:00.002597+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:42.958489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:02:42.958489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":627,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001097+0000","last_change":"2024-02-12T17:10:03.879478+0000","last_active":"2024-02-12T17:12:00.001097+0000","last_peered":"2024-02-12T17:12:00.001097+0000","last_clean":"2024-02-12T17:09:58.604742+0000","last_became_active":"2024-02-12T17:10:03.879478+0000","last_became_peered":"2024-02-12T17:10:03.879478+0000","last_unstale":"2024-02-12T17:12:00.001097+0000","last_undegraded":"2024-02-12T17:12:00.001097+0000","last_fullsized":"2024-02-12T17:10:03.861800+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"534'69","reported_seq":202,"reported_epoch":600,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:01.007705+0000","last_change":"2024-02-12T17:10:03.934831+0000","last_active":"2024-02-12T17:12:01.007705+0000","last_peered":"2024-02-12T17:12:01.007705+0000","last_clean":"2024-02-12T17:09:51.482838+0000","last_became_active":"2024-02-12T17:10:03.934831+0000","last_became_peered":"2024-02-12T17:10:03.934831+0000","last_unstale":"2024-02-12T17:12:01.007705+0000","last_undegraded":"2024-02-12T17:10:03.873010+0000","last_fullsized":"2024-02-12T17:10:03.872891+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":496,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.792281+0000","last_change":"2024-02-12T17:11:53.596949+0000","last_active":"2024-02-12T17:12:00.792281+0000","last_peered":"2024-02-12T17:12:00.792281+0000","last_clean":"2024-02-12T17:12:00.792281+0000","last_became_active":"2024-02-12T17:11:53.596550+0000","last_became_peered":"2024-02-12T17:11:53.596550+0000","last_unstale":"2024-02-12T17:12:00.792281+0000","last_undegraded":"2024-02-12T17:12:00.792281+0000","last_fullsized":"2024-02-12T17:12:00.792281+0000","mapping_epoch":592,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,2],"avail_no_missing":["6","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.001921+0000","last_change":"2024-02-12T17:11:59.008876+0000","last_active":"2024-02-12T17:12:00.001921+0000","last_peered":"2024-02-12T17:12:00.001921+0000","last_clean":"2024-02-12T17:12:00.001921+0000","last_became_active":"2024-02-12T17:11:59.008587+0000","last_became_peered":"2024-02-12T17:11:59.008587+0000","last_unstale":"2024-02-12T17:12:00.001921+0000","last_undegraded":"2024-02-12T17:12:00.001921+0000","last_fullsized":"2024-02-12T17:12:00.001921+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":600,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.961281+0000","last_change":"2024-02-12T17:09:21.502830+0000","last_active":"2024-02-12T17:12:00.961281+0000","last_peered":"2024-02-12T17:12:00.961281+0000","last_clean":"2024-02-12T17:12:00.961281+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:12:00.961281+0000","last_undegraded":"2024-02-12T17:12:00.961281+0000","last_fullsized":"2024-02-12T17:12:00.961281+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.001888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.001888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":299,"reported_epoch":600,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:02.046073+0000","last_change":"2024-02-12T17:10:03.874040+0000","last_active":"2024-02-12T17:12:02.046073+0000","last_peered":"2024-02-12T17:12:02.046073+0000","last_clean":"2024-02-12T17:09:20.333789+0000","last_became_active":"2024-02-12T17:10:03.874040+0000","last_became_peered":"2024-02-12T17:10:03.874040+0000","last_unstale":"2024-02-12T17:12:02.046073+0000","last_undegraded":"2024-02-12T17:10:03.860815+0000","last_fullsized":"2024-02-12T17:10:03.856356+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":602,"num_read_kb":597,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":266,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002168+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:12:00.002168+0000","last_peered":"2024-02-12T17:12:00.002168+0000","last_clean":"2024-02-12T17:12:00.002168+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:12:00.002168+0000","last_undegraded":"2024-02-12T17:12:00.002168+0000","last_fullsized":"2024-02-12T17:12:00.002168+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":135,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.001838+0000","last_change":"2024-02-12T17:10:03.886514+0000","last_active":"2024-02-12T17:12:00.001838+0000","last_peered":"2024-02-12T17:12:00.001838+0000","last_clean":"2024-02-12T17:09:50.311156+0000","last_became_active":"2024-02-12T17:10:03.886514+0000","last_became_peered":"2024-02-12T17:10:03.886514+0000","last_unstale":"2024-02-12T17:12:00.001838+0000","last_undegraded":"2024-02-12T17:10:03.872376+0000","last_fullsized":"2024-02-12T17:10:03.872265+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":524,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-12T17:12:01.859044+0000","last_change":"2024-02-12T17:12:01.859044+0000","last_active":"2024-02-12T17:12:01.859044+0000","last_peered":"2024-02-12T17:12:01.859044+0000","last_clean":"2024-02-12T17:12:01.859044+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:12:01.859044+0000","last_undegraded":"2024-02-12T17:12:01.859044+0000","last_fullsized":"2024-02-12T17:12:01.859044+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:01.858974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:01.858974+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002207+0000","last_change":"2024-02-12T17:11:59.007548+0000","last_active":"2024-02-12T17:12:00.002207+0000","last_peered":"2024-02-12T17:12:00.002207+0000","last_clean":"2024-02-12T17:12:00.002207+0000","last_became_active":"2024-02-12T17:11:59.007214+0000","last_became_peered":"2024-02-12T17:11:59.007214+0000","last_unstale":"2024-02-12T17:12:00.002207+0000","last_undegraded":"2024-02-12T17:12:00.002207+0000","last_fullsized":"2024-02-12T17:12:00.002207+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":643,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.001530+0000","last_change":"2024-02-12T17:11:51.604118+0000","last_active":"2024-02-12T17:12:00.001530+0000","last_peered":"2024-02-12T17:12:00.001530+0000","last_clean":"2024-02-12T17:12:00.001530+0000","last_became_active":"2024-02-12T17:11:51.603527+0000","last_became_peered":"2024-02-12T17:11:51.603527+0000","last_unstale":"2024-02-12T17:12:00.001530+0000","last_undegraded":"2024-02-12T17:12:00.001530+0000","last_fullsized":"2024-02-12T17:12:00.001530+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.058086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.058086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":293,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002197+0000","last_change":"2024-02-12T17:09:21.602904+0000","last_active":"2024-02-12T17:12:00.002197+0000","last_peered":"2024-02-12T17:12:00.002197+0000","last_clean":"2024-02-12T17:12:00.002197+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:12:00.002197+0000","last_undegraded":"2024-02-12T17:12:00.002197+0000","last_fullsized":"2024-02-12T17:12:00.002197+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:54.311805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:05:54.311805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":576,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.001195+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:12:00.001195+0000","last_peered":"2024-02-12T17:12:00.001195+0000","last_clean":"2024-02-12T17:12:00.001195+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:12:00.001195+0000","last_undegraded":"2024-02-12T17:12:00.001195+0000","last_fullsized":"2024-02-12T17:12:00.001195+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":627,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.961142+0000","last_change":"2024-02-12T17:10:03.877565+0000","last_active":"2024-02-12T17:12:00.961142+0000","last_peered":"2024-02-12T17:12:00.961142+0000","last_clean":"2024-02-12T17:09:58.611920+0000","last_became_active":"2024-02-12T17:10:03.877565+0000","last_became_peered":"2024-02-12T17:10:03.877565+0000","last_unstale":"2024-02-12T17:12:00.961142+0000","last_undegraded":"2024-02-12T17:12:00.961142+0000","last_fullsized":"2024-02-12T17:10:03.854437+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":155,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.792479+0000","last_change":"2024-02-12T17:11:51.651481+0000","last_active":"2024-02-12T17:12:00.792479+0000","last_peered":"2024-02-12T17:12:00.792479+0000","last_clean":"2024-02-12T17:12:00.792479+0000","last_became_active":"2024-02-12T17:11:51.603746+0000","last_became_peered":"2024-02-12T17:11:51.603746+0000","last_unstale":"2024-02-12T17:12:00.792479+0000","last_undegraded":"2024-02-12T17:12:00.792479+0000","last_fullsized":"2024-02-12T17:12:00.792479+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'33","last_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.886342+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":602,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.961293+0000","last_change":"2024-02-12T17:11:25.567058+0000","last_active":"2024-02-12T17:12:00.961293+0000","last_peered":"2024-02-12T17:12:00.961293+0000","last_clean":"2024-02-12T17:12:00.961293+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:12:00.961293+0000","last_undegraded":"2024-02-12T17:12:00.961293+0000","last_fullsized":"2024-02-12T17:12:00.961293+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001830+0000","last_change":"2024-02-12T17:11:59.006012+0000","last_active":"2024-02-12T17:12:00.001830+0000","last_peered":"2024-02-12T17:12:00.001830+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.006012+0000","last_became_peered":"2024-02-12T17:11:59.006012+0000","last_unstale":"2024-02-12T17:12:00.001830+0000","last_undegraded":"2024-02-12T17:12:00.001830+0000","last_fullsized":"2024-02-12T17:11:58.996554+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":493,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001603+0000","last_change":"2024-02-12T17:10:03.942690+0000","last_active":"2024-02-12T17:12:00.001603+0000","last_peered":"2024-02-12T17:12:00.001603+0000","last_clean":"2024-02-12T17:04:37.845034+0000","last_became_active":"2024-02-12T17:10:03.942690+0000","last_became_peered":"2024-02-12T17:10:03.942690+0000","last_unstale":"2024-02-12T17:12:00.001603+0000","last_undegraded":"2024-02-12T17:12:00.001603+0000","last_fullsized":"2024-02-12T17:10:03.872324+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.005049+0000","last_change":"2024-02-12T17:11:51.597614+0000","last_active":"2024-02-12T17:12:00.005049+0000","last_peered":"2024-02-12T17:12:00.005049+0000","last_clean":"2024-02-12T17:12:00.005049+0000","last_became_active":"2024-02-12T17:11:51.597060+0000","last_became_peered":"2024-02-12T17:11:51.597060+0000","last_unstale":"2024-02-12T17:12:00.005049+0000","last_undegraded":"2024-02-12T17:12:00.005049+0000","last_fullsized":"2024-02-12T17:12:00.005049+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.537630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.537630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":620,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.960964+0000","last_change":"2024-02-12T17:10:03.878327+0000","last_active":"2024-02-12T17:12:00.960964+0000","last_peered":"2024-02-12T17:12:00.960964+0000","last_clean":"2024-02-12T17:09:58.611363+0000","last_became_active":"2024-02-12T17:10:03.878327+0000","last_became_peered":"2024-02-12T17:10:03.878327+0000","last_unstale":"2024-02-12T17:12:00.960964+0000","last_undegraded":"2024-02-12T17:10:03.854352+0000","last_fullsized":"2024-02-12T17:10:03.854244+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":149,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.961316+0000","last_change":"2024-02-12T17:10:03.875711+0000","last_active":"2024-02-12T17:12:00.961316+0000","last_peered":"2024-02-12T17:12:00.961316+0000","last_clean":"2024-02-12T17:09:50.556595+0000","last_became_active":"2024-02-12T17:10:03.875711+0000","last_became_peered":"2024-02-12T17:10:03.875711+0000","last_unstale":"2024-02-12T17:12:00.961316+0000","last_undegraded":"2024-02-12T17:10:03.857938+0000","last_fullsized":"2024-02-12T17:10:03.857823+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":613,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001453+0000","last_change":"2024-02-12T17:10:03.873160+0000","last_active":"2024-02-12T17:12:00.001453+0000","last_peered":"2024-02-12T17:12:00.001453+0000","last_clean":"2024-02-12T17:09:58.606674+0000","last_became_active":"2024-02-12T17:10:03.873160+0000","last_became_peered":"2024-02-12T17:10:03.873160+0000","last_unstale":"2024-02-12T17:12:00.001453+0000","last_undegraded":"2024-02-12T17:12:00.001453+0000","last_fullsized":"2024-02-12T17:10:03.861159+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.005161+0000","last_change":"2024-02-12T17:11:59.002565+0000","last_active":"2024-02-12T17:12:00.005161+0000","last_peered":"2024-02-12T17:12:00.005161+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.002565+0000","last_became_peered":"2024-02-12T17:11:59.002565+0000","last_unstale":"2024-02-12T17:12:00.005161+0000","last_undegraded":"2024-02-12T17:12:00.005161+0000","last_fullsized":"2024-02-12T17:11:58.998876+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":107,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.792048+0000","last_change":"2024-02-12T17:10:03.868467+0000","last_active":"2024-02-12T17:12:00.792048+0000","last_peered":"2024-02-12T17:12:00.792048+0000","last_clean":"2024-02-12T17:09:19.536649+0000","last_became_active":"2024-02-12T17:10:03.868467+0000","last_became_peered":"2024-02-12T17:10:03.868467+0000","last_unstale":"2024-02-12T17:12:00.792048+0000","last_undegraded":"2024-02-12T17:12:00.792048+0000","last_fullsized":"2024-02-12T17:10:03.854942+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":625,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.960419+0000","last_change":"2024-02-12T17:11:52.611057+0000","last_active":"2024-02-12T17:12:00.960419+0000","last_peered":"2024-02-12T17:12:00.960419+0000","last_clean":"2024-02-12T17:12:00.960419+0000","last_became_active":"2024-02-12T17:11:52.610650+0000","last_became_peered":"2024-02-12T17:11:52.610650+0000","last_unstale":"2024-02-12T17:12:00.960419+0000","last_undegraded":"2024-02-12T17:12:00.960419+0000","last_fullsized":"2024-02-12T17:12:00.960419+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:40.961097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:40.961097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":631,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.001313+0000","last_change":"2024-02-12T17:10:03.886500+0000","last_active":"2024-02-12T17:12:00.001313+0000","last_peered":"2024-02-12T17:12:00.001313+0000","last_clean":"2024-02-12T17:09:58.605538+0000","last_became_active":"2024-02-12T17:10:03.886500+0000","last_became_peered":"2024-02-12T17:10:03.886500+0000","last_unstale":"2024-02-12T17:12:00.001313+0000","last_undegraded":"2024-02-12T17:10:03.861901+0000","last_fullsized":"2024-02-12T17:10:03.861792+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":96,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.792282+0000","last_change":"2024-02-12T17:09:21.560337+0000","last_active":"2024-02-12T17:12:00.792282+0000","last_peered":"2024-02-12T17:12:00.792282+0000","last_clean":"2024-02-12T17:12:00.792282+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:12:00.792282+0000","last_undegraded":"2024-02-12T17:12:00.792282+0000","last_fullsized":"2024-02-12T17:12:00.792282+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:46.526746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:46.526746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.961105+0000","last_change":"2024-02-12T17:11:59.002042+0000","last_active":"2024-02-12T17:12:00.961105+0000","last_peered":"2024-02-12T17:12:00.961105+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.002042+0000","last_became_peered":"2024-02-12T17:11:59.002042+0000","last_unstale":"2024-02-12T17:12:00.961105+0000","last_undegraded":"2024-02-12T17:12:00.961105+0000","last_fullsized":"2024-02-12T17:11:58.999050+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":293,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002294+0000","last_change":"2024-02-12T17:11:51.598581+0000","last_active":"2024-02-12T17:12:00.002294+0000","last_peered":"2024-02-12T17:12:00.002294+0000","last_clean":"2024-02-12T17:12:00.002294+0000","last_became_active":"2024-02-12T17:11:51.597611+0000","last_became_peered":"2024-02-12T17:11:51.597611+0000","last_unstale":"2024-02-12T17:12:00.002294+0000","last_undegraded":"2024-02-12T17:12:00.002294+0000","last_fullsized":"2024-02-12T17:12:00.002294+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:01.455969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.455969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":122,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.791425+0000","last_change":"2024-02-12T17:11:58.772251+0000","last_active":"2024-02-12T17:12:00.791425+0000","last_peered":"2024-02-12T17:12:00.791425+0000","last_clean":"2024-02-12T17:12:00.791425+0000","last_became_active":"2024-02-12T17:11:52.602659+0000","last_became_peered":"2024-02-12T17:11:52.602659+0000","last_unstale":"2024-02-12T17:12:00.791425+0000","last_undegraded":"2024-02-12T17:12:00.791425+0000","last_fullsized":"2024-02-12T17:12:00.791425+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,3],"avail_no_missing":["6","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":330,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.002127+0000","last_change":"2024-02-12T17:10:03.894531+0000","last_active":"2024-02-12T17:12:00.002127+0000","last_peered":"2024-02-12T17:12:00.002127+0000","last_clean":"2024-02-12T17:09:58.609394+0000","last_became_active":"2024-02-12T17:10:03.894531+0000","last_became_peered":"2024-02-12T17:10:03.894531+0000","last_unstale":"2024-02-12T17:12:00.002127+0000","last_undegraded":"2024-02-12T17:10:03.855132+0000","last_fullsized":"2024-02-12T17:10:03.855063+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":19,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.004904+0000","last_change":"2024-02-12T17:11:51.597397+0000","last_active":"2024-02-12T17:12:00.004904+0000","last_peered":"2024-02-12T17:12:00.004904+0000","last_clean":"2024-02-12T17:12:00.004904+0000","last_became_active":"2024-02-12T17:11:51.596535+0000","last_became_peered":"2024-02-12T17:11:51.596535+0000","last_unstale":"2024-02-12T17:12:00.004904+0000","last_undegraded":"2024-02-12T17:12:00.004904+0000","last_fullsized":"2024-02-12T17:12:00.004904+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.006011+0000","last_change":"2024-02-12T17:11:59.001873+0000","last_active":"2024-02-12T17:12:00.006011+0000","last_peered":"2024-02-12T17:12:00.006011+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.001873+0000","last_became_peered":"2024-02-12T17:11:59.001873+0000","last_unstale":"2024-02-12T17:12:00.006011+0000","last_undegraded":"2024-02-12T17:12:00.006011+0000","last_fullsized":"2024-02-12T17:11:58.996345+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":633,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.961020+0000","last_change":"2024-02-12T17:09:22.252402+0000","last_active":"2024-02-12T17:12:00.961020+0000","last_peered":"2024-02-12T17:12:00.961020+0000","last_clean":"2024-02-12T17:12:00.961020+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:12:00.961020+0000","last_undegraded":"2024-02-12T17:12:00.961020+0000","last_fullsized":"2024-02-12T17:12:00.961020+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:07:52.341731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.341731+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":96,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.792093+0000","last_change":"2024-02-12T17:09:21.594780+0000","last_active":"2024-02-12T17:12:00.792093+0000","last_peered":"2024-02-12T17:12:00.792093+0000","last_clean":"2024-02-12T17:12:00.792093+0000","last_became_active":"2024-02-12T17:09:21.539652+0000","last_became_peered":"2024-02-12T17:09:21.539652+0000","last_unstale":"2024-02-12T17:12:00.792093+0000","last_undegraded":"2024-02-12T17:12:00.792093+0000","last_fullsized":"2024-02-12T17:12:00.792093+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:42.272807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:42.272807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":729,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.001352+0000","last_change":"2024-02-12T17:11:52.208317+0000","last_active":"2024-02-12T17:12:00.001352+0000","last_peered":"2024-02-12T17:12:00.001352+0000","last_clean":"2024-02-12T17:12:00.001352+0000","last_became_active":"2024-02-12T17:11:51.597080+0000","last_became_peered":"2024-02-12T17:11:51.597080+0000","last_unstale":"2024-02-12T17:12:00.001352+0000","last_undegraded":"2024-02-12T17:12:00.001352+0000","last_fullsized":"2024-02-12T17:12:00.001352+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'28","last_scrub_stamp":"2024-02-12T17:06:28.073131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:28.073131+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"595'26","reported_seq":466,"reported_epoch":600,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:01.006835+0000","last_change":"2024-02-12T17:11:55.606456+0000","last_active":"2024-02-12T17:12:01.006835+0000","last_peered":"2024-02-12T17:12:01.006835+0000","last_clean":"2024-02-12T17:12:01.006835+0000","last_became_active":"2024-02-12T17:11:55.605944+0000","last_became_peered":"2024-02-12T17:11:55.605944+0000","last_unstale":"2024-02-12T17:12:01.006835+0000","last_undegraded":"2024-02-12T17:12:01.006835+0000","last_fullsized":"2024-02-12T17:12:01.006835+0000","mapping_epoch":594,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.791724+0000","last_change":"2024-02-12T17:11:59.007425+0000","last_active":"2024-02-12T17:12:00.791724+0000","last_peered":"2024-02-12T17:12:00.791724+0000","last_clean":"2024-02-12T17:12:00.791724+0000","last_became_active":"2024-02-12T17:11:59.007213+0000","last_became_peered":"2024-02-12T17:11:59.007213+0000","last_unstale":"2024-02-12T17:12:00.791724+0000","last_undegraded":"2024-02-12T17:12:00.791724+0000","last_fullsized":"2024-02-12T17:12:00.791724+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":309,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.003193+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:12:00.003193+0000","last_peered":"2024-02-12T17:12:00.003193+0000","last_clean":"2024-02-12T17:12:00.003193+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:12:00.003193+0000","last_undegraded":"2024-02-12T17:12:00.003193+0000","last_fullsized":"2024-02-12T17:12:00.003193+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"596'28","reported_seq":925,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-12T17:12:01.621697+0000","last_change":"2024-02-12T17:11:55.613487+0000","last_active":"2024-02-12T17:12:01.621697+0000","last_peered":"2024-02-12T17:12:01.621697+0000","last_clean":"2024-02-12T17:12:01.621697+0000","last_became_active":"2024-02-12T17:11:55.613082+0000","last_became_peered":"2024-02-12T17:11:55.613082+0000","last_unstale":"2024-02-12T17:12:01.621697+0000","last_undegraded":"2024-02-12T17:12:01.621697+0000","last_fullsized":"2024-02-12T17:12:01.621697+0000","mapping_epoch":594,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'21","last_scrub_stamp":"2024-02-12T17:06:32.977637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:06:32.977637+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":261,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.960617+0000","last_change":"2024-02-12T17:11:52.609210+0000","last_active":"2024-02-12T17:12:00.960617+0000","last_peered":"2024-02-12T17:12:00.960617+0000","last_clean":"2024-02-12T17:12:00.960617+0000","last_became_active":"2024-02-12T17:11:52.608148+0000","last_became_peered":"2024-02-12T17:11:52.608148+0000","last_unstale":"2024-02-12T17:12:00.960617+0000","last_undegraded":"2024-02-12T17:12:00.960617+0000","last_fullsized":"2024-02-12T17:12:00.960617+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:24.976048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:06:24.976048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,5],"avail_no_missing":["7","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":148,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.791592+0000","last_change":"2024-02-12T17:11:52.659678+0000","last_active":"2024-02-12T17:12:00.791592+0000","last_peered":"2024-02-12T17:12:00.791592+0000","last_clean":"2024-02-12T17:12:00.791592+0000","last_became_active":"2024-02-12T17:11:52.602752+0000","last_became_peered":"2024-02-12T17:11:52.602752+0000","last_unstale":"2024-02-12T17:12:00.791592+0000","last_undegraded":"2024-02-12T17:12:00.791592+0000","last_fullsized":"2024-02-12T17:12:00.791592+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:05:55.344999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.344999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,3],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001756+0000","last_change":"2024-02-12T17:11:59.001433+0000","last_active":"2024-02-12T17:12:00.001756+0000","last_peered":"2024-02-12T17:12:00.001756+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.001433+0000","last_became_peered":"2024-02-12T17:11:59.001433+0000","last_unstale":"2024-02-12T17:12:00.001756+0000","last_undegraded":"2024-02-12T17:12:00.001756+0000","last_fullsized":"2024-02-12T17:11:58.996543+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":625,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001579+0000","last_change":"2024-02-12T17:10:03.874296+0000","last_active":"2024-02-12T17:12:00.001579+0000","last_peered":"2024-02-12T17:12:00.001579+0000","last_clean":"2024-02-12T17:09:58.605834+0000","last_became_active":"2024-02-12T17:10:03.874296+0000","last_became_peered":"2024-02-12T17:10:03.874296+0000","last_unstale":"2024-02-12T17:12:00.001579+0000","last_undegraded":"2024-02-12T17:12:00.001579+0000","last_fullsized":"2024-02-12T17:10:03.864117+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"596'24","reported_seq":946,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-12T17:12:01.621481+0000","last_change":"2024-02-12T17:11:55.612967+0000","last_active":"2024-02-12T17:12:01.621481+0000","last_peered":"2024-02-12T17:12:01.621481+0000","last_clean":"2024-02-12T17:12:01.621481+0000","last_became_active":"2024-02-12T17:11:55.612531+0000","last_became_peered":"2024-02-12T17:11:55.612531+0000","last_unstale":"2024-02-12T17:12:01.621481+0000","last_undegraded":"2024-02-12T17:12:01.621481+0000","last_fullsized":"2024-02-12T17:12:01.621481+0000","mapping_epoch":594,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'21","last_scrub_stamp":"2024-02-12T17:07:41.785018+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.785018+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":200,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.960679+0000","last_change":"2024-02-12T17:10:03.879456+0000","last_active":"2024-02-12T17:12:00.960679+0000","last_peered":"2024-02-12T17:12:00.960679+0000","last_clean":"2024-02-12T16:59:07.210188+0000","last_became_active":"2024-02-12T17:10:03.879456+0000","last_became_peered":"2024-02-12T17:10:03.879456+0000","last_unstale":"2024-02-12T17:12:00.960679+0000","last_undegraded":"2024-02-12T17:12:00.960679+0000","last_fullsized":"2024-02-12T17:10:03.854791+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":177,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.002940+0000","last_change":"2024-02-12T17:11:53.452018+0000","last_active":"2024-02-12T17:12:00.002940+0000","last_peered":"2024-02-12T17:12:00.002940+0000","last_clean":"2024-02-12T17:12:00.002940+0000","last_became_active":"2024-02-12T17:11:52.628397+0000","last_became_peered":"2024-02-12T17:11:52.628397+0000","last_unstale":"2024-02-12T17:12:00.002940+0000","last_undegraded":"2024-02-12T17:12:00.002940+0000","last_fullsized":"2024-02-12T17:12:00.002940+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.792131+0000","last_change":"2024-02-12T17:11:59.005258+0000","last_active":"2024-02-12T17:12:00.792131+0000","last_peered":"2024-02-12T17:12:00.792131+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.005258+0000","last_became_peered":"2024-02-12T17:11:59.005258+0000","last_unstale":"2024-02-12T17:12:00.792131+0000","last_undegraded":"2024-02-12T17:12:00.792131+0000","last_fullsized":"2024-02-12T17:11:59.003935+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":25,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.902058+0000","last_change":"2024-02-12T17:12:00.902004+0000","last_active":"2024-02-12T17:12:00.902058+0000","last_peered":"2024-02-12T17:12:00.902058+0000","last_clean":"2024-02-12T17:12:00.902058+0000","last_became_active":"2024-02-12T17:11:52.613393+0000","last_became_peered":"2024-02-12T17:11:52.613393+0000","last_unstale":"2024-02-12T17:12:00.902058+0000","last_undegraded":"2024-02-12T17:12:00.902058+0000","last_fullsized":"2024-02-12T17:12:00.902058+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":481,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-12T17:12:01.857388+0000","last_change":"2024-02-12T17:07:41.239456+0000","last_active":"2024-02-12T17:12:01.857388+0000","last_peered":"2024-02-12T17:12:01.857388+0000","last_clean":"2024-02-12T17:12:01.857388+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:12:01.857388+0000","last_undegraded":"2024-02-12T17:12:01.857388+0000","last_fullsized":"2024-02-12T17:12:01.857388+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:07:41.239350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.239350+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":230,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.002349+0000","last_change":"2024-02-12T17:10:03.893191+0000","last_active":"2024-02-12T17:12:00.002349+0000","last_peered":"2024-02-12T17:12:00.002349+0000","last_clean":"2024-02-12T17:08:07.415377+0000","last_became_active":"2024-02-12T17:10:03.893191+0000","last_became_peered":"2024-02-12T17:10:03.893191+0000","last_unstale":"2024-02-12T17:12:00.002349+0000","last_undegraded":"2024-02-12T17:12:00.002349+0000","last_fullsized":"2024-02-12T17:10:03.856411+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":383,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.003239+0000","last_change":"2024-02-12T17:11:59.133004+0000","last_active":"2024-02-12T17:12:00.003239+0000","last_peered":"2024-02-12T17:12:00.003239+0000","last_clean":"2024-02-12T17:12:00.003239+0000","last_became_active":"2024-02-12T17:11:51.601852+0000","last_became_peered":"2024-02-12T17:11:51.601852+0000","last_unstale":"2024-02-12T17:12:00.003239+0000","last_undegraded":"2024-02-12T17:12:00.003239+0000","last_fullsized":"2024-02-12T17:12:00.003239+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.001916+0000","last_change":"2024-02-12T17:11:59.496950+0000","last_active":"2024-02-12T17:12:00.001916+0000","last_peered":"2024-02-12T17:12:00.001916+0000","last_clean":"2024-02-12T17:12:00.001916+0000","last_became_active":"2024-02-12T17:11:59.496585+0000","last_became_peered":"2024-02-12T17:11:59.496585+0000","last_unstale":"2024-02-12T17:12:00.001916+0000","last_undegraded":"2024-02-12T17:12:00.001916+0000","last_fullsized":"2024-02-12T17:12:00.001916+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":320,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.003021+0000","last_change":"2024-02-12T17:11:51.597590+0000","last_active":"2024-02-12T17:12:00.003021+0000","last_peered":"2024-02-12T17:12:00.003021+0000","last_clean":"2024-02-12T17:12:00.003021+0000","last_became_active":"2024-02-12T17:11:51.597221+0000","last_became_peered":"2024-02-12T17:11:51.597221+0000","last_unstale":"2024-02-12T17:12:00.003021+0000","last_undegraded":"2024-02-12T17:12:00.003021+0000","last_fullsized":"2024-02-12T17:12:00.003021+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":20,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.005342+0000","last_change":"2024-02-12T17:11:52.605479+0000","last_active":"2024-02-12T17:12:00.005342+0000","last_peered":"2024-02-12T17:12:00.005342+0000","last_clean":"2024-02-12T17:12:00.005342+0000","last_became_active":"2024-02-12T17:11:52.605068+0000","last_became_peered":"2024-02-12T17:11:52.605068+0000","last_unstale":"2024-02-12T17:12:00.005342+0000","last_undegraded":"2024-02-12T17:12:00.005342+0000","last_fullsized":"2024-02-12T17:12:00.005342+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.567340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.567340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,6],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":591,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.960705+0000","last_change":"2024-02-12T17:11:51.602622+0000","last_active":"2024-02-12T17:12:00.960705+0000","last_peered":"2024-02-12T17:12:00.960705+0000","last_clean":"2024-02-12T17:12:00.960705+0000","last_became_active":"2024-02-12T17:11:51.602161+0000","last_became_peered":"2024-02-12T17:11:51.602161+0000","last_unstale":"2024-02-12T17:12:00.960705+0000","last_undegraded":"2024-02-12T17:12:00.960705+0000","last_fullsized":"2024-02-12T17:12:00.960705+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.877830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.877830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":38,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.005699+0000","last_change":"2024-02-12T17:11:53.984861+0000","last_active":"2024-02-12T17:12:00.005699+0000","last_peered":"2024-02-12T17:12:00.005699+0000","last_clean":"2024-02-12T17:12:00.005699+0000","last_became_active":"2024-02-12T17:11:52.603933+0000","last_became_peered":"2024-02-12T17:11:52.603933+0000","last_unstale":"2024-02-12T17:12:00.005699+0000","last_undegraded":"2024-02-12T17:12:00.005699+0000","last_fullsized":"2024-02-12T17:12:00.005699+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:07:37.774472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.774472+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,6],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.960962+0000","last_change":"2024-02-12T17:11:59.009775+0000","last_active":"2024-02-12T17:12:00.960962+0000","last_peered":"2024-02-12T17:12:00.960962+0000","last_clean":"2024-02-12T17:12:00.960962+0000","last_became_active":"2024-02-12T17:11:59.009543+0000","last_became_peered":"2024-02-12T17:11:59.009543+0000","last_unstale":"2024-02-12T17:12:00.960962+0000","last_undegraded":"2024-02-12T17:12:00.960962+0000","last_fullsized":"2024-02-12T17:12:00.960962+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":618,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.960872+0000","last_change":"2024-02-12T17:05:57.890025+0000","last_active":"2024-02-12T17:12:00.960872+0000","last_peered":"2024-02-12T17:12:00.960872+0000","last_clean":"2024-02-12T17:12:00.960872+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:12:00.960872+0000","last_undegraded":"2024-02-12T17:12:00.960872+0000","last_fullsized":"2024-02-12T17:12:00.960872+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:57.889955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:05:57.889955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":515,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.000157+0000","last_change":"2024-02-12T17:07:20.629548+0000","last_active":"2024-02-12T17:12:00.000157+0000","last_peered":"2024-02-12T17:12:00.000157+0000","last_clean":"2024-02-12T17:12:00.000157+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:12:00.000157+0000","last_undegraded":"2024-02-12T17:12:00.000157+0000","last_fullsized":"2024-02-12T17:12:00.000157+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.629467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.629467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":342,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.001790+0000","last_change":"2024-02-12T17:10:03.873123+0000","last_active":"2024-02-12T17:12:00.001790+0000","last_peered":"2024-02-12T17:12:00.001790+0000","last_clean":"2024-02-12T17:09:58.605059+0000","last_became_active":"2024-02-12T17:10:03.873123+0000","last_became_peered":"2024-02-12T17:10:03.873123+0000","last_unstale":"2024-02-12T17:12:00.001790+0000","last_undegraded":"2024-02-12T17:12:00.001790+0000","last_fullsized":"2024-02-12T17:10:03.863800+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":146,"reported_epoch":599,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:00.792321+0000","last_change":"2024-02-12T17:10:03.883757+0000","last_active":"2024-02-12T17:12:00.792321+0000","last_peered":"2024-02-12T17:12:00.792321+0000","last_clean":"2024-02-12T17:09:58.608650+0000","last_became_active":"2024-02-12T17:10:03.883757+0000","last_became_peered":"2024-02-12T17:10:03.883757+0000","last_unstale":"2024-02-12T17:12:00.792321+0000","last_undegraded":"2024-02-12T17:10:03.857329+0000","last_fullsized":"2024-02-12T17:10:03.857219+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.961292+0000","last_change":"2024-02-12T17:11:59.007574+0000","last_active":"2024-02-12T17:12:00.961292+0000","last_peered":"2024-02-12T17:12:00.961292+0000","last_clean":"2024-02-12T17:12:00.961292+0000","last_became_active":"2024-02-12T17:11:59.007300+0000","last_became_peered":"2024-02-12T17:11:59.007300+0000","last_unstale":"2024-02-12T17:12:00.961292+0000","last_undegraded":"2024-02-12T17:12:00.961292+0000","last_fullsized":"2024-02-12T17:12:00.961292+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":302,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.960723+0000","last_change":"2024-02-12T17:06:43.014800+0000","last_active":"2024-02-12T17:12:00.960723+0000","last_peered":"2024-02-12T17:12:00.960723+0000","last_clean":"2024-02-12T17:12:00.960723+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:12:00.960723+0000","last_undegraded":"2024-02-12T17:12:00.960723+0000","last_fullsized":"2024-02-12T17:12:00.960723+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:06:43.014737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":24,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.005842+0000","last_change":"2024-02-12T17:11:58.848344+0000","last_active":"2024-02-12T17:12:00.005842+0000","last_peered":"2024-02-12T17:12:00.005842+0000","last_clean":"2024-02-12T17:12:00.005842+0000","last_became_active":"2024-02-12T17:11:51.593680+0000","last_became_peered":"2024-02-12T17:11:51.593680+0000","last_unstale":"2024-02-12T17:12:00.005842+0000","last_undegraded":"2024-02-12T17:12:00.005842+0000","last_fullsized":"2024-02-12T17:12:00.005842+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.848275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.848275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":276,"reported_epoch":600,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:01.016307+0000","last_change":"2024-02-12T17:12:01.016307+0000","last_active":"2024-02-12T17:12:01.016307+0000","last_peered":"2024-02-12T17:12:01.016307+0000","last_clean":"2024-02-12T17:12:01.016307+0000","last_became_active":"2024-02-12T17:12:01.015764+0000","last_became_peered":"2024-02-12T17:12:01.015764+0000","last_unstale":"2024-02-12T17:12:01.016307+0000","last_undegraded":"2024-02-12T17:12:01.016307+0000","last_fullsized":"2024-02-12T17:12:01.016307+0000","mapping_epoch":591,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":30,"reported_epoch":599,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:00.007117+0000","last_change":"2024-02-12T17:11:54.090713+0000","last_active":"2024-02-12T17:12:00.007117+0000","last_peered":"2024-02-12T17:12:00.007117+0000","last_clean":"2024-02-12T17:12:00.007117+0000","last_became_active":"2024-02-12T17:11:52.619575+0000","last_became_peered":"2024-02-12T17:11:52.619575+0000","last_unstale":"2024-02-12T17:12:00.007117+0000","last_undegraded":"2024-02-12T17:12:00.007117+0000","last_fullsized":"2024-02-12T17:12:00.007117+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:38.887701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.887701+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,5],"avail_no_missing":["4","3","5"],"object_location_counts":[{"shards":"3,4,5","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":8,"reported_epoch":599,"state":"active+undersized","last_fresh":"2024-02-12T17:12:00.000673+0000","last_change":"2024-02-12T17:11:58.999911+0000","last_active":"2024-02-12T17:12:00.000673+0000","last_peered":"2024-02-12T17:12:00.000673+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:58.999911+0000","last_became_peered":"2024-02-12T17:11:58.999911+0000","last_unstale":"2024-02-12T17:12:00.000673+0000","last_undegraded":"2024-02-12T17:12:00.000673+0000","last_fullsized":"2024-02-12T17:11:58.993560+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":601,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.000874+0000","last_change":"2024-02-12T17:09:21.531721+0000","last_active":"2024-02-12T17:12:00.000874+0000","last_peered":"2024-02-12T17:12:00.000874+0000","last_clean":"2024-02-12T17:12:00.000874+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:12:00.000874+0000","last_undegraded":"2024-02-12T17:12:00.000874+0000","last_fullsized":"2024-02-12T17:12:00.000874+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:47.945301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:47.945301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":284,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.002688+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:12:00.002688+0000","last_peered":"2024-02-12T17:12:00.002688+0000","last_clean":"2024-02-12T17:12:00.002688+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:12:00.002688+0000","last_undegraded":"2024-02-12T17:12:00.002688+0000","last_fullsized":"2024-02-12T17:12:00.002688+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":717,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-12T17:12:00.000551+0000","last_change":"2024-02-12T17:09:22.375867+0000","last_active":"2024-02-12T17:12:00.000551+0000","last_peered":"2024-02-12T17:12:00.000551+0000","last_clean":"2024-02-12T17:12:00.000551+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:12:00.000551+0000","last_undegraded":"2024-02-12T17:12:00.000551+0000","last_fullsized":"2024-02-12T17:12:00.000551+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'34","last_scrub_stamp":"2024-02-12T17:07:47.772969+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.772969+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"595'29","reported_seq":1019,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-12T17:12:01.856932+0000","last_change":"2024-02-12T17:11:55.609567+0000","last_active":"2024-02-12T17:12:01.856932+0000","last_peered":"2024-02-12T17:12:01.856932+0000","last_clean":"2024-02-12T17:12:01.856932+0000","last_became_active":"2024-02-12T17:11:55.609231+0000","last_became_peered":"2024-02-12T17:11:55.609231+0000","last_unstale":"2024-02-12T17:12:01.856932+0000","last_undegraded":"2024-02-12T17:12:01.856932+0000","last_fullsized":"2024-02-12T17:12:01.856932+0000","mapping_epoch":594,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-02-12T17:06:35.066866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.066866+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":39,"acting":39,"num_store_stats":7},{"poolid":2,"num_pg":29,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":69,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":602,"num_read_kb":597,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":50,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1512,"num_read_kb":1269,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":14918,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1023,"ondisk_log_size":1023,"up":73,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":58,"acting":62,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":60,"acting":61,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234256,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370776,"kb_used_data":475216,"kb_used_omap":0,"kb_used_meta":895552,"kb_avail":92374376,"statfs":{"total":95995035648,"available":94591361024,"internally_reserved":0,"allocated":486621184,"data_stored":485686804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917045248},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.9300000000000002,"5min":1.6459999999999999,"15min":1.9430000000000001},"min":{"1min":0.221,"5min":0.218,"15min":0.16700000000000001},"max":{"1min":8.8510000000000009,"5min":8.8510000000000009,"15min":16.759},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.8540000000000001,"5min":1.3480000000000001,"15min":1.95},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.19600000000000001},"max":{"1min":6.633,"5min":7.8440000000000003,"15min":17.844000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":2.23,"15min":1.8560000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":6.4539999999999997,"5min":13.224,"15min":13.224},"last":1.234},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.6679999999999999,"15min":1.7549999999999999},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.22700000000000001},"max":{"1min":6.4660000000000002,"5min":10.194000000000001,"15min":17.324999999999999},"last":1.165}]},{"osd":2,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.1299999999999999,"5min":4.4089999999999998,"15min":2.6579999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":10.401,"5min":218.35300000000001,"15min":218.35300000000001},"last":0.443},{"interface":"front","average":{"1min":1.706,"5min":4.0010000000000003,"15min":2.8969999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.23000000000000001},"max":{"1min":7.8600000000000003,"5min":218.42099999999999,"15min":218.42099999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":3.226,"5min":1.651,"15min":1.8779999999999999},"min":{"1min":0.317,"5min":0.255,"15min":0.255},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":13.608000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.7879999999999998,"5min":1.8899999999999999,"15min":2.3039999999999998},"min":{"1min":0.29899999999999999,"5min":0.28399999999999997,"15min":0.25600000000000001},"max":{"1min":8.1600000000000001,"5min":13.699,"15min":31.152000000000001},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.67500000000000004,"15min":2.012},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8439999999999999,"5min":3.9279999999999999,"15min":50.406999999999996},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.70499999999999996,"15min":2.431},"min":{"1min":0.29999999999999999,"5min":0.249,"15min":0.249},"max":{"1min":3.988,"5min":4.5609999999999999,"15min":50.168999999999997},"last":0.252}]},{"osd":5,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.2090000000000001,"15min":2.2970000000000002},"min":{"1min":0.26000000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.8660000000000001,"5min":56.560000000000002,"15min":56.560000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.149,"15min":2.3620000000000001},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.117,"5min":53.963999999999999,"15min":53.963999999999999},"last":0.30499999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.621,"15min":1.786},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.1800000000000002,"5min":2.645,"15min":17.234999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.78700000000000003,"15min":1.8149999999999999},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":2.9079999999999999,"5min":3.5329999999999999,"15min":13.676},"last":0.38300000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528597,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311632,"kb_used_data":2120,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433520,"statfs":{"total":95995035648,"available":95675924480,"internally_reserved":0,"allocated":2170880,"data_stored":1244890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.7250000000000001,"15min":1.835},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":3.1179999999999999,"5min":27.744,"15min":27.744},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.665,"5min":1.794,"15min":1.8220000000000001},"min":{"1min":0.246,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":8.0190000000000001,"5min":27.614000000000001,"15min":27.614000000000001},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Feb 12 17:09:40 2024","interfaces":[{"interface":"back","average":{"1min":2.9870000000000001,"5min":1.74,"15min":1.629},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":14.884,"5min":14.884,"15min":14.884},"last":1.77},{"interface":"front","average":{"1min":3.0430000000000001,"5min":1.8220000000000001,"15min":1.8340000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":17.106999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":2.5990000000000002,"15min":2.327},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":7.2480000000000002,"5min":24.300999999999998,"15min":56.798000000000002},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.9159999999999999,"15min":2.1349999999999998},"min":{"1min":0.248,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.0190000000000001,"5min":24.143999999999998,"15min":56.780999999999999},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":4.4870000000000001,"5min":3.109,"15min":2.1339999999999999},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.25},"max":{"1min":58.420999999999999,"5min":58.420999999999999,"15min":58.420999999999999},"last":3.6349999999999998},{"interface":"front","average":{"1min":5.3499999999999996,"5min":3.173,"15min":2.4399999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":58.709000000000003,"5min":58.709000000000003,"15min":58.709000000000003},"last":1.004}]},{"osd":4,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.47399999999999998,"15min":1.6479999999999999},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.7210000000000001,"5min":2.456,"15min":24.956},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.59899999999999998,"15min":1.542},"min":{"1min":0.29299999999999998,"5min":0.222,"15min":0.222},"max":{"1min":1.1799999999999999,"5min":3.8690000000000002,"15min":13.728},"last":3.6720000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.90600000000000003,"15min":2.036},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.245},"max":{"1min":1.054,"5min":34.869,"15min":34.869},"last":3.706},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.95999999999999996,"15min":2.302},"min":{"1min":0.27500000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":8.8870000000000005,"5min":33.591000000000001,"15min":33.591000000000001},"last":0.38}]},{"osd":7,"last update":"Mon Feb 12 17:11:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.71699999999999997,"15min":2.0859999999999999},"min":{"1min":0.28299999999999997,"5min":0.255,"15min":0.255},"max":{"1min":6.9279999999999999,"5min":6.9279999999999999,"15min":31.033000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.72799999999999998,"15min":2.0110000000000001},"min":{"1min":0.34300000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.7090000000000001,"5min":5.2389999999999999,"15min":29.34},"last":0.47399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575125,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917072,"kb_used_data":319176,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":92828080,"statfs":{"total":95995035648,"available":95055953920,"internally_reserved":0,"allocated":326836224,"data_stored":325617050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2009999999999996,"5min":2.5369999999999999,"15min":1.7230000000000001},"min":{"1min":0.23499999999999999,"5min":0.20699999999999999,"15min":0.17499999999999999},"max":{"1min":65.328999999999994,"5min":65.328999999999994,"15min":65.328999999999994},"last":3.5190000000000001},{"interface":"front","average":{"1min":6.6280000000000001,"5min":2.516,"15min":1.7050000000000001},"min":{"1min":0.22800000000000001,"5min":0.193,"15min":0.189},"max":{"1min":65.372,"5min":65.372,"15min":65.372},"last":10.164}]},{"osd":2,"last update":"Mon Feb 12 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":5.9429999999999996,"5min":2.6989999999999998,"15min":2.0019999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.191},"max":{"1min":51.820999999999998,"5min":51.820999999999998,"15min":51.820999999999998},"last":2.2229999999999999},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.395,"15min":1.847},"min":{"1min":0.223,"5min":0.223,"15min":0.19},"max":{"1min":53.802999999999997,"5min":53.802999999999997,"15min":53.802999999999997},"last":9.8190000000000008}]},{"osd":3,"last update":"Mon Feb 12 17:09:37 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":2.7789999999999999,"15min":1.823},"min":{"1min":0.308,"5min":0.28999999999999998,"15min":0.23100000000000001},"max":{"1min":10.977,"5min":64.528000000000006,"15min":64.528000000000006},"last":12.565},{"interface":"front","average":{"1min":2.9289999999999998,"5min":2.4449999999999998,"15min":1.647},"min":{"1min":0.34200000000000003,"5min":0.312,"15min":0.246},"max":{"1min":11.124000000000001,"5min":63.753999999999998,"15min":63.753999999999998},"last":3.452}]},{"osd":4,"last update":"Mon Feb 12 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.6859999999999999,"15min":1.8879999999999999},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":7.556,"5min":12.02,"15min":12.91},"last":3.419},{"interface":"front","average":{"1min":1.849,"5min":1.5449999999999999,"15min":1.8260000000000001},"min":{"1min":0.374,"5min":0.33000000000000002,"15min":0.25900000000000001},"max":{"1min":9.6159999999999997,"5min":12.193,"15min":16.074000000000002},"last":4.04}]},{"osd":5,"last update":"Mon Feb 12 17:09:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.8819999999999999,"15min":2.0739999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.306,"5min":22.154,"15min":22.154},"last":3.4980000000000002},{"interface":"front","average":{"1min":1.423,"5min":1.754,"15min":1.913},"min":{"1min":0.317,"5min":0.317,"15min":0.27300000000000002},"max":{"1min":8.5820000000000007,"5min":21.803000000000001,"15min":21.803000000000001},"last":4.0060000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.7170000000000001,"15min":1.9430000000000001},"min":{"1min":0.38,"5min":0.36499999999999999,"15min":0.30599999999999999},"max":{"1min":7.6210000000000004,"5min":7.6210000000000004,"15min":13.834},"last":3.4750000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.7470000000000001,"15min":1.7789999999999999},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.307},"max":{"1min":4.3700000000000001,"5min":16.172999999999998,"15min":16.172999999999998},"last":4.0640000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.9239999999999999,"15min":2.0289999999999999},"min":{"1min":0.377,"5min":0.33100000000000002,"15min":0.28799999999999998},"max":{"1min":7.3099999999999996,"5min":8.7560000000000002,"15min":9.5239999999999991},"last":2.1880000000000002},{"interface":"front","average":{"1min":1.8200000000000001,"5min":2.0790000000000002,"15min":2.1669999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":9.3510000000000009,"5min":13.965999999999999,"15min":20.548999999999999},"last":5.835}]}]},{"osd":0,"up_from":8,"seq":34359738674,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107536,"kb_used_data":207240,"kb_used_omap":0,"kb_used_meta":900288,"kb_avail":92637616,"statfs":{"total":95995035648,"available":94860918784,"internally_reserved":0,"allocated":212213760,"data_stored":211351309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":921894912},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:09:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8809999999999998,"5min":1.694,"15min":1.484},"min":{"1min":0.253,"5min":0.22,"15min":0.17399999999999999},"max":{"1min":9.7420000000000009,"5min":10.807,"15min":23.510999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":4.3070000000000004,"5min":2.5259999999999998,"15min":1.6970000000000001},"min":{"1min":0.29699999999999999,"5min":0.20399999999999999,"15min":0.193},"max":{"1min":10.092000000000001,"5min":10.741,"15min":16.16},"last":6.2830000000000004}]},{"osd":2,"last update":"Mon Feb 12 17:11:20 2024","interfaces":[{"interface":"back","average":{"1min":4.5679999999999996,"5min":3.2109999999999999,"15min":1.8160000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.20200000000000001},"max":{"1min":13.791,"5min":15.472,"15min":15.472},"last":0.308},{"interface":"front","average":{"1min":4.4429999999999996,"5min":4.1059999999999999,"15min":2.2290000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":11.875999999999999,"5min":26.039999999999999,"15min":26.039999999999999},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:11:39 2024","interfaces":[{"interface":"back","average":{"1min":3.1230000000000002,"5min":4.79,"15min":2.4870000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":15.113,"5min":87.311000000000007,"15min":87.311000000000007},"last":0.35499999999999998},{"interface":"front","average":{"1min":5.6989999999999998,"5min":5.1589999999999998,"15min":2.6469999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.221},"max":{"1min":20.911000000000001,"5min":90.991,"15min":90.991},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.9319999999999999,"5min":1.5429999999999999,"15min":1.6100000000000001},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.23599999999999999},"max":{"1min":15.141999999999999,"5min":15.141999999999999,"15min":16.888999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.8009999999999999,"5min":2.8199999999999998,"15min":2.1560000000000001},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":11.057,"5min":26.082999999999998,"15min":26.082999999999998},"last":0.499}]},{"osd":5,"last update":"Mon Feb 12 17:11:27 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":2.254,"15min":2.4100000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28000000000000003},"max":{"1min":14.063000000000001,"5min":17.754000000000001,"15min":50.404000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.0569999999999999,"5min":1.601,"15min":2.2029999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28299999999999997},"max":{"1min":10.298,"5min":11.398,"15min":49.979999999999997},"last":0.48199999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.931,"5min":3.0609999999999999,"15min":2.1960000000000002},"min":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.29999999999999999},"max":{"1min":11.035,"5min":26.109999999999999,"15min":26.109999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.748,"5min":2.4060000000000001,"15min":2.137},"min":{"1min":0.39700000000000002,"5min":0.372,"15min":0.33600000000000002},"max":{"1min":15.06,"5min":15.847,"15min":16.294},"last":0.41999999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:11:20 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":1.7849999999999999,"15min":2.0430000000000001},"min":{"1min":0.44600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":11.459,"5min":11.459,"15min":19.373000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":3.3210000000000002,"5min":2.8180000000000001,"15min":2.4940000000000002},"min":{"1min":0.44600000000000001,"5min":0.38200000000000001,"15min":0.35599999999999998},"max":{"1min":8.0250000000000004,"5min":26.135000000000002,"15min":34.722999999999999},"last":0.53700000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411623,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1579396,"kb_used_data":83388,"kb_used_omap":0,"kb_used_meta":1494080,"kb_avail":92165756,"statfs":{"total":95995035648,"available":94377734144,"internally_reserved":0,"allocated":85389312,"data_stored":84475658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1529937920},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:19 2024","interfaces":[{"interface":"back","average":{"1min":2.5270000000000001,"5min":2.6459999999999999,"15min":1.7310000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17000000000000001},"max":{"1min":11.253,"5min":21.445,"15min":36.100999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":3.3570000000000002,"5min":2.7559999999999998,"15min":1.9750000000000001},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.193},"max":{"1min":7.617,"5min":19.879000000000001,"15min":36.299999999999997},"last":0.28000000000000003}]},{"osd":1,"last update":"Mon Feb 12 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":3.282,"5min":1.948,"15min":1.776},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.19600000000000001},"max":{"1min":10.922000000000001,"5min":20.763999999999999,"15min":20.763999999999999},"last":6.2999999999999998},{"interface":"front","average":{"1min":4.3310000000000004,"5min":2.3029999999999999,"15min":1.984},"min":{"1min":0.31900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":12.552,"5min":21.361999999999998,"15min":21.361999999999998},"last":6.657}]},{"osd":3,"last update":"Mon Feb 12 17:11:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":4.0300000000000002,"15min":2.101},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.1740000000000004,"5min":40.759999999999998,"15min":40.759999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.6480000000000001,"5min":3.4380000000000002,"15min":1.9159999999999999},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.23699999999999999},"max":{"1min":10.912000000000001,"5min":21.702999999999999,"15min":21.702999999999999},"last":0.25}]},{"osd":4,"last update":"Mon Feb 12 17:11:11 2024","interfaces":[{"interface":"back","average":{"1min":2.6440000000000001,"5min":2.2109999999999999,"15min":2.0169999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":11.204000000000001,"5min":21.515000000000001,"15min":21.515000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.998,"15min":2.1440000000000001},"min":{"1min":0.307,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6900000000000004,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.42999999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:11:19 2024","interfaces":[{"interface":"back","average":{"1min":2.597,"5min":2.2000000000000002,"15min":2.0339999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":11.351000000000001,"5min":21.542999999999999,"15min":32.57},"last":0.32400000000000001},{"interface":"front","average":{"1min":2.2850000000000001,"5min":2.121,"15min":2.2759999999999998},"min":{"1min":0.35099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":10.896000000000001,"5min":20.937000000000001,"15min":31.905999999999999},"last":0.42099999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:11:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":2.1539999999999999,"15min":1.9830000000000001},"min":{"1min":0.44600000000000001,"5min":0.313,"15min":0.30199999999999999},"max":{"1min":2.3839999999999999,"5min":20.966999999999999,"15min":20.966999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":2.016,"5min":1.9470000000000001,"15min":1.875},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.524,"5min":20.193000000000001,"15min":20.193000000000001},"last":0.39700000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:11:19 2024","interfaces":[{"interface":"back","average":{"1min":3.1299999999999999,"5min":2.1139999999999999,"15min":2.5640000000000001},"min":{"1min":0.34899999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":7.5919999999999996,"5min":20.221,"15min":45.468000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":2.2309999999999999,"5min":2.254,"15min":2.5489999999999999},"min":{"1min":0.40600000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":10.484999999999999,"5min":20.995999999999999,"15min":46.656999999999996},"last":0.35699999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149986,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907688,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":905536,"kb_avail":92837464,"statfs":{"total":95995035648,"available":95065563136,"internally_reserved":0,"allocated":2195456,"data_stored":1246005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":927268864},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:45 2024","interfaces":[{"interface":"back","average":{"1min":4.9969999999999999,"5min":4.1900000000000004,"15min":2.1720000000000002},"min":{"1min":0.186,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":18.297999999999998,"5min":18.497,"15min":58.451999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":6.1719999999999997,"5min":4.6280000000000001,"15min":2.5859999999999999},"min":{"1min":0.216,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":72.688000000000002,"5min":72.688000000000002,"15min":72.688000000000002},"last":0.36499999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:09:40 2024","interfaces":[{"interface":"back","average":{"1min":5.2539999999999996,"5min":2.9020000000000001,"15min":1.8779999999999999},"min":{"1min":0.28799999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.859999999999999,"5min":18.859999999999999,"15min":18.859999999999999},"last":8.9280000000000008},{"interface":"front","average":{"1min":3.4009999999999998,"5min":1.7889999999999999,"15min":1.3759999999999999},"min":{"1min":0.48499999999999999,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":14.654,"5min":14.654,"15min":14.962},"last":5.1630000000000003}]},{"osd":2,"last update":"Mon Feb 12 17:11:45 2024","interfaces":[{"interface":"back","average":{"1min":3.6920000000000002,"5min":3.6669999999999998,"15min":1.8400000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":40.954999999999998,"5min":40.954999999999998,"15min":40.954999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":6.6769999999999996,"5min":5.0549999999999997,"15min":2.2490000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.22},"max":{"1min":46.408000000000001,"5min":46.408000000000001,"15min":46.408000000000001},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:11:05 2024","interfaces":[{"interface":"back","average":{"1min":3.286,"5min":2.6779999999999999,"15min":2.246},"min":{"1min":0.40999999999999998,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":7.9169999999999998,"5min":18.401,"15min":19.010000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":2.831,"5min":1.5329999999999999,"15min":1.77},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.8470000000000004,"5min":7.9880000000000004,"15min":19.033999999999999},"last":0.33900000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:11:28 2024","interfaces":[{"interface":"back","average":{"1min":4.3570000000000002,"5min":3.0979999999999999,"15min":2.4180000000000001},"min":{"1min":0.38,"5min":0.253,"15min":0.253},"max":{"1min":16.943000000000001,"5min":17.940999999999999,"15min":17.940999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":3.1619999999999999,"5min":2.9940000000000002,"15min":2.5600000000000001},"min":{"1min":0.45500000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":10.089,"5min":16.564,"15min":16.564},"last":0.40400000000000003}]},{"osd":6,"last update":"Mon Feb 12 17:11:45 2024","interfaces":[{"interface":"back","average":{"1min":3.3860000000000001,"5min":2.3889999999999998,"15min":1.855},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":40.889000000000003,"5min":40.889000000000003,"15min":40.889000000000003},"last":0.377},{"interface":"front","average":{"1min":3.702,"5min":3.1880000000000002,"15min":2.1440000000000001},"min":{"1min":0.42899999999999999,"5min":0.35599999999999998,"15min":0.32600000000000001},"max":{"1min":16.989999999999998,"5min":18.265000000000001,"15min":18.265000000000001},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:11:05 2024","interfaces":[{"interface":"back","average":{"1min":4.8719999999999999,"5min":3.1389999999999998,"15min":2.7269999999999999},"min":{"1min":0.45200000000000001,"5min":0.35399999999999998,"15min":0.315},"max":{"1min":13.159000000000001,"5min":18.484000000000002,"15min":37.719999999999999},"last":0.505},{"interface":"front","average":{"1min":5.1920000000000002,"5min":3.0019999999999998,"15min":2.5950000000000002},"min":{"1min":0.40799999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":17.015000000000001,"5min":17.992999999999999,"15min":38.948},"last":0.55000000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953759,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893644,"kb_used_data":2116,"kb_used_omap":0,"kb_used_meta":891520,"kb_avail":92851508,"statfs":{"total":95995035648,"available":95079944192,"internally_reserved":0,"allocated":2166784,"data_stored":1244974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912916480},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:05 2024","interfaces":[{"interface":"back","average":{"1min":2.3460000000000001,"5min":1.3959999999999999,"15min":1.9830000000000001},"min":{"1min":0.26000000000000001,"5min":0.21199999999999999,"15min":0.20000000000000001},"max":{"1min":12.771000000000001,"5min":12.771000000000001,"15min":14.146000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.931,"5min":1.5860000000000001,"15min":1.968},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19800000000000001},"max":{"1min":12.65,"5min":12.65,"15min":20.244},"last":0.438}]},{"osd":1,"last update":"Mon Feb 12 17:09:01 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.528,"15min":1.8420000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.218},"max":{"1min":9.7349999999999994,"5min":9.7349999999999994,"15min":11.077999999999999},"last":3.7450000000000001},{"interface":"front","average":{"1min":1.488,"5min":1.877,"15min":2.0059999999999998},"min":{"1min":0.25900000000000001,"5min":0.252,"15min":0.23699999999999999},"max":{"1min":6.9569999999999999,"5min":14.082000000000001,"15min":14.082000000000001},"last":13.465999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:11:05 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":5.0960000000000001,"15min":3.3149999999999999},"min":{"1min":0.26200000000000001,"5min":0.255,"15min":0.252},"max":{"1min":25.587,"5min":191.05600000000001,"15min":191.05600000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":2.5459999999999998,"5min":5.085,"15min":3.1789999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.224},"max":{"1min":25.57,"5min":191.083,"15min":191.083},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:11:05 2024","interfaces":[{"interface":"back","average":{"1min":3.9689999999999999,"5min":2.6640000000000001,"15min":2.395},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":25.509,"5min":28.992000000000001,"15min":28.992000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":5.6539999999999999,"5min":2.5950000000000002,"15min":2.3300000000000001},"min":{"1min":0.314,"5min":0.30299999999999999,"15min":0.28899999999999998},"max":{"1min":25.475999999999999,"5min":29.084,"15min":29.084},"last":0.44800000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:11:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.91600000000000004,"15min":2.5249999999999999},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.8109999999999999,"5min":26.510000000000002,"15min":79.573999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.82199999999999995,"15min":2.766},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.219},"max":{"1min":2.5409999999999999,"5min":23.986999999999998,"15min":80.497},"last":0.33000000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.53400000000000003,"15min":1.7789999999999999},"min":{"1min":0.248,"5min":0.246,"15min":0.246},"max":{"1min":0.51600000000000001,"5min":3.0670000000000002,"15min":27.678000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.622,"15min":1.9239999999999999},"min":{"1min":0.30099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.52400000000000002,"5min":4.133,"15min":15.593},"last":0.317}]},{"osd":7,"last update":"Mon Feb 12 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67500000000000004,"15min":2.081},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27500000000000002},"max":{"1min":2.6200000000000001,"5min":4.0010000000000003,"15min":15.836},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.70399999999999996,"15min":2.1280000000000001},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":4.4980000000000002,"5min":4.4980000000000002,"15min":27.773},"last":0.378}]}]},{"osd":5,"up_from":38,"seq":163208757531,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1227912,"kb_used_data":34752,"kb_used_omap":0,"kb_used_meta":1193152,"kb_avail":92517240,"statfs":{"total":95995035648,"available":94737653760,"internally_reserved":0,"allocated":35586048,"data_stored":34667676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1221787648},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.4139999999999999,"15min":1.7729999999999999},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":7.4059999999999997,"5min":10.858000000000001,"15min":46.365000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.9510000000000001,"5min":1.536,"15min":2.1509999999999998},"min":{"1min":0.29899999999999999,"5min":0.22500000000000001,"15min":0.19900000000000001},"max":{"1min":15.368,"5min":15.368,"15min":46.134},"last":0.47499999999999998}]},{"osd":1,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.387,"15min":1.7450000000000001},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.23200000000000001},"max":{"1min":8.3620000000000001,"5min":11.19,"15min":16.655000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":2.3149999999999999,"5min":2.02,"15min":2.097},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.21199999999999999},"max":{"1min":10.973000000000001,"5min":12.154,"15min":16.712},"last":1.319}]},{"osd":2,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":2.6560000000000001,"5min":1.7270000000000001,"15min":2.012},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":15.612,"5min":15.612,"15min":16.940999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.8570000000000002,"5min":1.538,"15min":1.821},"min":{"1min":0.32000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":13.845000000000001,"5min":13.845000000000001,"15min":17.103999999999999},"last":0.48699999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":9.0950000000000006,"5min":3.6949999999999998,"15min":2.4910000000000001},"min":{"1min":0.30299999999999999,"5min":0.26100000000000001,"15min":0.22600000000000001},"max":{"1min":43.259,"5min":43.259,"15min":43.259},"last":0.496},{"interface":"front","average":{"1min":5.7750000000000004,"5min":3.1459999999999999,"15min":2.2719999999999998},"min":{"1min":0.34799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":43.198,"5min":43.198,"15min":43.198},"last":0.45700000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.55500000000000005,"15min":1.833},"min":{"1min":0.246,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":4.625,"5min":4.625,"15min":12.656000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.53700000000000003,"15min":1.9259999999999999},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":4.3250000000000002,"5min":4.3250000000000002,"15min":20.550000000000001},"last":0.255}]},{"osd":6,"last update":"Mon Feb 12 17:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.61199999999999999,"15min":1.724},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":12.281000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.60399999999999998,"15min":1.839},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":2.5630000000000002,"5min":4.3659999999999997,"15min":22.035},"last":0.33400000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.60599999999999998,"15min":1.905},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":5.3230000000000004,"5min":5.3230000000000004,"15min":19.184999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.72399999999999998,"15min":2.0299999999999998},"min":{"1min":0.35799999999999998,"5min":0.27800000000000002,"15min":0.26900000000000002},"max":{"1min":4.8159999999999998,"5min":5.1239999999999997,"15min":20.274000000000001},"last":0.34499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:12:09.702 INFO:tasks.thrashosds.thrasher:Setting 5.7 to [1, 4, 5] 2024-02-12T17:12:09.702 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.7', '1', '4', '5'] 2024-02-12T17:12:09.702 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pg-upmap 5.7 1 4 5 2024-02-12T17:12:09.889 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T17:12:10.160 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:12:10.172 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:10.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:10.212 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:08.559611+0000 mgr.y (mgr.24481) 848 : audit [DBG] from='client.16074 v1:172.21.15.33:0/1269112845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:12:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:08.793035+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:12:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:08.941902+0000 mgr.y (mgr.24481) 849 : cluster [DBG] pgmap v1199: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:08.951956+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:12:10.278 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:10.303 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:08.559611+0000 mgr.y (mgr.24481) 848 : audit [DBG] from='client.16074 v1:172.21.15.33:0/1269112845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:12:10.303 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:08.793035+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:12:10.304 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:08.941902+0000 mgr.y (mgr.24481) 849 : cluster [DBG] pgmap v1199: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:10.304 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:08.951956+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:12:10.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:08.559611+0000 mgr.y (mgr.24481) 848 : audit [DBG] from='client.16074 v1:172.21.15.33:0/1269112845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:12:10.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:08.793035+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:12:10.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:08.941902+0000 mgr.y (mgr.24481) 849 : cluster [DBG] pgmap v1199: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:10.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:08.951956+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:12:10.305 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:10.555 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:10.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:10.584 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:10.607 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:12:10.659 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:10.688 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:10.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:11.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:11.067 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:11.203 DEBUG:teuthology.orchestra.run.smithi033:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-02-12T17:12:11.206 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T17:12:11.206 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T17:12:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:09.878785+0000 osd.4 (osd.4) 185 : cluster [DBG] 2.f scrub starts 2024-02-12T17:12:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:09.882243+0000 osd.4 (osd.4) 186 : cluster [DBG] 2.f scrub ok 2024-02-12T17:12:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:10.013005+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T17:12:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:10.028487+0000 osd.7 (osd.7) 279 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T17:12:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:10.659893+0000 osd.0 (osd.0) 230 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:12:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:10.780373+0000 osd.0 (osd.0) 231 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:12:11.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:12:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:10] "GET /metrics HTTP/1.1" 200 176805 "" "Prometheus/2.33.4" 2024-02-12T17:12:11.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:09.878785+0000 osd.4 (osd.4) 185 : cluster [DBG] 2.f scrub starts 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:09.882243+0000 osd.4 (osd.4) 186 : cluster [DBG] 2.f scrub ok 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:10.013005+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:10.028487+0000 osd.7 (osd.7) 279 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:10.659893+0000 osd.0 (osd.0) 230 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:10.780373+0000 osd.0 (osd.0) 231 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:09.878785+0000 osd.4 (osd.4) 185 : cluster [DBG] 2.f scrub starts 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:09.882243+0000 osd.4 (osd.4) 186 : cluster [DBG] 2.f scrub ok 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:10.013005+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.12 deep-scrub starts 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:10.028487+0000 osd.7 (osd.7) 279 : cluster [DBG] 6.12 deep-scrub ok 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:10.659893+0000 osd.0 (osd.0) 230 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:12:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:10.780373+0000 osd.0 (osd.0) 231 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:12:11.412 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:hints = 1 2024-02-12T17:12:11.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T17:12:11.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object prefix: benchmark_data_smithi033_188107 2024-02-12T17:12:11.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:12:11.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 0 0 0 0 0 0 - 0 2024-02-12T17:12:11.816 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:11.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:11.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:11.922 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:11.971 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:12.042 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:10.942477+0000 mgr.y (mgr.24481) 850 : cluster [DBG] pgmap v1200: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:12.185 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:12.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:12.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:10.942477+0000 mgr.y (mgr.24481) 850 : cluster [DBG] pgmap v1200: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:12.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:10.942477+0000 mgr.y (mgr.24481) 850 : cluster [DBG] pgmap v1200: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 25 active+undersized, 16 active+undersized+degraded, 58 active+clean; 5.7 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/576 objects degraded (13.889%); 53/576 objects misplaced (9.201%) 2024-02-12T17:12:12.290 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:12.337 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:12.381 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:12:12.381 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:12:12.381 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:12:12.381 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:12:12.382 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:12:12.412 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 1 16 3824 3808 237.987 238 0.00291678 0.00419118 2024-02-12T17:12:12.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:12:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:12:13.003 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:13.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:11.788056+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.a scrub starts 2024-02-12T17:12:13.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:11.906732+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.a scrub ok 2024-02-12T17:12:13.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:12.131240+0000 osd.3 (osd.3) 206 : cluster [DBG] 4.11 deep-scrub starts 2024-02-12T17:12:13.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:12.141194+0000 osd.3 (osd.3) 207 : cluster [DBG] 4.11 deep-scrub ok 2024-02-12T17:12:13.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:11.788056+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.a scrub starts 2024-02-12T17:12:13.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:11.906732+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.a scrub ok 2024-02-12T17:12:13.164 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:12.131240+0000 osd.3 (osd.3) 206 : cluster [DBG] 4.11 deep-scrub starts 2024-02-12T17:12:13.164 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:12.141194+0000 osd.3 (osd.3) 207 : cluster [DBG] 4.11 deep-scrub ok 2024-02-12T17:12:13.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 2 16 7531 7515 234.827 231.688 0.00555584 0.00424796 2024-02-12T17:12:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:11.788056+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.a scrub starts 2024-02-12T17:12:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:11.906732+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.a scrub ok 2024-02-12T17:12:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:12.131240+0000 osd.3 (osd.3) 206 : cluster [DBG] 4.11 deep-scrub starts 2024-02-12T17:12:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:12.141194+0000 osd.3 (osd.3) 207 : cluster [DBG] 4.11 deep-scrub ok 2024-02-12T17:12:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:14.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:14.121 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:14.170 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:12.795954+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.17 scrub starts 2024-02-12T17:12:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:12.800427+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.17 scrub ok 2024-02-12T17:12:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:12.836325+0000 osd.5 (osd.5) 291 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:12:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:12.838559+0000 osd.5 (osd.5) 292 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:12:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:12.943141+0000 mgr.y (mgr.24481) 851 : cluster [DBG] pgmap v1201: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 22 active+undersized, 19 active+undersized+degraded, 58 active+clean; 5.5 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 704 KiB/s wr, 11 op/s; 133/840 objects degraded (15.833%); 53/840 objects misplaced (6.310%) 2024-02-12T17:12:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:13.039472+0000 mon.a (mon.0) 1971 : cluster [WRN] Health check update: Degraded data redundancy: 133/840 objects degraded (15.833%), 19 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:13.714159+0000 mon.a (mon.0) 1972 : audit [DBG] from='client.? v1:172.21.15.33:0/2747192149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:12.795954+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.17 scrub starts 2024-02-12T17:12:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:12.800427+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.17 scrub ok 2024-02-12T17:12:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:12.836325+0000 osd.5 (osd.5) 291 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:12:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:12.838559+0000 osd.5 (osd.5) 292 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:12:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:12.943141+0000 mgr.y (mgr.24481) 851 : cluster [DBG] pgmap v1201: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 22 active+undersized, 19 active+undersized+degraded, 58 active+clean; 5.5 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 704 KiB/s wr, 11 op/s; 133/840 objects degraded (15.833%); 53/840 objects misplaced (6.310%) 2024-02-12T17:12:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:13.039472+0000 mon.a (mon.0) 1971 : cluster [WRN] Health check update: Degraded data redundancy: 133/840 objects degraded (15.833%), 19 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:14.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:13.714159+0000 mon.a (mon.0) 1972 : audit [DBG] from='client.? v1:172.21.15.33:0/2747192149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:14.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:14.383 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:14.409 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:12.795954+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.17 scrub starts 2024-02-12T17:12:14.409 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:12.800427+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.17 scrub ok 2024-02-12T17:12:14.409 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:12.836325+0000 osd.5 (osd.5) 291 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:12:14.410 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:12.838559+0000 osd.5 (osd.5) 292 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:12:14.410 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:12.943141+0000 mgr.y (mgr.24481) 851 : cluster [DBG] pgmap v1201: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 22 active+undersized, 19 active+undersized+degraded, 58 active+clean; 5.5 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 704 KiB/s wr, 11 op/s; 133/840 objects degraded (15.833%); 53/840 objects misplaced (6.310%) 2024-02-12T17:12:14.410 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:13.039472+0000 mon.a (mon.0) 1971 : cluster [WRN] Health check update: Degraded data redundancy: 133/840 objects degraded (15.833%), 19 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:14.410 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:13.714159+0000 mon.a (mon.0) 1972 : audit [DBG] from='client.? v1:172.21.15.33:0/2747192149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:14.410 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:12:14.410 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:12:14.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:12:14.412 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:12:14.412 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:12:14.412 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:12:14.412 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:14.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 3 16 10377 10361 215.838 177.875 0.00141698 0.00449107 2024-02-12T17:12:15.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:12:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:12:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:12:15.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:12:15.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 4 16 12964 12948 202.296 161.688 0.00141157 0.00486937 2024-02-12T17:12:15.808 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:15.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:15.855 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T17:12:15.914 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:15.957 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:16.059 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:14.812617+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.16 scrub starts 2024-02-12T17:12:16.059 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:14.813494+0000 mgr.y (mgr.24481) 852 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:16.059 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:14.817349+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.16 scrub ok 2024-02-12T17:12:16.059 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:14.943990+0000 mgr.y (mgr.24481) 853 : cluster [DBG] pgmap v1202: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 16 active+undersized, 25 active+undersized+degraded, 58 active+clean; 325 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 651 op/s; 3172/16182 objects degraded (19.602%); 53/16182 objects misplaced (0.328%) 2024-02-12T17:12:16.059 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:15.214572+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:12:16.059 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:15.289692+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.33:0/849921718' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]: dispatch 2024-02-12T17:12:16.060 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:15.290020+0000 mon.a (mon.0) 1973 : audit [INF] from='client.36032 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]: dispatch 2024-02-12T17:12:16.079 INFO:teuthology.orchestra.run.smithi033.stderr:set 5.7 pg_upmap mapping to [1,4,5] 2024-02-12T17:12:16.236 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:14.812617+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.16 scrub starts 2024-02-12T17:12:16.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:14.813494+0000 mgr.y (mgr.24481) 852 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:16.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:14.817349+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.16 scrub ok 2024-02-12T17:12:16.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:14.943990+0000 mgr.y (mgr.24481) 853 : cluster [DBG] pgmap v1202: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 16 active+undersized, 25 active+undersized+degraded, 58 active+clean; 325 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 651 op/s; 3172/16182 objects degraded (19.602%); 53/16182 objects misplaced (0.328%) 2024-02-12T17:12:16.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:15.214572+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:12:16.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:15.289692+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.33:0/849921718' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]: dispatch 2024-02-12T17:12:16.237 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:15.290020+0000 mon.a (mon.0) 1973 : audit [INF] from='client.36032 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]: dispatch 2024-02-12T17:12:16.237 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:14.812617+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.16 scrub starts 2024-02-12T17:12:16.238 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:14.813494+0000 mgr.y (mgr.24481) 852 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:16.238 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:14.817349+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.16 scrub ok 2024-02-12T17:12:16.238 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:14.943990+0000 mgr.y (mgr.24481) 853 : cluster [DBG] pgmap v1202: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 16 active+undersized, 25 active+undersized+degraded, 58 active+clean; 325 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 651 op/s; 3172/16182 objects degraded (19.602%); 53/16182 objects misplaced (0.328%) 2024-02-12T17:12:16.238 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:15.214572+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:12:16.238 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:15.289692+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.33:0/849921718' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]: dispatch 2024-02-12T17:12:16.238 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:15.290020+0000 mon.a (mon.0) 1973 : audit [INF] from='client.36032 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]: dispatch 2024-02-12T17:12:16.328 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:12:16.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 5 16 15699 15683 196.019 170.938 0.00636984 0.00509584 2024-02-12T17:12:16.716 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:16.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:16.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:16.875 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:17.252 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:12:17.381 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:15.262126+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:15.814062+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:15.817744+0000 osd.4 (osd.4) 188 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:15.897054+0000 osd.7 (osd.7) 280 : cluster [DBG] 2.18 scrub starts 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:15.902103+0000 osd.7 (osd.7) 281 : cluster [DBG] 2.18 scrub ok 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:16.064295+0000 mon.a (mon.0) 1974 : audit [INF] from='client.36032 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]': finished 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:16.064383+0000 mon.a (mon.0) 1975 : cluster [DBG] osdmap e603: 8 total, 7 up, 8 in 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:16.588879+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e604: 8 total, 7 up, 8 in 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:17.013402+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.33:0/1665499772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:12:17.382 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:17.015916+0000 mon.a (mon.0) 1977 : audit [INF] from='client.25933 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:12:17.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 6 15 18273 18258 190.17 160.938 0.00535361 0.00525276 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:15.262126+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:15.814062+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:15.817744+0000 osd.4 (osd.4) 188 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:15.897054+0000 osd.7 (osd.7) 280 : cluster [DBG] 2.18 scrub starts 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:15.902103+0000 osd.7 (osd.7) 281 : cluster [DBG] 2.18 scrub ok 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:16.064295+0000 mon.a (mon.0) 1974 : audit [INF] from='client.36032 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]': finished 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:16.064383+0000 mon.a (mon.0) 1975 : cluster [DBG] osdmap e603: 8 total, 7 up, 8 in 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:16.588879+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e604: 8 total, 7 up, 8 in 2024-02-12T17:12:17.473 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:17.013402+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.33:0/1665499772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:12:17.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:17.015916+0000 mon.a (mon.0) 1977 : audit [INF] from='client.25933 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:12:17.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:15.262126+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:12:17.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:15.814062+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:12:17.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:15.817744+0000 osd.4 (osd.4) 188 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:12:17.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:15.897054+0000 osd.7 (osd.7) 280 : cluster [DBG] 2.18 scrub starts 2024-02-12T17:12:17.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:15.902103+0000 osd.7 (osd.7) 281 : cluster [DBG] 2.18 scrub ok 2024-02-12T17:12:17.474 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:16.064295+0000 mon.a (mon.0) 1974 : audit [INF] from='client.36032 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.7", "id": [1, 4, 5]}]': finished 2024-02-12T17:12:17.475 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:16.064383+0000 mon.a (mon.0) 1975 : cluster [DBG] osdmap e603: 8 total, 7 up, 8 in 2024-02-12T17:12:17.475 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:16.588879+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e604: 8 total, 7 up, 8 in 2024-02-12T17:12:17.475 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:17.013402+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.33:0/1665499772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:12:17.475 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:17.015916+0000 mon.a (mon.0) 1977 : audit [INF] from='client.25933 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:12:17.618 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:12:18.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 7 16 20741 20725 185.028 154.188 0.00258171 0.00539882 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:16.816425+0000 osd.5 (osd.5) 293 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:16.820776+0000 osd.5 (osd.5) 294 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:16.868297+0000 osd.7 (osd.7) 282 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:16.873562+0000 osd.7 (osd.7) 283 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:16.944633+0000 mgr.y (mgr.24481) 854 : cluster [DBG] pgmap v1205: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 16 active+undersized, 25 active+undersized+degraded, 58 active+clean; 694 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s; 6123/33909 objects degraded (18.057%); 53/33909 objects misplaced (0.156%) 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:17.526038+0000 mon.a (mon.0) 1978 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:17.595122+0000 mon.a (mon.0) 1979 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:17.609975+0000 mon.a (mon.0) 1980 : audit [INF] from='client.25933 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:12:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:17.610054+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e605: 8 total, 7 up, 8 in 2024-02-12T17:12:18.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:16.816425+0000 osd.5 (osd.5) 293 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:12:18.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:16.820776+0000 osd.5 (osd.5) 294 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:12:18.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:16.868297+0000 osd.7 (osd.7) 282 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:12:18.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:16.873562+0000 osd.7 (osd.7) 283 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:12:18.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:16.944633+0000 mgr.y (mgr.24481) 854 : cluster [DBG] pgmap v1205: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 16 active+undersized, 25 active+undersized+degraded, 58 active+clean; 694 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s; 6123/33909 objects degraded (18.057%); 53/33909 objects misplaced (0.156%) 2024-02-12T17:12:18.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:17.526038+0000 mon.a (mon.0) 1978 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:18.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:17.595122+0000 mon.a (mon.0) 1979 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:17.609975+0000 mon.a (mon.0) 1980 : audit [INF] from='client.25933 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:17.610054+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e605: 8 total, 7 up, 8 in 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:16.816425+0000 osd.5 (osd.5) 293 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:16.820776+0000 osd.5 (osd.5) 294 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:16.868297+0000 osd.7 (osd.7) 282 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:16.873562+0000 osd.7 (osd.7) 283 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:16.944633+0000 mgr.y (mgr.24481) 854 : cluster [DBG] pgmap v1205: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 16 active+undersized, 25 active+undersized+degraded, 58 active+clean; 694 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s; 6123/33909 objects degraded (18.057%); 53/33909 objects misplaced (0.156%) 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:17.526038+0000 mon.a (mon.0) 1978 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:17.595122+0000 mon.a (mon.0) 1979 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:17.609975+0000 mon.a (mon.0) 1980 : audit [INF] from='client.25933 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:12:18.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:17.610054+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e605: 8 total, 7 up, 8 in 2024-02-12T17:12:19.374 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:19.065755+0000 mon.a (mon.0) 1982 : cluster [WRN] Health check update: Degraded data redundancy: 11035/58212 objects degraded (18.957%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:19.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 8 16 23093 23077 180.273 147 0.00433701 0.00554192 2024-02-12T17:12:19.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:19.065755+0000 mon.a (mon.0) 1982 : cluster [WRN] Health check update: Degraded data redundancy: 11035/58212 objects degraded (18.957%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:19.065755+0000 mon.a (mon.0) 1982 : cluster [WRN] Health check update: Degraded data redundancy: 11035/58212 objects degraded (18.957%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:20.080 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:12:20.081 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:12:20.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 9 16 25381 25365 176.13 143 0.00378015 0.0056722 2024-02-12T17:12:20.465 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:18.945498+0000 mgr.y (mgr.24481) 855 : cluster [DBG] pgmap v1207: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 199 MiB/s wr, 3.19k op/s; 11035/58212 objects degraded (18.957%); 53/58212 objects misplaced (0.091%) 2024-02-12T17:12:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:18.945498+0000 mgr.y (mgr.24481) 855 : cluster [DBG] pgmap v1207: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 199 MiB/s wr, 3.19k op/s; 11035/58212 objects degraded (18.957%); 53/58212 objects misplaced (0.091%) 2024-02-12T17:12:20.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:18.945498+0000 mgr.y (mgr.24481) 855 : cluster [DBG] pgmap v1207: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 199 MiB/s wr, 3.19k op/s; 11035/58212 objects degraded (18.957%); 53/58212 objects misplaced (0.091%) 2024-02-12T17:12:20.841 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T17:12:21.132 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:12:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:20] "GET /metrics HTTP/1.1" 200 176884 "" "Prometheus/2.33.4" 2024-02-12T17:12:21.176 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:21.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:21.282 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:21.289 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:12:21.335 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:21.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 10 16 27616 27600 172.485 139.688 0.00618442 0.00579287 2024-02-12T17:12:21.557 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:21.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:21.661 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:22.084 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:22.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:22.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:22.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:20.946341+0000 mgr.y (mgr.24481) 856 : cluster [DBG] pgmap v1208: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 146 MiB/s wr, 2.33k op/s; 11035/58212 objects degraded (18.957%); 53/58212 objects misplaced (0.091%) 2024-02-12T17:12:22.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:21.592749+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e606: 8 total, 7 up, 8 in 2024-02-12T17:12:22.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:20.946341+0000 mgr.y (mgr.24481) 856 : cluster [DBG] pgmap v1208: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 146 MiB/s wr, 2.33k op/s; 11035/58212 objects degraded (18.957%); 53/58212 objects misplaced (0.091%) 2024-02-12T17:12:22.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:21.592749+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e606: 8 total, 7 up, 8 in 2024-02-12T17:12:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:22.413 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 11 16 29677 29661 168.514 128.812 0.00332267 0.00592932 2024-02-12T17:12:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:20.946341+0000 mgr.y (mgr.24481) 856 : cluster [DBG] pgmap v1208: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 146 MiB/s wr, 2.33k op/s; 11035/58212 objects degraded (18.957%); 53/58212 objects misplaced (0.091%) 2024-02-12T17:12:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:21.592749+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e606: 8 total, 7 up, 8 in 2024-02-12T17:12:22.686 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:22.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:22.792 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:22.838 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:12:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:12:22.842 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:22.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:23.067 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:23.127 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:23.245 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:23.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:23.303 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:12:23.303 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:12:23.350 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:23.401 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:23.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 12 16 31801 31785 165.532 132.75 0.00568162 0.00603502 2024-02-12T17:12:23.609 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:23.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:23.714 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:23.769 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:23.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:23.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:24.016 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:24.035 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T17:12:24.048 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:12:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:24.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:24.160 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:22.946981+0000 mgr.y (mgr.24481) 857 : cluster [DBG] pgmap v1210: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 805 B/s rd, 124 MiB/s wr, 1.98k op/s; 13288/71694 objects degraded (18.534%); 53/71694 objects misplaced (0.074%) 2024-02-12T17:12:24.173 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:24.250 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:24.254 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:22.946981+0000 mgr.y (mgr.24481) 857 : cluster [DBG] pgmap v1210: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 805 B/s rd, 124 MiB/s wr, 1.98k op/s; 13288/71694 objects degraded (18.534%); 53/71694 objects misplaced (0.074%) 2024-02-12T17:12:24.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:24.278 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:12:24.308 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:12:24.344 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:12:24.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:24.384 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:12:24.398 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:24.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 13 16 33771 33755 162.269 123.125 0.0108105 0.00615471 2024-02-12T17:12:24.427 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:12:24.432 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:24.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:24.438 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:22.946981+0000 mgr.y (mgr.24481) 857 : cluster [DBG] pgmap v1210: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 805 B/s rd, 124 MiB/s wr, 1.98k op/s; 13288/71694 objects degraded (18.534%); 53/71694 objects misplaced (0.074%) 2024-02-12T17:12:24.452 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:12:24.459 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:12:24.492 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:24.497 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:12:24.497 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:12:24.497 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:12:24.497 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:12:24.497 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:12:24.497 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:12:24.498 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:12:24.498 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:12:24.498 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:12:24.539 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:24.605 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:24.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:24.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:24.864 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:24.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:12:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:12:25.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:24.236013+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.33:0/1916103333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:25.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 14 16 35691 35675 159.25 120 0.0019315 0.00626927 2024-02-12T17:12:25.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:24.236013+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.33:0/1916103333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:25.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:24.236013+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.33:0/1916103333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:26.163 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-12T17:12:26.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:12:26.164 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-12T17:12:26.164 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T17:12:26.222 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:26.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:24.823642+0000 mgr.y (mgr.24481) 858 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:24.947721+0000 mgr.y (mgr.24481) 859 : cluster [DBG] pgmap v1211: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.43k op/s; 17427/92322 objects degraded (18.876%); 53/92322 objects misplaced (0.057%) 2024-02-12T17:12:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:25.154566+0000 mon.a (mon.0) 1984 : cluster [WRN] Health check update: Degraded data redundancy: 17427/92322 objects degraded (18.876%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:26.278 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:12:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:12:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:12:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:12:26.278 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:12:26.333 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:26.376 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:26.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 15 16 37540 37524 156.337 115.562 0.00780018 0.00639171 2024-02-12T17:12:26.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:24.823642+0000 mgr.y (mgr.24481) 858 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:26.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:24.947721+0000 mgr.y (mgr.24481) 859 : cluster [DBG] pgmap v1211: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.43k op/s; 17427/92322 objects degraded (18.876%); 53/92322 objects misplaced (0.057%) 2024-02-12T17:12:26.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:25.154566+0000 mon.a (mon.0) 1984 : cluster [WRN] Health check update: Degraded data redundancy: 17427/92322 objects degraded (18.876%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:24.823642+0000 mgr.y (mgr.24481) 858 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:24.947721+0000 mgr.y (mgr.24481) 859 : cluster [DBG] pgmap v1211: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.43k op/s; 17427/92322 objects degraded (18.876%); 53/92322 objects misplaced (0.057%) 2024-02-12T17:12:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:25.154566+0000 mon.a (mon.0) 1984 : cluster [WRN] Health check update: Degraded data redundancy: 17427/92322 objects degraded (18.876%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:26.938 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:26.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:27.045 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:27.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T17:12:27.098 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:27.306 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:27.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:27.412 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:27.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 16 16 39446 39430 154.011 119.125 0.00757869 0.00648665 2024-02-12T17:12:27.451 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:27.479 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:12:27.603 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:26.948682+0000 mgr.y (mgr.24481) 860 : cluster [DBG] pgmap v1212: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 19288/103515 objects degraded (18.633%); 53/103515 objects misplaced (0.051%) 2024-02-12T17:12:27.603 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:26.948682+0000 mgr.y (mgr.24481) 860 : cluster [DBG] pgmap v1212: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 19288/103515 objects degraded (18.633%); 53/103515 objects misplaced (0.051%) 2024-02-12T17:12:27.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:26.948682+0000 mgr.y (mgr.24481) 860 : cluster [DBG] pgmap v1212: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 19288/103515 objects degraded (18.633%); 53/103515 objects misplaced (0.051%) 2024-02-12T17:12:28.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 17 16 41367 41351 152.013 120.062 0.00749464 0.00657302 2024-02-12T17:12:28.809 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:28.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:28.842 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:12:28.842 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:12:28.915 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:28.968 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:28.997 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:29.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:29.008 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:29.103 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:29.157 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:29.290 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:29.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:29.397 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:29.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 18 16 43078 43062 149.508 106.938 0.0100142 0.00668299 2024-02-12T17:12:29.440 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:29.480 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:12:29.480 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24355.0:2083 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94508044033920 gen 6] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.519335+0000", 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 519.95554415799995, 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1633644109999999, 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24355", 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2083 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.481 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519335+0000", 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519335+0000", 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519333+0000", 2024-02-12T17:12:29.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3970000000000002e-06 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519339+0000", 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9990000000000008e-06 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519346+0000", 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.699e-06 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519352+0000", 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017726999999999999 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519529+0000", 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8419999999999999e-06 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519536+0000", 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.034153570000000001 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.553690+0000", 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0160000000000008e-06 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.553698+0000", 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.022487199 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.576185+0000", 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8740000000000007e-06 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.576195+0000", 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0615738669999999 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.637769+0000", 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120844 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.637890+0000", 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.044809557999999999 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.682699+0000", 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1633644109999999 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183796 9.9 9:9b9d44ff:::benchmark_data_smithi033_131576_object46206:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.519476+0000", 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 519.955402854, 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.163018455, 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183796 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519476+0000", 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519476+0000", 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:29.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519475+0000", 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8999999999999997e-06 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519480+0000", 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1370000000000002e-06 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519488+0000", 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1850000000000001e-06 2024-02-12T17:12:29.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519494+0000", 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0197000000000004e-05 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519584+0000", 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012856599999999999 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519713+0000", 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015817699999999999 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,5", 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.519871+0000", 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0096412979999999995 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.529512+0000", 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082318930000000005 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.537744+0000", 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-05 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.537754+0000", 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1369527939999999 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.674707+0000", 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000119757 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.674827+0000", 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075961400000000004 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.682423+0000", 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1493000000000001e-05 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.682495+0000", 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.163018455 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.493 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183799 9.3 9:c6d8c0ff:::benchmark_data_smithi033_131576_object46209:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.521648+0000", 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 519.95323088299995, 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1647798149999999, 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183799 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521648+0000", 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521648+0000", 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521647+0000", 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8599999999999998e-06 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521653+0000", 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1588e-05 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521664+0000", 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3470000000000004e-06 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.521670+0000", 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056849500000000002 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522239+0000", 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120591 2024-02-12T17:12:29.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522360+0000", 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015625200000000001 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,5", 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.522516+0000", 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014359825 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.536876+0000", 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8340000000000001e-06 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.536882+0000", 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1419698709999999 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.678852+0000", 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012575800000000001 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.678978+0000", 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.007399203 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.686377+0000", 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0636000000000001e-05 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.686428+0000", 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1647798149999999 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183801 9.9 9:96e80f0d:::benchmark_data_smithi033_131576_object46211:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.526679+0000", 2024-02-12T17:12:29.499 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 519.94820040499997, 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1562766440000001, 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183801 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526679+0000", 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526679+0000", 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526677+0000", 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8290000000000002e-06 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526683+0000", 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4666e-05 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526698+0000", 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9229999999999996e-06 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526705+0000", 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2431000000000001e-05 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526747+0000", 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6534999999999997e-05 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526834+0000", 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010707 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,4,5", 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.526941+0000", 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010838854 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.537780+0000", 2024-02-12T17:12:29.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0490000000000004e-06 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.537788+0000", 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.137077535 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.674865+0000", 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010332600000000001 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.674969+0000", 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079168629999999997 2024-02-12T17:12:29.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.682886+0000", 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9560000000000005e-05 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.682955+0000", 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1562766440000001 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183806 9.2 9:401f2b8c:::benchmark_data_smithi033_131576_object46216:head [delete] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.566880+0000", 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 519.90799888599997, 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1056333119999999, 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183806 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566880+0000", 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566880+0000", 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566878+0000", 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.486e-06 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566885+0000", 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7730000000000005e-06 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566894+0000", 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4620000000000001e-06 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566900+0000", 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494170000000001 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.573450+0000", 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3419999999999997e-06 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.573459+0000", 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.085010405 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.658470+0000", 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012053300000000001 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.658590+0000", 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015071 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,3", 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.658741+0000", 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006408815 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.665150+0000", 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8158000000000001e-05 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.665208+0000", 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045370510000000003 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.669745+0000", 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025681050000000002 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672313+0000", 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1764e-05 2024-02-12T17:12:29.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672375+0000", 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138634 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672513+0000", 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1056333119999999 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.15459.0:183808 9.2 9:4994a024:::benchmark_data_smithi033_131576_object46218:head [delete] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:49.566913+0000", 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 519.90796638200004, 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.105857358, 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.15459", 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 183808 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566913+0000", 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566913+0000", 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566912+0000", 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1810000000000003e-06 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566916+0000", 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4219999999999999e-06 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566921+0000", 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4379999999999997e-06 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.566926+0000", 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065533589999999999 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.573479+0000", 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3410000000000002e-06 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:49.573484+0000", 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.087181733 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.660666+0000", 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012873100000000001 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.660795+0000", 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015155499999999999 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,3", 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.660946+0000", 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042465819999999996 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.665193+0000", 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1631e-05 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.665215+0000", 2024-02-12T17:12:29.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045630260000000004 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.669778+0000", 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027785520000000001 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672556+0000", 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1248999999999996e-05 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672617+0000", 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000152478 2024-02-12T17:12:29.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.672770+0000", 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.105857358 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.24355.0:2085 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94508044033920 gen 7] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:03:50.561581+0000", 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 518.91329757799997, 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1212566980000001, 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:12:29.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.24355", 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3785705817", 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2085 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561581+0000", 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561581+0000", 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:12:29.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561579+0000", 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9589999999999995e-06 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561587+0000", 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1561e-05 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561598+0000", 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2279999999999998e-06 2024-02-12T17:12:29.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.561606+0000", 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016483583999999999 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.578090+0000", 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1689999999999992e-06 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:50.578098+0000", 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0600229800000001 2024-02-12T17:12:29.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.638121+0000", 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7311000000000007e-05 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.638189+0000", 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.044649568000000001 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:03:51.682838+0000", 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1212566980000001 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.523 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:121711 10.0 10:07955a90:::benchmark_data_smithi033_160181_object121710:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e480)", 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:51.065001+0000", 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 278.40987800599999, 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99851951800000005, 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 121711 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.065001+0000", 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.065001+0000", 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.065000+0000", 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9873000000000001e-05 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.065060+0000", 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3989999999999998e-06 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.065067+0000", 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4610000000000001e-06 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.065072+0000", 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059086828000000001 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.124159+0000", 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5570000000000001e-06 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.124168+0000", 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93926208600000005 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.063430+0000", 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0227000000000002e-05 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.063520+0000", 2024-02-12T17:12:29.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99851951800000005 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:121726 10.11 10:88750a9f:::benchmark_data_smithi033_160181_object121725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:51.067624+0000", 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 278.40725486399998, 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0345230029999999, 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.528 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 121726 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.067624+0000", 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.067624+0000", 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999962 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.067620+0000", 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4978e-05 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.067685+0000", 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9633e-05 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.067705+0000", 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7830000000000004e-06 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.067714+0000", 2024-02-12T17:12:29.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092611870000000006 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.076975+0000", 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8230000000000001e-06 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:51.076984+0000", 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.012772126 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.089756+0000", 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7099000000000002e-05 2024-02-12T17:12:29.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.089833+0000", 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017032700000000001 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.090003+0000", 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001505773 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.091509+0000", 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00027710499999999998 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.091786+0000", 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010303069999999999 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.102089+0000", 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3800999999999997e-05 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.102123+0000", 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.393e-05 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:52.102147+0000", 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0345230029999999 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:123098 10.13 10.aa7455d3 (undecoded) ondisk+retry+write+known_if_redirected e482)", 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:53.075600+0000", 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 276.399278616, 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99416470700000004, 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.534 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 123098 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.075600+0000", 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.075600+0000", 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.075599+0000", 2024-02-12T17:12:29.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2896000000000002e-05 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.075662+0000", 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2309e-05 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.075674+0000", 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1200000000000002e-06 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.075682+0000", 2024-02-12T17:12:29.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9586999999999999e-05 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.075722+0000", 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.959e-06 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.075728+0000", 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99402557800000002 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.069753+0000", 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1601e-05 2024-02-12T17:12:29.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.069765+0000", 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99416470700000004 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:123122 10.13 10.6c9c1413 (undecoded) ondisk+write+known_if_redirected e482)", 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:53.099481+0000", 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 276.37539813900003, 2024-02-12T17:12:29.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97032434899999997, 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 123122 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.099481+0000", 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.099481+0000", 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.099480+0000", 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0372999999999999e-05 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.099540+0000", 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7390000000000002e-06 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.099547+0000", 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3499999999999999e-06 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.099551+0000", 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9895999999999999e-05 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.099591+0000", 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8100000000000003e-06 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.099597+0000", 2024-02-12T17:12:29.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97020114999999996 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.069798+0000", 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1940000000000001e-06 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.069805+0000", 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97032434899999997 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:123132 10.13 10.d2149573 (undecoded) ondisk+write+known_if_redirected e482)", 2024-02-12T17:12:29.543 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:07:53.110341+0000", 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 276.36453784299999, 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95949594400000005, 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 123132 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.110341+0000", 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.110341+0000", 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.110339+0000", 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4023000000000002e-05 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.110383+0000", 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4085e-05 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.110397+0000", 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1510000000000004e-06 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.110405+0000", 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6761000000000002e-05 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.110452+0000", 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1120000000000002e-06 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:53.110461+0000", 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95936775100000005 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.069829+0000", 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5830000000000007e-06 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:07:54.069837+0000", 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95949594400000005 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:174491 10.c 10:377f7530:::benchmark_data_smithi033_160181_object51319:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:20.547771+0000", 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 188.927107836, 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.02031161, 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 174491 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547771+0000", 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547771+0000", 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547769+0000", 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3609999999999997e-06 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547776+0000", 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7280000000000007e-06 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547785+0000", 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547790+0000", 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8473999999999998e-05 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547828+0000", 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1046999999999998e-05 2024-02-12T17:12:29.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.547919+0000", 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024493399999999998 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,3", 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.548164+0000", 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011109290000000001 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.549275+0000", 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056100059999999998 2024-02-12T17:12:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.554885+0000", 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0041192240000000003 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.559005+0000", 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017330979 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.576335+0000", 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1763000000000001e-05 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.576347+0000", 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99164042600000002 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.567988+0000", 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5088999999999997e-05 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.568083+0000", 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.02031161 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:174507 10.1c 10:3d1ef335:::benchmark_data_smithi033_160181_object51335:head [delete] snapc 0=[] ondisk+write+known_if_redirected e513)", 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:20.629865+0000", 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 188.84501364900001, 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93946509700000003, 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 174507 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.629865+0000", 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.629865+0000", 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.629862+0000", 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1224e-05 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.629873+0000", 2024-02-12T17:12:29.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9701e-05 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.629893+0000", 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0699999999999996e-06 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.629902+0000", 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0269999999999997e-05 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.629962+0000", 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1119e-05 2024-02-12T17:12:29.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:20.629973+0000", 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.91849143300000002 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.548465+0000", 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014890300000000001 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.548614+0000", 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030395500000000001 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2,3,6", 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.548918+0000", 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012750473 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.561668+0000", 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064640050000000001 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.568132+0000", 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7803999999999999e-05 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.568170+0000", 2024-02-12T17:12:29.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7876999999999997e-05 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.568208+0000", 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091347900000000003 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.569121+0000", 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9012999999999999e-05 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.569180+0000", 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000150044 2024-02-12T17:12:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.569330+0000", 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93946509700000003 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:174528 10.10 10:0eea8b7b:::benchmark_data_smithi033_160181_object51356:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:21.631594+0000", 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 187.84328485, 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98311103499999997, 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 174528 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.631594+0000", 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.631594+0000", 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.631592+0000", 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2020000000000004e-06 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.631600+0000", 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.188e-05 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.631612+0000", 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9599999999999997e-06 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.631618+0000", 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8640000000000002e-05 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.631666+0000", 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4569999999999994e-06 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:21.631676+0000", 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94676047399999996 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.578436+0000", 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124805 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.578561+0000", 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00031168400000000001 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5,6", 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.578873+0000", 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031792069999999999 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.582052+0000", 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002067995 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.584120+0000", 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020121940000000001 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.586132+0000", 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.028375351 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.614507+0000", 2024-02-12T17:12:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2728e-05 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.614570+0000", 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000135038 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:22.614705+0000", 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98311103499999997 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:235925 10.10 10:0cd20b3f:::benchmark_data_smithi033_160181_object112753:head [delete] snapc 0=[] ondisk+write+known_if_redirected e557)", 2024-02-12T17:12:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:03.463200+0000", 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.011679209999997, 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0324135839999999, 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 235925 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.463200+0000", 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.463200+0000", 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.463198+0000", 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9190000000000002e-06 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.463205+0000", 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2836e-05 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.463217+0000", 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5980000000000002e-06 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.463224+0000", 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1946000000000001e-05 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.463266+0000", 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.781e-06 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.463276+0000", 2024-02-12T17:12:29.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.01713589 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.480412+0000", 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120303 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.480532+0000", 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000374297 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 5,6", 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.480906+0000", 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014475406999999999 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495382+0000", 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4838000000000003e-05 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495467+0000", 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6888000000000001e-05 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495533+0000", 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4484000000000003e-05 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495578+0000", 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5429000000000002e-05 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495613+0000", 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0324135839999999 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:235941 10.10 10:0df72f5e:::benchmark_data_smithi033_160181_object112769:head [delete] snapc 0=[] ondisk+write+known_if_redirected e557)", 2024-02-12T17:12:29.572 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:03.517690+0000", 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.957189478999993, 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97800865100000001, 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 235941 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.517690+0000", 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.517690+0000", 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.517689+0000", 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1109999999999999e-06 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.517692+0000", 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3689999999999998e-06 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.517697+0000", 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0640000000000004e-06 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.517701+0000", 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7997000000000002e-05 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.517739+0000", 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2349999999999996e-06 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.517745+0000", 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96463816000000002 2024-02-12T17:12:29.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.482384+0000", 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010077399999999999 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.482484+0000", 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016963500000000001 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 5,6", 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.482654+0000", 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012735318000000001 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495389+0000", 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113738 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495503+0000", 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014283899999999999 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495646+0000", 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3578999999999999e-05 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495670+0000", 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8642000000000001e-05 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495698+0000", 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97800865100000001 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:235947 10.10 10:0b6ed6b5:::benchmark_data_smithi033_160181_object112775:head [delete] snapc 0=[] ondisk+write+known_if_redirected e557)", 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:03.547333+0000", 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.927546145999997, 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95005526399999995, 2024-02-12T17:12:29.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 235947 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.547333+0000", 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.547333+0000", 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.547331+0000", 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1299999999999998e-06 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.547337+0000", 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1736000000000001e-05 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.547349+0000", 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5549999999999999e-06 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.547356+0000", 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9677e-05 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.547395+0000", 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9379999999999997e-06 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:03.547401+0000", 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93628937599999995 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.483691+0000", 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7993999999999998e-05 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.483779+0000", 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015845599999999999 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 5,6", 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.483937+0000", 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01145734 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495394+0000", 2024-02-12T17:12:29.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00034479500000000001 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.495739+0000", 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015914449999999999 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.497331+0000", 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4498000000000002e-05 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.497355+0000", 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2866e-05 2024-02-12T17:12:29.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:04.497388+0000", 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95005526399999995 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:246177 10.1f 10:fcd04f19:::benchmark_data_smithi033_160181_object123005:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e569)", 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:20.151118+0000", 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.323761207999993, 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0918610040000001, 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:12:29.584 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 246177 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151118+0000", 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151118+0000", 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151117+0000", 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1620000000000001e-06 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151121+0000", 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.64e-06 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151128+0000", 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6199999999999998e-06 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151132+0000", 2024-02-12T17:12:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5536000000000004e-05 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151198+0000", 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6799999999999993e-06 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151206+0000", 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.08207377 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.233279+0000", 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9981000000000006e-05 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.233349+0000", 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095798459999999995 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.242929+0000", 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9617999999999998e-05 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.242979+0000", 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0918610040000001 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.588 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:246181 10.1f 10:fcdd8d24:::benchmark_data_smithi033_160181_object123009:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e569)", 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:20.151145+0000", 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.323734154999997, 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.096291109, 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 246181 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:12:29.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151145+0000", 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151145+0000", 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151144+0000", 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9280000000000002e-06 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151147+0000", 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4700000000000001e-06 2024-02-12T17:12:29.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151153+0000", 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9140000000000001e-06 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151157+0000", 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00032654100000000002 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151483+0000", 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6610000000000003e-06 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151490+0000", 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0818926790000001 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.233382+0000", 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115574 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.233498+0000", 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396246 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.233894+0000", 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066555260000000002 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.240550+0000", 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044043770000000001 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.244954+0000", 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022997880000000001 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:12:29.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.247254+0000", 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4873999999999997e-05 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.247309+0000", 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000127143 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.247436+0000", 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.096291109 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:12:29.594 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:12:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:29.595 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:29.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:29.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:29.748 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:29.781 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:29.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:29.887 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:29.953 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:30.021 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:12:30.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:28.949710+0000 mgr.y (mgr.24481) 861 : cluster [DBG] pgmap v1213: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s; 22828/121293 objects degraded (18.821%); 53/121293 objects misplaced (0.044%) 2024-02-12T17:12:30.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:29.139550+0000 mon.c (mon.2) 282 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:12:30.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:29.886704+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.33:0/2253572665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:30.158 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:29.887239+0000 mon.a (mon.0) 1985 : audit [INF] from='client.25939 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:30.158 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:28.949710+0000 mgr.y (mgr.24481) 861 : cluster [DBG] pgmap v1213: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s; 22828/121293 objects degraded (18.821%); 53/121293 objects misplaced (0.044%) 2024-02-12T17:12:30.158 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:29.139550+0000 mon.c (mon.2) 282 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:12:30.158 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:29.886704+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.33:0/2253572665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:30.158 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:29.887239+0000 mon.a (mon.0) 1985 : audit [INF] from='client.25939 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:28.949710+0000 mgr.y (mgr.24481) 861 : cluster [DBG] pgmap v1213: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s; 22828/121293 objects degraded (18.821%); 53/121293 objects misplaced (0.044%) 2024-02-12T17:12:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:29.139550+0000 mon.c (mon.2) 282 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:12:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:29.886704+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.33:0/2253572665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:29.887239+0000 mon.a (mon.0) 1985 : audit [INF] from='client.25939 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:30.285 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T17:12:30.323 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:30.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:30.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 19 16 44582 44566 146.586 94 0.00237658 0.00681432 2024-02-12T17:12:30.434 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:30.460 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:30.560 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:30.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:30.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:30.708 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:30.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:12:31.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:30.002631+0000 mon.a (mon.0) 1986 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:31.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:30.010500+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25939 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:12:31.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:30.010579+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e607: 8 total, 7 up, 8 in 2024-02-12T17:12:31.029 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:12:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:30] "GET /metrics HTTP/1.1" 200 176884 "" "Prometheus/2.33.4" 2024-02-12T17:12:31.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:30.002631+0000 mon.a (mon.0) 1986 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:31.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:30.010500+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25939 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:12:31.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:30.010579+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e607: 8 total, 7 up, 8 in 2024-02-12T17:12:31.098 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:31.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:31.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:31.263 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:31.295 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:31.300 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:30.002631+0000 mon.a (mon.0) 1986 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:31.300 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:30.010500+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25939 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:12:31.300 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:30.010579+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e607: 8 total, 7 up, 8 in 2024-02-12T17:12:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:31.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:31.402 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:31.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:12:31.414040+0000 min lat: 0.00107612 max lat: 0.102422 avg lat: 0.00693962 2024-02-12T17:12:31.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:12:31.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 20 16 46099 46083 143.997 94.8125 0.0115269 0.00693962 2024-02-12T17:12:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:31.512 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:31.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:31.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:31.657 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:32.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:32.236 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:32.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:30.950588+0000 mgr.y (mgr.24481) 862 : cluster [DBG] pgmap v1215: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 22828/121293 objects degraded (18.821%); 53/121293 objects misplaced (0.044%) 2024-02-12T17:12:32.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:31.586409+0000 mon.a (mon.0) 1989 : cluster [WRN] Health check update: Degraded data redundancy: 22828/121293 objects degraded (18.821%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:32.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:31.592748+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e608: 8 total, 7 up, 8 in 2024-02-12T17:12:32.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:30.950588+0000 mgr.y (mgr.24481) 862 : cluster [DBG] pgmap v1215: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 22828/121293 objects degraded (18.821%); 53/121293 objects misplaced (0.044%) 2024-02-12T17:12:32.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:31.586409+0000 mon.a (mon.0) 1989 : cluster [WRN] Health check update: Degraded data redundancy: 22828/121293 objects degraded (18.821%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:32.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:31.592748+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e608: 8 total, 7 up, 8 in 2024-02-12T17:12:32.287 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:32.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 21 16 47654 47638 141.768 97.1875 0.00454596 0.00704838 2024-02-12T17:12:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:30.950588+0000 mgr.y (mgr.24481) 862 : cluster [DBG] pgmap v1215: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 22828/121293 objects degraded (18.821%); 53/121293 objects misplaced (0.044%) 2024-02-12T17:12:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:31.586409+0000 mon.a (mon.0) 1989 : cluster [WRN] Health check update: Degraded data redundancy: 22828/121293 objects degraded (18.821%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:31.592748+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e608: 8 total, 7 up, 8 in 2024-02-12T17:12:32.497 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:32.498 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:32.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:32.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:32.661 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:32.704 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:32.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:32.798 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:12:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:12:32.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:32.862 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:32.996 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:33.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:33.104 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:33.171 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:33.284 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:33.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:33.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:33.414 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 22 16 49221 49205 139.775 97.9375 0.0170057 0.00714882 2024-02-12T17:12:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:33.641 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:33.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:32.527139+0000 mon.a (mon.0) 1991 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:33.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:32.953200+0000 mgr.y (mgr.24481) 863 : cluster [DBG] pgmap v1217: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s; 24433/131007 objects degraded (18.650%); 53/131007 objects misplaced (0.040%) 2024-02-12T17:12:33.748 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:33.791 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:34.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:32.527139+0000 mon.a (mon.0) 1991 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:34.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:32.953200+0000 mgr.y (mgr.24481) 863 : cluster [DBG] pgmap v1217: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s; 24433/131007 objects degraded (18.650%); 53/131007 objects misplaced (0.040%) 2024-02-12T17:12:34.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:32.527139+0000 mon.a (mon.0) 1991 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:34.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:32.953200+0000 mgr.y (mgr.24481) 863 : cluster [DBG] pgmap v1217: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s; 24433/131007 objects degraded (18.650%); 53/131007 objects misplaced (0.040%) 2024-02-12T17:12:34.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 23 16 50860 50844 138.152 102.438 0.0122971 0.00723368 2024-02-12T17:12:34.535 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:12:35.235 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:35.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:12:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:12:35.253 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:35.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:35.360 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:35.400 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:35.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 24 16 52442 52426 136.515 98.875 0.012216 0.00732034 2024-02-12T17:12:36.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:34.836468+0000 mgr.y (mgr.24481) 864 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:36.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:34.954314+0000 mgr.y (mgr.24481) 865 : cluster [DBG] pgmap v1218: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 27288/145383 objects degraded (18.770%); 53/145383 objects misplaced (0.036%) 2024-02-12T17:12:36.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 25 16 54039 54023 135.046 99.8125 0.00812774 0.0074007 2024-02-12T17:12:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:34.836468+0000 mgr.y (mgr.24481) 864 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:34.954314+0000 mgr.y (mgr.24481) 865 : cluster [DBG] pgmap v1218: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 27288/145383 objects degraded (18.770%); 53/145383 objects misplaced (0.036%) 2024-02-12T17:12:36.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:34.836468+0000 mgr.y (mgr.24481) 864 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:36.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:34.954314+0000 mgr.y (mgr.24481) 865 : cluster [DBG] pgmap v1218: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 27288/145383 objects degraded (18.770%); 53/145383 objects misplaced (0.036%) 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 608, 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "cf6ad642-c9c5-11ee-95b9-87774f69a715", 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2024-02-12T16:43:50.282468+0000", 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2024-02-12T17:12:31.586749+0000", 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2024-02-12T17:10:02.814427+0000", 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2024-02-12T17:11:51.074104+0000", 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5806080, 2024-02-12T17:12:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "nodeep-scrub", 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "noscrub", 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 18, 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T17:12:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 11, 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 8, 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "luminous", 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "pacific", 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "device_health_metrics", 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:47:33.056945+0000", 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:12:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2024-02-12T17:12:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "71", 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:12:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:12:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:12:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:12:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_devicehealth": {} 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".rgw.root", 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:37.647343+0000", 2024-02-12T17:12:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:12:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 29, 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 29, 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "2.1d", 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 599, 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 598, 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 598, 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "128'2" 2024-02-12T17:12:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "599", 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "599", 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "597", 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:12:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:12:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:12:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "iscsi", 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:38.076200+0000", 2024-02-12T17:12:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:12:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 3, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 3, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 3, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 3, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 3, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "527", 2024-02-12T17:12:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:12:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:12:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:12:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 4, 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.log", 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:39.589565+0000", 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:12:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 32, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 32, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 32, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 32, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 32, 2024-02-12T17:12:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "61", 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:12:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:12:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:12:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:12:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:12:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:12:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:12:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:12:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:12:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:12:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:12:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:12:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:12:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:12:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:12:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:12:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:12:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:12:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 5, 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.control", 2024-02-12T17:12:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:41.623856+0000", 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:12:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "5.10", 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 429, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 428, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 428, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "380'16" 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "433", 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "429", 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "431", 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:12:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:12:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:12:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 6, 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "default.rgw.meta", 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T16:49:43.693806+0000", 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:12:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 22, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 22, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 22, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 22, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 22, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "6.16", 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 396, 2024-02-12T17:12:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 395, 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 395, 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "396", 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "396", 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "394", 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:12:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:12:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:12:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_bias": 4 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw": {} 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 11, 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_4", 2024-02-12T17:12:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-02-12T17:11:57.982074+0000", 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 3, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 2, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2024-02-12T17:12:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "602", 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-02-12T17:12:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-02-12T17:12:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-02-12T17:12:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rados": {} 2024-02-12T17:12:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "5dd172eb-ee9d-4700-9b80-60f91c5f2cb7", 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0.2548370361328125, 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 8, 2024-02-12T17:12:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 533, 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2545411201 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/2545411201", 2024-02-12T17:12:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6802/2545411201", 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6804/2545411201", 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6803/2545411201", 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "71231bc4-1716-431d-bd88-f6a3cad0bd24", 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 0, 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:12:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 13, 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 516, 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 533, 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:12:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1723231708 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6805/1723231708", 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6806/1723231708", 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6808/1723231708", 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6807/1723231708", 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "exists" 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8a9c7127-d17c-4c44-b666-4211006de47c", 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:12:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0, 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 18, 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:12:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3126580659 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3126580659", 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6810/3126580659", 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6812/3126580659", 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6811/3126580659", 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3cc40671-c8f7-412c-a0f1-987f48a016b9", 2024-02-12T17:12:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 26, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 599, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3941314957 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6813/3941314957", 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6814/3941314957", 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6816/3941314957", 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6815/3941314957", 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0498129c-861c-4f7e-b8ba-219949da3130", 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.54083251953125, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 32, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6800", 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6801", 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6803", 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6802", 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 544126779 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6800/544126779", 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6801/544126779", 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6803/544126779", 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6802/544126779", 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "05718a1e-1164-4b54-a493-23c1cd3adb50", 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 38, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 604, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:12:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6804", 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6805", 2024-02-12T17:12:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6807", 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6806", 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3229884005 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6804/3229884005", 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6805/3229884005", 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6807/3229884005", 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6806/3229884005", 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "dc48640f-8f27-405c-a9a4-cf1e3e6c8821", 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 45, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:12:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6808", 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6809", 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:12:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6811", 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6810", 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 507305558 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6808/507305558", 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6809/507305558", 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6811/507305558", 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6810/507305558", 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "7d55aa3f-06b6-4fc3-97fd-62b535f78b66", 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 54, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 597, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-02-12T17:12:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6812", 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6813", 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6815", 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-02-12T17:12:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.94:6814", 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3401977873 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.94:6812/3401977873", 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.94:6813/3401977873", 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.94:6815/3401977873", 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.94:6814/3401977873", 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:12:06.439494+0000", 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-02-12T17:12:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:09:44.771155+0000", 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 532 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:12:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:54.538230+0000", 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:33.499743+0000", 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-02-12T17:12:36.666 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:51.353050+0000", 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:12:36.667 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:35.185056+0000", 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:49.922567+0000", 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.668 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138314316775423, 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-02-12T17:11:49.490108+0000", 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [ 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.7", 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: 1, 2024-02-12T17:12:36.669 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [], 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [ 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.5", 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.670 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.6", 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: 3 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.8", 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.671 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.d", 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.11", 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.16", 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: 3 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.18", 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.673 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.19", 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.1a", 2024-02-12T17:12:36.674 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "2.1c", 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: 7 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.2", 2024-02-12T17:12:36.675 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.9", 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.676 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.b", 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.d", 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.e", 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: 3 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.f", 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: 7 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.13", 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.18", 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: 3 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.1b", 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "4.1d", 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:12:36.680 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.0", 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.5", 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: 2 2024-02-12T17:12:36.681 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.7", 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: 0 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.c", 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: 0, 2024-02-12T17:12:36.682 INFO:teuthology.orchestra.run.smithi033.stdout: 6 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.10", 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: 7, 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: 6, 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "5.13", 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: 3, 2024-02-12T17:12:36.683 INFO:teuthology.orchestra.run.smithi033.stdout: 5, 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: 7 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "6.e", 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: 4, 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: 2, 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: 5 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": { 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/2829464018": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:12:36.684 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/1835128904": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2277321750": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2116602769": "2024-02-13T16:50:02.257936+0000", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3967118778": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2298361247": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/218227980": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/2070582473": "2024-02-13T16:44:22.727241+0000", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:0/4291575956": "2024-02-13T16:44:46.313346+0000", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "172.21.15.33:6800/3157562110": "2024-02-13T16:44:22.727241+0000" 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2024-02-12T17:12:36.685 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "ruleset-failure-domain": "osd", 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [], 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2024-02-12T17:12:36.686 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2024-02-12T17:12:36.687 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:36.687 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:12:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:36.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:37.075 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:37.120 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:37.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:36.583850+0000 mon.c (mon.2) 283 : audit [DBG] from='client.? v1:172.21.15.33:0/3612639692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:12:37.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:36.587901+0000 mon.a (mon.0) 1992 : cluster [WRN] Health check update: Degraded data redundancy: 27288/145383 objects degraded (18.770%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:37.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:36.583850+0000 mon.c (mon.2) 283 : audit [DBG] from='client.? v1:172.21.15.33:0/3612639692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:12:37.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:36.587901+0000 mon.a (mon.0) 1992 : cluster [WRN] Health check update: Degraded data redundancy: 27288/145383 objects degraded (18.770%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:37.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:36.583850+0000 mon.c (mon.2) 283 : audit [DBG] from='client.? v1:172.21.15.33:0/3612639692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T17:12:37.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:36.587901+0000 mon.a (mon.0) 1992 : cluster [WRN] Health check update: Degraded data redundancy: 27288/145383 objects degraded (18.770%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:37.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 26 16 55475 55459 133.304 89.75 0.022441 0.00749738 2024-02-12T17:12:37.864 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.36023.0:55576 11.2 e608/597 11:45baea21:::benchmark_data_smithi033_188107_object55575:head v 608'3522, mlcod=608'3519)", 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:12:37.483174+0000", 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.001493819, 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015208369999999999, 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:12:37.865 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.483174+0000", 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.483174+0000", 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.483172+0000", 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1704000000000001e-05 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.483194+0000", 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0273e-05 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.483204+0000", 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6450000000000005e-06 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.483212+0000", 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9721e-05 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:12:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.483261+0000", 2024-02-12T17:12:37.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3189000000000001e-05 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.483285+0000", 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110987 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.870 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.36023.0:55582 11.2 e608/597)", 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:12:37.484697+0000", 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4294967295.9999709, 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8186e-05, 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.484697+0000", 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:12:37.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:12:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.484697+0000", 2024-02-12T17:12:37.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:12:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:12:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.484695+0000", 2024-02-12T17:12:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6426000000000001e-05 2024-02-12T17:12:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:12:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.484721+0000", 2024-02-12T17:12:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1770000000000001e-05 2024-02-12T17:12:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.484733+0000", 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1940000000000003e-06 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:12:37.484739+0000", 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2880000000000003e-05 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:37.875 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:12:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:12:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:12:37.876 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 2 2024-02-12T17:12:37.876 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:12:38.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 27 16 56994 56978 131.883 94.9375 0.00650765 0.00757792 2024-02-12T17:12:38.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:36.955074+0000 mgr.y (mgr.24481) 866 : cluster [DBG] pgmap v1219: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.38k op/s; 28785/154314 objects degraded (18.654%); 53/154314 objects misplaced (0.034%) 2024-02-12T17:12:38.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:37.803723+0000 mon.a (mon.0) 1993 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:38.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:36.955074+0000 mgr.y (mgr.24481) 866 : cluster [DBG] pgmap v1219: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.38k op/s; 28785/154314 objects degraded (18.654%); 53/154314 objects misplaced (0.034%) 2024-02-12T17:12:38.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:37.803723+0000 mon.a (mon.0) 1993 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:36.955074+0000 mgr.y (mgr.24481) 866 : cluster [DBG] pgmap v1219: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.38k op/s; 28785/154314 objects degraded (18.654%); 53/154314 objects misplaced (0.034%) 2024-02-12T17:12:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:37.803723+0000 mon.a (mon.0) 1993 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:38.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T17:12:38.882 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:38.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:38.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:38.992 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:39.039 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:39.095 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:39.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:39.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:39.246 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:39.280 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:39.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:39.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:39.387 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:39.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 28 16 58470 58454 130.467 92.25 0.00465738 0.00766018 2024-02-12T17:12:39.436 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:40.199 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:38.957775+0000 mgr.y (mgr.24481) 867 : cluster [DBG] pgmap v1220: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 31694/169077 objects degraded (18.745%); 53/169077 objects misplaced (0.031%) 2024-02-12T17:12:40.199 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:39.700077+0000 mon.a (mon.0) 1994 : audit [DBG] from='client.? v1:172.21.15.33:0/3387902028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:40.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:38.957775+0000 mgr.y (mgr.24481) 867 : cluster [DBG] pgmap v1220: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 31694/169077 objects degraded (18.745%); 53/169077 objects misplaced (0.031%) 2024-02-12T17:12:40.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:39.700077+0000 mon.a (mon.0) 1994 : audit [DBG] from='client.? v1:172.21.15.33:0/3387902028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:40.200 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:40.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:40.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:40.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 29 16 59928 59912 129.11 91.125 0.00856319 0.00774094 2024-02-12T17:12:40.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:38.957775+0000 mgr.y (mgr.24481) 867 : cluster [DBG] pgmap v1220: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 31694/169077 objects degraded (18.745%); 53/169077 objects misplaced (0.031%) 2024-02-12T17:12:40.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:39.700077+0000 mon.a (mon.0) 1994 : audit [DBG] from='client.? v1:172.21.15.33:0/3387902028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:12:41.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:12:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:40] "GET /metrics HTTP/1.1" 200 176892 "" "Prometheus/2.33.4" 2024-02-12T17:12:41.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 30 16 61490 61474 128.06 97.625 0.00795445 0.00780426 2024-02-12T17:12:41.496 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:41.500 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:41.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:41.529 INFO:tasks.thrashosds.thrasher:j is {'epoch': 608, 'fsid': 'cf6ad642-c9c5-11ee-95b9-87774f69a715', 'created': '2024-02-12T16:43:50.282468+0000', 'modified': '2024-02-12T17:12:31.586749+0000', 'last_up_change': '2024-02-12T17:10:02.814427+0000', 'last_in_change': '2024-02-12T17:11:51.074104+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T16:47:33.056945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-12T16:49:37.647343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 29, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 29, 'last_pg_merge_meta': {'source_pgid': '2.1d', 'ready_epoch': 599, 'last_epoch_started': 598, 'last_epoch_clean': 598, 'source_version': "0'0", 'target_version': "128'2"}, 'last_change': '599', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '599', 'last_force_op_resend_preluminous': '597', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-12T16:49:38.076200+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '527', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-12T16:49:39.589565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-12T16:49:41.623856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.10', 'ready_epoch': 429, 'last_epoch_started': 428, 'last_epoch_clean': 428, 'source_version': "0'0", 'target_version': "380'16"}, 'last_change': '433', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '429', 'last_force_op_resend_preluminous': '431', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-12T16:49:43.693806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '6.16', 'ready_epoch': 396, 'last_epoch_started': 395, 'last_epoch_clean': 395, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '396', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '396', 'last_force_op_resend_preluminous': '394', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-02-12T17:11:57.982074+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '602', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '5dd172eb-ee9d-4700-9b80-60f91c5f2cb7', 'up': 1, 'in': 1, 'weight': 0.2548370361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 533, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 2545411201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6802', 'nonce': 2545411201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6804', 'nonce': 2545411201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 2545411201}]}, 'public_addr': '172.21.15.33:6801/2545411201', 'cluster_addr': '172.21.15.33:6802/2545411201', 'heartbeat_back_addr': '172.21.15.33:6804/2545411201', 'heartbeat_front_addr': '172.21.15.33:6803/2545411201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71231bc4-1716-431d-bd88-f6a3cad0bd24', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 516, 'down_at': 533, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1723231708}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6806', 'nonce': 1723231708}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6808', 'nonce': 1723231708}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1723231708}]}, 'public_addr': '172.21.15.33:6805/1723231708', 'cluster_addr': '172.21.15.33:6806/1723231708', 'heartbeat_back_addr': '172.21.15.33:6808/1723231708', 'heartbeat_front_addr': '172.21.15.33:6807/1723231708', 'state': ['exists']}, {'osd': 2, 'uuid': '8a9c7127-d17c-4c44-b666-4211006de47c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3126580659}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6810', 'nonce': 3126580659}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6812', 'nonce': 3126580659}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3126580659}]}, 'public_addr': '172.21.15.33:6809/3126580659', 'cluster_addr': '172.21.15.33:6810/3126580659', 'heartbeat_back_addr': '172.21.15.33:6812/3126580659', 'heartbeat_front_addr': '172.21.15.33:6811/3126580659', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3cc40671-c8f7-412c-a0f1-987f48a016b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 599, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3941314957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6814', 'nonce': 3941314957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6816', 'nonce': 3941314957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3941314957}]}, 'public_addr': '172.21.15.33:6813/3941314957', 'cluster_addr': '172.21.15.33:6814/3941314957', 'heartbeat_back_addr': '172.21.15.33:6816/3941314957', 'heartbeat_front_addr': '172.21.15.33:6815/3941314957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0498129c-861c-4f7e-b8ba-219949da3130', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.54083251953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6800', 'nonce': 544126779}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 544126779}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 544126779}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 544126779}]}, 'public_addr': '172.21.15.94:6800/544126779', 'cluster_addr': '172.21.15.94:6801/544126779', 'heartbeat_back_addr': '172.21.15.94:6803/544126779', 'heartbeat_front_addr': '172.21.15.94:6802/544126779', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '05718a1e-1164-4b54-a493-23c1cd3adb50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 604, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 3229884005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 3229884005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 3229884005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 3229884005}]}, 'public_addr': '172.21.15.94:6804/3229884005', 'cluster_addr': '172.21.15.94:6805/3229884005', 'heartbeat_back_addr': '172.21.15.94:6807/3229884005', 'heartbeat_front_addr': '172.21.15.94:6806/3229884005', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dc48640f-8f27-405c-a9a4-cf1e3e6c8821', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 507305558}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 507305558}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 507305558}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 507305558}]}, 'public_addr': '172.21.15.94:6808/507305558', 'cluster_addr': '172.21.15.94:6809/507305558', 'heartbeat_back_addr': '172.21.15.94:6811/507305558', 'heartbeat_front_addr': '172.21.15.94:6810/507305558', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7d55aa3f-06b6-4fc3-97fd-62b535f78b66', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 597, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3401977873}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 3401977873}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 3401977873}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 3401977873}]}, 'public_addr': '172.21.15.94:6812/3401977873', 'cluster_addr': '172.21.15.94:6813/3401977873', 'heartbeat_back_addr': '172.21.15.94:6815/3401977873', 'heartbeat_front_addr': '172.21.15.94:6814/3401977873', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:12:06.439494+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:09:44.771155+0000', 'dead_epoch': 532}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:54.538230+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:33.499743+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:51.353050+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:35.185056+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:49.922567+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T17:11:49.490108+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.7', 'osds': [1, 4, 5]}], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '2.5', 'osds': [7, 4, 6]}, {'pgid': '2.6', 'osds': [6, 4, 3]}, {'pgid': '2.8', 'osds': [7, 4, 5]}, {'pgid': '2.d', 'osds': [3, 4, 2]}, {'pgid': '2.11', 'osds': [6, 4, 5]}, {'pgid': '2.16', 'osds': [6, 4, 3]}, {'pgid': '2.18', 'osds': [7, 4, 5]}, {'pgid': '2.19', 'osds': [7, 3, 0]}, {'pgid': '2.1a', 'osds': [3, 7, 2]}, {'pgid': '2.1c', 'osds': [5, 2, 7]}, {'pgid': '4.2', 'osds': [5, 4, 2]}, {'pgid': '4.9', 'osds': [3, 4, 6]}, {'pgid': '4.b', 'osds': [4, 3, 5]}, {'pgid': '4.d', 'osds': [4, 2, 6]}, {'pgid': '4.e', 'osds': [6, 4, 3]}, {'pgid': '4.f', 'osds': [3, 4, 7]}, {'pgid': '4.13', 'osds': [4, 6, 5]}, {'pgid': '4.18', 'osds': [4, 6, 3]}, {'pgid': '4.1b', 'osds': [3, 4, 2]}, {'pgid': '4.1d', 'osds': [3, 4, 6]}, {'pgid': '5.0', 'osds': [3, 4, 2]}, {'pgid': '5.5', 'osds': [6, 4, 2]}, {'pgid': '5.7', 'osds': [5, 4, 0]}, {'pgid': '5.c', 'osds': [4, 0, 6]}, {'pgid': '5.10', 'osds': [7, 6, 5]}, {'pgid': '5.13', 'osds': [3, 5, 7]}, {'pgid': '6.e', 'osds': [4, 2, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.33:6800/2829464018': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/1835128904': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2277321750': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:0/2116602769': '2024-02-13T16:50:02.257936+0000', '172.21.15.33:6800/3967118778': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/2298361247': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:0/218227980': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/2070582473': '2024-02-13T16:44:22.727241+0000', '172.21.15.33:0/4291575956': '2024-02-13T16:44:46.313346+0000', '172.21.15.33:6800/3157562110': '2024-02-13T16:44:22.727241+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T17:12:41.529 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:12:41.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:41.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:41.797 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:41.843 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:41.917 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:12:41.917 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:12:41.917 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:12:41.917 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:12:41.918 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:12:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:40.958581+0000 mgr.y (mgr.24481) 868 : cluster [DBG] pgmap v1221: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s; 31694/169077 objects degraded (18.745%); 53/169077 objects misplaced (0.031%) 2024-02-12T17:12:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:41.595337+0000 mon.a (mon.0) 1995 : cluster [WRN] Health check update: Degraded data redundancy: 31694/169077 objects degraded (18.745%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:42.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:40.958581+0000 mgr.y (mgr.24481) 868 : cluster [DBG] pgmap v1221: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s; 31694/169077 objects degraded (18.745%); 53/169077 objects misplaced (0.031%) 2024-02-12T17:12:42.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:41.595337+0000 mon.a (mon.0) 1995 : cluster [WRN] Health check update: Degraded data redundancy: 31694/169077 objects degraded (18.745%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:42.414 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:42.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 31 16 62914 62898 126.8 89 0.0150618 0.00788217 2024-02-12T17:12:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:42.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:42.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:40.958581+0000 mgr.y (mgr.24481) 868 : cluster [DBG] pgmap v1221: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s; 31694/169077 objects degraded (18.745%); 53/169077 objects misplaced (0.031%) 2024-02-12T17:12:42.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:41.595337+0000 mon.a (mon.0) 1995 : cluster [WRN] Health check update: Degraded data redundancy: 31694/169077 objects degraded (18.745%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:42.519 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:42.605 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:42.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:42.715 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:42.766 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:12:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:12:42.766 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:42.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:42.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:42.951 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:43.004 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:43.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:43.108 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:43.154 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:43.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 32 16 64354 64338 125.65 90 0.00980795 0.00795471 2024-02-12T17:12:43.518 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:43.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:43.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:43.686 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:43.720 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:43.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:43.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:43.827 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:43.866 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:43.891 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:43.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:43.902 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:12:43.935 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:12:43.999 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:44.065 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:44.102 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:44.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:44.197 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:42.960918+0000 mgr.y (mgr.24481) 869 : cluster [DBG] pgmap v1222: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 900 B/s rd, 85 MiB/s wr, 1.36k op/s; 33094/177534 objects degraded (18.641%); 53/177534 objects misplaced (0.030%) 2024-02-12T17:12:44.198 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:42.960918+0000 mgr.y (mgr.24481) 869 : cluster [DBG] pgmap v1222: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 900 B/s rd, 85 MiB/s wr, 1.36k op/s; 33094/177534 objects degraded (18.641%); 53/177534 objects misplaced (0.030%) 2024-02-12T17:12:44.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:44.266 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:44.303 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:44.307 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:42.960918+0000 mgr.y (mgr.24481) 869 : cluster [DBG] pgmap v1222: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 900 B/s rd, 85 MiB/s wr, 1.36k op/s; 33094/177534 objects degraded (18.641%); 53/177534 objects misplaced (0.030%) 2024-02-12T17:12:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:44.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:44.409 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:44.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 33 16 65748 65732 124.482 87.125 0.0125926 0.00802834 2024-02-12T17:12:44.471 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:44.549 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:44.639 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:44.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:44.747 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:44.799 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:44.828 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:44.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:44.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:44.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:44.977 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:45.074 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:44.849817+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.33:0/752207691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:12:45.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:12:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:12:45.092 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:12:45.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:44.849817+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.33:0/752207691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:12:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:44.849817+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.33:0/752207691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:12:45.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 34 16 67058 67042 123.229 81.875 0.0136949 0.00811072 2024-02-12T17:12:45.936 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:12:46.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:44.847632+0000 mgr.y (mgr.24481) 870 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:46.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:44.965104+0000 mgr.y (mgr.24481) 871 : cluster [DBG] pgmap v1223: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 35789/191022 objects degraded (18.736%); 53/191022 objects misplaced (0.028%) 2024-02-12T17:12:46.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:45.063116+0000 mon.a (mon.0) 1997 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:46.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:45.081076+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? v1:172.21.15.33:0/752207691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:12:46.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:45.081143+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e609: 8 total, 7 up, 8 in 2024-02-12T17:12:46.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:44.847632+0000 mgr.y (mgr.24481) 870 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:46.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:44.965104+0000 mgr.y (mgr.24481) 871 : cluster [DBG] pgmap v1223: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 35789/191022 objects degraded (18.736%); 53/191022 objects misplaced (0.028%) 2024-02-12T17:12:46.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:45.063116+0000 mon.a (mon.0) 1997 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:46.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:45.081076+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? v1:172.21.15.33:0/752207691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:12:46.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:45.081143+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e609: 8 total, 7 up, 8 in 2024-02-12T17:12:46.415 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 35 16 68490 68474 122.265 89.5 0.00554107 0.0081746 2024-02-12T17:12:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:44.847632+0000 mgr.y (mgr.24481) 870 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:44.965104+0000 mgr.y (mgr.24481) 871 : cluster [DBG] pgmap v1223: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 35789/191022 objects degraded (18.736%); 53/191022 objects misplaced (0.028%) 2024-02-12T17:12:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:45.063116+0000 mon.a (mon.0) 1997 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:12:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:45.081076+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? v1:172.21.15.33:0/752207691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:12:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:45.081143+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e609: 8 total, 7 up, 8 in 2024-02-12T17:12:47.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:45.760781+0000 osd.7 (osd.7) 284 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:12:47.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:45.764950+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:12:47.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:46.596757+0000 mon.a (mon.0) 2000 : cluster [WRN] Health check update: Degraded data redundancy: 35789/191022 objects degraded (18.736%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:47.077 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:46.608252+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e610: 8 total, 7 up, 8 in 2024-02-12T17:12:47.252 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:47.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:47.360 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:47.416 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 36 16 69780 69764 121.108 80.625 0.00530704 0.00825195 2024-02-12T17:12:47.417 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:45.760781+0000 osd.7 (osd.7) 284 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:12:47.417 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:45.764950+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:12:47.417 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:46.596757+0000 mon.a (mon.0) 2000 : cluster [WRN] Health check update: Degraded data redundancy: 35789/191022 objects degraded (18.736%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:47.417 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:46.608252+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e610: 8 total, 7 up, 8 in 2024-02-12T17:12:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:47.464 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:47.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:47.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:45.760781+0000 osd.7 (osd.7) 284 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:12:47.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:45.764950+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:12:47.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:46.596757+0000 mon.a (mon.0) 2000 : cluster [WRN] Health check update: Degraded data redundancy: 35789/191022 objects degraded (18.736%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:47.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:46.608252+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e610: 8 total, 7 up, 8 in 2024-02-12T17:12:47.569 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:47.616 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:48.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:46.965696+0000 mgr.y (mgr.24481) 872 : cluster [DBG] pgmap v1226: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s; 37163/199167 objects degraded (18.659%); 53/199167 objects misplaced (0.027%) 2024-02-12T17:12:48.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:46.965696+0000 mgr.y (mgr.24481) 872 : cluster [DBG] pgmap v1226: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s; 37163/199167 objects degraded (18.659%); 53/199167 objects misplaced (0.027%) 2024-02-12T17:12:48.413 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:48.416 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 37 16 71155 71139 120.157 85.9375 0.0151718 0.0083181 2024-02-12T17:12:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:48.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:48.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:46.965696+0000 mgr.y (mgr.24481) 872 : cluster [DBG] pgmap v1226: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s; 37163/199167 objects degraded (18.659%); 53/199167 objects misplaced (0.027%) 2024-02-12T17:12:48.519 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:48.561 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:48.591 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:48.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:48.700 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:48.799 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:48.841 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:48.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:48.951 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:48.992 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:49.103 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:47.747325+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:12:49.104 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:47.763256+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:12:49.104 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:47.781518+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:12:49.104 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:47.788322+0000 osd.7 (osd.7) 287 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:12:49.151 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:12:49.204 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:49.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:49.240 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:47.747325+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:12:49.240 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:47.763256+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:12:49.240 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:47.781518+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:12:49.240 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:47.788322+0000 osd.7 (osd.7) 287 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:12:49.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:47.747325+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:12:49.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:47.763256+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:12:49.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:47.781518+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:12:49.244 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:47.788322+0000 osd.7 (osd.7) 287 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:12:49.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:49.416 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:49.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:49.416 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 38 16 72366 72350 118.987 75.6875 0.00970145 0.00839978 2024-02-12T17:12:49.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:49.589 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:49.631 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:49.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:49.735 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:49.780 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:50.416 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 39 16 73685 73669 118.049 82.4375 0.00946438 0.00846731 2024-02-12T17:12:50.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:48.631066+0000 osd.5 (osd.5) 295 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:12:50.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:48.698720+0000 osd.5 (osd.5) 296 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:12:50.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:48.773732+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:12:50.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:48.882059+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:12:50.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:48.955251+0000 osd.5 (osd.5) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:50.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:48.955313+0000 osd.5 (osd.5) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:50.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:48.966791+0000 mgr.y (mgr.24481) 873 : cluster [DBG] pgmap v1227: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 39677/211746 objects degraded (18.738%); 53/211746 objects misplaced (0.025%) 2024-02-12T17:12:50.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:49.103946+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e611: 8 total, 7 up, 8 in 2024-02-12T17:12:50.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:49.774096+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:12:50.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:49.809838+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:12:50.506 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:12:50.508 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:12:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:48.631066+0000 osd.5 (osd.5) 295 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:12:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:48.698720+0000 osd.5 (osd.5) 296 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:12:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:48.773732+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:12:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:48.882059+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:12:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:48.955251+0000 osd.5 (osd.5) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:48.955313+0000 osd.5 (osd.5) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:48.966791+0000 mgr.y (mgr.24481) 873 : cluster [DBG] pgmap v1227: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 39677/211746 objects degraded (18.738%); 53/211746 objects misplaced (0.025%) 2024-02-12T17:12:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:49.103946+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e611: 8 total, 7 up, 8 in 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:49.774096+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:49.809838+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:48.631066+0000 osd.5 (osd.5) 295 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:48.698720+0000 osd.5 (osd.5) 296 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:48.773732+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:48.882059+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:48.955251+0000 osd.5 (osd.5) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:48.955313+0000 osd.5 (osd.5) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:48.966791+0000 mgr.y (mgr.24481) 873 : cluster [DBG] pgmap v1227: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 39677/211746 objects degraded (18.738%); 53/211746 objects misplaced (0.025%) 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:49.103946+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e611: 8 total, 7 up, 8 in 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:49.774096+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:12:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:49.809838+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:12:50.543 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:50.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:50.652 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:50.700 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:50.736 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:50.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:50.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:50.895 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:51.192 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:12:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:50] "GET /metrics HTTP/1.1" 200 176885 "" "Prometheus/2.33.4" 2024-02-12T17:12:51.192 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:50.175338+0000 mgr.y (mgr.24481) 874 : audit [DBG] from='client.16131 v1:172.21.15.33:0/2258189920' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:12:51.192 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:50.756419+0000 osd.4 (osd.4) 191 : cluster [DBG] 4.b scrub starts 2024-02-12T17:12:51.192 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:50.764677+0000 osd.4 (osd.4) 192 : cluster [DBG] 4.b scrub ok 2024-02-12T17:12:51.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:50.175338+0000 mgr.y (mgr.24481) 874 : audit [DBG] from='client.16131 v1:172.21.15.33:0/2258189920' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:12:51.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:50.756419+0000 osd.4 (osd.4) 191 : cluster [DBG] 4.b scrub starts 2024-02-12T17:12:51.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:50.764677+0000 osd.4 (osd.4) 192 : cluster [DBG] 4.b scrub ok 2024-02-12T17:12:51.416 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:12:51.415872+0000 min lat: 0.00107612 max lat: 0.109485 avg lat: 0.00853247 2024-02-12T17:12:51.416 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:12:51.416 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 40 16 74990 74974 117.137 81.5625 0.00813289 0.00853247 2024-02-12T17:12:51.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:50.175338+0000 mgr.y (mgr.24481) 874 : audit [DBG] from='client.16131 v1:172.21.15.33:0/2258189920' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:12:51.495 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:50.756419+0000 osd.4 (osd.4) 191 : cluster [DBG] 4.b scrub starts 2024-02-12T17:12:51.496 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:50.764677+0000 osd.4 (osd.4) 192 : cluster [DBG] 4.b scrub ok 2024-02-12T17:12:52.307 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:52.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:52.404 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:50.776506+0000 osd.6 (osd.6) 159 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:12:52.404 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:50.786893+0000 osd.6 (osd.6) 160 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:12:52.404 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:50.792310+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.1b scrub starts 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:50.797352+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.1b scrub ok 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:50.967567+0000 mgr.y (mgr.24481) 875 : cluster [DBG] pgmap v1229: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s; 39677/211746 objects degraded (18.738%); 53/211746 objects misplaced (0.025%) 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:51.119030+0000 osd.3 (osd.3) 210 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:51.126049+0000 osd.3 (osd.3) 211 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:51.598254+0000 mon.a (mon.0) 2003 : cluster [WRN] Health check update: Degraded data redundancy: 39677/211746 objects degraded (18.738%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:51.604350+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e612: 8 total, 7 up, 8 in 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:52.116453+0000 mon.a (mon.0) 2005 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:50.776506+0000 osd.6 (osd.6) 159 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:50.786893+0000 osd.6 (osd.6) 160 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:12:52.405 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:50.792310+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.1b scrub starts 2024-02-12T17:12:52.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:50.797352+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.1b scrub ok 2024-02-12T17:12:52.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:50.967567+0000 mgr.y (mgr.24481) 875 : cluster [DBG] pgmap v1229: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s; 39677/211746 objects degraded (18.738%); 53/211746 objects misplaced (0.025%) 2024-02-12T17:12:52.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:51.119030+0000 osd.3 (osd.3) 210 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:12:52.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:51.126049+0000 osd.3 (osd.3) 211 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:12:52.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:51.598254+0000 mon.a (mon.0) 2003 : cluster [WRN] Health check update: Degraded data redundancy: 39677/211746 objects degraded (18.738%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:52.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:51.604350+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e612: 8 total, 7 up, 8 in 2024-02-12T17:12:52.406 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:52.116453+0000 mon.a (mon.0) 2005 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:52.414 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:52.418 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 41 16 76214 76198 116.14 76.5 0.0135669 0.00860624 2024-02-12T17:12:52.469 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:50.776506+0000 osd.6 (osd.6) 159 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:12:52.469 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:50.786893+0000 osd.6 (osd.6) 160 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:12:52.469 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:50.792310+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.1b scrub starts 2024-02-12T17:12:52.469 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:50.797352+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.1b scrub ok 2024-02-12T17:12:52.469 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:50.967567+0000 mgr.y (mgr.24481) 875 : cluster [DBG] pgmap v1229: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s; 39677/211746 objects degraded (18.738%); 53/211746 objects misplaced (0.025%) 2024-02-12T17:12:52.469 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:51.119030+0000 osd.3 (osd.3) 210 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:12:52.469 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:51.126049+0000 osd.3 (osd.3) 211 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:12:52.470 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:51.598254+0000 mon.a (mon.0) 2003 : cluster [WRN] Health check update: Degraded data redundancy: 39677/211746 objects degraded (18.738%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:52.470 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:51.604350+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e612: 8 total, 7 up, 8 in 2024-02-12T17:12:52.470 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:52.116453+0000 mon.a (mon.0) 2005 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:52.474 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:52.503 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:52.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:52.609 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:12:52.661 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:52.701 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:12:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:52.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:12:52.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:12:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:12:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:12:52.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:52.867 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:52.906 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:52.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:52.954 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1228,"stamp":"2024-02-12T17:12:49.102684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4613084836,"num_objects":70582,"num_object_clones":0,"num_object_copies":211746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39677,"num_objects_misplaced":53,"num_objects_unfound":0,"num_objects_dirty":70582,"num_whiteouts":0,"num_read":2181,"num_read_kb":1933,"num_write":71390,"num_write_kb":4505001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":777,"num_bytes_recovered":21641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71525,"ondisk_log_size":71525,"up":309,"acting":335,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":422,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19372908,"kb_used_data":11592492,"kb_used_omap":0,"kb_used_meta":7780352,"kb_avail":730588308,"statfs":{"total":767960285184,"available":748122427392,"internally_reserved":0,"allocated":11870711808,"data_stored":11862752411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7967080448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":747372544,"num_objects":11404,"num_object_clones":0,"num_object_copies":34212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11404,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11404,"num_write_kb":729856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.142585"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":309,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.599809+0000","last_change":"2024-02-12T17:07:35.884990+0000","last_active":"2024-02-12T17:12:31.599809+0000","last_peered":"2024-02-12T17:12:31.599809+0000","last_clean":"2024-02-12T17:12:31.599809+0000","last_became_active":"2024-02-12T17:03:50.545370+0000","last_became_peered":"2024-02-12T17:03:50.545370+0000","last_unstale":"2024-02-12T17:12:31.599809+0000","last_undegraded":"2024-02-12T17:12:31.599809+0000","last_fullsized":"2024-02-12T17:12:31.599809+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":40,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.088998+0000","last_change":"2024-02-12T17:11:53.988743+0000","last_active":"2024-02-12T17:12:45.088998+0000","last_peered":"2024-02-12T17:12:45.088998+0000","last_clean":"2024-02-12T17:12:45.088998+0000","last_became_active":"2024-02-12T17:11:52.621153+0000","last_became_peered":"2024-02-12T17:11:52.621153+0000","last_unstale":"2024-02-12T17:12:45.088998+0000","last_undegraded":"2024-02-12T17:12:45.088998+0000","last_fullsized":"2024-02-12T17:12:45.088998+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'19","last_scrub_stamp":"2024-02-12T17:07:47.845930+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.845930+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,3],"avail_no_missing":["4","3","6"],"object_location_counts":[{"shards":"3,4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":336,"reported_epoch":608,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:31.598775+0000","last_change":"2024-02-12T17:11:52.849834+0000","last_active":"2024-02-12T17:12:31.598775+0000","last_peered":"2024-02-12T17:12:31.598775+0000","last_clean":"2024-02-12T17:12:31.598775+0000","last_became_active":"2024-02-12T17:11:52.621205+0000","last_became_peered":"2024-02-12T17:11:52.621205+0000","last_unstale":"2024-02-12T17:12:31.598775+0000","last_undegraded":"2024-02-12T17:12:31.598775+0000","last_fullsized":"2024-02-12T17:12:31.598775+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'7","last_scrub_stamp":"2024-02-12T17:05:56.388366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:56.388366+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":45,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.097091+0000","last_change":"2024-02-12T17:11:37.549503+0000","last_active":"2024-02-12T17:12:45.097091+0000","last_peered":"2024-02-12T17:12:45.097091+0000","last_clean":"2024-02-12T17:12:45.097091+0000","last_became_active":"2024-02-12T17:11:37.549052+0000","last_became_peered":"2024-02-12T17:11:37.549052+0000","last_unstale":"2024-02-12T17:12:45.097091+0000","last_undegraded":"2024-02-12T17:12:45.097091+0000","last_fullsized":"2024-02-12T17:12:45.097091+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.235654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:34.502332+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.235654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,7],"avail_no_missing":["5","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":346,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.598453+0000","last_change":"2024-02-12T17:11:52.013271+0000","last_active":"2024-02-12T17:12:31.598453+0000","last_peered":"2024-02-12T17:12:31.598453+0000","last_clean":"2024-02-12T17:12:31.598453+0000","last_became_active":"2024-02-12T17:11:51.598092+0000","last_became_peered":"2024-02-12T17:11:51.598092+0000","last_unstale":"2024-02-12T17:12:31.598453+0000","last_undegraded":"2024-02-12T17:12:31.598453+0000","last_fullsized":"2024-02-12T17:12:31.598453+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:03:16.782772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:16.782772+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":286,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.089224+0000","last_change":"2024-02-12T17:11:51.620071+0000","last_active":"2024-02-12T17:12:45.089224+0000","last_peered":"2024-02-12T17:12:45.089224+0000","last_clean":"2024-02-12T17:12:45.089224+0000","last_became_active":"2024-02-12T17:11:51.619753+0000","last_became_peered":"2024-02-12T17:11:51.619753+0000","last_unstale":"2024-02-12T17:12:45.089224+0000","last_undegraded":"2024-02-12T17:12:45.089224+0000","last_fullsized":"2024-02-12T17:12:45.089224+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:29.479610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.724890+0000","last_clean_scrub_stamp":"2024-02-12T17:06:29.479610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":384,"reported_epoch":608,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:31.599875+0000","last_change":"2024-02-12T17:12:05.116184+0000","last_active":"2024-02-12T17:12:31.599875+0000","last_peered":"2024-02-12T17:12:31.599875+0000","last_clean":"2024-02-12T17:12:31.599875+0000","last_became_active":"2024-02-12T17:11:52.628218+0000","last_became_peered":"2024-02-12T17:11:52.628218+0000","last_unstale":"2024-02-12T17:12:31.599875+0000","last_undegraded":"2024-02-12T17:12:31.599875+0000","last_fullsized":"2024-02-12T17:12:31.599875+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":1540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":42,"reported_epoch":608,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:31.597919+0000","last_change":"2024-02-12T17:11:37.552612+0000","last_active":"2024-02-12T17:12:31.597919+0000","last_peered":"2024-02-12T17:12:31.597919+0000","last_clean":"2024-02-12T17:12:31.597919+0000","last_became_active":"2024-02-12T17:11:37.552178+0000","last_became_peered":"2024-02-12T17:11:37.552178+0000","last_unstale":"2024-02-12T17:12:31.597919+0000","last_undegraded":"2024-02-12T17:12:31.597919+0000","last_fullsized":"2024-02-12T17:12:31.597919+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:52.828449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:44.303379+0000","last_clean_scrub_stamp":"2024-02-12T17:06:52.828449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,2],"avail_no_missing":["3","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":160,"reported_epoch":608,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:31.598411+0000","last_change":"2024-02-12T17:10:03.934455+0000","last_active":"2024-02-12T17:12:31.598411+0000","last_peered":"2024-02-12T17:12:31.598411+0000","last_clean":"2024-02-12T17:09:50.594583+0000","last_became_active":"2024-02-12T17:10:03.934455+0000","last_became_peered":"2024-02-12T17:10:03.934455+0000","last_unstale":"2024-02-12T17:12:31.598411+0000","last_undegraded":"2024-02-12T17:10:03.866782+0000","last_fullsized":"2024-02-12T17:10:03.866683+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":48,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.088140+0000","last_change":"2024-02-12T17:12:01.996493+0000","last_active":"2024-02-12T17:12:45.088140+0000","last_peered":"2024-02-12T17:12:45.088140+0000","last_clean":"2024-02-12T17:12:45.088140+0000","last_became_active":"2024-02-12T17:11:37.565444+0000","last_became_peered":"2024-02-12T17:11:37.565444+0000","last_unstale":"2024-02-12T17:12:45.088140+0000","last_undegraded":"2024-02-12T17:12:45.088140+0000","last_fullsized":"2024-02-12T17:12:45.088140+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:01.996397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:01.996397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":136,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:45.096971+0000","last_change":"2024-02-12T17:10:03.869766+0000","last_active":"2024-02-12T17:12:45.096971+0000","last_peered":"2024-02-12T17:12:45.096971+0000","last_clean":"2024-02-12T17:09:58.609101+0000","last_became_active":"2024-02-12T17:10:03.869766+0000","last_became_peered":"2024-02-12T17:10:03.869766+0000","last_unstale":"2024-02-12T17:12:45.096971+0000","last_undegraded":"2024-02-12T17:10:03.860263+0000","last_fullsized":"2024-02-12T17:10:03.860160+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":563,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.090078+0000","last_change":"2024-02-12T17:12:15.902270+0000","last_active":"2024-02-12T17:12:45.090078+0000","last_peered":"2024-02-12T17:12:45.090078+0000","last_clean":"2024-02-12T17:12:45.090078+0000","last_became_active":"2024-02-12T17:11:52.608576+0000","last_became_peered":"2024-02-12T17:11:52.608576+0000","last_unstale":"2024-02-12T17:12:45.090078+0000","last_undegraded":"2024-02-12T17:12:45.090078+0000","last_fullsized":"2024-02-12T17:12:45.090078+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.902112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.902112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,5],"avail_no_missing":["7","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":672,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:45.089349+0000","last_change":"2024-02-12T17:10:03.919345+0000","last_active":"2024-02-12T17:12:45.089349+0000","last_peered":"2024-02-12T17:12:45.089349+0000","last_clean":"2024-02-12T17:09:58.612919+0000","last_became_active":"2024-02-12T17:10:03.919345+0000","last_became_peered":"2024-02-12T17:10:03.919345+0000","last_unstale":"2024-02-12T17:12:45.089349+0000","last_undegraded":"2024-02-12T17:10:03.858535+0000","last_fullsized":"2024-02-12T17:10:03.858425+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":310,"reported_epoch":609,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:12:45.090214+0000","last_change":"2024-02-12T17:12:22.616320+0000","last_active":"2024-02-12T17:12:45.090214+0000","last_peered":"2024-02-12T17:12:45.090214+0000","last_clean":"2024-02-12T17:11:51.020252+0000","last_became_active":"2024-02-12T17:11:52.607640+0000","last_became_peered":"2024-02-12T17:11:52.607640+0000","last_unstale":"2024-02-12T17:12:45.090214+0000","last_undegraded":"2024-02-12T17:12:45.090214+0000","last_fullsized":"2024-02-12T17:12:45.090214+0000","mapping_epoch":591,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":293,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.598855+0000","last_change":"2024-02-12T17:09:21.547851+0000","last_active":"2024-02-12T17:12:31.598855+0000","last_peered":"2024-02-12T17:12:31.598855+0000","last_clean":"2024-02-12T17:12:31.598855+0000","last_became_active":"2024-02-12T17:09:21.547628+0000","last_became_peered":"2024-02-12T17:09:21.547628+0000","last_unstale":"2024-02-12T17:12:31.598855+0000","last_undegraded":"2024-02-12T17:12:31.598855+0000","last_fullsized":"2024-02-12T17:12:31.598855+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":120,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.097106+0000","last_change":"2024-02-12T17:12:12.800505+0000","last_active":"2024-02-12T17:12:45.097106+0000","last_peered":"2024-02-12T17:12:45.097106+0000","last_clean":"2024-02-12T17:12:45.097106+0000","last_became_active":"2024-02-12T17:11:36.552546+0000","last_became_peered":"2024-02-12T17:11:36.552546+0000","last_unstale":"2024-02-12T17:12:45.097106+0000","last_undegraded":"2024-02-12T17:12:45.097106+0000","last_fullsized":"2024-02-12T17:12:45.097106+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:12.800431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.800431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":325,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.600282+0000","last_change":"2024-02-12T17:12:12.141297+0000","last_active":"2024-02-12T17:12:31.600282+0000","last_peered":"2024-02-12T17:12:31.600282+0000","last_clean":"2024-02-12T17:12:31.600282+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:12:31.600282+0000","last_undegraded":"2024-02-12T17:12:31.600282+0000","last_fullsized":"2024-02-12T17:12:31.600282+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.141204+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":121,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.095971+0000","last_change":"2024-02-12T17:11:51.604630+0000","last_active":"2024-02-12T17:12:45.095971+0000","last_peered":"2024-02-12T17:12:45.095971+0000","last_clean":"2024-02-12T17:12:45.095971+0000","last_became_active":"2024-02-12T17:11:51.604349+0000","last_became_peered":"2024-02-12T17:11:51.604349+0000","last_unstale":"2024-02-12T17:12:45.095971+0000","last_undegraded":"2024-02-12T17:12:45.095971+0000","last_fullsized":"2024-02-12T17:12:45.095971+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":595,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.088955+0000","last_change":"2024-02-12T17:12:10.028633+0000","last_active":"2024-02-12T17:12:45.088955+0000","last_peered":"2024-02-12T17:12:45.088955+0000","last_clean":"2024-02-12T17:12:45.088955+0000","last_became_active":"2024-02-12T17:11:51.599882+0000","last_became_peered":"2024-02-12T17:11:51.599882+0000","last_unstale":"2024-02-12T17:12:45.088955+0000","last_undegraded":"2024-02-12T17:12:45.088955+0000","last_fullsized":"2024-02-12T17:12:45.088955+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.028495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":581,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.096468+0000","last_change":"2024-02-12T17:12:14.817507+0000","last_active":"2024-02-12T17:12:45.096468+0000","last_peered":"2024-02-12T17:12:45.096468+0000","last_clean":"2024-02-12T17:12:45.096468+0000","last_became_active":"2024-02-12T17:11:52.602172+0000","last_became_peered":"2024-02-12T17:11:52.602172+0000","last_unstale":"2024-02-12T17:12:45.096468+0000","last_undegraded":"2024-02-12T17:12:45.096468+0000","last_fullsized":"2024-02-12T17:12:45.096468+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:14.817357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:11.135261+0000","last_clean_scrub_stamp":"2024-02-12T17:12:14.817357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,3],"avail_no_missing":["6","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":338,"reported_epoch":608,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:31.597761+0000","last_change":"2024-02-12T17:10:03.867728+0000","last_active":"2024-02-12T17:12:31.597761+0000","last_peered":"2024-02-12T17:12:31.597761+0000","last_clean":"2024-02-12T17:09:58.609134+0000","last_became_active":"2024-02-12T17:10:03.867728+0000","last_became_peered":"2024-02-12T17:10:03.867728+0000","last_unstale":"2024-02-12T17:12:31.597761+0000","last_undegraded":"2024-02-12T17:10:03.854546+0000","last_fullsized":"2024-02-12T17:10:03.854479+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":360,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.098011+0000","last_change":"2024-02-12T17:10:03.943360+0000","last_active":"2024-02-12T17:12:45.098011+0000","last_peered":"2024-02-12T17:12:45.098011+0000","last_clean":"2024-02-12T17:05:38.389408+0000","last_became_active":"2024-02-12T17:10:03.943360+0000","last_became_peered":"2024-02-12T17:10:03.943360+0000","last_unstale":"2024-02-12T17:12:45.098011+0000","last_undegraded":"2024-02-12T17:12:45.098011+0000","last_fullsized":"2024-02-12T17:10:03.861151+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":275,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.600713+0000","last_change":"2024-02-12T17:07:36.839451+0000","last_active":"2024-02-12T17:12:31.600713+0000","last_peered":"2024-02-12T17:12:31.600713+0000","last_clean":"2024-02-12T17:12:31.600713+0000","last_became_active":"2024-02-12T17:03:50.545207+0000","last_became_peered":"2024-02-12T17:03:50.545207+0000","last_unstale":"2024-02-12T17:12:31.600713+0000","last_undegraded":"2024-02-12T17:12:31.600713+0000","last_fullsized":"2024-02-12T17:12:31.600713+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":406,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.098629+0000","last_change":"2024-02-12T17:10:03.880787+0000","last_active":"2024-02-12T17:12:45.098629+0000","last_peered":"2024-02-12T17:12:45.098629+0000","last_clean":"2024-02-12T17:09:19.537752+0000","last_became_active":"2024-02-12T17:10:03.880787+0000","last_became_peered":"2024-02-12T17:10:03.880787+0000","last_unstale":"2024-02-12T17:12:45.098629+0000","last_undegraded":"2024-02-12T17:12:45.098629+0000","last_fullsized":"2024-02-12T17:10:03.861487+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":47,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.089328+0000","last_change":"2024-02-12T17:12:15.817919+0000","last_active":"2024-02-12T17:12:45.089328+0000","last_peered":"2024-02-12T17:12:45.089328+0000","last_clean":"2024-02-12T17:12:45.089328+0000","last_became_active":"2024-02-12T17:11:52.612653+0000","last_became_peered":"2024-02-12T17:11:52.612653+0000","last_unstale":"2024-02-12T17:12:45.089328+0000","last_undegraded":"2024-02-12T17:12:45.089328+0000","last_fullsized":"2024-02-12T17:12:45.089328+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:15.817751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.817751+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,5],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":647,"reported_epoch":608,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:31.598233+0000","last_change":"2024-02-12T17:10:03.941991+0000","last_active":"2024-02-12T17:12:31.598233+0000","last_peered":"2024-02-12T17:12:31.598233+0000","last_clean":"2024-02-12T17:12:31.598233+0000","last_became_active":"2024-02-12T17:10:03.935291+0000","last_became_peered":"2024-02-12T17:10:03.935291+0000","last_unstale":"2024-02-12T17:12:31.598233+0000","last_undegraded":"2024-02-12T17:12:31.598233+0000","last_fullsized":"2024-02-12T17:12:31.598233+0000","mapping_epoch":533,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":548,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.100385+0000","last_change":"2024-02-12T17:10:03.879412+0000","last_active":"2024-02-12T17:12:45.100385+0000","last_peered":"2024-02-12T17:12:45.100385+0000","last_clean":"2024-02-12T17:09:58.605818+0000","last_became_active":"2024-02-12T17:10:03.879412+0000","last_became_peered":"2024-02-12T17:10:03.879412+0000","last_unstale":"2024-02-12T17:12:45.100385+0000","last_undegraded":"2024-02-12T17:12:45.100385+0000","last_fullsized":"2024-02-12T17:10:03.862546+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":113,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.096642+0000","last_change":"2024-02-12T17:12:02.752352+0000","last_active":"2024-02-12T17:12:45.096642+0000","last_peered":"2024-02-12T17:12:45.096642+0000","last_clean":"2024-02-12T17:12:45.096642+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:12:45.096642+0000","last_undegraded":"2024-02-12T17:12:45.096642+0000","last_fullsized":"2024-02-12T17:12:45.096642+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:02.752280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:02.752280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":165,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:45.092479+0000","last_change":"2024-02-12T17:10:03.869410+0000","last_active":"2024-02-12T17:12:45.092479+0000","last_peered":"2024-02-12T17:12:45.092479+0000","last_clean":"2024-02-12T17:09:49.876861+0000","last_became_active":"2024-02-12T17:10:03.869410+0000","last_became_peered":"2024-02-12T17:10:03.869410+0000","last_unstale":"2024-02-12T17:12:45.092479+0000","last_undegraded":"2024-02-12T17:10:03.856610+0000","last_fullsized":"2024-02-12T17:10:03.856493+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":659,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:42.588719+0000","last_change":"2024-02-12T17:05:55.414790+0000","last_active":"2024-02-12T17:12:42.588719+0000","last_peered":"2024-02-12T17:12:42.588719+0000","last_clean":"2024-02-12T17:12:42.588719+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:12:42.588719+0000","last_undegraded":"2024-02-12T17:12:42.588719+0000","last_fullsized":"2024-02-12T17:12:42.588719+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.796130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.796130+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":34,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.089473+0000","last_change":"2024-02-12T17:12:06.904420+0000","last_active":"2024-02-12T17:12:45.089473+0000","last_peered":"2024-02-12T17:12:45.089473+0000","last_clean":"2024-02-12T17:12:45.089473+0000","last_became_active":"2024-02-12T17:11:51.595076+0000","last_became_peered":"2024-02-12T17:11:51.595076+0000","last_unstale":"2024-02-12T17:12:45.089473+0000","last_undegraded":"2024-02-12T17:12:45.089473+0000","last_fullsized":"2024-02-12T17:12:45.089473+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:06.904354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:06.904354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":700,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.098541+0000","last_change":"2024-02-12T17:12:12.838640+0000","last_active":"2024-02-12T17:12:45.098541+0000","last_peered":"2024-02-12T17:12:45.098541+0000","last_clean":"2024-02-12T17:12:45.098541+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:12:45.098541+0000","last_undegraded":"2024-02-12T17:12:45.098541+0000","last_fullsized":"2024-02-12T17:12:45.098541+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:12.838563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.838563+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"534'21","reported_seq":642,"reported_epoch":610,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:46.623071+0000","last_change":"2024-02-12T17:10:03.995272+0000","last_active":"2024-02-12T17:12:46.623071+0000","last_peered":"2024-02-12T17:12:46.623071+0000","last_clean":"2024-02-12T17:10:02.526826+0000","last_became_active":"2024-02-12T17:10:03.995272+0000","last_became_peered":"2024-02-12T17:10:03.995272+0000","last_unstale":"2024-02-12T17:12:46.623071+0000","last_undegraded":"2024-02-12T17:10:03.873726+0000","last_fullsized":"2024-02-12T17:10:03.873602+0000","mapping_epoch":533,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":599,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.097654+0000","last_change":"2024-02-12T17:07:35.798822+0000","last_active":"2024-02-12T17:12:45.097654+0000","last_peered":"2024-02-12T17:12:45.097654+0000","last_clean":"2024-02-12T17:12:45.097654+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:12:45.097654+0000","last_undegraded":"2024-02-12T17:12:45.097654+0000","last_fullsized":"2024-02-12T17:12:45.097654+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:35.798720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.798720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":329,"reported_epoch":608,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:31.598577+0000","last_change":"2024-02-12T17:10:03.901247+0000","last_active":"2024-02-12T17:12:31.598577+0000","last_peered":"2024-02-12T17:12:31.598577+0000","last_clean":"2024-02-12T17:09:58.611807+0000","last_became_active":"2024-02-12T17:10:03.901247+0000","last_became_peered":"2024-02-12T17:10:03.901247+0000","last_unstale":"2024-02-12T17:12:31.598577+0000","last_undegraded":"2024-02-12T17:10:03.867068+0000","last_fullsized":"2024-02-12T17:10:03.866968+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":614,"reported_epoch":610,"state":"active+clean+scrubbing","last_fresh":"2024-02-12T17:12:48.769268+0000","last_change":"2024-02-12T17:12:48.769268+0000","last_active":"2024-02-12T17:12:48.769268+0000","last_peered":"2024-02-12T17:12:48.769268+0000","last_clean":"2024-02-12T17:12:48.769268+0000","last_became_active":"2024-02-12T17:11:51.608628+0000","last_became_peered":"2024-02-12T17:11:51.608628+0000","last_unstale":"2024-02-12T17:12:48.769268+0000","last_undegraded":"2024-02-12T17:12:48.769268+0000","last_fullsized":"2024-02-12T17:12:48.769268+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:37.116609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:06:37.116609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":122,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.095774+0000","last_change":"2024-02-12T17:11:52.607460+0000","last_active":"2024-02-12T17:12:45.095774+0000","last_peered":"2024-02-12T17:12:45.095774+0000","last_clean":"2024-02-12T17:12:45.095774+0000","last_became_active":"2024-02-12T17:11:52.607042+0000","last_became_peered":"2024-02-12T17:11:52.607042+0000","last_unstale":"2024-02-12T17:12:45.095774+0000","last_undegraded":"2024-02-12T17:12:45.095774+0000","last_fullsized":"2024-02-12T17:12:45.095774+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:34.780676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:34.780676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,5],"avail_no_missing":["6","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":318,"reported_epoch":608,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:31.599860+0000","last_change":"2024-02-12T17:10:03.919074+0000","last_active":"2024-02-12T17:12:31.599860+0000","last_peered":"2024-02-12T17:12:31.599860+0000","last_clean":"2024-02-12T17:09:58.621311+0000","last_became_active":"2024-02-12T17:10:03.919074+0000","last_became_peered":"2024-02-12T17:10:03.919074+0000","last_unstale":"2024-02-12T17:12:31.599860+0000","last_undegraded":"2024-02-12T17:10:03.875183+0000","last_fullsized":"2024-02-12T17:10:03.875072+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":29,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.090943+0000","last_change":"2024-02-12T17:11:51.596024+0000","last_active":"2024-02-12T17:12:45.090943+0000","last_peered":"2024-02-12T17:12:45.090943+0000","last_clean":"2024-02-12T17:12:45.090943+0000","last_became_active":"2024-02-12T17:11:51.595762+0000","last_became_peered":"2024-02-12T17:11:51.595762+0000","last_unstale":"2024-02-12T17:12:45.090943+0000","last_undegraded":"2024-02-12T17:12:45.090943+0000","last_fullsized":"2024-02-12T17:12:45.090943+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.272939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.272939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":469,"reported_epoch":607,"state":"active+undersized","last_fresh":"2024-02-12T17:12:30.037924+0000","last_change":"2024-02-12T17:10:03.867952+0000","last_active":"2024-02-12T17:12:30.037924+0000","last_peered":"2024-02-12T17:12:30.037924+0000","last_clean":"2024-02-12T17:08:07.412687+0000","last_became_active":"2024-02-12T17:10:03.867952+0000","last_became_peered":"2024-02-12T17:10:03.867952+0000","last_unstale":"2024-02-12T17:12:30.037924+0000","last_undegraded":"2024-02-12T17:12:30.037924+0000","last_fullsized":"2024-02-12T17:10:03.854012+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":294,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.600481+0000","last_change":"2024-02-12T17:06:45.502715+0000","last_active":"2024-02-12T17:12:31.600481+0000","last_peered":"2024-02-12T17:12:31.600481+0000","last_clean":"2024-02-12T17:12:31.600481+0000","last_became_active":"2024-02-12T17:03:50.558331+0000","last_became_peered":"2024-02-12T17:03:50.558331+0000","last_unstale":"2024-02-12T17:12:31.600481+0000","last_undegraded":"2024-02-12T17:12:31.600481+0000","last_fullsized":"2024-02-12T17:12:31.600481+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"610'4535","reported_seq":4553,"reported_epoch":610,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:48.553591+0000","last_change":"2024-02-12T17:12:11.416644+0000","last_active":"2024-02-12T17:12:48.553591+0000","last_peered":"2024-02-12T17:12:48.553591+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.000993+0000","last_became_peered":"2024-02-12T17:11:59.000993+0000","last_unstale":"2024-02-12T17:12:48.553591+0000","last_undegraded":"2024-02-12T17:12:11.416490+0000","last_fullsized":"2024-02-12T17:11:58.993807+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4535,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":297205760,"num_objects":4535,"num_object_clones":0,"num_object_copies":13605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4535,"num_write_kb":290240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":4535}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":317,"reported_epoch":608,"state":"active+undersized","last_fresh":"2024-02-12T17:12:31.598700+0000","last_change":"2024-02-12T17:10:03.893996+0000","last_active":"2024-02-12T17:12:31.598700+0000","last_peered":"2024-02-12T17:12:31.598700+0000","last_clean":"2024-02-12T17:09:58.614534+0000","last_became_active":"2024-02-12T17:10:03.893996+0000","last_became_peered":"2024-02-12T17:10:03.893996+0000","last_unstale":"2024-02-12T17:12:31.598700+0000","last_undegraded":"2024-02-12T17:12:31.598700+0000","last_fullsized":"2024-02-12T17:10:03.859995+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":386,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.091304+0000","last_change":"2024-02-12T17:12:09.882327+0000","last_active":"2024-02-12T17:12:45.091304+0000","last_peered":"2024-02-12T17:12:45.091304+0000","last_clean":"2024-02-12T17:12:45.091304+0000","last_became_active":"2024-02-12T17:11:55.607676+0000","last_became_peered":"2024-02-12T17:11:55.607676+0000","last_unstale":"2024-02-12T17:12:45.091304+0000","last_undegraded":"2024-02-12T17:12:45.091304+0000","last_fullsized":"2024-02-12T17:12:45.091304+0000","mapping_epoch":594,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:12:09.882247+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:12:09.882247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":190,"reported_epoch":608,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:31.600525+0000","last_change":"2024-02-12T17:11:53.100445+0000","last_active":"2024-02-12T17:12:31.600525+0000","last_peered":"2024-02-12T17:12:31.600525+0000","last_clean":"2024-02-12T17:12:31.600525+0000","last_became_active":"2024-02-12T17:11:52.623783+0000","last_became_peered":"2024-02-12T17:11:52.623783+0000","last_unstale":"2024-02-12T17:12:31.600525+0000","last_undegraded":"2024-02-12T17:12:31.600525+0000","last_fullsized":"2024-02-12T17:12:31.600525+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'18","last_scrub_stamp":"2024-02-12T17:03:19.689433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:19.689433+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":473,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.100110+0000","last_change":"2024-02-12T17:10:03.891860+0000","last_active":"2024-02-12T17:12:45.100110+0000","last_peered":"2024-02-12T17:12:45.100110+0000","last_clean":"2024-02-12T17:04:37.847517+0000","last_became_active":"2024-02-12T17:10:03.891860+0000","last_became_peered":"2024-02-12T17:10:03.891860+0000","last_unstale":"2024-02-12T17:12:45.100110+0000","last_undegraded":"2024-02-12T17:12:45.100110+0000","last_fullsized":"2024-02-12T17:10:03.873207+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"610'4418","reported_seq":4439,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:48.651159+0000","last_change":"2024-02-12T17:11:59.009932+0000","last_active":"2024-02-12T17:12:48.651159+0000","last_peered":"2024-02-12T17:12:48.651159+0000","last_clean":"2024-02-12T17:12:48.651159+0000","last_became_active":"2024-02-12T17:11:59.009600+0000","last_became_peered":"2024-02-12T17:11:59.009600+0000","last_unstale":"2024-02-12T17:12:48.651159+0000","last_undegraded":"2024-02-12T17:12:48.651159+0000","last_fullsized":"2024-02-12T17:12:48.651159+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4418,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289538048,"num_objects":4418,"num_object_clones":0,"num_object_copies":13254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":282752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":599,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T17:12:30.037834+0000","last_change":"2024-02-12T17:12:10.780511+0000","last_active":"2024-02-12T17:12:30.037834+0000","last_peered":"2024-02-12T17:12:30.037834+0000","last_clean":"2024-02-12T17:12:30.037834+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:12:30.037834+0000","last_undegraded":"2024-02-12T17:12:30.037834+0000","last_fullsized":"2024-02-12T17:12:30.037834+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.780381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.780381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":391,"reported_epoch":607,"state":"active+undersized","last_fresh":"2024-02-12T17:12:30.038054+0000","last_change":"2024-02-12T17:10:03.864945+0000","last_active":"2024-02-12T17:12:30.038054+0000","last_peered":"2024-02-12T17:12:30.038054+0000","last_clean":"2024-02-12T17:08:07.411291+0000","last_became_active":"2024-02-12T17:10:03.864945+0000","last_became_peered":"2024-02-12T17:10:03.864945+0000","last_unstale":"2024-02-12T17:12:30.038054+0000","last_undegraded":"2024-02-12T17:12:30.038054+0000","last_fullsized":"2024-02-12T17:10:03.855363+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":181,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.098139+0000","last_change":"2024-02-12T17:11:55.829278+0000","last_active":"2024-02-12T17:12:45.098139+0000","last_peered":"2024-02-12T17:12:45.098139+0000","last_clean":"2024-02-12T17:12:45.098139+0000","last_became_active":"2024-02-12T17:11:52.612854+0000","last_became_peered":"2024-02-12T17:11:52.612854+0000","last_unstale":"2024-02-12T17:12:45.098139+0000","last_undegraded":"2024-02-12T17:12:45.098139+0000","last_fullsized":"2024-02-12T17:12:45.098139+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:55.829155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:55.829155+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"519'21","reported_seq":877,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:46.617426+0000","last_change":"2024-02-12T17:09:25.723429+0000","last_active":"2024-02-12T17:12:46.617426+0000","last_peered":"2024-02-12T17:12:46.617426+0000","last_clean":"2024-02-12T17:12:46.617426+0000","last_became_active":"2024-02-12T17:09:25.723083+0000","last_became_peered":"2024-02-12T17:09:25.723083+0000","last_unstale":"2024-02-12T17:12:46.617426+0000","last_undegraded":"2024-02-12T17:12:46.617426+0000","last_fullsized":"2024-02-12T17:12:46.617426+0000","mapping_epoch":517,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"609'4347","reported_seq":4364,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:46.415289+0000","last_change":"2024-02-12T17:12:11.418822+0000","last_active":"2024-02-12T17:12:46.415289+0000","last_peered":"2024-02-12T17:12:46.415289+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.003539+0000","last_became_peered":"2024-02-12T17:11:59.003539+0000","last_unstale":"2024-02-12T17:12:46.415289+0000","last_undegraded":"2024-02-12T17:12:11.415351+0000","last_fullsized":"2024-02-12T17:11:58.996599+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4347,"ondisk_log_size":4347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":284884992,"num_objects":4347,"num_object_clones":0,"num_object_copies":13041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4347,"num_write_kb":278208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":4347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":320,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.600580+0000","last_change":"2024-02-12T17:12:15.262269+0000","last_active":"2024-02-12T17:12:31.600580+0000","last_peered":"2024-02-12T17:12:31.600580+0000","last_clean":"2024-02-12T17:12:31.600580+0000","last_became_active":"2024-02-12T17:11:51.607338+0000","last_became_peered":"2024-02-12T17:11:51.607338+0000","last_unstale":"2024-02-12T17:12:31.600580+0000","last_undegraded":"2024-02-12T17:12:31.600580+0000","last_fullsized":"2024-02-12T17:12:31.600580+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.262134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.262134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":637,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.097740+0000","last_change":"2024-02-12T17:10:03.879478+0000","last_active":"2024-02-12T17:12:45.097740+0000","last_peered":"2024-02-12T17:12:45.097740+0000","last_clean":"2024-02-12T17:09:58.604742+0000","last_became_active":"2024-02-12T17:10:03.879478+0000","last_became_peered":"2024-02-12T17:10:03.879478+0000","last_unstale":"2024-02-12T17:12:45.097740+0000","last_undegraded":"2024-02-12T17:12:45.097740+0000","last_fullsized":"2024-02-12T17:10:03.861800+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"534'69","reported_seq":219,"reported_epoch":608,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:42.588795+0000","last_change":"2024-02-12T17:10:03.934831+0000","last_active":"2024-02-12T17:12:42.588795+0000","last_peered":"2024-02-12T17:12:42.588795+0000","last_clean":"2024-02-12T17:09:51.482838+0000","last_became_active":"2024-02-12T17:10:03.934831+0000","last_became_peered":"2024-02-12T17:10:03.934831+0000","last_unstale":"2024-02-12T17:12:42.588795+0000","last_undegraded":"2024-02-12T17:10:03.873010+0000","last_fullsized":"2024-02-12T17:10:03.872891+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":506,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.096950+0000","last_change":"2024-02-12T17:11:53.596949+0000","last_active":"2024-02-12T17:12:45.096950+0000","last_peered":"2024-02-12T17:12:45.096950+0000","last_clean":"2024-02-12T17:12:45.096950+0000","last_became_active":"2024-02-12T17:11:53.596550+0000","last_became_peered":"2024-02-12T17:11:53.596550+0000","last_unstale":"2024-02-12T17:12:45.096950+0000","last_undegraded":"2024-02-12T17:12:45.096950+0000","last_fullsized":"2024-02-12T17:12:45.096950+0000","mapping_epoch":592,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,2],"avail_no_missing":["6","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"609'4406","reported_seq":4426,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:46.458882+0000","last_change":"2024-02-12T17:11:59.008876+0000","last_active":"2024-02-12T17:12:46.458882+0000","last_peered":"2024-02-12T17:12:46.458882+0000","last_clean":"2024-02-12T17:12:46.458882+0000","last_became_active":"2024-02-12T17:11:59.008587+0000","last_became_peered":"2024-02-12T17:11:59.008587+0000","last_unstale":"2024-02-12T17:12:46.458882+0000","last_undegraded":"2024-02-12T17:12:46.458882+0000","last_fullsized":"2024-02-12T17:12:46.458882+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4406,"ondisk_log_size":4406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":288751616,"num_objects":4406,"num_object_clones":0,"num_object_copies":13218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":281984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":610,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.090484+0000","last_change":"2024-02-12T17:09:21.502830+0000","last_active":"2024-02-12T17:12:45.090484+0000","last_peered":"2024-02-12T17:12:45.090484+0000","last_clean":"2024-02-12T17:12:45.090484+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:12:45.090484+0000","last_undegraded":"2024-02-12T17:12:45.090484+0000","last_fullsized":"2024-02-12T17:12:45.090484+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:46.001888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:46.001888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":354,"reported_epoch":610,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:47.172658+0000","last_change":"2024-02-12T17:10:03.874040+0000","last_active":"2024-02-12T17:12:47.172658+0000","last_peered":"2024-02-12T17:12:47.172658+0000","last_clean":"2024-02-12T17:09:20.333789+0000","last_became_active":"2024-02-12T17:10:03.874040+0000","last_became_peered":"2024-02-12T17:10:03.874040+0000","last_unstale":"2024-02-12T17:12:47.172658+0000","last_undegraded":"2024-02-12T17:10:03.860815+0000","last_fullsized":"2024-02-12T17:10:03.856356+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":647,"num_read_kb":642,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":275,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.600371+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:12:31.600371+0000","last_peered":"2024-02-12T17:12:31.600371+0000","last_clean":"2024-02-12T17:12:31.600371+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:12:31.600371+0000","last_undegraded":"2024-02-12T17:12:31.600371+0000","last_fullsized":"2024-02-12T17:12:31.600371+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":145,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:45.097304+0000","last_change":"2024-02-12T17:10:03.886514+0000","last_active":"2024-02-12T17:12:45.097304+0000","last_peered":"2024-02-12T17:12:45.097304+0000","last_clean":"2024-02-12T17:09:50.311156+0000","last_became_active":"2024-02-12T17:10:03.886514+0000","last_became_peered":"2024-02-12T17:10:03.886514+0000","last_unstale":"2024-02-12T17:12:45.097304+0000","last_undegraded":"2024-02-12T17:10:03.872376+0000","last_fullsized":"2024-02-12T17:10:03.872265+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":534,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.097852+0000","last_change":"2024-02-12T17:12:01.859044+0000","last_active":"2024-02-12T17:12:45.097852+0000","last_peered":"2024-02-12T17:12:45.097852+0000","last_clean":"2024-02-12T17:12:45.097852+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:12:45.097852+0000","last_undegraded":"2024-02-12T17:12:45.097852+0000","last_fullsized":"2024-02-12T17:12:45.097852+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:01.858974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:01.858974+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"609'4251","reported_seq":4271,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:46.446550+0000","last_change":"2024-02-12T17:11:59.007548+0000","last_active":"2024-02-12T17:12:46.446550+0000","last_peered":"2024-02-12T17:12:46.446550+0000","last_clean":"2024-02-12T17:12:46.446550+0000","last_became_active":"2024-02-12T17:11:59.007214+0000","last_became_peered":"2024-02-12T17:11:59.007214+0000","last_unstale":"2024-02-12T17:12:46.446550+0000","last_undegraded":"2024-02-12T17:12:46.446550+0000","last_fullsized":"2024-02-12T17:12:46.446550+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4251,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":278593536,"num_objects":4251,"num_object_clones":0,"num_object_copies":12753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4251,"num_write_kb":272064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":658,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.098994+0000","last_change":"2024-02-12T17:12:08.952096+0000","last_active":"2024-02-12T17:12:45.098994+0000","last_peered":"2024-02-12T17:12:45.098994+0000","last_clean":"2024-02-12T17:12:45.098994+0000","last_became_active":"2024-02-12T17:11:51.603527+0000","last_became_peered":"2024-02-12T17:11:51.603527+0000","last_unstale":"2024-02-12T17:12:45.098994+0000","last_undegraded":"2024-02-12T17:12:45.098994+0000","last_fullsized":"2024-02-12T17:12:45.098994+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.951964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.951964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":302,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.598192+0000","last_change":"2024-02-12T17:09:21.602904+0000","last_active":"2024-02-12T17:12:31.598192+0000","last_peered":"2024-02-12T17:12:31.598192+0000","last_clean":"2024-02-12T17:12:31.598192+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:12:31.598192+0000","last_undegraded":"2024-02-12T17:12:31.598192+0000","last_fullsized":"2024-02-12T17:12:31.598192+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:05:54.311805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:05:54.311805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":586,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.097911+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:12:45.097911+0000","last_peered":"2024-02-12T17:12:45.097911+0000","last_clean":"2024-02-12T17:12:45.097911+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:12:45.097911+0000","last_undegraded":"2024-02-12T17:12:45.097911+0000","last_fullsized":"2024-02-12T17:12:45.097911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":637,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.090013+0000","last_change":"2024-02-12T17:10:03.877565+0000","last_active":"2024-02-12T17:12:45.090013+0000","last_peered":"2024-02-12T17:12:45.090013+0000","last_clean":"2024-02-12T17:09:58.611920+0000","last_became_active":"2024-02-12T17:10:03.877565+0000","last_became_peered":"2024-02-12T17:10:03.877565+0000","last_unstale":"2024-02-12T17:12:45.090013+0000","last_undegraded":"2024-02-12T17:12:45.090013+0000","last_fullsized":"2024-02-12T17:10:03.854437+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":170,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:47.763371+0000","last_change":"2024-02-12T17:12:47.763371+0000","last_active":"2024-02-12T17:12:47.763371+0000","last_peered":"2024-02-12T17:12:47.763371+0000","last_clean":"2024-02-12T17:12:47.763371+0000","last_became_active":"2024-02-12T17:11:51.603746+0000","last_became_peered":"2024-02-12T17:11:51.603746+0000","last_unstale":"2024-02-12T17:12:47.763371+0000","last_undegraded":"2024-02-12T17:12:47.763371+0000","last_fullsized":"2024-02-12T17:12:47.763371+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:47.763264+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.763264+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":617,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.091576+0000","last_change":"2024-02-12T17:12:03.922963+0000","last_active":"2024-02-12T17:12:45.091576+0000","last_peered":"2024-02-12T17:12:45.091576+0000","last_clean":"2024-02-12T17:12:45.091576+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:12:45.091576+0000","last_undegraded":"2024-02-12T17:12:45.091576+0000","last_fullsized":"2024-02-12T17:12:45.091576+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:12:03.922893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.922893+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"609'4337","reported_seq":4354,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:46.469572+0000","last_change":"2024-02-12T17:12:11.421593+0000","last_active":"2024-02-12T17:12:46.469572+0000","last_peered":"2024-02-12T17:12:46.469572+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.006012+0000","last_became_peered":"2024-02-12T17:11:59.006012+0000","last_unstale":"2024-02-12T17:12:46.469572+0000","last_undegraded":"2024-02-12T17:12:11.415937+0000","last_fullsized":"2024-02-12T17:11:58.996554+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4337,"ondisk_log_size":4337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":284229632,"num_objects":4337,"num_object_clones":0,"num_object_copies":13011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4337,"num_write_kb":277568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":4337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":503,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.096964+0000","last_change":"2024-02-12T17:10:03.942690+0000","last_active":"2024-02-12T17:12:45.096964+0000","last_peered":"2024-02-12T17:12:45.096964+0000","last_clean":"2024-02-12T17:04:37.845034+0000","last_became_active":"2024-02-12T17:10:03.942690+0000","last_became_peered":"2024-02-12T17:10:03.942690+0000","last_unstale":"2024-02-12T17:12:45.096964+0000","last_undegraded":"2024-02-12T17:12:45.096964+0000","last_fullsized":"2024-02-12T17:10:03.872324+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.088516+0000","last_change":"2024-02-12T17:12:03.905061+0000","last_active":"2024-02-12T17:12:45.088516+0000","last_peered":"2024-02-12T17:12:45.088516+0000","last_clean":"2024-02-12T17:12:45.088516+0000","last_became_active":"2024-02-12T17:11:51.597060+0000","last_became_peered":"2024-02-12T17:11:51.597060+0000","last_unstale":"2024-02-12T17:12:45.088516+0000","last_undegraded":"2024-02-12T17:12:45.088516+0000","last_fullsized":"2024-02-12T17:12:45.088516+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:03.904976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.904976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":630,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:45.089881+0000","last_change":"2024-02-12T17:10:03.878327+0000","last_active":"2024-02-12T17:12:45.089881+0000","last_peered":"2024-02-12T17:12:45.089881+0000","last_clean":"2024-02-12T17:09:58.611363+0000","last_became_active":"2024-02-12T17:10:03.878327+0000","last_became_peered":"2024-02-12T17:10:03.878327+0000","last_unstale":"2024-02-12T17:12:45.089881+0000","last_undegraded":"2024-02-12T17:10:03.854352+0000","last_fullsized":"2024-02-12T17:10:03.854244+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":159,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:45.089132+0000","last_change":"2024-02-12T17:10:03.875711+0000","last_active":"2024-02-12T17:12:45.089132+0000","last_peered":"2024-02-12T17:12:45.089132+0000","last_clean":"2024-02-12T17:09:50.556595+0000","last_became_active":"2024-02-12T17:10:03.875711+0000","last_became_peered":"2024-02-12T17:10:03.875711+0000","last_unstale":"2024-02-12T17:12:45.089132+0000","last_undegraded":"2024-02-12T17:10:03.857938+0000","last_fullsized":"2024-02-12T17:10:03.857823+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":633,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.097476+0000","last_change":"2024-02-12T17:12:17.628714+0000","last_active":"2024-02-12T17:12:45.097476+0000","last_peered":"2024-02-12T17:12:45.097476+0000","last_clean":"2024-02-12T17:12:45.097476+0000","last_became_active":"2024-02-12T17:12:17.628441+0000","last_became_peered":"2024-02-12T17:12:17.628441+0000","last_unstale":"2024-02-12T17:12:45.097476+0000","last_undegraded":"2024-02-12T17:12:45.097476+0000","last_fullsized":"2024-02-12T17:12:45.097476+0000","mapping_epoch":604,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:33.737556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:33.737556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,0],"avail_no_missing":["5","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"610'4414","reported_seq":4432,"reported_epoch":610,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:48.675316+0000","last_change":"2024-02-12T17:12:11.419723+0000","last_active":"2024-02-12T17:12:48.675316+0000","last_peered":"2024-02-12T17:12:48.675316+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.002565+0000","last_became_peered":"2024-02-12T17:11:59.002565+0000","last_unstale":"2024-02-12T17:12:48.675316+0000","last_undegraded":"2024-02-12T17:12:11.415309+0000","last_fullsized":"2024-02-12T17:11:58.998876+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4414,"ondisk_log_size":4414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289275904,"num_objects":4414,"num_object_clones":0,"num_object_copies":13242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4414,"num_write_kb":282496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":4414}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":117,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.095573+0000","last_change":"2024-02-12T17:10:03.868467+0000","last_active":"2024-02-12T17:12:45.095573+0000","last_peered":"2024-02-12T17:12:45.095573+0000","last_clean":"2024-02-12T17:09:19.536649+0000","last_became_active":"2024-02-12T17:10:03.868467+0000","last_became_peered":"2024-02-12T17:10:03.868467+0000","last_unstale":"2024-02-12T17:12:45.095573+0000","last_undegraded":"2024-02-12T17:12:45.095573+0000","last_fullsized":"2024-02-12T17:10:03.854942+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":640,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.765196+0000","last_change":"2024-02-12T17:12:45.765100+0000","last_active":"2024-02-12T17:12:45.765196+0000","last_peered":"2024-02-12T17:12:45.765196+0000","last_clean":"2024-02-12T17:12:45.765196+0000","last_became_active":"2024-02-12T17:11:52.610650+0000","last_became_peered":"2024-02-12T17:11:52.610650+0000","last_unstale":"2024-02-12T17:12:45.765196+0000","last_undegraded":"2024-02-12T17:12:45.765196+0000","last_fullsized":"2024-02-12T17:12:45.765196+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:45.764955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:45.764955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":641,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:45.098002+0000","last_change":"2024-02-12T17:10:03.886500+0000","last_active":"2024-02-12T17:12:45.098002+0000","last_peered":"2024-02-12T17:12:45.098002+0000","last_clean":"2024-02-12T17:09:58.605538+0000","last_became_active":"2024-02-12T17:10:03.886500+0000","last_became_peered":"2024-02-12T17:10:03.886500+0000","last_unstale":"2024-02-12T17:12:45.098002+0000","last_undegraded":"2024-02-12T17:10:03.861901+0000","last_fullsized":"2024-02-12T17:10:03.861792+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":106,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.092657+0000","last_change":"2024-02-12T17:09:21.560337+0000","last_active":"2024-02-12T17:12:45.092657+0000","last_peered":"2024-02-12T17:12:45.092657+0000","last_clean":"2024-02-12T17:12:45.092657+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:12:45.092657+0000","last_undegraded":"2024-02-12T17:12:45.092657+0000","last_fullsized":"2024-02-12T17:12:45.092657+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:46.526746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:46.526746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"610'4391","reported_seq":4409,"reported_epoch":610,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:48.854240+0000","last_change":"2024-02-12T17:12:11.426363+0000","last_active":"2024-02-12T17:12:48.854240+0000","last_peered":"2024-02-12T17:12:48.854240+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.002042+0000","last_became_peered":"2024-02-12T17:11:59.002042+0000","last_unstale":"2024-02-12T17:12:48.854240+0000","last_undegraded":"2024-02-12T17:12:11.415497+0000","last_fullsized":"2024-02-12T17:11:58.999050+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4391,"ondisk_log_size":4391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":287768576,"num_objects":4391,"num_object_clones":0,"num_object_copies":13173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":281024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4391}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":302,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.598279+0000","last_change":"2024-02-12T17:11:51.598581+0000","last_active":"2024-02-12T17:12:31.598279+0000","last_peered":"2024-02-12T17:12:31.598279+0000","last_clean":"2024-02-12T17:12:31.598279+0000","last_became_active":"2024-02-12T17:11:51.597611+0000","last_became_peered":"2024-02-12T17:11:51.597611+0000","last_unstale":"2024-02-12T17:12:31.598279+0000","last_undegraded":"2024-02-12T17:12:31.598279+0000","last_fullsized":"2024-02-12T17:12:31.598279+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:01.455969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.455969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":132,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.096733+0000","last_change":"2024-02-12T17:11:58.772251+0000","last_active":"2024-02-12T17:12:45.096733+0000","last_peered":"2024-02-12T17:12:45.096733+0000","last_clean":"2024-02-12T17:12:45.096733+0000","last_became_active":"2024-02-12T17:11:52.602659+0000","last_became_peered":"2024-02-12T17:11:52.602659+0000","last_unstale":"2024-02-12T17:12:45.096733+0000","last_undegraded":"2024-02-12T17:12:45.096733+0000","last_fullsized":"2024-02-12T17:12:45.096733+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,3],"avail_no_missing":["6","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":339,"reported_epoch":608,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:31.598078+0000","last_change":"2024-02-12T17:10:03.894531+0000","last_active":"2024-02-12T17:12:31.598078+0000","last_peered":"2024-02-12T17:12:31.598078+0000","last_clean":"2024-02-12T17:09:58.609394+0000","last_became_active":"2024-02-12T17:10:03.894531+0000","last_became_peered":"2024-02-12T17:10:03.894531+0000","last_unstale":"2024-02-12T17:12:31.598078+0000","last_undegraded":"2024-02-12T17:10:03.855132+0000","last_fullsized":"2024-02-12T17:10:03.855063+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":29,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.088386+0000","last_change":"2024-02-12T17:11:51.597397+0000","last_active":"2024-02-12T17:12:45.088386+0000","last_peered":"2024-02-12T17:12:45.088386+0000","last_clean":"2024-02-12T17:12:45.088386+0000","last_became_active":"2024-02-12T17:11:51.596535+0000","last_became_peered":"2024-02-12T17:11:51.596535+0000","last_unstale":"2024-02-12T17:12:45.088386+0000","last_undegraded":"2024-02-12T17:12:45.088386+0000","last_fullsized":"2024-02-12T17:12:45.088386+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:30.025325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:02:30.025325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"610'4449","reported_seq":4467,"reported_epoch":610,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:48.678420+0000","last_change":"2024-02-12T17:12:11.419367+0000","last_active":"2024-02-12T17:12:48.678420+0000","last_peered":"2024-02-12T17:12:48.678420+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.001873+0000","last_became_peered":"2024-02-12T17:11:59.001873+0000","last_unstale":"2024-02-12T17:12:48.678420+0000","last_undegraded":"2024-02-12T17:12:11.416817+0000","last_fullsized":"2024-02-12T17:11:58.996345+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4449,"ondisk_log_size":4449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":291569664,"num_objects":4449,"num_object_clones":0,"num_object_copies":13347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4449,"num_write_kb":284736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":4449}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":643,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.089453+0000","last_change":"2024-02-12T17:09:22.252402+0000","last_active":"2024-02-12T17:12:45.089453+0000","last_peered":"2024-02-12T17:12:45.089453+0000","last_clean":"2024-02-12T17:12:45.089453+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:12:45.089453+0000","last_undegraded":"2024-02-12T17:12:45.089453+0000","last_fullsized":"2024-02-12T17:12:45.089453+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:07:52.341731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.341731+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":106,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.096809+0000","last_change":"2024-02-12T17:09:21.594780+0000","last_active":"2024-02-12T17:12:45.096809+0000","last_peered":"2024-02-12T17:12:45.096809+0000","last_clean":"2024-02-12T17:12:45.096809+0000","last_became_active":"2024-02-12T17:09:21.539652+0000","last_became_peered":"2024-02-12T17:09:21.539652+0000","last_unstale":"2024-02-12T17:12:45.096809+0000","last_undegraded":"2024-02-12T17:12:45.096809+0000","last_fullsized":"2024-02-12T17:12:45.096809+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:42.272807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:42.272807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":744,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.096909+0000","last_change":"2024-02-12T17:12:05.850523+0000","last_active":"2024-02-12T17:12:45.096909+0000","last_peered":"2024-02-12T17:12:45.096909+0000","last_clean":"2024-02-12T17:12:45.096909+0000","last_became_active":"2024-02-12T17:11:51.597080+0000","last_became_peered":"2024-02-12T17:11:51.597080+0000","last_unstale":"2024-02-12T17:12:45.096909+0000","last_undegraded":"2024-02-12T17:12:45.096909+0000","last_fullsized":"2024-02-12T17:12:45.096909+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:12:05.850456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.850456+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"595'26","reported_seq":483,"reported_epoch":608,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:42.588774+0000","last_change":"2024-02-12T17:11:55.606456+0000","last_active":"2024-02-12T17:12:42.588774+0000","last_peered":"2024-02-12T17:12:42.588774+0000","last_clean":"2024-02-12T17:12:42.588774+0000","last_became_active":"2024-02-12T17:11:55.605944+0000","last_became_peered":"2024-02-12T17:11:55.605944+0000","last_unstale":"2024-02-12T17:12:42.588774+0000","last_undegraded":"2024-02-12T17:12:42.588774+0000","last_fullsized":"2024-02-12T17:12:42.588774+0000","mapping_epoch":594,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"610'4484","reported_seq":4505,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:48.593321+0000","last_change":"2024-02-12T17:11:59.007425+0000","last_active":"2024-02-12T17:12:48.593321+0000","last_peered":"2024-02-12T17:12:48.593321+0000","last_clean":"2024-02-12T17:12:48.593321+0000","last_became_active":"2024-02-12T17:11:59.007213+0000","last_became_peered":"2024-02-12T17:11:59.007213+0000","last_unstale":"2024-02-12T17:12:48.593321+0000","last_undegraded":"2024-02-12T17:12:48.593321+0000","last_fullsized":"2024-02-12T17:12:48.593321+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4484,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":293863424,"num_objects":4484,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":286976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":318,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.598192+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:12:31.598192+0000","last_peered":"2024-02-12T17:12:31.598192+0000","last_clean":"2024-02-12T17:12:31.598192+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:12:31.598192+0000","last_undegraded":"2024-02-12T17:12:31.598192+0000","last_fullsized":"2024-02-12T17:12:31.598192+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"596'28","reported_seq":949,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:46.623780+0000","last_change":"2024-02-12T17:12:07.826309+0000","last_active":"2024-02-12T17:12:46.623780+0000","last_peered":"2024-02-12T17:12:46.623780+0000","last_clean":"2024-02-12T17:12:46.623780+0000","last_became_active":"2024-02-12T17:11:55.613082+0000","last_became_peered":"2024-02-12T17:11:55.613082+0000","last_unstale":"2024-02-12T17:12:46.623780+0000","last_undegraded":"2024-02-12T17:12:46.623780+0000","last_fullsized":"2024-02-12T17:12:46.623780+0000","mapping_epoch":594,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'28","last_scrub_stamp":"2024-02-12T17:12:07.826227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:12:07.826227+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":276,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.089699+0000","last_change":"2024-02-12T17:12:16.873712+0000","last_active":"2024-02-12T17:12:45.089699+0000","last_peered":"2024-02-12T17:12:45.089699+0000","last_clean":"2024-02-12T17:12:45.089699+0000","last_became_active":"2024-02-12T17:11:52.608148+0000","last_became_peered":"2024-02-12T17:11:52.608148+0000","last_unstale":"2024-02-12T17:12:45.089699+0000","last_undegraded":"2024-02-12T17:12:45.089699+0000","last_fullsized":"2024-02-12T17:12:45.089699+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:16.873570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.873570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,5],"avail_no_missing":["7","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":163,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.096623+0000","last_change":"2024-02-12T17:12:04.705755+0000","last_active":"2024-02-12T17:12:45.096623+0000","last_peered":"2024-02-12T17:12:45.096623+0000","last_clean":"2024-02-12T17:12:45.096623+0000","last_became_active":"2024-02-12T17:11:52.602752+0000","last_became_peered":"2024-02-12T17:11:52.602752+0000","last_unstale":"2024-02-12T17:12:45.096623+0000","last_undegraded":"2024-02-12T17:12:45.096623+0000","last_fullsized":"2024-02-12T17:12:45.096623+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:04.705646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.705646+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,3],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"609'4310","reported_seq":4325,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:46.479010+0000","last_change":"2024-02-12T17:12:11.416306+0000","last_active":"2024-02-12T17:12:46.479010+0000","last_peered":"2024-02-12T17:12:46.479010+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.001433+0000","last_became_peered":"2024-02-12T17:11:59.001433+0000","last_unstale":"2024-02-12T17:12:46.479010+0000","last_undegraded":"2024-02-12T17:12:11.415676+0000","last_fullsized":"2024-02-12T17:11:58.996543+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4310,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":282460160,"num_objects":4310,"num_object_clones":0,"num_object_copies":12930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":275840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":4310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":635,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.098818+0000","last_change":"2024-02-12T17:10:03.874296+0000","last_active":"2024-02-12T17:12:45.098818+0000","last_peered":"2024-02-12T17:12:45.098818+0000","last_clean":"2024-02-12T17:09:58.605834+0000","last_became_active":"2024-02-12T17:10:03.874296+0000","last_became_peered":"2024-02-12T17:10:03.874296+0000","last_unstale":"2024-02-12T17:12:45.098818+0000","last_undegraded":"2024-02-12T17:12:45.098818+0000","last_fullsized":"2024-02-12T17:10:03.864117+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"596'24","reported_seq":965,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:46.623397+0000","last_change":"2024-02-12T17:11:55.612967+0000","last_active":"2024-02-12T17:12:46.623397+0000","last_peered":"2024-02-12T17:12:46.623397+0000","last_clean":"2024-02-12T17:12:46.623397+0000","last_became_active":"2024-02-12T17:11:55.612531+0000","last_became_peered":"2024-02-12T17:11:55.612531+0000","last_unstale":"2024-02-12T17:12:46.623397+0000","last_undegraded":"2024-02-12T17:12:46.623397+0000","last_fullsized":"2024-02-12T17:12:46.623397+0000","mapping_epoch":594,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'21","last_scrub_stamp":"2024-02-12T17:07:41.785018+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-02-12T16:50:50.799984+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.785018+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":210,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.088331+0000","last_change":"2024-02-12T17:10:03.879456+0000","last_active":"2024-02-12T17:12:45.088331+0000","last_peered":"2024-02-12T17:12:45.088331+0000","last_clean":"2024-02-12T16:59:07.210188+0000","last_became_active":"2024-02-12T17:10:03.879456+0000","last_became_peered":"2024-02-12T17:10:03.879456+0000","last_unstale":"2024-02-12T17:12:45.088331+0000","last_undegraded":"2024-02-12T17:12:45.088331+0000","last_fullsized":"2024-02-12T17:10:03.854791+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":186,"reported_epoch":608,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:31.599602+0000","last_change":"2024-02-12T17:11:53.452018+0000","last_active":"2024-02-12T17:12:31.599602+0000","last_peered":"2024-02-12T17:12:31.599602+0000","last_clean":"2024-02-12T17:12:31.599602+0000","last_became_active":"2024-02-12T17:11:52.628397+0000","last_became_peered":"2024-02-12T17:11:52.628397+0000","last_unstale":"2024-02-12T17:12:31.599602+0000","last_undegraded":"2024-02-12T17:12:31.599602+0000","last_fullsized":"2024-02-12T17:12:31.599602+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"610'4419","reported_seq":4437,"reported_epoch":610,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:48.610844+0000","last_change":"2024-02-12T17:12:11.418667+0000","last_active":"2024-02-12T17:12:48.610844+0000","last_peered":"2024-02-12T17:12:48.610844+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:59.005258+0000","last_became_peered":"2024-02-12T17:11:59.005258+0000","last_unstale":"2024-02-12T17:12:48.610844+0000","last_undegraded":"2024-02-12T17:12:11.415841+0000","last_fullsized":"2024-02-12T17:11:59.003935+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4419,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289603584,"num_objects":4419,"num_object_clones":0,"num_object_copies":13257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4419,"num_write_kb":282816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":4419}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.092107+0000","last_change":"2024-02-12T17:12:00.902004+0000","last_active":"2024-02-12T17:12:45.092107+0000","last_peered":"2024-02-12T17:12:45.092107+0000","last_clean":"2024-02-12T17:12:45.092107+0000","last_became_active":"2024-02-12T17:11:52.613393+0000","last_became_peered":"2024-02-12T17:11:52.613393+0000","last_unstale":"2024-02-12T17:12:45.092107+0000","last_undegraded":"2024-02-12T17:12:45.092107+0000","last_fullsized":"2024-02-12T17:12:45.092107+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":500,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:46.622130+0000","last_change":"2024-02-12T17:07:41.239456+0000","last_active":"2024-02-12T17:12:46.622130+0000","last_peered":"2024-02-12T17:12:46.622130+0000","last_clean":"2024-02-12T17:12:46.622130+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:12:46.622130+0000","last_undegraded":"2024-02-12T17:12:46.622130+0000","last_fullsized":"2024-02-12T17:12:46.622130+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:07:41.239350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:07:41.239350+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":239,"reported_epoch":608,"state":"active+undersized","last_fresh":"2024-02-12T17:12:31.598311+0000","last_change":"2024-02-12T17:10:03.893191+0000","last_active":"2024-02-12T17:12:31.598311+0000","last_peered":"2024-02-12T17:12:31.598311+0000","last_clean":"2024-02-12T17:08:07.415377+0000","last_became_active":"2024-02-12T17:10:03.893191+0000","last_became_peered":"2024-02-12T17:10:03.893191+0000","last_unstale":"2024-02-12T17:12:31.598311+0000","last_undegraded":"2024-02-12T17:12:31.598311+0000","last_fullsized":"2024-02-12T17:10:03.856411+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":392,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.598290+0000","last_change":"2024-02-12T17:11:59.133004+0000","last_active":"2024-02-12T17:12:31.598290+0000","last_peered":"2024-02-12T17:12:31.598290+0000","last_clean":"2024-02-12T17:12:31.598290+0000","last_became_active":"2024-02-12T17:11:51.601852+0000","last_became_peered":"2024-02-12T17:11:51.601852+0000","last_unstale":"2024-02-12T17:12:31.598290+0000","last_undegraded":"2024-02-12T17:12:31.598290+0000","last_fullsized":"2024-02-12T17:12:31.598290+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"609'4320","reported_seq":4339,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:46.464568+0000","last_change":"2024-02-12T17:11:59.496950+0000","last_active":"2024-02-12T17:12:46.464568+0000","last_peered":"2024-02-12T17:12:46.464568+0000","last_clean":"2024-02-12T17:12:46.464568+0000","last_became_active":"2024-02-12T17:11:59.496585+0000","last_became_peered":"2024-02-12T17:11:59.496585+0000","last_unstale":"2024-02-12T17:12:46.464568+0000","last_undegraded":"2024-02-12T17:12:46.464568+0000","last_fullsized":"2024-02-12T17:12:46.464568+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4320,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":283115520,"num_objects":4320,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4320,"num_write_kb":276480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":329,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.599753+0000","last_change":"2024-02-12T17:11:51.597590+0000","last_active":"2024-02-12T17:12:31.599753+0000","last_peered":"2024-02-12T17:12:31.599753+0000","last_clean":"2024-02-12T17:12:31.599753+0000","last_became_active":"2024-02-12T17:11:51.597221+0000","last_became_peered":"2024-02-12T17:11:51.597221+0000","last_unstale":"2024-02-12T17:12:31.599753+0000","last_undegraded":"2024-02-12T17:12:31.599753+0000","last_fullsized":"2024-02-12T17:12:31.599753+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":30,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.090806+0000","last_change":"2024-02-12T17:11:52.605479+0000","last_active":"2024-02-12T17:12:45.090806+0000","last_peered":"2024-02-12T17:12:45.090806+0000","last_clean":"2024-02-12T17:12:45.090806+0000","last_became_active":"2024-02-12T17:11:52.605068+0000","last_became_peered":"2024-02-12T17:11:52.605068+0000","last_unstale":"2024-02-12T17:12:45.090806+0000","last_undegraded":"2024-02-12T17:12:45.090806+0000","last_fullsized":"2024-02-12T17:12:45.090806+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:45.567340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:07:45.567340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,6],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":601,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.089033+0000","last_change":"2024-02-12T17:11:51.602622+0000","last_active":"2024-02-12T17:12:45.089033+0000","last_peered":"2024-02-12T17:12:45.089033+0000","last_clean":"2024-02-12T17:12:45.089033+0000","last_became_active":"2024-02-12T17:11:51.602161+0000","last_became_peered":"2024-02-12T17:11:51.602161+0000","last_unstale":"2024-02-12T17:12:45.089033+0000","last_undegraded":"2024-02-12T17:12:45.089033+0000","last_fullsized":"2024-02-12T17:12:45.089033+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:51.877830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:51.877830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":48,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.091491+0000","last_change":"2024-02-12T17:11:53.984861+0000","last_active":"2024-02-12T17:12:45.091491+0000","last_peered":"2024-02-12T17:12:45.091491+0000","last_clean":"2024-02-12T17:12:45.091491+0000","last_became_active":"2024-02-12T17:11:52.603933+0000","last_became_peered":"2024-02-12T17:11:52.603933+0000","last_unstale":"2024-02-12T17:12:45.091491+0000","last_undegraded":"2024-02-12T17:12:45.091491+0000","last_fullsized":"2024-02-12T17:12:45.091491+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'31","last_scrub_stamp":"2024-02-12T17:07:37.774472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.774472+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,6],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"610'4421","reported_seq":4442,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:48.794277+0000","last_change":"2024-02-12T17:11:59.009775+0000","last_active":"2024-02-12T17:12:48.794277+0000","last_peered":"2024-02-12T17:12:48.794277+0000","last_clean":"2024-02-12T17:12:48.794277+0000","last_became_active":"2024-02-12T17:11:59.009543+0000","last_became_peered":"2024-02-12T17:11:59.009543+0000","last_unstale":"2024-02-12T17:12:48.794277+0000","last_undegraded":"2024-02-12T17:12:48.794277+0000","last_fullsized":"2024-02-12T17:12:48.794277+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4421,"ondisk_log_size":4421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289734656,"num_objects":4421,"num_object_clones":0,"num_object_copies":13263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4421,"num_write_kb":282944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":633,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.089822+0000","last_change":"2024-02-12T17:12:08.013330+0000","last_active":"2024-02-12T17:12:45.089822+0000","last_peered":"2024-02-12T17:12:45.089822+0000","last_clean":"2024-02-12T17:12:45.089822+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:12:45.089822+0000","last_undegraded":"2024-02-12T17:12:45.089822+0000","last_fullsized":"2024-02-12T17:12:45.089822+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.013247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.013247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":528,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T17:12:30.037961+0000","last_change":"2024-02-12T17:12:04.711418+0000","last_active":"2024-02-12T17:12:30.037961+0000","last_peered":"2024-02-12T17:12:30.037961+0000","last_clean":"2024-02-12T17:12:30.037961+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:12:30.037961+0000","last_undegraded":"2024-02-12T17:12:30.037961+0000","last_fullsized":"2024-02-12T17:12:30.037961+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:04.711333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.711333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":352,"reported_epoch":609,"state":"active+undersized","last_fresh":"2024-02-12T17:12:45.097250+0000","last_change":"2024-02-12T17:10:03.873123+0000","last_active":"2024-02-12T17:12:45.097250+0000","last_peered":"2024-02-12T17:12:45.097250+0000","last_clean":"2024-02-12T17:09:58.605059+0000","last_became_active":"2024-02-12T17:10:03.873123+0000","last_became_peered":"2024-02-12T17:10:03.873123+0000","last_unstale":"2024-02-12T17:12:45.097250+0000","last_undegraded":"2024-02-12T17:12:45.097250+0000","last_fullsized":"2024-02-12T17:10:03.863800+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":156,"reported_epoch":609,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:45.092744+0000","last_change":"2024-02-12T17:10:03.883757+0000","last_active":"2024-02-12T17:12:45.092744+0000","last_peered":"2024-02-12T17:12:45.092744+0000","last_clean":"2024-02-12T17:09:58.608650+0000","last_became_active":"2024-02-12T17:10:03.883757+0000","last_became_peered":"2024-02-12T17:10:03.883757+0000","last_unstale":"2024-02-12T17:12:45.092744+0000","last_undegraded":"2024-02-12T17:10:03.857329+0000","last_fullsized":"2024-02-12T17:10:03.857219+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"610'4493","reported_seq":4514,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:48.866328+0000","last_change":"2024-02-12T17:11:59.007574+0000","last_active":"2024-02-12T17:12:48.866328+0000","last_peered":"2024-02-12T17:12:48.866328+0000","last_clean":"2024-02-12T17:12:48.866328+0000","last_became_active":"2024-02-12T17:11:59.007300+0000","last_became_peered":"2024-02-12T17:11:59.007300+0000","last_unstale":"2024-02-12T17:12:48.866328+0000","last_undegraded":"2024-02-12T17:12:48.866328+0000","last_fullsized":"2024-02-12T17:12:48.866328+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4493,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":294453248,"num_objects":4493,"num_object_clones":0,"num_object_copies":13479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4493,"num_write_kb":287552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":312,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.088444+0000","last_change":"2024-02-12T17:06:43.014800+0000","last_active":"2024-02-12T17:12:45.088444+0000","last_peered":"2024-02-12T17:12:45.088444+0000","last_clean":"2024-02-12T17:12:45.088444+0000","last_became_active":"2024-02-12T17:04:46.024228+0000","last_became_peered":"2024-02-12T17:04:46.024228+0000","last_unstale":"2024-02-12T17:12:45.088444+0000","last_undegraded":"2024-02-12T17:12:45.088444+0000","last_fullsized":"2024-02-12T17:12:45.088444+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:06:43.014737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.088834+0000","last_change":"2024-02-12T17:11:58.848344+0000","last_active":"2024-02-12T17:12:45.088834+0000","last_peered":"2024-02-12T17:12:45.088834+0000","last_clean":"2024-02-12T17:12:45.088834+0000","last_became_active":"2024-02-12T17:11:51.593680+0000","last_became_peered":"2024-02-12T17:11:51.593680+0000","last_unstale":"2024-02-12T17:12:45.088834+0000","last_undegraded":"2024-02-12T17:12:45.088834+0000","last_fullsized":"2024-02-12T17:12:45.088834+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.848275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.848275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":285,"reported_epoch":608,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:31.600038+0000","last_change":"2024-02-12T17:12:01.016307+0000","last_active":"2024-02-12T17:12:31.600038+0000","last_peered":"2024-02-12T17:12:31.600038+0000","last_clean":"2024-02-12T17:12:31.600038+0000","last_became_active":"2024-02-12T17:12:01.015764+0000","last_became_peered":"2024-02-12T17:12:01.015764+0000","last_unstale":"2024-02-12T17:12:31.600038+0000","last_undegraded":"2024-02-12T17:12:31.600038+0000","last_fullsized":"2024-02-12T17:12:31.600038+0000","mapping_epoch":591,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":40,"reported_epoch":609,"state":"active+clean+remapped","last_fresh":"2024-02-12T17:12:45.089535+0000","last_change":"2024-02-12T17:11:54.090713+0000","last_active":"2024-02-12T17:12:45.089535+0000","last_peered":"2024-02-12T17:12:45.089535+0000","last_clean":"2024-02-12T17:12:45.089535+0000","last_became_active":"2024-02-12T17:11:52.619575+0000","last_became_peered":"2024-02-12T17:11:52.619575+0000","last_unstale":"2024-02-12T17:12:45.089535+0000","last_undegraded":"2024-02-12T17:12:45.089535+0000","last_fullsized":"2024-02-12T17:12:45.089535+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'15","last_scrub_stamp":"2024-02-12T17:07:38.887701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:38.887701+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,5],"avail_no_missing":["4","3","5"],"object_location_counts":[{"shards":"3,4,5","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"610'4395","reported_seq":4412,"reported_epoch":610,"state":"active+undersized+degraded","last_fresh":"2024-02-12T17:12:48.428085+0000","last_change":"2024-02-12T17:12:11.428956+0000","last_active":"2024-02-12T17:12:48.428085+0000","last_peered":"2024-02-12T17:12:48.428085+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:11:58.999911+0000","last_became_peered":"2024-02-12T17:11:58.999911+0000","last_unstale":"2024-02-12T17:12:48.428085+0000","last_undegraded":"2024-02-12T17:12:11.424883+0000","last_fullsized":"2024-02-12T17:11:58.993560+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4395,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":288030720,"num_objects":4395,"num_object_clones":0,"num_object_copies":13185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4395,"num_write_kb":281280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":4395}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":616,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.097579+0000","last_change":"2024-02-12T17:12:11.906816+0000","last_active":"2024-02-12T17:12:45.097579+0000","last_peered":"2024-02-12T17:12:45.097579+0000","last_clean":"2024-02-12T17:12:45.097579+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:12:45.097579+0000","last_undegraded":"2024-02-12T17:12:45.097579+0000","last_fullsized":"2024-02-12T17:12:45.097579+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:11.906737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:11.906737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":293,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T17:12:31.600394+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:12:31.600394+0000","last_peered":"2024-02-12T17:12:31.600394+0000","last_clean":"2024-02-12T17:12:31.600394+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:12:31.600394+0000","last_undegraded":"2024-02-12T17:12:31.600394+0000","last_fullsized":"2024-02-12T17:12:31.600394+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":734,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-12T17:12:45.100242+0000","last_change":"2024-02-12T17:12:16.820857+0000","last_active":"2024-02-12T17:12:45.100242+0000","last_peered":"2024-02-12T17:12:45.100242+0000","last_clean":"2024-02-12T17:12:45.100242+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:12:45.100242+0000","last_undegraded":"2024-02-12T17:12:45.100242+0000","last_fullsized":"2024-02-12T17:12:45.100242+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:12:16.820782+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.820782+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"595'29","reported_seq":1043,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-12T17:12:47.788400+0000","last_change":"2024-02-12T17:12:47.788400+0000","last_active":"2024-02-12T17:12:47.788400+0000","last_peered":"2024-02-12T17:12:47.788400+0000","last_clean":"2024-02-12T17:12:47.788400+0000","last_became_active":"2024-02-12T17:11:55.609231+0000","last_became_peered":"2024-02-12T17:11:55.609231+0000","last_unstale":"2024-02-12T17:12:47.788400+0000","last_undegraded":"2024-02-12T17:12:47.788400+0000","last_fullsized":"2024-02-12T17:12:47.788400+0000","mapping_epoch":594,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'29","last_scrub_stamp":"2024-02-12T17:12:47.788327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.788327+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4613079040,"num_objects":70390,"num_object_clones":0,"num_object_copies":211170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70390,"num_write_kb":4504960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11311841280,"data_stored":11311841280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70390,"ondisk_log_size":70390,"up":39,"acting":39,"num_store_stats":7},{"poolid":2,"num_pg":29,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":69,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":647,"num_read_kb":642,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":50,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1512,"num_read_kb":1269,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":14918,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1028,"ondisk_log_size":1028,"up":73,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":58,"acting":63,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":60,"acting":61,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234265,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2797776,"kb_used_data":1902216,"kb_used_omap":0,"kb_used_meta":895552,"kb_avail":90947376,"statfs":{"total":95995035648,"available":93130113024,"internally_reserved":0,"allocated":1947869184,"data_stored":1946887908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917045248},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.7050000000000001,"15min":1.966},"min":{"1min":0.26000000000000001,"5min":0.218,"15min":0.16700000000000001},"max":{"1min":7.0629999999999997,"5min":8.8510000000000009,"15min":16.759},"last":3.3889999999999998},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.54,"15min":2.0169999999999999},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.19600000000000001},"max":{"1min":10.965999999999999,"5min":10.965999999999999,"15min":17.844000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":2.23,"15min":1.8560000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":6.4539999999999997,"5min":13.224,"15min":13.224},"last":1.234},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.6679999999999999,"15min":1.7549999999999999},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.22700000000000001},"max":{"1min":6.4660000000000002,"5min":10.194000000000001,"15min":17.324999999999999},"last":1.165}]},{"osd":2,"last update":"Mon Feb 12 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":4.3609999999999998,"15min":2.653},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":4.742,"5min":218.35300000000001,"15min":218.35300000000001},"last":6.6639999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":3.8839999999999999,"15min":2.8879999999999999},"min":{"1min":0.26800000000000002,"5min":0.247,"15min":0.23000000000000001},"max":{"1min":2.7440000000000002,"5min":218.42099999999999,"15min":218.42099999999999},"last":0.375}]},{"osd":3,"last update":"Mon Feb 12 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.792,"15min":1.9139999999999999},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":8.7309999999999999,"5min":12.781000000000001,"15min":13.608000000000001},"last":3.4159999999999999},{"interface":"front","average":{"1min":2.7410000000000001,"5min":2.2269999999999999,"15min":2.4369999999999998},"min":{"1min":0.32700000000000001,"5min":0.28399999999999997,"15min":0.25600000000000001},"max":{"1min":13.452,"5min":13.699,"15min":31.152000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.73199999999999998,"15min":2.0310000000000001},"min":{"1min":0.23499999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":10.789999999999999,"5min":10.789999999999999,"15min":50.406999999999996},"last":3.6749999999999998},{"interface":"front","average":{"1min":1.167,"5min":0.79700000000000004,"15min":2.4660000000000002},"min":{"1min":0.252,"5min":0.249,"15min":0.249},"max":{"1min":13.423,"5min":13.423,"15min":50.168999999999997},"last":1.5049999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.2010000000000001,"15min":2.294},"min":{"1min":0.27300000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.8099999999999996,"5min":56.560000000000002,"15min":56.560000000000002},"last":3.359},{"interface":"front","average":{"1min":0.40100000000000002,"5min":1.1240000000000001,"15min":2.3559999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65900000000000003,"5min":53.963999999999999,"15min":53.963999999999999},"last":1.022}]},{"osd":6,"last update":"Mon Feb 12 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":0.81499999999999995,"15min":1.849},"min":{"1min":0.32400000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":21.620000000000001,"5min":21.620000000000001,"15min":21.620000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.81299999999999994,"15min":1.825},"min":{"1min":0.30399999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":8.6319999999999997,"5min":8.6319999999999997,"15min":13.676},"last":4.0140000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528606,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2848904,"kb_used_data":2251008,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":90896248,"statfs":{"total":95995035648,"available":93077757952,"internally_reserved":0,"allocated":2305032192,"data_stored":2304059306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.7250000000000001,"15min":1.835},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":3.1179999999999999,"5min":27.744,"15min":27.744},"last":2.742},{"interface":"front","average":{"1min":1.665,"5min":1.794,"15min":1.8220000000000001},"min":{"1min":0.246,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":8.0190000000000001,"5min":27.614000000000001,"15min":27.614000000000001},"last":5.069}]},{"osd":1,"last update":"Mon Feb 12 17:09:40 2024","interfaces":[{"interface":"back","average":{"1min":2.9870000000000001,"5min":1.74,"15min":1.629},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":14.884,"5min":14.884,"15min":14.884},"last":1.77},{"interface":"front","average":{"1min":3.0430000000000001,"5min":1.8220000000000001,"15min":1.8340000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":17.106999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":2.5990000000000002,"15min":2.327},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":7.2480000000000002,"5min":24.300999999999998,"15min":56.798000000000002},"last":5.109},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.9159999999999999,"15min":2.1349999999999998},"min":{"1min":0.248,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.0190000000000001,"5min":24.143999999999998,"15min":56.780999999999999},"last":2.7770000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":4.4870000000000001,"5min":3.109,"15min":2.1339999999999999},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.25},"max":{"1min":58.420999999999999,"5min":58.420999999999999,"15min":58.420999999999999},"last":2.6070000000000002},{"interface":"front","average":{"1min":5.3499999999999996,"5min":3.173,"15min":2.4399999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":58.709000000000003,"5min":58.709000000000003,"15min":58.709000000000003},"last":5.1319999999999997}]},{"osd":4,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.47399999999999998,"15min":1.6479999999999999},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.7210000000000001,"5min":2.456,"15min":24.956},"last":7.6440000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.59899999999999998,"15min":1.542},"min":{"1min":0.29299999999999998,"5min":0.222,"15min":0.222},"max":{"1min":1.1799999999999999,"5min":3.8690000000000002,"15min":13.728},"last":6.8570000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.90600000000000003,"15min":2.036},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.245},"max":{"1min":1.054,"5min":34.869,"15min":34.869},"last":3.8700000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.95999999999999996,"15min":2.302},"min":{"1min":0.27500000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":8.8870000000000005,"5min":33.591000000000001,"15min":33.591000000000001},"last":2.8090000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:12:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.67800000000000005,"15min":2.036},"min":{"1min":0.29399999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.94899999999999995,"5min":6.9279999999999999,"15min":31.033000000000001},"last":6.6550000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.69499999999999995,"15min":1.994},"min":{"1min":0.33900000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.6619999999999999,"5min":5.2389999999999999,"15min":29.34},"last":5.1799999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575125,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917072,"kb_used_data":319176,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":92828080,"statfs":{"total":95995035648,"available":95055953920,"internally_reserved":0,"allocated":326836224,"data_stored":325617050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2009999999999996,"5min":2.5369999999999999,"15min":1.7230000000000001},"min":{"1min":0.23499999999999999,"5min":0.20699999999999999,"15min":0.17499999999999999},"max":{"1min":65.328999999999994,"5min":65.328999999999994,"15min":65.328999999999994},"last":3.5190000000000001},{"interface":"front","average":{"1min":6.6280000000000001,"5min":2.516,"15min":1.7050000000000001},"min":{"1min":0.22800000000000001,"5min":0.193,"15min":0.189},"max":{"1min":65.372,"5min":65.372,"15min":65.372},"last":10.164}]},{"osd":2,"last update":"Mon Feb 12 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":5.9429999999999996,"5min":2.6989999999999998,"15min":2.0019999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.191},"max":{"1min":51.820999999999998,"5min":51.820999999999998,"15min":51.820999999999998},"last":2.2229999999999999},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.395,"15min":1.847},"min":{"1min":0.223,"5min":0.223,"15min":0.19},"max":{"1min":53.802999999999997,"5min":53.802999999999997,"15min":53.802999999999997},"last":9.8190000000000008}]},{"osd":3,"last update":"Mon Feb 12 17:09:37 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":2.7789999999999999,"15min":1.823},"min":{"1min":0.308,"5min":0.28999999999999998,"15min":0.23100000000000001},"max":{"1min":10.977,"5min":64.528000000000006,"15min":64.528000000000006},"last":12.565},{"interface":"front","average":{"1min":2.9289999999999998,"5min":2.4449999999999998,"15min":1.647},"min":{"1min":0.34200000000000003,"5min":0.312,"15min":0.246},"max":{"1min":11.124000000000001,"5min":63.753999999999998,"15min":63.753999999999998},"last":3.452}]},{"osd":4,"last update":"Mon Feb 12 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.6859999999999999,"15min":1.8879999999999999},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":7.556,"5min":12.02,"15min":12.91},"last":3.419},{"interface":"front","average":{"1min":1.849,"5min":1.5449999999999999,"15min":1.8260000000000001},"min":{"1min":0.374,"5min":0.33000000000000002,"15min":0.25900000000000001},"max":{"1min":9.6159999999999997,"5min":12.193,"15min":16.074000000000002},"last":4.04}]},{"osd":5,"last update":"Mon Feb 12 17:09:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.8819999999999999,"15min":2.0739999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.306,"5min":22.154,"15min":22.154},"last":3.4980000000000002},{"interface":"front","average":{"1min":1.423,"5min":1.754,"15min":1.913},"min":{"1min":0.317,"5min":0.317,"15min":0.27300000000000002},"max":{"1min":8.5820000000000007,"5min":21.803000000000001,"15min":21.803000000000001},"last":4.0060000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.7170000000000001,"15min":1.9430000000000001},"min":{"1min":0.38,"5min":0.36499999999999999,"15min":0.30599999999999999},"max":{"1min":7.6210000000000004,"5min":7.6210000000000004,"15min":13.834},"last":3.4750000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.7470000000000001,"15min":1.7789999999999999},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.307},"max":{"1min":4.3700000000000001,"5min":16.172999999999998,"15min":16.172999999999998},"last":4.0640000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.9239999999999999,"15min":2.0289999999999999},"min":{"1min":0.377,"5min":0.33100000000000002,"15min":0.28799999999999998},"max":{"1min":7.3099999999999996,"5min":8.7560000000000002,"15min":9.5239999999999991},"last":2.1880000000000002},{"interface":"front","average":{"1min":1.8200000000000001,"5min":2.0790000000000002,"15min":2.1669999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":9.3510000000000009,"5min":13.965999999999999,"15min":20.548999999999999},"last":5.835}]}]},{"osd":0,"up_from":8,"seq":34359738682,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1293168,"kb_used_data":392872,"kb_used_omap":0,"kb_used_meta":900288,"kb_avail":92451984,"statfs":{"total":95995035648,"available":94670831616,"internally_reserved":0,"allocated":402300928,"data_stored":401401019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":921894912},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:09:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8809999999999998,"5min":1.694,"15min":1.484},"min":{"1min":0.253,"5min":0.22,"15min":0.17399999999999999},"max":{"1min":9.7420000000000009,"5min":10.807,"15min":23.510999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":4.3070000000000004,"5min":2.5259999999999998,"15min":1.6970000000000001},"min":{"1min":0.29699999999999999,"5min":0.20399999999999999,"15min":0.193},"max":{"1min":10.092000000000001,"5min":10.741,"15min":16.16},"last":6.2830000000000004}]},{"osd":2,"last update":"Mon Feb 12 17:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":3.2909999999999999,"15min":1.8600000000000001},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.312000000000001,"5min":19.312000000000001,"15min":19.312000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.476,"5min":4.0739999999999998,"15min":2.2480000000000002},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":11.455,"5min":26.039999999999999,"15min":26.039999999999999},"last":0.26900000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":4.6040000000000001,"15min":2.4740000000000002},"min":{"1min":0.221,"5min":0.186,"15min":0.186},"max":{"1min":3.133,"5min":87.311000000000007,"15min":87.311000000000007},"last":0.221},{"interface":"front","average":{"1min":1.032,"5min":4.9290000000000003,"15min":2.6469999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.221},"max":{"1min":10.271000000000001,"5min":90.991,"15min":90.991},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.9319999999999999,"5min":1.5429999999999999,"15min":1.6100000000000001},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.23599999999999999},"max":{"1min":15.141999999999999,"5min":15.141999999999999,"15min":16.888999999999999},"last":0.308},{"interface":"front","average":{"1min":1.8009999999999999,"5min":2.8199999999999998,"15min":2.1560000000000001},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":11.057,"5min":26.082999999999998,"15min":26.082999999999998},"last":0.30299999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:12:30 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":2.3210000000000002,"15min":2.3929999999999998},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28000000000000003},"max":{"1min":15.090999999999999,"5min":17.754000000000001,"15min":50.404000000000003},"last":2.145},{"interface":"front","average":{"1min":1.405,"5min":1.7070000000000001,"15min":2.2029999999999998},"min":{"1min":0.36399999999999999,"5min":0.31900000000000001,"15min":0.28299999999999997},"max":{"1min":15.164,"5min":15.164,"15min":49.979999999999997},"last":2.0070000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.931,"5min":3.0609999999999999,"15min":2.1960000000000002},"min":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.29999999999999999},"max":{"1min":11.035,"5min":26.109999999999999,"15min":26.109999999999999},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.748,"5min":2.4060000000000001,"15min":2.137},"min":{"1min":0.39700000000000002,"5min":0.372,"15min":0.33600000000000002},"max":{"1min":15.06,"5min":15.847,"15min":16.294},"last":4.5759999999999996}]},{"osd":7,"last update":"Mon Feb 12 17:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.8069999999999999,"15min":2.077},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":15.183999999999999,"5min":15.183999999999999,"15min":19.373000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.0900000000000001,"5min":2.7909999999999999,"15min":2.4809999999999999},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.375},"max":{"1min":11.006,"5min":26.135000000000002,"15min":34.722999999999999},"last":3.5670000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411632,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3499324,"kb_used_data":1992948,"kb_used_omap":0,"kb_used_meta":1506368,"kb_avail":90245828,"statfs":{"total":95995035648,"available":92411727872,"internally_reserved":0,"allocated":2040778752,"data_stored":2039818202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1542520832},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":2.5009999999999999,"15min":1.675},"min":{"1min":0.20499999999999999,"5min":0.189,"15min":0.17000000000000001},"max":{"1min":1.7230000000000001,"5min":21.445,"15min":36.100999999999999},"last":2.9319999999999999},{"interface":"front","average":{"1min":0.495,"5min":2.4950000000000001,"15min":1.9199999999999999},"min":{"1min":0.214,"5min":0.20300000000000001,"15min":0.193},"max":{"1min":1.8169999999999999,"5min":19.879000000000001,"15min":36.299999999999997},"last":0.19900000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":3.282,"5min":1.948,"15min":1.776},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.19600000000000001},"max":{"1min":10.922000000000001,"5min":20.763999999999999,"15min":20.763999999999999},"last":6.2999999999999998},{"interface":"front","average":{"1min":4.3310000000000004,"5min":2.3029999999999999,"15min":1.984},"min":{"1min":0.31900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":12.552,"5min":21.361999999999998,"15min":21.361999999999998},"last":6.657}]},{"osd":3,"last update":"Mon Feb 12 17:11:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":4.0300000000000002,"15min":2.101},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.1740000000000004,"5min":40.759999999999998,"15min":40.759999999999998},"last":2.8959999999999999},{"interface":"front","average":{"1min":2.6480000000000001,"5min":3.4380000000000002,"15min":1.9159999999999999},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.23699999999999999},"max":{"1min":10.912000000000001,"5min":21.702999999999999,"15min":21.702999999999999},"last":3.008}]},{"osd":4,"last update":"Mon Feb 12 17:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":2.2469999999999999,"15min":1.9810000000000001},"min":{"1min":0.28799999999999998,"5min":0.245,"15min":0.245},"max":{"1min":5.5739999999999998,"5min":21.515000000000001,"15min":21.515000000000001},"last":2.9630000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":2.0800000000000001,"15min":2.1219999999999999},"min":{"1min":0.41299999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.5659999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":1.8300000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":2.1240000000000001,"15min":1.9690000000000001},"min":{"1min":0.26000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":1.708,"5min":21.542999999999999,"15min":32.57},"last":2.984},{"interface":"front","average":{"1min":0.70399999999999996,"5min":2.0129999999999999,"15min":2.23},"min":{"1min":0.33300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":2.3140000000000001,"5min":20.937000000000001,"15min":31.905999999999999},"last":3.8900000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:11:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":2.1539999999999999,"15min":1.9830000000000001},"min":{"1min":0.44600000000000001,"5min":0.313,"15min":0.30199999999999999},"max":{"1min":2.3839999999999999,"5min":20.966999999999999,"15min":20.966999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":2.016,"5min":1.9470000000000001,"15min":1.875},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.524,"5min":20.193000000000001,"15min":20.193000000000001},"last":0.31}]},{"osd":7,"last update":"Mon Feb 12 17:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.9530000000000001,"15min":2.5139999999999998},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.7470000000000001,"5min":20.221,"15min":45.468000000000004},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.78500000000000003,"5min":2.1429999999999998,"15min":2.5089999999999999},"min":{"1min":0.35699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.3540000000000001,"5min":20.995999999999999,"15min":46.656999999999996},"last":0.54100000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149994,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2838932,"kb_used_data":1929292,"kb_used_omap":0,"kb_used_meta":909632,"kb_avail":90906220,"statfs":{"total":95995035648,"available":93087969280,"internally_reserved":0,"allocated":1975595008,"data_stored":1974603380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":931463168},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:11:45 2024","interfaces":[{"interface":"back","average":{"1min":4.9969999999999999,"5min":4.1900000000000004,"15min":2.1720000000000002},"min":{"1min":0.186,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":18.297999999999998,"5min":18.497,"15min":58.451999999999998},"last":0.20300000000000001},{"interface":"front","average":{"1min":6.1719999999999997,"5min":4.6280000000000001,"15min":2.5859999999999999},"min":{"1min":0.216,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":72.688000000000002,"5min":72.688000000000002,"15min":72.688000000000002},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:09:40 2024","interfaces":[{"interface":"back","average":{"1min":5.2539999999999996,"5min":2.9020000000000001,"15min":1.8779999999999999},"min":{"1min":0.28799999999999998,"5min":0.218,"15min":0.218},"max":{"1min":18.859999999999999,"5min":18.859999999999999,"15min":18.859999999999999},"last":8.9280000000000008},{"interface":"front","average":{"1min":3.4009999999999998,"5min":1.7889999999999999,"15min":1.3759999999999999},"min":{"1min":0.48499999999999999,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":14.654,"5min":14.654,"15min":14.962},"last":5.1630000000000003}]},{"osd":2,"last update":"Mon Feb 12 17:11:45 2024","interfaces":[{"interface":"back","average":{"1min":3.6920000000000002,"5min":3.6669999999999998,"15min":1.8400000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":40.954999999999998,"5min":40.954999999999998,"15min":40.954999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":6.6769999999999996,"5min":5.0549999999999997,"15min":2.2490000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.22},"max":{"1min":46.408000000000001,"5min":46.408000000000001,"15min":46.408000000000001},"last":0.26400000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:12:07 2024","interfaces":[{"interface":"back","average":{"1min":4.7290000000000001,"5min":3.3130000000000002,"15min":2.5},"min":{"1min":0.34100000000000003,"5min":0.317,"15min":0.29599999999999999},"max":{"1min":72.659000000000006,"5min":72.659000000000006,"15min":72.659000000000006},"last":5.5650000000000004},{"interface":"front","average":{"1min":3.2570000000000001,"5min":2.0630000000000002,"15min":1.9299999999999999},"min":{"1min":0.28100000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":40.930999999999997,"5min":40.930999999999997,"15min":40.930999999999997},"last":0.434}]},{"osd":5,"last update":"Mon Feb 12 17:12:30 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":3.085,"15min":2.4409999999999998},"min":{"1min":0.38700000000000001,"5min":0.30299999999999999,"15min":0.253},"max":{"1min":6.3209999999999997,"5min":17.940999999999999,"15min":17.940999999999999},"last":5.4249999999999998},{"interface":"front","average":{"1min":4.6669999999999998,"5min":3.456,"15min":2.823},"min":{"1min":0.371,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":72.620000000000005,"5min":72.620000000000005,"15min":72.620000000000005},"last":0.75900000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:11:45 2024","interfaces":[{"interface":"back","average":{"1min":3.3860000000000001,"5min":2.3889999999999998,"15min":1.855},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":40.889000000000003,"5min":40.889000000000003,"15min":40.889000000000003},"last":1.625},{"interface":"front","average":{"1min":3.702,"5min":3.1880000000000002,"15min":2.1440000000000001},"min":{"1min":0.42899999999999999,"5min":0.35599999999999998,"15min":0.32600000000000001},"max":{"1min":16.989999999999998,"5min":18.265000000000001,"15min":18.265000000000001},"last":0.98499999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:12:07 2024","interfaces":[{"interface":"back","average":{"1min":4.7859999999999996,"5min":3.7909999999999999,"15min":2.9790000000000001},"min":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.315},"max":{"1min":72.712999999999994,"5min":72.712999999999994,"15min":72.712999999999994},"last":0.375},{"interface":"front","average":{"1min":1.4770000000000001,"5min":3.137,"15min":2.641},"min":{"1min":0.443,"5min":0.33500000000000002,"15min":0.28299999999999997},"max":{"1min":9.3780000000000001,"5min":17.992999999999999,"15min":38.948},"last":0.50800000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953768,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2571012,"kb_used_data":1683580,"kb_used_omap":0,"kb_used_meta":887424,"kb_avail":91174140,"statfs":{"total":95995035648,"available":93362319360,"internally_reserved":0,"allocated":1723985920,"data_stored":1723017214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":908722176},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.869,"5min":1.607,"15min":2.04},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20000000000000001},"max":{"1min":25.829000000000001,"5min":25.829000000000001,"15min":25.829000000000001},"last":1.512},{"interface":"front","average":{"1min":1.992,"5min":1.8660000000000001,"15min":2.0259999999999998},"min":{"1min":0.28799999999999998,"5min":0.22800000000000001,"15min":0.19800000000000001},"max":{"1min":22.286000000000001,"5min":22.286000000000001,"15min":22.286000000000001},"last":11}]},{"osd":1,"last update":"Mon Feb 12 17:09:01 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.528,"15min":1.8420000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.218},"max":{"1min":9.7349999999999994,"5min":9.7349999999999994,"15min":11.077999999999999},"last":3.7450000000000001},{"interface":"front","average":{"1min":1.488,"5min":1.877,"15min":2.0059999999999998},"min":{"1min":0.25900000000000001,"5min":0.252,"15min":0.23699999999999999},"max":{"1min":6.9569999999999999,"5min":14.082000000000001,"15min":14.082000000000001},"last":13.465999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":5.1479999999999997,"15min":3.3210000000000002},"min":{"1min":0.30099999999999999,"5min":0.255,"15min":0.252},"max":{"1min":24.361999999999998,"5min":191.05600000000001,"15min":191.05600000000001},"last":1.494},{"interface":"front","average":{"1min":1.7529999999999999,"5min":5.1289999999999996,"15min":3.2109999999999999},"min":{"1min":0.35099999999999998,"5min":0.29499999999999998,"15min":0.24399999999999999},"max":{"1min":24.390000000000001,"5min":191.083,"15min":191.083},"last":0.66500000000000004}]},{"osd":3,"last update":"Mon Feb 12 17:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":2.806,"15min":2.4009999999999998},"min":{"1min":0.30199999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":21.876999999999999,"5min":28.992000000000001,"15min":28.992000000000001},"last":4.4199999999999999},{"interface":"front","average":{"1min":1.645,"5min":2.7010000000000001,"15min":2.359},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":21.914000000000001,"5min":29.084,"15min":29.084},"last":0.745}]},{"osd":5,"last update":"Mon Feb 12 17:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.873,"15min":2.5249999999999999},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.2130000000000001,"5min":26.510000000000002,"15min":79.573999999999998},"last":4.2380000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.82199999999999995,"15min":2.75},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.219},"max":{"1min":3.407,"5min":23.986999999999998,"15min":80.497},"last":1.891}]},{"osd":6,"last update":"Mon Feb 12 17:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.53400000000000003,"15min":1.7789999999999999},"min":{"1min":0.248,"5min":0.246,"15min":0.246},"max":{"1min":0.51600000000000001,"5min":3.0670000000000002,"15min":27.678000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.622,"15min":1.9239999999999999},"min":{"1min":0.30099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.52400000000000002,"5min":4.133,"15min":15.593},"last":4.2629999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.623,"15min":2.0430000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72899999999999998,"5min":2.855,"15min":15.836},"last":1.468},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.66200000000000003,"15min":2.0939999999999999},"min":{"1min":0.34599999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.70499999999999996,"5min":4.4980000000000002,"15min":27.773},"last":0.82999999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757540,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2606720,"kb_used_data":1121400,"kb_used_omap":0,"kb_used_meta":1485312,"kb_avail":91138432,"statfs":{"total":95995035648,"available":93325754368,"internally_reserved":0,"allocated":1148313600,"data_stored":1147348332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1520959488},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:12:19 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.45,"15min":1.8100000000000001},"min":{"1min":0.28899999999999998,"5min":0.20699999999999999,"15min":0.20300000000000001},"max":{"1min":6.0199999999999996,"5min":10.858000000000001,"15min":46.365000000000002},"last":1.887},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.5329999999999999,"15min":2.161},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.19900000000000001},"max":{"1min":6.0910000000000002,"5min":15.368,"15min":46.134},"last":0.92600000000000005}]},{"osd":1,"last update":"Mon Feb 12 17:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.387,"15min":1.7450000000000001},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.23200000000000001},"max":{"1min":8.3620000000000001,"5min":11.19,"15min":16.655000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":2.3149999999999999,"5min":2.02,"15min":2.097},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.21199999999999999},"max":{"1min":10.973000000000001,"5min":12.154,"15min":16.712},"last":1.319}]},{"osd":2,"last update":"Mon Feb 12 17:12:19 2024","interfaces":[{"interface":"back","average":{"1min":3.4009999999999998,"5min":2.1760000000000002,"15min":2.0880000000000001},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":57.390999999999998,"5min":57.390999999999998,"15min":57.390999999999998},"last":1.9570000000000001},{"interface":"front","average":{"1min":4.0510000000000002,"5min":2.1890000000000001,"15min":1.944},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":57.134999999999998,"5min":57.134999999999998,"15min":57.134999999999998},"last":5.9580000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:12:19 2024","interfaces":[{"interface":"back","average":{"1min":3.653,"5min":4.218,"15min":2.677},"min":{"1min":0.29699999999999999,"5min":0.26100000000000001,"15min":0.22600000000000001},"max":{"1min":57.170999999999999,"5min":57.170999999999999,"15min":57.170999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":3.4009999999999998,"5min":3.6419999999999999,"15min":2.4369999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":57.484999999999999,"5min":57.484999999999999,"15min":57.484999999999999},"last":5.923}]},{"osd":4,"last update":"Mon Feb 12 17:12:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.52100000000000002,"15min":1.8340000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.84299999999999997,"5min":4.625,"15min":12.656000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.57499999999999996,"15min":1.9379999999999999},"min":{"1min":0.255,"5min":0.254,"15min":0.254},"max":{"1min":5.335,"5min":5.335,"15min":20.550000000000001},"last":0.32300000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.61199999999999999,"15min":1.724},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":12.281000000000001},"last":1.016},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.60399999999999998,"15min":1.839},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":2.5630000000000002,"5min":4.3659999999999997,"15min":22.035},"last":0.377}]},{"osd":7,"last update":"Mon Feb 12 17:12:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.76800000000000002,"15min":1.9590000000000001},"min":{"1min":0.29399999999999998,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":14.337999999999999,"5min":14.337999999999999,"15min":19.184999999999999},"last":5.9800000000000004},{"interface":"front","average":{"1min":1.1950000000000001,"5min":0.83799999999999997,"15min":2.0760000000000001},"min":{"1min":0.32300000000000001,"5min":0.27800000000000002,"15min":0.26900000000000002},"max":{"1min":14.637,"5min":14.637,"15min":20.274000000000001},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270204928,"data_stored":270204928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2042429440,"data_stored":2042429440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1984626688,"data_stored":1984626688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1753939968,"data_stored":1753939968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1165623296,"data_stored":1165623296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2342453248,"data_stored":2342453248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1752563712,"data_stored":1752563712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:12:52.960 INFO:tasks.thrashosds.thrasher:Setting 2.8 to [7, 4, 5] 2024-02-12T17:12:52.961 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.8', '7', '4', '5'] 2024-02-12T17:12:52.961 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pg-upmap 2.8 7 4 5 2024-02-12T17:12:53.012 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:12:53.012 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:12:53.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:53.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:53.134 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:51.498013+0000 osd.3 (osd.3) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:53.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:51.498053+0000 osd.3 (osd.3) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:53.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:52.619912+0000 mon.c (mon.2) 284 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:12:53.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:52.620733+0000 mon.c (mon.2) 285 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:12:53.135 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:52.635619+0000 mon.a (mon.0) 2006 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:53.136 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:51.498013+0000 osd.3 (osd.3) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:53.136 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:51.498053+0000 osd.3 (osd.3) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:53.136 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:52.619912+0000 mon.c (mon.2) 284 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:12:53.136 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:52.620733+0000 mon.c (mon.2) 285 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:12:53.211 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:53.277 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:53.368 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:12:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:53.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:12:53.418 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 42 16 77436 77420 115.193 76.375 0.0178356 0.00867683 2024-02-12T17:12:53.479 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:53.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:52.635619+0000 mon.a (mon.0) 2006 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:53.519 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:51.498013+0000 osd.3 (osd.3) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:12:53.519 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:51.498053+0000 osd.3 (osd.3) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:12:53.519 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:52.619912+0000 mon.c (mon.2) 284 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:12:53.519 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:52.620733+0000 mon.c (mon.2) 285 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:12:53.520 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:52.635619+0000 mon.a (mon.0) 2006 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:12:53.520 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:53.838 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:53.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:53.860 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:53.862 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T17:12:53.920 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:53.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:53.991 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:54.025 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:54.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:54.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:54.141 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:52.970335+0000 mgr.y (mgr.24481) 876 : cluster [DBG] pgmap v1231: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 804 B/s rd, 65 MiB/s wr, 1.04k op/s; 40892/218967 objects degraded (18.675%); 53/218967 objects misplaced (0.024%) 2024-02-12T17:12:54.141 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:53.789727+0000 osd.4 (osd.4) 193 : cluster [DBG] 5.c scrub starts 2024-02-12T17:12:54.141 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:53.793561+0000 osd.4 (osd.4) 194 : cluster [DBG] 5.c scrub ok 2024-02-12T17:12:54.200 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:54.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:52.970335+0000 mgr.y (mgr.24481) 876 : cluster [DBG] pgmap v1231: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 804 B/s rd, 65 MiB/s wr, 1.04k op/s; 40892/218967 objects degraded (18.675%); 53/218967 objects misplaced (0.024%) 2024-02-12T17:12:54.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:53.789727+0000 osd.4 (osd.4) 193 : cluster [DBG] 5.c scrub starts 2024-02-12T17:12:54.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:53.793561+0000 osd.4 (osd.4) 194 : cluster [DBG] 5.c scrub ok 2024-02-12T17:12:54.346 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:54.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:54.418 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 43 16 78650 78634 114.278 75.875 0.0152312 0.00874582 2024-02-12T17:12:54.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:54.455 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:52.970335+0000 mgr.y (mgr.24481) 876 : cluster [DBG] pgmap v1231: 125 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 57 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 804 B/s rd, 65 MiB/s wr, 1.04k op/s; 40892/218967 objects degraded (18.675%); 53/218967 objects misplaced (0.024%) 2024-02-12T17:12:54.455 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:53.789727+0000 osd.4 (osd.4) 193 : cluster [DBG] 5.c scrub starts 2024-02-12T17:12:54.455 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:53.793561+0000 osd.4 (osd.4) 194 : cluster [DBG] 5.c scrub ok 2024-02-12T17:12:54.507 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:54.541 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:54.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:54.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:54.714 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:54.745 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:54.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:54.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:12:54.860 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:12:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:12:54.907 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:54.936 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:12:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:54.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:12:55.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:55.076 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:55.112 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:55.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:55.159 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:53.740597+0000 osd.6 (osd.6) 161 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:12:55.159 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:53.746851+0000 osd.6 (osd.6) 162 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:12:55.159 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:54.048444+0000 osd.3 (osd.3) 214 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:12:55.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:55.315 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:55.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:55.419 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 44 16 79874 79858 113.419 76.5 0.0140266 0.00881246 2024-02-12T17:12:55.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:55.479 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:54.052000+0000 osd.3 (osd.3) 215 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:12:55.486 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:55.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:53.740597+0000 osd.6 (osd.6) 161 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:12:55.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:53.746851+0000 osd.6 (osd.6) 162 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:12:55.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:54.048444+0000 osd.3 (osd.3) 214 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:12:55.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:54.052000+0000 osd.3 (osd.3) 215 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:12:55.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:53.740597+0000 osd.6 (osd.6) 161 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:12:55.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:53.746851+0000 osd.6 (osd.6) 162 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:12:55.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:54.048444+0000 osd.3 (osd.3) 214 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:12:55.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:54.052000+0000 osd.3 (osd.3) 215 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:12:55.520 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:55.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:55.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:12:55.671 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:55.784 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:12:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:55.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:12:55.892 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:55.935 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:56.419 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 45 16 81121 81105 112.631 77.9375 0.018212 0.00887315 2024-02-12T17:12:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:54.650301+0000 osd.5 (osd.5) 299 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:12:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:54.656452+0000 osd.5 (osd.5) 300 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:12:56.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:54.857509+0000 mgr.y (mgr.24481) 877 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:56.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:54.971062+0000 mgr.y (mgr.24481) 878 : cluster [DBG] pgmap v1232: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 43266/230967 objects degraded (18.733%); 53/230967 objects misplaced (0.023%) 2024-02-12T17:12:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:54.650301+0000 osd.5 (osd.5) 299 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:12:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:54.656452+0000 osd.5 (osd.5) 300 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:12:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:54.857509+0000 mgr.y (mgr.24481) 877 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:54.971062+0000 mgr.y (mgr.24481) 878 : cluster [DBG] pgmap v1232: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 43266/230967 objects degraded (18.733%); 53/230967 objects misplaced (0.023%) 2024-02-12T17:12:56.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:54.650301+0000 osd.5 (osd.5) 299 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:12:56.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:54.656452+0000 osd.5 (osd.5) 300 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:12:56.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:54.857509+0000 mgr.y (mgr.24481) 877 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:12:56.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:54.971062+0000 mgr.y (mgr.24481) 878 : cluster [DBG] pgmap v1232: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 43266/230967 objects degraded (18.733%); 53/230967 objects misplaced (0.023%) 2024-02-12T17:12:57.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:55.859859+0000 osd.4 (osd.4) 195 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:12:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:55.862337+0000 osd.4 (osd.4) 196 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:12:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:56.600539+0000 mon.a (mon.0) 2007 : cluster [WRN] Health check update: Degraded data redundancy: 43266/230967 objects degraded (18.733%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:55.859859+0000 osd.4 (osd.4) 195 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:12:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:55.862337+0000 osd.4 (osd.4) 196 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:12:57.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:56.600539+0000 mon.a (mon.0) 2007 : cluster [WRN] Health check update: Degraded data redundancy: 43266/230967 objects degraded (18.733%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:57.419 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 46 15 82370 82355 111.881 78.125 0.0126585 0.00893354 2024-02-12T17:12:57.496 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:55.859859+0000 osd.4 (osd.4) 195 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:12:57.497 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:55.862337+0000 osd.4 (osd.4) 196 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:12:57.497 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:56.600539+0000 mon.a (mon.0) 2007 : cluster [WRN] Health check update: Degraded data redundancy: 43266/230967 objects degraded (18.733%), 25 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-02-12T17:12:58.197 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:58.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:58.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:12:58.356 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:58.401 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:12:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:58.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:12:58.420 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 47 16 83579 83563 111.103 75.5 0.020734 0.00899583 2024-02-12T17:12:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:56.686338+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:12:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:56.688971+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:12:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:56.869340+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.d scrub starts 2024-02-12T17:12:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:56.880277+0000 osd.4 (osd.4) 198 : cluster [DBG] 4.d scrub ok 2024-02-12T17:12:58.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:56.971686+0000 mgr.y (mgr.24481) 879 : cluster [DBG] pgmap v1233: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 44377/237837 objects degraded (18.659%); 53/237837 objects misplaced (0.022%) 2024-02-12T17:12:58.508 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:12:58.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:56.686338+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:12:58.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:56.688971+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:12:58.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:56.869340+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.d scrub starts 2024-02-12T17:12:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:56.880277+0000 osd.4 (osd.4) 198 : cluster [DBG] 4.d scrub ok 2024-02-12T17:12:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:56.971686+0000 mgr.y (mgr.24481) 879 : cluster [DBG] pgmap v1233: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 44377/237837 objects degraded (18.659%); 53/237837 objects misplaced (0.022%) 2024-02-12T17:12:58.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:56.686338+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:12:58.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:56.688971+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:12:58.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:56.869340+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.d scrub starts 2024-02-12T17:12:58.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:56.880277+0000 osd.4 (osd.4) 198 : cluster [DBG] 4.d scrub ok 2024-02-12T17:12:58.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:56.971686+0000 mgr.y (mgr.24481) 879 : cluster [DBG] pgmap v1233: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 44377/237837 objects degraded (18.659%); 53/237837 objects misplaced (0.022%) 2024-02-12T17:12:58.571 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:58.579 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:58.834 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:12:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:58.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:12:58.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:12:58.995 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:59.031 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:12:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:12:59.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:12:59.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:12:59.179 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:12:59.218 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:12:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:57.638722+0000 osd.5 (osd.5) 301 : cluster [DBG] 2.1c scrub starts 2024-02-12T17:12:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:57.645457+0000 osd.5 (osd.5) 302 : cluster [DBG] 2.1c scrub ok 2024-02-12T17:12:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:57.737418+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:12:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:57.741791+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:12:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:58.013397+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:12:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:58.018704+0000 osd.3 (osd.3) 217 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:12:59.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:12:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:58.692034+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? v1:172.21.15.33:0/522409850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:59.404 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:12:59.420 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 48 16 84657 84641 110.192 67.375 0.00840289 0.00907105 2024-02-12T17:12:59.431 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:12:59.459 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:12:59.489 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:12:59.521 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:12:59.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:57.638722+0000 osd.5 (osd.5) 301 : cluster [DBG] 2.1c scrub starts 2024-02-12T17:12:59.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:57.645457+0000 osd.5 (osd.5) 302 : cluster [DBG] 2.1c scrub ok 2024-02-12T17:12:59.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:57.737418+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:12:59.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:57.741791+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:12:59.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:58.013397+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:12:59.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:58.018704+0000 osd.3 (osd.3) 217 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:12:59.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:58.692034+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? v1:172.21.15.33:0/522409850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:59.531 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:57.638722+0000 osd.5 (osd.5) 301 : cluster [DBG] 2.1c scrub starts 2024-02-12T17:12:59.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:57.645457+0000 osd.5 (osd.5) 302 : cluster [DBG] 2.1c scrub ok 2024-02-12T17:12:59.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:57.737418+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:12:59.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:57.741791+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:12:59.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:58.013397+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:12:59.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:58.018704+0000 osd.3 (osd.3) 217 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:12:59.532 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:12:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:58.692034+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? v1:172.21.15.33:0/522409850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:12:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:12:59.564 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:12:59.590 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:12:59.590 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:12:59.590 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:12:59.590 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:12:59.591 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:12:59.591 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:12:59.591 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:12:59.591 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:12:59.591 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:13:00.420 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 49 16 85880 85864 109.503 76.4375 0.0104317 0.00912754 2024-02-12T17:13:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:58.865610+0000 osd.4 (osd.4) 199 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:13:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:58.871562+0000 osd.4 (osd.4) 200 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:13:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:58.972495+0000 mgr.y (mgr.24481) 880 : cluster [DBG] pgmap v1234: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 46684/249234 objects degraded (18.731%); 53/249234 objects misplaced (0.021%) 2024-02-12T17:13:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:58.999851+0000 osd.3 (osd.3) 218 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:13:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:59.006254+0000 osd.3 (osd.3) 219 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:13:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:59.168840+0000 mon.a (mon.0) 2009 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:13:00.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:59.186703+0000 mon.a (mon.0) 2010 : audit [INF] from='client.? v1:172.21.15.33:0/522409850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:13:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:59.186765+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 7 up, 8 in 2024-02-12T17:13:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:12:59.368476+0000 mon.b (mon.1) 109 : audit [DBG] from='client.? v1:172.21.15.33:0/1642518514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:59.606858+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.e deep-scrub starts 2024-02-12T17:13:00.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:59.647202+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.e deep-scrub ok 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:58.865610+0000 osd.4 (osd.4) 199 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:58.871562+0000 osd.4 (osd.4) 200 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:58.972495+0000 mgr.y (mgr.24481) 880 : cluster [DBG] pgmap v1234: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 46684/249234 objects degraded (18.731%); 53/249234 objects misplaced (0.021%) 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:58.999851+0000 osd.3 (osd.3) 218 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:59.006254+0000 osd.3 (osd.3) 219 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:59.168840+0000 mon.a (mon.0) 2009 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:59.186703+0000 mon.a (mon.0) 2010 : audit [INF] from='client.? v1:172.21.15.33:0/522409850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:59.186765+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 7 up, 8 in 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:12:59.368476+0000 mon.b (mon.1) 109 : audit [DBG] from='client.? v1:172.21.15.33:0/1642518514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:59.606858+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.e deep-scrub starts 2024-02-12T17:13:00.493 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:59.647202+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.e deep-scrub ok 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:58.865610+0000 osd.4 (osd.4) 199 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:58.871562+0000 osd.4 (osd.4) 200 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:58.972495+0000 mgr.y (mgr.24481) 880 : cluster [DBG] pgmap v1234: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 46684/249234 objects degraded (18.731%); 53/249234 objects misplaced (0.021%) 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:58.999851+0000 osd.3 (osd.3) 218 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:59.006254+0000 osd.3 (osd.3) 219 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:59.168840+0000 mon.a (mon.0) 2009 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:59.186703+0000 mon.a (mon.0) 2010 : audit [INF] from='client.? v1:172.21.15.33:0/522409850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:59.186765+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 7 up, 8 in 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:12:59.368476+0000 mon.b (mon.1) 109 : audit [DBG] from='client.? v1:172.21.15.33:0/1642518514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:00.494 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:59.606858+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.e deep-scrub starts 2024-02-12T17:13:00.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:59.647202+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.e deep-scrub ok 2024-02-12T17:13:01.199 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:13:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:00] "GET /metrics HTTP/1.1" 200 176885 "" "Prometheus/2.33.4" 2024-02-12T17:13:01.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:59.677461+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub starts 2024-02-12T17:13:01.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:12:59.685595+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.b scrub ok 2024-02-12T17:13:01.302 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:01.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:01.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:01.410 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:01.421 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 50 16 87092 87076 108.828 75.75 0.0157506 0.00918381 2024-02-12T17:13:01.460 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:01.464 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:59.677461+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub starts 2024-02-12T17:13:01.464 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:12:59.685595+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.b scrub ok 2024-02-12T17:13:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:01.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:01.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:59.677461+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub starts 2024-02-12T17:13:01.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:12:59.685595+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.b scrub ok 2024-02-12T17:13:01.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:01.696 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:01.752 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:01.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:01.856 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:01.899 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:02.209 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:00.684516+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.5 deep-scrub starts 2024-02-12T17:13:02.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:00.684516+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.5 deep-scrub starts 2024-02-12T17:13:02.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:00.766576+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.5 deep-scrub ok 2024-02-12T17:13:02.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:00.771511+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:13:02.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:00.783520+0000 osd.6 (osd.6) 168 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:13:02.210 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:00.973225+0000 mgr.y (mgr.24481) 881 : cluster [DBG] pgmap v1236: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 46684/249234 objects degraded (18.731%); 53/249234 objects misplaced (0.021%) 2024-02-12T17:13:02.210 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:01.034913+0000 osd.3 (osd.3) 220 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:13:02.210 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:01.059566+0000 osd.3 (osd.3) 221 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:13:02.210 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:01.538754+0000 osd.5 (osd.5) 305 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:13:02.210 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:01.543040+0000 osd.5 (osd.5) 306 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:13:02.210 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:01.601687+0000 mon.a (mon.0) 2012 : cluster [WRN] Health check update: Degraded data redundancy: 46684/249234 objects degraded (18.731%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:02.210 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:01.611171+0000 mon.a (mon.0) 2013 : cluster [DBG] osdmap e614: 8 total, 7 up, 8 in 2024-02-12T17:13:02.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:00.684516+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.5 deep-scrub starts 2024-02-12T17:13:02.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:00.766576+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.5 deep-scrub ok 2024-02-12T17:13:02.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:00.771511+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:13:02.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:00.783520+0000 osd.6 (osd.6) 168 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:13:02.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:00.973225+0000 mgr.y (mgr.24481) 881 : cluster [DBG] pgmap v1236: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 46684/249234 objects degraded (18.731%); 53/249234 objects misplaced (0.021%) 2024-02-12T17:13:02.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:01.034913+0000 osd.3 (osd.3) 220 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:13:02.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:01.059566+0000 osd.3 (osd.3) 221 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:13:02.211 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:01.538754+0000 osd.5 (osd.5) 305 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:13:02.212 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:01.543040+0000 osd.5 (osd.5) 306 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:13:02.212 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:01.601687+0000 mon.a (mon.0) 2012 : cluster [WRN] Health check update: Degraded data redundancy: 46684/249234 objects degraded (18.731%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:02.212 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:01.611171+0000 mon.a (mon.0) 2013 : cluster [DBG] osdmap e614: 8 total, 7 up, 8 in 2024-02-12T17:13:02.392 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:13:02.421 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 51 16 88225 88209 108.082 70.8125 0.0185909 0.00924826 2024-02-12T17:13:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:00.766576+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.5 deep-scrub ok 2024-02-12T17:13:02.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:00.771511+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:13:02.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:00.783520+0000 osd.6 (osd.6) 168 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:13:02.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:00.973225+0000 mgr.y (mgr.24481) 881 : cluster [DBG] pgmap v1236: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 46684/249234 objects degraded (18.731%); 53/249234 objects misplaced (0.021%) 2024-02-12T17:13:02.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:01.034913+0000 osd.3 (osd.3) 220 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:13:02.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:01.059566+0000 osd.3 (osd.3) 221 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:13:02.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:01.538754+0000 osd.5 (osd.5) 305 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:13:02.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:01.543040+0000 osd.5 (osd.5) 306 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:13:02.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:01.601687+0000 mon.a (mon.0) 2012 : cluster [WRN] Health check update: Degraded data redundancy: 46684/249234 objects degraded (18.731%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:02.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:01.611171+0000 mon.a (mon.0) 2013 : cluster [DBG] osdmap e614: 8 total, 7 up, 8 in 2024-02-12T17:13:02.967 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:02.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:13:02.996 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:13:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:13:02.997 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:13:03.076 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:03.128 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:03.193 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:03.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:03.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:03.350 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:03.392 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:03.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:03.421 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 52 16 89432 89416 107.454 75.4375 0.0116073 0.00930169 2024-02-12T17:13:03.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:03.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:03.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:03.726 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:02.528398+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:03.726 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:02.841146+0000 osd.6 (osd.6) 169 : cluster [DBG] 5.5 scrub starts 2024-02-12T17:13:03.726 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:02.852555+0000 osd.6 (osd.6) 170 : cluster [DBG] 5.5 scrub ok 2024-02-12T17:13:03.726 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:02.973816+0000 mgr.y (mgr.24481) 882 : cluster [DBG] pgmap v1238: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 47780/255942 objects degraded (18.668%); 53/255942 objects misplaced (0.021%) 2024-02-12T17:13:03.726 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:03.754 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:03.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:03.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:03.904 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:03.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:04.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:02.528398+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:02.841146+0000 osd.6 (osd.6) 169 : cluster [DBG] 5.5 scrub starts 2024-02-12T17:13:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:02.852555+0000 osd.6 (osd.6) 170 : cluster [DBG] 5.5 scrub ok 2024-02-12T17:13:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:02.973816+0000 mgr.y (mgr.24481) 882 : cluster [DBG] pgmap v1238: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 47780/255942 objects degraded (18.668%); 53/255942 objects misplaced (0.021%) 2024-02-12T17:13:04.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:02.528398+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:04.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:02.841146+0000 osd.6 (osd.6) 169 : cluster [DBG] 5.5 scrub starts 2024-02-12T17:13:04.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:02.852555+0000 osd.6 (osd.6) 170 : cluster [DBG] 5.5 scrub ok 2024-02-12T17:13:04.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:02.973816+0000 mgr.y (mgr.24481) 882 : cluster [DBG] pgmap v1238: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 47780/255942 objects degraded (18.668%); 53/255942 objects misplaced (0.021%) 2024-02-12T17:13:04.063 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:04.123 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:04.156 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:04.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:04.264 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:04.326 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:04.351 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:04.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:04.421 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 53 16 90544 90528 106.738 69.5 0.0154972 0.00936415 2024-02-12T17:13:04.459 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:04.518 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:04.564 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:04.568 INFO:teuthology.orchestra.run.smithi033.stderr:set 2.8 pg_upmap mapping to [7,4,5] 2024-02-12T17:13:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:04.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:04.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:04.765 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:04.878 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:03.639300+0000 osd.7 (osd.7) 296 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:13:04.878 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:03.705881+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:13:04.878 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:03.881451+0000 mon.c (mon.2) 286 : audit [INF] from='client.? v1:172.21.15.33:0/4277377212' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]: dispatch 2024-02-12T17:13:04.878 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:03.883284+0000 mon.a (mon.0) 2015 : audit [INF] from='client.36068 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]: dispatch 2024-02-12T17:13:04.878 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:03.938493+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:04.878 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:03.938536+0000 osd.6 (osd.6) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:04.882 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:03.639300+0000 osd.7 (osd.7) 296 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:13:04.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:03.705881+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:13:04.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:03.881451+0000 mon.c (mon.2) 286 : audit [INF] from='client.? v1:172.21.15.33:0/4277377212' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]: dispatch 2024-02-12T17:13:04.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:03.883284+0000 mon.a (mon.0) 2015 : audit [INF] from='client.36068 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]: dispatch 2024-02-12T17:13:04.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:03.938493+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:04.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:03.938536+0000 osd.6 (osd.6) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:04.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:03.639300+0000 osd.7 (osd.7) 296 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:13:04.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:03.705881+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:13:04.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:03.881451+0000 mon.c (mon.2) 286 : audit [INF] from='client.? v1:172.21.15.33:0/4277377212' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]: dispatch 2024-02-12T17:13:04.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:03.883284+0000 mon.a (mon.0) 2015 : audit [INF] from='client.36068 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]: dispatch 2024-02-12T17:13:04.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:03.938493+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:04.884 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:03.938536+0000 osd.6 (osd.6) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:04.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:04.985 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:04.998 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:13:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:05.247 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:13:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:13:05.370 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:05.403 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:05.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:05.421 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 54 16 91619 91603 106.004 67.1875 0.00504741 0.00942757 2024-02-12T17:13:05.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:05.574 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:05.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:03.931914+0000 osd.4 (osd.4) 201 : cluster [DBG] 11.d scrub starts 2024-02-12T17:13:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:04.547768+0000 mon.a (mon.0) 2016 : audit [INF] from='client.36068 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]': finished 2024-02-12T17:13:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:04.547840+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e615: 8 total, 7 up, 8 in 2024-02-12T17:13:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:04.874179+0000 mgr.y (mgr.24481) 883 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:05.656 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:04.974411+0000 mgr.y (mgr.24481) 884 : cluster [DBG] pgmap v1240: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 985 op/s; 49935/266970 objects degraded (18.704%); 53/266970 objects misplaced (0.020%) 2024-02-12T17:13:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:03.931914+0000 osd.4 (osd.4) 201 : cluster [DBG] 11.d scrub starts 2024-02-12T17:13:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:04.547768+0000 mon.a (mon.0) 2016 : audit [INF] from='client.36068 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]': finished 2024-02-12T17:13:05.657 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:04.547840+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e615: 8 total, 7 up, 8 in 2024-02-12T17:13:05.657 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:04.874179+0000 mgr.y (mgr.24481) 883 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:05.657 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:04.974411+0000 mgr.y (mgr.24481) 884 : cluster [DBG] pgmap v1240: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 985 op/s; 49935/266970 objects degraded (18.704%); 53/266970 objects misplaced (0.020%) 2024-02-12T17:13:05.702 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:05.755 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:05.898 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:05.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:05.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:03.931914+0000 osd.4 (osd.4) 201 : cluster [DBG] 11.d scrub starts 2024-02-12T17:13:05.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:04.547768+0000 mon.a (mon.0) 2016 : audit [INF] from='client.36068 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [7, 4, 5]}]': finished 2024-02-12T17:13:05.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:04.547840+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e615: 8 total, 7 up, 8 in 2024-02-12T17:13:05.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:04.874179+0000 mgr.y (mgr.24481) 883 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:05.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:04.974411+0000 mgr.y (mgr.24481) 884 : cluster [DBG] pgmap v1240: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 985 op/s; 49935/266970 objects degraded (18.704%); 53/266970 objects misplaced (0.020%) 2024-02-12T17:13:06.003 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:06.068 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:06.111 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:06.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:06.219 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:06.287 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:06.325 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:06.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:06.421 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 55 16 92668 92652 105.269 65.5625 0.0105145 0.00949426 2024-02-12T17:13:06.436 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:06.493 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:06.527 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:06.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:06.605 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:13:06.605 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T17:13:06.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T17:13:06.605 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:13:06.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.605 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:183803 9.4 9:2795d152:::benchmark_data_smithi033_131576_object46213:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e348)", 2024-02-12T17:13:06.605 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:50.520156+0000", 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 555.71880876099999, 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2623986300000001, 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 183803 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520156+0000", 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520156+0000", 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520154+0000", 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8210000000000008e-06 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520163+0000", 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4144999999999999e-05 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520177+0000", 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2660000000000001e-06 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520185+0000", 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0112000000000003e-05 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520245+0000", 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2449999999999998e-06 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520253+0000", 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055836549999999999 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.525837+0000", 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3610000000000007e-06 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.525846+0000", 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2330762159999999 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.758923+0000", 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114481 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.759037+0000", 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013508899999999999 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,5", 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.759172+0000", 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091537259999999992 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.768326+0000", 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011621948 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.779948+0000", 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022625689999999999 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782210+0000", 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017038200000000001 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782381+0000", 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3735e-05 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782434+0000", 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120133 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782555+0000", 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2623986300000001 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.612 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:183805 9.4 9:24010594:::benchmark_data_smithi033_131576_object46215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e348)", 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:50.520212+0000", 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 555.71875304100001, 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.262464222, 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 183805 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520212+0000", 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520212+0000", 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520211+0000", 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6809999999999997e-06 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520215+0000", 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9650000000000008e-06 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520223+0000", 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6289999999999998e-06 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520229+0000", 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4175e-05 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520313+0000", 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7139999999999996e-06 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.520320+0000", 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055488869999999997 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.525869+0000", 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0460000000000002e-06 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.525875+0000", 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2359034259999999 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.761778+0000", 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011102100000000001 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.761889+0000", 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000134628 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,5", 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.762024+0000", 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087307259999999994 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.770754+0000", 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092316619999999999 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.779986+0000", 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022892709999999998 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782275+0000", 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000305803 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782581+0000", 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5931999999999998e-05 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782607+0000", 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8731000000000003e-05 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782676+0000", 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.262464222 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:183813 9.4 9:2c1c8d35:::benchmark_data_smithi033_131576_object46223:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:50.677411+0000", 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 555.56155402000002, 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1053829159999999, 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.620 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 183813 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.677411+0000", 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.677411+0000", 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.677408+0000", 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9859999999999994e-06 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.677416+0000", 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2823000000000001e-05 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.677429+0000", 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3390000000000004e-06 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.677436+0000", 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1465999999999998e-05 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.677488+0000", 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5419999999999994e-06 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.677497+0000", 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0868096920000001 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.764307+0000", 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000124611 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.764432+0000", 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000142643 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,5", 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.764574+0000", 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061880149999999998 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.770762+0000", 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092563679999999992 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.780019+0000", 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022899159999999999 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782309+0000", 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00039726799999999999 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782706+0000", 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6237e-05 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782732+0000", 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1530000000000002e-05 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782794+0000", 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1053829159999999 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:183817 9.8 9:1f48e57e:::benchmark_data_smithi033_131576_object46227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:50.683200+0000", 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 555.55576438000003, 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.063261365, 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 183817 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683200+0000", 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683200+0000", 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683198+0000", 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6729999999999992e-06 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683206+0000", 2024-02-12T17:13:06.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3166999999999999e-05 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683219+0000", 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3790000000000003e-06 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683225+0000", 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5511000000000003e-05 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683291+0000", 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8400000000000001e-06 2024-02-12T17:13:06.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683300+0000", 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.021374564 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.704674+0000", 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1409000000000005e-05 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.704766+0000", 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1197000000000001e-05 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,7", 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.704857+0000", 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053563869999999998 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.710213+0000", 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032572159000000003 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.742785+0000", 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7687e-05 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.742823+0000", 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0034524849999999999 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.746276+0000", 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0677999999999998e-05 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.746326+0000", 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013537299999999999 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.746462+0000", 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.063261365 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:183818 9.4 9:263bc9db:::benchmark_data_smithi033_131576_object46228:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:50.686244+0000", 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 555.55272027499996, 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.096677415, 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.633 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 183818 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.686244+0000", 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.686244+0000", 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.686242+0000", 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1820000000000006e-06 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.686251+0000", 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5492999999999999e-05 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.686266+0000", 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1659999999999997e-06 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.686273+0000", 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9295999999999997e-05 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.686322+0000", 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1730000000000001e-06 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.686332+0000", 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.082030225 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.768362+0000", 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012505799999999999 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.768487+0000", 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000156114 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,5", 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.768643+0000", 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052279630000000004 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.773871+0000", 2024-02-12T17:13:06.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083707629999999998 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782242+0000", 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8815999999999997e-05 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782341+0000", 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047327799999999999 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782814+0000", 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4684000000000002e-05 2024-02-12T17:13:06.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782849+0000", 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3314999999999999e-05 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.782922+0000", 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.096677415 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121694 10.15 10:ac83d55d:::benchmark_data_smithi033_160181_object121693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e480)", 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.060072+0000", 2024-02-12T17:13:06.639 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 315.17889240300002, 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0400345070000001, 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121694 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060072+0000", 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060072+0000", 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060069+0000", 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7141999999999998e-05 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060106+0000", 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4969000000000001e-05 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060131+0000", 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0489999999999993e-06 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060140+0000", 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.078440560000000006 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.138581+0000", 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5939999999999996e-06 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.138589+0000", 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96144408400000003 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.100033+0000", 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4197999999999997e-05 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.100107+0000", 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0400345070000001 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121710 10.15 10:ae0bc385:::benchmark_data_smithi033_160181_object121709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e480)", 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.060148+0000", 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 315.17881653000001, 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.040033067, 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121710 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060148+0000", 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060148+0000", 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060147+0000", 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7807e-05 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060175+0000", 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.579e-06 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060182+0000", 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0799999999999999e-06 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060186+0000", 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.078434377999999999 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.138620+0000", 2024-02-12T17:13:06.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4099999999999996e-06 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.138627+0000", 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96150227700000002 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.100129+0000", 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2435999999999997e-05 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.100181+0000", 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.040033067 2024-02-12T17:13:06.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121719 10.5 10:a4d22be1:::benchmark_data_smithi033_160181_object121718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e480)", 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.060190+0000", 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 315.17877497400002, 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.038694056, 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121719 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.648 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060190+0000", 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060190+0000", 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060189+0000", 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7509e-05 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060217+0000", 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8140000000000003e-06 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060223+0000", 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1200000000000004e-06 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060228+0000", 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.078386676000000002 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.138614+0000", 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6909999999999997e-06 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.138622+0000", 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.94237256199999997 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.080994+0000", 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014881700000000001 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.081143+0000", 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019436900000000001 2024-02-12T17:13:06.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.081338+0000", 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010351961 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.091690+0000", 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068291769999999996 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.098519+0000", 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027540399999999998 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.098794+0000", 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9523e-05 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.098844+0000", 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0086999999999998e-05 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.098884+0000", 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.038694056 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.653 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121722 10.15 10:a8be5d21:::benchmark_data_smithi033_160181_object121721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e480)", 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.060232+0000", 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 315.17873307299999, 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.071353386, 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121722 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.654 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060232+0000", 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060232+0000", 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060231+0000", 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2234999999999999e-05 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060263+0000", 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6599999999999998e-06 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060270+0000", 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1309999999999999e-06 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.060274+0000", 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.078375714999999999 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.138650+0000", 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.13e-06 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.138655+0000", 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96154582499999997 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.100201+0000", 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111895 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.100312+0000", 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000175835 2024-02-12T17:13:06.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.100488+0000", 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044126720000000003 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.104901+0000", 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021090569999999999 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.125992+0000", 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055013730000000004 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.131493+0000", 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0971999999999999e-05 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.131544+0000", 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1109999999999998e-05 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.131585+0000", 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.071353386 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121728 10.8 10:1796702f:::benchmark_data_smithi033_160181_object121727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.193863+0000", 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 315.04510181299997, 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.91139806700000003, 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121728 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.660 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193863+0000", 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193863+0000", 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999952 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193858+0000", 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7508000000000001e-05 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193895+0000", 2024-02-12T17:13:06.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9050000000000001e-05 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193924+0000", 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5829999999999996e-06 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.193934+0000", 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028842400000000002 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.194222+0000", 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7110000000000001e-06 2024-02-12T17:13:06.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.194231+0000", 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.89220195300000005 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.086433+0000", 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012886700000000001 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.086562+0000", 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016759000000000001 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.086730+0000", 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047494349999999998 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.091479+0000", 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000221082 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.091700+0000", 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013462244999999999 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.105162+0000", 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3739000000000002e-05 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.105216+0000", 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4879999999999997e-05 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.105261+0000", 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.91139806700000003 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:121730 10.5 10:a6668925:::benchmark_data_smithi033_160181_object121729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:51.196953+0000", 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 315.042011694, 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90909190200000001, 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 121730 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.196953+0000", 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.196953+0000", 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999957 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.196949+0000", 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9313e-05 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.196998+0000", 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0593000000000003e-05 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.197029+0000", 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3869999999999999e-05 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.197042+0000", 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0189000000000001e-05 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.197123+0000", 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3679000000000001e-05 2024-02-12T17:13:06.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:51.197136+0000", 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88785056900000003 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.084987+0000", 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014834599999999999 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.085135+0000", 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019177500000000001 2024-02-12T17:13:06.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.085327+0000", 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0064050799999999996 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.091732+0000", 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068231029999999996 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.098555+0000", 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074014069999999996 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.105957+0000", 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9462000000000002e-05 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.106006+0000", 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8934999999999999e-05 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:52.106045+0000", 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90909190200000001 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.671 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123093 10.18 10:18fb724f:::benchmark_data_smithi033_160181_object123092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e483)", 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:54.070403+0000", 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 312.16856211499999, 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.263490236, 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123093 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.672 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070403+0000", 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070403+0000", 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070400+0000", 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.862e-05 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070429+0000", 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0112000000000001e-05 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070439+0000", 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5670000000000002e-06 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070444+0000", 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041556500000000001 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070860+0000", 2024-02-12T17:13:06.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8399999999999997e-06 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.070867+0000", 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2028627570000001 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.273729+0000", 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011330489999999999 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for degraded object", 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.274862+0000", 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.058922970999999998 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.333785+0000", 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000107392 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.333893+0000", 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.263490236 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123158 10.18 10:1f239b08:::benchmark_data_smithi033_160181_object123157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e483)", 2024-02-12T17:13:06.676 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:54.087854+0000", 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 312.15111041099999, 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.247987739, 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123158 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.087854+0000", 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.677 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.087854+0000", 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.087851+0000", 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6679999999999999e-05 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.087878+0000", 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1097000000000001e-05 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.087899+0000", 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8449999999999998e-06 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.087909+0000", 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7018999999999997e-05 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.087966+0000", 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4769e-05 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.087981+0000", 2024-02-12T17:13:06.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1869174250000001 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.274898+0000", 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000165138 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.275063+0000", 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022293700000000001 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3,5", 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.275286+0000", 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.053196886999999998 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.328483+0000", 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2924e-05 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.328496+0000", 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001439369 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.329935+0000", 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040019419999999997 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.333937+0000", 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1838000000000001e-05 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.333989+0000", 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018527870000000001 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.335842+0000", 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.247987739 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:231230 10.5 10:a39da56d:::benchmark_data_smithi033_160181_object108058:head [delete] snapc 0=[] ondisk+write+known_if_redirected e550)", 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:10:55.250570+0000", 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 130.988394844, 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.106648994, 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 231230 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.250570+0000", 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.250570+0000", 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.250568+0000", 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4729999999999999e-06 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.250574+0000", 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5230000000000001e-06 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.250580+0000", 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.18e-06 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.250584+0000", 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2929000000000005e-05 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.250666+0000", 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000121505 2024-02-12T17:13:06.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.250788+0000", 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000604982 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.251393+0000", 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004036095 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.255429+0000", 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019219969 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.274649+0000", 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.082474277999999998 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.357123+0000", 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1570000000000003e-05 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.357175+0000", 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3977000000000003e-05 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:55.357219+0000", 2024-02-12T17:13:06.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.106648994 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:231902 10.18 10:1b1665fb:::benchmark_data_smithi033_160181_object108730:head [delete] snapc 0=[] ondisk+write+known_if_redirected e551)", 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:10:56.301000+0000", 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 129.93796504400001, 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.089167437000000002, 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 231902 2024-02-12T17:13:06.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.301000+0000", 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.301000+0000", 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.300997+0000", 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3540000000000002e-06 2024-02-12T17:13:06.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.301005+0000", 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4443e-05 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.301019+0000", 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.421e-06 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.301026+0000", 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5099000000000003e-05 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.301071+0000", 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1571999999999998e-05 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.301152+0000", 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069530299999999998 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.301848+0000", 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010619666 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.312467+0000", 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011284419999999999 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.313596+0000", 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.075575479000000001 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.389171+0000", 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1574000000000004e-05 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.692 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.389233+0000", 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093434200000000001 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.390167+0000", 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.089167437000000002 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:232148 10.18 10:1a09a475:::benchmark_data_smithi033_160181_object108976:head [delete] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:10:56.781522+0000", 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 129.45744288, 2024-02-12T17:13:06.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.11285524199999999, 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 232148 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.781522+0000", 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.781522+0000", 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.781519+0000", 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2449999999999998e-06 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.781528+0000", 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2405999999999999e-05 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.781540+0000", 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0210000000000002e-06 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.781547+0000", 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061503000000000003 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.782162+0000", 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115633 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T17:13:06.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.782278+0000", 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.090660699999999997 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.872938+0000", 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4351999999999995e-05 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.873023+0000", 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060004000000000004 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.873623+0000", 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081485659999999994 2024-02-12T17:13:06.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.881771+0000", 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048827109999999996 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.886654+0000", 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067384530000000002 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.893393+0000", 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4736999999999999e-05 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.893477+0000", 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000899795 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:10:56.894377+0000", 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.11285524199999999 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:234522 10.18 10:185fd310:::benchmark_data_smithi033_160181_object111350:head [delete] snapc 0=[] ondisk+write+known_if_redirected e554)", 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:11:00.507580+0000", 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 125.731384917, 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.11274622500000001, 2024-02-12T17:13:06.699 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 234522 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.507580+0000", 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.507580+0000", 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.507578+0000", 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9550000000000003e-06 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.507585+0000", 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0866000000000001e-05 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.507595+0000", 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1340000000000003e-06 2024-02-12T17:13:06.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.507601+0000", 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0328999999999998e-05 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.507661+0000", 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000137999 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.507799+0000", 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029849962000000001 2024-02-12T17:13:06.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.537649+0000", 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1583999999999996e-05 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.537740+0000", 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061260299999999998 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.538353+0000", 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054291349999999999 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.543782+0000", 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.062442461999999997 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.606225+0000", 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013198013999999999 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.619423+0000", 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1316999999999996e-05 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.619484+0000", 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00084200300000000002 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:00.620326+0000", 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.11274622500000001 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:235036 10.18 10:18932b07:::benchmark_data_smithi033_160181_object111864:head [delete] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:11:01.826837+0000", 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 124.412127259, 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12107155, 2024-02-12T17:13:06.705 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 235036 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.826837+0000", 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.826837+0000", 2024-02-12T17:13:06.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.826835+0000", 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6810000000000004e-06 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.826844+0000", 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.27e-05 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.826856+0000", 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0360000000000001e-06 2024-02-12T17:13:06.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.826863+0000", 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042268760000000001 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.831090+0000", 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105568 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.831196+0000", 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.095655144999999997 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.926851+0000", 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2001000000000002e-05 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.926943+0000", 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00065815000000000003 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.927601+0000", 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010297021999999999 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.937898+0000", 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087704229999999994 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.946668+0000", 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00044218600000000001 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.947111+0000", 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8619000000000002e-05 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.947169+0000", 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000739739 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:01.947909+0000", 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.12107155 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:235938 10.18 10:1fce010e:::benchmark_data_smithi033_160181_object112766:head [delete] snapc 0=[] ondisk+write+known_if_redirected e557)", 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:11:03.507638+0000", 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 122.731326482, 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.18946811399999999, 2024-02-12T17:13:06.711 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 235938 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.507638+0000", 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.507638+0000", 2024-02-12T17:13:06.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999967 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.507635+0000", 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0586e-05 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.507646+0000", 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6229e-05 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.507662+0000", 2024-02-12T17:13:06.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3819999999999998e-06 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.507669+0000", 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059849589999999998 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.513654+0000", 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012656599999999999 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.513781+0000", 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.15970046500000001 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.673481+0000", 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1830999999999995e-05 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.673573+0000", 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00056540900000000001 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.674139+0000", 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016811606 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.690950+0000", 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011179 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.691062+0000", 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051953240000000003 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.696257+0000", 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7587999999999998e-05 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.696315+0000", 2024-02-12T17:13:06.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00079150399999999995 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:03.697106+0000", 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.18946811399999999 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:236420 10.18 10:18b62814:::benchmark_data_smithi033_160181_object113248:head [delete] snapc 0=[] ondisk+write+known_if_redirected e558)", 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:11:04.773766+0000", 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 121.465198972, 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.095047433000000001, 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 236420 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.773766+0000", 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.773766+0000", 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.773763+0000", 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.095e-05 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.773774+0000", 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2237000000000001e-05 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.773796+0000", 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1079999999999993e-06 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:06.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.773804+0000", 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010868 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.773913+0000", 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011765 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.774031+0000", 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.063561530000000005 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.837592+0000", 2024-02-12T17:13:06.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4912000000000003e-05 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.837677+0000", 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00070189999999999998 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.838379+0000", 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012249384 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.850628+0000", 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053079799999999997 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.851159+0000", 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016820087000000001 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.867979+0000", 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3414000000000001e-05 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.868033+0000", 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00078053400000000005 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:06.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:11:04.868813+0000", 2024-02-12T17:13:06.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.095047433000000001 2024-02-12T17:13:06.723 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.723 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.723 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.723 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:06.723 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:06.723 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:13:06.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:05.360974+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:05.361005+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:05.559828+0000 mon.a (mon.0) 2018 : cluster [DBG] osdmap e616: 8 total, 7 up, 8 in 2024-02-12T17:13:06.773 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:06.804 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:06.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:06.916 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:06.978 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:07.015 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:07.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:07.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:05.360974+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:07.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:05.361005+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:07.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:05.559828+0000 mon.a (mon.0) 2018 : cluster [DBG] osdmap e616: 8 total, 7 up, 8 in 2024-02-12T17:13:07.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:05.360974+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:07.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:05.361005+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:07.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:05.559828+0000 mon.a (mon.0) 2018 : cluster [DBG] osdmap e616: 8 total, 7 up, 8 in 2024-02-12T17:13:07.121 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:07.162 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:07.422 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 56 16 93785 93769 104.636 69.8125 0.00567083 0.00955238 2024-02-12T17:13:07.923 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:07.925 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:07.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:07.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:06.603423+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: Degraded data redundancy: 49935/266970 objects degraded (18.704%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:07.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:06.975211+0000 mgr.y (mgr.24481) 885 : cluster [DBG] pgmap v1242: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s; 51018/273150 objects degraded (18.678%); 53/273150 objects misplaced (0.019%) 2024-02-12T17:13:07.932 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:06.603423+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: Degraded data redundancy: 49935/266970 objects degraded (18.704%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:07.932 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:06.975211+0000 mgr.y (mgr.24481) 885 : cluster [DBG] pgmap v1242: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s; 51018/273150 objects degraded (18.678%); 53/273150 objects misplaced (0.019%) 2024-02-12T17:13:07.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:06.603423+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: Degraded data redundancy: 49935/266970 objects degraded (18.704%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:07.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:06.975211+0000 mgr.y (mgr.24481) 885 : cluster [DBG] pgmap v1242: 125 pgs: 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 58 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s; 51018/273150 objects degraded (18.678%); 53/273150 objects misplaced (0.019%) 2024-02-12T17:13:08.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:08.107 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:08.151 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:08.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:08.259 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:08.311 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:08.422 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 57 16 94985 94969 104.116 75 0.00759825 0.00960018 2024-02-12T17:13:08.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:06.847785+0000 osd.4 (osd.4) 204 : cluster [DBG] 11.d scrub ok 2024-02-12T17:13:08.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:07.705005+0000 osd.7 (osd.7) 298 : cluster [DBG] 5.d scrub starts 2024-02-12T17:13:08.996 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:07.711593+0000 osd.7 (osd.7) 299 : cluster [DBG] 5.d scrub ok 2024-02-12T17:13:09.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:06.847785+0000 osd.4 (osd.4) 204 : cluster [DBG] 11.d scrub ok 2024-02-12T17:13:09.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:07.705005+0000 osd.7 (osd.7) 298 : cluster [DBG] 5.d scrub starts 2024-02-12T17:13:09.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:07.711593+0000 osd.7 (osd.7) 299 : cluster [DBG] 5.d scrub ok 2024-02-12T17:13:09.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:06.847785+0000 osd.4 (osd.4) 204 : cluster [DBG] 11.d scrub ok 2024-02-12T17:13:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:07.705005+0000 osd.7 (osd.7) 298 : cluster [DBG] 5.d scrub starts 2024-02-12T17:13:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:07.711593+0000 osd.7 (osd.7) 299 : cluster [DBG] 5.d scrub ok 2024-02-12T17:13:09.373 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:09.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:09.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 58 16 96164 96148 103.589 73.6875 0.0160376 0.0096492 2024-02-12T17:13:09.484 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:09.532 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:09.559 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:09.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:09.631 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:08.720705+0000 osd.7 (osd.7) 300 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:13:09.631 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:08.873954+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:13:09.631 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:08.976146+0000 mgr.y (mgr.24481) 886 : cluster [DBG] pgmap v1243: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 53142/283635 objects degraded (18.736%); 53/283635 objects misplaced (0.019%) 2024-02-12T17:13:09.663 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:09.702 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:09.728 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:09.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:09.759 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T17:13:09.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:09.890 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:08.720705+0000 osd.7 (osd.7) 300 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:13:09.890 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:08.873954+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:13:09.890 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:08.976146+0000 mgr.y (mgr.24481) 886 : cluster [DBG] pgmap v1243: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 53142/283635 objects degraded (18.736%); 53/283635 objects misplaced (0.019%) 2024-02-12T17:13:09.893 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:09.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:08.720705+0000 osd.7 (osd.7) 300 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:13:09.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:08.873954+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:13:09.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:08.976146+0000 mgr.y (mgr.24481) 886 : cluster [DBG] pgmap v1243: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 53142/283635 objects degraded (18.736%); 53/283635 objects misplaced (0.019%) 2024-02-12T17:13:10.039 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:10.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:10.147 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:10.205 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:10.250 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:10.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:10.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:10.358 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:10.358 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-12T17:13:10.359 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:13:10.359 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 5, 7, 3, 1, 6, 4] 2024-02-12T17:13:10.359 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd out 4 2024-02-12T17:13:10.398 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:10.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 59 16 97317 97301 103.055 72.0625 0.0222812 0.00969916 2024-02-12T17:13:10.431 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:10.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:10.538 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:10.606 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:10.641 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:10.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:10.649 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:09.036564+0000 osd.4 (osd.4) 205 : cluster [DBG] 5.a scrub starts 2024-02-12T17:13:10.649 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:09.044311+0000 osd.4 (osd.4) 206 : cluster [DBG] 5.a scrub ok 2024-02-12T17:13:10.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:10.794 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:10.836 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:10.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:10.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:10.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:09.036564+0000 osd.4 (osd.4) 205 : cluster [DBG] 5.a scrub starts 2024-02-12T17:13:10.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:09.044311+0000 osd.4 (osd.4) 206 : cluster [DBG] 5.a scrub ok 2024-02-12T17:13:10.996 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:11.012 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:13:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:10] "GET /metrics HTTP/1.1" 200 176880 "" "Prometheus/2.33.4" 2024-02-12T17:13:11.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:09.036564+0000 osd.4 (osd.4) 205 : cluster [DBG] 5.a scrub starts 2024-02-12T17:13:11.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:09.044311+0000 osd.4 (osd.4) 206 : cluster [DBG] 5.a scrub ok 2024-02-12T17:13:11.039 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:11.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:11.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:11.229 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:11.279 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:11.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:11.384 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:11.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:13:11.422999+0000 min lat: 0.00107612 max lat: 0.162663 avg lat: 0.00975961 2024-02-12T17:13:11.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:13:11.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 60 16 98348 98332 102.411 64.4375 0.0104222 0.00975961 2024-02-12T17:13:11.459 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:11.521 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:11.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:11.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:11.661 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:10.245329+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:11.661 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:10.245396+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:11.662 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:10.407316+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.6 deep-scrub starts 2024-02-12T17:13:11.662 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:10.412023+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.6 deep-scrub ok 2024-02-12T17:13:11.662 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:10.639064+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e617: 8 total, 7 up, 8 in 2024-02-12T17:13:11.662 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:10.785013+0000 osd.7 (osd.7) 304 : cluster [DBG] 11.5 scrub starts 2024-02-12T17:13:11.662 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:10.976764+0000 mgr.y (mgr.24481) 887 : cluster [DBG] pgmap v1245: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 796 B/s rd, 54 MiB/s wr, 865 op/s; 53142/283635 objects degraded (18.736%); 53/283635 objects misplaced (0.019%) 2024-02-12T17:13:11.662 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:11.606912+0000 mon.a (mon.0) 2021 : cluster [WRN] Health check update: Degraded data redundancy: 53142/283635 objects degraded (18.736%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:11.662 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:11.618245+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e618: 8 total, 7 up, 8 in 2024-02-12T17:13:11.685 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:11.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:10.245329+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:11.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:10.245396+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:10.407316+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.6 deep-scrub starts 2024-02-12T17:13:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:10.412023+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.6 deep-scrub ok 2024-02-12T17:13:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:10.639064+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e617: 8 total, 7 up, 8 in 2024-02-12T17:13:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:10.785013+0000 osd.7 (osd.7) 304 : cluster [DBG] 11.5 scrub starts 2024-02-12T17:13:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:10.976764+0000 mgr.y (mgr.24481) 887 : cluster [DBG] pgmap v1245: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 796 B/s rd, 54 MiB/s wr, 865 op/s; 53142/283635 objects degraded (18.736%); 53/283635 objects misplaced (0.019%) 2024-02-12T17:13:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:11.606912+0000 mon.a (mon.0) 2021 : cluster [WRN] Health check update: Degraded data redundancy: 53142/283635 objects degraded (18.736%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:11.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:11.618245+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e618: 8 total, 7 up, 8 in 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:10.245329+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:10.245396+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:10.407316+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.6 deep-scrub starts 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:10.412023+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.6 deep-scrub ok 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:10.639064+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e617: 8 total, 7 up, 8 in 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:10.785013+0000 osd.7 (osd.7) 304 : cluster [DBG] 11.5 scrub starts 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:10.976764+0000 mgr.y (mgr.24481) 887 : cluster [DBG] pgmap v1245: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 796 B/s rd, 54 MiB/s wr, 865 op/s; 53142/283635 objects degraded (18.736%); 53/283635 objects misplaced (0.019%) 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:11.606912+0000 mon.a (mon.0) 2021 : cluster [WRN] Health check update: Degraded data redundancy: 53142/283635 objects degraded (18.736%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:11.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:11.618245+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e618: 8 total, 7 up, 8 in 2024-02-12T17:13:12.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 61 16 99388 99372 101.798 65 0.00928522 0.00981897 2024-02-12T17:13:12.651 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:13:12.651 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:13:12.651 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:13:12.652 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:13:12.652 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:13:12.652 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:13:12.652 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:13:12.652 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:13:12.652 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:13:12.653 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:13:12.653 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:13:12.653 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:13:12.653 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:13:12.653 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:13:12.653 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:13:12.657 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:11.665814+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? v1:172.21.15.33:0/2692001174' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:13:12.658 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:12.008280+0000 osd.3 (osd.3) 222 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:13:12.658 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:12.016219+0000 osd.3 (osd.3) 223 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:13:12.658 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:12.119879+0000 mon.a (mon.0) 2024 : audit [DBG] from='client.? v1:172.21.15.33:0/905121789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:12.679 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:13:12.856 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:12.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:12.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:12.963 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:12.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:11.665814+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? v1:172.21.15.33:0/2692001174' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:13:12.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:12.008280+0000 osd.3 (osd.3) 222 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:13:12.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:12.016219+0000 osd.3 (osd.3) 223 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:13:12.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:12.119879+0000 mon.a (mon.0) 2024 : audit [DBG] from='client.? v1:172.21.15.33:0/905121789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:12.993 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:13:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:13:13.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:11.665814+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? v1:172.21.15.33:0/2692001174' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:13:13.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:12.008280+0000 osd.3 (osd.3) 222 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:13:13.005 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:12.016219+0000 osd.3 (osd.3) 223 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:13:13.006 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:12.119879+0000 mon.a (mon.0) 2024 : audit [DBG] from='client.? v1:172.21.15.33:0/905121789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:13.009 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:13.111 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:13.112 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:13.123 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:13.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:13.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:13.277 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:13.305 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:13.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:13.412 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:13.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 62 16 100463 100447 101.239 67.1875 0.0093363 0.00987328 2024-02-12T17:13:13.473 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:13.496 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:13.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:13.602 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:13.663 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:13.673 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:12.651747+0000 mon.a (mon.0) 2025 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:13.673 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:12.667160+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? v1:172.21.15.33:0/2692001174' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:13:13.674 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:12.667208+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e619: 8 total, 7 up, 8 in 2024-02-12T17:13:13.674 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:12.977336+0000 mgr.y (mgr.24481) 888 : cluster [DBG] pgmap v1248: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 934 op/s; 54207/289956 objects degraded (18.695%); 53/289956 objects misplaced (0.018%) 2024-02-12T17:13:13.925 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:13.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:13.930 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:12.651747+0000 mon.a (mon.0) 2025 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:13.930 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:12.667160+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? v1:172.21.15.33:0/2692001174' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:13:13.930 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:12.667208+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e619: 8 total, 7 up, 8 in 2024-02-12T17:13:13.930 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:12.977336+0000 mgr.y (mgr.24481) 888 : cluster [DBG] pgmap v1248: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 934 op/s; 54207/289956 objects degraded (18.695%); 53/289956 objects misplaced (0.018%) 2024-02-12T17:13:13.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:12.651747+0000 mon.a (mon.0) 2025 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:13.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:12.667160+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? v1:172.21.15.33:0/2692001174' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:13:13.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:12.667208+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e619: 8 total, 7 up, 8 in 2024-02-12T17:13:13.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:12.977336+0000 mgr.y (mgr.24481) 888 : cluster [DBG] pgmap v1248: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 934 op/s; 54207/289956 objects degraded (18.695%); 53/289956 objects misplaced (0.018%) 2024-02-12T17:13:14.031 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:14.078 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:14.211 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:14.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:14.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:14.397 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:14.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 63 16 101642 101626 100.802 73.6875 0.00717589 0.00991584 2024-02-12T17:13:14.436 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:14.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:14.541 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:14.581 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:14.842 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:13:14.846 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:13:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:14.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:14.948 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:14.987 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:15.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:13:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:13:15.347 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:15.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:15.423 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 64 16 102852 102836 100.409 75.625 0.00995035 0.00995489 2024-02-12T17:13:15.452 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:15.497 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:15.668 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:13:16.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 65 16 103956 103940 99.9254 69 0.0128509 0.0100028 2024-02-12T17:13:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:14.884681+0000 mgr.y (mgr.24481) 889 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:16.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:14.978186+0000 mgr.y (mgr.24481) 890 : cluster [DBG] pgmap v1249: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 922 op/s; 56297/300222 objects degraded (18.752%); 53/300222 objects misplaced (0.018%) 2024-02-12T17:13:16.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:15.064794+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:16.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:15.064845+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:14.884681+0000 mgr.y (mgr.24481) 889 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:16.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:14.978186+0000 mgr.y (mgr.24481) 890 : cluster [DBG] pgmap v1249: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 922 op/s; 56297/300222 objects degraded (18.752%); 53/300222 objects misplaced (0.018%) 2024-02-12T17:13:16.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:15.064794+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:16.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:15.064845+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:14.884681+0000 mgr.y (mgr.24481) 889 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:14.978186+0000 mgr.y (mgr.24481) 890 : cluster [DBG] pgmap v1249: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 922 op/s; 56297/300222 objects degraded (18.752%); 53/300222 objects misplaced (0.018%) 2024-02-12T17:13:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:15.064794+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:15.064845+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:16.848 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:13:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:16.040963+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e620: 8 total, 7 up, 8 in 2024-02-12T17:13:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:16.608584+0000 mon.a (mon.0) 2029 : cluster [WRN] Health check update: Degraded data redundancy: 56297/300222 objects degraded (18.752%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:16.040963+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e620: 8 total, 7 up, 8 in 2024-02-12T17:13:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:16.608584+0000 mon.a (mon.0) 2029 : cluster [WRN] Health check update: Degraded data redundancy: 56297/300222 objects degraded (18.752%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:17.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 66 16 105034 105018 99.4321 67.375 0.020814 0.0100536 2024-02-12T17:13:17.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:16.040963+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e620: 8 total, 7 up, 8 in 2024-02-12T17:13:17.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:16.608584+0000 mon.a (mon.0) 2029 : cluster [WRN] Health check update: Degraded data redundancy: 56297/300222 objects degraded (18.752%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:17.660 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:17.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:17.669 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:17.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:17.882 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:17.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:17.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:18.041 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:18.071 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:18.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:18.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:18.240 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:18.288 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:18.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:18.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:18.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 67 16 106169 106153 99.0068 70.9375 0.0201526 0.010096 2024-02-12T17:13:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:18.479 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:18.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:18.586 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:18.633 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:18.738 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:16.978832+0000 mgr.y (mgr.24481) 891 : cluster [DBG] pgmap v1251: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s; 57360/306597 objects degraded (18.709%); 53/306597 objects misplaced (0.017%) 2024-02-12T17:13:18.739 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:17.522297+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:18.739 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:16.978832+0000 mgr.y (mgr.24481) 891 : cluster [DBG] pgmap v1251: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s; 57360/306597 objects degraded (18.709%); 53/306597 objects misplaced (0.017%) 2024-02-12T17:13:18.739 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:17.522297+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:18.741 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:18.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:18.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:18.908 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:18.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:16.978832+0000 mgr.y (mgr.24481) 891 : cluster [DBG] pgmap v1251: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s; 57360/306597 objects degraded (18.709%); 53/306597 objects misplaced (0.017%) 2024-02-12T17:13:18.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:17.522297+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:18.992 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:18.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:19.097 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:19.153 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:19.417 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:13:19.417 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:13:19.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 68 16 107327 107311 98.6151 72.375 0.00699413 0.0101371 2024-02-12T17:13:19.713 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:19.714 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:18.979575+0000 mgr.y (mgr.24481) 892 : cluster [DBG] pgmap v1252: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 59487/317241 objects degraded (18.751%); 53/317241 objects misplaced (0.017%) 2024-02-12T17:13:19.714 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:13:19.566+0000 7fd96a9de700 -1 mon.a@0(leader).osd e620 definitely_dead 0 2024-02-12T17:13:19.714 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:18.979575+0000 mgr.y (mgr.24481) 892 : cluster [DBG] pgmap v1252: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 59487/317241 objects degraded (18.751%); 53/317241 objects misplaced (0.017%) 2024-02-12T17:13:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:19.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:19.816 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:19.868 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:18.979575+0000 mgr.y (mgr.24481) 892 : cluster [DBG] pgmap v1252: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 59487/317241 objects degraded (18.751%); 53/317241 objects misplaced (0.017%) 2024-02-12T17:13:19.868 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:19.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:20.005 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:20.052 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:20.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 69 16 108503 108487 98.251 73.5 0.00784108 0.0101738 2024-02-12T17:13:20.570 INFO:teuthology.orchestra.run.smithi033.stderr:marked out osd.4. 2024-02-12T17:13:20.878 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:19.566505+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.33:0/3139872515' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:13:20.878 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:19.566984+0000 mon.a (mon.0) 2031 : audit [INF] from='client.36080 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:13:20.878 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:19.567046+0000 mon.a (mon.0) 2032 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-12T17:13:20.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:19.566505+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.33:0/3139872515' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:13:20.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:19.566984+0000 mon.a (mon.0) 2031 : audit [INF] from='client.36080 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:13:20.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:19.567046+0000 mon.a (mon.0) 2032 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-12T17:13:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:19.566505+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.33:0/3139872515' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:13:21.022 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:19.566984+0000 mon.a (mon.0) 2031 : audit [INF] from='client.36080 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-12T17:13:21.023 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:19.567046+0000 mon.a (mon.0) 2032 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-12T17:13:21.275 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:21.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:13:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:20] "GET /metrics HTTP/1.1" 200 176894 "" "Prometheus/2.33.4" 2024-02-12T17:13:21.310 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:21.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:21.414 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:21.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 70 16 108694 108678 97.0181 11.9375 0.00323207 0.0101817 2024-02-12T17:13:21.445 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:21.467 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:21.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:21.570 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:20.547303+0000 mon.a (mon.0) 2033 : audit [INF] from='client.36080 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-12T17:13:21.570 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:20.547396+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-12T17:13:21.570 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:20.980334+0000 mgr.y (mgr.24481) 893 : cluster [DBG] pgmap v1254: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 59487/317241 objects degraded (18.751%); 53/317241 objects misplaced (0.017%) 2024-02-12T17:13:21.570 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:21.571 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:20.547303+0000 mon.a (mon.0) 2033 : audit [INF] from='client.36080 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-12T17:13:21.571 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:20.547396+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-12T17:13:21.571 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:20.980334+0000 mgr.y (mgr.24481) 893 : cluster [DBG] pgmap v1254: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 59487/317241 objects degraded (18.751%); 53/317241 objects misplaced (0.017%) 2024-02-12T17:13:21.611 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:21.671 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:21.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:20.547303+0000 mon.a (mon.0) 2033 : audit [INF] from='client.36080 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-12T17:13:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:20.547396+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-12T17:13:21.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:20.980334+0000 mgr.y (mgr.24481) 893 : cluster [DBG] pgmap v1254: 125 pgs: 1 active+remapped+backfill_toofull, 25 active+clean+remapped, 15 active+undersized, 25 active+undersized+degraded, 59 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 59487/317241 objects degraded (18.751%); 53/317241 objects misplaced (0.017%) 2024-02-12T17:13:21.782 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:21.810 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:22.258 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:22.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:22.362 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:22.390 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:22.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 71 16 108694 108678 95.6517 0 - 0.0101817 2024-02-12T17:13:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:22.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:22.568 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:22.627 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:22.673 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:22.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:22.777 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:22.816 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:21.561663+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2024-02-12T17:13:22.816 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:21.565511+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.5 starting backfill to osd.7 from (0'0,0'0] MAX to 410'15 2024-02-12T17:13:22.816 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:21.610129+0000 mon.a (mon.0) 2036 : cluster [WRN] Health check update: Degraded data redundancy: 59487/317241 objects degraded (18.751%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:22.817 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:21.561663+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2024-02-12T17:13:22.817 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:21.565511+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.5 starting backfill to osd.7 from (0'0,0'0] MAX to 410'15 2024-02-12T17:13:22.817 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:21.610129+0000 mon.a (mon.0) 2036 : cluster [WRN] Health check update: Degraded data redundancy: 59487/317241 objects degraded (18.751%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:22.818 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T17:13:22.825 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:13:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:13:22.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:21.561663+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2024-02-12T17:13:22.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:21.565511+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.5 starting backfill to osd.7 from (0'0,0'0] MAX to 410'15 2024-02-12T17:13:22.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:21.610129+0000 mon.a (mon.0) 2036 : cluster [WRN] Health check update: Degraded data redundancy: 59487/317241 objects degraded (18.751%), 25 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:23.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 72 16 109369 109353 94.9092 21.0938 0.0144352 0.0105325 2024-02-12T17:13:23.694 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:23.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:23.806 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.563751+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.564665+0000 osd.3 (osd.3) 224 : cluster [DBG] 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 620'6840 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.565387+0000 osd.7 (osd.7) 305 : cluster [DBG] 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 620'6829 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.566157+0000 osd.2 (osd.2) 223 : cluster [DBG] 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 620'6743 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.567629+0000 osd.2 (osd.2) 224 : cluster [DBG] 11.6 starting backfill to osd.5 from (0'0,0'0] MAX to 620'6890 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.570710+0000 osd.5 (osd.5) 309 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 596'24 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.570790+0000 osd.5 (osd.5) 310 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 596'28 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.571881+0000 osd.7 (osd.7) 306 : cluster [DBG] 2.f starting backfill to osd.6 from (0'0,0'0] MAX to 91'4 2024-02-12T17:13:23.873 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.575453+0000 osd.5 (osd.5) 311 : cluster [DBG] 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 620'6741 2024-02-12T17:13:23.874 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:22.980898+0000 mgr.y (mgr.24481) 894 : cluster [DBG] pgmap v1257: 125 pgs: 10 peering, 6 remapped+peering, 1 active+remapped+backfill_toofull, 19 active+clean+remapped, 15 active+undersized, 24 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s; 53527/322530 objects degraded (16.596%); 33/322530 objects misplaced (0.010%) 2024-02-12T17:13:23.889 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:23.916 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:23.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.563751+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.564665+0000 osd.3 (osd.3) 224 : cluster [DBG] 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 620'6840 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.565387+0000 osd.7 (osd.7) 305 : cluster [DBG] 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 620'6829 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.566157+0000 osd.2 (osd.2) 223 : cluster [DBG] 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 620'6743 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.567629+0000 osd.2 (osd.2) 224 : cluster [DBG] 11.6 starting backfill to osd.5 from (0'0,0'0] MAX to 620'6890 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.570710+0000 osd.5 (osd.5) 309 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 596'24 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.570790+0000 osd.5 (osd.5) 310 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 596'28 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.571881+0000 osd.7 (osd.7) 306 : cluster [DBG] 2.f starting backfill to osd.6 from (0'0,0'0] MAX to 91'4 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.575453+0000 osd.5 (osd.5) 311 : cluster [DBG] 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 620'6741 2024-02-12T17:13:24.016 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:22.980898+0000 mgr.y (mgr.24481) 894 : cluster [DBG] pgmap v1257: 125 pgs: 10 peering, 6 remapped+peering, 1 active+remapped+backfill_toofull, 19 active+clean+remapped, 15 active+undersized, 24 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s; 53527/322530 objects degraded (16.596%); 33/322530 objects misplaced (0.010%) 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.563751+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.564665+0000 osd.3 (osd.3) 224 : cluster [DBG] 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 620'6840 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.565387+0000 osd.7 (osd.7) 305 : cluster [DBG] 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 620'6829 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.566157+0000 osd.2 (osd.2) 223 : cluster [DBG] 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 620'6743 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.567629+0000 osd.2 (osd.2) 224 : cluster [DBG] 11.6 starting backfill to osd.5 from (0'0,0'0] MAX to 620'6890 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.570710+0000 osd.5 (osd.5) 309 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 596'24 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.570790+0000 osd.5 (osd.5) 310 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 596'28 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.571881+0000 osd.7 (osd.7) 306 : cluster [DBG] 2.f starting backfill to osd.6 from (0'0,0'0] MAX to 91'4 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.575453+0000 osd.5 (osd.5) 311 : cluster [DBG] 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 620'6741 2024-02-12T17:13:24.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:22.980898+0000 mgr.y (mgr.24481) 894 : cluster [DBG] pgmap v1257: 125 pgs: 10 peering, 6 remapped+peering, 1 active+remapped+backfill_toofull, 19 active+clean+remapped, 15 active+undersized, 24 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s; 53527/322530 objects degraded (16.596%); 33/322530 objects misplaced (0.010%) 2024-02-12T17:13:24.026 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:24.073 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:24.115 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:24.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:24.219 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:24.299 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:24.338 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:24.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:24.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 73 16 110272 110256 94.3821 56.4375 0.0120663 0.0105911 2024-02-12T17:13:24.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:24.524 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:24.566 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:24.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:24.593 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:23.561026+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-02-12T17:13:24.593 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:23.578651+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2024-02-12T17:13:24.677 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:24.733 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:24.761 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:24.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:24.871 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:24.900 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:13:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:13:24.903 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:23.561026+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-02-12T17:13:24.903 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:23.578651+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2024-02-12T17:13:24.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:23.561026+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-02-12T17:13:24.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:23.578651+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2024-02-12T17:13:24.937 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:24.976 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:24.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:25.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:25.138 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:25.183 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:25.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:25.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:25.310 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:13:25.310 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:13:25.322 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:25.329 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:25.348 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:13:25.356 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:25.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:25.372 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:13:25.400 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:13:25.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 74 16 111103 111087 93.8086 51.9375 0.0194476 0.0106555 2024-02-12T17:13:25.427 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:13:25.463 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:25.467 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:13:25.467 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:13:25.468 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:13:25.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:13:25.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:13:25.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:13:25.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:13:25.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:13:25.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:13:25.513 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:25.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:25.608 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:24.584421+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2024-02-12T17:13:25.608 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:24.899311+0000 mgr.y (mgr.24481) 895 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:25.608 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:24.919807+0000 mon.c (mon.2) 288 : audit [INF] from='client.? v1:172.21.15.33:0/2261435494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:13:25.608 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:24.920435+0000 mon.a (mon.0) 2041 : audit [INF] from='client.36086 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:13:25.608 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:24.984843+0000 mgr.y (mgr.24481) 896 : cluster [DBG] pgmap v1260: 125 pgs: 4 active+undersized+degraded+remapped+backfilling, 11 peering, 6 remapped+peering, 2 active+remapped+backfill_toofull, 24 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 43 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 930 op/s; 54453/328395 objects degraded (16.582%); 34251/328395 objects misplaced (10.430%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T17:13:25.609 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:25.251584+0000 mon.c (mon.2) 289 : audit [DBG] from='client.? v1:172.21.15.33:0/1795287206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:25.609 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:24.584421+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2024-02-12T17:13:25.609 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:24.899311+0000 mgr.y (mgr.24481) 895 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:25.609 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:24.919807+0000 mon.c (mon.2) 288 : audit [INF] from='client.? v1:172.21.15.33:0/2261435494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:13:25.609 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:24.920435+0000 mon.a (mon.0) 2041 : audit [INF] from='client.36086 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:13:25.609 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:24.984843+0000 mgr.y (mgr.24481) 896 : cluster [DBG] pgmap v1260: 125 pgs: 4 active+undersized+degraded+remapped+backfilling, 11 peering, 6 remapped+peering, 2 active+remapped+backfill_toofull, 24 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 43 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 930 op/s; 54453/328395 objects degraded (16.582%); 34251/328395 objects misplaced (10.430%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T17:13:25.609 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:25.251584+0000 mon.c (mon.2) 289 : audit [DBG] from='client.? v1:172.21.15.33:0/1795287206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:25.618 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:13:25.671 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:25.702 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:24.584421+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2024-02-12T17:13:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:24.899311+0000 mgr.y (mgr.24481) 895 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:24.919807+0000 mon.c (mon.2) 288 : audit [INF] from='client.? v1:172.21.15.33:0/2261435494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:13:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:24.920435+0000 mon.a (mon.0) 2041 : audit [INF] from='client.36086 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:13:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:24.984843+0000 mgr.y (mgr.24481) 896 : cluster [DBG] pgmap v1260: 125 pgs: 4 active+undersized+degraded+remapped+backfilling, 11 peering, 6 remapped+peering, 2 active+remapped+backfill_toofull, 24 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 43 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 930 op/s; 54453/328395 objects degraded (16.582%); 34251/328395 objects misplaced (10.430%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T17:13:25.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:25.251584+0000 mon.c (mon.2) 289 : audit [DBG] from='client.? v1:172.21.15.33:0/1795287206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:26.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 75 15 111704 111689 93.0595 37.625 0.0292082 0.0107414 2024-02-12T17:13:26.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:25.590417+0000 mon.a (mon.0) 2042 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:26.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:25.591229+0000 mon.a (mon.0) 2043 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:26.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:25.609072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.36086 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:13:26.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:25.609154+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2024-02-12T17:13:27.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:25.590417+0000 mon.a (mon.0) 2042 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:27.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:25.591229+0000 mon.a (mon.0) 2043 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:27.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:25.609072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.36086 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:13:27.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:25.609154+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2024-02-12T17:13:27.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:25.590417+0000 mon.a (mon.0) 2042 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:27.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:25.591229+0000 mon.a (mon.0) 2043 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:27.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:25.609072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.36086 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:13:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:25.609154+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2024-02-12T17:13:27.247 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3, 1, 6] out_osds: [4] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-12T17:13:27.247 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-02-12T17:13:27.247 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-02-12T17:13:27.247 DEBUG:teuthology.orchestra.run.smithi033:osd.1> sudo journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.1.service 2024-02-12T17:13:27.250 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl start ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.1 2024-02-12T17:13:27.307 INFO:journalctl@ceph.osd.1.smithi033.stdout:-- Logs begin at Mon 2024-02-12 16:32:30 UTC. -- 2024-02-12T17:13:27.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 76 16 112631 112615 92.5965 57.875 0.0109551 0.0107944 2024-02-12T17:13:27.618 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:27 smithi033 systemd[1]: Starting Ceph osd.1 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:13:27.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:26.611474+0000 mon.a (mon.0) 2046 : cluster [WRN] Health check update: Degraded data redundancy: 54453/328395 objects degraded (16.582%), 24 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:27.620 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:26.985774+0000 mgr.y (mgr.24481) 897 : cluster [DBG] pgmap v1262: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 50 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.6 KiB/s rd, 44 MiB/s wr, 700 op/s; 61849/331809 objects degraded (18.640%); 41313/331809 objects misplaced (12.451%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:13:27.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:26.611474+0000 mon.a (mon.0) 2046 : cluster [WRN] Health check update: Degraded data redundancy: 54453/328395 objects degraded (16.582%), 24 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:27.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:26.985774+0000 mgr.y (mgr.24481) 897 : cluster [DBG] pgmap v1262: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 50 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.6 KiB/s rd, 44 MiB/s wr, 700 op/s; 61849/331809 objects degraded (18.640%); 41313/331809 objects misplaced (12.451%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:13:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:26.611474+0000 mon.a (mon.0) 2046 : cluster [WRN] Health check update: Degraded data redundancy: 54453/328395 objects degraded (16.582%), 24 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:26.985774+0000 mgr.y (mgr.24481) 897 : cluster [DBG] pgmap v1262: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 50 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.6 KiB/s rd, 44 MiB/s wr, 700 op/s; 61849/331809 objects degraded (18.640%); 41313/331809 objects misplaced (12.451%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T17:13:28.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 77 16 113621 113605 92.1975 61.875 0.015954 0.010842 2024-02-12T17:13:28.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:27.623251+0000 mon.a (mon.0) 2047 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-02-12T17:13:29.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:27.623251+0000 mon.a (mon.0) 2047 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-02-12T17:13:29.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:27.623251+0000 mon.a (mon.0) 2047 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-02-12T17:13:29.424 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 78 16 114631 114615 91.8247 63.125 0.0311218 0.0108858 2024-02-12T17:13:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:13:29.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:13:29.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:28.986796+0000 mgr.y (mgr.24481) 898 : cluster [DBG] pgmap v1263: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s rd, 60 MiB/s wr, 968 op/s; 61235/339945 objects degraded (18.013%); 42664/339945 objects misplaced (12.550%); 25 MiB/s, 405 objects/s recovering 2024-02-12T17:13:29.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:28.986796+0000 mgr.y (mgr.24481) 898 : cluster [DBG] pgmap v1263: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s rd, 60 MiB/s wr, 968 op/s; 61235/339945 objects degraded (18.013%); 42664/339945 objects misplaced (12.550%); 25 MiB/s, 405 objects/s recovering 2024-02-12T17:13:29.888 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:29.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:29.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:28.986796+0000 mgr.y (mgr.24481) 898 : cluster [DBG] pgmap v1263: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.0 KiB/s rd, 60 MiB/s wr, 968 op/s; 61235/339945 objects degraded (18.013%); 42664/339945 objects misplaced (12.550%); 25 MiB/s, 405 objects/s recovering 2024-02-12T17:13:29.996 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:30.056 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:30.139 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:30.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:30.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:30.309 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:30.342 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:30.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:30.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 79 16 115594 115578 91.4242 60.1875 0.0133944 0.0109339 2024-02-12T17:13:30.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:30.507 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:30.551 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:30.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:30.628 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:13:30.629 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:13:30.629 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:13:30.629 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:13:30.629 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:13:30.640 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:29.852487+0000 osd.0 (osd.0) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:30.640 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:29.852539+0000 osd.0 (osd.0) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:30.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:30.737 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:30.772 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:30.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:29.852487+0000 osd.0 (osd.0) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:30.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:29.852539+0000 osd.0 (osd.0) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:29.852487+0000 osd.0 (osd.0) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:30.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:29.852539+0000 osd.0 (osd.0) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:30.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:30.880 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:30.934 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:31.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:13:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:30] "GET /metrics HTTP/1.1" 200 176894 "" "Prometheus/2.33.4" 2024-02-12T17:13:31.278 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:30 smithi033 bash[194946]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-12T17:13:31.278 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:30 smithi033 bash[194946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-12T17:13:31.278 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:30 smithi033 bash[194946]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-02-12T17:13:31.278 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:30 smithi033 bash[194946]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-12T17:13:31.278 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:30 smithi033 bash[194946]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-02-12T17:13:31.278 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:30 smithi033 bash[194946]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-12T17:13:31.278 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:30 smithi033 bash[194946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-12T17:13:31.278 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:30 smithi033 bash[194946]: --> ceph-volume lvm activate successful for osd ID: 1 2024-02-12T17:13:31.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:2024-02-12T17:13:31.424653+0000 min lat: 0.00107612 max lat: 2.19819 avg lat: 0.0109865 2024-02-12T17:13:31.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T17:13:31.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 80 16 116499 116483 90.9884 56.5625 0.0101007 0.0109865 2024-02-12T17:13:31.677 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:30.647212+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e627: 8 total, 7 up, 7 in 2024-02-12T17:13:31.677 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:30.987580+0000 mgr.y (mgr.24481) 899 : cluster [DBG] pgmap v1265: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 799 B/s rd, 38 MiB/s wr, 601 op/s; 61235/339945 objects degraded (18.013%); 42664/339945 objects misplaced (12.550%); 21 MiB/s, 338 objects/s recovering 2024-02-12T17:13:31.677 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:31.612887+0000 mon.a (mon.0) 2049 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:31.677 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:31.612924+0000 mon.a (mon.0) 2050 : cluster [WRN] Health check update: Degraded data redundancy: 61235/339945 objects degraded (18.013%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:31.678 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:31.635773+0000 mon.a (mon.0) 2051 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2024-02-12T17:13:31.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:30.647212+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e627: 8 total, 7 up, 7 in 2024-02-12T17:13:31.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:30.987580+0000 mgr.y (mgr.24481) 899 : cluster [DBG] pgmap v1265: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 799 B/s rd, 38 MiB/s wr, 601 op/s; 61235/339945 objects degraded (18.013%); 42664/339945 objects misplaced (12.550%); 21 MiB/s, 338 objects/s recovering 2024-02-12T17:13:31.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:31.612887+0000 mon.a (mon.0) 2049 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:31.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:31.612924+0000 mon.a (mon.0) 2050 : cluster [WRN] Health check update: Degraded data redundancy: 61235/339945 objects degraded (18.013%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:31.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:31.635773+0000 mon.a (mon.0) 2051 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2024-02-12T17:13:31.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:13:31.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:30.647212+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e627: 8 total, 7 up, 7 in 2024-02-12T17:13:31.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:30.987580+0000 mgr.y (mgr.24481) 899 : cluster [DBG] pgmap v1265: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 799 B/s rd, 38 MiB/s wr, 601 op/s; 61235/339945 objects degraded (18.013%); 42664/339945 objects misplaced (12.550%); 21 MiB/s, 338 objects/s recovering 2024-02-12T17:13:31.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:31.612887+0000 mon.a (mon.0) 2049 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:31.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:31.612924+0000 mon.a (mon.0) 2050 : cluster [WRN] Health check update: Degraded data redundancy: 61235/339945 objects degraded (18.013%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:31.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:31.635773+0000 mon.a (mon.0) 2051 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2024-02-12T17:13:32.269 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T17:13:32.296 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:32.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:32.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:32.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 81 16 117447 117431 90.5965 59.25 0.0173459 0.0110335 2024-02-12T17:13:32.448 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:32.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:13:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:13:33.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 82 16 118390 118374 90.2104 58.9375 0.0122017 0.0110805 2024-02-12T17:13:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:34.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:34.295 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:34.314 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:34.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:34.400 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:32.988338+0000 mgr.y (mgr.24481) 900 : cluster [DBG] pgmap v1267: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 693 B/s rd, 48 MiB/s wr, 761 op/s; 62115/345237 objects degraded (17.992%); 42961/345237 objects misplaced (12.444%); 18 MiB/s, 294 objects/s recovering 2024-02-12T17:13:34.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 83 16 119335 119319 89.8351 59.0625 0.00353262 0.011127 2024-02-12T17:13:34.431 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:34.473 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:34.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:34.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:32.988338+0000 mgr.y (mgr.24481) 900 : cluster [DBG] pgmap v1267: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 693 B/s rd, 48 MiB/s wr, 761 op/s; 62115/345237 objects degraded (17.992%); 42961/345237 objects misplaced (12.444%); 18 MiB/s, 294 objects/s recovering 2024-02-12T17:13:34.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:32.988338+0000 mgr.y (mgr.24481) 900 : cluster [DBG] pgmap v1267: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 693 B/s rd, 48 MiB/s wr, 761 op/s; 62115/345237 objects degraded (17.992%); 42961/345237 objects misplaced (12.444%); 18 MiB/s, 294 objects/s recovering 2024-02-12T17:13:34.579 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:34.639 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:34.701 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:13:34.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:34.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:13:34.807 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:34.879 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:34.924 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:34.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:35.031 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:35.069 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:35.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:13:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:13:35.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 84 16 120245 120229 89.4427 56.875 0.0162953 0.0111759 2024-02-12T17:13:35.749 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:13:36.068 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:34.909721+0000 mgr.y (mgr.24481) 901 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:36.068 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:34.989090+0000 mgr.y (mgr.24481) 902 : cluster [DBG] pgmap v1268: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 936 op/s; 62130/354261 objects degraded (17.538%); 44429/354261 objects misplaced (12.541%); 30 MiB/s, 476 objects/s recovering 2024-02-12T17:13:36.069 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:34.909721+0000 mgr.y (mgr.24481) 901 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:36.069 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:34.989090+0000 mgr.y (mgr.24481) 902 : cluster [DBG] pgmap v1268: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 936 op/s; 62130/354261 objects degraded (17.538%); 44429/354261 objects misplaced (12.541%); 30 MiB/s, 476 objects/s recovering 2024-02-12T17:13:36.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 85 16 121121 121105 89.0345 54.75 0.0135003 0.0112274 2024-02-12T17:13:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:34.909721+0000 mgr.y (mgr.24481) 901 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:36.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:34.989090+0000 mgr.y (mgr.24481) 902 : cluster [DBG] pgmap v1268: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 936 op/s; 62130/354261 objects degraded (17.538%); 44429/354261 objects misplaced (12.541%); 30 MiB/s, 476 objects/s recovering 2024-02-12T17:13:37.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 86 16 121957 121941 88.6067 52.25 0.0235213 0.0112816 2024-02-12T17:13:37.476 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:36.614421+0000 mon.a (mon.0) 2052 : cluster [WRN] Health check update: Degraded data redundancy: 62130/354261 objects degraded (17.538%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:37.477 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:36.614421+0000 mon.a (mon.0) 2052 : cluster [WRN] Health check update: Degraded data redundancy: 62130/354261 objects degraded (17.538%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:37.482 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:36.614421+0000 mon.a (mon.0) 2052 : cluster [WRN] Health check update: Degraded data redundancy: 62130/354261 objects degraded (17.538%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:37.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:37.594 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:37.632 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:38.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 87 16 122944 122928 88.2973 61.6875 0.00578078 0.011321 2024-02-12T17:13:38.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:36.989986+0000 mgr.y (mgr.24481) 903 : cluster [DBG] pgmap v1269: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 817 op/s; 62991/359553 objects degraded (17.519%); 44683/359553 objects misplaced (12.427%); 13 MiB/s, 204 objects/s recovering 2024-02-12T17:13:38.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:36.989986+0000 mgr.y (mgr.24481) 903 : cluster [DBG] pgmap v1269: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 817 op/s; 62991/359553 objects degraded (17.519%); 44683/359553 objects misplaced (12.427%); 13 MiB/s, 204 objects/s recovering 2024-02-12T17:13:38.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:36.989986+0000 mgr.y (mgr.24481) 903 : cluster [DBG] pgmap v1269: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 817 op/s; 62991/359553 objects degraded (17.519%); 44683/359553 objects misplaced (12.427%); 13 MiB/s, 204 objects/s recovering 2024-02-12T17:13:38.695 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:38.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:38.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:38.798 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T17:13:38.801 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:38.886 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:38.930 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:38.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:39.027 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:38 smithi033 bash[195654]: ab24d21a08382af2273528f65614f38358a973bc7f65bae1b54d3c30a2916684 2024-02-12T17:13:39.027 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:38 smithi033 systemd[1]: Started Ceph osd.1 for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T17:13:39.027 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[195731]: debug 2024-02-12T17:13:38.915+0000 7f2cb9fb7380 -1 Falling back to public interface 2024-02-12T17:13:39.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:39.125 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:39.170 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:39.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:39.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:39.328 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:39.372 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:39.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:39.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 88 16 123887 123871 87.9636 58.9375 0.018158 0.0113643 2024-02-12T17:13:39.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:39.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:38.463538+0000 mon.a (mon.0) 2053 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:39.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:38.463538+0000 mon.a (mon.0) 2053 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:39.537 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:39.587 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:39.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:39.695 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:39.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:38.463538+0000 mon.a (mon.0) 2053 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:39.794 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:39.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:39.901 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:39.952 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:40.411 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:40.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:40.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 89 16 124849 124833 87.6508 60.125 0.0176553 0.0114048 2024-02-12T17:13:40.452 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:13:40.483 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:13:40.512 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:13:40.519 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:40.535 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:13:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-02-12T17:13:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-02-12T17:13:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:13:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.15459.0:183815 9.1 9:8de3b1cd:::benchmark_data_smithi033_131576_object46225:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-02-12T17:13:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:03:50.683095+0000", 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 589.59463480800002, 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.026021957, 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.15459", 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/260956091", 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 183815 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683095+0000", 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683095+0000", 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683093+0000", 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6199999999999999e-06 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683100+0000", 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0569e-05 2024-02-12T17:13:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683111+0000", 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6089999999999997e-06 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683117+0000", 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8913999999999999e-05 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683176+0000", 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0161e-05 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:50.683186+0000", 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99201056899999995 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.675197+0000", 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012823820000000001 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.676479+0000", 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018419700000000001 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,4,7", 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.676664+0000", 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059324800000000004 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.682596+0000", 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5244999999999999e-05 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.682631+0000", 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0032687770000000001 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.685900+0000", 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023019000000000001 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.708919+0000", 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3932999999999999e-05 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.708983+0000", 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013393199999999999 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:03:51.709117+0000", 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.026021957 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123098 10.13 10:cbaa2e55:::benchmark_data_smithi033_160181_object123097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e483)", 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:54.073995+0000", 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 346.20373448200002, 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0604953479999999, 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123098 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.073995+0000", 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.073995+0000", 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.073993+0000", 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7579000000000001e-05 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074020+0000", 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1139e-05 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074031+0000", 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7139999999999993e-06 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074039+0000", 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8359999999999998e-05 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074087+0000", 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2359e-05 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074100+0000", 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0351537180000001 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.109254+0000", 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000179882 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.109433+0000", 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000305352 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,7", 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.109739+0000", 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010413054999999999 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.120152+0000", 2024-02-12T17:13:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000238514 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.120390+0000", 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010396674999999999 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.130787+0000", 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003495761 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134283+0000", 2024-02-12T17:13:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4449999999999994e-05 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134347+0000", 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000143468 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134491+0000", 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0604953479999999 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123110 10.19 10:9c61a0a8:::benchmark_data_smithi033_160181_object123109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e483)", 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:54.074048+0000", 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 346.20368158799999, 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.084618791, 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123110 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074048+0000", 2024-02-12T17:13:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074048+0000", 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074047+0000", 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4651999999999999e-05 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074092+0000", 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9009999999999997e-06 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074099+0000", 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4999999999999996e-06 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074106+0000", 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012267499999999999 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074228+0000", 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6290000000000005e-06 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074238+0000", 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0680504829999999 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.142288+0000", 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000157546 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.142446+0000", 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029158500000000002 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,3,5,7", 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.142737+0000", 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002381097 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.145119+0000", 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7261e-05 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.145136+0000", 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017930299999999999 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.145315+0000", 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074436599999999999 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.146059+0000", 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012363074 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.158423+0000", 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8609000000000006e-05 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.158501+0000", 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016592399999999999 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.158667+0000", 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.084618791 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123122 10.13 10:c8283936:::benchmark_data_smithi033_160181_object123121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e483)", 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:54.074118+0000", 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 346.20361230499998, 2024-02-12T17:13:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0605462370000001, 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123122 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074118+0000", 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074118+0000", 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.999999 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074117+0000", 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0664000000000002e-05 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074157+0000", 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4209999999999998e-06 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074165+0000", 2024-02-12T17:13:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9869999999999999e-06 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074170+0000", 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8529000000000002e-05 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074228+0000", 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5509999999999999e-06 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074234+0000", 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0365176389999999 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.110752+0000", 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017214300000000001 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.110924+0000", 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000298094 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,7", 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.111222+0000", 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089842110000000006 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.120206+0000", 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033086600000000003 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.120537+0000", 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010285898 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.130823+0000", 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036797319999999998 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134503+0000", 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6190000000000003e-05 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134549+0000", 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115091 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134664+0000", 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0605462370000001 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123132 10.13 10:cea9284b:::benchmark_data_smithi033_160181_object123131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e483)", 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:54.074180+0000", 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 346.20354937600001, 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.060649545, 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123132 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074180+0000", 2024-02-12T17:13:40.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074180+0000", 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074180+0000", 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3299e-05 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074223+0000", 2024-02-12T17:13:40.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1300000000000003e-06 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074230+0000", 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0430000000000002e-06 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074237+0000", 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8770000000000003e-05 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074276+0000", 2024-02-12T17:13:40.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5370000000000003e-06 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.074283+0000", 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0379018229999999 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.112184+0000", 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000149653 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.112334+0000", 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000260525 2024-02-12T17:13:40.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,7", 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.112595+0000", 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079791549999999999 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.120574+0000", 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010281096 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.130855+0000", 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9346e-05 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.130884+0000", 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037880000000000001 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134672+0000", 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5039e-05 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134717+0000", 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011283800000000001 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134830+0000", 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.060649545 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:123161 10.13 10:c82c69d9:::benchmark_data_smithi033_160181_object123160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e483)", 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:07:54.091793+0000", 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 346.18593720699999, 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0432007210000001, 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.568 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 123161 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.091793+0000", 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.091793+0000", 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.091790+0000", 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2110000000000001e-05 2024-02-12T17:13:40.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.091842+0000", 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1511e-05 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.091853+0000", 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0599999999999996e-06 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.091859+0000", 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6609999999999997e-05 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.091896+0000", 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3720000000000001e-06 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:54.091902+0000", 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.021801626 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.113704+0000", 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014969199999999999 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.113854+0000", 2024-02-12T17:13:40.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029043599999999998 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,7", 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.114144+0000", 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0066095939999999999 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.120754+0000", 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010160515 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.130914+0000", 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001764205 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.132679+0000", 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021599940000000001 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134839+0000", 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3686999999999997e-05 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134882+0000", 2024-02-12T17:13:40.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111166 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:07:55.134993+0000", 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0432007210000001 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174484 10.b 10:d217cf23:::benchmark_data_smithi033_160181_object51312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.548227+0000", 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 259.72950267599998, 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.075691562, 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.574 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174484 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548227+0000", 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548227+0000", 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:13:40.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548225+0000", 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7138000000000001e-05 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548242+0000", 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1348999999999999e-05 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548254+0000", 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3720000000000001e-06 2024-02-12T17:13:40.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548260+0000", 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013573799999999999 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548396+0000", 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013111500000000001 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548527+0000", 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033629599999999998 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,7", 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548863+0000", 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044791179999999998 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.553342+0000", 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022465739999999999 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.555589+0000", 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037435110000000001 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.559332+0000", 2024-02-12T17:13:40.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023322775 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.582655+0000", 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1654000000000001e-05 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.582667+0000", 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0411751439999999 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.623842+0000", 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6817999999999995e-05 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.623919+0000", 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.075691562 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174489 10.1d 10:be0e53bc:::benchmark_data_smithi033_160181_object51317:head [delete] snapc 0=[] ondisk+write+known_if_redirected e512)", 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.548289+0000", 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 259.72944039100003, 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0610144699999999, 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.580 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174489 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548289+0000", 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548289+0000", 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548289+0000", 2024-02-12T17:13:40.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.394e-06 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548292+0000", 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1460000000000004e-06 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548297+0000", 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0899999999999998e-06 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.548301+0000", 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089486180000000002 2024-02-12T17:13:40.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.557250+0000", 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013864699999999999 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.557389+0000", 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026830700000000002 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5,7", 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.557657+0000", 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045124550000000003 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.562170+0000", 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048272059999999997 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.566997+0000", 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7989e-05 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.567015+0000", 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024101498999999998 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.591116+0000", 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7630000000000006e-06 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.591125+0000", 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0181139560000001 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.609239+0000", 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4999999999999994e-05 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.609304+0000", 2024-02-12T17:13:40.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0610144699999999 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174502 10.1d 10:baa8bd85:::benchmark_data_smithi033_160181_object51330:head [delete] snapc 0=[] ondisk+write+known_if_redirected e513)", 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.604524+0000", 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 259.67320606499999, 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0191680860000001, 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174502 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.604524+0000", 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.604524+0000", 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999967 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.604521+0000", 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1549999999999996e-06 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.604530+0000", 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1834000000000001e-05 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.604542+0000", 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9310000000000008e-06 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.604550+0000", 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000209168 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.604759+0000", 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0091e-05 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.604769+0000", 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.004855338 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.609624+0000", 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000112334 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.609737+0000", 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020363799999999999 2024-02-12T17:13:40.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,5,6,7", 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.609940+0000", 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001542542 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.611483+0000", 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016735739999999999 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.613156+0000", 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00068856199999999998 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.613845+0000", 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017859830000000001 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.615631+0000", 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00078360099999999998 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.616414+0000", 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070639810000000004 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.623478+0000", 2024-02-12T17:13:40.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9277000000000002e-05 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.623538+0000", 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000154198 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.623692+0000", 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0191680860000001 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174512 10.0 10:00c8f393:::benchmark_data_smithi033_160181_object51340:head [delete] snapc 0=[] ondisk+write+known_if_redirected e513)", 2024-02-12T17:13:40.592 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.657567+0000", 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 259.62016329699998, 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90916702100000002, 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174512 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657567+0000", 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657567+0000", 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657564+0000", 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.168e-05 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657575+0000", 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1532e-05 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657597+0000", 2024-02-12T17:13:40.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0269999999999995e-06 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657605+0000", 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001813853 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.659419+0000", 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1812e-05 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.659430+0000", 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.89088120199999998 2024-02-12T17:13:40.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.550312+0000", 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000142638 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.550454+0000", 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028575399999999999 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,5,6,7", 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.550740+0000", 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054289170000000001 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.556169+0000", 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022056279999999998 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.558375+0000", 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040584460000000003 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.562433+0000", 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010226160000000001 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.563456+0000", 2024-02-12T17:13:40.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003069268 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.566525+0000", 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8956000000000003e-05 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.566584+0000", 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014971799999999999 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.566734+0000", 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90916702100000002 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.598 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174513 10.b 10:d36aebfd:::benchmark_data_smithi033_160181_object51341:head [delete] snapc 0=[] ondisk+write+known_if_redirected e513)", 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:20.657619+0000", 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 259.62011102299999, 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.97533211900000005, 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174513 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657619+0000", 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657619+0000", 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657618+0000", 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5329999999999999e-06 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657622+0000", 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3990000000000003e-06 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657627+0000", 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.138e-06 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657631+0000", 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8789000000000002e-05 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657690+0000", 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2862e-05 2024-02-12T17:13:40.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:20.657703+0000", 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96652830099999998 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.624231+0000", 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012301 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.624354+0000", 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031729399999999997 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,3,6,7", 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.624672+0000", 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016559649999999999 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.626327+0000", 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002190727 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.628518+0000", 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071698400000000005 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.629235+0000", 2024-02-12T17:13:40.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019611519999999999 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631196+0000", 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015544580000000001 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.632751+0000", 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5576999999999999e-05 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.632816+0000", 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013456300000000001 2024-02-12T17:13:40.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.632951+0000", 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.97533211900000005 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174508 10.a 10:52de054a:::benchmark_data_smithi033_160181_object51336:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e514)", 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.469078+0000", 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.80865157400001, 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.154072464, 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.605 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174508 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.469078+0000", 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.469078+0000", 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.469076+0000", 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7970000000000006e-06 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.469085+0000", 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070817049999999998 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.476166+0000", 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1833e-05 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.476178+0000", 2024-02-12T17:13:40.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014700582 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.490879+0000", 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0589e-05 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.490889+0000", 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1053605499999999 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.596250+0000", 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116493 2024-02-12T17:13:40.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.596366+0000", 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027547999999999998 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,6", 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.596642+0000", 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012376638000000001 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609018+0000", 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4773000000000001e-05 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609053+0000", 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000112477 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609166+0000", 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013819605 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.622985+0000", 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8933000000000001e-05 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.623034+0000", 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116536 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.623151+0000", 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.154072464 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174521 10.a 10:54c5c51b:::benchmark_data_smithi033_160181_object51349:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.604767+0000", 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.67296328200001, 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0185510849999999, 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.611 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174521 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.604767+0000", 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.604767+0000", 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:40.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.604764+0000", 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2789999999999998e-06 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.604772+0000", 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1887e-05 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.604784+0000", 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5930000000000002e-06 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.604791+0000", 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028254439999999999 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.607616+0000", 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0753e-05 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.607627+0000", 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99034537600000005 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.597972+0000", 2024-02-12T17:13:40.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013059700000000001 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.598103+0000", 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000209741 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,6", 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.598313+0000", 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01078282 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609095+0000", 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7951000000000003e-05 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609133+0000", 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125926 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609259+0000", 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013899662 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.623159+0000", 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5683999999999998e-05 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.623205+0000", 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000112961 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.623318+0000", 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0185510849999999 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174522 10.7 10:e0fe2426:::benchmark_data_smithi033_160181_object51350:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.622446+0000", 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.65528344500001, 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98831616600000005, 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.617 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174522 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.622446+0000", 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.622446+0000", 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.622444+0000", 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6020000000000002e-06 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.622452+0000", 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9089999999999995e-06 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.622462+0000", 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8680000000000004e-06 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.622468+0000", 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015180599999999999 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.622619+0000", 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7960000000000001e-06 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.622628+0000", 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.97095370199999997 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.593582+0000", 2024-02-12T17:13:40.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000137188 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.593719+0000", 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00037865799999999998 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,6", 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.594098+0000", 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003890527 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.597988+0000", 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081123700000000003 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.598800+0000", 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057355790000000002 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.604535+0000", 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060482909999999999 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.610583+0000", 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0067e-05 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.610633+0000", 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000129123 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.610763+0000", 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98831616600000005 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174529 10.7 10:e6e85cc1:::benchmark_data_smithi033_160181_object51357:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.631624+0000", 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.64610611900002, 2024-02-12T17:13:40.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99066993000000003, 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174529 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631624+0000", 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631624+0000", 2024-02-12T17:13:40.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631621+0000", 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7239999999999998e-06 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631630+0000", 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2036e-05 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631642+0000", 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1539999999999996e-06 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631649+0000", 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0243e-05 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631699+0000", 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.85e-06 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.631708+0000", 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.96421781699999998 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.595926+0000", 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114298 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.596040+0000", 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027874600000000002 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,6", 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.596319+0000", 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002901654 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.599221+0000", 2024-02-12T17:13:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003162083 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.602383+0000", 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.005197719 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.607580+0000", 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014532346999999999 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.622113+0000", 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3263999999999997e-05 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.622166+0000", 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000127682 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.622294+0000", 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.99066993000000003 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174531 10.d 10:b0084bee:::benchmark_data_smithi033_160181_object51359:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.633406+0000", 2024-02-12T17:13:40.629 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.64432385999999, 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0123133099999999, 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174531 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.633406+0000", 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.633406+0000", 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.633405+0000", 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6579999999999999e-06 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.633410+0000", 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8869999999999998e-06 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.633419+0000", 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0889999999999996e-06 2024-02-12T17:13:40.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.633425+0000", 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5796e-05 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.633471+0000", 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0590000000000003e-06 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.633479+0000", 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.982750444 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.616229+0000", 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3417000000000004e-05 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.616313+0000", 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029387400000000001 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,6,7", 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.616607+0000", 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014957570999999999 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.631564+0000", 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5345000000000001e-05 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.631600+0000", 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023462819999999999 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.633946+0000", 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011651392999999999 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.645597+0000", 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6603000000000001e-05 2024-02-12T17:13:40.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.645644+0000", 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5323999999999999e-05 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.645719+0000", 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0123133099999999 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174535 10.d 10:b71c7904:::benchmark_data_smithi033_160181_object51363:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.636948+0000", 2024-02-12T17:13:40.635 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.640781774, 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0088868900000001, 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174535 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.636948+0000", 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.636948+0000", 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.636946+0000", 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6120000000000004e-06 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.636953+0000", 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9509999999999995e-06 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.636962+0000", 2024-02-12T17:13:40.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.668e-06 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.636966+0000", 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4350000000000001e-05 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.637011+0000", 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6510000000000007e-06 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.637019+0000", 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.98068510200000003 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.617704+0000", 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2959000000000004e-05 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.617777+0000", 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014504900000000001 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,6,7", 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.617922+0000", 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013712929 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.631635+0000", 2024-02-12T17:13:40.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4044000000000001e-05 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.631669+0000", 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022848870000000002 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.633954+0000", 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011791061 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.645745+0000", 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5636999999999999e-05 2024-02-12T17:13:40.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.645771+0000", 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4070000000000002e-05 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.645835+0000", 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0088868900000001 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174544 10.a 10:57ea5c56:::benchmark_data_smithi033_160181_object51372:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.701027+0000", 2024-02-12T17:13:40.641 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.57670304999999, 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.922455735, 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174544 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.701027+0000", 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.701027+0000", 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.701025+0000", 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0509999999999999e-06 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.701032+0000", 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1266e-05 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.701043+0000", 2024-02-12T17:13:40.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.105e-06 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.701049+0000", 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1243999999999999e-05 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.701100+0000", 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0809999999999996e-06 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.701107+0000", 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.90109221299999998 2024-02-12T17:13:40.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.602200+0000", 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117868 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.602317+0000", 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000344784 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,6", 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.602662+0000", 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0065357899999999997 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609198+0000", 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0959e-05 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609229+0000", 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1136999999999994e-05 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609290+0000", 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014035306000000001 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.623325+0000", 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7542e-05 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.623373+0000", 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010959399999999999 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.623483+0000", 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.922455735 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.647 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174546 10.e 10:75f79f81:::benchmark_data_smithi033_160181_object51374:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.703536+0000", 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.57419405299999, 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.91917628900000004, 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174546 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.703536+0000", 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.703536+0000", 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.703534+0000", 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5209999999999999e-06 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.703541+0000", 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9409999999999996e-06 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.703550+0000", 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9070000000000004e-06 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.703555+0000", 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6690999999999997e-05 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.703602+0000", 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1790000000000008e-06 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.703610+0000", 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.89768130599999996 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.601292+0000", 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3429000000000003e-05 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.601385+0000", 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028175599999999998 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,5,6", 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.601667+0000", 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.005526034 2024-02-12T17:13:40.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.607193+0000", 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002273438 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609466+0000", 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4174999999999997e-05 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.609500+0000", 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013003796999999999 2024-02-12T17:13:40.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.622504+0000", 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5485999999999997e-05 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.622560+0000", 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015230699999999999 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.622712+0000", 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.91917628900000004 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.653 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.25603.0:174565 10.d 10:b5ab7e2a:::benchmark_data_smithi033_160181_object51393:head [delete] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:09:21.731933+0000", 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 258.54579663999999, 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.91974358199999995, 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.25603", 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 174565 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.654 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.731933+0000", 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.731933+0000", 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.731931+0000", 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0219999999999999e-06 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.731938+0000", 2024-02-12T17:13:40.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3739e-05 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.731952+0000", 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7209999999999997e-06 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.731959+0000", 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7416000000000006e-05 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.732036+0000", 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1097999999999999e-05 2024-02-12T17:13:40.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:21.732047+0000", 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.88801510400000006 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.620062+0000", 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012525000000000001 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.620188+0000", 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026701300000000003 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 3,6,7", 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.620455+0000", 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011246643000000001 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.631701+0000", 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001963457 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.633665+0000", 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029515599999999998 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.633960+0000", 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017525424000000001 2024-02-12T17:13:40.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.651485+0000", 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7821000000000001e-05 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.651543+0000", 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000133654 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:09:22.651677+0000", 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.91974358199999995 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.659 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.660 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:40.660 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:40.660 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:13:40.660 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:13:40.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:38.990874+0000 mgr.y (mgr.24481) 904 : cluster [DBG] pgmap v1270: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 62465/367926 objects degraded (16.978%); 46063/367926 objects misplaced (12.520%); 27 MiB/s, 432 objects/s recovering 2024-02-12T17:13:40.660 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:40.414605+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? v1:172.21.15.33:0/628304235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:40.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:38.990874+0000 mgr.y (mgr.24481) 904 : cluster [DBG] pgmap v1270: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 62465/367926 objects degraded (16.978%); 46063/367926 objects misplaced (12.520%); 27 MiB/s, 432 objects/s recovering 2024-02-12T17:13:40.661 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:40.414605+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? v1:172.21.15.33:0/628304235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:40.661 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:40.661 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:13:40.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:13:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:38.990874+0000 mgr.y (mgr.24481) 904 : cluster [DBG] pgmap v1270: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 62465/367926 objects degraded (16.978%); 46063/367926 objects misplaced (12.520%); 27 MiB/s, 432 objects/s recovering 2024-02-12T17:13:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:40.414605+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? v1:172.21.15.33:0/628304235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:41.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:13:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:40] "GET /metrics HTTP/1.1" 200 176930 "" "Prometheus/2.33.4" 2024-02-12T17:13:41.425 INFO:tasks.radosbench.radosbench.0.smithi033.stdout: 90 1 125687 125686 87.2693 53.3125 0.0347032 0.011455 2024-02-12T17:13:41.937 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total time run: 90.0133 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Total writes made: 125687 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Write size: 65536 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Object size: 65536 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Bandwidth (MB/sec): 87.2697 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Bandwidth: 39.1625 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max bandwidth (MB/sec): 238 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min bandwidth (MB/sec): 0 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average IOPS: 1396 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev IOPS: 626.609 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max IOPS: 3808 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min IOPS: 0 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Average Latency(s): 0.0114551 2024-02-12T17:13:42.426 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Stddev Latency(s): 0.025326 2024-02-12T17:13:42.427 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Max latency(s): 2.19819 2024-02-12T17:13:42.427 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Min latency(s): 0.00107612 2024-02-12T17:13:42.430 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T17:13:42.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:40.993379+0000 mgr.y (mgr.24481) 905 : cluster [DBG] pgmap v1271: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 933 op/s; 62465/367926 objects degraded (16.978%); 46063/367926 objects misplaced (12.520%); 23 MiB/s, 360 objects/s recovering 2024-02-12T17:13:42.489 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:41.615292+0000 mon.a (mon.0) 2054 : cluster [WRN] Health check update: Degraded data redundancy: 62465/367926 objects degraded (16.978%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:42.726 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T17:13:42.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:40.993379+0000 mgr.y (mgr.24481) 905 : cluster [DBG] pgmap v1271: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 933 op/s; 62465/367926 objects degraded (16.978%); 46063/367926 objects misplaced (12.520%); 23 MiB/s, 360 objects/s recovering 2024-02-12T17:13:42.743 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:41.615292+0000 mon.a (mon.0) 2054 : cluster [WRN] Health check update: Degraded data redundancy: 62465/367926 objects degraded (16.978%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:42.744 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:13:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:13:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:40.993379+0000 mgr.y (mgr.24481) 905 : cluster [DBG] pgmap v1271: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 933 op/s; 62465/367926 objects degraded (16.978%); 46063/367926 objects misplaced (12.520%); 23 MiB/s, 360 objects/s recovering 2024-02-12T17:13:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:41.615292+0000 mon.a (mon.0) 2054 : cluster [WRN] Health check update: Degraded data redundancy: 62465/367926 objects degraded (16.978%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:44.272 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[195731]: debug 2024-02-12T17:13:43.814+0000 7f2cb9fb7380 -1 osd.1 532 log_to_monitors {default=true} 2024-02-12T17:13:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T17:13:44.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T17:13:44.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:42.995273+0000 mgr.y (mgr.24481) 906 : cluster [DBG] pgmap v1272: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 901 B/s rd, 51 MiB/s wr, 819 op/s; 63334/373134 objects degraded (16.974%); 46335/373134 objects misplaced (12.418%); 20 MiB/s, 317 objects/s recovering 2024-02-12T17:13:44.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:43.823068+0000 mon.c (mon.2) 290 : audit [INF] from='osd.1 v1:172.21.15.33:6805/2528921223' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T17:13:44.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:43.824449+0000 mon.a (mon.0) 2055 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T17:13:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:42.995273+0000 mgr.y (mgr.24481) 906 : cluster [DBG] pgmap v1272: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 901 B/s rd, 51 MiB/s wr, 819 op/s; 63334/373134 objects degraded (16.974%); 46335/373134 objects misplaced (12.418%); 20 MiB/s, 317 objects/s recovering 2024-02-12T17:13:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:43.823068+0000 mon.c (mon.2) 290 : audit [INF] from='osd.1 v1:172.21.15.33:6805/2528921223' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T17:13:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:43.824449+0000 mon.a (mon.0) 2055 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T17:13:44.755 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:44.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:44.864 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:44.926 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:44.932 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:42.995273+0000 mgr.y (mgr.24481) 906 : cluster [DBG] pgmap v1272: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 901 B/s rd, 51 MiB/s wr, 819 op/s; 63334/373134 objects degraded (16.974%); 46335/373134 objects misplaced (12.418%); 20 MiB/s, 317 objects/s recovering 2024-02-12T17:13:44.932 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:43.823068+0000 mon.c (mon.2) 290 : audit [INF] from='osd.1 v1:172.21.15.33:6805/2528921223' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T17:13:44.932 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:43.824449+0000 mon.a (mon.0) 2055 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T17:13:44.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:13:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:13:45.027 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:13:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[195731]: debug 2024-02-12T17:13:44.892+0000 7f2caa49c700 -1 osd.1 629 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T17:13:45.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:44.501792+0000 mon.a (mon.0) 2056 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T17:13:45.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:44.501859+0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2024-02-12T17:13:45.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:44.513066+0000 mon.c (mon.2) 291 : audit [INF] from='osd.1 v1:172.21.15.33:6805/2528921223' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T17:13:45.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:44.513389+0000 mon.a (mon.0) 2058 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T17:13:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:44.501792+0000 mon.a (mon.0) 2056 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T17:13:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:44.501859+0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2024-02-12T17:13:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:44.513066+0000 mon.c (mon.2) 291 : audit [INF] from='osd.1 v1:172.21.15.33:6805/2528921223' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T17:13:45.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:44.513389+0000 mon.a (mon.0) 2058 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T17:13:45.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:44.501792+0000 mon.a (mon.0) 2056 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T17:13:45.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:44.501859+0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2024-02-12T17:13:45.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:44.513066+0000 mon.c (mon.2) 291 : audit [INF] from='osd.1 v1:172.21.15.33:6805/2528921223' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T17:13:45.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:44.513389+0000 mon.a (mon.0) 2058 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-02-12T17:13:46.291 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:13:46.594 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:46.598 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:44.930519+0000 mgr.y (mgr.24481) 907 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:46.598 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:44.995935+0000 mgr.y (mgr.24481) 908 : cluster [DBG] pgmap v1274: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 858 op/s; 60379/375288 objects degraded (16.089%); 46713/375288 objects misplaced (12.447%); 33 MiB/s, 535 objects/s recovering 2024-02-12T17:13:46.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:45.509385+0000 mon.a (mon.0) 2059 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T17:13:46.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:45.521197+0000 mon.a (mon.0) 2060 : cluster [INF] osd.1 v1:172.21.15.33:6805/2528921223 boot 2024-02-12T17:13:46.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:45.521239+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e630: 8 total, 8 up, 7 in 2024-02-12T17:13:46.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:45.522386+0000 mon.c (mon.2) 292 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T17:13:46.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:44.930519+0000 mgr.y (mgr.24481) 907 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:46.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:44.995935+0000 mgr.y (mgr.24481) 908 : cluster [DBG] pgmap v1274: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 858 op/s; 60379/375288 objects degraded (16.089%); 46713/375288 objects misplaced (12.447%); 33 MiB/s, 535 objects/s recovering 2024-02-12T17:13:46.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:45.509385+0000 mon.a (mon.0) 2059 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T17:13:46.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:45.521197+0000 mon.a (mon.0) 2060 : cluster [INF] osd.1 v1:172.21.15.33:6805/2528921223 boot 2024-02-12T17:13:46.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:45.521239+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e630: 8 total, 8 up, 7 in 2024-02-12T17:13:46.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:45.522386+0000 mon.c (mon.2) 292 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T17:13:46.676 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:46.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:46.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:46.846 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:44.930519+0000 mgr.y (mgr.24481) 907 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:46.847 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:44.995935+0000 mgr.y (mgr.24481) 908 : cluster [DBG] pgmap v1274: 125 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 32 active+clean+remapped, 15 active+undersized, 20 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 858 op/s; 60379/375288 objects degraded (16.089%); 46713/375288 objects misplaced (12.447%); 33 MiB/s, 535 objects/s recovering 2024-02-12T17:13:46.847 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:45.509385+0000 mon.a (mon.0) 2059 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T17:13:46.847 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:45.521197+0000 mon.a (mon.0) 2060 : cluster [INF] osd.1 v1:172.21.15.33:6805/2528921223 boot 2024-02-12T17:13:46.847 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:45.521239+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e630: 8 total, 8 up, 7 in 2024-02-12T17:13:46.847 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:45.522386+0000 mon.c (mon.2) 292 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T17:13:46.859 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:46.914 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:46.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:47.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:47.071 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:47.112 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:47.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:47.217 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:47.284 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:47.451 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:47.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:47.559 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:47.624 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:47.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:44.864411+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:47.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:44.864508+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:47.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.529435+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e631: 8 total, 8 up, 7 in 2024-02-12T17:13:47.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.531217+0000 osd.2 (osd.2) 225 : cluster [DBG] 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 629'7955 2024-02-12T17:13:47.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.531233+0000 osd.2 (osd.2) 226 : cluster [DBG] 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8127 2024-02-12T17:13:47.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.532400+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 629'7972 2024-02-12T17:13:47.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.534354+0000 osd.3 (osd.3) 225 : cluster [INF] 2.d continuing backfill to osd.1 from (128'2,128'2] MIN to 128'2 2024-02-12T17:13:47.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.534971+0000 osd.3 (osd.3) 226 : cluster [DBG] 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 629'8054 2024-02-12T17:13:47.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.538312+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.e starting backfill to osd.1 from (0'0,0'0] MAX to 629'8046 2024-02-12T17:13:47.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.542942+0000 osd.3 (osd.3) 227 : cluster [INF] 5.13 continuing backfill to osd.1 from (348'12,419'14] MIN to 419'14 2024-02-12T17:13:47.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.542971+0000 osd.5 (osd.5) 312 : cluster [DBG] 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8001 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.556641+0000 osd.7 (osd.7) 308 : cluster [INF] 11.e continuing backfill to osd.3 from (608'3342,629'8046] 11:707e1c6b:::benchmark_data_smithi033_188107_object35338:head to 629'8046 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.558518+0000 osd.3 (osd.3) 228 : cluster [INF] 11.b continuing backfill to osd.5 from (608'3563,629'8054] MIN to 629'8054 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.563433+0000 osd.6 (osd.6) 175 : cluster [INF] 5.5 continuing backfill to osd.1 from (406'13,410'15] MIN to 410'15 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.563830+0000 osd.5 (osd.5) 313 : cluster [INF] 11.7 continuing backfill to osd.7 from (606'2678,629'8001] 11:e74718db:::benchmark_data_smithi033_188107_object26925:head to 629'8001 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.572607+0000 osd.2 (osd.2) 227 : cluster [INF] 11.6 continuing backfill to osd.5 from (606'2221,629'8127] 11:671a30a6:::benchmark_data_smithi033_188107_object84353:head to 629'8127 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.572920+0000 osd.2 (osd.2) 228 : cluster [INF] 11.c continuing backfill to osd.7 from (606'1974,629'7955] 11:366e84f1:::benchmark_data_smithi033_188107_object41569:head to 629'7955 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.616770+0000 mon.a (mon.0) 2063 : cluster [WRN] Health check update: Degraded data redundancy: 60379/375288 objects degraded (16.089%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.642258+0000 osd.7 (osd.7) 309 : cluster [DBG] 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 629'8045 2024-02-12T17:13:47.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.642893+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e632: 8 total, 8 up, 7 in 2024-02-12T17:13:47.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.643263+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 629'7888 2024-02-12T17:13:47.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.648750+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8121 2024-02-12T17:13:47.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.657487+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8078 2024-02-12T17:13:47.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:47.526828+0000 mon.a (mon.0) 2065 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:47.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:44.864411+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:47.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:44.864508+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:47.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.529435+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e631: 8 total, 8 up, 7 in 2024-02-12T17:13:47.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.531217+0000 osd.2 (osd.2) 225 : cluster [DBG] 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 629'7955 2024-02-12T17:13:47.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.531233+0000 osd.2 (osd.2) 226 : cluster [DBG] 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8127 2024-02-12T17:13:47.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.532400+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 629'7972 2024-02-12T17:13:47.785 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.534354+0000 osd.3 (osd.3) 225 : cluster [INF] 2.d continuing backfill to osd.1 from (128'2,128'2] MIN to 128'2 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.534971+0000 osd.3 (osd.3) 226 : cluster [DBG] 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 629'8054 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.538312+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.e starting backfill to osd.1 from (0'0,0'0] MAX to 629'8046 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.542942+0000 osd.3 (osd.3) 227 : cluster [INF] 5.13 continuing backfill to osd.1 from (348'12,419'14] MIN to 419'14 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.542971+0000 osd.5 (osd.5) 312 : cluster [DBG] 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8001 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.556641+0000 osd.7 (osd.7) 308 : cluster [INF] 11.e continuing backfill to osd.3 from (608'3342,629'8046] 11:707e1c6b:::benchmark_data_smithi033_188107_object35338:head to 629'8046 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.558518+0000 osd.3 (osd.3) 228 : cluster [INF] 11.b continuing backfill to osd.5 from (608'3563,629'8054] MIN to 629'8054 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.563433+0000 osd.6 (osd.6) 175 : cluster [INF] 5.5 continuing backfill to osd.1 from (406'13,410'15] MIN to 410'15 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.563830+0000 osd.5 (osd.5) 313 : cluster [INF] 11.7 continuing backfill to osd.7 from (606'2678,629'8001] 11:e74718db:::benchmark_data_smithi033_188107_object26925:head to 629'8001 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.572607+0000 osd.2 (osd.2) 227 : cluster [INF] 11.6 continuing backfill to osd.5 from (606'2221,629'8127] 11:671a30a6:::benchmark_data_smithi033_188107_object84353:head to 629'8127 2024-02-12T17:13:47.786 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.572920+0000 osd.2 (osd.2) 228 : cluster [INF] 11.c continuing backfill to osd.7 from (606'1974,629'7955] 11:366e84f1:::benchmark_data_smithi033_188107_object41569:head to 629'7955 2024-02-12T17:13:47.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.616770+0000 mon.a (mon.0) 2063 : cluster [WRN] Health check update: Degraded data redundancy: 60379/375288 objects degraded (16.089%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:47.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.642258+0000 osd.7 (osd.7) 309 : cluster [DBG] 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 629'8045 2024-02-12T17:13:47.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.642893+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e632: 8 total, 8 up, 7 in 2024-02-12T17:13:47.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.643263+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 629'7888 2024-02-12T17:13:47.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.648750+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8121 2024-02-12T17:13:47.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.657487+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8078 2024-02-12T17:13:47.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:47.526828+0000 mon.a (mon.0) 2065 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:47.787 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:13:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw-foo-a[65833]: debug 2024-02-12T17:13:47.610+0000 7efdb0cdd700 -1 RGWWatcher::handle_error cookie 94508044033920 err (110) Connection timed out 2024-02-12T17:13:47.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:44.864411+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:47.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:44.864508+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:47.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.529435+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e631: 8 total, 8 up, 7 in 2024-02-12T17:13:47.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.531217+0000 osd.2 (osd.2) 225 : cluster [DBG] 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 629'7955 2024-02-12T17:13:47.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.531233+0000 osd.2 (osd.2) 226 : cluster [DBG] 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8127 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.532400+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 629'7972 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.534354+0000 osd.3 (osd.3) 225 : cluster [INF] 2.d continuing backfill to osd.1 from (128'2,128'2] MIN to 128'2 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.534971+0000 osd.3 (osd.3) 226 : cluster [DBG] 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 629'8054 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.538312+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.e starting backfill to osd.1 from (0'0,0'0] MAX to 629'8046 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.542942+0000 osd.3 (osd.3) 227 : cluster [INF] 5.13 continuing backfill to osd.1 from (348'12,419'14] MIN to 419'14 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.542971+0000 osd.5 (osd.5) 312 : cluster [DBG] 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8001 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.556641+0000 osd.7 (osd.7) 308 : cluster [INF] 11.e continuing backfill to osd.3 from (608'3342,629'8046] 11:707e1c6b:::benchmark_data_smithi033_188107_object35338:head to 629'8046 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.558518+0000 osd.3 (osd.3) 228 : cluster [INF] 11.b continuing backfill to osd.5 from (608'3563,629'8054] MIN to 629'8054 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.563433+0000 osd.6 (osd.6) 175 : cluster [INF] 5.5 continuing backfill to osd.1 from (406'13,410'15] MIN to 410'15 2024-02-12T17:13:47.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.563830+0000 osd.5 (osd.5) 313 : cluster [INF] 11.7 continuing backfill to osd.7 from (606'2678,629'8001] 11:e74718db:::benchmark_data_smithi033_188107_object26925:head to 629'8001 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.572607+0000 osd.2 (osd.2) 227 : cluster [INF] 11.6 continuing backfill to osd.5 from (606'2221,629'8127] 11:671a30a6:::benchmark_data_smithi033_188107_object84353:head to 629'8127 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.572920+0000 osd.2 (osd.2) 228 : cluster [INF] 11.c continuing backfill to osd.7 from (606'1974,629'7955] 11:366e84f1:::benchmark_data_smithi033_188107_object41569:head to 629'7955 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.616770+0000 mon.a (mon.0) 2063 : cluster [WRN] Health check update: Degraded data redundancy: 60379/375288 objects degraded (16.089%), 25 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.642258+0000 osd.7 (osd.7) 309 : cluster [DBG] 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 629'8045 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.642893+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e632: 8 total, 8 up, 7 in 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.643263+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 629'7888 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.648750+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8121 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.657487+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 629'8078 2024-02-12T17:13:47.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:47.526828+0000 mon.a (mon.0) 2065 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:48.298 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:48.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:48.411 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:48.466 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:48.496 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:48.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:48.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:46.996656+0000 mgr.y (mgr.24481) 909 : cluster [DBG] pgmap v1278: 125 pgs: 6 peering, 12 remapped+peering, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 21 active+clean+remapped, 14 active+undersized, 15 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 23 MiB/s wr, 817 op/s; 44627/372354 objects degraded (11.985%); 38823/372354 objects misplaced (10.426%); 53 MiB/s, 847 objects/s recovering 2024-02-12T17:13:48.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:47.654538+0000 mon.a (mon.0) 2066 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:48.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:47.668112+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e633: 8 total, 8 up, 7 in 2024-02-12T17:13:48.567 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:48.371972+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? v1:172.21.15.33:0/1177008327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:13:48.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:48.671 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:48.672 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:48.693 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:13:48.718 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:48.725 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:48.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:46.996656+0000 mgr.y (mgr.24481) 909 : cluster [DBG] pgmap v1278: 125 pgs: 6 peering, 12 remapped+peering, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 21 active+clean+remapped, 14 active+undersized, 15 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 23 MiB/s wr, 817 op/s; 44627/372354 objects degraded (11.985%); 38823/372354 objects misplaced (10.426%); 53 MiB/s, 847 objects/s recovering 2024-02-12T17:13:48.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:47.654538+0000 mon.a (mon.0) 2066 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:48.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:47.668112+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e633: 8 total, 8 up, 7 in 2024-02-12T17:13:48.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:48.371972+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? v1:172.21.15.33:0/1177008327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:13:48.832 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:48.905 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:46.996656+0000 mgr.y (mgr.24481) 909 : cluster [DBG] pgmap v1278: 125 pgs: 6 peering, 12 remapped+peering, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 21 active+clean+remapped, 14 active+undersized, 15 active+undersized+degraded, 51 active+clean; 7.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 23 MiB/s wr, 817 op/s; 44627/372354 objects degraded (11.985%); 38823/372354 objects misplaced (10.426%); 53 MiB/s, 847 objects/s recovering 2024-02-12T17:13:48.905 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:47.654538+0000 mon.a (mon.0) 2066 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:48.905 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:47.668112+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e633: 8 total, 8 up, 7 in 2024-02-12T17:13:48.905 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:48.371972+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? v1:172.21.15.33:0/1177008327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:13:48.906 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:49.015 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:49.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:49.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:49.195 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:49.582 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:49.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:49.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:49.768 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:49.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:48.666516+0000 mon.a (mon.0) 2069 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:49.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:48.673319+0000 mon.a (mon.0) 2070 : audit [INF] from='client.? v1:172.21.15.33:0/1177008327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:13:49.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:48.673396+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e634: 8 total, 8 up, 7 in 2024-02-12T17:13:49.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:48.997458+0000 mgr.y (mgr.24481) 910 : cluster [DBG] pgmap v1281: 125 pgs: 5 active+undersized+degraded+remapped+backfill_toofull, 6 peering, 12 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 8 active+clean+remapped, 9 active+undersized, 7 active+undersized+degraded, 74 active+clean; 7.4 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 42686/366285 objects degraded (11.654%); 37802/366285 objects misplaced (10.320%); 17 MiB/s, 272 objects/s recovering 2024-02-12T17:13:49.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:48.998057+0000 mon.c (mon.2) 293 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:13:49.864 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:48.999286+0000 mon.a (mon.0) 2072 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:13:49.868 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:48.666516+0000 mon.a (mon.0) 2069 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:49.868 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:48.673319+0000 mon.a (mon.0) 2070 : audit [INF] from='client.? v1:172.21.15.33:0/1177008327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:13:49.868 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:48.673396+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e634: 8 total, 8 up, 7 in 2024-02-12T17:13:49.868 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:48.997458+0000 mgr.y (mgr.24481) 910 : cluster [DBG] pgmap v1281: 125 pgs: 5 active+undersized+degraded+remapped+backfill_toofull, 6 peering, 12 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 8 active+clean+remapped, 9 active+undersized, 7 active+undersized+degraded, 74 active+clean; 7.4 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 42686/366285 objects degraded (11.654%); 37802/366285 objects misplaced (10.320%); 17 MiB/s, 272 objects/s recovering 2024-02-12T17:13:49.868 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:48.998057+0000 mon.c (mon.2) 293 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:13:49.868 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:48.999286+0000 mon.a (mon.0) 2072 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:13:49.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:48.666516+0000 mon.a (mon.0) 2069 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:13:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:48.673319+0000 mon.a (mon.0) 2070 : audit [INF] from='client.? v1:172.21.15.33:0/1177008327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:13:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:48.673396+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e634: 8 total, 8 up, 7 in 2024-02-12T17:13:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:48.997458+0000 mgr.y (mgr.24481) 910 : cluster [DBG] pgmap v1281: 125 pgs: 5 active+undersized+degraded+remapped+backfill_toofull, 6 peering, 12 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 8 active+clean+remapped, 9 active+undersized, 7 active+undersized+degraded, 74 active+clean; 7.4 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 42686/366285 objects degraded (11.654%); 37802/366285 objects misplaced (10.320%); 17 MiB/s, 272 objects/s recovering 2024-02-12T17:13:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:48.998057+0000 mon.c (mon.2) 293 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:13:49.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:48.999286+0000 mon.a (mon.0) 2072 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-12T17:13:50.892 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:13:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:50] "GET /metrics HTTP/1.1" 200 176572 "" "Prometheus/2.33.4" 2024-02-12T17:13:50.892 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:49.690030+0000 mon.a (mon.0) 2073 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T17:13:50.892 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:49.690092+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e635: 8 total, 8 up, 7 in 2024-02-12T17:13:50.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:49.690030+0000 mon.a (mon.0) 2073 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T17:13:50.893 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:49.690092+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e635: 8 total, 8 up, 7 in 2024-02-12T17:13:50.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:49.690030+0000 mon.a (mon.0) 2073 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-12T17:13:50.994 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:49.690092+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e635: 8 total, 8 up, 7 in 2024-02-12T17:13:51.250 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:51.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:51.366 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:51.426 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:51.454 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:51.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:51.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:51.604 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:51.606 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:13:51.629 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:51.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:51.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:51.768 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:51.789 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:51.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:51.894 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:51.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:50.697752+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e636: 8 total, 8 up, 7 in 2024-02-12T17:13:51.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:50.998160+0000 mgr.y (mgr.24481) 911 : cluster [DBG] pgmap v1284: 125 pgs: 5 active+undersized+degraded+remapped+backfill_toofull, 6 peering, 8 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+undersized+degraded, 101 active+clean; 7.4 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 505 op/s; 42653/366285 objects degraded (11.645%); 37789/366285 objects misplaced (10.317%); 15 MiB/s, 236 objects/s recovering 2024-02-12T17:13:51.971 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:51.628350+0000 mon.a (mon.0) 2076 : cluster [WRN] Health check update: Degraded data redundancy: 42686/366285 objects degraded (11.654%), 13 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:51.973 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:50.697752+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e636: 8 total, 8 up, 7 in 2024-02-12T17:13:51.973 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:50.998160+0000 mgr.y (mgr.24481) 911 : cluster [DBG] pgmap v1284: 125 pgs: 5 active+undersized+degraded+remapped+backfill_toofull, 6 peering, 8 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+undersized+degraded, 101 active+clean; 7.4 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 505 op/s; 42653/366285 objects degraded (11.645%); 37789/366285 objects misplaced (10.317%); 15 MiB/s, 236 objects/s recovering 2024-02-12T17:13:51.973 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:51.628350+0000 mon.a (mon.0) 2076 : cluster [WRN] Health check update: Degraded data redundancy: 42686/366285 objects degraded (11.654%), 13 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:51.981 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:51.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:50.697752+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e636: 8 total, 8 up, 7 in 2024-02-12T17:13:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:50.998160+0000 mgr.y (mgr.24481) 911 : cluster [DBG] pgmap v1284: 125 pgs: 5 active+undersized+degraded+remapped+backfill_toofull, 6 peering, 8 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+undersized+degraded, 101 active+clean; 7.4 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 505 op/s; 42653/366285 objects degraded (11.645%); 37789/366285 objects misplaced (10.317%); 15 MiB/s, 236 objects/s recovering 2024-02-12T17:13:51.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:51.628350+0000 mon.a (mon.0) 2076 : cluster [WRN] Health check update: Degraded data redundancy: 42686/366285 objects degraded (11.654%), 13 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-02-12T17:13:52.211 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:52.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:52.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:52.320 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:52.370 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:52.396 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:52.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:52.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:52.544 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:52.575 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:52.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:52.681 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:13:52.694 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:13:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:13:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.36023.0:133738 11.c e637/630 11:3bb9444f:::benchmark_data_smithi033_188107_object8048:head v 637'8258, mlcod=637'8256)", 2024-02-12T17:13:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:13:52.655621+0000", 2024-02-12T17:13:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0016892789999999999, 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001716579, 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:52.655621+0000", 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:52.655621+0000", 2024-02-12T17:13:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:52.655620+0000", 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2460000000000003e-06 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:52.655628+0000", 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2278e-05 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:52.655640+0000", 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7249999999999997e-06 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:52.655645+0000", 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6003999999999999e-05 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:52.655731+0000", 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5072e-05 2024-02-12T17:13:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:52.655756+0000", 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013478900000000001 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 1 2024-02-12T17:13:52.698 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:13:52.724 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:13:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:13:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:13:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:52.766 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:13:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:52.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:13:52.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:52.924 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:52.958 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:52.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:52.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:51.741346+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e637: 8 total, 8 up, 7 in 2024-02-12T17:13:52.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:52.549255+0000 mon.a (mon.0) 2078 : audit [DBG] from='client.? v1:172.21.15.33:0/2221977396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:52.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:52.655076+0000 mon.c (mon.2) 294 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:13:52.980 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:51.741346+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e637: 8 total, 8 up, 7 in 2024-02-12T17:13:52.980 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:52.549255+0000 mon.a (mon.0) 2078 : audit [DBG] from='client.? v1:172.21.15.33:0/2221977396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:52.980 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:52.655076+0000 mon.c (mon.2) 294 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:13:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:51.741346+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e637: 8 total, 8 up, 7 in 2024-02-12T17:13:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:52.549255+0000 mon.a (mon.0) 2078 : audit [DBG] from='client.? v1:172.21.15.33:0/2221977396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:13:52.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:52.655076+0000 mon.c (mon.2) 294 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:13:53.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:53.114 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:53.153 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:53.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:53.259 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:53.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:53.366 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:53.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:53.409 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:13:53.409 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:13:53.409 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:13:53.409 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:13:53.409 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:13:53.410 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:13:53.410 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:13:53.410 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:13:53.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:13:53.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:13:53.410 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:13:53.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:13:53.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:13:53.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:13:53.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:13:53.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:13:53.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:53.570 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:53.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:53.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:53.739 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:53.767 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:53.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:53.873 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:53.943 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:53.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:52.758875+0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e638: 8 total, 8 up, 7 in 2024-02-12T17:13:53.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:52.998807+0000 mgr.y (mgr.24481) 912 : cluster [DBG] pgmap v1287: 124 pgs: 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 110 active+clean; 7.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 427 op/s; 56947/361149 objects degraded (15.768%); 45169/361149 objects misplaced (12.507%); 1.6 MiB/s, 24 objects/s recovering 2024-02-12T17:13:53.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:52.999812+0000 mon.c (mon.2) 295 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:13:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:53.001441+0000 mon.a (mon.0) 2080 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:13:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:53.260930+0000 mon.c (mon.2) 296 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:13:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:53.271177+0000 mon.c (mon.2) 297 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:13:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:53.287502+0000 mon.a (mon.0) 2081 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:53.755676+0000 mon.a (mon.0) 2082 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T17:13:53.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:53.755738+0000 mon.a (mon.0) 2083 : cluster [DBG] osdmap e639: 8 total, 8 up, 7 in 2024-02-12T17:13:54.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:52.758875+0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e638: 8 total, 8 up, 7 in 2024-02-12T17:13:54.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:52.998807+0000 mgr.y (mgr.24481) 912 : cluster [DBG] pgmap v1287: 124 pgs: 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 110 active+clean; 7.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 427 op/s; 56947/361149 objects degraded (15.768%); 45169/361149 objects misplaced (12.507%); 1.6 MiB/s, 24 objects/s recovering 2024-02-12T17:13:54.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:52.999812+0000 mon.c (mon.2) 295 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:13:54.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:53.001441+0000 mon.a (mon.0) 2080 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:13:54.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:53.260930+0000 mon.c (mon.2) 296 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:13:54.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:53.271177+0000 mon.c (mon.2) 297 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:13:54.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:53.287502+0000 mon.a (mon.0) 2081 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:54.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:53.755676+0000 mon.a (mon.0) 2082 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T17:13:54.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:53.755738+0000 mon.a (mon.0) 2083 : cluster [DBG] osdmap e639: 8 total, 8 up, 7 in 2024-02-12T17:13:54.033 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:52.758875+0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e638: 8 total, 8 up, 7 in 2024-02-12T17:13:54.033 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:52.998807+0000 mgr.y (mgr.24481) 912 : cluster [DBG] pgmap v1287: 124 pgs: 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 110 active+clean; 7.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 427 op/s; 56947/361149 objects degraded (15.768%); 45169/361149 objects misplaced (12.507%); 1.6 MiB/s, 24 objects/s recovering 2024-02-12T17:13:54.033 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:52.999812+0000 mon.c (mon.2) 295 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:13:54.033 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:53.001441+0000 mon.a (mon.0) 2080 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-12T17:13:54.033 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:53.260930+0000 mon.c (mon.2) 296 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:13:54.033 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:53.271177+0000 mon.c (mon.2) 297 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:13:54.034 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:53.287502+0000 mon.a (mon.0) 2081 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:13:54.034 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:53.755676+0000 mon.a (mon.0) 2082 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-12T17:13:54.034 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:53.755738+0000 mon.a (mon.0) 2083 : cluster [DBG] osdmap e639: 8 total, 8 up, 7 in 2024-02-12T17:13:54.156 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:54.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:54.261 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:54.586 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:54.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:54.691 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:54.738 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:54.768 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:54.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:54.875 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:13:54.927 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:54.952 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:13:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:54.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:13:54.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:13:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:13:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:53.780066+0000 mon.a (mon.0) 2084 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:54.765134+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e640: 8 total, 8 up, 7 in 2024-02-12T17:13:55.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:53.780066+0000 mon.a (mon.0) 2084 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:55.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:54.765134+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e640: 8 total, 8 up, 7 in 2024-02-12T17:13:55.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:53.780066+0000 mon.a (mon.0) 2084 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:13:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:54.765134+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e640: 8 total, 8 up, 7 in 2024-02-12T17:13:55.059 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:13:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:55.127 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:13:55.131 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T17:13:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:55.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:13:55.157 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:13:55.233 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:13:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:55.450 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:13:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:55.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:13:55.555 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:55.673 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:55.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:55.783 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:136419 11.b 11:dc2f4e0f:::benchmark_data_smithi033_188107_object10729:head [delete] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:13:55.855492+0000", 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0046339470000000002, 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0046552579999999998, 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:55.864 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 136419 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855492+0000", 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855492+0000", 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855492+0000", 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.711e-06 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855494+0000", 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8540000000000001e-06 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855497+0000", 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.249e-06 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855499+0000", 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003555613 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859055+0000", 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012923700000000001 2024-02-12T17:13:55.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859184+0000", 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014585999999999999 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 1,4,5", 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859330+0000", 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038372189999999998 2024-02-12T17:13:55.868 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.871 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:55.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.871 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:136420 11.2 11.640caf32 (undecoded) ondisk+write+known_if_redirected e641)", 2024-02-12T17:13:55.871 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:13:55.859033+0000", 2024-02-12T17:13:55.871 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0010936990000000001, 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001168014, 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 136420 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:55.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859033+0000", 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859033+0000", 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999976 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859030+0000", 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0246e-05 2024-02-12T17:13:55.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859040+0000", 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5959999999999993e-06 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859050+0000", 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1879999999999999e-06 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:55.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859056+0000", 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3448999999999999e-05 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:136422 11.8 11.9938e448 (undecoded) ondisk+write+known_if_redirected e641)", 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:13:55.859062+0000", 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0010639759999999999, 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011739039999999999, 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 136422 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859062+0000", 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859062+0000", 2024-02-12T17:13:55.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859062+0000", 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.253e-06 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859065+0000", 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7359999999999998e-06 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:55.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859071+0000", 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0520000000000003e-06 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859075+0000", 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2368000000000001e-05 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.36023.0:136415 11.1 e641/597 11:88005180:::benchmark_data_smithi033_188107_object10725:head v 641'8463, mlcod=641'8460)", 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:13:55.855385+0000", 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0047410029999999997, 2024-02-12T17:13:55.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048880870000000002, 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855385+0000", 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855385+0000", 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855384+0000", 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.234e-05 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855396+0000", 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7564999999999999e-05 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855414+0000", 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8650000000000001e-06 2024-02-12T17:13:55.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855422+0000", 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2354000000000001e-05 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855464+0000", 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1979000000000001e-05 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855486+0000", 2024-02-12T17:13:55.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010049599999999999 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:136412 11.8 11:163e83b6:::benchmark_data_smithi033_188107_object10722:head [delete] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:13:55.855446+0000", 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0046805650000000002, 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048687540000000003, 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:55.882 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 136412 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855446+0000", 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855446+0000", 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999986 2024-02-12T17:13:55.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855444+0000", 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3620000000000002e-06 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855451+0000", 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1739999999999997e-06 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:55.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855457+0000", 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8979999999999998e-06 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855461+0000", 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013961799999999999 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855600+0000", 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1218999999999996e-05 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855681+0000", 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6099000000000004e-05 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855768+0000", 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032174500000000002 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:136413 11.a 11:5644a4fb:::benchmark_data_smithi033_188107_object10723:head [delete] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:13:55.855468+0000", 2024-02-12T17:13:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0046586049999999997, 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049197659999999999, 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 136413 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855468+0000", 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855468+0000", 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855467+0000", 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.818e-06 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855470+0000", 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9459999999999998e-06 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855473+0000", 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2239999999999998e-06 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855475+0000", 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022198460000000001 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.857695+0000", 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012281799999999999 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.857818+0000", 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013421700000000001 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 5,6", 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.857952+0000", 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024843949999999999 2024-02-12T17:13:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:136416 11.2 11:402c4995:::benchmark_data_smithi033_188107_object10726:head [delete] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:13:55.855484+0000", 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0046428199999999998, 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050004910000000001, 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:13:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 136416 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855484+0000", 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855484+0000", 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855483+0000", 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9829999999999999e-06 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855485+0000", 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9220000000000001e-06 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855488+0000", 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0959999999999999e-06 2024-02-12T17:13:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.855490+0000", 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041580260000000004 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859648+0000", 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010539000000000001 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859754+0000", 2024-02-12T17:13:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000109638 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,2", 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:13:55.859863+0000", 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043796629999999998 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 7 2024-02-12T17:13:55.895 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:13:55.896 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:56.022 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:56.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:56.134 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:13:56.158 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:13:56.174 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:54.941380+0000 mgr.y (mgr.24481) 913 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:56.174 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:54.999431+0000 mgr.y (mgr.24481) 914 : cluster [DBG] pgmap v1290: 124 pgs: 3 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 55070/352914 objects degraded (15.604%); 43758/352914 objects misplaced (12.399%); 11 MiB/s, 170 objects/s recovering 2024-02-12T17:13:56.174 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:55.778508+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2024-02-12T17:13:56.176 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:56.209 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:13:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:56.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:13:56.220 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:54.941380+0000 mgr.y (mgr.24481) 913 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:56.220 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:54.999431+0000 mgr.y (mgr.24481) 914 : cluster [DBG] pgmap v1290: 124 pgs: 3 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 55070/352914 objects degraded (15.604%); 43758/352914 objects misplaced (12.399%); 11 MiB/s, 170 objects/s recovering 2024-02-12T17:13:56.220 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:55.778508+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2024-02-12T17:13:56.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:54.941380+0000 mgr.y (mgr.24481) 913 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:13:56.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:54.999431+0000 mgr.y (mgr.24481) 914 : cluster [DBG] pgmap v1290: 124 pgs: 3 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 55070/352914 objects degraded (15.604%); 43758/352914 objects misplaced (12.399%); 11 MiB/s, 170 objects/s recovering 2024-02-12T17:13:56.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:55.778508+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2024-02-12T17:13:56.318 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:13:56.378 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:57.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:55.786675+0000 osd.5 (osd.5) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:57.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:55.786715+0000 osd.5 (osd.5) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:57.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:56.630661+0000 mon.a (mon.0) 2087 : cluster [WRN] Health check update: Degraded data redundancy: 55070/352914 objects degraded (15.604%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:13:57.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:56.638905+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e642: 8 total, 8 up, 7 in 2024-02-12T17:13:57.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:55.786675+0000 osd.5 (osd.5) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:57.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:55.786715+0000 osd.5 (osd.5) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:57.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:56.630661+0000 mon.a (mon.0) 2087 : cluster [WRN] Health check update: Degraded data redundancy: 55070/352914 objects degraded (15.604%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:13:57.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:56.638905+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e642: 8 total, 8 up, 7 in 2024-02-12T17:13:57.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:55.786675+0000 osd.5 (osd.5) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:13:57.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:55.786715+0000 osd.5 (osd.5) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:13:57.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:56.630661+0000 mon.a (mon.0) 2087 : cluster [WRN] Health check update: Degraded data redundancy: 55070/352914 objects degraded (15.604%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:13:57.281 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:56.638905+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e642: 8 total, 8 up, 7 in 2024-02-12T17:13:58.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:57.003740+0000 mgr.y (mgr.24481) 915 : cluster [DBG] pgmap v1293: 123 pgs: 2 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 54161/348099 objects degraded (15.559%); 43449/348099 objects misplaced (12.482%); 11 MiB/s, 172 objects/s recovering 2024-02-12T17:13:58.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:57.004250+0000 mon.c (mon.2) 298 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:13:58.244 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:57.005699+0000 mon.a (mon.0) 2089 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:13:58.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:57.003740+0000 mgr.y (mgr.24481) 915 : cluster [DBG] pgmap v1293: 123 pgs: 2 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 54161/348099 objects degraded (15.559%); 43449/348099 objects misplaced (12.482%); 11 MiB/s, 172 objects/s recovering 2024-02-12T17:13:58.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:57.004250+0000 mon.c (mon.2) 298 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:13:58.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:57.005699+0000 mon.a (mon.0) 2089 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:13:58.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:57.003740+0000 mgr.y (mgr.24481) 915 : cluster [DBG] pgmap v1293: 123 pgs: 2 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 54161/348099 objects degraded (15.559%); 43449/348099 objects misplaced (12.482%); 11 MiB/s, 172 objects/s recovering 2024-02-12T17:13:58.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:57.004250+0000 mon.c (mon.2) 298 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:13:58.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:57.005699+0000 mon.a (mon.0) 2089 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T17:13:59.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:13:57.861276+0000 mon.a (mon.0) 2090 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:13:59.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:57.861341+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e643: 8 total, 8 up, 7 in 2024-02-12T17:13:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:13:57.861276+0000 mon.a (mon.0) 2090 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:13:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:57.861341+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e643: 8 total, 8 up, 7 in 2024-02-12T17:13:59.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:13:57.861276+0000 mon.a (mon.0) 2090 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T17:13:59.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:57.861341+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e643: 8 total, 8 up, 7 in 2024-02-12T17:13:59.291 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:13:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:59.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:13:59.394 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:13:59.449 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:59.474 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:13:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:13:59.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:13:59.579 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:13:59.659 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:13:59.952 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:58.929386+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e644: 8 total, 8 up, 7 in 2024-02-12T17:13:59.953 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:13:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:59.007029+0000 mgr.y (mgr.24481) 916 : cluster [DBG] pgmap v1296: 123 pgs: 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 110 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 52284/339993 objects degraded (15.378%); 42052/339993 objects misplaced (12.368%); 12 MiB/s, 187 objects/s recovering 2024-02-12T17:14:00.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:58.929386+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e644: 8 total, 8 up, 7 in 2024-02-12T17:14:00.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:13:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:59.007029+0000 mgr.y (mgr.24481) 916 : cluster [DBG] pgmap v1296: 123 pgs: 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 110 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 52284/339993 objects degraded (15.378%); 42052/339993 objects misplaced (12.368%); 12 MiB/s, 187 objects/s recovering 2024-02-12T17:14:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:58.929386+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e644: 8 total, 8 up, 7 in 2024-02-12T17:14:00.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:13:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:59.007029+0000 mgr.y (mgr.24481) 916 : cluster [DBG] pgmap v1296: 123 pgs: 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 110 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 52284/339993 objects degraded (15.378%); 42052/339993 objects misplaced (12.368%); 12 MiB/s, 187 objects/s recovering 2024-02-12T17:14:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:13:59.949952+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e645: 8 total, 8 up, 7 in 2024-02-12T17:14:01.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:14:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:00] "GET /metrics HTTP/1.1" 200 176572 "" "Prometheus/2.33.4" 2024-02-12T17:14:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:13:59.949952+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e645: 8 total, 8 up, 7 in 2024-02-12T17:14:01.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:13:59.949952+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e645: 8 total, 8 up, 7 in 2024-02-12T17:14:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:01.422 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T17:14:01.739 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:01.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:01.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:14:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:01.932 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:14:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:01.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:14:02.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:02.081 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:02.116 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:02.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:02.224 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:00.967476+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e646: 8 total, 8 up, 7 in 2024-02-12T17:14:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:01.010266+0000 mgr.y (mgr.24481) 917 : cluster [DBG] pgmap v1299: 122 pgs: 1 clean+premerge+peered, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 52284/339993 objects degraded (15.378%); 42052/339993 objects misplaced (12.368%); 10 MiB/s, 159 objects/s recovering 2024-02-12T17:14:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:01.010642+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:14:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:01.012284+0000 mon.a (mon.0) 2095 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:14:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:01.633209+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:01.633246+0000 mon.a (mon.0) 2097 : cluster [WRN] Health check update: Degraded data redundancy: 52284/339993 objects degraded (15.378%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:02.273 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:00.967476+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e646: 8 total, 8 up, 7 in 2024-02-12T17:14:02.274 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:01.010266+0000 mgr.y (mgr.24481) 917 : cluster [DBG] pgmap v1299: 122 pgs: 1 clean+premerge+peered, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 52284/339993 objects degraded (15.378%); 42052/339993 objects misplaced (12.368%); 10 MiB/s, 159 objects/s recovering 2024-02-12T17:14:02.274 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:01.010642+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:14:02.274 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:01.012284+0000 mon.a (mon.0) 2095 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:14:02.274 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:01.633209+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:02.274 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:01.633246+0000 mon.a (mon.0) 2097 : cluster [WRN] Health check update: Degraded data redundancy: 52284/339993 objects degraded (15.378%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:02.274 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:00.967476+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e646: 8 total, 8 up, 7 in 2024-02-12T17:14:02.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:01.010266+0000 mgr.y (mgr.24481) 917 : cluster [DBG] pgmap v1299: 122 pgs: 1 clean+premerge+peered, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 52284/339993 objects degraded (15.378%); 42052/339993 objects misplaced (12.368%); 10 MiB/s, 159 objects/s recovering 2024-02-12T17:14:02.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:01.010642+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:14:02.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:01.012284+0000 mon.a (mon.0) 2095 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-12T17:14:02.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:01.633209+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:02.275 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:01.633246+0000 mon.a (mon.0) 2097 : cluster [WRN] Health check update: Degraded data redundancy: 52284/339993 objects degraded (15.378%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:02.279 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:02.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:14:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:14:03.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:01.993324+0000 mon.a (mon.0) 2098 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T17:14:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:01.993449+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e647: 8 total, 8 up, 7 in 2024-02-12T17:14:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:02.791542+0000 mon.c (mon.2) 300 : audit [INF] from='client.? v1:172.21.15.33:0/3888666317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:02.791840+0000 mon.a (mon.0) 2100 : audit [INF] from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:03.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:01.993324+0000 mon.a (mon.0) 2098 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T17:14:03.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:01.993449+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e647: 8 total, 8 up, 7 in 2024-02-12T17:14:03.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:02.791542+0000 mon.c (mon.2) 300 : audit [INF] from='client.? v1:172.21.15.33:0/3888666317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:03.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:02.791840+0000 mon.a (mon.0) 2100 : audit [INF] from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:01.993324+0000 mon.a (mon.0) 2098 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-12T17:14:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:01.993449+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e647: 8 total, 8 up, 7 in 2024-02-12T17:14:03.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:02.791542+0000 mon.c (mon.2) 300 : audit [INF] from='client.? v1:172.21.15.33:0/3888666317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:03.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:02.791840+0000 mon.a (mon.0) 2100 : audit [INF] from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:04.159 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:14:04.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:02.993663+0000 mon.a (mon.0) 2101 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:14:04.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:03.013197+0000 mgr.y (mgr.24481) 918 : cluster [DBG] pgmap v1301: 122 pgs: 1 clean+premerge+peered, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 673 op/s; 51337/334959 objects degraded (15.326%); 41715/334959 objects misplaced (12.454%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T17:14:04.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:03.021943+0000 mon.a (mon.0) 2102 : audit [INF] from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:14:04.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:03.022000+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2024-02-12T17:14:04.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:02.993663+0000 mon.a (mon.0) 2101 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:14:04.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:03.013197+0000 mgr.y (mgr.24481) 918 : cluster [DBG] pgmap v1301: 122 pgs: 1 clean+premerge+peered, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 673 op/s; 51337/334959 objects degraded (15.326%); 41715/334959 objects misplaced (12.454%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T17:14:04.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:03.021943+0000 mon.a (mon.0) 2102 : audit [INF] from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:14:04.281 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:03.022000+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2024-02-12T17:14:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:02.993663+0000 mon.a (mon.0) 2101 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:14:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:03.013197+0000 mgr.y (mgr.24481) 918 : cluster [DBG] pgmap v1301: 122 pgs: 1 clean+premerge+peered, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 108 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 673 op/s; 51337/334959 objects degraded (15.326%); 41715/334959 objects misplaced (12.454%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T17:14:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:03.021943+0000 mon.a (mon.0) 2102 : audit [INF] from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:14:04.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:03.022000+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2024-02-12T17:14:04.792 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:04.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:04.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:14:04.951 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:04.976 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:14:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:04.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:14:05.071 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:04.050949+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e649: 8 total, 8 up, 7 in 2024-02-12T17:14:05.071 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:04.050949+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e649: 8 total, 8 up, 7 in 2024-02-12T17:14:05.071 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:05.018213+0000 mon.c (mon.2) 301 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.072 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:05.018368+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.072 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:05.019981+0000 mon.a (mon.0) 2105 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.072 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:05.020163+0000 mon.a (mon.0) 2106 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.088 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:05.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:14:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:14:05.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:04.050949+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e649: 8 total, 8 up, 7 in 2024-02-12T17:14:05.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:05.018213+0000 mon.c (mon.2) 301 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:05.018368+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:05.019981+0000 mon.a (mon.0) 2105 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:05.020163+0000 mon.a (mon.0) 2106 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.482 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:05.018213+0000 mon.c (mon.2) 301 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:05.018368+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:05.019981+0000 mon.a (mon.0) 2105 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:05.020163+0000 mon.a (mon.0) 2106 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]: dispatch 2024-02-12T17:14:05.483 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:05.534 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:05.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:05.642 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:05.708 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:05.932 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:05.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:06.041 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:06.104 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:06.208 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:06.211 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:04.954539+0000 mgr.y (mgr.24481) 919 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:06.211 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:05.017703+0000 mgr.y (mgr.24481) 920 : cluster [DBG] pgmap v1304: 121 pgs: 1 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 107 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 49540/326877 objects degraded (15.156%); 40412/326877 objects misplaced (12.363%); 11 MiB/s, 181 objects/s recovering 2024-02-12T17:14:06.211 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:05.072049+0000 mon.a (mon.0) 2107 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]': finished 2024-02-12T17:14:06.211 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:05.072102+0000 mon.a (mon.0) 2108 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]': finished 2024-02-12T17:14:06.211 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:05.072145+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e650: 8 total, 8 up, 7 in 2024-02-12T17:14:06.212 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:04.954539+0000 mgr.y (mgr.24481) 919 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:06.212 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:05.017703+0000 mgr.y (mgr.24481) 920 : cluster [DBG] pgmap v1304: 121 pgs: 1 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 107 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 49540/326877 objects degraded (15.156%); 40412/326877 objects misplaced (12.363%); 11 MiB/s, 181 objects/s recovering 2024-02-12T17:14:06.212 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:05.072049+0000 mon.a (mon.0) 2107 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]': finished 2024-02-12T17:14:06.212 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:05.072102+0000 mon.a (mon.0) 2108 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]': finished 2024-02-12T17:14:06.212 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:05.072145+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e650: 8 total, 8 up, 7 in 2024-02-12T17:14:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:06.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:06.326 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:06.384 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:04.954539+0000 mgr.y (mgr.24481) 919 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:06.384 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:05.017703+0000 mgr.y (mgr.24481) 920 : cluster [DBG] pgmap v1304: 121 pgs: 1 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 107 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 49540/326877 objects degraded (15.156%); 40412/326877 objects misplaced (12.363%); 11 MiB/s, 181 objects/s recovering 2024-02-12T17:14:06.384 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:05.072049+0000 mon.a (mon.0) 2107 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.8"}]': finished 2024-02-12T17:14:06.384 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:05.072102+0000 mon.a (mon.0) 2108 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.8"}]': finished 2024-02-12T17:14:06.384 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:05.072145+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e650: 8 total, 8 up, 7 in 2024-02-12T17:14:06.386 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:06.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:06.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:06.566 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:06.593 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:06.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:06.701 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:14:06.727 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:06.747 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:14:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:06.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:14:06.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:06.901 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:06.923 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:06.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:07.029 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:14:07.072 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:07.109 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:14:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:07.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:14:07.214 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:07.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:06.640559+0000 mon.a (mon.0) 2110 : cluster [WRN] Health check update: Degraded data redundancy: 49540/326877 objects degraded (15.156%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:07.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:06.936821+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.33:0/1289537509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:14:07.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:06.938234+0000 mon.a (mon.0) 2111 : audit [INF] from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:14:07.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:07.018849+0000 mon.c (mon.2) 303 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:14:07.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:07.020990+0000 mon.a (mon.0) 2112 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:14:07.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:06.640559+0000 mon.a (mon.0) 2110 : cluster [WRN] Health check update: Degraded data redundancy: 49540/326877 objects degraded (15.156%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:07.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:06.936821+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.33:0/1289537509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:14:07.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:06.938234+0000 mon.a (mon.0) 2111 : audit [INF] from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:14:07.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:07.018849+0000 mon.c (mon.2) 303 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:14:07.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:07.020990+0000 mon.a (mon.0) 2112 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:14:07.269 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:07.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:06.640559+0000 mon.a (mon.0) 2110 : cluster [WRN] Health check update: Degraded data redundancy: 49540/326877 objects degraded (15.156%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:07.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:06.936821+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.33:0/1289537509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:14:07.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:06.938234+0000 mon.a (mon.0) 2111 : audit [INF] from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:14:07.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:07.018849+0000 mon.c (mon.2) 303 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:14:07.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:07.020990+0000 mon.a (mon.0) 2112 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-12T17:14:07.720 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T17:14:07.952 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:14:08.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:07.018483+0000 mgr.y (mgr.24481) 921 : cluster [DBG] pgmap v1306: 121 pgs: 1 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 107 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1020 B/s rd, 0 B/s wr, 869 op/s; 48612/321882 objects degraded (15.102%); 40074/321882 objects misplaced (12.450%); 9.1 MiB/s, 146 objects/s recovering 2024-02-12T17:14:08.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:07.681905+0000 mon.a (mon.0) 2113 : audit [INF] from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:14:08.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:07.681971+0000 mon.a (mon.0) 2114 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T17:14:08.268 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:07.682017+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e651: 8 total, 8 up, 7 in 2024-02-12T17:14:08.268 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:07.018483+0000 mgr.y (mgr.24481) 921 : cluster [DBG] pgmap v1306: 121 pgs: 1 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 107 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1020 B/s rd, 0 B/s wr, 869 op/s; 48612/321882 objects degraded (15.102%); 40074/321882 objects misplaced (12.450%); 9.1 MiB/s, 146 objects/s recovering 2024-02-12T17:14:08.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:07.681905+0000 mon.a (mon.0) 2113 : audit [INF] from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:14:08.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:07.681971+0000 mon.a (mon.0) 2114 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T17:14:08.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:07.682017+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e651: 8 total, 8 up, 7 in 2024-02-12T17:14:08.489 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:07.018483+0000 mgr.y (mgr.24481) 921 : cluster [DBG] pgmap v1306: 121 pgs: 1 peering, 6 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 107 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1020 B/s rd, 0 B/s wr, 869 op/s; 48612/321882 objects degraded (15.102%); 40074/321882 objects misplaced (12.450%); 9.1 MiB/s, 146 objects/s recovering 2024-02-12T17:14:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:07.681905+0000 mon.a (mon.0) 2113 : audit [INF] from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:14:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:07.681971+0000 mon.a (mon.0) 2114 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-12T17:14:08.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:07.682017+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e651: 8 total, 8 up, 7 in 2024-02-12T17:14:09.499 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:09.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:09.609 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:09.646 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:09.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.701475+0000 osd.7 (osd.7) 310 : cluster [DBG] 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 650'9192 2024-02-12T17:14:09.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.704444+0000 osd.3 (osd.3) 231 : cluster [DBG] 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9220 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.704913+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.707904+0000 osd.5 (osd.5) 316 : cluster [DBG] 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9138 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.708657+0000 osd.3 (osd.3) 232 : cluster [DBG] 5.13 starting backfill to osd.0 from (0'0,0'0] MAX to 419'14 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.708741+0000 osd.3 (osd.3) 233 : cluster [INF] 5.13 continuing backfill to osd.1 from (348'12,419'14] MIN to 419'14 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.709528+0000 osd.3 (osd.3) 234 : cluster [DBG] 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 650'9193 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.712762+0000 osd.2 (osd.2) 229 : cluster [DBG] 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9254 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.714848+0000 osd.7 (osd.7) 311 : cluster [INF] 11.e continuing backfill to osd.1 from (608'4042,650'9192] MIN to 650'9192 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.719518+0000 osd.5 (osd.5) 317 : cluster [INF] 11.7 continuing backfill to osd.1 from (608'3678,650'9138] MIN to 650'9138 2024-02-12T17:14:09.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.732845+0000 osd.3 (osd.3) 235 : cluster [INF] 11.b continuing backfill to osd.1 from (610'4463,650'9193] 11:d0777a54:::benchmark_data_smithi033_188107_object107623:head to 650'9193 2024-02-12T17:14:09.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:08.734748+0000 osd.2 (osd.2) 230 : cluster [INF] 11.6 continuing backfill to osd.1 from (608'3121,650'9254] MIN to 650'9254 2024-02-12T17:14:09.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.019003+0000 mgr.y (mgr.24481) 922 : cluster [DBG] pgmap v1309: 121 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 clean+premerge+peered, 3 remapped+peering, 8 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 97 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 770 op/s; 39572/315405 objects degraded (12.546%); 25889/315405 objects misplaced (8.208%); 17 MiB/s, 274 objects/s recovering 2024-02-12T17:14:09.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.701475+0000 osd.7 (osd.7) 310 : cluster [DBG] 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 650'9192 2024-02-12T17:14:09.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.704444+0000 osd.3 (osd.3) 231 : cluster [DBG] 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9220 2024-02-12T17:14:09.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.704913+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2024-02-12T17:14:09.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.707904+0000 osd.5 (osd.5) 316 : cluster [DBG] 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9138 2024-02-12T17:14:09.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.708657+0000 osd.3 (osd.3) 232 : cluster [DBG] 5.13 starting backfill to osd.0 from (0'0,0'0] MAX to 419'14 2024-02-12T17:14:09.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.708741+0000 osd.3 (osd.3) 233 : cluster [INF] 5.13 continuing backfill to osd.1 from (348'12,419'14] MIN to 419'14 2024-02-12T17:14:09.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.709528+0000 osd.3 (osd.3) 234 : cluster [DBG] 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 650'9193 2024-02-12T17:14:09.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.712762+0000 osd.2 (osd.2) 229 : cluster [DBG] 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9254 2024-02-12T17:14:09.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.714848+0000 osd.7 (osd.7) 311 : cluster [INF] 11.e continuing backfill to osd.1 from (608'4042,650'9192] MIN to 650'9192 2024-02-12T17:14:09.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.719518+0000 osd.5 (osd.5) 317 : cluster [INF] 11.7 continuing backfill to osd.1 from (608'3678,650'9138] MIN to 650'9138 2024-02-12T17:14:09.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.732845+0000 osd.3 (osd.3) 235 : cluster [INF] 11.b continuing backfill to osd.1 from (610'4463,650'9193] 11:d0777a54:::benchmark_data_smithi033_188107_object107623:head to 650'9193 2024-02-12T17:14:09.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:08.734748+0000 osd.2 (osd.2) 230 : cluster [INF] 11.6 continuing backfill to osd.1 from (608'3121,650'9254] MIN to 650'9254 2024-02-12T17:14:09.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.019003+0000 mgr.y (mgr.24481) 922 : cluster [DBG] pgmap v1309: 121 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 clean+premerge+peered, 3 remapped+peering, 8 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 97 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 770 op/s; 39572/315405 objects degraded (12.546%); 25889/315405 objects misplaced (8.208%); 17 MiB/s, 274 objects/s recovering 2024-02-12T17:14:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.701475+0000 osd.7 (osd.7) 310 : cluster [DBG] 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 650'9192 2024-02-12T17:14:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.704444+0000 osd.3 (osd.3) 231 : cluster [DBG] 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9220 2024-02-12T17:14:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.704913+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2024-02-12T17:14:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.707904+0000 osd.5 (osd.5) 316 : cluster [DBG] 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9138 2024-02-12T17:14:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.708657+0000 osd.3 (osd.3) 232 : cluster [DBG] 5.13 starting backfill to osd.0 from (0'0,0'0] MAX to 419'14 2024-02-12T17:14:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.708741+0000 osd.3 (osd.3) 233 : cluster [INF] 5.13 continuing backfill to osd.1 from (348'12,419'14] MIN to 419'14 2024-02-12T17:14:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.709528+0000 osd.3 (osd.3) 234 : cluster [DBG] 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 650'9193 2024-02-12T17:14:09.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.712762+0000 osd.2 (osd.2) 229 : cluster [DBG] 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9254 2024-02-12T17:14:09.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.714848+0000 osd.7 (osd.7) 311 : cluster [INF] 11.e continuing backfill to osd.1 from (608'4042,650'9192] MIN to 650'9192 2024-02-12T17:14:09.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.719518+0000 osd.5 (osd.5) 317 : cluster [INF] 11.7 continuing backfill to osd.1 from (608'3678,650'9138] MIN to 650'9138 2024-02-12T17:14:09.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.732845+0000 osd.3 (osd.3) 235 : cluster [INF] 11.b continuing backfill to osd.1 from (610'4463,650'9193] 11:d0777a54:::benchmark_data_smithi033_188107_object107623:head to 650'9193 2024-02-12T17:14:09.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:08.734748+0000 osd.2 (osd.2) 230 : cluster [INF] 11.6 continuing backfill to osd.1 from (608'3121,650'9254] MIN to 650'9254 2024-02-12T17:14:09.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.019003+0000 mgr.y (mgr.24481) 922 : cluster [DBG] pgmap v1309: 121 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 clean+premerge+peered, 3 remapped+peering, 8 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 97 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 770 op/s; 39572/315405 objects degraded (12.546%); 25889/315405 objects misplaced (8.208%); 17 MiB/s, 274 objects/s recovering 2024-02-12T17:14:10.796 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:10.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:10.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:10.906 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:11.012 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:10] "GET /metrics HTTP/1.1" 200 176825 "" "Prometheus/2.33.4" 2024-02-12T17:14:11.013 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.699722+0000 mon.a (mon.0) 2117 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:11.013 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.713323+0000 osd.7 (osd.7) 312 : cluster [DBG] 5.10 starting backfill to osd.0 from (0'0,0'0] MAX to 430'18 2024-02-12T17:14:11.013 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.714340+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e653: 8 total, 8 up, 7 in 2024-02-12T17:14:11.013 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.715011+0000 osd.6 (osd.6) 177 : cluster [DBG] 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9123 2024-02-12T17:14:11.013 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.717837+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.f starting backfill to osd.0 from (0'0,0'0] MAX to 91'4 2024-02-12T17:14:11.013 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.717864+0000 osd.1 (osd.1) 3 : cluster [DBG] 5.5 starting backfill to osd.0 from (0'0,0'0] MAX to 410'15 2024-02-12T17:14:11.013 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.719133+0000 osd.5 (osd.5) 319 : cluster [DBG] 5.15 starting backfill to osd.0 from (0'0,0'0] MAX to 632'22 2024-02-12T17:14:11.014 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.726490+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 632'30 2024-02-12T17:14:11.014 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:09.733657+0000 osd.3 (osd.3) 237 : cluster [DBG] 5.3 starting backfill to osd.0 from (0'0,0'0] MAX to 519'21 2024-02-12T17:14:11.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.699722+0000 mon.a (mon.0) 2117 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:11.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.713323+0000 osd.7 (osd.7) 312 : cluster [DBG] 5.10 starting backfill to osd.0 from (0'0,0'0] MAX to 430'18 2024-02-12T17:14:11.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.714340+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e653: 8 total, 8 up, 7 in 2024-02-12T17:14:11.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.715011+0000 osd.6 (osd.6) 177 : cluster [DBG] 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9123 2024-02-12T17:14:11.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.717837+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.f starting backfill to osd.0 from (0'0,0'0] MAX to 91'4 2024-02-12T17:14:11.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.717864+0000 osd.1 (osd.1) 3 : cluster [DBG] 5.5 starting backfill to osd.0 from (0'0,0'0] MAX to 410'15 2024-02-12T17:14:11.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.719133+0000 osd.5 (osd.5) 319 : cluster [DBG] 5.15 starting backfill to osd.0 from (0'0,0'0] MAX to 632'22 2024-02-12T17:14:11.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.726490+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 632'30 2024-02-12T17:14:11.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:09.733657+0000 osd.3 (osd.3) 237 : cluster [DBG] 5.3 starting backfill to osd.0 from (0'0,0'0] MAX to 519'21 2024-02-12T17:14:11.015 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:11.101 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:11.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:11.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:14:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.699722+0000 mon.a (mon.0) 2117 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:11.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.713323+0000 osd.7 (osd.7) 312 : cluster [DBG] 5.10 starting backfill to osd.0 from (0'0,0'0] MAX to 430'18 2024-02-12T17:14:11.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.714340+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e653: 8 total, 8 up, 7 in 2024-02-12T17:14:11.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.715011+0000 osd.6 (osd.6) 177 : cluster [DBG] 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 650'9123 2024-02-12T17:14:11.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.717837+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.f starting backfill to osd.0 from (0'0,0'0] MAX to 91'4 2024-02-12T17:14:11.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.717864+0000 osd.1 (osd.1) 3 : cluster [DBG] 5.5 starting backfill to osd.0 from (0'0,0'0] MAX to 410'15 2024-02-12T17:14:11.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.719133+0000 osd.5 (osd.5) 319 : cluster [DBG] 5.15 starting backfill to osd.0 from (0'0,0'0] MAX to 632'22 2024-02-12T17:14:11.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.726490+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 632'30 2024-02-12T17:14:11.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:09.733657+0000 osd.3 (osd.3) 237 : cluster [DBG] 5.3 starting backfill to osd.0 from (0'0,0'0] MAX to 519'21 2024-02-12T17:14:11.252 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:11.281 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:14:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:11.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:14:11.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:11.430 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:11.460 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:11.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:11.566 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:11.588 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:14:11.588 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:14:11.588 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:14:11.588 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:14:11.588 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:14:11.657 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:11.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:10.730215+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e654: 8 total, 8 up, 7 in 2024-02-12T17:14:11.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:11.021795+0000 mgr.y (mgr.24481) 923 : cluster [DBG] pgmap v1312: 120 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering, 1 clean+premerge+peered, 4 remapped+peering, 7 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 95 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 539 op/s; 39572/315405 objects degraded (12.546%); 25900/315405 objects misplaced (8.212%); 20 MiB/s, 319 objects/s recovering 2024-02-12T17:14:11.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:11.022315+0000 mon.c (mon.2) 304 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:14:11.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:11.023622+0000 mon.a (mon.0) 2120 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:14:11.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:11.642439+0000 mon.a (mon.0) 2121 : cluster [WRN] Health check update: Degraded data redundancy: 39572/315405 objects degraded (12.546%), 7 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:12.050 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:14:12.050 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:14:12.051 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:14:12.075 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:14:12.133 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:14:12.162 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:10.730215+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e654: 8 total, 8 up, 7 in 2024-02-12T17:14:12.162 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:11.021795+0000 mgr.y (mgr.24481) 923 : cluster [DBG] pgmap v1312: 120 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering, 1 clean+premerge+peered, 4 remapped+peering, 7 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 95 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 539 op/s; 39572/315405 objects degraded (12.546%); 25900/315405 objects misplaced (8.212%); 20 MiB/s, 319 objects/s recovering 2024-02-12T17:14:12.162 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:11.022315+0000 mon.c (mon.2) 304 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:14:12.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:11.023622+0000 mon.a (mon.0) 2120 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:14:12.163 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:11.642439+0000 mon.a (mon.0) 2121 : cluster [WRN] Health check update: Degraded data redundancy: 39572/315405 objects degraded (12.546%), 7 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:12.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:14:12.235 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:14:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:10.730215+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e654: 8 total, 8 up, 7 in 2024-02-12T17:14:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:11.021795+0000 mgr.y (mgr.24481) 923 : cluster [DBG] pgmap v1312: 120 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering, 1 clean+premerge+peered, 4 remapped+peering, 7 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 95 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 539 op/s; 39572/315405 objects degraded (12.546%); 25900/315405 objects misplaced (8.212%); 20 MiB/s, 319 objects/s recovering 2024-02-12T17:14:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:11.022315+0000 mon.c (mon.2) 304 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:14:12.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:11.023622+0000 mon.a (mon.0) 2120 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-12T17:14:12.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:11.642439+0000 mon.a (mon.0) 2121 : cluster [WRN] Health check update: Degraded data redundancy: 39572/315405 objects degraded (12.546%), 7 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:12.269 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:14:12.300 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:14:12.301 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:14:12.301 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:14:12.301 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:14:12.301 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:14:12.301 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:14:12.301 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:14:12.301 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:14:12.650 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:12.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:12.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T17:14:12.757 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:12.794 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:11.775710+0000 mon.a (mon.0) 2122 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T17:14:12.795 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:14:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:14:12.890 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:11.775710+0000 mon.a (mon.0) 2122 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T17:14:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:11.775775+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e655: 8 total, 8 up, 7 in 2024-02-12T17:14:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:11.897487+0000 mon.b (mon.1) 112 : audit [DBG] from='client.? v1:172.21.15.33:0/1253184428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:13.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:11.775710+0000 mon.a (mon.0) 2122 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-12T17:14:13.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:11.775775+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e655: 8 total, 8 up, 7 in 2024-02-12T17:14:13.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:11.897487+0000 mon.b (mon.1) 112 : audit [DBG] from='client.? v1:172.21.15.33:0/1253184428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:13.182 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:13.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:13.213 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T17:14:13.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:11.775775+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e655: 8 total, 8 up, 7 in 2024-02-12T17:14:13.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:11.897487+0000 mon.b (mon.1) 112 : audit [DBG] from='client.? v1:172.21.15.33:0/1253184428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:13.289 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:13.397 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:13.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:13.551 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:13.606 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:14.084 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:14.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:12.779195+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e656: 8 total, 8 up, 7 in 2024-02-12T17:14:14.088 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:13.022418+0000 mgr.y (mgr.24481) 924 : cluster [DBG] pgmap v1315: 120 pgs: 1 active+recovering+degraded, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 1 active+recovering, 1 clean+premerge+peered, 4 remapped+peering, 7 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 93 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 241 op/s; 39018/312513 objects degraded (12.485%); 25927/312513 objects misplaced (8.296%); 1.1 MiB/s, 28 objects/s recovering 2024-02-12T17:14:14.089 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:12.779195+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e656: 8 total, 8 up, 7 in 2024-02-12T17:14:14.089 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:13.022418+0000 mgr.y (mgr.24481) 924 : cluster [DBG] pgmap v1315: 120 pgs: 1 active+recovering+degraded, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 1 active+recovering, 1 clean+premerge+peered, 4 remapped+peering, 7 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 93 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 241 op/s; 39018/312513 objects degraded (12.485%); 25927/312513 objects misplaced (8.296%); 1.1 MiB/s, 28 objects/s recovering 2024-02-12T17:14:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:12.779195+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e656: 8 total, 8 up, 7 in 2024-02-12T17:14:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:13.022418+0000 mgr.y (mgr.24481) 924 : cluster [DBG] pgmap v1315: 120 pgs: 1 active+recovering+degraded, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 1 active+recovering, 1 clean+premerge+peered, 4 remapped+peering, 7 peering, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 93 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 241 op/s; 39018/312513 objects degraded (12.485%); 25927/312513 objects misplaced (8.296%); 1.1 MiB/s, 28 objects/s recovering 2024-02-12T17:14:14.371 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:14.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:14.479 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:14.530 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:14.616 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:14.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:14.726 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:14.781 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:15.015 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:15.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:13.808953+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e657: 8 total, 8 up, 7 in 2024-02-12T17:14:15.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:13.808953+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e657: 8 total, 8 up, 7 in 2024-02-12T17:14:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:15.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:15.121 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:15.195 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:15.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:13.808953+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e657: 8 total, 8 up, 7 in 2024-02-12T17:14:15.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:14:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:14:15.320 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:15.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:15.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:15.477 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:15.502 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:15.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:15.608 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:15.690 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:15.734 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:15.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:15.841 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:15.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:14.826322+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e658: 8 total, 8 up, 7 in 2024-02-12T17:14:15.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:14.964516+0000 mgr.y (mgr.24481) 925 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:15.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:15.023759+0000 mgr.y (mgr.24481) 926 : cluster [DBG] pgmap v1318: 119 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 721 op/s; 50166/306756 objects degraded (16.354%); 44371/306756 objects misplaced (14.465%); 16 MiB/s, 271 objects/s recovering 2024-02-12T17:14:15.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:15.024174+0000 mon.c (mon.2) 305 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:14:15.868 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:15.029756+0000 mon.a (mon.0) 2127 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:14:15.904 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:15.936 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:15.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:16.041 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:16.110 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:16.120 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:14:16.191 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:14.826322+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e658: 8 total, 8 up, 7 in 2024-02-12T17:14:16.191 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:14.964516+0000 mgr.y (mgr.24481) 925 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:16.191 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:15.023759+0000 mgr.y (mgr.24481) 926 : cluster [DBG] pgmap v1318: 119 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 721 op/s; 50166/306756 objects degraded (16.354%); 44371/306756 objects misplaced (14.465%); 16 MiB/s, 271 objects/s recovering 2024-02-12T17:14:16.191 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:15.024174+0000 mon.c (mon.2) 305 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:14:16.191 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:15.029756+0000 mon.a (mon.0) 2127 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:14:16.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:14.826322+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e658: 8 total, 8 up, 7 in 2024-02-12T17:14:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:14.964516+0000 mgr.y (mgr.24481) 925 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:15.023759+0000 mgr.y (mgr.24481) 926 : cluster [DBG] pgmap v1318: 119 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 721 op/s; 50166/306756 objects degraded (16.354%); 44371/306756 objects misplaced (14.465%); 16 MiB/s, 271 objects/s recovering 2024-02-12T17:14:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:15.024174+0000 mon.c (mon.2) 305 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:14:16.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:15.029756+0000 mon.a (mon.0) 2127 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-12T17:14:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:15.842209+0000 mon.a (mon.0) 2128 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:15.864081+0000 mon.a (mon.0) 2129 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:14:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:15.864136+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e659: 8 total, 8 up, 7 in 2024-02-12T17:14:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:16.341416+0000 osd.3 (osd.3) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:16.341477+0000 osd.3 (osd.3) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:16.644693+0000 mon.a (mon.0) 2131 : cluster [WRN] Health check update: Degraded data redundancy: 50166/306756 objects degraded (16.354%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:17.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:16.656595+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2024-02-12T17:14:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:15.842209+0000 mon.a (mon.0) 2128 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:15.864081+0000 mon.a (mon.0) 2129 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:14:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:15.864136+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e659: 8 total, 8 up, 7 in 2024-02-12T17:14:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:16.341416+0000 osd.3 (osd.3) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:16.341477+0000 osd.3 (osd.3) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:16.644693+0000 mon.a (mon.0) 2131 : cluster [WRN] Health check update: Degraded data redundancy: 50166/306756 objects degraded (16.354%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:16.656595+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2024-02-12T17:14:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:15.842209+0000 mon.a (mon.0) 2128 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:15.864081+0000 mon.a (mon.0) 2129 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-12T17:14:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:15.864136+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e659: 8 total, 8 up, 7 in 2024-02-12T17:14:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:16.341416+0000 osd.3 (osd.3) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:16.341477+0000 osd.3 (osd.3) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:16.644693+0000 mon.a (mon.0) 2131 : cluster [WRN] Health check update: Degraded data redundancy: 50166/306756 objects degraded (16.354%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:17.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:16.656595+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2024-02-12T17:14:18.479 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:18.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:18.588 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:14:18.637 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:18.672 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:14:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:18.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:14:18.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:17.024301+0000 mgr.y (mgr.24481) 927 : cluster [DBG] pgmap v1321: 119 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 812 op/s; 49432/302769 objects degraded (16.327%); 44152/302769 objects misplaced (14.583%); 17 MiB/s, 274 objects/s recovering 2024-02-12T17:14:18.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:17.541123+0000 mon.a (mon.0) 2133 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:18.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:17.675377+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2024-02-12T17:14:18.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:17.024301+0000 mgr.y (mgr.24481) 927 : cluster [DBG] pgmap v1321: 119 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 812 op/s; 49432/302769 objects degraded (16.327%); 44152/302769 objects misplaced (14.583%); 17 MiB/s, 274 objects/s recovering 2024-02-12T17:14:18.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:17.541123+0000 mon.a (mon.0) 2133 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:18.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:17.675377+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2024-02-12T17:14:18.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:18.817 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:17.024301+0000 mgr.y (mgr.24481) 927 : cluster [DBG] pgmap v1321: 119 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 812 op/s; 49432/302769 objects degraded (16.327%); 44152/302769 objects misplaced (14.583%); 17 MiB/s, 274 objects/s recovering 2024-02-12T17:14:18.817 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:17.541123+0000 mon.a (mon.0) 2133 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:18.817 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:17.675377+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2024-02-12T17:14:18.824 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:18.850 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:18.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:18.955 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:19.019 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:19.239 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:19.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:19.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:14:19.430 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:19.464 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:14:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:19.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:14:19.570 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:19.631 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:19.660 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:19.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:19.722 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:18.687867+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e662: 8 total, 8 up, 7 in 2024-02-12T17:14:19.722 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:19.025141+0000 mgr.y (mgr.24481) 928 : cluster [DBG] pgmap v1324: 118 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 47609/295689 objects degraded (16.101%); 42664/295689 objects misplaced (14.429%); 11 MiB/s, 183 objects/s recovering 2024-02-12T17:14:19.766 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:19.818 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:18.687867+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e662: 8 total, 8 up, 7 in 2024-02-12T17:14:19.818 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:19.025141+0000 mgr.y (mgr.24481) 928 : cluster [DBG] pgmap v1324: 118 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 47609/295689 objects degraded (16.101%); 42664/295689 objects misplaced (14.429%); 11 MiB/s, 183 objects/s recovering 2024-02-12T17:14:19.819 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:18.687867+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e662: 8 total, 8 up, 7 in 2024-02-12T17:14:19.819 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:19.025141+0000 mgr.y (mgr.24481) 928 : cluster [DBG] pgmap v1324: 118 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 47609/295689 objects degraded (16.101%); 42664/295689 objects misplaced (14.429%); 11 MiB/s, 183 objects/s recovering 2024-02-12T17:14:19.821 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:19.870 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:19.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:19.975 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:20.038 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:20.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:20.260 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:20.331 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:20.685 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:21.174 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:14:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:20] "GET /metrics HTTP/1.1" 200 176831 "" "Prometheus/2.33.4" 2024-02-12T17:14:21.192 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:14:21.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:21.085556+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? v1:172.21.15.33:0/1523939004' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:14:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:21.085556+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? v1:172.21.15.33:0/1523939004' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:14:21.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:21.085556+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? v1:172.21.15.33:0/1523939004' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:14:22.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:21.025711+0000 mgr.y (mgr.24481) 929 : cluster [DBG] pgmap v1325: 118 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 993 B/s rd, 0 B/s wr, 716 op/s; 47609/295689 objects degraded (16.101%); 42664/295689 objects misplaced (14.429%); 8.9 MiB/s, 142 objects/s recovering 2024-02-12T17:14:22.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:21.137672+0000 mon.a (mon.0) 2137 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:22.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:21.144646+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? v1:172.21.15.33:0/1523939004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:14:22.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:21.144713+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e663: 8 total, 8 up, 7 in 2024-02-12T17:14:22.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:21.355538+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.33:0/3736216078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:14:22.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:21.355940+0000 mon.a (mon.0) 2140 : audit [INF] from='client.26005 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:14:22.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:21.649438+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: Degraded data redundancy: 47609/295689 objects degraded (16.101%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:22.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:21.658466+0000 mon.a (mon.0) 2142 : audit [INF] from='client.26005 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:14:22.494 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:21.658531+0000 mon.a (mon.0) 2143 : cluster [DBG] osdmap e664: 8 total, 8 up, 7 in 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:21.025711+0000 mgr.y (mgr.24481) 929 : cluster [DBG] pgmap v1325: 118 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 993 B/s rd, 0 B/s wr, 716 op/s; 47609/295689 objects degraded (16.101%); 42664/295689 objects misplaced (14.429%); 8.9 MiB/s, 142 objects/s recovering 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:21.137672+0000 mon.a (mon.0) 2137 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:21.144646+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? v1:172.21.15.33:0/1523939004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:21.144713+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e663: 8 total, 8 up, 7 in 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:21.355538+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.33:0/3736216078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:21.355940+0000 mon.a (mon.0) 2140 : audit [INF] from='client.26005 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:21.649438+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: Degraded data redundancy: 47609/295689 objects degraded (16.101%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:21.658466+0000 mon.a (mon.0) 2142 : audit [INF] from='client.26005 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:14:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:21.658531+0000 mon.a (mon.0) 2143 : cluster [DBG] osdmap e664: 8 total, 8 up, 7 in 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:21.025711+0000 mgr.y (mgr.24481) 929 : cluster [DBG] pgmap v1325: 118 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 993 B/s rd, 0 B/s wr, 716 op/s; 47609/295689 objects degraded (16.101%); 42664/295689 objects misplaced (14.429%); 8.9 MiB/s, 142 objects/s recovering 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:21.137672+0000 mon.a (mon.0) 2137 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:21.144646+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? v1:172.21.15.33:0/1523939004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:21.144713+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e663: 8 total, 8 up, 7 in 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:21.355538+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.33:0/3736216078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:21.355940+0000 mon.a (mon.0) 2140 : audit [INF] from='client.26005 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:21.649438+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: Degraded data redundancy: 47609/295689 objects degraded (16.101%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:21.658466+0000 mon.a (mon.0) 2142 : audit [INF] from='client.26005 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:14:22.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:21.658531+0000 mon.a (mon.0) 2143 : cluster [DBG] osdmap e664: 8 total, 8 up, 7 in 2024-02-12T17:14:22.744 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:22.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:22.851 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:22.935 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:14:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:14:23.001 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:23.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:23.106 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:23.191 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:23.477 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:23.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:23.592 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:23.655 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:23.704 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T17:14:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:23.028398+0000 mgr.y (mgr.24481) 930 : cluster [DBG] pgmap v1328: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 956 B/s rd, 0 B/s wr, 701 op/s; 46813/291510 objects degraded (16.059%); 42422/291510 objects misplaced (14.553%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T17:14:24.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:23.181346+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2024-02-12T17:14:24.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:23.028398+0000 mgr.y (mgr.24481) 930 : cluster [DBG] pgmap v1328: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 956 B/s rd, 0 B/s wr, 701 op/s; 46813/291510 objects degraded (16.059%); 42422/291510 objects misplaced (14.553%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T17:14:24.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:23.181346+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2024-02-12T17:14:24.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:23.028398+0000 mgr.y (mgr.24481) 930 : cluster [DBG] pgmap v1328: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped, 98 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 956 B/s rd, 0 B/s wr, 701 op/s; 46813/291510 objects degraded (16.059%); 42422/291510 objects misplaced (14.553%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T17:14:24.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:23.181346+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2024-02-12T17:14:24.720 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T17:14:24.931 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:24.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:25.045 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:25.118 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:25.198 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:25.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:25.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:14:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:14:25.304 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:25.383 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:25.509 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:14:25.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:24.217713+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e666: 8 total, 8 up, 7 in 2024-02-12T17:14:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:24.217713+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e666: 8 total, 8 up, 7 in 2024-02-12T17:14:25.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:24.217713+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e666: 8 total, 8 up, 7 in 2024-02-12T17:14:26.152 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:26.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:26.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:26.316 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:26.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:26.450 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:26.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:24.974524+0000 mgr.y (mgr.24481) 931 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:26.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:25.030264+0000 mgr.y (mgr.24481) 932 : cluster [DBG] pgmap v1331: 118 pgs: 1 active+undersized+degraded+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 98 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 898 op/s; 45097/284904 objects degraded (15.829%); 41066/284904 objects misplaced (14.414%); 12 MiB/s, 196 objects/s recovering 2024-02-12T17:14:26.508 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:25.214750+0000 mon.a (mon.0) 2146 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:26.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:24.974524+0000 mgr.y (mgr.24481) 931 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:26.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:25.030264+0000 mgr.y (mgr.24481) 932 : cluster [DBG] pgmap v1331: 118 pgs: 1 active+undersized+degraded+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 98 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 898 op/s; 45097/284904 objects degraded (15.829%); 41066/284904 objects misplaced (14.414%); 12 MiB/s, 196 objects/s recovering 2024-02-12T17:14:26.509 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:25.214750+0000 mon.a (mon.0) 2146 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:26.513 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:24.974524+0000 mgr.y (mgr.24481) 931 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:25.030264+0000 mgr.y (mgr.24481) 932 : cluster [DBG] pgmap v1331: 118 pgs: 1 active+undersized+degraded+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 98 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 898 op/s; 45097/284904 objects degraded (15.829%); 41066/284904 objects misplaced (14.414%); 12 MiB/s, 196 objects/s recovering 2024-02-12T17:14:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:25.214750+0000 mon.a (mon.0) 2146 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:14:26.775 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:14:26.776 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:14:27.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:25.700252+0000 osd.6 (osd.6) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:27.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:25.700300+0000 osd.6 (osd.6) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:27.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:26.227309+0000 osd.7 (osd.7) 313 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:27.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:26.227364+0000 osd.7 (osd.7) 314 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:27.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:26.268503+0000 mon.a (mon.0) 2147 : cluster [DBG] osdmap e667: 8 total, 8 up, 7 in 2024-02-12T17:14:27.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:26.652161+0000 mon.a (mon.0) 2148 : cluster [WRN] Health check update: Degraded data redundancy: 45097/284904 objects degraded (15.829%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:27.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:26.661014+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2024-02-12T17:14:27.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:25.700252+0000 osd.6 (osd.6) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:27.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:25.700300+0000 osd.6 (osd.6) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:27.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:26.227309+0000 osd.7 (osd.7) 313 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:27.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:26.227364+0000 osd.7 (osd.7) 314 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:27.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:26.268503+0000 mon.a (mon.0) 2147 : cluster [DBG] osdmap e667: 8 total, 8 up, 7 in 2024-02-12T17:14:27.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:26.652161+0000 mon.a (mon.0) 2148 : cluster [WRN] Health check update: Degraded data redundancy: 45097/284904 objects degraded (15.829%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:27.559 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:26.661014+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2024-02-12T17:14:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:25.700252+0000 osd.6 (osd.6) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:25.700300+0000 osd.6 (osd.6) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:26.227309+0000 osd.7 (osd.7) 313 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:27.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:26.227364+0000 osd.7 (osd.7) 314 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:27.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:26.268503+0000 mon.a (mon.0) 2147 : cluster [DBG] osdmap e667: 8 total, 8 up, 7 in 2024-02-12T17:14:27.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:26.652161+0000 mon.a (mon.0) 2148 : cluster [WRN] Health check update: Degraded data redundancy: 45097/284904 objects degraded (15.829%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:27.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:26.661014+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2024-02-12T17:14:28.213 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T17:14:28.344 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:14:28.344 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:14:28.344 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:14:28.344 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:14:28.345 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:14:28.345 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:14:28.345 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:14:28.389 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:14:28.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:14:28.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:14:28.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:14:28.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:14:28.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:14:28.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:14:28.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:14:28.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:14:28.525 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:27.033520+0000 mgr.y (mgr.24481) 933 : cluster [DBG] pgmap v1334: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 98 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 896 op/s; 44320/280752 objects degraded (15.786%); 40810/280752 objects misplaced (14.536%); 12 MiB/s, 196 objects/s recovering 2024-02-12T17:14:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:27.140653+0000 osd.4 (osd.4) 207 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:27.140702+0000 osd.4 (osd.4) 208 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:27.679564+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e669: 8 total, 8 up, 7 in 2024-02-12T17:14:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:28.028019+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? v1:172.21.15.33:0/3164673974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:27.033520+0000 mgr.y (mgr.24481) 933 : cluster [DBG] pgmap v1334: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 98 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 896 op/s; 44320/280752 objects degraded (15.786%); 40810/280752 objects misplaced (14.536%); 12 MiB/s, 196 objects/s recovering 2024-02-12T17:14:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:27.140653+0000 osd.4 (osd.4) 207 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:27.140702+0000 osd.4 (osd.4) 208 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:27.679564+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e669: 8 total, 8 up, 7 in 2024-02-12T17:14:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:28.028019+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? v1:172.21.15.33:0/3164673974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:28.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:28.628 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:28.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:28.708 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:27.033520+0000 mgr.y (mgr.24481) 933 : cluster [DBG] pgmap v1334: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 98 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 896 op/s; 44320/280752 objects degraded (15.786%); 40810/280752 objects misplaced (14.536%); 12 MiB/s, 196 objects/s recovering 2024-02-12T17:14:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:27.140653+0000 osd.4 (osd.4) 207 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:27.140702+0000 osd.4 (osd.4) 208 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:27.679564+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e669: 8 total, 8 up, 7 in 2024-02-12T17:14:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:28.028019+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? v1:172.21.15.33:0/3164673974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:28.748 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:28.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:28.771 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:28.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:28.916 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:28.950 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:28.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:29.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:29.114 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:29.340 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:29.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:29.350 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:29.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:29.497 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:29.530 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:29.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:29.641 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:29.744 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:30.426 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:29.034320+0000 mgr.y (mgr.24481) 934 : cluster [DBG] pgmap v1336: 118 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 753 op/s; 42703/274053 objects degraded (15.582%); 39466/274053 objects misplaced (14.401%); 9.4 MiB/s, 150 objects/s recovering 2024-02-12T17:14:30.774 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.36023.0:162315 11.6 e669/651 11:64032c1b:::benchmark_data_smithi033_188107_object36625:head v 669'10207, mlcod=669'10205)", 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:30.769472+0000", 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.002497661, 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025247059999999998, 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:30.769472+0000", 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:30.769472+0000", 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:30.769470+0000", 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6509999999999996e-06 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:30.769480+0000", 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0349e-05 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:30.769490+0000", 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4479999999999996e-06 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:30.769495+0000", 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8756e-05 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:30.769533+0000", 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6699999999999999e-05 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:30.769550+0000", 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8233999999999997e-05 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 1 2024-02-12T17:14:30.778 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:14:30.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:29.034320+0000 mgr.y (mgr.24481) 934 : cluster [DBG] pgmap v1336: 118 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 753 op/s; 42703/274053 objects degraded (15.582%); 39466/274053 objects misplaced (14.401%); 9.4 MiB/s, 150 objects/s recovering 2024-02-12T17:14:30.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:29.034320+0000 mgr.y (mgr.24481) 934 : cluster [DBG] pgmap v1336: 118 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 753 op/s; 42703/274053 objects degraded (15.582%); 39466/274053 objects misplaced (14.401%); 9.4 MiB/s, 150 objects/s recovering 2024-02-12T17:14:31.278 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:14:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:30] "GET /metrics HTTP/1.1" 200 176831 "" "Prometheus/2.33.4" 2024-02-12T17:14:31.342 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:31.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:31.370 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:14:31.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:31.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T17:14:31.513 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:31.539 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:31.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:31.643 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:31.702 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:31.897 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:31.036081+0000 mgr.y (mgr.24481) 935 : cluster [DBG] pgmap v1337: 118 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 603 op/s; 42703/274053 objects degraded (15.582%); 39466/274053 objects misplaced (14.401%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T17:14:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:31.303568+0000 osd.2 (osd.2) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:31.303622+0000 osd.2 (osd.2) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:31.345654+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e670: 8 total, 8 up, 7 in 2024-02-12T17:14:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:31.655312+0000 mon.a (mon.0) 2152 : cluster [WRN] Health check update: Degraded data redundancy: 42703/274053 objects degraded (15.582%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:31.675004+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e671: 8 total, 8 up, 7 in 2024-02-12T17:14:32.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:31.036081+0000 mgr.y (mgr.24481) 935 : cluster [DBG] pgmap v1337: 118 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 603 op/s; 42703/274053 objects degraded (15.582%); 39466/274053 objects misplaced (14.401%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:31.303568+0000 osd.2 (osd.2) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:31.303622+0000 osd.2 (osd.2) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:31.345654+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e670: 8 total, 8 up, 7 in 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:31.655312+0000 mon.a (mon.0) 2152 : cluster [WRN] Health check update: Degraded data redundancy: 42703/274053 objects degraded (15.582%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:31.675004+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e671: 8 total, 8 up, 7 in 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:31.036081+0000 mgr.y (mgr.24481) 935 : cluster [DBG] pgmap v1337: 118 pgs: 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 603 op/s; 42703/274053 objects degraded (15.582%); 39466/274053 objects misplaced (14.401%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:31.303568+0000 osd.2 (osd.2) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:31.303622+0000 osd.2 (osd.2) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:31.345654+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e670: 8 total, 8 up, 7 in 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:31.655312+0000 mon.a (mon.0) 2152 : cluster [WRN] Health check update: Degraded data redundancy: 42703/274053 objects degraded (15.582%), 9 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:32.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:31.675004+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e671: 8 total, 8 up, 7 in 2024-02-12T17:14:32.780 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:14:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:14:33.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:32.528188+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:33.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:32.689587+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-12T17:14:33.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:32.528188+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:33.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:32.689587+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-12T17:14:33.794 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:14:33.794 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:14:33.794 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:14:33.795 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:14:33.795 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:14:33.821 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:32.528188+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:33.821 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:32.689587+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-12T17:14:34.528 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:34.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:34.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T17:14:34.657 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T17:14:34.726 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:34.750 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:14:34.781 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:34.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:34.887 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:34.976 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:33.036549+0000 mgr.y (mgr.24481) 936 : cluster [DBG] pgmap v1341: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 956 B/s rd, 0 B/s wr, 668 op/s; 41986/270027 objects degraded (15.549%); 39245/270027 objects misplaced (14.534%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T17:14:34.976 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:34.376973+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? v1:172.21.15.33:0/3029551955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:14:34.976 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:34.516743+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? v1:172.21.15.33:0/698078295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:34.976 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:33.036549+0000 mgr.y (mgr.24481) 936 : cluster [DBG] pgmap v1341: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 956 B/s rd, 0 B/s wr, 668 op/s; 41986/270027 objects degraded (15.549%); 39245/270027 objects misplaced (14.534%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T17:14:34.976 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:34.376973+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? v1:172.21.15.33:0/3029551955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:14:34.977 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:34.516743+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? v1:172.21.15.33:0/698078295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:34.984 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:34.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:33.036549+0000 mgr.y (mgr.24481) 936 : cluster [DBG] pgmap v1341: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 99 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 956 B/s rd, 0 B/s wr, 668 op/s; 41986/270027 objects degraded (15.549%); 39245/270027 objects misplaced (14.534%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T17:14:34.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:34.376973+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? v1:172.21.15.33:0/3029551955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:14:34.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:34.516743+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? v1:172.21.15.33:0/698078295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:14:35.006 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:35.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:14:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:14:35.452 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:35.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:35.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:14:35.567 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:34.687522+0000 mon.a (mon.0) 2158 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:35.567 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:34.691513+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.33:0/3029551955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:14:35.567 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:34.691590+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? v1:172.21.15.33:0/698078295' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:14:35.567 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:34.691629+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-12T17:14:35.629 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:35.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:34.687522+0000 mon.a (mon.0) 2158 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:35.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:34.691513+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.33:0/3029551955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:14:35.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:34.691590+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? v1:172.21.15.33:0/698078295' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:14:35.715 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:34.691629+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-12T17:14:35.717 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:34.687522+0000 mon.a (mon.0) 2158 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:35.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:34.691513+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.33:0/3029551955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:14:35.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:34.691590+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? v1:172.21.15.33:0/698078295' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:14:35.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:34.691629+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-12T17:14:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:14:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:35.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:14:35.998 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:36.061 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:36.231 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:36.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:36.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:36.421 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:36.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:36.538 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:36.637 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:34.987890+0000 mgr.y (mgr.24481) 937 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:35.037002+0000 mgr.y (mgr.24481) 938 : cluster [DBG] pgmap v1343: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 829 op/s; 40721/264114 objects degraded (15.418%); 38015/264114 objects misplaced (14.393%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T17:14:36.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:34.987890+0000 mgr.y (mgr.24481) 937 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:36.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:35.037002+0000 mgr.y (mgr.24481) 938 : cluster [DBG] pgmap v1343: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 829 op/s; 40721/264114 objects degraded (15.418%); 38015/264114 objects misplaced (14.393%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T17:14:36.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:34.987890+0000 mgr.y (mgr.24481) 937 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:36.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:35.037002+0000 mgr.y (mgr.24481) 938 : cluster [DBG] pgmap v1343: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 829 op/s; 40721/264114 objects degraded (15.418%); 38015/264114 objects misplaced (14.393%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T17:14:37.104 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:51713 11.d 11:b0e110fd:::benchmark_data_smithi033_188107_object51712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e608)", 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:12:34.941295+0000", 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 122.159189969, 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052866593000000003, 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51713 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941295+0000", 2024-02-12T17:14:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941295+0000", 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941293+0000", 2024-02-12T17:14:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2028999999999997e-05 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941375+0000", 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1749999999999994e-06 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941384+0000", 2024-02-12T17:14:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.959e-06 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941390+0000", 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2905999999999999e-05 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941433+0000", 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015072999999999999 2024-02-12T17:14:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941584+0000", 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017116599999999999 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,6", 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.941755+0000", 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033923270000000001 2024-02-12T17:14:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.945147+0000", 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030684100000000003 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.945454+0000", 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048621189000000002 2024-02-12T17:14:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.994075+0000", 2024-02-12T17:14:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7298e-05 2024-02-12T17:14:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.994123+0000", 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8658000000000002e-05 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:34.994161+0000", 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052866593000000003 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:54369 11.c 11:35c8ea78:::benchmark_data_smithi033_188107_object54368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e608)", 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:12:36.635843+0000", 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 120.46464224, 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050907442999999997, 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 54369 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.635843+0000", 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.635843+0000", 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.635840+0000", 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8925000000000003e-05 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.635889+0000", 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2490999999999999e-05 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.635901+0000", 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4300000000000003e-06 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.635908+0000", 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8359999999999998e-05 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.635956+0000", 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121214 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.636077+0000", 2024-02-12T17:14:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015633500000000001 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.636234+0000", 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011527615999999999 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.647761+0000", 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.038741539999999998 2024-02-12T17:14:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.686503+0000", 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014588900000000001 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.686649+0000", 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000101263 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.686750+0000", 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050907442999999997 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:54373 11.c 11:3c82dacf:::benchmark_data_smithi033_188107_object54372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e608)", 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:12:36.636990+0000", 2024-02-12T17:14:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 120.46349496800001, 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058181929, 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 54373 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.636990+0000", 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.636990+0000", 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.636988+0000", 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9984e-05 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.637048+0000", 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4360000000000008e-06 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.637058+0000", 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4279999999999998e-06 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.637062+0000", 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010380109999999999 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.638100+0000", 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012864800000000001 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.638229+0000", 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000153979 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.638383+0000", 2024-02-12T17:14:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0093861469999999992 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.647769+0000", 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047272084999999998 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.695041+0000", 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7321999999999999e-05 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.695088+0000", 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3566000000000006e-05 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.695172+0000", 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058181929 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:54431 11.d 11:bce3f729:::benchmark_data_smithi033_188107_object54430:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e608)", 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:12:36.717780+0000", 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 120.382704889, 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052057306999999997, 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 54431 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.717780+0000", 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.717780+0000", 2024-02-12T17:14:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.717778+0000", 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7187999999999994e-05 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.717846+0000", 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5669999999999995e-06 2024-02-12T17:14:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.717854+0000", 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.284e-06 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.717858+0000", 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7784999999999998e-05 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.717896+0000", 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012632 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718022+0000", 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017942199999999999 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,6", 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718202+0000", 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022246190000000002 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.720427+0000", 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038301600000000002 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.720810+0000", 2024-02-12T17:14:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048971537000000002 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.769781+0000", 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2848000000000001e-05 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.769814+0000", 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3238999999999999e-05 2024-02-12T17:14:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.769837+0000", 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.052057306999999997 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:54435 11.c 11:3687baae:::benchmark_data_smithi033_188107_object54434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e608)", 2024-02-12T17:14:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:12:36.718112+0000", 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 120.382372457, 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051806918, 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 54435 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718112+0000", 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718112+0000", 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718111+0000", 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4566000000000004e-05 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718185+0000", 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9590000000000007e-06 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718195+0000", 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8300000000000001e-06 2024-02-12T17:14:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718201+0000", 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1233999999999999e-05 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718242+0000", 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010156 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718344+0000", 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020506999999999999 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.718549+0000", 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022659820000000002 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.720815+0000", 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048958913999999999 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.769774+0000", 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8912000000000002e-05 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.769823+0000", 2024-02-12T17:14:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6416000000000007e-05 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.769919+0000", 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.051806918 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:54438 11.c 11:35ede375:::benchmark_data_smithi033_188107_object54437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e608)", 2024-02-12T17:14:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:12:36.721452+0000", 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 120.379033165, 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048593865999999999, 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 54438 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721452+0000", 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721452+0000", 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721449+0000", 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5446999999999999e-05 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721515+0000", 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1849e-05 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721527+0000", 2024-02-12T17:14:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9399999999999999e-06 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721533+0000", 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0179999999999999e-05 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721563+0000", 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4905e-05 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721658+0000", 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106572 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.721764+0000", 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016867449999999999 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.723451+0000", 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.046502174 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.769953+0000", 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9569000000000001e-05 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.769983+0000", 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2631000000000004e-05 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:12:36.770045+0000", 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048593865999999999 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:87344 11.c 11:363f990f:::benchmark_data_smithi033_188107_object87343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:01.656498+0000", 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.443986860999999, 2024-02-12T17:14:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045631532000000002, 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87344 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656498+0000", 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656498+0000", 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999967 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656495+0000", 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0933999999999998e-05 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656536+0000", 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9208999999999999e-05 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656555+0000", 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9319999999999996e-06 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656563+0000", 2024-02-12T17:14:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2577000000000002e-05 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656615+0000", 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014113400000000001 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656757+0000", 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015785899999999999 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.656914+0000", 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019389884 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.676304+0000", 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025674879000000001 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.701979+0000", 2024-02-12T17:14:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5315999999999997e-05 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.702034+0000", 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4948e-05 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:01.702129+0000", 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045631532000000002 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:91779 11.e 11:70a49202:::benchmark_data_smithi033_188107_object91778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e615)", 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:05.565790+0000", 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.534694771000005, 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050668315999999998, 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 91779 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.565790+0000", 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.565790+0000", 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:14:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.565789+0000", 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4054e-05 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.565813+0000", 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.784e-06 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.565819+0000", 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8550000000000004e-06 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.565823+0000", 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9846999999999996e-05 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.565903+0000", 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000139567 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.566042+0000", 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000174704 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 7", 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.566217+0000", 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012804830999999999 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.579022+0000", 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037306082999999997 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.616328+0000", 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8378999999999999e-05 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.616376+0000", 2024-02-12T17:14:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2006000000000001e-05 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.616458+0000", 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.050668315999999998 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:91785 11.e 11:7f35517c:::benchmark_data_smithi033_188107_object91784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e615)", 2024-02-12T17:14:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:05.572813+0000", 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.527671807999994, 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043789643000000003, 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 91785 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.572813+0000", 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.572813+0000", 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.572811+0000", 2024-02-12T17:14:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1897000000000001e-05 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.572863+0000", 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5529999999999999e-05 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.572878+0000", 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5909999999999998e-06 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.572886+0000", 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023781710000000001 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.575264+0000", 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014074000000000001 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.575405+0000", 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000168887 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 7", 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.575574+0000", 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022895769999999999 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.598469+0000", 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018022153999999999 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.616492+0000", 2024-02-12T17:14:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8160000000000001e-05 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.616530+0000", 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2911999999999996e-05 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:05.616603+0000", 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043789643000000003 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:92898 11.d 11:b96b9380:::benchmark_data_smithi033_188107_object92897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:06.637457+0000", 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.463027929000006, 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059857136999999998, 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92898 2024-02-12T17:14:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637457+0000", 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637457+0000", 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637455+0000", 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1131000000000003e-05 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637496+0000", 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2240000000000001e-06 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637505+0000", 2024-02-12T17:14:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8220000000000003e-06 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637511+0000", 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015734599999999999 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637668+0000", 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014034 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637809+0000", 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000113372 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,6", 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.637922+0000", 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037053329999999999 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.641627+0000", 2024-02-12T17:14:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011263899999999999 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642754+0000", 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054427534 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697181+0000", 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9660999999999994e-05 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697271+0000", 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2985999999999998e-05 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697314+0000", 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.059857136999999998 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:92904 11.c 11:309e22d9:::benchmark_data_smithi033_188107_object92903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:06.642273+0000", 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.458211628000001, 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054846694000000001, 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92904 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642273+0000", 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642273+0000", 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642271+0000", 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7241000000000003e-05 2024-02-12T17:14:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642328+0000", 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0618000000000001e-05 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642339+0000", 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4509999999999997e-06 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642345+0000", 2024-02-12T17:14:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9818000000000003e-05 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642395+0000", 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000144404 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642539+0000", 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000180027 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642719+0000", 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0085767650000000001 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.651296+0000", 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.045598090000000001 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.696894+0000", 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000102505 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.696997+0000", 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000123125 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697120+0000", 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054846694000000001 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:92905 11.c 11:3b22e803:::benchmark_data_smithi033_188107_object92904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:06.642745+0000", 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.457739941, 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054578684000000002, 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92905 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642745+0000", 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642745+0000", 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642743+0000", 2024-02-12T17:14:37.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4720000000000004e-05 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642808+0000", 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1497e-05 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642820+0000", 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8420000000000001e-06 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642826+0000", 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022797659999999999 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.645105+0000", 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014310899999999999 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.645248+0000", 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016929400000000001 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.645418+0000", 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0089136170000000004 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.654331+0000", 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042859429999999997 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697191+0000", 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7790999999999997e-05 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697239+0000", 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4989999999999998e-05 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697324+0000", 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054578684000000002 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:92906 11.d 11:b4e24a75:::benchmark_data_smithi033_188107_object92905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:06.642881+0000", 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.457603511000002, 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054554931000000001, 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92906 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642881+0000", 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642881+0000", 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642880+0000", 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4064999999999998e-05 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642934+0000", 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7990000000000005e-06 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642944+0000", 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0689999999999998e-06 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.642950+0000", 2024-02-12T17:14:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025141679999999998 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.645464+0000", 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012543799999999999 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.645589+0000", 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106278 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,6", 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.645696+0000", 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086190429999999998 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.654315+0000", 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001228289 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.655543+0000", 2024-02-12T17:14:37.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.041818778000000001 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697362+0000", 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6806e-05 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697399+0000", 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7673000000000003e-05 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:06.697436+0000", 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054554931000000001 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.36023.0:134676 11.7 e639/630 11:e49ed952:::benchmark_data_smithi033_188107_object8986:head v 639'8367, mlcod=638'8365)", 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:53.785467+0000", 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.315017449000003, 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.055273223000000003, 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.195 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.785467+0000", 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.785467+0000", 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.785466+0000", 2024-02-12T17:14:37.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2080000000000003e-06 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.785473+0000", 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8910000000000007e-06 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.785483+0000", 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.669e-06 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.785489+0000", 2024-02-12T17:14:37.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6078e-05 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.785556+0000", 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8496999999999998e-05 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.785584+0000", 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.055132694000000003 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.840717+0000", 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3836e-05 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:53.840741+0000", 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.055273223000000003 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.36023.0:135679 11.e e640/630 11:73663adb:::benchmark_data_smithi033_188107_object9989:head v 640'8513, mlcod=640'8510)", 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:54.916215+0000", 2024-02-12T17:14:37.199 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.184269778000001, 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.060361338, 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.916215+0000", 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.916215+0000", 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:14:37.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.916213+0000", 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2459e-05 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.916226+0000", 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0448000000000001e-05 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.916236+0000", 2024-02-12T17:14:37.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4389999999999997e-06 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.916242+0000", 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013624700000000001 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.916379+0000", 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3799e-05 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.916403+0000", 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.060140539 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.976543+0000", 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3259999999999997e-05 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.976576+0000", 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.060361338 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.36023.0:135696 11.7 e640/630 11:e13c799e:::benchmark_data_smithi033_188107_object10006:head v 640'8425, mlcod=640'8421)", 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:54.919421+0000", 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.181063532000003, 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063489004000000002, 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.919421+0000", 2024-02-12T17:14:37.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.919421+0000", 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.919419+0000", 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1235e-05 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.919431+0000", 2024-02-12T17:14:37.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1925e-05 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.919443+0000", 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8919999999999999e-06 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.919447+0000", 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5862999999999997e-05 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.919503+0000", 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9165e-05 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.919522+0000", 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063334614999999997 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.982857+0000", 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3133000000000001e-05 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.982910+0000", 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063489004000000002 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.36023.0:135695 11.6 e640/630 11:657ac936:::benchmark_data_smithi033_188107_object10005:head v 640'8567, mlcod=639'8565)", 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:54.920206+0000", 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.180278838, 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056396368000000002, 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.920206+0000", 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.920206+0000", 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.920204+0000", 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1800000000000002e-06 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.920214+0000", 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.429e-05 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.920228+0000", 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6119999999999998e-06 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.920234+0000", 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4607000000000001e-05 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.920288+0000", 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0361e-05 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.920308+0000", 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056270494999999997 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.976579+0000", 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3303e-05 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.976602+0000", 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056396368000000002 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.36023.0:135701 11.c e640/630 11:344058df:::benchmark_data_smithi033_188107_object10011:head v 640'8399, mlcod=640'8397)", 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:54.923183+0000", 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.177301757000002, 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054421842999999998, 2024-02-12T17:14:37.212 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923183+0000", 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923183+0000", 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923181+0000", 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1689999999999998e-06 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923190+0000", 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4930000000000004e-06 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923200+0000", 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1250000000000001e-06 2024-02-12T17:14:37.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923205+0000", 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5728999999999994e-05 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923271+0000", 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4525e-05 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923295+0000", 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054277684999999999 2024-02-12T17:14:37.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.977573+0000", 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1940000000000003e-05 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.977605+0000", 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.054421842999999998 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.36023.0:135697 11.6 e640/630 11:6060fc50:::benchmark_data_smithi033_188107_object10007:head v 640'8568, mlcod=639'8565)", 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:54.923218+0000", 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.177267182999998, 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.053863188999999999, 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923218+0000", 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923218+0000", 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923217+0000", 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0219999999999998e-06 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923221+0000", 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.048e-06 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923225+0000", 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1999999999999999e-06 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.923228+0000", 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.048476789999999999 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.971705+0000", 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0293999999999999e-05 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.971725+0000", 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005327788 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.977053+0000", 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7512999999999999e-05 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.977081+0000", 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.053863188999999999 2024-02-12T17:14:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.36023.0:135709 11.b e640/630 11:dd43dfe3:::benchmark_data_smithi033_188107_object10019:head v 640'8475, mlcod=640'8473)", 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:13:54.928146+0000", 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.172339145000002, 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047319430000000003, 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.928146+0000", 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.928146+0000", 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.928143+0000", 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4463e-05 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.928158+0000", 2024-02-12T17:14:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4030999999999999e-05 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.928202+0000", 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3690000000000007e-06 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.928210+0000", 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1736999999999995e-05 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.928272+0000", 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4966000000000001e-05 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.928297+0000", 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047135265000000003 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.975432+0000", 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3049999999999997e-05 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:13:54.975465+0000", 2024-02-12T17:14:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047319430000000003 2024-02-12T17:14:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:37.225 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:14:37.717 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T17:14:37.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:36.656641+0000 mon.a (mon.0) 2162 : cluster [WRN] Health check update: Degraded data redundancy: 40721/264114 objects degraded (15.418%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:37.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:36.682804+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-12T17:14:37.848 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T17:14:38.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:36.656641+0000 mon.a (mon.0) 2162 : cluster [WRN] Health check update: Degraded data redundancy: 40721/264114 objects degraded (15.418%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:38.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:36.682804+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-12T17:14:38.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:36.656641+0000 mon.a (mon.0) 2162 : cluster [WRN] Health check update: Degraded data redundancy: 40721/264114 objects degraded (15.418%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:38.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:36.682804+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-12T17:14:38.037 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:38.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:38.146 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:38.207 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:38.217 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:38.253 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:14:38.266 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:38.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:38.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:38.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:38.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:37.037532+0000 mgr.y (mgr.24481) 939 : cluster [DBG] pgmap v1345: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 955 B/s rd, 0 B/s wr, 578 op/s; 40093/260736 objects degraded (15.377%); 37787/260736 objects misplaced (14.492%); 5.8 MiB/s, 92 objects/s recovering 2024-02-12T17:14:38.606 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:38.694 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:38.720 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:38.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:37.037532+0000 mgr.y (mgr.24481) 939 : cluster [DBG] pgmap v1345: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 955 B/s rd, 0 B/s wr, 578 op/s; 40093/260736 objects degraded (15.377%); 37787/260736 objects misplaced (14.492%); 5.8 MiB/s, 92 objects/s recovering 2024-02-12T17:14:38.819 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:38.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:38.928 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:38.980 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:38.981 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:37.037532+0000 mgr.y (mgr.24481) 939 : cluster [DBG] pgmap v1345: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 955 B/s rd, 0 B/s wr, 578 op/s; 40093/260736 objects degraded (15.377%); 37787/260736 objects misplaced (14.492%); 5.8 MiB/s, 92 objects/s recovering 2024-02-12T17:14:39.019 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:39.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:39.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:39.193 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:39.231 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:39.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:39.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:39.380 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:39.505 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:39.646 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:14:39.647 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:39.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:39.678 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:14:39.752 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:39.764 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:14:39.813 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:14:39.845 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:39.852 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:14:39.892 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:14:39.892 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:39.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:39.902 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:39.574158+0000 mon.b (mon.1) 115 : audit [DBG] from='client.? v1:172.21.15.33:0/2295990885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:39.902 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:39.574158+0000 mon.b (mon.1) 115 : audit [DBG] from='client.? v1:172.21.15.33:0/2295990885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:39.924 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:39.574158+0000 mon.b (mon.1) 115 : audit [DBG] from='client.? v1:172.21.15.33:0/2295990885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:39.939 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:14:39.968 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:14:39.999 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:40.014 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:14:40.015 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:14:40.015 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:14:40.015 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:14:40.015 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:14:40.015 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:14:40.015 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:14:40.015 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:14:40.045 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:40.194 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:40.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:40.221 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:14:40.221 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:14:40.300 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:40.382 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:39.038237+0000 mgr.y (mgr.24481) 940 : cluster [DBG] pgmap v1346: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 806 B/s rd, 0 B/s wr, 810 op/s; 39109/254604 objects degraded (15.361%); 36496/254604 objects misplaced (14.334%); 5.4 MiB/s, 85 objects/s recovering 2024-02-12T17:14:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:39.038237+0000 mgr.y (mgr.24481) 940 : cluster [DBG] pgmap v1346: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 806 B/s rd, 0 B/s wr, 810 op/s; 39109/254604 objects degraded (15.361%); 36496/254604 objects misplaced (14.334%); 5.4 MiB/s, 85 objects/s recovering 2024-02-12T17:14:40.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:39.038237+0000 mgr.y (mgr.24481) 940 : cluster [DBG] pgmap v1346: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 806 B/s rd, 0 B/s wr, 810 op/s; 39109/254604 objects degraded (15.361%); 36496/254604 objects misplaced (14.334%); 5.4 MiB/s, 85 objects/s recovering 2024-02-12T17:14:41.029 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T17:14:41.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:14:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:40] "GET /metrics HTTP/1.1" 200 176813 "" "Prometheus/2.33.4" 2024-02-12T17:14:41.491 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:41.776 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:41.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:41.887 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:41.939 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:41.042241+0000 mgr.y (mgr.24481) 941 : cluster [DBG] pgmap v1347: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 643 op/s; 39109/254604 objects degraded (15.361%); 36496/254604 objects misplaced (14.334%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:14:41.942 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:41.987 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:41.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:42.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:41.042241+0000 mgr.y (mgr.24481) 941 : cluster [DBG] pgmap v1347: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 643 op/s; 39109/254604 objects degraded (15.361%); 36496/254604 objects misplaced (14.334%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:14:42.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:41.042241+0000 mgr.y (mgr.24481) 941 : cluster [DBG] pgmap v1347: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 1 active+remapped, 100 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 643 op/s; 39109/254604 objects degraded (15.361%); 36496/254604 objects misplaced (14.334%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T17:14:42.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:14:42.138 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:42.347 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:14:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:42.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:14:42.453 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:42.493 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:42.541 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:42.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:42.651 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:14:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:14:42.654 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:41.667444+0000 mon.a (mon.0) 2164 : cluster [WRN] Health check update: Degraded data redundancy: 39109/254604 objects degraded (15.361%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:42.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:42.698 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:41.667444+0000 mon.a (mon.0) 2164 : cluster [WRN] Health check update: Degraded data redundancy: 39109/254604 objects degraded (15.361%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:43.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:41.667444+0000 mon.a (mon.0) 2164 : cluster [WRN] Health check update: Degraded data redundancy: 39109/254604 objects degraded (15.361%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:43.063 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:43.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:43.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:43.220 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:43.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:43.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:14:43.453 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:43.579 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:14:43.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:43.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:14:43.635 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:14:43.635 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:14:43.684 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:43.745 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:43.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:43.047275+0000 mgr.y (mgr.24481) 942 : cluster [DBG] pgmap v1348: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 613 B/s rd, 0 B/s wr, 529 op/s; 38409/250854 objects degraded (15.311%); 36253/250854 objects misplaced (14.452%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:14:44.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:43.047275+0000 mgr.y (mgr.24481) 942 : cluster [DBG] pgmap v1348: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 613 B/s rd, 0 B/s wr, 529 op/s; 38409/250854 objects degraded (15.311%); 36253/250854 objects misplaced (14.452%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:14:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:43.047275+0000 mgr.y (mgr.24481) 942 : cluster [DBG] pgmap v1348: 118 pgs: 1 active+undersized+degraded+remapped, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 613 B/s rd, 0 B/s wr, 529 op/s; 38409/250854 objects degraded (15.311%); 36253/250854 objects misplaced (14.452%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:14:44.464 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T17:14:44.855 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:45.028 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:45.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:45.134 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:45.172 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:45.205 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:45.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:45.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:45.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:14:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:14:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:45.393 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:45.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:45.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:45.549 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:45.759 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:45.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:45.863 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:45.944 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:45.999 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:46.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:46.106 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:46.187 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:46.219 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:14:46.222 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T17:14:46.253 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:46.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:46.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:46.460 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:45.001237+0000 mgr.y (mgr.24481) 943 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:46.460 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:45.050323+0000 mgr.y (mgr.24481) 944 : cluster [DBG] pgmap v1349: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 653 op/s; 37416/244500 objects degraded (15.303%); 34904/244500 objects misplaced (14.276%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:14:46.460 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:45.100867+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:46.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:45.100931+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:46.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:45.173841+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-12T17:14:46.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:45.431970+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.33:0/2470971782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:14:46.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:45.432667+0000 mon.a (mon.0) 2166 : audit [INF] from='client.36158 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:14:46.461 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:46.125817+0000 mon.a (mon.0) 2167 : audit [INF] from='client.? v1:172.21.15.33:0/1258297807' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:14:46.461 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:45.001237+0000 mgr.y (mgr.24481) 943 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:46.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:45.050323+0000 mgr.y (mgr.24481) 944 : cluster [DBG] pgmap v1349: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 653 op/s; 37416/244500 objects degraded (15.303%); 34904/244500 objects misplaced (14.276%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:14:46.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:45.100867+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:46.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:45.100931+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:46.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:45.173841+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-12T17:14:46.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:45.431970+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.33:0/2470971782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:14:46.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:45.432667+0000 mon.a (mon.0) 2166 : audit [INF] from='client.36158 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:14:46.462 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:46.125817+0000 mon.a (mon.0) 2167 : audit [INF] from='client.? v1:172.21.15.33:0/1258297807' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:14:46.472 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:46.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:45.001237+0000 mgr.y (mgr.24481) 943 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:46.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:45.050323+0000 mgr.y (mgr.24481) 944 : cluster [DBG] pgmap v1349: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 653 op/s; 37416/244500 objects degraded (15.303%); 34904/244500 objects misplaced (14.276%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:14:46.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:45.100867+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:46.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:45.100931+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:46.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:45.173841+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-12T17:14:46.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:45.431970+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.33:0/2470971782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:14:46.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:45.432667+0000 mon.a (mon.0) 2166 : audit [INF] from='client.36158 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:14:46.493 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:46.125817+0000 mon.a (mon.0) 2167 : audit [INF] from='client.? v1:172.21.15.33:0/1258297807' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:14:46.509 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:46.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:46.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:46.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:14:46.717 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:46.769 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:46.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:46.876 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:46.944 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:46.958 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:14:46.959 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:14:47.033 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:47.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:47.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:47.192 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:47.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:46.179262+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:47.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:46.186806+0000 mon.a (mon.0) 2169 : audit [INF] from='client.36158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:14:47.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:46.186873+0000 mon.a (mon.0) 2170 : audit [INF] from='client.? v1:172.21.15.33:0/1258297807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:14:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:46.186915+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-12T17:14:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:46.675778+0000 mon.a (mon.0) 2172 : cluster [WRN] Health check update: Degraded data redundancy: 37416/244500 objects degraded (15.303%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:46.893318+0000 osd.0 (osd.0) 234 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:46.893377+0000 osd.0 (osd.0) 235 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:46.179262+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:46.186806+0000 mon.a (mon.0) 2169 : audit [INF] from='client.36158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:14:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:46.186873+0000 mon.a (mon.0) 2170 : audit [INF] from='client.? v1:172.21.15.33:0/1258297807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:14:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:46.186915+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-12T17:14:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:46.675778+0000 mon.a (mon.0) 2172 : cluster [WRN] Health check update: Degraded data redundancy: 37416/244500 objects degraded (15.303%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:46.893318+0000 osd.0 (osd.0) 234 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:46.893377+0000 osd.0 (osd.0) 235 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:46.179262+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:14:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:46.186806+0000 mon.a (mon.0) 2169 : audit [INF] from='client.36158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:14:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:46.186873+0000 mon.a (mon.0) 2170 : audit [INF] from='client.? v1:172.21.15.33:0/1258297807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:14:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:46.186915+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-12T17:14:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:46.675778+0000 mon.a (mon.0) 2172 : cluster [WRN] Health check update: Degraded data redundancy: 37416/244500 objects degraded (15.303%), 8 pgs degraded (PG_DEGRADED) 2024-02-12T17:14:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:46.893318+0000 osd.0 (osd.0) 234 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:14:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:46.893377+0000 osd.0 (osd.0) 235 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:14:47.767 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T17:14:48.172 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:48.174 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 3 1 2024-02-12T17:14:48.175 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:48.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:48.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:48.453 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:48.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:48.528 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:47.051003+0000 mgr.y (mgr.24481) 945 : cluster [DBG] pgmap v1352: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 580 op/s; 36728/240678 objects degraded (15.260%); 34644/240678 objects misplaced (14.394%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:14:48.528 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:47.699892+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e677: 8 total, 8 up, 7 in 2024-02-12T17:14:48.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:47.051003+0000 mgr.y (mgr.24481) 945 : cluster [DBG] pgmap v1352: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 580 op/s; 36728/240678 objects degraded (15.260%); 34644/240678 objects misplaced (14.394%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:14:48.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:47.699892+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e677: 8 total, 8 up, 7 in 2024-02-12T17:14:48.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:47.051003+0000 mgr.y (mgr.24481) 945 : cluster [DBG] pgmap v1352: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 580 op/s; 36728/240678 objects degraded (15.260%); 34644/240678 objects misplaced (14.394%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:14:48.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:47.699892+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e677: 8 total, 8 up, 7 in 2024-02-12T17:14:48.562 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:14:48.606 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:48.717 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:14:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:48.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:14:48.824 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:48.867 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:48.980 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:49.279 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:49.637 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:49.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:49.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:50.218 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:14:50.218 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:14:50.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:49.052123+0000 mgr.y (mgr.24481) 946 : cluster [DBG] pgmap v1354: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 906 op/s; 35784/234531 objects degraded (15.258%); 33397/234531 objects misplaced (14.240%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:14:50.339 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:50.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:50.448 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:14:50.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:49.052123+0000 mgr.y (mgr.24481) 946 : cluster [DBG] pgmap v1354: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 906 op/s; 35784/234531 objects degraded (15.258%); 33397/234531 objects misplaced (14.240%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:14:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:49.052123+0000 mgr.y (mgr.24481) 946 : cluster [DBG] pgmap v1354: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 906 op/s; 35784/234531 objects degraded (15.258%); 33397/234531 objects misplaced (14.240%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:14:50.537 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:50.883 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:14:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:50] "GET /metrics HTTP/1.1" 200 176827 "" "Prometheus/2.33.4" 2024-02-12T17:14:50.925 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:14:50.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T17:14:51.358 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:52.150 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:14:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:52.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:14:52.266 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:52.368 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:52.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:51.054916+0000 mgr.y (mgr.24481) 947 : cluster [DBG] pgmap v1355: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 554 op/s; 35784/234531 objects degraded (15.258%); 33397/234531 objects misplaced (14.240%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:14:52.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:51.678680+0000 mon.a (mon.0) 2174 : cluster [WRN] Health check update: Degraded data redundancy: 35784/234531 objects degraded (15.258%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:14:52.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:51.704806+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2024-02-12T17:14:52.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:51.054916+0000 mgr.y (mgr.24481) 947 : cluster [DBG] pgmap v1355: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 554 op/s; 35784/234531 objects degraded (15.258%); 33397/234531 objects misplaced (14.240%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:14:52.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:51.678680+0000 mon.a (mon.0) 2174 : cluster [WRN] Health check update: Degraded data redundancy: 35784/234531 objects degraded (15.258%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:14:52.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:51.704806+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2024-02-12T17:14:52.661 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:52.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:52.675 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:52.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:14:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:14:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:14:52.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:51.054916+0000 mgr.y (mgr.24481) 947 : cluster [DBG] pgmap v1355: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 554 op/s; 35784/234531 objects degraded (15.258%); 33397/234531 objects misplaced (14.240%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:14:52.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:51.678680+0000 mon.a (mon.0) 2174 : cluster [WRN] Health check update: Degraded data redundancy: 35784/234531 objects degraded (15.258%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:14:52.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:51.704806+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2024-02-12T17:14:52.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:52.876 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:53.518 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:14:53.518 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:14:54.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T17:14:54.365 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.3 to 1 (10000) 2024-02-12T17:14:54.523 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:14:54.559 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:14:54.615 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:14:54.678 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:14:54.697 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:54.721 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:14:54.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:53.055324+0000 mgr.y (mgr.24481) 948 : cluster [DBG] pgmap v1357: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 672 op/s; 35072/230664 objects degraded (15.205%); 33148/230664 objects misplaced (14.371%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T17:14:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:53.309567+0000 mon.c (mon.2) 307 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:14:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:53.778388+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.33:0/1794798084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:14:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:53.783059+0000 mon.a (mon.0) 2176 : audit [INF] from='client.26035 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:14:54.771 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:14:54.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:53.055324+0000 mgr.y (mgr.24481) 948 : cluster [DBG] pgmap v1357: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 672 op/s; 35072/230664 objects degraded (15.205%); 33148/230664 objects misplaced (14.371%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T17:14:54.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:53.309567+0000 mon.c (mon.2) 307 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:14:54.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:53.778388+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.33:0/1794798084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:14:54.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:53.783059+0000 mon.a (mon.0) 2176 : audit [INF] from='client.26035 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:14:54.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:53.055324+0000 mgr.y (mgr.24481) 948 : cluster [DBG] pgmap v1357: 118 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 672 op/s; 35072/230664 objects degraded (15.205%); 33148/230664 objects misplaced (14.371%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T17:14:54.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:53.309567+0000 mon.c (mon.2) 307 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:14:54.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:53.778388+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.33:0/1794798084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:14:54.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:53.783059+0000 mon.a (mon.0) 2176 : audit [INF] from='client.26035 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:14:54.803 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:14:54.841 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:14:54.875 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:14:54.875 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:14:54.875 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:14:54.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:14:54.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:14:54.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:14:54.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:14:54.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:14:55.269 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:55.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:55.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:55.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:14:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:14:55.304 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-12T17:14:55.379 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:55.496 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:55.550 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:14:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:55.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:14:55.563 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:54.303769+0000 mon.a (mon.0) 2177 : audit [INF] from='client.26035 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:14:55.563 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:54.303838+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e679: 8 total, 8 up, 7 in 2024-02-12T17:14:55.563 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:54.476262+0000 mon.c (mon.2) 308 : audit [DBG] from='client.? v1:172.21.15.33:0/647359312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:55.564 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:54.815488+0000 mon.c (mon.2) 309 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:14:55.564 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:54.830241+0000 mon.c (mon.2) 310 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:14:55.564 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:54.860601+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:55.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:54.303769+0000 mon.a (mon.0) 2177 : audit [INF] from='client.26035 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:14:55.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:54.303838+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e679: 8 total, 8 up, 7 in 2024-02-12T17:14:55.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:54.476262+0000 mon.c (mon.2) 308 : audit [DBG] from='client.? v1:172.21.15.33:0/647359312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:55.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:54.815488+0000 mon.c (mon.2) 309 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:14:55.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:54.830241+0000 mon.c (mon.2) 310 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:14:55.566 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:54.860601+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:55.660 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:14:55.707 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:54.303769+0000 mon.a (mon.0) 2177 : audit [INF] from='client.26035 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:14:55.707 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:54.303838+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e679: 8 total, 8 up, 7 in 2024-02-12T17:14:55.707 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:54.476262+0000 mon.c (mon.2) 308 : audit [DBG] from='client.? v1:172.21.15.33:0/647359312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:14:55.707 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:54.815488+0000 mon.c (mon.2) 309 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:14:55.707 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:54.830241+0000 mon.c (mon.2) 310 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:14:55.708 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:54.860601+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:14:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:56.078 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:14:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:56.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:14:56.189 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:56.279 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:56.386 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:56.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:56.495 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:14:56.543 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:56.627 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:14:56.630 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:55.015670+0000 mgr.y (mgr.24481) 949 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:56.630 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:55.058907+0000 mgr.y (mgr.24481) 950 : cluster [DBG] pgmap v1359: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 738 op/s; 34063/224406 objects degraded (15.179%); 31830/224406 objects misplaced (14.184%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:14:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:56.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:14:56.680 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:14:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-02-12T17:14:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-02-12T17:14:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:191955 10.d 10:b6526e74:::benchmark_data_smithi033_160181_object68783:head [delete] snapc 0=[] ondisk+write+known_if_redirected e527)", 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:52.564883+0000", 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 304.11140779300001, 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.005525148, 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 191955 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.564883+0000", 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.564883+0000", 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.564880+0000", 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0900000000000005e-06 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.564888+0000", 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5560999999999999e-05 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.564904+0000", 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6850000000000004e-06 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.564910+0000", 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5602000000000001e-05 2024-02-12T17:14:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.564956+0000", 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3089999999999998e-06 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.564963+0000", 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98631608100000001 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.551279+0000", 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010980399999999999 2024-02-12T17:14:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.551389+0000", 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000277519 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 3,7", 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.551667+0000", 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092235570000000003 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.560890+0000", 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072327479999999998 2024-02-12T17:14:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.568123+0000", 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022039469999999999 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.570327+0000", 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3501999999999998e-05 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.570370+0000", 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7375e-05 2024-02-12T17:14:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.570408+0000", 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.005525148 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:192026 10.d 10:b2bcc46f:::benchmark_data_smithi033_160181_object68854:head [delete] snapc 0=[] ondisk+write+known_if_redirected e527)", 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:09:52.717730+0000", 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 303.95856074099999, 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.85309621999999996, 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 192026 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.717730+0000", 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.717730+0000", 2024-02-12T17:14:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999971 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.717727+0000", 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0760000000000003e-06 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.717735+0000", 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6918999999999998e-05 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.717752+0000", 2024-02-12T17:14:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4549999999999998e-06 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.717759+0000", 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6638999999999998e-05 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.717806+0000", 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4660000000000005e-06 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:52.717816+0000", 2024-02-12T17:14:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83529187500000002 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.553107+0000", 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6137000000000005e-05 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.553204+0000", 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179416 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 3,7", 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.553383+0000", 2024-02-12T17:14:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075805459999999996 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.560964+0000", 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0083648209999999997 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.569328+0000", 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001413011 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.570741+0000", 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5917000000000001e-05 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.570787+0000", 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8584999999999997e-05 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:09:53.570826+0000", 2024-02-12T17:14:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.85309621999999996 2024-02-12T17:14:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:246173 10.1d 10:bf850e6e:::benchmark_data_smithi033_160181_object123001:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e569)", 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:20.150611+0000", 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.52567937399999, 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0609963950000001, 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 246173 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.150611+0000", 2024-02-12T17:14:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.150611+0000", 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.150609+0000", 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6399999999999997e-06 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.150616+0000", 2024-02-12T17:14:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3988999999999999e-05 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.150630+0000", 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0430000000000002e-06 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.150637+0000", 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048623299999999998 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151123+0000", 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3599999999999998e-06 2024-02-12T17:14:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.151131+0000", 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015739893000000001 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.166871+0000", 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.418e-06 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.166878+0000", 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0446638580000001 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.211542+0000", 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5462000000000004e-05 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.211607+0000", 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0609963950000001 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:246211 10.1d 10:b8079877:::benchmark_data_smithi033_160181_object123039:head [delete] snapc 0=[] ondisk+write+known_if_redirected e569)", 2024-02-12T17:14:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:20.279241+0000", 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.397049447, 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96402668400000002, 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 246211 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.279241+0000", 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.279241+0000", 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.279239+0000", 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8989999999999996e-06 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.279249+0000", 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7440000000000002e-06 2024-02-12T17:14:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.279259+0000", 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6099999999999999e-06 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.279263+0000", 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4159000000000002e-05 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.279308+0000", 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5740000000000001e-06 2024-02-12T17:14:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.279314+0000", 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93283547899999997 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.212150+0000", 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9770000000000002e-05 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.212249+0000", 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00035051199999999998 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.212600+0000", 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0098279590000000007 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.222428+0000", 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011152308 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.233580+0000", 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095328359999999994 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243113+0000", 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8189999999999998e-05 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243161+0000", 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106462 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243268+0000", 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96402668400000002 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:246217 10.1d 10:bdd8e4af:::benchmark_data_smithi033_160181_object123045:head [delete] snapc 0=[] ondisk+write+known_if_redirected e569)", 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:20.284975+0000", 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.39131538500001, 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95846038099999997, 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 246217 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.284975+0000", 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.284975+0000", 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.284974+0000", 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3900000000000002e-06 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.284978+0000", 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0839999999999999e-06 2024-02-12T17:14:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.284983+0000", 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1239999999999998e-06 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.284987+0000", 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5991999999999997e-05 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.285023+0000", 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0108e-05 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.285033+0000", 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.92926335800000004 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.214296+0000", 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4234999999999995e-05 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.214391+0000", 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021193199999999999 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.214603+0000", 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0078313129999999995 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.222434+0000", 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011177631 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.233611+0000", 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0096894359999999992 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243301+0000", 2024-02-12T17:14:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1236999999999998e-05 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243342+0000", 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3281000000000004e-05 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243435+0000", 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.95846038099999997 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:246221 10.1d 10:bb0a1227:::benchmark_data_smithi033_160181_object123049:head [delete] snapc 0=[] ondisk+write+known_if_redirected e569)", 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:20.294716+0000", 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.38157491699999, 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94888310799999998, 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 246221 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.294716+0000", 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.294716+0000", 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.294715+0000", 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8279999999999997e-06 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.294722+0000", 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5399999999999997e-06 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.294726+0000", 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8720000000000002e-06 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.294729+0000", 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9612e-05 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.294769+0000", 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4370000000000004e-06 2024-02-12T17:14:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.294775+0000", 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.92118047599999997 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.215955+0000", 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4351000000000005e-05 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.216050+0000", 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021433800000000001 2024-02-12T17:14:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,5", 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.216264+0000", 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00617528 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.222439+0000", 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011195164000000001 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.233635+0000", 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0098316189999999998 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243466+0000", 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9477999999999997e-05 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243506+0000", 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2905999999999994e-05 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.243599+0000", 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94888310799999998 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.25603.0:246255 10.1c 10:39d9e6aa:::benchmark_data_smithi033_160181_object123083:head [delete] snapc 0=[] ondisk+write+known_if_redirected e569)", 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:11:20.363547+0000", 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.31274337100001, 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87286993099999999, 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.25603", 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/3173055907", 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 246255 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.363547+0000", 2024-02-12T17:14:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.363547+0000", 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.363546+0000", 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3079999999999998e-06 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.363551+0000", 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9420000000000007e-06 2024-02-12T17:14:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.363561+0000", 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7119999999999999e-06 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.363566+0000", 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5770000000000001e-05 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.363641+0000", 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2910000000000004e-06 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:20.363648+0000", 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84370628800000003 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.207354+0000", 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012389000000000001 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.207478+0000", 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000391708 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,7", 2024-02-12T17:14:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.207870+0000", 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014371261 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.222241+0000", 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090909830000000004 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.231332+0000", 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0049113969999999996 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.236243+0000", 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6230000000000002e-05 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.236299+0000", 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011758 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:11:21.236417+0000", 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87286993099999999 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:146795 11.1 11:846d59f6:::benchmark_data_smithi033_188107_object21105:head [delete] snapc 0=[] ondisk+write+known_if_redirected e650)", 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:07.690117+0000", 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.986173219999998, 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1163995679999998, 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146795 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.690117+0000", 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.690117+0000", 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.690115+0000", 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1720000000000001e-06 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.690122+0000", 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1391e-05 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.690134+0000", 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2340000000000002e-06 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.690139+0000", 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061448999999999998 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.690753+0000", 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120596 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.690874+0000", 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000134191 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,3", 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.691008+0000", 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030931370000000001 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.694101+0000", 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012506888000000001 2024-02-12T17:14:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.706608+0000", 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.721e-06 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.706617+0000", 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.99849447899999999 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705111+0000", 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2867000000000001e-05 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705124+0000", 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.087673779 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.792798+0000", 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5685000000000003e-05 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for missing object", 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.792894+0000", 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013527834000000001 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.806422+0000", 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5270000000000001e-05 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.806517+0000", 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1163995679999998 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:146810 11.1 11:83d44dd0:::benchmark_data_smithi033_188107_object21120:head [delete] snapc 0=[] ondisk+write+known_if_redirected e651)", 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:07.719456+0000", 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.956834673000003, 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0961500759999998, 2024-02-12T17:14:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146810 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.719456+0000", 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.719456+0000", 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.719454+0000", 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9070000000000001e-06 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.719460+0000", 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1693e-05 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.719472+0000", 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1180000000000003e-06 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.719479+0000", 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5485999999999998e-05 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.719525+0000", 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8389999999999996e-06 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.719534+0000", 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98561511300000004 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705149+0000", 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7260000000000002e-06 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705156+0000", 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0879546659999999 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.793111+0000", 2024-02-12T17:14:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2150000000000004e-05 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.793203+0000", 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2332999999999998e-05 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,3", 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.793296+0000", 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047728650000000003 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.798068+0000", 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5024000000000002e-05 2024-02-12T17:14:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.798103+0000", 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0084346479999999995 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.806538+0000", 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0089459090000000002 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.815484+0000", 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5368999999999999e-05 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.815529+0000", 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6410999999999997e-05 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.815606+0000", 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0961500759999998 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:146816 11.1 11:8b576d6a:::benchmark_data_smithi033_188107_object21126:head [delete] snapc 0=[] ondisk+write+known_if_redirected e651)", 2024-02-12T17:14:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:07.725866+0000", 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.950424001000002, 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1015891390000001, 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146816 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.725866+0000", 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.725866+0000", 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999995 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.725866+0000", 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0290000000000001e-06 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.725869+0000", 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1359999999999996e-06 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.725874+0000", 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999998e-06 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.725878+0000", 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3424999999999998e-05 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.725922+0000", 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2690000000000005e-06 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.725931+0000", 2024-02-12T17:14:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97925692099999995 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705188+0000", 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9709999999999993e-06 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705197+0000", 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.090073834 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.795271+0000", 2024-02-12T17:14:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7908e-05 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.795349+0000", 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000200562 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,3", 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.795549+0000", 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0039960600000000001 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.799545+0000", 2024-02-12T17:14:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016764049 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.816309+0000", 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075865700000000004 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.817068+0000", 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010202330000000001 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827270+0000", 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1588000000000001e-05 2024-02-12T17:14:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827322+0000", 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013373899999999999 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827456+0000", 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1015891390000001 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:146823 11.1 11:8112fd3b:::benchmark_data_smithi033_188107_object21133:head [delete] snapc 0=[] ondisk+write+known_if_redirected e651)", 2024-02-12T17:14:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:07.739466+0000", 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.936824270000002, 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.088145704, 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146823 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.739466+0000", 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.739466+0000", 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.739464+0000", 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2230000000000001e-06 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.739471+0000", 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2354000000000001e-05 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.739483+0000", 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9699999999999996e-06 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.739489+0000", 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1625000000000002e-05 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.739531+0000", 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5599999999999999e-06 2024-02-12T17:14:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.739538+0000", 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96569379 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705231+0000", 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5199999999999997e-06 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705240+0000", 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0915566720000001 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.796797+0000", 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2193999999999994e-05 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.796869+0000", 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0448e-05 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,3", 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.796949+0000", 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052663220000000004 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.802216+0000", 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014143334 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.816359+0000", 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00074697800000000003 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.817106+0000", 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010392731000000001 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827499+0000", 2024-02-12T17:14:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6439000000000001e-05 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827535+0000", 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6797000000000003e-05 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827612+0000", 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.088145704 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:146824 11.1 11:8130f07e:::benchmark_data_smithi033_188107_object21134:head [delete] snapc 0=[] ondisk+write+known_if_redirected e651)", 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:07.740926+0000", 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.935364444999998, 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.086933368, 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 146824 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.740926+0000", 2024-02-12T17:14:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.740926+0000", 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.740925+0000", 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5759999999999999e-06 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.740929+0000", 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7779999999999996e-06 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.740938+0000", 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3409999999999996e-06 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.740943+0000", 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8435000000000001e-05 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.740971+0000", 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1440000000000002e-06 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:07.740976+0000", 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.96428572899999998 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705262+0000", 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5550000000000003e-06 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:08.705270+0000", 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0928571069999999 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.798128+0000", 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8110999999999998e-05 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.798206+0000", 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2608999999999995e-05 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0,2,3", 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.798288+0000", 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064388850000000001 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.804727+0000", 2024-02-12T17:14:56.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011653604999999999 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.816381+0000", 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076957299999999996 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.817150+0000", 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010524995000000001 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827675+0000", 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5108999999999997e-05 2024-02-12T17:14:56.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827730+0000", 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892900000000001 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:09.827859+0000", 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.086933368 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:162950 11.3 11:caea9473:::benchmark_data_smithi033_188107_object37260:head [delete] snapc 0=[] ondisk+write+known_if_redirected e670)", 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:31.671831+0000", 2024-02-12T17:14:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.004459925999999, 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0383080650000001, 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 162950 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.671831+0000", 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.671831+0000", 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.671830+0000", 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4809999999999998e-06 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.671833+0000", 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1019999999999996e-06 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.671838+0000", 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8430000000000001e-06 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.671841+0000", 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1353e-05 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.671862+0000", 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4494999999999997e-05 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.671937+0000", 2024-02-12T17:14:56.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012940900000000001 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,7", 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.672066+0000", 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0032582599999999998 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.675325+0000", 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6818999999999997e-05 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.675411+0000", 2024-02-12T17:14:56.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052453910000000003 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.680657+0000", 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0232e-05 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.680667+0000", 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.029425093 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.710092+0000", 2024-02-12T17:14:56.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6430000000000001e-05 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.710139+0000", 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0383080650000001 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:162976 11.3 11:c0d58bb7:::benchmark_data_smithi033_188107_object37286:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:31.738019+0000", 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.938270936999999, 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98113885899999997, 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.765 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 162976 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.738019+0000", 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.738019+0000", 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999999 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.738019+0000", 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0589999999999996e-06 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.738023+0000", 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7069999999999997e-06 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.738028+0000", 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9350000000000002e-06 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.738033+0000", 2024-02-12T17:14:56.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1214999999999998e-05 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.738074+0000", 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2220000000000002e-06 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.738083+0000", 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.97238838599999999 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.710471+0000", 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000132356 2024-02-12T17:14:56.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.710603+0000", 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3502000000000001e-05 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,7", 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.710687+0000", 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0083112629999999993 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.718998+0000", 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2537999999999998e-05 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719041+0000", 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5011999999999994e-05 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719106+0000", 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9958999999999999e-05 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719136+0000", 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2673000000000001e-05 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719158+0000", 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.98113885899999997 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:162991 11.3 11:c514435a:::benchmark_data_smithi033_188107_object37301:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:31.770929+0000", 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.905361785, 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94836378899999996, 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.771 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 162991 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.770929+0000", 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.770929+0000", 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.770926+0000", 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1732999999999999e-05 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.770938+0000", 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6877000000000001e-05 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.770955+0000", 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8059999999999997e-06 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.770962+0000", 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-05 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.771012+0000", 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.038e-06 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.771020+0000", 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94216705099999998 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.713187+0000", 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112688 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.713300+0000", 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000142829 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,7", 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.713443+0000", 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056330369999999996 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719076+0000", 2024-02-12T17:14:56.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9488999999999996e-05 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719165+0000", 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5276000000000001e-05 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719251+0000", 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2881999999999999e-05 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719273+0000", 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8977000000000001e-05 2024-02-12T17:14:56.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719292+0000", 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.94836378899999996 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:163004 11.3 11:c6eb52eb:::benchmark_data_smithi033_188107_object37314:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:31.783827+0000", 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.892463672000002, 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93559342000000001, 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.777 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 163004 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.783827+0000", 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.783827+0000", 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.783824+0000", 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.783832+0000", 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1948e-05 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.783844+0000", 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5519999999999999e-06 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.783851+0000", 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5927999999999998e-05 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.783897+0000", 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4960000000000002e-06 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.783907+0000", 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93232424999999997 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.716231+0000", 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000110888 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.716342+0000", 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014089899999999999 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,7", 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.716483+0000", 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026872810000000001 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719170+0000", 2024-02-12T17:14:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9932999999999998e-05 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719220+0000", 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111623 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719331+0000", 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7239e-05 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719379+0000", 2024-02-12T17:14:56.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1542000000000003e-05 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719420+0000", 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.93559342000000001 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:163045 11.3 11:c26c5b14:::benchmark_data_smithi033_188107_object37355:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:31.845464+0000", 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.830826308999999, 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.88680725599999999, 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.783 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 163045 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.845464+0000", 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.845464+0000", 2024-02-12T17:14:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.845462+0000", 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9670000000000001e-06 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.845470+0000", 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1558e-05 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.845481+0000", 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1839999999999998e-06 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.845487+0000", 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4566000000000001e-05 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.845521+0000", 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2020000000000001e-06 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.845527+0000", 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.87393487599999997 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719462+0000", 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111006 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719573+0000", 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013528100000000001 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,7", 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.719709+0000", 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012383198 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732092+0000", 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2648000000000003e-05 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732124+0000", 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9672000000000006e-05 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732204+0000", 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0531999999999998e-05 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732235+0000", 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6810000000000002e-05 2024-02-12T17:14:56.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732271+0000", 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.88680725599999999 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:163068 11.3 11:c14072ab:::benchmark_data_smithi033_188107_object37378:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:31.878647+0000", 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.797643667999999, 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.85377366200000004, 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.789 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 163068 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878647+0000", 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878647+0000", 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878644+0000", 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4869999999999996e-06 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878652+0000", 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2517000000000001e-05 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878664+0000", 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9160000000000004e-06 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878671+0000", 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6264000000000003e-05 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878718+0000", 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4560000000000006e-06 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878727+0000", 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84366044100000004 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.722387+0000", 2024-02-12T17:14:56.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8978e-05 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.722486+0000", 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012523699999999999 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,7", 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.722612+0000", 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095397339999999994 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732151+0000", 2024-02-12T17:14:56.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.443e-06 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732158+0000", 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018636299999999999 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732344+0000", 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9947000000000002e-05 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732384+0000", 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6251000000000002e-05 2024-02-12T17:14:56.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732420+0000", 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.85377366200000004 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:163072 11.3 11:c080d389:::benchmark_data_smithi033_188107_object37382:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:31.878945+0000", 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.797345932999999, 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.85356154699999998, 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 163072 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878945+0000", 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878945+0000", 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878943+0000", 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3289999999999994e-06 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878951+0000", 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4188999999999999e-05 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878965+0000", 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8709999999999999e-06 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.878972+0000", 2024-02-12T17:14:56.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7234e-05 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.879040+0000", 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.909e-06 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.879047+0000", 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.84582911000000005 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.724877+0000", 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7873999999999994e-05 2024-02-12T17:14:56.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.724944+0000", 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5875000000000001e-05 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,7", 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.725020+0000", 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0071435989999999996 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732164+0000", 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014436200000000001 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732308+0000", 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014342900000000001 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732452+0000", 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6483e-05 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732478+0000", 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7861999999999999e-05 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732506+0000", 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.85356154699999998 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:163091 11.3 11:c6ad9b96:::benchmark_data_smithi033_188107_object37401:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:14:31.909749+0000", 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.766540977999998, 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83074871699999997, 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 163091 2024-02-12T17:14:56.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.909749+0000", 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.909749+0000", 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999976 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.909747+0000", 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7090000000000002e-06 2024-02-12T17:14:56.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.909755+0000", 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2490000000000007e-06 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.909764+0000", 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9730000000000002e-06 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.909770+0000", 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8247000000000001e-05 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.909818+0000", 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7970000000000006e-06 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:31.909827+0000", 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.817384946 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.727212+0000", 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5484999999999999e-05 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:14:56.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.727298+0000", 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010953199999999999 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,7", 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.727407+0000", 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047623120000000003 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732169+0000", 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038505700000000001 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.732555+0000", 2024-02-12T17:14:56.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0078554450000000008 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.740410+0000", 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6776000000000001e-05 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.740457+0000", 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1436000000000001e-05 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:14:32.740498+0000", 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.83074871699999997 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.806 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:14:56.807 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:14:56.807 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:14:56.807 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:14:56.808 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:55.015670+0000 mgr.y (mgr.24481) 949 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:56.808 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:55.058907+0000 mgr.y (mgr.24481) 950 : cluster [DBG] pgmap v1359: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 738 op/s; 34063/224406 objects degraded (15.179%); 31830/224406 objects misplaced (14.184%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:14:56.808 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:55.015670+0000 mgr.y (mgr.24481) 949 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:14:56.808 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:55.058907+0000 mgr.y (mgr.24481) 950 : cluster [DBG] pgmap v1359: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 738 op/s; 34063/224406 objects degraded (15.179%); 31830/224406 objects misplaced (14.184%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:14:56.841 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:56.957 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:14:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:56.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:14:57.063 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:14:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:57.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T17:14:57.590 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:14:57.596 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:56.685542+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check update: Degraded data redundancy: 34063/224406 objects degraded (15.179%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:14:57.596 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:56.698636+0000 mon.a (mon.0) 2181 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2024-02-12T17:14:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:57.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:14:57.698 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:14:57.760 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:56.685542+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check update: Degraded data redundancy: 34063/224406 objects degraded (15.179%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:14:57.760 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:56.698636+0000 mon.a (mon.0) 2181 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2024-02-12T17:14:57.760 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:56.685542+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check update: Degraded data redundancy: 34063/224406 objects degraded (15.179%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:14:57.760 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:56.698636+0000 mon.a (mon.0) 2181 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2024-02-12T17:14:57.763 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:57.994 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:14:58.271 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:14:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:58.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:14:58.368 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:14:58.378 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:14:58.453 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:58.578 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:14:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:58.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:14:58.588 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:14:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:57.059316+0000 mgr.y (mgr.24481) 951 : cluster [DBG] pgmap v1361: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 762 op/s; 33357/220818 objects degraded (15.106%); 31594/220818 objects misplaced (14.308%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:14:58.588 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:14:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:57.059316+0000 mgr.y (mgr.24481) 951 : cluster [DBG] pgmap v1361: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 762 op/s; 33357/220818 objects degraded (15.106%); 31594/220818 objects misplaced (14.308%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:14:58.589 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:14:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:57.059316+0000 mgr.y (mgr.24481) 951 : cluster [DBG] pgmap v1361: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 762 op/s; 33357/220818 objects degraded (15.106%); 31594/220818 objects misplaced (14.308%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:14:58.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:14:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:58.861 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:14:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:14:58.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:14:58.965 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:14:59.071 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:14:59.450 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:15:00.087 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:15:00.087 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-02-12T17:15:00.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.087 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:181458 11.f 11:f1ea7f05:::benchmark_data_smithi033_188107_object55768:head [delete] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:15:00.074167+0000", 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0093709039999999993, 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0094014609999999998, 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 181458 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:15:00.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074167+0000", 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074167+0000", 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999986 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074165+0000", 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2045000000000001e-05 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:15:00.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074177+0000", 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.25e-06 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074185+0000", 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3039999999999998e-06 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074189+0000", 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9877000000000006e-05 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074289+0000", 2024-02-12T17:15:00.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109434 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074398+0000", 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1623e-05 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1,2", 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.074490+0000", 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026241160000000001 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.077114+0000", 2024-02-12T17:15:00.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099509100000000003 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078109+0000", 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0039422659999999998 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:181460 11.4 11:2aaff034:::benchmark_data_smithi033_188107_object55770:head [delete] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:15:00.078480+0000", 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.0050579149999999996, 2024-02-12T17:15:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051840590000000004, 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 181460 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078480+0000", 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078480+0000", 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078478+0000", 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3470000000000004e-06 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078484+0000", 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9760000000000007e-06 2024-02-12T17:15:00.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078494+0000", 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1404e-05 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078506+0000", 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2499000000000001e-05 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078548+0000", 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100652 2024-02-12T17:15:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078649+0000", 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6426000000000001e-05 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2,6", 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.078745+0000", 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003024905 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.081770+0000", 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0032902420000000001 2024-02-12T17:15:00.096 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.36023.0:181463 11.5 11:ac3c0593:::benchmark_data_smithi033_188107_object55773:head [delete] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-02-12T17:15:00.082531+0000", 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.001006404, 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001214262, 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.36023", 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:15:00.097 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 181463 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082531+0000", 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082531+0000", 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999981 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-02-12T17:15:00.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082530+0000", 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3210000000000001e-06 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082536+0000", 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0016e-05 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082546+0000", 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.434e-06 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082551+0000", 2024-02-12T17:15:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0713999999999997e-05 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082592+0000", 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9275000000000002e-05 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082671+0000", 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9952999999999998e-05 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 5,6", 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-02-12T17:15:00.082751+0000", 2024-02-12T17:15:00.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021992299999999999 2024-02-12T17:15:00.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:15:00.101 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-02-12T17:15:00.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:15:00.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-02-12T17:15:00.101 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-02-12T17:15:00.101 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 3 2024-02-12T17:15:00.101 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:15:00.389 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:15:00.400 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:14:59.061052+0000 mgr.y (mgr.24481) 952 : cluster [DBG] pgmap v1362: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 697 B/s rd, 0 B/s wr, 735 op/s; 32363/214485 objects degraded (15.089%); 30288/214485 objects misplaced (14.121%); 2.1 MiB/s, 34 objects/s recovering 2024-02-12T17:15:00.400 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:14:59.500516+0000 mon.a (mon.0) 2182 : audit [INF] from='client.? v1:172.21.15.33:0/2727367298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:14:59.061052+0000 mgr.y (mgr.24481) 952 : cluster [DBG] pgmap v1362: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 697 B/s rd, 0 B/s wr, 735 op/s; 32363/214485 objects degraded (15.089%); 30288/214485 objects misplaced (14.121%); 2.1 MiB/s, 34 objects/s recovering 2024-02-12T17:15:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:14:59.500516+0000 mon.a (mon.0) 2182 : audit [INF] from='client.? v1:172.21.15.33:0/2727367298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:14:59.061052+0000 mgr.y (mgr.24481) 952 : cluster [DBG] pgmap v1362: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 697 B/s rd, 0 B/s wr, 735 op/s; 32363/214485 objects degraded (15.089%); 30288/214485 objects misplaced (14.121%); 2.1 MiB/s, 34 objects/s recovering 2024-02-12T17:15:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:14:59.500516+0000 mon.a (mon.0) 2182 : audit [INF] from='client.? v1:172.21.15.33:0/2727367298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:00.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T17:15:01.178 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:15:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:00] "GET /metrics HTTP/1.1" 200 176827 "" "Prometheus/2.33.4" 2024-02-12T17:15:01.240 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:15:01.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:00.348896+0000 mon.a (mon.0) 2183 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:15:01.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:00.374020+0000 mon.a (mon.0) 2184 : audit [INF] from='client.? v1:172.21.15.33:0/2727367298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:15:01.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:00.374081+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e681: 8 total, 8 up, 7 in 2024-02-12T17:15:01.429 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:00.862261+0000 osd.5 (osd.5) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:01.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:00.348896+0000 mon.a (mon.0) 2183 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:15:01.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:00.374020+0000 mon.a (mon.0) 2184 : audit [INF] from='client.? v1:172.21.15.33:0/2727367298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:15:01.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:00.374081+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e681: 8 total, 8 up, 7 in 2024-02-12T17:15:01.429 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:00.862261+0000 osd.5 (osd.5) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:01.464 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:01.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:01.574 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:01.625 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:01.733 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:00.862317+0000 osd.5 (osd.5) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:01.734 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:00.862317+0000 osd.5 (osd.5) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:00.348896+0000 mon.a (mon.0) 2183 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:15:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:00.374020+0000 mon.a (mon.0) 2184 : audit [INF] from='client.? v1:172.21.15.33:0/2727367298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:15:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:00.374081+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e681: 8 total, 8 up, 7 in 2024-02-12T17:15:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:00.862261+0000 osd.5 (osd.5) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:00.862317+0000 osd.5 (osd.5) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:02.685 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:02.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:15:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:15:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:01.061604+0000 mgr.y (mgr.24481) 953 : cluster [DBG] pgmap v1364: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 491 op/s; 32363/214485 objects degraded (15.089%); 30288/214485 objects misplaced (14.121%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T17:15:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:01.427714+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e682: 8 total, 8 up, 7 in 2024-02-12T17:15:02.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:01.687469+0000 mon.a (mon.0) 2187 : cluster [WRN] Health check update: Degraded data redundancy: 32363/214485 objects degraded (15.089%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:02.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:01.061604+0000 mgr.y (mgr.24481) 953 : cluster [DBG] pgmap v1364: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 491 op/s; 32363/214485 objects degraded (15.089%); 30288/214485 objects misplaced (14.121%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T17:15:02.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:01.427714+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e682: 8 total, 8 up, 7 in 2024-02-12T17:15:02.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:01.687469+0000 mon.a (mon.0) 2187 : cluster [WRN] Health check update: Degraded data redundancy: 32363/214485 objects degraded (15.089%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:02.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:01.061604+0000 mgr.y (mgr.24481) 953 : cluster [DBG] pgmap v1364: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 491 op/s; 32363/214485 objects degraded (15.089%); 30288/214485 objects misplaced (14.121%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T17:15:02.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:01.427714+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e682: 8 total, 8 up, 7 in 2024-02-12T17:15:02.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:01.687469+0000 mon.a (mon.0) 2187 : cluster [WRN] Health check update: Degraded data redundancy: 32363/214485 objects degraded (15.089%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:02.987 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:02.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:03.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:03.141 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:03.255 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:03.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:03.289 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-02-12T17:15:03.289 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-02-12T17:15:03.289 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-02-12T17:15:03.289 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-02-12T17:15:03.289 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-02-12T17:15:03.363 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:03.453 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:03.595 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.1 to 1 (10000) 2024-02-12T17:15:03.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:02.546489+0000 mon.a (mon.0) 2188 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:03.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:02.600649+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? v1:172.21.15.33:0/408672230' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:15:03.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:02.546489+0000 mon.a (mon.0) 2188 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:03.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:02.600649+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? v1:172.21.15.33:0/408672230' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:15:03.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:02.546489+0000 mon.a (mon.0) 2188 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:03.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:02.600649+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? v1:172.21.15.33:0/408672230' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:15:04.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T17:15:04.536 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:15:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:15:04.858 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:03.066913+0000 mgr.y (mgr.24481) 954 : cluster [DBG] pgmap v1366: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 804 B/s rd, 0 B/s wr, 527 op/s; 31628/210771 objects degraded (15.006%); 30048/210771 objects misplaced (14.256%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:04.859 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:03.577570+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.33:0/408672230' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:15:04.859 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:03.577646+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e683: 8 total, 8 up, 7 in 2024-02-12T17:15:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:03.066913+0000 mgr.y (mgr.24481) 954 : cluster [DBG] pgmap v1366: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 804 B/s rd, 0 B/s wr, 527 op/s; 31628/210771 objects degraded (15.006%); 30048/210771 objects misplaced (14.256%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:03.577570+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.33:0/408672230' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:15:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:03.577646+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e683: 8 total, 8 up, 7 in 2024-02-12T17:15:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:03.066913+0000 mgr.y (mgr.24481) 954 : cluster [DBG] pgmap v1366: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 804 B/s rd, 0 B/s wr, 527 op/s; 31628/210771 objects degraded (15.006%); 30048/210771 objects misplaced (14.256%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:03.577570+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.33:0/408672230' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:15:05.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:03.577646+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e683: 8 total, 8 up, 7 in 2024-02-12T17:15:05.097 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-12T17:15:05.129 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:15:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:15:05.348 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:05.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:05.464 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:05.518 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:05.579 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:05.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:05.684 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:05.752 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:05.788 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:05.818 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:15:05.832 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:05.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:05.882 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:15:05.929 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:15:05.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:05.955 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:15:05.974 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:05.987 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:15:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:06.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:06.036 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:15:06.061 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:15:06.114 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:06.123 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:15:06.131 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:15:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:15:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:15:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:15:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:15:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:15:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:15:06.132 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:15:06.150 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:06.232 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:06.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:06.337 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:15:06.436 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:06.590 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:15:06.590 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:15:06.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:05.025871+0000 mgr.y (mgr.24481) 955 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:06.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:05.067712+0000 mgr.y (mgr.24481) 956 : cluster [DBG] pgmap v1368: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 549 op/s; 30635/204609 objects degraded (14.972%); 28792/204609 objects misplaced (14.072%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:15:06.787 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:05.766059+0000 mon.b (mon.1) 117 : audit [DBG] from='client.? v1:172.21.15.33:0/3139086553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:06.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:05.025871+0000 mgr.y (mgr.24481) 955 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:06.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:05.067712+0000 mgr.y (mgr.24481) 956 : cluster [DBG] pgmap v1368: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 549 op/s; 30635/204609 objects degraded (14.972%); 28792/204609 objects misplaced (14.072%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:15:06.788 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:05.766059+0000 mon.b (mon.1) 117 : audit [DBG] from='client.? v1:172.21.15.33:0/3139086553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:06.965 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:05.025871+0000 mgr.y (mgr.24481) 955 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:06.965 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:05.067712+0000 mgr.y (mgr.24481) 956 : cluster [DBG] pgmap v1368: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 549 op/s; 30635/204609 objects degraded (14.972%); 28792/204609 objects misplaced (14.072%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:15:06.966 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:05.766059+0000 mon.b (mon.1) 117 : audit [DBG] from='client.? v1:172.21.15.33:0/3139086553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:07.101 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:15:07.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:07.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:15:07.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:07.247 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:07.332 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:07.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:07.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T17:15:07.436 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:07.469 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:07.489 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:07.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:07.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:07.610 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:07.628 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:07.668 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:07.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:07.771 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:06.688892+0000 mon.a (mon.0) 2192 : cluster [WRN] Health check update: Degraded data redundancy: 30635/204609 objects degraded (14.972%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:06.711163+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e684: 8 total, 8 up, 7 in 2024-02-12T17:15:07.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:06.688892+0000 mon.a (mon.0) 2192 : cluster [WRN] Health check update: Degraded data redundancy: 30635/204609 objects degraded (14.972%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:07.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:06.711163+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e684: 8 total, 8 up, 7 in 2024-02-12T17:15:07.844 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:07.932 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:07.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:07.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:06.688892+0000 mon.a (mon.0) 2192 : cluster [WRN] Health check update: Degraded data redundancy: 30635/204609 objects degraded (14.972%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:06.711163+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e684: 8 total, 8 up, 7 in 2024-02-12T17:15:08.039 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:08.081 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:08.107 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:08.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:08.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:08.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:08.275 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:08.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:08.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:08.469 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:08.708 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:08.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:08.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:07.070925+0000 mgr.y (mgr.24481) 957 : cluster [DBG] pgmap v1370: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s; 29932/200940 objects degraded (14.896%); 28566/200940 objects misplaced (14.216%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:15:08.718 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:07.070925+0000 mgr.y (mgr.24481) 957 : cluster [DBG] pgmap v1370: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s; 29932/200940 objects degraded (14.896%); 28566/200940 objects misplaced (14.216%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:15:08.814 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:08.848 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:08.880 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:08.882 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:07.070925+0000 mgr.y (mgr.24481) 957 : cluster [DBG] pgmap v1370: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s; 29932/200940 objects degraded (14.896%); 28566/200940 objects misplaced (14.216%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T17:15:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:08.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:08.985 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:09.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:15:09.060 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:09.100 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:09.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:09.209 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:09.237 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:09.260 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:09.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:09.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:09.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:09.528 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:10.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:09.072104+0000 mgr.y (mgr.24481) 958 : cluster [DBG] pgmap v1371: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 670 B/s rd, 0 B/s wr, 852 op/s; 28962/194958 objects degraded (14.856%); 27326/194958 objects misplaced (14.016%); 3.3 MiB/s, 53 objects/s recovering 2024-02-12T17:15:11.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:09.072104+0000 mgr.y (mgr.24481) 958 : cluster [DBG] pgmap v1371: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 670 B/s rd, 0 B/s wr, 852 op/s; 28962/194958 objects degraded (14.856%); 27326/194958 objects misplaced (14.016%); 3.3 MiB/s, 53 objects/s recovering 2024-02-12T17:15:11.035 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:15:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:10] "GET /metrics HTTP/1.1" 200 176828 "" "Prometheus/2.33.4" 2024-02-12T17:15:11.035 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:09.072104+0000 mgr.y (mgr.24481) 958 : cluster [DBG] pgmap v1371: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 670 B/s rd, 0 B/s wr, 852 op/s; 28962/194958 objects degraded (14.856%); 27326/194958 objects misplaced (14.016%); 3.3 MiB/s, 53 objects/s recovering 2024-02-12T17:15:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:10.807678+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.33:0/1932969655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:15:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:10.810124+0000 mon.a (mon.0) 2194 : audit [INF] from='client.26050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:15:12.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:10.807678+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.33:0/1932969655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:15:12.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:10.810124+0000 mon.a (mon.0) 2194 : audit [INF] from='client.26050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:15:12.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:10.807678+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.33:0/1932969655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:15:12.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:10.810124+0000 mon.a (mon.0) 2194 : audit [INF] from='client.26050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:15:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:15:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:15:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:11.072814+0000 mgr.y (mgr.24481) 959 : cluster [DBG] pgmap v1372: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 659 op/s; 28962/194958 objects degraded (14.856%); 27326/194958 objects misplaced (14.016%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:11.641704+0000 mon.a (mon.0) 2195 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:11.660002+0000 mon.a (mon.0) 2196 : audit [INF] from='client.26050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:15:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:11.660061+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2024-02-12T17:15:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:11.692500+0000 mon.a (mon.0) 2198 : cluster [WRN] Health check update: Degraded data redundancy: 28962/194958 objects degraded (14.856%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:11.722299+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2024-02-12T17:15:12.999 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:15:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:11.072814+0000 mgr.y (mgr.24481) 959 : cluster [DBG] pgmap v1372: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 659 op/s; 28962/194958 objects degraded (14.856%); 27326/194958 objects misplaced (14.016%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:11.641704+0000 mon.a (mon.0) 2195 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:11.660002+0000 mon.a (mon.0) 2196 : audit [INF] from='client.26050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:15:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:11.660061+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2024-02-12T17:15:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:11.692500+0000 mon.a (mon.0) 2198 : cluster [WRN] Health check update: Degraded data redundancy: 28962/194958 objects degraded (14.856%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:11.722299+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2024-02-12T17:15:13.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:11.072814+0000 mgr.y (mgr.24481) 959 : cluster [DBG] pgmap v1372: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 659 op/s; 28962/194958 objects degraded (14.856%); 27326/194958 objects misplaced (14.016%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:13.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:11.641704+0000 mon.a (mon.0) 2195 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:13.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:11.660002+0000 mon.a (mon.0) 2196 : audit [INF] from='client.26050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:15:13.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:11.660061+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2024-02-12T17:15:13.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:11.692500+0000 mon.a (mon.0) 2198 : cluster [WRN] Health check update: Degraded data redundancy: 28962/194958 objects degraded (14.856%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:13.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:11.722299+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2024-02-12T17:15:13.366 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:13.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:13.073372+0000 mgr.y (mgr.24481) 960 : cluster [DBG] pgmap v1375: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 520 op/s; 28262/191043 objects degraded (14.794%); 27083/191043 objects misplaced (14.176%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:14.002 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:13.073372+0000 mgr.y (mgr.24481) 960 : cluster [DBG] pgmap v1375: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 520 op/s; 28262/191043 objects degraded (14.794%); 27083/191043 objects misplaced (14.176%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:14.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:13.073372+0000 mgr.y (mgr.24481) 960 : cluster [DBG] pgmap v1375: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 520 op/s; 28262/191043 objects degraded (14.794%); 27083/191043 objects misplaced (14.176%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:14.249 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:14.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:14.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:14.296 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:14.357 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:14.433 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:14.904 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:14.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:15.017 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:15.065 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:15.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:15.200 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:15.289 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:15.340 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:15.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:15.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:15.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:15:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:15:15.490 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:15.523 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:15.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:15.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:15.732 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:16.027 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:15:16.215 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.6 to 1 (10000) 2024-02-12T17:15:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:15.036131+0000 mgr.y (mgr.24481) 961 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:15.077075+0000 mgr.y (mgr.24481) 962 : cluster [DBG] pgmap v1376: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 27296/184758 objects degraded (14.774%); 25785/184758 objects misplaced (13.956%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:15.189410+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.33:0/3110608700' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:15:16.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:15.199403+0000 mon.a (mon.0) 2200 : audit [INF] from='client.26056 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:15:16.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:15.036131+0000 mgr.y (mgr.24481) 961 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:16.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:15.077075+0000 mgr.y (mgr.24481) 962 : cluster [DBG] pgmap v1376: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 27296/184758 objects degraded (14.774%); 25785/184758 objects misplaced (13.956%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:16.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:15.189410+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.33:0/3110608700' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:15:16.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:15.199403+0000 mon.a (mon.0) 2200 : audit [INF] from='client.26056 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:15:16.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:15.036131+0000 mgr.y (mgr.24481) 961 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:16.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:15.077075+0000 mgr.y (mgr.24481) 962 : cluster [DBG] pgmap v1376: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 27296/184758 objects degraded (14.774%); 25785/184758 objects misplaced (13.956%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:16.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:15.189410+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.33:0/3110608700' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:15:16.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:15.199403+0000 mon.a (mon.0) 2200 : audit [INF] from='client.26056 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:15:17.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:16.184810+0000 mon.a (mon.0) 2201 : audit [INF] from='client.26056 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:15:17.222 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:16.184874+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2024-02-12T17:15:17.222 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:16.694366+0000 mon.a (mon.0) 2203 : cluster [WRN] Health check update: Degraded data redundancy: 27296/184758 objects degraded (14.774%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:17.222 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:16.184810+0000 mon.a (mon.0) 2201 : audit [INF] from='client.26056 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:15:17.222 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:16.184874+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2024-02-12T17:15:17.222 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:16.694366+0000 mon.a (mon.0) 2203 : cluster [WRN] Health check update: Degraded data redundancy: 27296/184758 objects degraded (14.774%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:17.223 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:16.721465+0000 mon.a (mon.0) 2204 : cluster [DBG] osdmap e688: 8 total, 8 up, 7 in 2024-02-12T17:15:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:16.184810+0000 mon.a (mon.0) 2201 : audit [INF] from='client.26056 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:15:17.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:16.184874+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2024-02-12T17:15:17.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:16.694366+0000 mon.a (mon.0) 2203 : cluster [WRN] Health check update: Degraded data redundancy: 27296/184758 objects degraded (14.774%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:17.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:16.721465+0000 mon.a (mon.0) 2204 : cluster [DBG] osdmap e688: 8 total, 8 up, 7 in 2024-02-12T17:15:17.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:16.721465+0000 mon.a (mon.0) 2204 : cluster [DBG] osdmap e688: 8 total, 8 up, 7 in 2024-02-12T17:15:18.269 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T17:15:18.269 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:15:18.419 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:18.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:18.525 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:18.617 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:18.909 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:18.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:18.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:17.077653+0000 mgr.y (mgr.24481) 963 : cluster [DBG] pgmap v1379: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 946 B/s rd, 0 B/s wr, 627 op/s; 26559/180879 objects degraded (14.683%); 25524/180879 objects misplaced (14.111%); 2.3 MiB/s, 37 objects/s recovering 2024-02-12T17:15:18.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:17.529000+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:18.914 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:17.077653+0000 mgr.y (mgr.24481) 963 : cluster [DBG] pgmap v1379: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 946 B/s rd, 0 B/s wr, 627 op/s; 26559/180879 objects degraded (14.683%); 25524/180879 objects misplaced (14.111%); 2.3 MiB/s, 37 objects/s recovering 2024-02-12T17:15:18.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:17.529000+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:18.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:17.077653+0000 mgr.y (mgr.24481) 963 : cluster [DBG] pgmap v1379: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 946 B/s rd, 0 B/s wr, 627 op/s; 26559/180879 objects degraded (14.683%); 25524/180879 objects misplaced (14.111%); 2.3 MiB/s, 37 objects/s recovering 2024-02-12T17:15:18.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:17.529000+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:19.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:19.447 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:19.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:19.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:19.625 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:19.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:19.732 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:19.833 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:19.841 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:15:19.841 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:15:20.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:19.078310+0000 mgr.y (mgr.24481) 964 : cluster [DBG] pgmap v1380: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 733 op/s; 25607/174873 objects degraded (14.643%); 24316/174873 objects misplaced (13.905%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:15:20.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:19.078310+0000 mgr.y (mgr.24481) 964 : cluster [DBG] pgmap v1380: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 733 op/s; 25607/174873 objects degraded (14.643%); 24316/174873 objects misplaced (13.905%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:15:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:19.078310+0000 mgr.y (mgr.24481) 964 : cluster [DBG] pgmap v1380: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 733 op/s; 25607/174873 objects degraded (14.643%); 24316/174873 objects misplaced (13.905%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:15:21.130 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:15:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:20] "GET /metrics HTTP/1.1" 200 176846 "" "Prometheus/2.33.4" 2024-02-12T17:15:21.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:20.750881+0000 mon.b (mon.1) 120 : audit [DBG] from='client.? v1:172.21.15.33:0/185062610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:22.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:20.750881+0000 mon.b (mon.1) 120 : audit [DBG] from='client.? v1:172.21.15.33:0/185062610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:22.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:20.750881+0000 mon.b (mon.1) 120 : audit [DBG] from='client.? v1:172.21.15.33:0/185062610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:22.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:15:22.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:15:22.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:15:22.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:15:22.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:15:22.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:15:22.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:15:22.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:15:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:15:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:15:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:15:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:15:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:15:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:15:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:15:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:15:22.343 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:22.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:22.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:22.457 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:22.457 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:22.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T17:15:22.493 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:22.524 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:22.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:22.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:22.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:21.079771+0000 mgr.y (mgr.24481) 965 : cluster [DBG] pgmap v1381: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 25607/174873 objects degraded (14.643%); 24316/174873 objects misplaced (13.905%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:22.718 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:21.696163+0000 mon.a (mon.0) 2206 : cluster [WRN] Health check update: Degraded data redundancy: 25607/174873 objects degraded (14.643%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:22.720 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:21.079771+0000 mgr.y (mgr.24481) 965 : cluster [DBG] pgmap v1381: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 25607/174873 objects degraded (14.643%); 24316/174873 objects misplaced (13.905%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:22.720 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:21.696163+0000 mon.a (mon.0) 2206 : cluster [WRN] Health check update: Degraded data redundancy: 25607/174873 objects degraded (14.643%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:22.722 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:22.877 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:22.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:15:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:15:22.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:21.079771+0000 mgr.y (mgr.24481) 965 : cluster [DBG] pgmap v1381: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 25607/174873 objects degraded (14.643%); 24316/174873 objects misplaced (13.905%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:22.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:21.696163+0000 mon.a (mon.0) 2206 : cluster [WRN] Health check update: Degraded data redundancy: 25607/174873 objects degraded (14.643%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:22.998 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:23.760 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:23.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:23.875 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:24.090 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:24.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:24.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:24.237 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:24.263 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:24.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:24.369 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:24.441 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:23.081114+0000 mgr.y (mgr.24481) 966 : cluster [DBG] pgmap v1382: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 576 op/s; 24883/170934 objects degraded (14.557%); 24069/170934 objects misplaced (14.081%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:15:25.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:23.081114+0000 mgr.y (mgr.24481) 966 : cluster [DBG] pgmap v1382: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 576 op/s; 24883/170934 objects degraded (14.557%); 24069/170934 objects misplaced (14.081%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:15:25.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:23.081114+0000 mgr.y (mgr.24481) 966 : cluster [DBG] pgmap v1382: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 576 op/s; 24883/170934 objects degraded (14.557%); 24069/170934 objects misplaced (14.081%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:15:25.292 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:15:25.493 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:15:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:15:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:26.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:25.046512+0000 mgr.y (mgr.24481) 967 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:26.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:25.081933+0000 mgr.y (mgr.24481) 968 : cluster [DBG] pgmap v1383: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 755 op/s; 23938/164661 objects degraded (14.538%); 22792/164661 objects misplaced (13.842%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:27.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:25.046512+0000 mgr.y (mgr.24481) 967 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:27.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:25.081933+0000 mgr.y (mgr.24481) 968 : cluster [DBG] pgmap v1383: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 755 op/s; 23938/164661 objects degraded (14.538%); 22792/164661 objects misplaced (13.842%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:27.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:25.046512+0000 mgr.y (mgr.24481) 967 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:27.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:25.081933+0000 mgr.y (mgr.24481) 968 : cluster [DBG] pgmap v1383: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 755 op/s; 23938/164661 objects degraded (14.538%); 22792/164661 objects misplaced (13.842%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:27.548 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:27.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:27.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:27.703 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:27.723 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:27.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:27.828 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:15:27.830 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:26.701229+0000 mon.a (mon.0) 2207 : cluster [WRN] Health check update: Degraded data redundancy: 23938/164661 objects degraded (14.538%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:27.830 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:27.339740+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.33:0/3846180012' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:27.830 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:27.340467+0000 mon.a (mon.0) 2208 : audit [INF] from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:27.837 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:26.701229+0000 mon.a (mon.0) 2207 : cluster [WRN] Health check update: Degraded data redundancy: 23938/164661 objects degraded (14.538%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:27.837 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:27.339740+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.33:0/3846180012' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:27.837 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:27.340467+0000 mon.a (mon.0) 2208 : audit [INF] from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:27.916 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:26.701229+0000 mon.a (mon.0) 2207 : cluster [WRN] Health check update: Degraded data redundancy: 23938/164661 objects degraded (14.538%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:27.339740+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.33:0/3846180012' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:27.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:27.340467+0000 mon.a (mon.0) 2208 : audit [INF] from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:15:28.767 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:15:28.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:27.088185+0000 mgr.y (mgr.24481) 969 : cluster [DBG] pgmap v1384: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 642 op/s; 23275/160944 objects degraded (14.462%); 22573/160944 objects misplaced (14.025%); 2.5 MiB/s, 39 objects/s recovering 2024-02-12T17:15:28.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:27.729139+0000 mon.a (mon.0) 2209 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:28.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:27.757938+0000 mon.a (mon.0) 2210 : audit [INF] from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:15:28.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:27.758051+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e689: 8 total, 8 up, 7 in 2024-02-12T17:15:29.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:27.088185+0000 mgr.y (mgr.24481) 969 : cluster [DBG] pgmap v1384: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 642 op/s; 23275/160944 objects degraded (14.462%); 22573/160944 objects misplaced (14.025%); 2.5 MiB/s, 39 objects/s recovering 2024-02-12T17:15:29.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:27.729139+0000 mon.a (mon.0) 2209 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:29.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:27.757938+0000 mon.a (mon.0) 2210 : audit [INF] from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:15:29.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:27.758051+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e689: 8 total, 8 up, 7 in 2024-02-12T17:15:29.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:27.088185+0000 mgr.y (mgr.24481) 969 : cluster [DBG] pgmap v1384: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 642 op/s; 23275/160944 objects degraded (14.462%); 22573/160944 objects misplaced (14.025%); 2.5 MiB/s, 39 objects/s recovering 2024-02-12T17:15:29.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:27.729139+0000 mon.a (mon.0) 2209 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:29.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:27.757938+0000 mon.a (mon.0) 2210 : audit [INF] from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:15:29.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:27.758051+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e689: 8 total, 8 up, 7 in 2024-02-12T17:15:29.311 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:15:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:29.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:15:29.423 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:29.462 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:29.488 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:29.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:29.601 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:29.643 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:29.672 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:29.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:29.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:29.858 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:30.749 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:30.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:30.844 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:15:30.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:30.875 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:15:30.888 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:30.928 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:29.088980+0000 mgr.y (mgr.24481) 970 : cluster [DBG] pgmap v1386: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 680 op/s; 22282/154476 objects degraded (14.424%); 21286/154476 objects misplaced (13.779%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:15:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:30.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:31.023 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:15:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:30] "GET /metrics HTTP/1.1" 200 176846 "" "Prometheus/2.33.4" 2024-02-12T17:15:31.024 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:29.088980+0000 mgr.y (mgr.24481) 970 : cluster [DBG] pgmap v1386: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 680 op/s; 22282/154476 objects degraded (14.424%); 21286/154476 objects misplaced (13.779%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:15:31.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:29.088980+0000 mgr.y (mgr.24481) 970 : cluster [DBG] pgmap v1386: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 680 op/s; 22282/154476 objects degraded (14.424%); 21286/154476 objects misplaced (13.779%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:15:31.030 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:15:31.105 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:31.572 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:15:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:31.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:15:31.684 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:31.720 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:31.744 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:31.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:30.234536+0000 mgr.y (mgr.24481) 971 : audit [DBG] from='client.36194 v1:172.21.15.33:0/325662125' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:15:31.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:30.234536+0000 mgr.y (mgr.24481) 971 : audit [DBG] from='client.36194 v1:172.21.15.33:0/325662125' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:15:31.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:31.863 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:31.907 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:31.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:31.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:30.234536+0000 mgr.y (mgr.24481) 971 : audit [DBG] from='client.36194 v1:172.21.15.33:0/325662125' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:15:32.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:32.066 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:32.355 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1386,"stamp":"2024-02-12T17:15:29.088398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3361937084,"num_objects":51492,"num_object_clones":0,"num_object_copies":154476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22282,"num_objects_misplaced":21286,"num_objects_unfound":0,"num_objects_dirty":51492,"num_whiteouts":0,"num_read":1844,"num_read_kb":1596,"num_write":201076,"num_write_kb":8044010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20726,"num_bytes_recovered":1303010080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92538,"ondisk_log_size":92538,"up":354,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":368,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15219528,"kb_used_data":7723024,"kb_used_omap":3,"kb_used_meta":7496060,"kb_avail":640996536,"statfs":{"total":671965249536,"available":656380452864,"internally_reserved":0,"allocated":7908376576,"data_stored":7898917464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":7675965501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,7,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-445579264,"num_objects":-6799,"num_object_clones":0,"num_object_copies":-20397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3325,"num_objects_misplaced":-3030,"num_objects_unfound":0,"num_objects_dirty":-6799,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6799,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":27000832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.010598"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":413,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.229172+0000","last_change":"2024-02-12T17:14:10.348720+0000","last_active":"2024-02-12T17:15:16.229172+0000","last_peered":"2024-02-12T17:15:16.229172+0000","last_clean":"2024-02-12T17:15:16.229172+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:15:16.229172+0000","last_undegraded":"2024-02-12T17:15:16.229172+0000","last_fullsized":"2024-02-12T17:15:16.229172+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":197,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.715626+0000","last_change":"2024-02-12T17:13:46.694822+0000","last_active":"2024-02-12T17:15:16.715626+0000","last_peered":"2024-02-12T17:15:16.715626+0000","last_clean":"2024-02-12T17:15:16.715626+0000","last_became_active":"2024-02-12T17:13:46.694448+0000","last_became_peered":"2024-02-12T17:13:46.694448+0000","last_unstale":"2024-02-12T17:15:16.715626+0000","last_undegraded":"2024-02-12T17:15:16.715626+0000","last_fullsized":"2024-02-12T17:15:16.715626+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:12:55.862345+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:12:55.862345+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":440,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.232524+0000","last_change":"2024-02-12T17:13:46.787428+0000","last_active":"2024-02-12T17:15:16.232524+0000","last_peered":"2024-02-12T17:15:16.232524+0000","last_clean":"2024-02-12T17:15:16.232524+0000","last_became_active":"2024-02-12T17:13:46.786996+0000","last_became_peered":"2024-02-12T17:13:46.786996+0000","last_unstale":"2024-02-12T17:15:16.232524+0000","last_undegraded":"2024-02-12T17:15:16.232524+0000","last_fullsized":"2024-02-12T17:15:16.232524+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:13:12.016227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:12.016227+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":468,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.231644+0000","last_change":"2024-02-12T17:14:09.685503+0000","last_active":"2024-02-12T17:15:16.231644+0000","last_peered":"2024-02-12T17:15:16.231644+0000","last_clean":"2024-02-12T17:15:16.231644+0000","last_became_active":"2024-02-12T17:14:08.761132+0000","last_became_peered":"2024-02-12T17:14:08.761132+0000","last_unstale":"2024-02-12T17:15:16.231644+0000","last_undegraded":"2024-02-12T17:15:16.231644+0000","last_fullsized":"2024-02-12T17:15:16.231644+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:54.052006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.052006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":483,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.231798+0000","last_change":"2024-02-12T17:13:46.753228+0000","last_active":"2024-02-12T17:15:16.231798+0000","last_peered":"2024-02-12T17:15:16.231798+0000","last_clean":"2024-02-12T17:15:16.231798+0000","last_became_active":"2024-02-12T17:13:46.752755+0000","last_became_peered":"2024-02-12T17:13:46.752755+0000","last_unstale":"2024-02-12T17:15:16.231798+0000","last_undegraded":"2024-02-12T17:15:16.231798+0000","last_fullsized":"2024-02-12T17:15:16.231798+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":1540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":148,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223684+0000","last_change":"2024-02-12T17:13:46.607814+0000","last_active":"2024-02-12T17:15:16.223684+0000","last_peered":"2024-02-12T17:15:16.223684+0000","last_clean":"2024-02-12T17:15:16.223684+0000","last_became_active":"2024-02-12T17:13:46.606367+0000","last_became_peered":"2024-02-12T17:13:46.606367+0000","last_unstale":"2024-02-12T17:15:16.223684+0000","last_undegraded":"2024-02-12T17:15:16.223684+0000","last_fullsized":"2024-02-12T17:15:16.223684+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":224,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.718066+0000","last_change":"2024-02-12T17:13:46.567244+0000","last_active":"2024-02-12T17:15:16.718066+0000","last_peered":"2024-02-12T17:15:16.718066+0000","last_clean":"2024-02-12T17:15:16.718066+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:15:16.718066+0000","last_undegraded":"2024-02-12T17:15:16.718066+0000","last_fullsized":"2024-02-12T17:15:16.718066+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":58,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.242419+0000","last_change":"2024-02-12T17:14:08.993489+0000","last_active":"2024-02-12T17:15:16.242419+0000","last_peered":"2024-02-12T17:15:16.242419+0000","last_clean":"2024-02-12T17:15:16.242419+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:15:16.242419+0000","last_undegraded":"2024-02-12T17:15:16.242419+0000","last_fullsized":"2024-02-12T17:15:16.242419+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":412,"reported_epoch":688,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:16.708541+0000","last_change":"2024-02-12T17:14:09.723827+0000","last_active":"2024-02-12T17:15:16.708541+0000","last_peered":"2024-02-12T17:15:16.708541+0000","last_clean":"2024-02-12T17:14:05.085957+0000","last_became_active":"2024-02-12T17:14:09.723480+0000","last_became_peered":"2024-02-12T17:14:09.723480+0000","last_unstale":"2024-02-12T17:15:16.708541+0000","last_undegraded":"2024-02-12T17:15:16.708541+0000","last_fullsized":"2024-02-12T17:15:16.708541+0000","mapping_epoch":652,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":381,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.232596+0000","last_change":"2024-02-12T17:14:08.755098+0000","last_active":"2024-02-12T17:15:16.232596+0000","last_peered":"2024-02-12T17:15:16.232596+0000","last_clean":"2024-02-12T17:15:16.232596+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:15:16.232596+0000","last_undegraded":"2024-02-12T17:15:16.232596+0000","last_fullsized":"2024-02-12T17:15:16.232596+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":404,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.229478+0000","last_change":"2024-02-12T17:12:12.141297+0000","last_active":"2024-02-12T17:15:16.229478+0000","last_peered":"2024-02-12T17:15:16.229478+0000","last_clean":"2024-02-12T17:15:16.229478+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:15:16.229478+0000","last_undegraded":"2024-02-12T17:15:16.229478+0000","last_fullsized":"2024-02-12T17:15:16.229478+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.141204+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":234,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.716641+0000","last_change":"2024-02-12T17:14:08.743209+0000","last_active":"2024-02-12T17:15:16.716641+0000","last_peered":"2024-02-12T17:15:16.716641+0000","last_clean":"2024-02-12T17:15:16.716641+0000","last_became_active":"2024-02-12T17:14:08.742886+0000","last_became_peered":"2024-02-12T17:14:08.742886+0000","last_unstale":"2024-02-12T17:15:16.716641+0000","last_undegraded":"2024-02-12T17:15:16.716641+0000","last_fullsized":"2024-02-12T17:15:16.716641+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.783527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.783527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":683,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708691+0000","last_change":"2024-02-12T17:13:21.591627+0000","last_active":"2024-02-12T17:15:16.708691+0000","last_peered":"2024-02-12T17:15:16.708691+0000","last_clean":"2024-02-12T17:15:16.708691+0000","last_became_active":"2024-02-12T17:13:21.591246+0000","last_became_peered":"2024-02-12T17:13:21.591246+0000","last_unstale":"2024-02-12T17:15:16.708691+0000","last_undegraded":"2024-02-12T17:15:16.708691+0000","last_fullsized":"2024-02-12T17:15:16.708691+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.028495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":426,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.229453+0000","last_change":"2024-02-12T17:13:46.558817+0000","last_active":"2024-02-12T17:15:16.229453+0000","last_peered":"2024-02-12T17:15:16.229453+0000","last_clean":"2024-02-12T17:15:16.229453+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:15:16.229453+0000","last_undegraded":"2024-02-12T17:15:16.229453+0000","last_fullsized":"2024-02-12T17:15:16.229453+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":80,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223231+0000","last_change":"2024-02-12T17:13:46.576676+0000","last_active":"2024-02-12T17:15:16.223231+0000","last_peered":"2024-02-12T17:15:16.223231+0000","last_clean":"2024-02-12T17:15:16.223231+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:15:16.223231+0000","last_undegraded":"2024-02-12T17:15:16.223231+0000","last_fullsized":"2024-02-12T17:15:16.223231+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":363,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.229309+0000","last_change":"2024-02-12T17:14:08.763317+0000","last_active":"2024-02-12T17:15:16.229309+0000","last_peered":"2024-02-12T17:15:16.229309+0000","last_clean":"2024-02-12T17:15:16.229309+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:15:16.229309+0000","last_undegraded":"2024-02-12T17:15:16.229309+0000","last_fullsized":"2024-02-12T17:15:16.229309+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":120,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.224054+0000","last_change":"2024-02-12T17:14:08.742170+0000","last_active":"2024-02-12T17:15:16.224054+0000","last_peered":"2024-02-12T17:15:16.224054+0000","last_clean":"2024-02-12T17:15:16.224054+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:15:16.224054+0000","last_undegraded":"2024-02-12T17:15:16.224054+0000","last_fullsized":"2024-02-12T17:15:16.224054+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":202,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.715996+0000","last_change":"2024-02-12T17:13:46.784204+0000","last_active":"2024-02-12T17:15:16.715996+0000","last_peered":"2024-02-12T17:15:16.715996+0000","last_clean":"2024-02-12T17:15:16.715996+0000","last_became_active":"2024-02-12T17:13:46.783737+0000","last_became_peered":"2024-02-12T17:13:46.783737+0000","last_unstale":"2024-02-12T17:15:16.715996+0000","last_undegraded":"2024-02-12T17:15:16.715996+0000","last_fullsized":"2024-02-12T17:15:16.715996+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:15.817751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.817751+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":750,"reported_epoch":687,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:16.231308+0000","last_change":"2024-02-12T17:14:40.177468+0000","last_active":"2024-02-12T17:15:16.231308+0000","last_peered":"2024-02-12T17:15:16.231308+0000","last_clean":"2024-02-12T17:13:44.507959+0000","last_became_active":"2024-02-12T17:14:08.758564+0000","last_became_peered":"2024-02-12T17:14:08.758564+0000","last_unstale":"2024-02-12T17:15:16.231308+0000","last_undegraded":"2024-02-12T17:15:16.231308+0000","last_fullsized":"2024-02-12T17:15:16.231308+0000","mapping_epoch":651,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":46,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.240037+0000","last_change":"2024-02-12T17:14:08.762208+0000","last_active":"2024-02-12T17:15:16.240037+0000","last_peered":"2024-02-12T17:15:16.240037+0000","last_clean":"2024-02-12T17:15:16.240037+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:15:16.240037+0000","last_undegraded":"2024-02-12T17:15:16.240037+0000","last_fullsized":"2024-02-12T17:15:16.240037+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":192,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.716236+0000","last_change":"2024-02-12T17:12:02.752352+0000","last_active":"2024-02-12T17:15:16.716236+0000","last_peered":"2024-02-12T17:15:16.716236+0000","last_clean":"2024-02-12T17:15:16.716236+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:15:16.716236+0000","last_undegraded":"2024-02-12T17:15:16.716236+0000","last_fullsized":"2024-02-12T17:15:16.716236+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:02.752280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:02.752280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":152,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223154+0000","last_change":"2024-02-12T17:13:46.582722+0000","last_active":"2024-02-12T17:15:16.223154+0000","last_peered":"2024-02-12T17:15:16.223154+0000","last_clean":"2024-02-12T17:15:16.223154+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:15:16.223154+0000","last_undegraded":"2024-02-12T17:15:16.223154+0000","last_fullsized":"2024-02-12T17:15:16.223154+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":775,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:17.643921+0000","last_change":"2024-02-12T17:12:58.020311+0000","last_active":"2024-02-12T17:15:17.643921+0000","last_peered":"2024-02-12T17:15:17.643921+0000","last_clean":"2024-02-12T17:15:17.643921+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:15:17.643921+0000","last_undegraded":"2024-02-12T17:15:17.643921+0000","last_fullsized":"2024-02-12T17:15:17.643921+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:12:58.018712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.018712+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":46,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.241676+0000","last_change":"2024-02-12T17:14:08.872636+0000","last_active":"2024-02-12T17:15:16.241676+0000","last_peered":"2024-02-12T17:15:16.241676+0000","last_clean":"2024-02-12T17:15:16.241676+0000","last_became_active":"2024-02-12T17:14:08.872123+0000","last_became_peered":"2024-02-12T17:14:08.872123+0000","last_unstale":"2024-02-12T17:15:16.241676+0000","last_undegraded":"2024-02-12T17:15:16.241676+0000","last_fullsized":"2024-02-12T17:15:16.241676+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:06.904354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:06.904354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":779,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.711650+0000","last_change":"2024-02-12T17:12:12.838640+0000","last_active":"2024-02-12T17:15:16.711650+0000","last_peered":"2024-02-12T17:15:16.711650+0000","last_clean":"2024-02-12T17:15:16.711650+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:15:16.711650+0000","last_undegraded":"2024-02-12T17:15:16.711650+0000","last_fullsized":"2024-02-12T17:15:16.711650+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:12.838563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.838563+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"655'24","reported_seq":774,"reported_epoch":688,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:27.649360+0000","last_change":"2024-02-12T17:14:09.729342+0000","last_active":"2024-02-12T17:15:27.649360+0000","last_peered":"2024-02-12T17:15:27.649360+0000","last_clean":"2024-02-12T17:14:07.616779+0000","last_became_active":"2024-02-12T17:14:09.728969+0000","last_became_peered":"2024-02-12T17:14:09.728969+0000","last_unstale":"2024-02-12T17:15:27.649360+0000","last_undegraded":"2024-02-12T17:15:27.649360+0000","last_fullsized":"2024-02-12T17:15:27.649360+0000","mapping_epoch":652,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":683,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.712661+0000","last_change":"2024-02-12T17:12:54.656523+0000","last_active":"2024-02-12T17:15:16.712661+0000","last_peered":"2024-02-12T17:15:16.712661+0000","last_clean":"2024-02-12T17:15:16.712661+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:15:16.712661+0000","last_undegraded":"2024-02-12T17:15:16.712661+0000","last_fullsized":"2024-02-12T17:15:16.712661+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:54.656456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.656456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":417,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.233969+0000","last_change":"2024-02-12T17:13:46.577769+0000","last_active":"2024-02-12T17:15:16.233969+0000","last_peered":"2024-02-12T17:15:16.233969+0000","last_clean":"2024-02-12T17:15:16.233969+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:15:16.233969+0000","last_undegraded":"2024-02-12T17:15:16.233969+0000","last_fullsized":"2024-02-12T17:15:16.233969+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":705,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708879+0000","last_change":"2024-02-12T17:13:21.591810+0000","last_active":"2024-02-12T17:15:16.708879+0000","last_peered":"2024-02-12T17:15:16.708879+0000","last_clean":"2024-02-12T17:15:16.708879+0000","last_became_active":"2024-02-12T17:13:21.591502+0000","last_became_peered":"2024-02-12T17:13:21.591502+0000","last_unstale":"2024-02-12T17:15:16.708879+0000","last_undegraded":"2024-02-12T17:15:16.708879+0000","last_fullsized":"2024-02-12T17:15:16.708879+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.882066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.882066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":226,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.716535+0000","last_change":"2024-02-12T17:13:46.651066+0000","last_active":"2024-02-12T17:15:16.716535+0000","last_peered":"2024-02-12T17:15:16.716535+0000","last_clean":"2024-02-12T17:15:16.716535+0000","last_became_active":"2024-02-12T17:13:46.650776+0000","last_became_peered":"2024-02-12T17:13:46.650776+0000","last_unstale":"2024-02-12T17:15:16.716535+0000","last_undegraded":"2024-02-12T17:15:16.716535+0000","last_fullsized":"2024-02-12T17:15:16.716535+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:50.786900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.786900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":406,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.232891+0000","last_change":"2024-02-12T17:13:46.822111+0000","last_active":"2024-02-12T17:15:16.232891+0000","last_peered":"2024-02-12T17:15:16.232891+0000","last_clean":"2024-02-12T17:15:16.232891+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:15:16.232891+0000","last_undegraded":"2024-02-12T17:15:16.232891+0000","last_fullsized":"2024-02-12T17:15:16.232891+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":317,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708786+0000","last_change":"2024-02-12T17:14:08.745944+0000","last_active":"2024-02-12T17:15:16.708786+0000","last_peered":"2024-02-12T17:15:16.708786+0000","last_clean":"2024-02-12T17:15:16.708786+0000","last_became_active":"2024-02-12T17:14:08.745729+0000","last_became_peered":"2024-02-12T17:14:08.745729+0000","last_unstale":"2024-02-12T17:15:16.708786+0000","last_undegraded":"2024-02-12T17:15:16.708786+0000","last_fullsized":"2024-02-12T17:15:16.708786+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:49.809844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:12:49.809844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":80,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.227716+0000","last_change":"2024-02-12T17:13:46.559519+0000","last_active":"2024-02-12T17:15:16.227716+0000","last_peered":"2024-02-12T17:15:16.227716+0000","last_clean":"2024-02-12T17:15:16.227716+0000","last_became_active":"2024-02-12T17:13:46.559095+0000","last_became_peered":"2024-02-12T17:13:46.559095+0000","last_unstale":"2024-02-12T17:15:16.227716+0000","last_undegraded":"2024-02-12T17:15:16.227716+0000","last_fullsized":"2024-02-12T17:15:16.227716+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":56,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.242508+0000","last_change":"2024-02-12T17:14:08.993503+0000","last_active":"2024-02-12T17:15:16.242508+0000","last_peered":"2024-02-12T17:15:16.242508+0000","last_clean":"2024-02-12T17:15:16.242508+0000","last_became_active":"2024-02-12T17:14:08.861886+0000","last_became_peered":"2024-02-12T17:14:08.861886+0000","last_unstale":"2024-02-12T17:15:16.242508+0000","last_undegraded":"2024-02-12T17:15:16.242508+0000","last_fullsized":"2024-02-12T17:15:16.242508+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"689'12646","reported_seq":19605,"reported_epoch":689,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-12T17:15:28.609534+0000","last_change":"2024-02-12T17:15:09.885129+0000","last_active":"2024-02-12T17:15:28.609534+0000","last_peered":"2024-02-12T17:15:28.609534+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.966441+0000","last_became_peered":"2024-02-12T17:14:08.966441+0000","last_unstale":"2024-02-12T17:15:28.609534+0000","last_undegraded":"2024-02-12T17:14:08.734833+0000","last_fullsized":"2024-02-12T17:14:08.712653+0000","mapping_epoch":651,"log_start":"614'5621","ondisk_log_start":"614'5621","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":7025,"ondisk_log_size":7025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208011264,"num_objects":3174,"num_object_clones":0,"num_object_copies":9522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3174,"num_objects_misplaced":3174,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":506240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3413,"num_bytes_recovered":223674368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":3174}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":405,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.232445+0000","last_change":"2024-02-12T17:13:46.787305+0000","last_active":"2024-02-12T17:15:16.232445+0000","last_peered":"2024-02-12T17:15:16.232445+0000","last_clean":"2024-02-12T17:15:16.232445+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:15:16.232445+0000","last_undegraded":"2024-02-12T17:15:16.232445+0000","last_fullsized":"2024-02-12T17:15:16.232445+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":405,"reported_epoch":688,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:16.710731+0000","last_change":"2024-02-12T17:14:09.723027+0000","last_active":"2024-02-12T17:15:16.710731+0000","last_peered":"2024-02-12T17:15:16.710731+0000","last_clean":"2024-02-12T17:11:53.687795+0000","last_became_active":"2024-02-12T17:14:09.722681+0000","last_became_peered":"2024-02-12T17:14:09.722681+0000","last_unstale":"2024-02-12T17:15:16.710731+0000","last_undegraded":"2024-02-12T17:15:16.710731+0000","last_fullsized":"2024-02-12T17:15:16.710731+0000","mapping_epoch":652,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:12:09.882247+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:12:09.882247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[5,7,6],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":154,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223052+0000","last_change":"2024-02-12T17:13:46.744739+0000","last_active":"2024-02-12T17:15:16.223052+0000","last_peered":"2024-02-12T17:15:16.223052+0000","last_clean":"2024-02-12T17:15:16.223052+0000","last_became_active":"2024-02-12T17:13:46.744003+0000","last_became_peered":"2024-02-12T17:13:46.744003+0000","last_unstale":"2024-02-12T17:15:16.223052+0000","last_undegraded":"2024-02-12T17:15:16.223052+0000","last_fullsized":"2024-02-12T17:15:16.223052+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:51.126056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:51.126056+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":46,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.240106+0000","last_change":"2024-02-12T17:14:08.769824+0000","last_active":"2024-02-12T17:15:16.240106+0000","last_peered":"2024-02-12T17:15:16.240106+0000","last_clean":"2024-02-12T17:15:16.240106+0000","last_became_active":"2024-02-12T17:14:08.769221+0000","last_became_peered":"2024-02-12T17:14:08.769221+0000","last_unstale":"2024-02-12T17:15:16.240106+0000","last_undegraded":"2024-02-12T17:15:16.240106+0000","last_fullsized":"2024-02-12T17:15:16.240106+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"689'12406","reported_seq":12539,"reported_epoch":689,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:15:28.676758+0000","last_change":"2024-02-12T17:14:16.786942+0000","last_active":"2024-02-12T17:15:28.676758+0000","last_peered":"2024-02-12T17:15:28.676758+0000","last_clean":"2024-02-12T17:13:45.521351+0000","last_became_active":"2024-02-12T17:13:46.743970+0000","last_became_peered":"2024-02-12T17:13:46.743970+0000","last_unstale":"2024-02-12T17:15:28.676758+0000","last_undegraded":"2024-02-12T17:15:28.676758+0000","last_fullsized":"2024-02-12T17:15:28.676758+0000","mapping_epoch":630,"log_start":"619'6368","ondisk_log_start":"619'6368","created":597,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'5769","last_scrub_stamp":"2024-02-12T17:13:06.847792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:13:06.847792+0000","log_size":6038,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":207421464,"num_objects":3166,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3166,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12406,"num_write_kb":498241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":3166}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":693,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.240340+0000","last_change":"2024-02-12T17:14:07.897458+0000","last_active":"2024-02-12T17:15:16.240340+0000","last_peered":"2024-02-12T17:15:16.240340+0000","last_clean":"2024-02-12T17:15:16.240340+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:15:16.240340+0000","last_undegraded":"2024-02-12T17:15:16.240340+0000","last_fullsized":"2024-02-12T17:15:16.240340+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.780381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.780381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":80,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223840+0000","last_change":"2024-02-12T17:13:46.791505+0000","last_active":"2024-02-12T17:15:16.223840+0000","last_peered":"2024-02-12T17:15:16.223840+0000","last_clean":"2024-02-12T17:15:16.223840+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:15:16.223840+0000","last_undegraded":"2024-02-12T17:15:16.223840+0000","last_fullsized":"2024-02-12T17:15:16.223840+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":160,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223276+0000","last_change":"2024-02-12T17:14:09.246132+0000","last_active":"2024-02-12T17:15:16.223276+0000","last_peered":"2024-02-12T17:15:16.223276+0000","last_clean":"2024-02-12T17:15:16.223276+0000","last_became_active":"2024-02-12T17:14:08.752149+0000","last_became_peered":"2024-02-12T17:14:08.752149+0000","last_unstale":"2024-02-12T17:15:16.223276+0000","last_undegraded":"2024-02-12T17:15:16.223276+0000","last_fullsized":"2024-02-12T17:15:16.223276+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:13:01.543047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.543047+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"655'23","reported_seq":501,"reported_epoch":688,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:17.643201+0000","last_change":"2024-02-12T17:14:09.749094+0000","last_active":"2024-02-12T17:15:17.643201+0000","last_peered":"2024-02-12T17:15:17.643201+0000","last_clean":"2024-02-12T17:03:47.420086+0000","last_became_active":"2024-02-12T17:14:09.748790+0000","last_became_peered":"2024-02-12T17:14:09.748790+0000","last_unstale":"2024-02-12T17:15:17.643201+0000","last_undegraded":"2024-02-12T17:15:17.643201+0000","last_fullsized":"2024-02-12T17:15:17.643201+0000","mapping_epoch":652,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'21","last_scrub_stamp":"2024-02-12T17:12:57.741798+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:12:57.741798+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[3,6,5],"avail_no_missing":["3","0","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"688'12500","reported_seq":14689,"reported_epoch":688,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:15:26.564436+0000","last_change":"2024-02-12T17:14:31.380762+0000","last_active":"2024-02-12T17:15:26.564436+0000","last_peered":"2024-02-12T17:15:26.564436+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.877848+0000","last_became_peered":"2024-02-12T17:14:08.877848+0000","last_unstale":"2024-02-12T17:15:26.564436+0000","last_undegraded":"2024-02-12T17:14:09.686965+0000","last_fullsized":"2024-02-12T17:14:08.709420+0000","mapping_epoch":651,"log_start":"628'7563","ondisk_log_start":"628'7563","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4937,"ondisk_log_size":4937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":211025920,"num_objects":3220,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":2831,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12500,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":741,"num_bytes_recovered":48431104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4","objects":3220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":408,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.232012+0000","last_change":"2024-02-12T17:13:21.603543+0000","last_active":"2024-02-12T17:15:16.232012+0000","last_peered":"2024-02-12T17:15:16.232012+0000","last_clean":"2024-02-12T17:15:16.232012+0000","last_became_active":"2024-02-12T17:13:21.603226+0000","last_became_peered":"2024-02-12T17:13:21.603226+0000","last_unstale":"2024-02-12T17:15:16.232012+0000","last_undegraded":"2024-02-12T17:15:16.232012+0000","last_fullsized":"2024-02-12T17:15:16.232012+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.262134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.262134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":725,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.712729+0000","last_change":"2024-02-12T17:13:46.566957+0000","last_active":"2024-02-12T17:15:16.712729+0000","last_peered":"2024-02-12T17:15:16.712729+0000","last_clean":"2024-02-12T17:15:16.712729+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:15:16.712729+0000","last_undegraded":"2024-02-12T17:15:16.712729+0000","last_fullsized":"2024-02-12T17:15:16.712729+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":194,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:17.642949+0000","last_change":"2024-02-12T17:13:46.607171+0000","last_active":"2024-02-12T17:15:17.642949+0000","last_peered":"2024-02-12T17:15:17.642949+0000","last_clean":"2024-02-12T17:15:17.642949+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:15:17.642949+0000","last_undegraded":"2024-02-12T17:15:17.642949+0000","last_fullsized":"2024-02-12T17:15:17.642949+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":639,"reported_epoch":687,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:16.223839+0000","last_change":"2024-02-12T17:14:09.723924+0000","last_active":"2024-02-12T17:15:16.223839+0000","last_peered":"2024-02-12T17:15:16.223839+0000","last_clean":"2024-02-12T17:14:05.074559+0000","last_became_active":"2024-02-12T17:14:09.723600+0000","last_became_peered":"2024-02-12T17:14:09.723600+0000","last_unstale":"2024-02-12T17:15:16.223839+0000","last_undegraded":"2024-02-12T17:15:16.223839+0000","last_fullsized":"2024-02-12T17:15:16.223839+0000","mapping_epoch":652,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:13:02.852563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:13:02.852563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[1,6,7],"avail_no_missing":["1","0","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"688'12707","reported_seq":12806,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:26.584920+0000","last_change":"2024-02-12T17:11:59.008876+0000","last_active":"2024-02-12T17:15:26.584920+0000","last_peered":"2024-02-12T17:15:26.584920+0000","last_clean":"2024-02-12T17:15:26.584920+0000","last_became_active":"2024-02-12T17:11:59.008587+0000","last_became_peered":"2024-02-12T17:11:59.008587+0000","last_unstale":"2024-02-12T17:15:26.584920+0000","last_undegraded":"2024-02-12T17:15:26.584920+0000","last_fullsized":"2024-02-12T17:15:26.584920+0000","mapping_epoch":597,"log_start":"628'7718","ondisk_log_start":"628'7718","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4989,"ondisk_log_size":4989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":214106112,"num_objects":3267,"num_object_clones":0,"num_object_copies":9801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12707,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":694,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.709211+0000","last_change":"2024-02-12T17:13:00.766698+0000","last_active":"2024-02-12T17:15:16.709211+0000","last_peered":"2024-02-12T17:15:16.709211+0000","last_clean":"2024-02-12T17:15:16.709211+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:15:16.709211+0000","last_undegraded":"2024-02-12T17:15:16.709211+0000","last_fullsized":"2024-02-12T17:15:16.709211+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.766590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":286,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:22.991532+0000","last_change":"2024-02-12T17:13:46.584941+0000","last_active":"2024-02-12T17:15:22.991532+0000","last_peered":"2024-02-12T17:15:22.991532+0000","last_clean":"2024-02-12T17:15:22.991532+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:15:22.991532+0000","last_undegraded":"2024-02-12T17:15:22.991532+0000","last_fullsized":"2024-02-12T17:15:22.991532+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":310,"num_read_kb":305,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":354,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.229556+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:15:16.229556+0000","last_peered":"2024-02-12T17:15:16.229556+0000","last_clean":"2024-02-12T17:15:16.229556+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:15:16.229556+0000","last_undegraded":"2024-02-12T17:15:16.229556+0000","last_fullsized":"2024-02-12T17:15:16.229556+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":153,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.233155+0000","last_change":"2024-02-12T17:14:09.321979+0000","last_active":"2024-02-12T17:15:16.233155+0000","last_peered":"2024-02-12T17:15:16.233155+0000","last_clean":"2024-02-12T17:15:16.233155+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:15:16.233155+0000","last_undegraded":"2024-02-12T17:15:16.233155+0000","last_fullsized":"2024-02-12T17:15:16.233155+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":618,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.711364+0000","last_change":"2024-02-12T17:13:10.412138+0000","last_active":"2024-02-12T17:15:16.711364+0000","last_peered":"2024-02-12T17:15:16.711364+0000","last_clean":"2024-02-12T17:15:16.711364+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:15:16.711364+0000","last_undegraded":"2024-02-12T17:15:16.711364+0000","last_fullsized":"2024-02-12T17:15:16.711364+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:13:10.412031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"688'12320","reported_seq":12419,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:26.581807+0000","last_change":"2024-02-12T17:11:59.007548+0000","last_active":"2024-02-12T17:15:26.581807+0000","last_peered":"2024-02-12T17:15:26.581807+0000","last_clean":"2024-02-12T17:15:26.581807+0000","last_became_active":"2024-02-12T17:11:59.007214+0000","last_became_peered":"2024-02-12T17:11:59.007214+0000","last_unstale":"2024-02-12T17:15:26.581807+0000","last_undegraded":"2024-02-12T17:15:26.581807+0000","last_fullsized":"2024-02-12T17:15:26.581807+0000","mapping_epoch":597,"log_start":"628'7355","ondisk_log_start":"628'7355","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4965,"ondisk_log_size":4965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":210894848,"num_objects":3218,"num_object_clones":0,"num_object_copies":9654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12320,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":757,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.711947+0000","last_change":"2024-02-12T17:13:46.826853+0000","last_active":"2024-02-12T17:15:16.711947+0000","last_peered":"2024-02-12T17:15:16.711947+0000","last_clean":"2024-02-12T17:15:16.711947+0000","last_became_active":"2024-02-12T17:13:46.821418+0000","last_became_peered":"2024-02-12T17:13:46.821418+0000","last_unstale":"2024-02-12T17:15:16.711947+0000","last_undegraded":"2024-02-12T17:15:16.711947+0000","last_fullsized":"2024-02-12T17:15:16.711947+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.951964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.951964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":386,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.231436+0000","last_change":"2024-02-12T17:12:59.006372+0000","last_active":"2024-02-12T17:15:16.231436+0000","last_peered":"2024-02-12T17:15:16.231436+0000","last_clean":"2024-02-12T17:15:16.231436+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:15:16.231436+0000","last_undegraded":"2024-02-12T17:15:16.231436+0000","last_fullsized":"2024-02-12T17:15:16.231436+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.006260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.006260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":665,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.710887+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:15:16.710887+0000","last_peered":"2024-02-12T17:15:16.710887+0000","last_clean":"2024-02-12T17:15:16.710887+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:15:16.710887+0000","last_undegraded":"2024-02-12T17:15:16.710887+0000","last_fullsized":"2024-02-12T17:15:16.710887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":734,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708295+0000","last_change":"2024-02-12T17:14:08.743668+0000","last_active":"2024-02-12T17:15:16.708295+0000","last_peered":"2024-02-12T17:15:16.708295+0000","last_clean":"2024-02-12T17:15:16.708295+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:15:16.708295+0000","last_undegraded":"2024-02-12T17:15:16.708295+0000","last_fullsized":"2024-02-12T17:15:16.708295+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":299,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.716596+0000","last_change":"2024-02-12T17:14:08.781462+0000","last_active":"2024-02-12T17:15:16.716596+0000","last_peered":"2024-02-12T17:15:16.716596+0000","last_clean":"2024-02-12T17:15:16.716596+0000","last_became_active":"2024-02-12T17:14:08.726463+0000","last_became_peered":"2024-02-12T17:14:08.726463+0000","last_unstale":"2024-02-12T17:15:16.716596+0000","last_undegraded":"2024-02-12T17:15:16.716596+0000","last_fullsized":"2024-02-12T17:15:16.716596+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:47.763264+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.763264+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":696,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708860+0000","last_change":"2024-02-12T17:12:03.922963+0000","last_active":"2024-02-12T17:15:16.708860+0000","last_peered":"2024-02-12T17:15:16.708860+0000","last_clean":"2024-02-12T17:15:16.708860+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:15:16.708860+0000","last_undegraded":"2024-02-12T17:15:16.708860+0000","last_fullsized":"2024-02-12T17:15:16.708860+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:12:03.922893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.922893+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"688'12503","reported_seq":14098,"reported_epoch":688,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:15:26.564183+0000","last_change":"2024-02-12T17:14:10.177438+0000","last_active":"2024-02-12T17:15:26.564183+0000","last_peered":"2024-02-12T17:15:26.564183+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.873608+0000","last_became_peered":"2024-02-12T17:14:08.873608+0000","last_unstale":"2024-02-12T17:15:26.564183+0000","last_undegraded":"2024-02-12T17:14:08.744781+0000","last_fullsized":"2024-02-12T17:14:08.704327+0000","mapping_epoch":651,"log_start":"628'7542","ondisk_log_start":"628'7542","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4961,"ondisk_log_size":4961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":212664320,"num_objects":3245,"num_object_clones":0,"num_object_copies":9735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12503,"num_write_kb":503936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":48496640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":3245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":80,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223921+0000","last_change":"2024-02-12T17:13:46.608049+0000","last_active":"2024-02-12T17:15:16.223921+0000","last_peered":"2024-02-12T17:15:16.223921+0000","last_clean":"2024-02-12T17:15:16.223921+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:15:16.223921+0000","last_undegraded":"2024-02-12T17:15:16.223921+0000","last_fullsized":"2024-02-12T17:15:16.223921+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.249487+0000","last_change":"2024-02-12T17:14:08.859925+0000","last_active":"2024-02-12T17:15:16.249487+0000","last_peered":"2024-02-12T17:15:16.249487+0000","last_clean":"2024-02-12T17:15:16.249487+0000","last_became_active":"2024-02-12T17:14:08.859396+0000","last_became_peered":"2024-02-12T17:14:08.859396+0000","last_unstale":"2024-02-12T17:15:16.249487+0000","last_undegraded":"2024-02-12T17:15:16.249487+0000","last_fullsized":"2024-02-12T17:15:16.249487+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:03.904976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.904976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":734,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708207+0000","last_change":"2024-02-12T17:14:05.666060+0000","last_active":"2024-02-12T17:15:16.708207+0000","last_peered":"2024-02-12T17:15:16.708207+0000","last_clean":"2024-02-12T17:15:16.708207+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:15:16.708207+0000","last_undegraded":"2024-02-12T17:15:16.708207+0000","last_fullsized":"2024-02-12T17:15:16.708207+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":60,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.242593+0000","last_change":"2024-02-12T17:14:09.046726+0000","last_active":"2024-02-12T17:15:16.242593+0000","last_peered":"2024-02-12T17:15:16.242593+0000","last_clean":"2024-02-12T17:15:16.242593+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:15:16.242593+0000","last_undegraded":"2024-02-12T17:15:16.242593+0000","last_fullsized":"2024-02-12T17:15:16.242593+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":737,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.710905+0000","last_change":"2024-02-12T17:13:46.827015+0000","last_active":"2024-02-12T17:15:16.710905+0000","last_peered":"2024-02-12T17:15:16.710905+0000","last_clean":"2024-02-12T17:15:16.710905+0000","last_became_active":"2024-02-12T17:13:46.821796+0000","last_became_peered":"2024-02-12T17:13:46.821796+0000","last_unstale":"2024-02-12T17:15:16.710905+0000","last_undegraded":"2024-02-12T17:15:16.710905+0000","last_fullsized":"2024-02-12T17:15:16.710905+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.698727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.698727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"689'12406","reported_seq":18494,"reported_epoch":689,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:15:28.691562+0000","last_change":"2024-02-12T17:14:46.835208+0000","last_active":"2024-02-12T17:15:28.691562+0000","last_peered":"2024-02-12T17:15:28.691562+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.806885+0000","last_became_peered":"2024-02-12T17:13:46.806885+0000","last_unstale":"2024-02-12T17:15:28.691562+0000","last_undegraded":"2024-02-12T17:13:46.572970+0000","last_fullsized":"2024-02-12T17:13:46.531119+0000","mapping_epoch":630,"log_start":"613'5374","ondisk_log_start":"613'5374","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":7032,"ondisk_log_size":7032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":206438400,"num_objects":3150,"num_object_clones":0,"num_object_copies":9450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1822,"num_objects_misplaced":3150,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":497792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2984,"num_bytes_recovered":195559424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":3150}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":111,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.222973+0000","last_change":"2024-02-12T17:13:46.579742+0000","last_active":"2024-02-12T17:15:16.222973+0000","last_peered":"2024-02-12T17:15:16.222973+0000","last_clean":"2024-02-12T17:15:16.222973+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:15:16.222973+0000","last_undegraded":"2024-02-12T17:15:16.222973+0000","last_fullsized":"2024-02-12T17:15:16.222973+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":748,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.709472+0000","last_change":"2024-02-12T17:14:08.746702+0000","last_active":"2024-02-12T17:15:16.709472+0000","last_peered":"2024-02-12T17:15:16.709472+0000","last_clean":"2024-02-12T17:15:16.709472+0000","last_became_active":"2024-02-12T17:14:08.746395+0000","last_became_peered":"2024-02-12T17:14:08.746395+0000","last_unstale":"2024-02-12T17:15:16.709472+0000","last_undegraded":"2024-02-12T17:15:16.709472+0000","last_fullsized":"2024-02-12T17:15:16.709472+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:45.764955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:45.764955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":64,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.241521+0000","last_change":"2024-02-12T17:14:11.148689+0000","last_active":"2024-02-12T17:15:16.241521+0000","last_peered":"2024-02-12T17:15:16.241521+0000","last_clean":"2024-02-12T17:15:16.241521+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:15:16.241521+0000","last_undegraded":"2024-02-12T17:15:16.241521+0000","last_fullsized":"2024-02-12T17:15:16.241521+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":190,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.716701+0000","last_change":"2024-02-12T17:12:53.746937+0000","last_active":"2024-02-12T17:15:16.716701+0000","last_peered":"2024-02-12T17:15:16.716701+0000","last_clean":"2024-02-12T17:15:16.716701+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:15:16.716701+0000","last_undegraded":"2024-02-12T17:15:16.716701+0000","last_fullsized":"2024-02-12T17:15:16.716701+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.746858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.746858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"689'12450","reported_seq":12564,"reported_epoch":689,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:15:28.899394+0000","last_change":"2024-02-12T17:14:46.848102+0000","last_active":"2024-02-12T17:15:28.899394+0000","last_peered":"2024-02-12T17:15:28.899394+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.832453+0000","last_became_peered":"2024-02-12T17:13:46.832453+0000","last_unstale":"2024-02-12T17:15:28.899394+0000","last_undegraded":"2024-02-12T17:13:46.655125+0000","last_fullsized":"2024-02-12T17:13:46.642131+0000","mapping_epoch":631,"log_start":"623'6770","ondisk_log_start":"623'6770","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5680,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":214433792,"num_objects":3272,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12450,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":3272}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":395,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.231482+0000","last_change":"2024-02-12T17:13:21.589965+0000","last_active":"2024-02-12T17:15:16.231482+0000","last_peered":"2024-02-12T17:15:16.231482+0000","last_clean":"2024-02-12T17:15:16.231482+0000","last_became_active":"2024-02-12T17:13:21.589584+0000","last_became_peered":"2024-02-12T17:13:21.589584+0000","last_unstale":"2024-02-12T17:15:16.231482+0000","last_undegraded":"2024-02-12T17:15:16.231482+0000","last_fullsized":"2024-02-12T17:15:16.231482+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:01.059570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.059570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":127,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.228372+0000","last_change":"2024-02-12T17:14:18.716530+0000","last_active":"2024-02-12T17:15:16.228372+0000","last_peered":"2024-02-12T17:15:16.228372+0000","last_clean":"2024-02-12T17:15:16.228372+0000","last_became_active":"2024-02-12T17:14:18.716232+0000","last_became_peered":"2024-02-12T17:14:18.716232+0000","last_unstale":"2024-02-12T17:15:16.228372+0000","last_undegraded":"2024-02-12T17:15:16.228372+0000","last_fullsized":"2024-02-12T17:15:16.228372+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":454,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.229738+0000","last_change":"2024-02-12T17:14:10.144923+0000","last_active":"2024-02-12T17:15:16.229738+0000","last_peered":"2024-02-12T17:15:16.229738+0000","last_clean":"2024-02-12T17:15:16.229738+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:15:16.229738+0000","last_undegraded":"2024-02-12T17:15:16.229738+0000","last_fullsized":"2024-02-12T17:15:16.229738+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":674,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.229394+0000","last_change":"2024-02-12T17:14:08.765556+0000","last_active":"2024-02-12T17:15:16.229394+0000","last_peered":"2024-02-12T17:15:16.229394+0000","last_clean":"2024-02-12T17:15:16.229394+0000","last_became_active":"2024-02-12T17:14:08.757310+0000","last_became_peered":"2024-02-12T17:14:08.757310+0000","last_unstale":"2024-02-12T17:15:16.229394+0000","last_undegraded":"2024-02-12T17:15:16.229394+0000","last_fullsized":"2024-02-12T17:15:16.229394+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:58.871569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.871569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"689'12543","reported_seq":15593,"reported_epoch":689,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:15:28.964685+0000","last_change":"2024-02-12T17:14:10.288517+0000","last_active":"2024-02-12T17:15:28.964685+0000","last_peered":"2024-02-12T17:15:28.964685+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.867083+0000","last_became_peered":"2024-02-12T17:14:08.867083+0000","last_unstale":"2024-02-12T17:15:28.964685+0000","last_undegraded":"2024-02-12T17:14:08.714899+0000","last_fullsized":"2024-02-12T17:14:08.701373+0000","mapping_epoch":651,"log_start":"623'6842","ondisk_log_start":"623'6842","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5701,"ondisk_log_size":5701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208994304,"num_objects":3189,"num_object_clones":0,"num_object_copies":9567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2641,"num_objects_misplaced":2641,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12543,"num_write_kb":503424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1058,"num_bytes_recovered":69337088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":3189}],"blocked_by":[],"up_primary":1,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":727,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708937+0000","last_change":"2024-02-12T17:13:08.874063+0000","last_active":"2024-02-12T17:15:16.708937+0000","last_peered":"2024-02-12T17:15:16.708937+0000","last_clean":"2024-02-12T17:15:16.708937+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:15:16.708937+0000","last_undegraded":"2024-02-12T17:15:16.708937+0000","last_fullsized":"2024-02-12T17:15:16.708937+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:13:08.873961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:13:08.873961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":208,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.717927+0000","last_change":"2024-02-12T17:14:14.843264+0000","last_active":"2024-02-12T17:15:16.717927+0000","last_peered":"2024-02-12T17:15:16.717927+0000","last_clean":"2024-02-12T17:15:16.717927+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:15:16.717927+0000","last_undegraded":"2024-02-12T17:15:16.717927+0000","last_fullsized":"2024-02-12T17:15:16.717927+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:56.688982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.688982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":873,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.711376+0000","last_change":"2024-02-12T17:14:09.517724+0000","last_active":"2024-02-12T17:15:16.711376+0000","last_peered":"2024-02-12T17:15:16.711376+0000","last_clean":"2024-02-12T17:15:16.711376+0000","last_became_active":"2024-02-12T17:14:08.740691+0000","last_became_peered":"2024-02-12T17:14:08.740691+0000","last_unstale":"2024-02-12T17:15:16.711376+0000","last_undegraded":"2024-02-12T17:15:16.711376+0000","last_fullsized":"2024-02-12T17:15:16.711376+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:12:05.850456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.850456+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"655'32","reported_seq":627,"reported_epoch":688,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:17.643241+0000","last_change":"2024-02-12T17:14:09.746021+0000","last_active":"2024-02-12T17:15:17.643241+0000","last_peered":"2024-02-12T17:15:17.643241+0000","last_clean":"2024-02-12T17:14:07.617720+0000","last_became_active":"2024-02-12T17:14:09.745704+0000","last_became_peered":"2024-02-12T17:14:09.745704+0000","last_unstale":"2024-02-12T17:15:17.643241+0000","last_undegraded":"2024-02-12T17:15:17.643241+0000","last_fullsized":"2024-02-12T17:15:17.643241+0000","mapping_epoch":652,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3","0","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"689'12532","reported_seq":12647,"reported_epoch":689,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:28.722705+0000","last_change":"2024-02-12T17:14:09.843467+0000","last_active":"2024-02-12T17:15:28.722705+0000","last_peered":"2024-02-12T17:15:28.722705+0000","last_clean":"2024-02-12T17:14:07.683370+0000","last_became_active":"2024-02-12T17:14:09.782739+0000","last_became_peered":"2024-02-12T17:14:09.782739+0000","last_unstale":"2024-02-12T17:15:28.722705+0000","last_undegraded":"2024-02-12T17:15:28.722705+0000","last_fullsized":"2024-02-12T17:15:28.722705+0000","mapping_epoch":652,"log_start":"619'6451","ondisk_log_start":"619'6451","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6081,"ondisk_log_size":6081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":204472320,"num_objects":3120,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3120,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12532,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":3120}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":397,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.233690+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:15:16.233690+0000","last_peered":"2024-02-12T17:15:16.233690+0000","last_clean":"2024-02-12T17:15:16.233690+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:15:16.233690+0000","last_undegraded":"2024-02-12T17:15:16.233690+0000","last_fullsized":"2024-02-12T17:15:16.233690+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"642'31","reported_seq":1090,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.649270+0000","last_change":"2024-02-12T17:13:53.773053+0000","last_active":"2024-02-12T17:15:27.649270+0000","last_peered":"2024-02-12T17:15:27.649270+0000","last_clean":"2024-02-12T17:15:27.649270+0000","last_became_active":"2024-02-12T17:13:53.772744+0000","last_became_peered":"2024-02-12T17:13:53.772744+0000","last_unstale":"2024-02-12T17:15:27.649270+0000","last_undegraded":"2024-02-12T17:15:27.649270+0000","last_fullsized":"2024-02-12T17:15:27.649270+0000","mapping_epoch":638,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'28","last_scrub_stamp":"2024-02-12T17:12:07.826227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:12:07.826227+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":402,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708111+0000","last_change":"2024-02-12T17:14:10.756477+0000","last_active":"2024-02-12T17:15:16.708111+0000","last_peered":"2024-02-12T17:15:16.708111+0000","last_clean":"2024-02-12T17:15:16.708111+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:15:16.708111+0000","last_undegraded":"2024-02-12T17:15:16.708111+0000","last_fullsized":"2024-02-12T17:15:16.708111+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:16.873570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.873570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":262,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.715865+0000","last_change":"2024-02-12T17:13:46.807564+0000","last_active":"2024-02-12T17:15:16.715865+0000","last_peered":"2024-02-12T17:15:16.715865+0000","last_clean":"2024-02-12T17:15:16.715865+0000","last_became_active":"2024-02-12T17:13:46.807214+0000","last_became_peered":"2024-02-12T17:13:46.807214+0000","last_unstale":"2024-02-12T17:15:16.715865+0000","last_undegraded":"2024-02-12T17:15:16.715865+0000","last_fullsized":"2024-02-12T17:15:16.715865+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:04.705646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.705646+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"688'12573","reported_seq":14908,"reported_epoch":688,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:15:26.597203+0000","last_change":"2024-02-12T17:14:09.819105+0000","last_active":"2024-02-12T17:15:26.597203+0000","last_peered":"2024-02-12T17:15:26.597203+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.810504+0000","last_became_peered":"2024-02-12T17:13:46.810504+0000","last_unstale":"2024-02-12T17:15:26.597203+0000","last_undegraded":"2024-02-12T17:13:46.690611+0000","last_fullsized":"2024-02-12T17:13:46.648648+0000","mapping_epoch":631,"log_start":"628'7571","ondisk_log_start":"628'7571","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5002,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":216596480,"num_objects":3305,"num_object_clones":0,"num_object_copies":9915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12573,"num_write_kb":508096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":72876032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3305}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":732,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.711849+0000","last_change":"2024-02-12T17:14:08.740896+0000","last_active":"2024-02-12T17:15:16.711849+0000","last_peered":"2024-02-12T17:15:16.711849+0000","last_clean":"2024-02-12T17:15:16.711849+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:15:16.711849+0000","last_undegraded":"2024-02-12T17:15:16.711849+0000","last_fullsized":"2024-02-12T17:15:16.711849+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"668'27","reported_seq":1115,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.649464+0000","last_change":"2024-02-12T17:14:24.235718+0000","last_active":"2024-02-12T17:15:27.649464+0000","last_peered":"2024-02-12T17:15:27.649464+0000","last_clean":"2024-02-12T17:15:27.649464+0000","last_became_active":"2024-02-12T17:14:24.235386+0000","last_became_peered":"2024-02-12T17:14:24.235386+0000","last_unstale":"2024-02-12T17:15:27.649464+0000","last_undegraded":"2024-02-12T17:15:27.649464+0000","last_fullsized":"2024-02-12T17:15:27.649464+0000","mapping_epoch":665,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'24","last_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.647212+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":98,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.222881+0000","last_change":"2024-02-12T17:14:05.106537+0000","last_active":"2024-02-12T17:15:16.222881+0000","last_peered":"2024-02-12T17:15:16.222881+0000","last_clean":"2024-02-12T17:15:16.222881+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:15:16.222881+0000","last_undegraded":"2024-02-12T17:15:16.222881+0000","last_fullsized":"2024-02-12T17:15:16.222881+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":146,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.233407+0000","last_change":"2024-02-12T17:13:46.819259+0000","last_active":"2024-02-12T17:15:16.233407+0000","last_peered":"2024-02-12T17:15:16.233407+0000","last_clean":"2024-02-12T17:15:16.233407+0000","last_became_active":"2024-02-12T17:13:46.818771+0000","last_became_peered":"2024-02-12T17:13:46.818771+0000","last_unstale":"2024-02-12T17:15:16.233407+0000","last_undegraded":"2024-02-12T17:15:16.233407+0000","last_fullsized":"2024-02-12T17:15:16.233407+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"689'12346","reported_seq":25267,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:28.694877+0000","last_change":"2024-02-12T17:14:32.710386+0000","last_active":"2024-02-12T17:15:28.694877+0000","last_peered":"2024-02-12T17:15:28.694877+0000","last_clean":"2024-02-12T17:15:28.694877+0000","last_became_active":"2024-02-12T17:14:32.696812+0000","last_became_peered":"2024-02-12T17:14:32.696812+0000","last_unstale":"2024-02-12T17:15:28.694877+0000","last_undegraded":"2024-02-12T17:15:28.694877+0000","last_fullsized":"2024-02-12T17:15:28.694877+0000","mapping_epoch":671,"log_start":"619'6300","ondisk_log_start":"619'6300","created":597,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6046,"ondisk_log_size":6046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":201588736,"num_objects":3076,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12346,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6402,"num_bytes_recovered":419561472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":78,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.228323+0000","last_change":"2024-02-12T17:13:46.805110+0000","last_active":"2024-02-12T17:15:16.228323+0000","last_peered":"2024-02-12T17:15:16.228323+0000","last_clean":"2024-02-12T17:15:16.228323+0000","last_became_active":"2024-02-12T17:13:46.802275+0000","last_became_peered":"2024-02-12T17:13:46.802275+0000","last_unstale":"2024-02-12T17:15:16.228323+0000","last_undegraded":"2024-02-12T17:15:16.228323+0000","last_fullsized":"2024-02-12T17:15:16.228323+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":618,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.649277+0000","last_change":"2024-02-12T17:13:07.711701+0000","last_active":"2024-02-12T17:15:27.649277+0000","last_peered":"2024-02-12T17:15:27.649277+0000","last_clean":"2024-02-12T17:15:27.649277+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:15:27.649277+0000","last_undegraded":"2024-02-12T17:15:27.649277+0000","last_fullsized":"2024-02-12T17:15:27.649277+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:13:07.711601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:07.711601+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":98,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223035+0000","last_change":"2024-02-12T17:14:00.999940+0000","last_active":"2024-02-12T17:15:16.223035+0000","last_peered":"2024-02-12T17:15:16.223035+0000","last_clean":"2024-02-12T17:15:16.223035+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:15:16.223035+0000","last_undegraded":"2024-02-12T17:15:16.223035+0000","last_fullsized":"2024-02-12T17:15:16.223035+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":491,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.233797+0000","last_change":"2024-02-12T17:13:46.806094+0000","last_active":"2024-02-12T17:15:16.233797+0000","last_peered":"2024-02-12T17:15:16.233797+0000","last_clean":"2024-02-12T17:15:16.233797+0000","last_became_active":"2024-02-12T17:13:46.805712+0000","last_became_peered":"2024-02-12T17:13:46.805712+0000","last_unstale":"2024-02-12T17:15:16.233797+0000","last_undegraded":"2024-02-12T17:15:16.233797+0000","last_fullsized":"2024-02-12T17:15:16.233797+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"688'12553","reported_seq":12652,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:26.568688+0000","last_change":"2024-02-12T17:11:59.496950+0000","last_active":"2024-02-12T17:15:26.568688+0000","last_peered":"2024-02-12T17:15:26.568688+0000","last_clean":"2024-02-12T17:15:26.568688+0000","last_became_active":"2024-02-12T17:11:59.496585+0000","last_became_peered":"2024-02-12T17:11:59.496585+0000","last_unstale":"2024-02-12T17:15:26.568688+0000","last_undegraded":"2024-02-12T17:15:26.568688+0000","last_fullsized":"2024-02-12T17:15:26.568688+0000","mapping_epoch":597,"log_start":"628'7548","ondisk_log_start":"628'7548","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5005,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":216989696,"num_objects":3311,"num_object_clones":0,"num_object_copies":9933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12553,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":417,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.232806+0000","last_change":"2024-02-12T17:13:21.603687+0000","last_active":"2024-02-12T17:15:16.232806+0000","last_peered":"2024-02-12T17:15:16.232806+0000","last_clean":"2024-02-12T17:15:16.232806+0000","last_became_active":"2024-02-12T17:13:21.603332+0000","last_became_peered":"2024-02-12T17:13:21.603332+0000","last_unstale":"2024-02-12T17:15:16.232806+0000","last_undegraded":"2024-02-12T17:15:16.232806+0000","last_fullsized":"2024-02-12T17:15:16.232806+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":109,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223764+0000","last_change":"2024-02-12T17:13:46.804446+0000","last_active":"2024-02-12T17:15:16.223764+0000","last_peered":"2024-02-12T17:15:16.223764+0000","last_clean":"2024-02-12T17:15:16.223764+0000","last_became_active":"2024-02-12T17:13:46.804033+0000","last_became_peered":"2024-02-12T17:13:46.804033+0000","last_unstale":"2024-02-12T17:15:16.223764+0000","last_undegraded":"2024-02-12T17:15:16.223764+0000","last_fullsized":"2024-02-12T17:15:16.223764+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.793569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.793569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":712,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708587+0000","last_change":"2024-02-12T17:13:56.661004+0000","last_active":"2024-02-12T17:15:16.708587+0000","last_peered":"2024-02-12T17:15:16.708587+0000","last_clean":"2024-02-12T17:15:16.708587+0000","last_became_active":"2024-02-12T17:13:56.660707+0000","last_became_peered":"2024-02-12T17:13:56.660707+0000","last_unstale":"2024-02-12T17:15:16.708587+0000","last_undegraded":"2024-02-12T17:15:16.708587+0000","last_fullsized":"2024-02-12T17:15:16.708587+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.685601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.685601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":178,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.223658+0000","last_change":"2024-02-12T17:13:46.757888+0000","last_active":"2024-02-12T17:15:16.223658+0000","last_peered":"2024-02-12T17:15:16.223658+0000","last_clean":"2024-02-12T17:15:16.223658+0000","last_became_active":"2024-02-12T17:13:46.757450+0000","last_became_peered":"2024-02-12T17:13:46.757450+0000","last_unstale":"2024-02-12T17:15:16.223658+0000","last_undegraded":"2024-02-12T17:15:16.223658+0000","last_fullsized":"2024-02-12T17:15:16.223658+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:56.880282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.880282+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.5","version":"689'12449","reported_seq":12552,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:28.949993+0000","last_change":"2024-02-12T17:13:12.678687+0000","last_active":"2024-02-12T17:15:28.949993+0000","last_peered":"2024-02-12T17:15:28.949993+0000","last_clean":"2024-02-12T17:15:28.949993+0000","last_became_active":"2024-02-12T17:11:59.009543+0000","last_became_peered":"2024-02-12T17:11:59.009543+0000","last_unstale":"2024-02-12T17:15:28.949993+0000","last_undegraded":"2024-02-12T17:15:28.949993+0000","last_fullsized":"2024-02-12T17:15:28.949993+0000","mapping_epoch":597,"log_start":"623'6729","ondisk_log_start":"623'6729","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5720,"ondisk_log_size":5720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208732160,"num_objects":3185,"num_object_clones":0,"num_object_copies":9555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12449,"num_write_kb":500288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":712,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.708207+0000","last_change":"2024-02-12T17:12:08.013330+0000","last_active":"2024-02-12T17:15:16.708207+0000","last_peered":"2024-02-12T17:15:16.708207+0000","last_clean":"2024-02-12T17:15:16.708207+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:15:16.708207+0000","last_undegraded":"2024-02-12T17:15:16.708207+0000","last_fullsized":"2024-02-12T17:15:16.708207+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.013247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.013247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":608,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.241755+0000","last_change":"2024-02-12T17:12:04.711418+0000","last_active":"2024-02-12T17:15:16.241755+0000","last_peered":"2024-02-12T17:15:16.241755+0000","last_clean":"2024-02-12T17:15:16.241755+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:15:16.241755+0000","last_undegraded":"2024-02-12T17:15:16.241755+0000","last_fullsized":"2024-02-12T17:15:16.241755+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:04.711333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.711333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":467,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.711167+0000","last_change":"2024-02-12T17:14:08.740752+0000","last_active":"2024-02-12T17:15:16.711167+0000","last_peered":"2024-02-12T17:15:16.711167+0000","last_clean":"2024-02-12T17:15:16.711167+0000","last_became_active":"2024-02-12T17:14:08.740430+0000","last_became_peered":"2024-02-12T17:14:08.740430+0000","last_unstale":"2024-02-12T17:15:16.711167+0000","last_undegraded":"2024-02-12T17:15:16.711167+0000","last_fullsized":"2024-02-12T17:15:16.711167+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":244,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.716775+0000","last_change":"2024-02-12T17:13:46.559877+0000","last_active":"2024-02-12T17:15:16.716775+0000","last_peered":"2024-02-12T17:15:16.716775+0000","last_clean":"2024-02-12T17:15:16.716775+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:15:16.716775+0000","last_undegraded":"2024-02-12T17:15:16.716775+0000","last_fullsized":"2024-02-12T17:15:16.716775+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"689'12690","reported_seq":12790,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:28.946712+0000","last_change":"2024-02-12T17:11:59.007574+0000","last_active":"2024-02-12T17:15:28.946712+0000","last_peered":"2024-02-12T17:15:28.946712+0000","last_clean":"2024-02-12T17:15:28.946712+0000","last_became_active":"2024-02-12T17:11:59.007300+0000","last_became_peered":"2024-02-12T17:11:59.007300+0000","last_unstale":"2024-02-12T17:15:28.946712+0000","last_undegraded":"2024-02-12T17:15:28.946712+0000","last_fullsized":"2024-02-12T17:15:28.946712+0000","mapping_epoch":597,"log_start":"624'6967","ondisk_log_start":"624'6967","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5723,"ondisk_log_size":5723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":209977344,"num_objects":3204,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12690,"num_write_kb":508608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":46,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.249309+0000","last_change":"2024-02-12T17:14:08.859503+0000","last_active":"2024-02-12T17:15:16.249309+0000","last_peered":"2024-02-12T17:15:16.249309+0000","last_clean":"2024-02-12T17:15:16.249309+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:15:16.249309+0000","last_undegraded":"2024-02-12T17:15:16.249309+0000","last_fullsized":"2024-02-12T17:15:16.249309+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:03.705889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:13:03.705889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":608,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.231243+0000","last_change":"2024-02-12T17:14:08.762783+0000","last_active":"2024-02-12T17:15:16.231243+0000","last_peered":"2024-02-12T17:15:16.231243+0000","last_clean":"2024-02-12T17:15:16.231243+0000","last_became_active":"2024-02-12T17:14:08.760654+0000","last_became_peered":"2024-02-12T17:14:08.760654+0000","last_unstale":"2024-02-12T17:15:16.231243+0000","last_undegraded":"2024-02-12T17:15:16.231243+0000","last_fullsized":"2024-02-12T17:15:16.231243+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:09.044325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:09.044325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":386,"reported_epoch":687,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:15:16.231909+0000","last_change":"2024-02-12T17:14:16.753241+0000","last_active":"2024-02-12T17:15:16.231909+0000","last_peered":"2024-02-12T17:15:16.231909+0000","last_clean":"2024-02-12T17:13:44.508652+0000","last_became_active":"2024-02-12T17:13:46.563455+0000","last_became_peered":"2024-02-12T17:13:46.563455+0000","last_unstale":"2024-02-12T17:15:16.231909+0000","last_undegraded":"2024-02-12T17:15:16.231909+0000","last_fullsized":"2024-02-12T17:15:16.231909+0000","mapping_epoch":630,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":56,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.241299+0000","last_change":"2024-02-12T17:14:10.285684+0000","last_active":"2024-02-12T17:15:16.241299+0000","last_peered":"2024-02-12T17:15:16.241299+0000","last_clean":"2024-02-12T17:15:16.241299+0000","last_became_active":"2024-02-12T17:14:08.871051+0000","last_became_peered":"2024-02-12T17:14:08.871051+0000","last_unstale":"2024-02-12T17:15:16.241299+0000","last_undegraded":"2024-02-12T17:15:16.241299+0000","last_fullsized":"2024-02-12T17:15:16.241299+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:50.764686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.764686+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"689'12452","reported_seq":19458,"reported_epoch":689,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:28.469227+0000","last_change":"2024-02-12T17:14:39.687781+0000","last_active":"2024-02-12T17:15:28.469227+0000","last_peered":"2024-02-12T17:15:28.469227+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.886954+0000","last_became_peered":"2024-02-12T17:14:08.886954+0000","last_unstale":"2024-02-12T17:15:28.469227+0000","last_undegraded":"2024-02-12T17:14:08.719570+0000","last_fullsized":"2024-02-12T17:14:08.707803+0000","mapping_epoch":651,"log_start":"616'5978","ondisk_log_start":"616'5978","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6474,"ondisk_log_size":6474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":209584128,"num_objects":3198,"num_object_clones":0,"num_object_copies":9594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3198,"num_objects_misplaced":3198,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3435,"num_bytes_recovered":225050624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":3198}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":695,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.712591+0000","last_change":"2024-02-12T17:12:11.906816+0000","last_active":"2024-02-12T17:15:16.712591+0000","last_peered":"2024-02-12T17:15:16.712591+0000","last_clean":"2024-02-12T17:15:16.712591+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:15:16.712591+0000","last_undegraded":"2024-02-12T17:15:16.712591+0000","last_fullsized":"2024-02-12T17:15:16.712591+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:11.906737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:11.906737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":372,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.232152+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:15:16.232152+0000","last_peered":"2024-02-12T17:15:16.232152+0000","last_clean":"2024-02-12T17:15:16.232152+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:15:16.232152+0000","last_undegraded":"2024-02-12T17:15:16.232152+0000","last_fullsized":"2024-02-12T17:15:16.232152+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":813,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:16.710652+0000","last_change":"2024-02-12T17:12:16.820857+0000","last_active":"2024-02-12T17:15:16.710652+0000","last_peered":"2024-02-12T17:15:16.710652+0000","last_clean":"2024-02-12T17:15:16.710652+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:15:16.710652+0000","last_undegraded":"2024-02-12T17:15:16.710652+0000","last_fullsized":"2024-02-12T17:15:16.710652+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:12:16.820782+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.820782+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"632'35","reported_seq":1179,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.649311+0000","last_change":"2024-02-12T17:13:46.691596+0000","last_active":"2024-02-12T17:15:27.649311+0000","last_peered":"2024-02-12T17:15:27.649311+0000","last_clean":"2024-02-12T17:15:27.649311+0000","last_became_active":"2024-02-12T17:13:46.691213+0000","last_became_peered":"2024-02-12T17:13:46.691213+0000","last_unstale":"2024-02-12T17:15:27.649311+0000","last_undegraded":"2024-02-12T17:15:27.649311+0000","last_fullsized":"2024-02-12T17:15:27.649311+0000","mapping_epoch":631,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'29","last_scrub_stamp":"2024-02-12T17:12:47.788327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.788327+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3361931288,"num_objects":51300,"num_object_clones":0,"num_object_copies":153900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22282,"num_objects_misplaced":21280,"num_objects_unfound":0,"num_objects_dirty":51300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200076,"num_write_kb":8043969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19887,"num_bytes_recovered":1302986752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9091493888,"data_stored":9091481672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91379,"ondisk_log_size":91379,"up":48,"acting":40,"num_store_stats":8},{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":310,"num_read_kb":305,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":16216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1029,"ondisk_log_size":1029,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234297,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3003896,"kb_used_data":1515696,"kb_used_omap":0,"kb_used_meta":1488064,"kb_avail":90741256,"statfs":{"total":95995035648,"available":92919046144,"internally_reserved":0,"allocated":1552072704,"data_stored":1550725025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1523777536},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":2.6899999999999999,"5min":1.9910000000000001,"15min":2.077},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":15.856999999999999,"5min":15.856999999999999,"15min":16.759},"last":0.65700000000000003},{"interface":"front","average":{"1min":2.79,"5min":1.837,"15min":1.915},"min":{"1min":0.27600000000000002,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":12.565,"5min":12.565,"15min":17.844000000000001},"last":0.251}]},{"osd":1,"last update":"Mon Feb 12 17:14:47 2024","interfaces":[{"interface":"back","average":{"1min":2.7890000000000001,"5min":2.7890000000000001,"15min":2.7890000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.8989999999999991,"5min":9.8989999999999991,"15min":9.8989999999999991},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.468,"5min":2.468,"15min":2.468},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":9.9120000000000008,"5min":9.9120000000000008,"15min":9.9120000000000008},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":3.431,"5min":2.8359999999999999,"15min":3.1419999999999999},"min":{"1min":0.34599999999999997,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":23.611000000000001,"5min":42.162999999999997,"15min":218.35300000000001},"last":1.595},{"interface":"front","average":{"1min":3.8090000000000002,"5min":2.8639999999999999,"15min":3.2269999999999999},"min":{"1min":0.47099999999999997,"5min":0.247,"15min":0.247},"max":{"1min":15.672000000000001,"5min":42.109999999999999,"15min":218.42099999999999},"last":2.3860000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":2.415,"15min":2.0939999999999999},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.255},"max":{"1min":7.5830000000000002,"5min":15.811999999999999,"15min":15.811999999999999},"last":5.8639999999999999},{"interface":"front","average":{"1min":4.1710000000000003,"5min":2.847,"15min":2.5350000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":11.009,"5min":13.452,"15min":31.152000000000001},"last":0.752}]},{"osd":4,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.421,"15min":1.8540000000000001},"min":{"1min":0.247,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":3.4969999999999999,"5min":20.672999999999998,"15min":20.672999999999998},"last":0.312},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.976,"15min":2.1509999999999998},"min":{"1min":0.29299999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.6769999999999996,"5min":38.390000000000001,"15min":38.390000000000001},"last":0.23300000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.8029999999999999,"15min":2.407},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":4.79,"5min":20.702000000000002,"15min":56.560000000000002},"last":0.28100000000000003},{"interface":"front","average":{"1min":2.5089999999999999,"5min":1.4690000000000001,"15min":2.1139999999999999},"min":{"1min":0.28499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":9.7170000000000005,"5min":10.669,"15min":53.963999999999999},"last":0.25800000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.4690000000000001,"15min":1.734},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":5.2539999999999996,"5min":21.620000000000001,"15min":21.620000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.722,"15min":1.8759999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29199999999999998},"max":{"1min":3.7519999999999998,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.24399999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528638,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2521672,"kb_used_data":1625216,"kb_used_omap":0,"kb_used_meta":896448,"kb_avail":91223480,"statfs":{"total":95995035648,"available":93412843520,"internally_reserved":0,"allocated":1664221184,"data_stored":1662857765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917962752},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":1.9930000000000001,"5min":1.7629999999999999,"15min":1.9510000000000001},"min":{"1min":0.23100000000000001,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":8.0920000000000005,"5min":33.576999999999998,"15min":33.576999999999998},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.321,"5min":2.125,"15min":1.919},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":6.1100000000000003,"5min":33.569000000000003,"15min":33.569000000000003},"last":0.85299999999999998}]},{"osd":1,"last update":"Mon Feb 12 17:14:46 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":13.805999999999999,"5min":13.805999999999999,"15min":13.805999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":2.194,"5min":2.194,"15min":2.194},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":13.778,"5min":13.778,"15min":13.778},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":2.5680000000000001,"5min":2.75,"15min":2.5649999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.311,"5min":33.594999999999999,"15min":56.798000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.968,"5min":2.234,"15min":2.3149999999999999},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":9.2479999999999993,"5min":33.526000000000003,"15min":56.780999999999999},"last":14.045999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":2.7469999999999999,"5min":3.3330000000000002,"15min":2.3170000000000002},"min":{"1min":0.35399999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":10.712,"5min":58.420999999999999,"15min":58.420999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":3.5590000000000002,"5min":3.5790000000000002,"15min":2.7450000000000001},"min":{"1min":0.33200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":14.69,"5min":58.709000000000003,"15min":58.709000000000003},"last":2.2360000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.514,"15min":1.585},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":11.012,"5min":33.706000000000003,"15min":33.706000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.823,"15min":1.635},"min":{"1min":0.28799999999999998,"5min":0.222,"15min":0.222},"max":{"1min":10.948,"5min":34.895000000000003,"15min":34.895000000000003},"last":0.372}]},{"osd":5,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.877,"15min":1.827},"min":{"1min":0.30599999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":10.77,"5min":34.869,"15min":34.869},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.4990000000000001,"15min":2.0249999999999999},"min":{"1min":0.32100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":11.1,"5min":33.591000000000001,"15min":33.591000000000001},"last":0.38100000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.677,"15min":1.871},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":6.1310000000000002,"5min":35.122999999999998,"15min":35.122999999999998},"last":0.44},{"interface":"front","average":{"1min":1.28,"5min":1.7390000000000001,"15min":1.901},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.26500000000000001},"max":{"1min":11.081,"5min":34.045000000000002,"15min":34.045000000000002},"last":0.35899999999999999}]}]},{"osd":1,"up_from":630,"seq":2705829396501,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":645516,"kb_used_data":315076,"kb_used_omap":3,"kb_used_meta":330428,"kb_avail":93099636,"statfs":{"total":95995035648,"available":95334027264,"internally_reserved":0,"allocated":322637824,"data_stored":321263510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":338358333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4849999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":14.67,"5min":14.67,"15min":14.67},"last":0.33900000000000002},{"interface":"front","average":{"1min":3.4209999999999998,"5min":3.4209999999999998,"15min":3.4209999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"last":6.2359999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":3.5819999999999999,"5min":3.5819999999999999,"15min":3.5819999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":8.766,"5min":8.766,"15min":8.766},"last":0.36499999999999999},{"interface":"front","average":{"1min":3.4969999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.880000000000001,"5min":12.880000000000001,"15min":12.880000000000001},"last":6.2110000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":3.6429999999999998,"5min":3.6429999999999998,"15min":3.6429999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":10.375,"5min":10.375,"15min":10.375},"last":0.83699999999999997},{"interface":"front","average":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":13.564,"5min":13.564,"15min":13.564},"last":5.9749999999999996}]},{"osd":4,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":5.9089999999999998},{"interface":"front","average":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":2.7810000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":12.805999999999999,"5min":12.805999999999999,"15min":12.805999999999999},"last":6.2830000000000004}]},{"osd":5,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":3.2160000000000002,"5min":3.2160000000000002,"15min":3.2160000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":12.769,"5min":12.769,"15min":12.769},"last":6.2610000000000001},{"interface":"front","average":{"1min":2.113,"5min":2.113,"15min":2.113},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":6.3239999999999998,"5min":6.3239999999999998,"15min":6.3239999999999998},"last":4.0090000000000003}]},{"osd":6,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":2.8300000000000001,"5min":2.8300000000000001,"15min":2.8300000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":12.720000000000001,"5min":12.720000000000001,"15min":12.720000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":8.0890000000000004,"5min":8.0890000000000004,"15min":8.0890000000000004},"last":5.9429999999999996}]},{"osd":7,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":2.964,"5min":2.964,"15min":2.964},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":12.829000000000001,"5min":12.829000000000001,"15min":12.829000000000001},"last":6.1180000000000003},{"interface":"front","average":{"1min":2.3919999999999999,"5min":2.3919999999999999,"15min":2.3919999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":6.4649999999999999},"last":1.9910000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738714,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215012,"kb_used_data":314716,"kb_used_omap":0,"kb_used_meta":900288,"kb_avail":92530140,"statfs":{"total":95995035648,"available":94750863360,"internally_reserved":0,"allocated":322269184,"data_stored":320934750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":921894912},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:14:48 2024","interfaces":[{"interface":"back","average":{"1min":3.4289999999999998,"5min":3.4289999999999998,"15min":3.4289999999999998},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":13.936999999999999,"5min":13.936999999999999,"15min":13.936999999999999},"last":1.534},{"interface":"front","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":13.787000000000001,"5min":13.787000000000001,"15min":13.787000000000001},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:14:29 2024","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.8700000000000001,"15min":1.9390000000000001},"min":{"1min":0.21299999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.1760000000000002,"5min":19.312000000000001,"15min":19.312000000000001},"last":1.776},{"interface":"front","average":{"1min":2.4849999999999999,"5min":3.6709999999999998,"15min":2.4020000000000001},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":7.9690000000000003,"5min":26.039999999999999,"15min":26.039999999999999},"last":6.6390000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:14:43 2024","interfaces":[{"interface":"back","average":{"1min":3.7280000000000002,"5min":3.0590000000000002,"15min":2.5859999999999999},"min":{"1min":0.23000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":25.143000000000001,"5min":25.143000000000001,"15min":87.311000000000007},"last":1.5149999999999999},{"interface":"front","average":{"1min":4.6440000000000001,"5min":3.4100000000000001,"15min":2.8519999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":24.690999999999999,"5min":24.690999999999999,"15min":90.991},"last":1.7450000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:15:04 2024","interfaces":[{"interface":"back","average":{"1min":2.363,"5min":1.6000000000000001,"15min":1.5629999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.23599999999999999},"max":{"1min":13.861000000000001,"5min":15.141999999999999,"15min":16.888999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.6030000000000002,"15min":2.1000000000000001},"min":{"1min":0.30499999999999999,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":13.503,"5min":26.082999999999998,"15min":26.082999999999998},"last":6.4249999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:14:37 2024","interfaces":[{"interface":"back","average":{"1min":1.9219999999999999,"5min":2.4729999999999999,"15min":2.2320000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":6.3609999999999998,"5min":17.754000000000001,"15min":50.404000000000003},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":1.9159999999999999,"15min":2.0659999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":13.837,"5min":15.164,"15min":49.979999999999997},"last":0.504}]},{"osd":6,"last update":"Mon Feb 12 17:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":3.4119999999999999,"15min":2.343},"min":{"1min":0.39900000000000002,"5min":0.35499999999999998,"15min":0.29999999999999999},"max":{"1min":13.48,"5min":48.780999999999999,"15min":48.780999999999999},"last":6.4610000000000003},{"interface":"front","average":{"1min":2.2250000000000001,"5min":3.1280000000000001,"15min":2.3109999999999999},"min":{"1min":0.36399999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":13.962999999999999,"5min":49.067999999999998,"15min":49.067999999999998},"last":1.4950000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:14:29 2024","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":2.0070000000000001,"15min":2.0339999999999998},"min":{"1min":0.38800000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":7.8609999999999998,"5min":15.183999999999999,"15min":19.373000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":2.04,"5min":2.8940000000000001,"15min":2.496},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.6420000000000003,"5min":26.135000000000002,"15min":34.722999999999999},"last":6.484}]}]},{"osd":2,"up_from":18,"seq":77309411664,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2935024,"kb_used_data":1436840,"kb_used_omap":0,"kb_used_meta":1498176,"kb_avail":90810128,"statfs":{"total":95995035648,"available":92989571072,"internally_reserved":0,"allocated":1471324160,"data_stored":1469985163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1534132224},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:14:29 2024","interfaces":[{"interface":"back","average":{"1min":2.9470000000000001,"5min":1.8680000000000001,"15min":1.6890000000000001},"min":{"1min":0.28999999999999998,"5min":0.17899999999999999,"15min":0.17000000000000001},"max":{"1min":9.7620000000000005,"5min":12.839,"15min":36.100999999999999},"last":0.437},{"interface":"front","average":{"1min":1.8280000000000001,"5min":1.988,"15min":1.9259999999999999},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.193},"max":{"1min":7.2590000000000003,"5min":10.193,"15min":36.299999999999997},"last":3.2050000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:14:47 2024","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":4.016,"15min":4.016},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":11.612,"5min":11.612,"15min":11.612},"last":0.46300000000000002},{"interface":"front","average":{"1min":5.8940000000000001,"5min":5.8940000000000001,"15min":5.8940000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":22.52,"5min":22.52,"15min":22.52},"last":6.0279999999999996}]},{"osd":3,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":8.657,"5min":3.8519999999999999,"15min":2.6160000000000001},"min":{"1min":0.32300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":29.489999999999998,"5min":40.759999999999998,"15min":40.759999999999998},"last":1.361},{"interface":"front","average":{"1min":4.4900000000000002,"5min":2.702,"15min":2.1899999999999999},"min":{"1min":0.38900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":12.33,"5min":17.358000000000001,"15min":21.702999999999999},"last":2.641}]},{"osd":4,"last update":"Mon Feb 12 17:15:22 2024","interfaces":[{"interface":"back","average":{"1min":3.008,"5min":2.0129999999999999,"15min":1.98},"min":{"1min":0.27600000000000002,"5min":0.245,"15min":0.245},"max":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.504},{"interface":"front","average":{"1min":2.8220000000000001,"5min":1.8939999999999999,"15min":2.0510000000000002},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.744,"5min":11.744,"15min":20.149000000000001},"last":2.5609999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:14:29 2024","interfaces":[{"interface":"back","average":{"1min":2.1339999999999999,"5min":1.7430000000000001,"15min":1.972},"min":{"1min":0.33700000000000002,"5min":0.217,"15min":0.217},"max":{"1min":8.9280000000000008,"5min":11.351000000000001,"15min":32.57},"last":0.52800000000000002},{"interface":"front","average":{"1min":3.1320000000000001,"5min":2.0579999999999998,"15min":2.2200000000000002},"min":{"1min":0.314,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":10.787000000000001,"5min":10.896000000000001,"15min":31.905999999999999},"last":0.30199999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":2.9550000000000001,"5min":2.0720000000000001,"15min":1.8979999999999999},"min":{"1min":0.36899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":10.714,"5min":10.714,"15min":20.966999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":2.8420000000000001,"5min":2.0880000000000001,"15min":1.9179999999999999},"min":{"1min":0.432,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":11.834,"5min":11.834,"15min":20.193000000000001},"last":2.5950000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:14:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2850000000000001,"5min":1.8660000000000001,"15min":2.4740000000000002},"min":{"1min":0.39900000000000002,"5min":0.32700000000000001,"15min":0.32600000000000001},"max":{"1min":11.545,"5min":11.545,"15min":45.468000000000004},"last":2.6190000000000002},{"interface":"front","average":{"1min":3.8319999999999999,"5min":2.298,"15min":2.5539999999999998},"min":{"1min":0.33800000000000002,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":10.762,"5min":10.762,"15min":46.656999999999996},"last":0.372}]}]},{"osd":3,"up_from":26,"seq":111669150026,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2380552,"kb_used_data":1482944,"kb_used_omap":0,"kb_used_meta":897344,"kb_avail":91364600,"statfs":{"total":95995035648,"available":93557350400,"internally_reserved":0,"allocated":1518534656,"data_stored":1517164996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":918880256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:14:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7669999999999999,"5min":3.4870000000000001,"15min":2.4470000000000001},"min":{"1min":0.22,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":20.861999999999998,"5min":20.861999999999998,"15min":58.451999999999998},"last":4.7859999999999996},{"interface":"front","average":{"1min":6.109,"5min":3.8210000000000002,"15min":2.8370000000000002},"min":{"1min":0.28499999999999998,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.984999999999999,"5min":72.688000000000002,"15min":72.688000000000002},"last":3.915}]},{"osd":1,"last update":"Mon Feb 12 17:14:48 2024","interfaces":[{"interface":"back","average":{"1min":4.4809999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":20.638000000000002,"5min":20.638000000000002,"15min":20.638000000000002},"last":2.0600000000000001},{"interface":"front","average":{"1min":5.4550000000000001,"5min":5.4550000000000001,"15min":5.4550000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":16.361999999999998,"5min":16.361999999999998,"15min":16.361999999999998},"last":3.944}]},{"osd":2,"last update":"Mon Feb 12 17:14:54 2024","interfaces":[{"interface":"back","average":{"1min":3.2509999999999999,"5min":3.1000000000000001,"15min":2.0859999999999999},"min":{"1min":0.27400000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":10.819000000000001,"5min":40.954999999999998,"15min":40.954999999999998},"last":4.9859999999999998},{"interface":"front","average":{"1min":3.577,"5min":4.367,"15min":2.5790000000000002},"min":{"1min":0.27100000000000002,"5min":0.22,"15min":0.22},"max":{"1min":17.347000000000001,"5min":46.408000000000001,"15min":46.408000000000001},"last":8.8439999999999994}]},{"osd":4,"last update":"Mon Feb 12 17:15:11 2024","interfaces":[{"interface":"back","average":{"1min":4.2880000000000003,"5min":3.3460000000000001,"15min":2.5169999999999999},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":8.7420000000000009,"5min":72.659000000000006,"15min":72.659000000000006},"last":3.9820000000000002},{"interface":"front","average":{"1min":2.9500000000000002,"5min":2.758,"15min":2.0419999999999998},"min":{"1min":0.313,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.087999999999999,"5min":40.930999999999997,"15min":40.930999999999997},"last":2.0910000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:14:30 2024","interfaces":[{"interface":"back","average":{"1min":2.6030000000000002,"5min":3.0150000000000001,"15min":2.3809999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.253},"max":{"1min":8.3390000000000004,"5min":16.943000000000001,"15min":17.940999999999999},"last":3.5569999999999999},{"interface":"front","average":{"1min":2.8610000000000002,"5min":3.5920000000000001,"15min":2.8410000000000002},"min":{"1min":0.29999999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.3799999999999999,"5min":72.620000000000005,"15min":72.620000000000005},"last":4.04}]},{"osd":6,"last update":"Mon Feb 12 17:14:54 2024","interfaces":[{"interface":"back","average":{"1min":2.141,"5min":2.6920000000000002,"15min":1.927},"min":{"1min":0.31900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.9029999999999996,"5min":40.889000000000003,"15min":40.889000000000003},"last":2.1150000000000002},{"interface":"front","average":{"1min":2.6829999999999998,"5min":3.2730000000000001,"15min":2.319},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":8.1769999999999996,"5min":16.989999999999998,"15min":18.265000000000001},"last":4.9240000000000004}]},{"osd":7,"last update":"Mon Feb 12 17:15:11 2024","interfaces":[{"interface":"back","average":{"1min":3.7810000000000001,"5min":3.5169999999999999,"15min":2.9940000000000002},"min":{"1min":0.378,"5min":0.375,"15min":0.315},"max":{"1min":8.6690000000000005,"5min":72.712999999999994,"15min":72.712999999999994},"last":4.0119999999999996},{"interface":"front","average":{"1min":3.298,"5min":2.9420000000000002,"15min":2.778},"min":{"1min":0.439,"5min":0.38600000000000001,"15min":0.28299999999999997},"max":{"1min":8.2170000000000005,"5min":17.015000000000001,"15min":38.948},"last":3.5110000000000001}]}]},{"osd":4,"up_from":0,"seq":137438953800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757572,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2517856,"kb_used_data":1032536,"kb_used_omap":0,"kb_used_meta":1485312,"kb_avail":91227296,"statfs":{"total":95995035648,"available":93416751104,"internally_reserved":0,"allocated":1057316864,"data_stored":1055986255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1520959488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.696,"15min":1.7629999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":3.927,"5min":8.0350000000000001,"15min":46.365000000000002},"last":1.5389999999999999},{"interface":"front","average":{"1min":2.133,"5min":1.659,"15min":1.9350000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.20100000000000001},"max":{"1min":7.157,"5min":15.368,"15min":46.134},"last":4.5730000000000004}]},{"osd":1,"last update":"Mon Feb 12 17:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":9.484,"5min":9.484,"15min":9.484},"last":0.28699999999999998},{"interface":"front","average":{"1min":3.0649999999999999,"5min":3.0649999999999999,"15min":3.0649999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.6319999999999997,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":2.6000000000000001,"15min":2.036},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":9.0269999999999992,"5min":57.390999999999998,"15min":57.390999999999998},"last":0.247},{"interface":"front","average":{"1min":2.7719999999999998,"5min":2.843,"15min":2.0219999999999998},"min":{"1min":0.32100000000000001,"5min":0.214,"15min":0.214},"max":{"1min":13.366,"5min":57.134999999999998,"15min":57.134999999999998},"last":0.32100000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":2.0030000000000001,"5min":3.996,"15min":2.6920000000000002},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":7.1630000000000003,"5min":57.170999999999999,"15min":57.170999999999999},"last":0.311},{"interface":"front","average":{"1min":3.3530000000000002,"5min":3.4420000000000002,"15min":2.5579999999999998},"min":{"1min":0.27900000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.3829999999999991,"5min":57.484999999999999,"15min":57.484999999999999},"last":5.4820000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.169,"15min":1.556},"min":{"1min":0.29799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.9939999999999998,"5min":9.0429999999999993,"15min":12.656000000000001},"last":4.5430000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.4370000000000001,"15min":1.573},"min":{"1min":0.255,"5min":0.255,"15min":0.254},"max":{"1min":3.3250000000000002,"5min":10.619999999999999,"15min":20.550000000000001},"last":0.33900000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:15:03 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.194,"15min":1.613},"min":{"1min":0.28899999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.4109999999999996,"5min":14.675000000000001,"15min":14.675000000000001},"last":4.5030000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.6399999999999999,"15min":1.7769999999999999},"min":{"1min":0.28299999999999997,"5min":0.245,"15min":0.245},"max":{"1min":3.5979999999999999,"5min":12.864000000000001,"15min":14.323},"last":5.2720000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.5669999999999999,"15min":1.8180000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":5.8289999999999997,"5min":14.337999999999999,"15min":19.184999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.454,"5min":1.5449999999999999,"15min":1.8819999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":6.6950000000000003,"5min":14.637,"15min":20.274000000000001},"last":5.0279999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317390848,"data_stored":317390848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319029248,"data_stored":319029248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1464799232,"data_stored":1464795160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1508442112,"data_stored":1508442112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1244663808,"data_stored":1244659736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1042677760,"data_stored":1042677760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1655705600,"data_stored":1655701528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1538785280,"data_stored":1538785280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:15:32.365 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:15:32.550 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:32.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:32.662 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:32.693 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:32.725 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:32.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:32.755 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:15:32.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:32.856 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:15:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:15:32.856 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:31.090264+0000 mgr.y (mgr.24481) 972 : cluster [DBG] pgmap v1387: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 680 op/s; 22282/154476 objects degraded (14.424%); 21286/154476 objects misplaced (13.779%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:15:32.856 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:31.722648+0000 mon.a (mon.0) 2212 : cluster [WRN] Health check update: Degraded data redundancy: 22282/154476 objects degraded (14.424%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:32.856 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:31.728477+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2024-02-12T17:15:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:32.885 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:32.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:32.961 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:31.090264+0000 mgr.y (mgr.24481) 972 : cluster [DBG] pgmap v1387: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 680 op/s; 22282/154476 objects degraded (14.424%); 21286/154476 objects misplaced (13.779%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:15:32.961 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:31.722648+0000 mon.a (mon.0) 2212 : cluster [WRN] Health check update: Degraded data redundancy: 22282/154476 objects degraded (14.424%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:32.961 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:31.728477+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2024-02-12T17:15:32.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:31.090264+0000 mgr.y (mgr.24481) 972 : cluster [DBG] pgmap v1387: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 680 op/s; 22282/154476 objects degraded (14.424%); 21286/154476 objects misplaced (13.779%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:15:32.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:31.722648+0000 mon.a (mon.0) 2212 : cluster [WRN] Health check update: Degraded data redundancy: 22282/154476 objects degraded (14.424%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:32.963 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:31.728477+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2024-02-12T17:15:32.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:33.023 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:33.049 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:33.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:33.156 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:33.264 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:33.355 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:33.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:33.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:33.493 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:33.519 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:33.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:33.626 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:33.696 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:33.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:33.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:33.090817+0000 mgr.y (mgr.24481) 973 : cluster [DBG] pgmap v1389: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 576 op/s; 21615/150822 objects degraded (14.331%); 21038/150822 objects misplaced (13.949%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:33.801 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:33.842 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:33.873 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:33.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:33.977 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:15:34.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:33.090817+0000 mgr.y (mgr.24481) 973 : cluster [DBG] pgmap v1389: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 576 op/s; 21615/150822 objects degraded (14.331%); 21038/150822 objects misplaced (13.949%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:34.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:33.090817+0000 mgr.y (mgr.24481) 973 : cluster [DBG] pgmap v1389: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 576 op/s; 21615/150822 objects degraded (14.331%); 21038/150822 objects misplaced (13.949%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:34.483 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:15:34.483 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:15:34.483 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:15:34.483 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:15:34.483 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:15:34.744 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:15:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:34.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:15:34.857 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:34.928 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:35.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:15:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:15:35.703 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:35.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:35.813 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:35.846 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:35.880 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:35.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:35.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:36.019 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:36.050 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:36.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:36.156 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:15:36.169 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:35.056517+0000 mgr.y (mgr.24481) 974 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:36.169 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:35.091714+0000 mgr.y (mgr.24481) 975 : cluster [DBG] pgmap v1390: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 683 op/s; 20626/144564 objects degraded (14.268%); 19812/144564 objects misplaced (13.705%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:36.240 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:36.280 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:15:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:36.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:15:36.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:36.443 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:36.477 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:36.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:36.535 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:35.056517+0000 mgr.y (mgr.24481) 974 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:36.535 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:35.091714+0000 mgr.y (mgr.24481) 975 : cluster [DBG] pgmap v1390: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 683 op/s; 20626/144564 objects degraded (14.268%); 19812/144564 objects misplaced (13.705%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:36.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:35.056517+0000 mgr.y (mgr.24481) 974 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:36.535 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:35.091714+0000 mgr.y (mgr.24481) 975 : cluster [DBG] pgmap v1390: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 683 op/s; 20626/144564 objects degraded (14.268%); 19812/144564 objects misplaced (13.705%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:15:36.582 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:36.638 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:36.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:36.745 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:36.783 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:36.803 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:36.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:36.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:36.966 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:37.114 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:37.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:37.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:37.256 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:15:37.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:36.729427+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: Degraded data redundancy: 20626/144564 objects degraded (14.268%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:37.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:36.729427+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: Degraded data redundancy: 20626/144564 objects degraded (14.268%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:37.342 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:37.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:36.729427+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: Degraded data redundancy: 20626/144564 objects degraded (14.268%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:37.948 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T17:15:38.242 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:38.242 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:38.265 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:15:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:38.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:15:38.371 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:38.445 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:38.462 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:15:38.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:37.092283+0000 mgr.y (mgr.24481) 976 : cluster [DBG] pgmap v1391: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 718 op/s; 20011/140853 objects degraded (14.207%); 19640/140853 objects misplaced (13.944%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:15:38.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:37.092283+0000 mgr.y (mgr.24481) 976 : cluster [DBG] pgmap v1391: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 718 op/s; 20011/140853 objects degraded (14.207%); 19640/140853 objects misplaced (13.944%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:15:38.536 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:37.092283+0000 mgr.y (mgr.24481) 976 : cluster [DBG] pgmap v1391: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 718 op/s; 20011/140853 objects degraded (14.207%); 19640/140853 objects misplaced (13.944%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:15:38.542 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:15:38.596 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:15:38.672 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:15:38.719 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:15:38.768 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:15:38.814 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:15:38.883 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:15:38.893 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:15:38.893 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:15:38.894 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:15:38.894 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:15:38.894 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:15:38.894 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:15:38.894 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:15:38.894 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:15:39.195 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:38.405724+0000 mon.b (mon.1) 121 : audit [DBG] from='client.? v1:172.21.15.33:0/1239785736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:39.482 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:38.405724+0000 mon.b (mon.1) 121 : audit [DBG] from='client.? v1:172.21.15.33:0/1239785736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:39.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:38.405724+0000 mon.b (mon.1) 121 : audit [DBG] from='client.? v1:172.21.15.33:0/1239785736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:40.333 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:40.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:40.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:40.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:40.466 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:39.093007+0000 mgr.y (mgr.24481) 977 : cluster [DBG] pgmap v1392: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 655 op/s; 19062/134835 objects degraded (14.137%); 18405/134835 objects misplaced (13.650%); 2.6 MiB/s, 40 objects/s recovering 2024-02-12T17:15:40.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:39.506526+0000 osd.3 (osd.3) 240 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:40.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:39.506576+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:40.467 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:39.093007+0000 mgr.y (mgr.24481) 977 : cluster [DBG] pgmap v1392: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 655 op/s; 19062/134835 objects degraded (14.137%); 18405/134835 objects misplaced (13.650%); 2.6 MiB/s, 40 objects/s recovering 2024-02-12T17:15:40.467 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:39.506526+0000 osd.3 (osd.3) 240 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:40.467 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:39.506576+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:40.474 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:39.093007+0000 mgr.y (mgr.24481) 977 : cluster [DBG] pgmap v1392: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 655 op/s; 19062/134835 objects degraded (14.137%); 18405/134835 objects misplaced (13.650%); 2.6 MiB/s, 40 objects/s recovering 2024-02-12T17:15:40.474 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:39.506526+0000 osd.3 (osd.3) 240 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:40.474 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:39.506576+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:40.476 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:40.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:40.611 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:40.659 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:40.683 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:40.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:40.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:40.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:40.825 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:40.853 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:40.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:40.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:40.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:41.001 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:41.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:15:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:40] "GET /metrics HTTP/1.1" 200 176849 "" "Prometheus/2.33.4" 2024-02-12T17:15:41.031 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:41.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:41.141 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:41.210 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:41.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:41.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:15:41.385 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:41.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:40.204811+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2024-02-12T17:15:41.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:40.204811+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2024-02-12T17:15:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:40.204811+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2024-02-12T17:15:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:41.093566+0000 mgr.y (mgr.24481) 978 : cluster [DBG] pgmap v1394: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 570 op/s; 19062/134835 objects degraded (14.137%); 18405/134835 objects misplaced (13.650%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T17:15:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:41.734201+0000 mon.a (mon.0) 2216 : cluster [WRN] Health check update: Degraded data redundancy: 19062/134835 objects degraded (14.137%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:42.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:41.746369+0000 mon.a (mon.0) 2217 : cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2024-02-12T17:15:42.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:41.093566+0000 mgr.y (mgr.24481) 978 : cluster [DBG] pgmap v1394: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 570 op/s; 19062/134835 objects degraded (14.137%); 18405/134835 objects misplaced (13.650%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T17:15:42.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:41.734201+0000 mon.a (mon.0) 2216 : cluster [WRN] Health check update: Degraded data redundancy: 19062/134835 objects degraded (14.137%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:42.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:41.746369+0000 mon.a (mon.0) 2217 : cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2024-02-12T17:15:42.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:41.093566+0000 mgr.y (mgr.24481) 978 : cluster [DBG] pgmap v1394: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 570 op/s; 19062/134835 objects degraded (14.137%); 18405/134835 objects misplaced (13.650%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T17:15:42.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:41.734201+0000 mon.a (mon.0) 2216 : cluster [WRN] Health check update: Degraded data redundancy: 19062/134835 objects degraded (14.137%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:42.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:41.746369+0000 mon.a (mon.0) 2217 : cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2024-02-12T17:15:42.671 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:15:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:42.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:15:42.783 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:42.822 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:42.822 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:15:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:15:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:42.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:42.963 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:43.040 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:43.228 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:43.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:43.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:43.367 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:43.396 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:43.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:43.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:43.532 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:43.563 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:43.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:43.667 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:43.699 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:43.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:43.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:43.876 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:43.902 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:43.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:44.005 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:44.083 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:44.269 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:43.096581+0000 mgr.y (mgr.24481) 979 : cluster [DBG] pgmap v1396: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 557 op/s; 18411/131181 objects degraded (14.035%); 18169/131181 objects misplaced (13.850%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:15:44.504 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:44.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:43.096581+0000 mgr.y (mgr.24481) 979 : cluster [DBG] pgmap v1396: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 557 op/s; 18411/131181 objects degraded (14.035%); 18169/131181 objects misplaced (13.850%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:15:44.566 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:44.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:44.682 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:44.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:15:44.705 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:15:44.730 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:15:44.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:43.096581+0000 mgr.y (mgr.24481) 979 : cluster [DBG] pgmap v1396: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 557 op/s; 18411/131181 objects degraded (14.035%); 18169/131181 objects misplaced (13.850%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:15:44.760 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:45.313 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:15:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:44.685007+0000 mgr.y (mgr.24481) 980 : audit [DBG] from='client.36206 v1:172.21.15.33:0/3470068032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:15:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:44.935576+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.33:0/1523693488' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:15:45.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:44.936199+0000 mon.a (mon.0) 2218 : audit [INF] from='client.26086 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:15:45.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:15:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:15:45.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:44.685007+0000 mgr.y (mgr.24481) 980 : audit [DBG] from='client.36206 v1:172.21.15.33:0/3470068032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:15:45.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:44.935576+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.33:0/1523693488' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:15:45.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:44.936199+0000 mon.a (mon.0) 2218 : audit [INF] from='client.26086 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:15:45.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:44.685007+0000 mgr.y (mgr.24481) 980 : audit [DBG] from='client.36206 v1:172.21.15.33:0/3470068032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:15:45.781 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:44.935576+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.33:0/1523693488' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:15:45.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:44.936199+0000 mon.a (mon.0) 2218 : audit [INF] from='client.26086 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:15:46.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:45.072660+0000 mgr.y (mgr.24481) 981 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:46.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:45.098201+0000 mgr.y (mgr.24481) 982 : cluster [DBG] pgmap v1397: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 670 op/s; 17400/124791 objects degraded (13.943%); 16853/124791 objects misplaced (13.505%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:46.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:45.262906+0000 mon.a (mon.0) 2219 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:46.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:45.287865+0000 mon.a (mon.0) 2220 : audit [INF] from='client.26086 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:15:46.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:45.287901+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2024-02-12T17:15:46.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:45.072660+0000 mgr.y (mgr.24481) 981 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:46.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:45.098201+0000 mgr.y (mgr.24481) 982 : cluster [DBG] pgmap v1397: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 670 op/s; 17400/124791 objects degraded (13.943%); 16853/124791 objects misplaced (13.505%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:46.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:45.262906+0000 mon.a (mon.0) 2219 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:46.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:45.287865+0000 mon.a (mon.0) 2220 : audit [INF] from='client.26086 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:15:46.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:45.287901+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2024-02-12T17:15:46.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:45.072660+0000 mgr.y (mgr.24481) 981 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:46.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:45.098201+0000 mgr.y (mgr.24481) 982 : cluster [DBG] pgmap v1397: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 670 op/s; 17400/124791 objects degraded (13.943%); 16853/124791 objects misplaced (13.505%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:46.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:45.262906+0000 mon.a (mon.0) 2219 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:15:46.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:45.287865+0000 mon.a (mon.0) 2220 : audit [INF] from='client.26086 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:15:46.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:45.287901+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2024-02-12T17:15:47.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:46.277989+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:47.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:46.278041+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:47.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:46.307847+0000 mon.a (mon.0) 2222 : cluster [DBG] osdmap e694: 8 total, 8 up, 7 in 2024-02-12T17:15:47.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:46.739441+0000 mon.a (mon.0) 2223 : cluster [WRN] Health check update: Degraded data redundancy: 17400/124791 objects degraded (13.943%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:47.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:46.277989+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:47.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:46.278041+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:47.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:46.307847+0000 mon.a (mon.0) 2222 : cluster [DBG] osdmap e694: 8 total, 8 up, 7 in 2024-02-12T17:15:47.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:46.739441+0000 mon.a (mon.0) 2223 : cluster [WRN] Health check update: Degraded data redundancy: 17400/124791 objects degraded (13.943%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:46.277989+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:46.278041+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:46.307847+0000 mon.a (mon.0) 2222 : cluster [DBG] osdmap e694: 8 total, 8 up, 7 in 2024-02-12T17:15:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:46.739441+0000 mon.a (mon.0) 2223 : cluster [WRN] Health check update: Degraded data redundancy: 17400/124791 objects degraded (13.943%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:47.771 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:47.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:47.889 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:47.977 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:48.071 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:48.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:48.179 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:15:48.258 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:47.098961+0000 mgr.y (mgr.24481) 983 : cluster [DBG] pgmap v1400: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 765 op/s; 16789/121059 objects degraded (13.868%); 16634/121059 objects misplaced (13.740%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:15:48.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:47.098961+0000 mgr.y (mgr.24481) 983 : cluster [DBG] pgmap v1400: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 765 op/s; 16789/121059 objects degraded (13.868%); 16634/121059 objects misplaced (13.740%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:15:48.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:47.098961+0000 mgr.y (mgr.24481) 983 : cluster [DBG] pgmap v1400: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 765 op/s; 16789/121059 objects degraded (13.868%); 16634/121059 objects misplaced (13.740%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T17:15:49.470 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:15:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:49.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:15:49.504 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1396,"stamp":"2024-02-12T17:15:43.096190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2853050044,"num_objects":43727,"num_object_clones":0,"num_object_copies":131181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18411,"num_objects_misplaced":18169,"num_objects_unfound":0,"num_objects_dirty":43727,"num_whiteouts":0,"num_read":1859,"num_read_kb":1611,"num_write":208841,"num_write_kb":8044010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21290,"num_bytes_recovered":1339972384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92403,"ondisk_log_size":92403,"up":354,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":368,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14223576,"kb_used_data":6710880,"kb_used_omap":3,"kb_used_meta":7512444,"kb_avail":641992488,"statfs":{"total":671965249536,"available":657400307712,"internally_reserved":0,"allocated":6871941120,"data_stored":6862421482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":7692742717},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,9,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-292356096,"num_objects":-4461,"num_object_clones":0,"num_object_copies":-13383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2215,"num_objects_misplaced":-1643,"num_objects_unfound":0,"num_objects_dirty":-4461,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4461,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":23592960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005159"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":416,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.747922+0000","last_change":"2024-02-12T17:14:10.348720+0000","last_active":"2024-02-12T17:15:31.747922+0000","last_peered":"2024-02-12T17:15:31.747922+0000","last_clean":"2024-02-12T17:15:31.747922+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:15:31.747922+0000","last_undegraded":"2024-02-12T17:15:31.747922+0000","last_fullsized":"2024-02-12T17:15:31.747922+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":198,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.764388+0000","last_change":"2024-02-12T17:13:46.694822+0000","last_active":"2024-02-12T17:15:27.764388+0000","last_peered":"2024-02-12T17:15:27.764388+0000","last_clean":"2024-02-12T17:15:27.764388+0000","last_became_active":"2024-02-12T17:13:46.694448+0000","last_became_peered":"2024-02-12T17:13:46.694448+0000","last_unstale":"2024-02-12T17:15:27.764388+0000","last_undegraded":"2024-02-12T17:15:27.764388+0000","last_fullsized":"2024-02-12T17:15:27.764388+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:12:55.862345+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:12:55.862345+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":443,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.749381+0000","last_change":"2024-02-12T17:13:46.787428+0000","last_active":"2024-02-12T17:15:31.749381+0000","last_peered":"2024-02-12T17:15:31.749381+0000","last_clean":"2024-02-12T17:15:31.749381+0000","last_became_active":"2024-02-12T17:13:46.786996+0000","last_became_peered":"2024-02-12T17:13:46.786996+0000","last_unstale":"2024-02-12T17:15:31.749381+0000","last_undegraded":"2024-02-12T17:15:31.749381+0000","last_fullsized":"2024-02-12T17:15:31.749381+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:13:12.016227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:12.016227+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":471,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.766588+0000","last_change":"2024-02-12T17:14:09.685503+0000","last_active":"2024-02-12T17:15:31.766588+0000","last_peered":"2024-02-12T17:15:31.766588+0000","last_clean":"2024-02-12T17:15:31.766588+0000","last_became_active":"2024-02-12T17:14:08.761132+0000","last_became_peered":"2024-02-12T17:14:08.761132+0000","last_unstale":"2024-02-12T17:15:31.766588+0000","last_undegraded":"2024-02-12T17:15:31.766588+0000","last_fullsized":"2024-02-12T17:15:31.766588+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:54.052006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.052006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":486,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.750026+0000","last_change":"2024-02-12T17:13:46.753228+0000","last_active":"2024-02-12T17:15:31.750026+0000","last_peered":"2024-02-12T17:15:31.750026+0000","last_clean":"2024-02-12T17:15:31.750026+0000","last_became_active":"2024-02-12T17:13:46.752755+0000","last_became_peered":"2024-02-12T17:13:46.752755+0000","last_unstale":"2024-02-12T17:15:31.750026+0000","last_undegraded":"2024-02-12T17:15:31.750026+0000","last_fullsized":"2024-02-12T17:15:31.750026+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":1540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":150,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.783006+0000","last_change":"2024-02-12T17:13:46.607814+0000","last_active":"2024-02-12T17:15:27.783006+0000","last_peered":"2024-02-12T17:15:27.783006+0000","last_clean":"2024-02-12T17:15:27.783006+0000","last_became_active":"2024-02-12T17:13:46.606367+0000","last_became_peered":"2024-02-12T17:13:46.606367+0000","last_unstale":"2024-02-12T17:15:27.783006+0000","last_undegraded":"2024-02-12T17:15:27.783006+0000","last_fullsized":"2024-02-12T17:15:27.783006+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":225,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.764928+0000","last_change":"2024-02-12T17:13:46.567244+0000","last_active":"2024-02-12T17:15:27.764928+0000","last_peered":"2024-02-12T17:15:27.764928+0000","last_clean":"2024-02-12T17:15:27.764928+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:15:27.764928+0000","last_undegraded":"2024-02-12T17:15:27.764928+0000","last_fullsized":"2024-02-12T17:15:27.764928+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":60,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.778336+0000","last_change":"2024-02-12T17:14:08.993489+0000","last_active":"2024-02-12T17:15:27.778336+0000","last_peered":"2024-02-12T17:15:27.778336+0000","last_clean":"2024-02-12T17:15:27.778336+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:15:27.778336+0000","last_undegraded":"2024-02-12T17:15:27.778336+0000","last_fullsized":"2024-02-12T17:15:27.778336+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":413,"reported_epoch":689,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:27.762368+0000","last_change":"2024-02-12T17:14:09.723827+0000","last_active":"2024-02-12T17:15:27.762368+0000","last_peered":"2024-02-12T17:15:27.762368+0000","last_clean":"2024-02-12T17:14:05.085957+0000","last_became_active":"2024-02-12T17:14:09.723480+0000","last_became_peered":"2024-02-12T17:14:09.723480+0000","last_unstale":"2024-02-12T17:15:27.762368+0000","last_undegraded":"2024-02-12T17:15:27.762368+0000","last_fullsized":"2024-02-12T17:15:27.762368+0000","mapping_epoch":652,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":384,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.749464+0000","last_change":"2024-02-12T17:14:08.755098+0000","last_active":"2024-02-12T17:15:31.749464+0000","last_peered":"2024-02-12T17:15:31.749464+0000","last_clean":"2024-02-12T17:15:31.749464+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:15:31.749464+0000","last_undegraded":"2024-02-12T17:15:31.749464+0000","last_fullsized":"2024-02-12T17:15:31.749464+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":407,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.748222+0000","last_change":"2024-02-12T17:12:12.141297+0000","last_active":"2024-02-12T17:15:31.748222+0000","last_peered":"2024-02-12T17:15:31.748222+0000","last_clean":"2024-02-12T17:15:31.748222+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:15:31.748222+0000","last_undegraded":"2024-02-12T17:15:31.748222+0000","last_fullsized":"2024-02-12T17:15:31.748222+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.141204+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":235,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.765733+0000","last_change":"2024-02-12T17:14:08.743209+0000","last_active":"2024-02-12T17:15:27.765733+0000","last_peered":"2024-02-12T17:15:27.765733+0000","last_clean":"2024-02-12T17:15:27.765733+0000","last_became_active":"2024-02-12T17:14:08.742886+0000","last_became_peered":"2024-02-12T17:14:08.742886+0000","last_unstale":"2024-02-12T17:15:27.765733+0000","last_undegraded":"2024-02-12T17:15:27.765733+0000","last_fullsized":"2024-02-12T17:15:27.765733+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.783527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.783527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":684,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762347+0000","last_change":"2024-02-12T17:13:21.591627+0000","last_active":"2024-02-12T17:15:27.762347+0000","last_peered":"2024-02-12T17:15:27.762347+0000","last_clean":"2024-02-12T17:15:27.762347+0000","last_became_active":"2024-02-12T17:13:21.591246+0000","last_became_peered":"2024-02-12T17:13:21.591246+0000","last_unstale":"2024-02-12T17:15:27.762347+0000","last_undegraded":"2024-02-12T17:15:27.762347+0000","last_fullsized":"2024-02-12T17:15:27.762347+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.028495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":429,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.754540+0000","last_change":"2024-02-12T17:13:46.558817+0000","last_active":"2024-02-12T17:15:31.754540+0000","last_peered":"2024-02-12T17:15:31.754540+0000","last_clean":"2024-02-12T17:15:31.754540+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:15:31.754540+0000","last_undegraded":"2024-02-12T17:15:31.754540+0000","last_fullsized":"2024-02-12T17:15:31.754540+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":82,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.779646+0000","last_change":"2024-02-12T17:13:46.576676+0000","last_active":"2024-02-12T17:15:27.779646+0000","last_peered":"2024-02-12T17:15:27.779646+0000","last_clean":"2024-02-12T17:15:27.779646+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:15:27.779646+0000","last_undegraded":"2024-02-12T17:15:27.779646+0000","last_fullsized":"2024-02-12T17:15:27.779646+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":366,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.748065+0000","last_change":"2024-02-12T17:14:08.763317+0000","last_active":"2024-02-12T17:15:31.748065+0000","last_peered":"2024-02-12T17:15:31.748065+0000","last_clean":"2024-02-12T17:15:31.748065+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:15:31.748065+0000","last_undegraded":"2024-02-12T17:15:31.748065+0000","last_fullsized":"2024-02-12T17:15:31.748065+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":122,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.782558+0000","last_change":"2024-02-12T17:14:08.742170+0000","last_active":"2024-02-12T17:15:27.782558+0000","last_peered":"2024-02-12T17:15:27.782558+0000","last_clean":"2024-02-12T17:15:27.782558+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:15:27.782558+0000","last_undegraded":"2024-02-12T17:15:27.782558+0000","last_fullsized":"2024-02-12T17:15:27.782558+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":203,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.764888+0000","last_change":"2024-02-12T17:13:46.784204+0000","last_active":"2024-02-12T17:15:27.764888+0000","last_peered":"2024-02-12T17:15:27.764888+0000","last_clean":"2024-02-12T17:15:27.764888+0000","last_became_active":"2024-02-12T17:13:46.783737+0000","last_became_peered":"2024-02-12T17:13:46.783737+0000","last_unstale":"2024-02-12T17:15:27.764888+0000","last_undegraded":"2024-02-12T17:15:27.764888+0000","last_fullsized":"2024-02-12T17:15:27.764888+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:15.817751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.817751+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":753,"reported_epoch":690,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:31.748586+0000","last_change":"2024-02-12T17:14:40.177468+0000","last_active":"2024-02-12T17:15:31.748586+0000","last_peered":"2024-02-12T17:15:31.748586+0000","last_clean":"2024-02-12T17:13:44.507959+0000","last_became_active":"2024-02-12T17:14:08.758564+0000","last_became_peered":"2024-02-12T17:14:08.758564+0000","last_unstale":"2024-02-12T17:15:31.748586+0000","last_undegraded":"2024-02-12T17:15:31.748586+0000","last_fullsized":"2024-02-12T17:15:31.748586+0000","mapping_epoch":651,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":48,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.776632+0000","last_change":"2024-02-12T17:14:08.762208+0000","last_active":"2024-02-12T17:15:27.776632+0000","last_peered":"2024-02-12T17:15:27.776632+0000","last_clean":"2024-02-12T17:15:27.776632+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:15:27.776632+0000","last_undegraded":"2024-02-12T17:15:27.776632+0000","last_fullsized":"2024-02-12T17:15:27.776632+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":193,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.766774+0000","last_change":"2024-02-12T17:12:02.752352+0000","last_active":"2024-02-12T17:15:27.766774+0000","last_peered":"2024-02-12T17:15:27.766774+0000","last_clean":"2024-02-12T17:15:27.766774+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:15:27.766774+0000","last_undegraded":"2024-02-12T17:15:27.766774+0000","last_fullsized":"2024-02-12T17:15:27.766774+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:02.752280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:02.752280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":154,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.779519+0000","last_change":"2024-02-12T17:13:46.582722+0000","last_active":"2024-02-12T17:15:27.779519+0000","last_peered":"2024-02-12T17:15:27.779519+0000","last_clean":"2024-02-12T17:15:27.779519+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:15:27.779519+0000","last_undegraded":"2024-02-12T17:15:27.779519+0000","last_fullsized":"2024-02-12T17:15:27.779519+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":781,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:37.656072+0000","last_change":"2024-02-12T17:12:58.020311+0000","last_active":"2024-02-12T17:15:37.656072+0000","last_peered":"2024-02-12T17:15:37.656072+0000","last_clean":"2024-02-12T17:15:37.656072+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:15:37.656072+0000","last_undegraded":"2024-02-12T17:15:37.656072+0000","last_fullsized":"2024-02-12T17:15:37.656072+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:12:58.018712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.018712+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":48,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.780142+0000","last_change":"2024-02-12T17:14:08.872636+0000","last_active":"2024-02-12T17:15:27.780142+0000","last_peered":"2024-02-12T17:15:27.780142+0000","last_clean":"2024-02-12T17:15:27.780142+0000","last_became_active":"2024-02-12T17:14:08.872123+0000","last_became_peered":"2024-02-12T17:14:08.872123+0000","last_unstale":"2024-02-12T17:15:27.780142+0000","last_undegraded":"2024-02-12T17:15:27.780142+0000","last_fullsized":"2024-02-12T17:15:27.780142+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:06.904354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:06.904354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":780,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768693+0000","last_change":"2024-02-12T17:12:12.838640+0000","last_active":"2024-02-12T17:15:27.768693+0000","last_peered":"2024-02-12T17:15:27.768693+0000","last_clean":"2024-02-12T17:15:27.768693+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:15:27.768693+0000","last_undegraded":"2024-02-12T17:15:27.768693+0000","last_fullsized":"2024-02-12T17:15:27.768693+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:12.838563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.838563+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"655'24","reported_seq":777,"reported_epoch":690,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:32.657231+0000","last_change":"2024-02-12T17:14:09.729342+0000","last_active":"2024-02-12T17:15:32.657231+0000","last_peered":"2024-02-12T17:15:32.657231+0000","last_clean":"2024-02-12T17:14:07.616779+0000","last_became_active":"2024-02-12T17:14:09.728969+0000","last_became_peered":"2024-02-12T17:14:09.728969+0000","last_unstale":"2024-02-12T17:15:32.657231+0000","last_undegraded":"2024-02-12T17:15:32.657231+0000","last_fullsized":"2024-02-12T17:15:32.657231+0000","mapping_epoch":652,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":684,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768680+0000","last_change":"2024-02-12T17:12:54.656523+0000","last_active":"2024-02-12T17:15:27.768680+0000","last_peered":"2024-02-12T17:15:27.768680+0000","last_clean":"2024-02-12T17:15:27.768680+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:15:27.768680+0000","last_undegraded":"2024-02-12T17:15:27.768680+0000","last_fullsized":"2024-02-12T17:15:27.768680+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:54.656456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.656456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":420,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.749572+0000","last_change":"2024-02-12T17:13:46.577769+0000","last_active":"2024-02-12T17:15:31.749572+0000","last_peered":"2024-02-12T17:15:31.749572+0000","last_clean":"2024-02-12T17:15:31.749572+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:15:31.749572+0000","last_undegraded":"2024-02-12T17:15:31.749572+0000","last_fullsized":"2024-02-12T17:15:31.749572+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":706,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762639+0000","last_change":"2024-02-12T17:13:21.591810+0000","last_active":"2024-02-12T17:15:27.762639+0000","last_peered":"2024-02-12T17:15:27.762639+0000","last_clean":"2024-02-12T17:15:27.762639+0000","last_became_active":"2024-02-12T17:13:21.591502+0000","last_became_peered":"2024-02-12T17:13:21.591502+0000","last_unstale":"2024-02-12T17:15:27.762639+0000","last_undegraded":"2024-02-12T17:15:27.762639+0000","last_fullsized":"2024-02-12T17:15:27.762639+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.882066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.882066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":227,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.765615+0000","last_change":"2024-02-12T17:13:46.651066+0000","last_active":"2024-02-12T17:15:27.765615+0000","last_peered":"2024-02-12T17:15:27.765615+0000","last_clean":"2024-02-12T17:15:27.765615+0000","last_became_active":"2024-02-12T17:13:46.650776+0000","last_became_peered":"2024-02-12T17:13:46.650776+0000","last_unstale":"2024-02-12T17:15:27.765615+0000","last_undegraded":"2024-02-12T17:15:27.765615+0000","last_fullsized":"2024-02-12T17:15:27.765615+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:50.786900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.786900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":409,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.751028+0000","last_change":"2024-02-12T17:13:46.822111+0000","last_active":"2024-02-12T17:15:31.751028+0000","last_peered":"2024-02-12T17:15:31.751028+0000","last_clean":"2024-02-12T17:15:31.751028+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:15:31.751028+0000","last_undegraded":"2024-02-12T17:15:31.751028+0000","last_fullsized":"2024-02-12T17:15:31.751028+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":318,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.766262+0000","last_change":"2024-02-12T17:14:08.745944+0000","last_active":"2024-02-12T17:15:27.766262+0000","last_peered":"2024-02-12T17:15:27.766262+0000","last_clean":"2024-02-12T17:15:27.766262+0000","last_became_active":"2024-02-12T17:14:08.745729+0000","last_became_peered":"2024-02-12T17:14:08.745729+0000","last_unstale":"2024-02-12T17:15:27.766262+0000","last_undegraded":"2024-02-12T17:15:27.766262+0000","last_fullsized":"2024-02-12T17:15:27.766262+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:49.809844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:12:49.809844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":82,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.780313+0000","last_change":"2024-02-12T17:13:46.559519+0000","last_active":"2024-02-12T17:15:27.780313+0000","last_peered":"2024-02-12T17:15:27.780313+0000","last_clean":"2024-02-12T17:15:27.780313+0000","last_became_active":"2024-02-12T17:13:46.559095+0000","last_became_peered":"2024-02-12T17:13:46.559095+0000","last_unstale":"2024-02-12T17:15:27.780313+0000","last_undegraded":"2024-02-12T17:15:27.780313+0000","last_fullsized":"2024-02-12T17:15:27.780313+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":58,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.778424+0000","last_change":"2024-02-12T17:14:08.993503+0000","last_active":"2024-02-12T17:15:27.778424+0000","last_peered":"2024-02-12T17:15:27.778424+0000","last_clean":"2024-02-12T17:15:27.778424+0000","last_became_active":"2024-02-12T17:14:08.861886+0000","last_became_peered":"2024-02-12T17:14:08.861886+0000","last_unstale":"2024-02-12T17:15:27.778424+0000","last_undegraded":"2024-02-12T17:15:27.778424+0000","last_fullsized":"2024-02-12T17:15:27.778424+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"690'13066","reported_seq":20028,"reported_epoch":690,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-12T17:15:38.665927+0000","last_change":"2024-02-12T17:15:09.885129+0000","last_active":"2024-02-12T17:15:38.665927+0000","last_peered":"2024-02-12T17:15:38.665927+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.966441+0000","last_became_peered":"2024-02-12T17:14:08.966441+0000","last_unstale":"2024-02-12T17:15:38.665927+0000","last_undegraded":"2024-02-12T17:14:08.734833+0000","last_fullsized":"2024-02-12T17:14:08.712653+0000","mapping_epoch":651,"log_start":"616'6021","ondisk_log_start":"616'6021","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":7045,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":180486144,"num_objects":2754,"num_object_clones":0,"num_object_copies":8262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2754,"num_objects_misplaced":2754,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13066,"num_write_kb":506240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3413,"num_bytes_recovered":223674368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[2,4],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4","objects":2754}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":408,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.749280+0000","last_change":"2024-02-12T17:13:46.787305+0000","last_active":"2024-02-12T17:15:31.749280+0000","last_peered":"2024-02-12T17:15:31.749280+0000","last_clean":"2024-02-12T17:15:31.749280+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:15:31.749280+0000","last_undegraded":"2024-02-12T17:15:31.749280+0000","last_fullsized":"2024-02-12T17:15:31.749280+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":406,"reported_epoch":689,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:27.768789+0000","last_change":"2024-02-12T17:14:09.723027+0000","last_active":"2024-02-12T17:15:27.768789+0000","last_peered":"2024-02-12T17:15:27.768789+0000","last_clean":"2024-02-12T17:11:53.687795+0000","last_became_active":"2024-02-12T17:14:09.722681+0000","last_became_peered":"2024-02-12T17:14:09.722681+0000","last_unstale":"2024-02-12T17:15:27.768789+0000","last_undegraded":"2024-02-12T17:15:27.768789+0000","last_fullsized":"2024-02-12T17:15:27.768789+0000","mapping_epoch":652,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:12:09.882247+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:12:09.882247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[5,7,6],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":156,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.786844+0000","last_change":"2024-02-12T17:13:46.744739+0000","last_active":"2024-02-12T17:15:27.786844+0000","last_peered":"2024-02-12T17:15:27.786844+0000","last_clean":"2024-02-12T17:15:27.786844+0000","last_became_active":"2024-02-12T17:13:46.744003+0000","last_became_peered":"2024-02-12T17:13:46.744003+0000","last_unstale":"2024-02-12T17:15:27.786844+0000","last_undegraded":"2024-02-12T17:15:27.786844+0000","last_fullsized":"2024-02-12T17:15:27.786844+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:51.126056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:51.126056+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":48,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.776513+0000","last_change":"2024-02-12T17:14:08.769824+0000","last_active":"2024-02-12T17:15:27.776513+0000","last_peered":"2024-02-12T17:15:27.776513+0000","last_clean":"2024-02-12T17:15:27.776513+0000","last_became_active":"2024-02-12T17:14:08.769221+0000","last_became_peered":"2024-02-12T17:14:08.769221+0000","last_unstale":"2024-02-12T17:15:27.776513+0000","last_undegraded":"2024-02-12T17:15:27.776513+0000","last_fullsized":"2024-02-12T17:15:27.776513+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"690'12805","reported_seq":12939,"reported_epoch":690,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:15:38.689341+0000","last_change":"2024-02-12T17:14:16.786942+0000","last_active":"2024-02-12T17:15:38.689341+0000","last_peered":"2024-02-12T17:15:38.689341+0000","last_clean":"2024-02-12T17:13:45.521351+0000","last_became_active":"2024-02-12T17:13:46.743970+0000","last_became_peered":"2024-02-12T17:13:46.743970+0000","last_unstale":"2024-02-12T17:15:38.689341+0000","last_undegraded":"2024-02-12T17:15:38.689341+0000","last_fullsized":"2024-02-12T17:15:38.689341+0000","mapping_epoch":630,"log_start":"623'6768","ondisk_log_start":"623'6768","created":597,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'5769","last_scrub_stamp":"2024-02-12T17:13:06.847792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:13:06.847792+0000","log_size":6037,"ondisk_log_size":6037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":181272600,"num_objects":2767,"num_object_clones":0,"num_object_copies":8301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2767,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12805,"num_write_kb":498241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":2767}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":695,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.777672+0000","last_change":"2024-02-12T17:14:07.897458+0000","last_active":"2024-02-12T17:15:27.777672+0000","last_peered":"2024-02-12T17:15:27.777672+0000","last_clean":"2024-02-12T17:15:27.777672+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:15:27.777672+0000","last_undegraded":"2024-02-12T17:15:27.777672+0000","last_fullsized":"2024-02-12T17:15:27.777672+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.780381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.780381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":82,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.783153+0000","last_change":"2024-02-12T17:13:46.791505+0000","last_active":"2024-02-12T17:15:27.783153+0000","last_peered":"2024-02-12T17:15:27.783153+0000","last_clean":"2024-02-12T17:15:27.783153+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:15:27.783153+0000","last_undegraded":"2024-02-12T17:15:27.783153+0000","last_fullsized":"2024-02-12T17:15:27.783153+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":162,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.779722+0000","last_change":"2024-02-12T17:14:09.246132+0000","last_active":"2024-02-12T17:15:27.779722+0000","last_peered":"2024-02-12T17:15:27.779722+0000","last_clean":"2024-02-12T17:15:27.779722+0000","last_became_active":"2024-02-12T17:14:08.752149+0000","last_became_peered":"2024-02-12T17:14:08.752149+0000","last_unstale":"2024-02-12T17:15:27.779722+0000","last_undegraded":"2024-02-12T17:15:27.779722+0000","last_fullsized":"2024-02-12T17:15:27.779722+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:13:01.543047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.543047+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"655'23","reported_seq":507,"reported_epoch":690,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:37.656353+0000","last_change":"2024-02-12T17:14:09.749094+0000","last_active":"2024-02-12T17:15:37.656353+0000","last_peered":"2024-02-12T17:15:37.656353+0000","last_clean":"2024-02-12T17:03:47.420086+0000","last_became_active":"2024-02-12T17:14:09.748790+0000","last_became_peered":"2024-02-12T17:14:09.748790+0000","last_unstale":"2024-02-12T17:15:37.656353+0000","last_undegraded":"2024-02-12T17:15:37.656353+0000","last_fullsized":"2024-02-12T17:15:37.656353+0000","mapping_epoch":652,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'21","last_scrub_stamp":"2024-02-12T17:12:57.741798+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:12:57.741798+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[3,6,5],"avail_no_missing":["3","0","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"691'13099","reported_seq":15726,"reported_epoch":691,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:15:41.651064+0000","last_change":"2024-02-12T17:14:31.380762+0000","last_active":"2024-02-12T17:15:41.651064+0000","last_peered":"2024-02-12T17:15:41.651064+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.877848+0000","last_became_peered":"2024-02-12T17:14:08.877848+0000","last_unstale":"2024-02-12T17:15:41.651064+0000","last_undegraded":"2024-02-12T17:14:09.686965+0000","last_fullsized":"2024-02-12T17:14:08.709420+0000","mapping_epoch":651,"log_start":"634'8163","ondisk_log_start":"634'8163","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4936,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":171769856,"num_objects":2621,"num_object_clones":0,"num_object_copies":7863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2175,"num_objects_misplaced":2175,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13099,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":886,"num_bytes_recovered":57933824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4","objects":2621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":411,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.750571+0000","last_change":"2024-02-12T17:13:21.603543+0000","last_active":"2024-02-12T17:15:31.750571+0000","last_peered":"2024-02-12T17:15:31.750571+0000","last_clean":"2024-02-12T17:15:31.750571+0000","last_became_active":"2024-02-12T17:13:21.603226+0000","last_became_peered":"2024-02-12T17:13:21.603226+0000","last_unstale":"2024-02-12T17:15:31.750571+0000","last_undegraded":"2024-02-12T17:15:31.750571+0000","last_fullsized":"2024-02-12T17:15:31.750571+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.262134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.262134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":726,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768755+0000","last_change":"2024-02-12T17:13:46.566957+0000","last_active":"2024-02-12T17:15:27.768755+0000","last_peered":"2024-02-12T17:15:27.768755+0000","last_clean":"2024-02-12T17:15:27.768755+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:15:27.768755+0000","last_undegraded":"2024-02-12T17:15:27.768755+0000","last_fullsized":"2024-02-12T17:15:27.768755+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":199,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:32.663855+0000","last_change":"2024-02-12T17:13:46.607171+0000","last_active":"2024-02-12T17:15:32.663855+0000","last_peered":"2024-02-12T17:15:32.663855+0000","last_clean":"2024-02-12T17:15:32.663855+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:15:32.663855+0000","last_undegraded":"2024-02-12T17:15:32.663855+0000","last_fullsized":"2024-02-12T17:15:32.663855+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":641,"reported_epoch":689,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:27.782318+0000","last_change":"2024-02-12T17:14:09.723924+0000","last_active":"2024-02-12T17:15:27.782318+0000","last_peered":"2024-02-12T17:15:27.782318+0000","last_clean":"2024-02-12T17:14:05.074559+0000","last_became_active":"2024-02-12T17:14:09.723600+0000","last_became_peered":"2024-02-12T17:14:09.723600+0000","last_unstale":"2024-02-12T17:15:27.782318+0000","last_undegraded":"2024-02-12T17:15:27.782318+0000","last_fullsized":"2024-02-12T17:15:27.782318+0000","mapping_epoch":652,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:13:02.852563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:13:02.852563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[1,6,7],"avail_no_missing":["1","0","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"691'13322","reported_seq":13424,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T17:15:41.626821+0000","last_change":"2024-02-12T17:11:59.008876+0000","last_active":"2024-02-12T17:15:41.626821+0000","last_peered":"2024-02-12T17:15:41.626821+0000","last_clean":"2024-02-12T17:15:41.626821+0000","last_became_active":"2024-02-12T17:11:59.008587+0000","last_became_peered":"2024-02-12T17:11:59.008587+0000","last_unstale":"2024-02-12T17:15:41.626821+0000","last_undegraded":"2024-02-12T17:15:41.626821+0000","last_fullsized":"2024-02-12T17:15:41.626821+0000","mapping_epoch":597,"log_start":"634'8318","ondisk_log_start":"634'8318","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5004,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":173801472,"num_objects":2652,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13322,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":695,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762760+0000","last_change":"2024-02-12T17:13:00.766698+0000","last_active":"2024-02-12T17:15:27.762760+0000","last_peered":"2024-02-12T17:15:27.762760+0000","last_clean":"2024-02-12T17:15:27.762760+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:15:27.762760+0000","last_undegraded":"2024-02-12T17:15:27.762760+0000","last_fullsized":"2024-02-12T17:15:27.762760+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.766590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":303,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:38.037577+0000","last_change":"2024-02-12T17:13:46.584941+0000","last_active":"2024-02-12T17:15:38.037577+0000","last_peered":"2024-02-12T17:15:38.037577+0000","last_clean":"2024-02-12T17:15:38.037577+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:15:38.037577+0000","last_undegraded":"2024-02-12T17:15:38.037577+0000","last_fullsized":"2024-02-12T17:15:38.037577+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":325,"num_read_kb":320,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":357,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.748297+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:15:31.748297+0000","last_peered":"2024-02-12T17:15:31.748297+0000","last_clean":"2024-02-12T17:15:31.748297+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:15:31.748297+0000","last_undegraded":"2024-02-12T17:15:31.748297+0000","last_fullsized":"2024-02-12T17:15:31.748297+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":155,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.783485+0000","last_change":"2024-02-12T17:14:09.321979+0000","last_active":"2024-02-12T17:15:27.783485+0000","last_peered":"2024-02-12T17:15:27.783485+0000","last_clean":"2024-02-12T17:15:27.783485+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:15:27.783485+0000","last_undegraded":"2024-02-12T17:15:27.783485+0000","last_fullsized":"2024-02-12T17:15:27.783485+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":619,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768663+0000","last_change":"2024-02-12T17:13:10.412138+0000","last_active":"2024-02-12T17:15:27.768663+0000","last_peered":"2024-02-12T17:15:27.768663+0000","last_clean":"2024-02-12T17:15:27.768663+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:15:27.768663+0000","last_undegraded":"2024-02-12T17:15:27.768663+0000","last_fullsized":"2024-02-12T17:15:27.768663+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:13:10.412031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"691'12930","reported_seq":13032,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T17:15:41.634600+0000","last_change":"2024-02-12T17:11:59.007548+0000","last_active":"2024-02-12T17:15:41.634600+0000","last_peered":"2024-02-12T17:15:41.634600+0000","last_clean":"2024-02-12T17:15:41.634600+0000","last_became_active":"2024-02-12T17:11:59.007214+0000","last_became_peered":"2024-02-12T17:11:59.007214+0000","last_unstale":"2024-02-12T17:15:41.634600+0000","last_undegraded":"2024-02-12T17:15:41.634600+0000","last_fullsized":"2024-02-12T17:15:41.634600+0000","mapping_epoch":597,"log_start":"630'7955","ondisk_log_start":"630'7955","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4975,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":170917888,"num_objects":2608,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12930,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":758,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.769078+0000","last_change":"2024-02-12T17:13:46.826853+0000","last_active":"2024-02-12T17:15:27.769078+0000","last_peered":"2024-02-12T17:15:27.769078+0000","last_clean":"2024-02-12T17:15:27.769078+0000","last_became_active":"2024-02-12T17:13:46.821418+0000","last_became_peered":"2024-02-12T17:13:46.821418+0000","last_unstale":"2024-02-12T17:15:27.769078+0000","last_undegraded":"2024-02-12T17:15:27.769078+0000","last_fullsized":"2024-02-12T17:15:27.769078+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.951964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.951964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":389,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.766349+0000","last_change":"2024-02-12T17:12:59.006372+0000","last_active":"2024-02-12T17:15:31.766349+0000","last_peered":"2024-02-12T17:15:31.766349+0000","last_clean":"2024-02-12T17:15:31.766349+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:15:31.766349+0000","last_undegraded":"2024-02-12T17:15:31.766349+0000","last_fullsized":"2024-02-12T17:15:31.766349+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.006260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.006260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":666,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768447+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:15:27.768447+0000","last_peered":"2024-02-12T17:15:27.768447+0000","last_clean":"2024-02-12T17:15:27.768447+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:15:27.768447+0000","last_undegraded":"2024-02-12T17:15:27.768447+0000","last_fullsized":"2024-02-12T17:15:27.768447+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":735,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762195+0000","last_change":"2024-02-12T17:14:08.743668+0000","last_active":"2024-02-12T17:15:27.762195+0000","last_peered":"2024-02-12T17:15:27.762195+0000","last_clean":"2024-02-12T17:15:27.762195+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:15:27.762195+0000","last_undegraded":"2024-02-12T17:15:27.762195+0000","last_fullsized":"2024-02-12T17:15:27.762195+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":300,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.765327+0000","last_change":"2024-02-12T17:14:08.781462+0000","last_active":"2024-02-12T17:15:27.765327+0000","last_peered":"2024-02-12T17:15:27.765327+0000","last_clean":"2024-02-12T17:15:27.765327+0000","last_became_active":"2024-02-12T17:14:08.726463+0000","last_became_peered":"2024-02-12T17:14:08.726463+0000","last_unstale":"2024-02-12T17:15:27.765327+0000","last_undegraded":"2024-02-12T17:15:27.765327+0000","last_fullsized":"2024-02-12T17:15:27.765327+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:47.763264+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.763264+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":697,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.766273+0000","last_change":"2024-02-12T17:12:03.922963+0000","last_active":"2024-02-12T17:15:27.766273+0000","last_peered":"2024-02-12T17:15:27.766273+0000","last_clean":"2024-02-12T17:15:27.766273+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:15:27.766273+0000","last_undegraded":"2024-02-12T17:15:27.766273+0000","last_fullsized":"2024-02-12T17:15:27.766273+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:12:03.922893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.922893+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"691'13106","reported_seq":14992,"reported_epoch":691,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:15:41.625239+0000","last_change":"2024-02-12T17:14:10.177438+0000","last_active":"2024-02-12T17:15:41.625239+0000","last_peered":"2024-02-12T17:15:41.625239+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.873608+0000","last_became_peered":"2024-02-12T17:14:08.873608+0000","last_unstale":"2024-02-12T17:15:41.625239+0000","last_undegraded":"2024-02-12T17:14:08.744781+0000","last_fullsized":"2024-02-12T17:14:08.704327+0000","mapping_epoch":651,"log_start":"633'8142","ondisk_log_start":"633'8142","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4964,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":7926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13106,"num_write_kb":503936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":884,"num_bytes_recovered":57933824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":82,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.783234+0000","last_change":"2024-02-12T17:13:46.608049+0000","last_active":"2024-02-12T17:15:27.783234+0000","last_peered":"2024-02-12T17:15:27.783234+0000","last_clean":"2024-02-12T17:15:27.783234+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:15:27.783234+0000","last_undegraded":"2024-02-12T17:15:27.783234+0000","last_fullsized":"2024-02-12T17:15:27.783234+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":48,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.780296+0000","last_change":"2024-02-12T17:14:08.859925+0000","last_active":"2024-02-12T17:15:27.780296+0000","last_peered":"2024-02-12T17:15:27.780296+0000","last_clean":"2024-02-12T17:15:27.780296+0000","last_became_active":"2024-02-12T17:14:08.859396+0000","last_became_peered":"2024-02-12T17:14:08.859396+0000","last_unstale":"2024-02-12T17:15:27.780296+0000","last_undegraded":"2024-02-12T17:15:27.780296+0000","last_fullsized":"2024-02-12T17:15:27.780296+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:03.904976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.904976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":735,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762105+0000","last_change":"2024-02-12T17:14:05.666060+0000","last_active":"2024-02-12T17:15:27.762105+0000","last_peered":"2024-02-12T17:15:27.762105+0000","last_clean":"2024-02-12T17:15:27.762105+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:15:27.762105+0000","last_undegraded":"2024-02-12T17:15:27.762105+0000","last_fullsized":"2024-02-12T17:15:27.762105+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":62,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.778498+0000","last_change":"2024-02-12T17:14:09.046726+0000","last_active":"2024-02-12T17:15:27.778498+0000","last_peered":"2024-02-12T17:15:27.778498+0000","last_clean":"2024-02-12T17:15:27.778498+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:15:27.778498+0000","last_undegraded":"2024-02-12T17:15:27.778498+0000","last_fullsized":"2024-02-12T17:15:27.778498+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":738,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.769052+0000","last_change":"2024-02-12T17:13:46.827015+0000","last_active":"2024-02-12T17:15:27.769052+0000","last_peered":"2024-02-12T17:15:27.769052+0000","last_clean":"2024-02-12T17:15:27.769052+0000","last_became_active":"2024-02-12T17:13:46.821796+0000","last_became_peered":"2024-02-12T17:13:46.821796+0000","last_unstale":"2024-02-12T17:15:27.769052+0000","last_undegraded":"2024-02-12T17:15:27.769052+0000","last_fullsized":"2024-02-12T17:15:27.769052+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.698727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.698727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"690'12818","reported_seq":18907,"reported_epoch":690,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:15:38.688154+0000","last_change":"2024-02-12T17:14:46.835208+0000","last_active":"2024-02-12T17:15:38.688154+0000","last_peered":"2024-02-12T17:15:38.688154+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.806885+0000","last_became_peered":"2024-02-12T17:13:46.806885+0000","last_unstale":"2024-02-12T17:15:38.688154+0000","last_undegraded":"2024-02-12T17:13:46.572970+0000","last_fullsized":"2024-02-12T17:13:46.531119+0000","mapping_epoch":630,"log_start":"616'5774","ondisk_log_start":"616'5774","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":7044,"ondisk_log_size":7044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179437568,"num_objects":2738,"num_object_clones":0,"num_object_copies":8214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1583,"num_objects_misplaced":2738,"num_objects_unfound":0,"num_objects_dirty":2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12818,"num_write_kb":497792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2984,"num_bytes_recovered":195559424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":2738}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":113,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.786768+0000","last_change":"2024-02-12T17:13:46.579742+0000","last_active":"2024-02-12T17:15:27.786768+0000","last_peered":"2024-02-12T17:15:27.786768+0000","last_clean":"2024-02-12T17:15:27.786768+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:15:27.786768+0000","last_undegraded":"2024-02-12T17:15:27.786768+0000","last_fullsized":"2024-02-12T17:15:27.786768+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":749,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762785+0000","last_change":"2024-02-12T17:14:08.746702+0000","last_active":"2024-02-12T17:15:27.762785+0000","last_peered":"2024-02-12T17:15:27.762785+0000","last_clean":"2024-02-12T17:15:27.762785+0000","last_became_active":"2024-02-12T17:14:08.746395+0000","last_became_peered":"2024-02-12T17:14:08.746395+0000","last_unstale":"2024-02-12T17:15:27.762785+0000","last_undegraded":"2024-02-12T17:15:27.762785+0000","last_fullsized":"2024-02-12T17:15:27.762785+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:45.764955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:45.764955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":66,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.780000+0000","last_change":"2024-02-12T17:14:11.148689+0000","last_active":"2024-02-12T17:15:27.780000+0000","last_peered":"2024-02-12T17:15:27.780000+0000","last_clean":"2024-02-12T17:15:27.780000+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:15:27.780000+0000","last_undegraded":"2024-02-12T17:15:27.780000+0000","last_fullsized":"2024-02-12T17:15:27.780000+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":191,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.766592+0000","last_change":"2024-02-12T17:12:53.746937+0000","last_active":"2024-02-12T17:15:27.766592+0000","last_peered":"2024-02-12T17:15:27.766592+0000","last_clean":"2024-02-12T17:15:27.766592+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:15:27.766592+0000","last_undegraded":"2024-02-12T17:15:27.766592+0000","last_fullsized":"2024-02-12T17:15:27.766592+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.746858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.746858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"690'12894","reported_seq":13009,"reported_epoch":690,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:15:38.973826+0000","last_change":"2024-02-12T17:14:46.848102+0000","last_active":"2024-02-12T17:15:38.973826+0000","last_peered":"2024-02-12T17:15:38.973826+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.832453+0000","last_became_peered":"2024-02-12T17:13:46.832453+0000","last_unstale":"2024-02-12T17:15:38.973826+0000","last_undegraded":"2024-02-12T17:13:46.655125+0000","last_fullsized":"2024-02-12T17:13:46.642131+0000","mapping_epoch":631,"log_start":"626'7170","ondisk_log_start":"626'7170","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5724,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":185335808,"num_objects":2828,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12894,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":2828}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":398,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.766448+0000","last_change":"2024-02-12T17:13:21.589965+0000","last_active":"2024-02-12T17:15:31.766448+0000","last_peered":"2024-02-12T17:15:31.766448+0000","last_clean":"2024-02-12T17:15:31.766448+0000","last_became_active":"2024-02-12T17:13:21.589584+0000","last_became_peered":"2024-02-12T17:13:21.589584+0000","last_unstale":"2024-02-12T17:15:31.766448+0000","last_undegraded":"2024-02-12T17:15:31.766448+0000","last_fullsized":"2024-02-12T17:15:31.766448+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:01.059570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.059570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":129,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.783433+0000","last_change":"2024-02-12T17:14:18.716530+0000","last_active":"2024-02-12T17:15:27.783433+0000","last_peered":"2024-02-12T17:15:27.783433+0000","last_clean":"2024-02-12T17:15:27.783433+0000","last_became_active":"2024-02-12T17:14:18.716232+0000","last_became_peered":"2024-02-12T17:14:18.716232+0000","last_unstale":"2024-02-12T17:15:27.783433+0000","last_undegraded":"2024-02-12T17:15:27.783433+0000","last_fullsized":"2024-02-12T17:15:27.783433+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":457,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.754842+0000","last_change":"2024-02-12T17:14:10.144923+0000","last_active":"2024-02-12T17:15:31.754842+0000","last_peered":"2024-02-12T17:15:31.754842+0000","last_clean":"2024-02-12T17:15:31.754842+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:15:31.754842+0000","last_undegraded":"2024-02-12T17:15:31.754842+0000","last_fullsized":"2024-02-12T17:15:31.754842+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":677,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.748142+0000","last_change":"2024-02-12T17:14:08.765556+0000","last_active":"2024-02-12T17:15:31.748142+0000","last_peered":"2024-02-12T17:15:31.748142+0000","last_clean":"2024-02-12T17:15:31.748142+0000","last_became_active":"2024-02-12T17:14:08.757310+0000","last_became_peered":"2024-02-12T17:14:08.757310+0000","last_unstale":"2024-02-12T17:15:31.748142+0000","last_undegraded":"2024-02-12T17:15:31.748142+0000","last_fullsized":"2024-02-12T17:15:31.748142+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:58.871569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.871569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"690'12940","reported_seq":16300,"reported_epoch":690,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:15:38.973047+0000","last_change":"2024-02-12T17:14:10.288517+0000","last_active":"2024-02-12T17:15:38.973047+0000","last_peered":"2024-02-12T17:15:38.973047+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.867083+0000","last_became_peered":"2024-02-12T17:14:08.867083+0000","last_unstale":"2024-02-12T17:15:38.973047+0000","last_undegraded":"2024-02-12T17:14:08.714899+0000","last_fullsized":"2024-02-12T17:14:08.701373+0000","mapping_epoch":651,"log_start":"626'7242","ondisk_log_start":"626'7242","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5698,"ondisk_log_size":5698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":182976512,"num_objects":2792,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":2224,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12940,"num_write_kb":503424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":76087296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2792}],"blocked_by":[],"up_primary":1,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":728,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762741+0000","last_change":"2024-02-12T17:13:08.874063+0000","last_active":"2024-02-12T17:15:27.762741+0000","last_peered":"2024-02-12T17:15:27.762741+0000","last_clean":"2024-02-12T17:15:27.762741+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:15:27.762741+0000","last_undegraded":"2024-02-12T17:15:27.762741+0000","last_fullsized":"2024-02-12T17:15:27.762741+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:13:08.873961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:13:08.873961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":209,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.764794+0000","last_change":"2024-02-12T17:14:14.843264+0000","last_active":"2024-02-12T17:15:27.764794+0000","last_peered":"2024-02-12T17:15:27.764794+0000","last_clean":"2024-02-12T17:15:27.764794+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:15:27.764794+0000","last_undegraded":"2024-02-12T17:15:27.764794+0000","last_fullsized":"2024-02-12T17:15:27.764794+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:56.688982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.688982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":874,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768318+0000","last_change":"2024-02-12T17:14:09.517724+0000","last_active":"2024-02-12T17:15:27.768318+0000","last_peered":"2024-02-12T17:15:27.768318+0000","last_clean":"2024-02-12T17:15:27.768318+0000","last_became_active":"2024-02-12T17:14:08.740691+0000","last_became_peered":"2024-02-12T17:14:08.740691+0000","last_unstale":"2024-02-12T17:15:27.768318+0000","last_undegraded":"2024-02-12T17:15:27.768318+0000","last_fullsized":"2024-02-12T17:15:27.768318+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:12:05.850456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.850456+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"655'32","reported_seq":633,"reported_epoch":690,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:37.657197+0000","last_change":"2024-02-12T17:14:09.746021+0000","last_active":"2024-02-12T17:15:37.657197+0000","last_peered":"2024-02-12T17:15:37.657197+0000","last_clean":"2024-02-12T17:14:07.617720+0000","last_became_active":"2024-02-12T17:14:09.745704+0000","last_became_peered":"2024-02-12T17:14:09.745704+0000","last_unstale":"2024-02-12T17:15:37.657197+0000","last_undegraded":"2024-02-12T17:15:37.657197+0000","last_fullsized":"2024-02-12T17:15:37.657197+0000","mapping_epoch":652,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3","0","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"690'12927","reported_seq":13044,"reported_epoch":690,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:38.702293+0000","last_change":"2024-02-12T17:14:09.843467+0000","last_active":"2024-02-12T17:15:38.702293+0000","last_peered":"2024-02-12T17:15:38.702293+0000","last_clean":"2024-02-12T17:14:07.683370+0000","last_became_active":"2024-02-12T17:14:09.782739+0000","last_became_peered":"2024-02-12T17:14:09.782739+0000","last_unstale":"2024-02-12T17:15:38.702293+0000","last_undegraded":"2024-02-12T17:15:38.702293+0000","last_fullsized":"2024-02-12T17:15:38.702293+0000","mapping_epoch":652,"log_start":"623'6851","ondisk_log_start":"623'6851","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6076,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":178585600,"num_objects":2725,"num_object_clones":0,"num_object_copies":8175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2725,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12927,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":2725}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":400,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.749335+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:15:31.749335+0000","last_peered":"2024-02-12T17:15:31.749335+0000","last_clean":"2024-02-12T17:15:31.749335+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:15:31.749335+0000","last_undegraded":"2024-02-12T17:15:31.749335+0000","last_fullsized":"2024-02-12T17:15:31.749335+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"642'31","reported_seq":1093,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:32.657206+0000","last_change":"2024-02-12T17:13:53.773053+0000","last_active":"2024-02-12T17:15:32.657206+0000","last_peered":"2024-02-12T17:15:32.657206+0000","last_clean":"2024-02-12T17:15:32.657206+0000","last_became_active":"2024-02-12T17:13:53.772744+0000","last_became_peered":"2024-02-12T17:13:53.772744+0000","last_unstale":"2024-02-12T17:15:32.657206+0000","last_undegraded":"2024-02-12T17:15:32.657206+0000","last_fullsized":"2024-02-12T17:15:32.657206+0000","mapping_epoch":638,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'28","last_scrub_stamp":"2024-02-12T17:12:07.826227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:12:07.826227+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":403,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.761950+0000","last_change":"2024-02-12T17:14:10.756477+0000","last_active":"2024-02-12T17:15:27.761950+0000","last_peered":"2024-02-12T17:15:27.761950+0000","last_clean":"2024-02-12T17:15:27.761950+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:15:27.761950+0000","last_undegraded":"2024-02-12T17:15:27.761950+0000","last_fullsized":"2024-02-12T17:15:27.761950+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:16.873570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.873570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":263,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.764539+0000","last_change":"2024-02-12T17:13:46.807564+0000","last_active":"2024-02-12T17:15:27.764539+0000","last_peered":"2024-02-12T17:15:27.764539+0000","last_clean":"2024-02-12T17:15:27.764539+0000","last_became_active":"2024-02-12T17:13:46.807214+0000","last_became_peered":"2024-02-12T17:13:46.807214+0000","last_unstale":"2024-02-12T17:15:27.764539+0000","last_undegraded":"2024-02-12T17:15:27.764539+0000","last_fullsized":"2024-02-12T17:15:27.764539+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:04.705646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.705646+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"691'13193","reported_seq":15873,"reported_epoch":691,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:15:41.657968+0000","last_change":"2024-02-12T17:14:09.819105+0000","last_active":"2024-02-12T17:15:41.657968+0000","last_peered":"2024-02-12T17:15:41.657968+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.810504+0000","last_became_peered":"2024-02-12T17:13:46.810504+0000","last_unstale":"2024-02-12T17:15:41.657968+0000","last_undegraded":"2024-02-12T17:13:46.690611+0000","last_fullsized":"2024-02-12T17:13:46.648648+0000","mapping_epoch":631,"log_start":"635'8271","ondisk_log_start":"635'8271","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4922,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":175964160,"num_objects":2685,"num_object_clones":0,"num_object_copies":8055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13193,"num_write_kb":508096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1284,"num_bytes_recovered":84148224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":2685}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":733,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768880+0000","last_change":"2024-02-12T17:14:08.740896+0000","last_active":"2024-02-12T17:15:27.768880+0000","last_peered":"2024-02-12T17:15:27.768880+0000","last_clean":"2024-02-12T17:15:27.768880+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:15:27.768880+0000","last_undegraded":"2024-02-12T17:15:27.768880+0000","last_fullsized":"2024-02-12T17:15:27.768880+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"668'27","reported_seq":1118,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:32.657239+0000","last_change":"2024-02-12T17:14:24.235718+0000","last_active":"2024-02-12T17:15:32.657239+0000","last_peered":"2024-02-12T17:15:32.657239+0000","last_clean":"2024-02-12T17:15:32.657239+0000","last_became_active":"2024-02-12T17:14:24.235386+0000","last_became_peered":"2024-02-12T17:14:24.235386+0000","last_unstale":"2024-02-12T17:15:32.657239+0000","last_undegraded":"2024-02-12T17:15:32.657239+0000","last_fullsized":"2024-02-12T17:15:32.657239+0000","mapping_epoch":665,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'24","last_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.647212+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":100,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.786693+0000","last_change":"2024-02-12T17:14:05.106537+0000","last_active":"2024-02-12T17:15:27.786693+0000","last_peered":"2024-02-12T17:15:27.786693+0000","last_clean":"2024-02-12T17:15:27.786693+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:15:27.786693+0000","last_undegraded":"2024-02-12T17:15:27.786693+0000","last_fullsized":"2024-02-12T17:15:27.786693+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":148,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.783630+0000","last_change":"2024-02-12T17:13:46.819259+0000","last_active":"2024-02-12T17:15:27.783630+0000","last_peered":"2024-02-12T17:15:27.783630+0000","last_clean":"2024-02-12T17:15:27.783630+0000","last_became_active":"2024-02-12T17:13:46.818771+0000","last_became_peered":"2024-02-12T17:13:46.818771+0000","last_unstale":"2024-02-12T17:15:27.783630+0000","last_undegraded":"2024-02-12T17:15:27.783630+0000","last_fullsized":"2024-02-12T17:15:27.783630+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"690'12743","reported_seq":25665,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:38.609538+0000","last_change":"2024-02-12T17:14:32.710386+0000","last_active":"2024-02-12T17:15:38.609538+0000","last_peered":"2024-02-12T17:15:38.609538+0000","last_clean":"2024-02-12T17:15:38.609538+0000","last_became_active":"2024-02-12T17:14:32.696812+0000","last_became_peered":"2024-02-12T17:14:32.696812+0000","last_unstale":"2024-02-12T17:15:38.609538+0000","last_undegraded":"2024-02-12T17:15:38.609538+0000","last_fullsized":"2024-02-12T17:15:38.609538+0000","mapping_epoch":671,"log_start":"623'6700","ondisk_log_start":"623'6700","created":597,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6043,"ondisk_log_size":6043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":175570944,"num_objects":2679,"num_object_clones":0,"num_object_copies":8037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12743,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6402,"num_bytes_recovered":419561472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":80,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.783409+0000","last_change":"2024-02-12T17:13:46.805110+0000","last_active":"2024-02-12T17:15:27.783409+0000","last_peered":"2024-02-12T17:15:27.783409+0000","last_clean":"2024-02-12T17:15:27.783409+0000","last_became_active":"2024-02-12T17:13:46.802275+0000","last_became_peered":"2024-02-12T17:13:46.802275+0000","last_unstale":"2024-02-12T17:15:27.783409+0000","last_undegraded":"2024-02-12T17:15:27.783409+0000","last_fullsized":"2024-02-12T17:15:27.783409+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":621,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:32.657229+0000","last_change":"2024-02-12T17:13:07.711701+0000","last_active":"2024-02-12T17:15:32.657229+0000","last_peered":"2024-02-12T17:15:32.657229+0000","last_clean":"2024-02-12T17:15:32.657229+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:15:32.657229+0000","last_undegraded":"2024-02-12T17:15:32.657229+0000","last_fullsized":"2024-02-12T17:15:32.657229+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:13:07.711601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:07.711601+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":100,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.779435+0000","last_change":"2024-02-12T17:14:00.999940+0000","last_active":"2024-02-12T17:15:27.779435+0000","last_peered":"2024-02-12T17:15:27.779435+0000","last_clean":"2024-02-12T17:15:27.779435+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:15:27.779435+0000","last_undegraded":"2024-02-12T17:15:27.779435+0000","last_fullsized":"2024-02-12T17:15:27.779435+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":494,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.749429+0000","last_change":"2024-02-12T17:13:46.806094+0000","last_active":"2024-02-12T17:15:31.749429+0000","last_peered":"2024-02-12T17:15:31.749429+0000","last_clean":"2024-02-12T17:15:31.749429+0000","last_became_active":"2024-02-12T17:13:46.805712+0000","last_became_peered":"2024-02-12T17:13:46.805712+0000","last_unstale":"2024-02-12T17:15:31.749429+0000","last_undegraded":"2024-02-12T17:15:31.749429+0000","last_fullsized":"2024-02-12T17:15:31.749429+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"691'13179","reported_seq":13281,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T17:15:41.623274+0000","last_change":"2024-02-12T17:11:59.496950+0000","last_active":"2024-02-12T17:15:41.623274+0000","last_peered":"2024-02-12T17:15:41.623274+0000","last_clean":"2024-02-12T17:15:41.623274+0000","last_became_active":"2024-02-12T17:11:59.496585+0000","last_became_peered":"2024-02-12T17:11:59.496585+0000","last_unstale":"2024-02-12T17:15:41.623274+0000","last_undegraded":"2024-02-12T17:15:41.623274+0000","last_fullsized":"2024-02-12T17:15:41.623274+0000","mapping_epoch":597,"log_start":"634'8248","ondisk_log_start":"634'8248","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4931,"ondisk_log_size":4931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":175964160,"num_objects":2685,"num_object_clones":0,"num_object_copies":8055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13179,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":420,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.750952+0000","last_change":"2024-02-12T17:13:21.603687+0000","last_active":"2024-02-12T17:15:31.750952+0000","last_peered":"2024-02-12T17:15:31.750952+0000","last_clean":"2024-02-12T17:15:31.750952+0000","last_became_active":"2024-02-12T17:13:21.603332+0000","last_became_peered":"2024-02-12T17:13:21.603332+0000","last_unstale":"2024-02-12T17:15:31.750952+0000","last_undegraded":"2024-02-12T17:15:31.750952+0000","last_fullsized":"2024-02-12T17:15:31.750952+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":111,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.783082+0000","last_change":"2024-02-12T17:13:46.804446+0000","last_active":"2024-02-12T17:15:27.783082+0000","last_peered":"2024-02-12T17:15:27.783082+0000","last_clean":"2024-02-12T17:15:27.783082+0000","last_became_active":"2024-02-12T17:13:46.804033+0000","last_became_peered":"2024-02-12T17:13:46.804033+0000","last_unstale":"2024-02-12T17:15:27.783082+0000","last_undegraded":"2024-02-12T17:15:27.783082+0000","last_fullsized":"2024-02-12T17:15:27.783082+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.793569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.793569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":713,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762392+0000","last_change":"2024-02-12T17:13:56.661004+0000","last_active":"2024-02-12T17:15:27.762392+0000","last_peered":"2024-02-12T17:15:27.762392+0000","last_clean":"2024-02-12T17:15:27.762392+0000","last_became_active":"2024-02-12T17:13:56.660707+0000","last_became_peered":"2024-02-12T17:13:56.660707+0000","last_unstale":"2024-02-12T17:15:27.762392+0000","last_undegraded":"2024-02-12T17:15:27.762392+0000","last_fullsized":"2024-02-12T17:15:27.762392+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.685601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.685601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":180,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.782142+0000","last_change":"2024-02-12T17:13:46.757888+0000","last_active":"2024-02-12T17:15:27.782142+0000","last_peered":"2024-02-12T17:15:27.782142+0000","last_clean":"2024-02-12T17:15:27.782142+0000","last_became_active":"2024-02-12T17:13:46.757450+0000","last_became_peered":"2024-02-12T17:13:46.757450+0000","last_unstale":"2024-02-12T17:15:27.782142+0000","last_undegraded":"2024-02-12T17:15:27.782142+0000","last_fullsized":"2024-02-12T17:15:27.782142+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:56.880282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.880282+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.5","version":"690'12864","reported_seq":12968,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:38.870406+0000","last_change":"2024-02-12T17:13:12.678687+0000","last_active":"2024-02-12T17:15:38.870406+0000","last_peered":"2024-02-12T17:15:38.870406+0000","last_clean":"2024-02-12T17:15:38.870406+0000","last_became_active":"2024-02-12T17:11:59.009543+0000","last_became_peered":"2024-02-12T17:11:59.009543+0000","last_unstale":"2024-02-12T17:15:38.870406+0000","last_undegraded":"2024-02-12T17:15:38.870406+0000","last_fullsized":"2024-02-12T17:15:38.870406+0000","mapping_epoch":597,"log_start":"626'7129","ondisk_log_start":"626'7129","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5735,"ondisk_log_size":5735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":181534720,"num_objects":2770,"num_object_clones":0,"num_object_copies":8310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12864,"num_write_kb":500288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":713,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.762028+0000","last_change":"2024-02-12T17:12:08.013330+0000","last_active":"2024-02-12T17:15:27.762028+0000","last_peered":"2024-02-12T17:15:27.762028+0000","last_clean":"2024-02-12T17:15:27.762028+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:15:27.762028+0000","last_undegraded":"2024-02-12T17:15:27.762028+0000","last_fullsized":"2024-02-12T17:15:27.762028+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.013247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.013247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":610,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.780210+0000","last_change":"2024-02-12T17:12:04.711418+0000","last_active":"2024-02-12T17:15:27.780210+0000","last_peered":"2024-02-12T17:15:27.780210+0000","last_clean":"2024-02-12T17:15:27.780210+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:15:27.780210+0000","last_undegraded":"2024-02-12T17:15:27.780210+0000","last_fullsized":"2024-02-12T17:15:27.780210+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:04.711333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.711333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":468,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768344+0000","last_change":"2024-02-12T17:14:08.740752+0000","last_active":"2024-02-12T17:15:27.768344+0000","last_peered":"2024-02-12T17:15:27.768344+0000","last_clean":"2024-02-12T17:15:27.768344+0000","last_became_active":"2024-02-12T17:14:08.740430+0000","last_became_peered":"2024-02-12T17:14:08.740430+0000","last_unstale":"2024-02-12T17:15:27.768344+0000","last_undegraded":"2024-02-12T17:15:27.768344+0000","last_fullsized":"2024-02-12T17:15:27.768344+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":245,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.766718+0000","last_change":"2024-02-12T17:13:46.559877+0000","last_active":"2024-02-12T17:15:27.766718+0000","last_peered":"2024-02-12T17:15:27.766718+0000","last_clean":"2024-02-12T17:15:27.766718+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:15:27.766718+0000","last_undegraded":"2024-02-12T17:15:27.766718+0000","last_fullsized":"2024-02-12T17:15:27.766718+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"690'13085","reported_seq":13186,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:38.924972+0000","last_change":"2024-02-12T17:11:59.007574+0000","last_active":"2024-02-12T17:15:38.924972+0000","last_peered":"2024-02-12T17:15:38.924972+0000","last_clean":"2024-02-12T17:15:38.924972+0000","last_became_active":"2024-02-12T17:11:59.007300+0000","last_became_peered":"2024-02-12T17:11:59.007300+0000","last_unstale":"2024-02-12T17:15:38.924972+0000","last_undegraded":"2024-02-12T17:15:38.924972+0000","last_fullsized":"2024-02-12T17:15:38.924972+0000","mapping_epoch":597,"log_start":"627'7367","ondisk_log_start":"627'7367","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5718,"ondisk_log_size":5718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":184090624,"num_objects":2809,"num_object_clones":0,"num_object_copies":8427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13085,"num_write_kb":508608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":48,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.780134+0000","last_change":"2024-02-12T17:14:08.859503+0000","last_active":"2024-02-12T17:15:27.780134+0000","last_peered":"2024-02-12T17:15:27.780134+0000","last_clean":"2024-02-12T17:15:27.780134+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:15:27.780134+0000","last_undegraded":"2024-02-12T17:15:27.780134+0000","last_fullsized":"2024-02-12T17:15:27.780134+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:03.705889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:13:03.705889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":611,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.747408+0000","last_change":"2024-02-12T17:14:08.762783+0000","last_active":"2024-02-12T17:15:31.747408+0000","last_peered":"2024-02-12T17:15:31.747408+0000","last_clean":"2024-02-12T17:15:31.747408+0000","last_became_active":"2024-02-12T17:14:08.760654+0000","last_became_peered":"2024-02-12T17:14:08.760654+0000","last_unstale":"2024-02-12T17:15:31.747408+0000","last_undegraded":"2024-02-12T17:15:31.747408+0000","last_fullsized":"2024-02-12T17:15:31.747408+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:09.044325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:09.044325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":389,"reported_epoch":690,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:15:31.750131+0000","last_change":"2024-02-12T17:14:16.753241+0000","last_active":"2024-02-12T17:15:31.750131+0000","last_peered":"2024-02-12T17:15:31.750131+0000","last_clean":"2024-02-12T17:13:44.508652+0000","last_became_active":"2024-02-12T17:13:46.563455+0000","last_became_peered":"2024-02-12T17:13:46.563455+0000","last_unstale":"2024-02-12T17:15:31.750131+0000","last_undegraded":"2024-02-12T17:15:31.750131+0000","last_fullsized":"2024-02-12T17:15:31.750131+0000","mapping_epoch":630,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":58,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.779797+0000","last_change":"2024-02-12T17:14:10.285684+0000","last_active":"2024-02-12T17:15:27.779797+0000","last_peered":"2024-02-12T17:15:27.779797+0000","last_clean":"2024-02-12T17:15:27.779797+0000","last_became_active":"2024-02-12T17:14:08.871051+0000","last_became_peered":"2024-02-12T17:14:08.871051+0000","last_unstale":"2024-02-12T17:15:27.779797+0000","last_undegraded":"2024-02-12T17:15:27.779797+0000","last_fullsized":"2024-02-12T17:15:27.779797+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:50.764686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.764686+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"690'12870","reported_seq":19877,"reported_epoch":690,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:38.436990+0000","last_change":"2024-02-12T17:14:39.687781+0000","last_active":"2024-02-12T17:15:38.436990+0000","last_peered":"2024-02-12T17:15:38.436990+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.886954+0000","last_became_peered":"2024-02-12T17:14:08.886954+0000","last_unstale":"2024-02-12T17:15:38.436990+0000","last_undegraded":"2024-02-12T17:14:08.719570+0000","last_fullsized":"2024-02-12T17:14:08.707803+0000","mapping_epoch":651,"log_start":"620'6478","ondisk_log_start":"620'6478","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6392,"ondisk_log_size":6392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":182190080,"num_objects":2780,"num_object_clones":0,"num_object_copies":8340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2780,"num_objects_misplaced":2780,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12870,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3435,"num_bytes_recovered":225050624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,4],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5","objects":2780}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":696,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768605+0000","last_change":"2024-02-12T17:12:11.906816+0000","last_active":"2024-02-12T17:15:27.768605+0000","last_peered":"2024-02-12T17:15:27.768605+0000","last_clean":"2024-02-12T17:15:27.768605+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:15:27.768605+0000","last_undegraded":"2024-02-12T17:15:27.768605+0000","last_fullsized":"2024-02-12T17:15:27.768605+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:11.906737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:11.906737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":375,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:31.750700+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:15:31.750700+0000","last_peered":"2024-02-12T17:15:31.750700+0000","last_clean":"2024-02-12T17:15:31.750700+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:15:31.750700+0000","last_undegraded":"2024-02-12T17:15:31.750700+0000","last_fullsized":"2024-02-12T17:15:31.750700+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":814,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T17:15:27.768099+0000","last_change":"2024-02-12T17:12:16.820857+0000","last_active":"2024-02-12T17:15:27.768099+0000","last_peered":"2024-02-12T17:15:27.768099+0000","last_clean":"2024-02-12T17:15:27.768099+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:15:27.768099+0000","last_undegraded":"2024-02-12T17:15:27.768099+0000","last_fullsized":"2024-02-12T17:15:27.768099+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:12:16.820782+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.820782+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"632'35","reported_seq":1182,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T17:15:32.657245+0000","last_change":"2024-02-12T17:13:46.691596+0000","last_active":"2024-02-12T17:15:32.657245+0000","last_peered":"2024-02-12T17:15:32.657245+0000","last_clean":"2024-02-12T17:15:32.657245+0000","last_became_active":"2024-02-12T17:13:46.691213+0000","last_became_peered":"2024-02-12T17:13:46.691213+0000","last_unstale":"2024-02-12T17:15:32.657245+0000","last_undegraded":"2024-02-12T17:15:32.657245+0000","last_fullsized":"2024-02-12T17:15:32.657245+0000","mapping_epoch":631,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'29","last_scrub_stamp":"2024-02-12T17:12:47.788327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.788327+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2853044248,"num_objects":43535,"num_object_clones":0,"num_object_copies":130605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18411,"num_objects_misplaced":18163,"num_objects_unfound":0,"num_objects_dirty":43535,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207841,"num_write_kb":8043969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20451,"num_bytes_recovered":1339949056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7898804224,"data_stored":7898792008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91244,"ondisk_log_size":91244,"up":48,"acting":40,"num_store_stats":8},{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":325,"num_read_kb":320,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":16216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1029,"ondisk_log_size":1029,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234299,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2828100,"kb_used_data":1344124,"kb_used_omap":0,"kb_used_meta":1483968,"kb_avail":90917052,"statfs":{"total":95995035648,"available":93099061248,"internally_reserved":0,"allocated":1376382976,"data_stored":1375030249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1519583232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":2.6899999999999999,"5min":1.9910000000000001,"15min":2.077},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":15.856999999999999,"5min":15.856999999999999,"15min":16.759},"last":0.70199999999999996},{"interface":"front","average":{"1min":2.79,"5min":1.837,"15min":1.915},"min":{"1min":0.27600000000000002,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":12.565,"5min":12.565,"15min":17.844000000000001},"last":0.45200000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:14:47 2024","interfaces":[{"interface":"back","average":{"1min":2.7890000000000001,"5min":2.7890000000000001,"15min":2.7890000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.8989999999999991,"5min":9.8989999999999991,"15min":9.8989999999999991},"last":1.8859999999999999},{"interface":"front","average":{"1min":2.468,"5min":2.468,"15min":2.468},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":9.9120000000000008,"5min":9.9120000000000008,"15min":9.9120000000000008},"last":1.827}]},{"osd":2,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":3.431,"5min":2.8359999999999999,"15min":3.1419999999999999},"min":{"1min":0.34599999999999997,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":23.611000000000001,"5min":42.162999999999997,"15min":218.35300000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":3.8090000000000002,"5min":2.8639999999999999,"15min":3.2269999999999999},"min":{"1min":0.47099999999999997,"5min":0.247,"15min":0.247},"max":{"1min":15.672000000000001,"5min":42.109999999999999,"15min":218.42099999999999},"last":0.375}]},{"osd":3,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":2.415,"15min":2.0939999999999999},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.255},"max":{"1min":7.5830000000000002,"5min":15.811999999999999,"15min":15.811999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":4.1710000000000003,"5min":2.847,"15min":2.5350000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":11.009,"5min":13.452,"15min":31.152000000000001},"last":3.4700000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.421,"15min":1.8540000000000001},"min":{"1min":0.247,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":3.4969999999999999,"5min":20.672999999999998,"15min":20.672999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.976,"15min":2.1509999999999998},"min":{"1min":0.29299999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.6769999999999996,"5min":38.390000000000001,"15min":38.390000000000001},"last":0.39700000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.8029999999999999,"15min":2.407},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":4.79,"5min":20.702000000000002,"15min":56.560000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":2.5089999999999999,"5min":1.4690000000000001,"15min":2.1139999999999999},"min":{"1min":0.28499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":9.7170000000000005,"5min":10.669,"15min":53.963999999999999},"last":0.41399999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.4690000000000001,"15min":1.734},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":5.2539999999999996,"5min":21.620000000000001,"15min":21.620000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.722,"15min":1.8759999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29199999999999998},"max":{"1min":3.7519999999999998,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.36199999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528640,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2311124,"kb_used_data":1418764,"kb_used_omap":0,"kb_used_meta":892352,"kb_avail":91434028,"statfs":{"total":95995035648,"available":93628444672,"internally_reserved":0,"allocated":1452814336,"data_stored":1451445869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913768448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":1.9930000000000001,"5min":1.7629999999999999,"15min":1.9510000000000001},"min":{"1min":0.23100000000000001,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":8.0920000000000005,"5min":33.576999999999998,"15min":33.576999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.321,"5min":2.125,"15min":1.919},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":6.1100000000000003,"5min":33.569000000000003,"15min":33.569000000000003},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:14:46 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":13.805999999999999,"5min":13.805999999999999,"15min":13.805999999999999},"last":3.0529999999999999},{"interface":"front","average":{"1min":2.194,"5min":2.194,"15min":2.194},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":13.778,"5min":13.778,"15min":13.778},"last":2.8199999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":2.5680000000000001,"5min":2.75,"15min":2.5649999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.311,"5min":33.594999999999999,"15min":56.798000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.968,"5min":2.234,"15min":2.3149999999999999},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":9.2479999999999993,"5min":33.526000000000003,"15min":56.780999999999999},"last":5.915}]},{"osd":3,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":2.7469999999999999,"5min":3.3330000000000002,"15min":2.3170000000000002},"min":{"1min":0.35399999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":10.712,"5min":58.420999999999999,"15min":58.420999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":3.5590000000000002,"5min":3.5790000000000002,"15min":2.7450000000000001},"min":{"1min":0.33200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":14.69,"5min":58.709000000000003,"15min":58.709000000000003},"last":2.6219999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.514,"15min":1.585},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":11.012,"5min":33.706000000000003,"15min":33.706000000000003},"last":1.49},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.823,"15min":1.635},"min":{"1min":0.28799999999999998,"5min":0.222,"15min":0.222},"max":{"1min":10.948,"5min":34.895000000000003,"15min":34.895000000000003},"last":0.54000000000000004}]},{"osd":5,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.877,"15min":1.827},"min":{"1min":0.30599999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":10.77,"5min":34.869,"15min":34.869},"last":1.28},{"interface":"front","average":{"1min":1.196,"5min":1.4990000000000001,"15min":2.0249999999999999},"min":{"1min":0.32100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":11.1,"5min":33.591000000000001,"15min":33.591000000000001},"last":1.5169999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.677,"15min":1.871},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":6.1310000000000002,"5min":35.122999999999998,"15min":35.122999999999998},"last":0.624},{"interface":"front","average":{"1min":1.28,"5min":1.7390000000000001,"15min":1.901},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.26500000000000001},"max":{"1min":11.081,"5min":34.045000000000002,"15min":34.045000000000002},"last":1.5369999999999999}]}]},{"osd":1,"up_from":630,"seq":2705829396504,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":619428,"kb_used_data":284892,"kb_used_omap":3,"kb_used_meta":334524,"kb_avail":93125724,"statfs":{"total":95995035648,"available":95360741376,"internally_reserved":0,"allocated":291729408,"data_stored":290344998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":342552637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4849999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":14.67,"5min":14.67,"15min":14.67},"last":2.605},{"interface":"front","average":{"1min":3.4209999999999998,"5min":3.4209999999999998,"15min":3.4209999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"last":6.7699999999999996}]},{"osd":2,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":3.5819999999999999,"5min":3.5819999999999999,"15min":3.5819999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":8.766,"5min":8.766,"15min":8.766},"last":2.5659999999999998},{"interface":"front","average":{"1min":3.4969999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.880000000000001,"5min":12.880000000000001,"15min":12.880000000000001},"last":6.7350000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":3.6429999999999998,"5min":3.6429999999999998,"15min":3.6429999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":10.375,"5min":10.375,"15min":10.375},"last":2.5790000000000002},{"interface":"front","average":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":13.564,"5min":13.564,"15min":13.564},"last":4.0609999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":4.5629999999999997},{"interface":"front","average":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":2.7810000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":12.805999999999999,"5min":12.805999999999999,"15min":12.805999999999999},"last":8.8249999999999993}]},{"osd":5,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":3.2160000000000002,"5min":3.2160000000000002,"15min":3.2160000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":12.769,"5min":12.769,"15min":12.769},"last":8.9120000000000008},{"interface":"front","average":{"1min":2.113,"5min":2.113,"15min":2.113},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":6.3239999999999998,"5min":6.3239999999999998,"15min":6.3239999999999998},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":2.8300000000000001,"5min":2.8300000000000001,"15min":2.8300000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":12.720000000000001,"5min":12.720000000000001,"15min":12.720000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":8.0890000000000004,"5min":8.0890000000000004,"15min":8.0890000000000004},"last":2.52}]},{"osd":7,"last update":"Mon Feb 12 17:14:50 2024","interfaces":[{"interface":"back","average":{"1min":2.964,"5min":2.964,"15min":2.964},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":12.829000000000001,"5min":12.829000000000001,"15min":12.829000000000001},"last":7.298},{"interface":"front","average":{"1min":2.3919999999999999,"5min":2.3919999999999999,"15min":2.3919999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":6.4649999999999999},"last":0.61399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738717,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1176764,"kb_used_data":280564,"kb_used_omap":0,"kb_used_meta":896192,"kb_avail":92568388,"statfs":{"total":95995035648,"available":94790029312,"internally_reserved":0,"allocated":287297536,"data_stored":285953006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":917700608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:14:48 2024","interfaces":[{"interface":"back","average":{"1min":3.4289999999999998,"5min":3.4289999999999998,"15min":3.4289999999999998},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":13.936999999999999,"5min":13.936999999999999,"15min":13.936999999999999},"last":5.3470000000000004},{"interface":"front","average":{"1min":2.802,"5min":2.802,"15min":2.802},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":13.787000000000001,"5min":13.787000000000001,"15min":13.787000000000001},"last":0.17699999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:15:31 2024","interfaces":[{"interface":"back","average":{"1min":3.6640000000000001,"5min":2.7759999999999998,"15min":2.0710000000000002},"min":{"1min":0.311,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":13.762,"5min":19.312000000000001,"15min":19.312000000000001},"last":3.7599999999999998},{"interface":"front","average":{"1min":4.1500000000000004,"5min":3.0539999999999998,"15min":2.6019999999999999},"min":{"1min":0.39700000000000002,"5min":0.218,"15min":0.20799999999999999},"max":{"1min":22.959,"5min":22.959,"15min":26.039999999999999},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:14:43 2024","interfaces":[{"interface":"back","average":{"1min":3.7280000000000002,"5min":3.0590000000000002,"15min":2.5859999999999999},"min":{"1min":0.23000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":25.143000000000001,"5min":25.143000000000001,"15min":87.311000000000007},"last":1.839},{"interface":"front","average":{"1min":4.6440000000000001,"5min":3.4100000000000001,"15min":2.8519999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":24.690999999999999,"5min":24.690999999999999,"15min":90.991},"last":13.433999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:15:04 2024","interfaces":[{"interface":"back","average":{"1min":2.363,"5min":1.6000000000000001,"15min":1.5629999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.23599999999999999},"max":{"1min":13.861000000000001,"5min":15.141999999999999,"15min":16.888999999999999},"last":3.7320000000000002},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.6030000000000002,"15min":2.1000000000000001},"min":{"1min":0.30499999999999999,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":13.503,"5min":26.082999999999998,"15min":26.082999999999998},"last":6.7220000000000004}]},{"osd":5,"last update":"Mon Feb 12 17:14:37 2024","interfaces":[{"interface":"back","average":{"1min":1.9219999999999999,"5min":2.4729999999999999,"15min":2.2320000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":6.3609999999999998,"5min":17.754000000000001,"15min":50.404000000000003},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":1.9159999999999999,"15min":2.0659999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":13.837,"5min":15.164,"15min":49.979999999999997},"last":3.7919999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":3.4119999999999999,"15min":2.343},"min":{"1min":0.39900000000000002,"5min":0.35499999999999998,"15min":0.29999999999999999},"max":{"1min":13.48,"5min":48.780999999999999,"15min":48.780999999999999},"last":6.758},{"interface":"front","average":{"1min":2.2250000000000001,"5min":3.1280000000000001,"15min":2.3109999999999999},"min":{"1min":0.36399999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":13.962999999999999,"5min":49.067999999999998,"15min":49.067999999999998},"last":1.3999999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:15:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.04,"15min":2.04},"min":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":13.813000000000001,"5min":15.183999999999999,"15min":19.373000000000001},"last":6.8339999999999996},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.3279999999999998,"15min":2.4430000000000001},"min":{"1min":0.38400000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":13.542,"5min":13.542,"15min":26.135000000000002},"last":1.1180000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411666,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2758716,"kb_used_data":1252276,"kb_used_omap":0,"kb_used_meta":1506368,"kb_avail":90986436,"statfs":{"total":95995035648,"available":93170110464,"internally_reserved":0,"allocated":1282330624,"data_stored":1280986579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1542520832},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":3.2559999999999998,"5min":1.9690000000000001,"15min":1.8440000000000001},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17000000000000001},"max":{"1min":11.663,"5min":11.663,"15min":36.100999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":5.1219999999999999,"5min":2.4489999999999998,"15min":2.181},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":16.350000000000001,"5min":16.350000000000001,"15min":36.299999999999997},"last":4.8019999999999996}]},{"osd":1,"last update":"Mon Feb 12 17:14:47 2024","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":4.016,"15min":4.016},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":11.612,"5min":11.612,"15min":11.612},"last":2.9510000000000001},{"interface":"front","average":{"1min":5.8940000000000001,"5min":5.8940000000000001,"15min":5.8940000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":22.52,"5min":22.52,"15min":22.52},"last":6.3319999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":8.657,"5min":3.8519999999999999,"15min":2.6160000000000001},"min":{"1min":0.32300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":29.489999999999998,"5min":40.759999999999998,"15min":40.759999999999998},"last":0.432},{"interface":"front","average":{"1min":4.4900000000000002,"5min":2.702,"15min":2.1899999999999999},"min":{"1min":0.38900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":12.33,"5min":17.358000000000001,"15min":21.702999999999999},"last":14.609999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:15:22 2024","interfaces":[{"interface":"back","average":{"1min":3.008,"5min":2.0129999999999999,"15min":1.98},"min":{"1min":0.27600000000000002,"5min":0.245,"15min":0.245},"max":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":2.8220000000000001,"5min":1.8939999999999999,"15min":2.0510000000000002},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.744,"5min":11.744,"15min":20.149000000000001},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":2.681,"5min":1.8700000000000001,"15min":2.0139999999999998},"min":{"1min":0.33900000000000002,"5min":0.217,"15min":0.217},"max":{"1min":11.581,"5min":11.581,"15min":32.57},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.881,"15min":2.1349999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.255},"max":{"1min":7.798,"5min":10.896000000000001,"15min":31.905999999999999},"last":1.8600000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:14:57 2024","interfaces":[{"interface":"back","average":{"1min":2.9550000000000001,"5min":2.0720000000000001,"15min":1.8979999999999999},"min":{"1min":0.36899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":10.714,"5min":10.714,"15min":20.966999999999999},"last":1.8899999999999999},{"interface":"front","average":{"1min":2.8420000000000001,"5min":2.0880000000000001,"15min":1.9179999999999999},"min":{"1min":0.432,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":11.834,"5min":11.834,"15min":20.193000000000001},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9590000000000001,"5min":2.121,"15min":2.4830000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":11.789,"5min":11.789,"15min":45.468000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.9990000000000001,"15min":2.4319999999999999},"min":{"1min":0.34499999999999997,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":7.8330000000000002,"5min":10.762,"15min":46.656999999999996},"last":1.9119999999999999}]}]},{"osd":3,"up_from":26,"seq":111669150029,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2115052,"kb_used_data":1205284,"kb_used_omap":0,"kb_used_meta":909632,"kb_avail":91630100,"statfs":{"total":95995035648,"available":93829222400,"internally_reserved":0,"allocated":1234210816,"data_stored":1232826062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":931463168},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:14:54 2024","interfaces":[{"interface":"back","average":{"1min":3.7669999999999999,"5min":3.4870000000000001,"15min":2.4470000000000001},"min":{"1min":0.22,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":20.861999999999998,"5min":20.861999999999998,"15min":58.451999999999998},"last":5.1600000000000001},{"interface":"front","average":{"1min":6.109,"5min":3.8210000000000002,"15min":2.8370000000000002},"min":{"1min":0.28499999999999998,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":42.984999999999999,"5min":72.688000000000002,"15min":72.688000000000002},"last":8.9100000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:14:48 2024","interfaces":[{"interface":"back","average":{"1min":4.4809999999999999,"5min":4.4809999999999999,"15min":4.4809999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":20.638000000000002,"5min":20.638000000000002,"15min":20.638000000000002},"last":2.6280000000000001},{"interface":"front","average":{"1min":5.4550000000000001,"5min":5.4550000000000001,"15min":5.4550000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":16.361999999999998,"5min":16.361999999999998,"15min":16.361999999999998},"last":8.8729999999999993}]},{"osd":2,"last update":"Mon Feb 12 17:14:54 2024","interfaces":[{"interface":"back","average":{"1min":3.2509999999999999,"5min":3.1000000000000001,"15min":2.0859999999999999},"min":{"1min":0.27400000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":10.819000000000001,"5min":40.954999999999998,"15min":40.954999999999998},"last":14.999000000000001},{"interface":"front","average":{"1min":3.577,"5min":4.367,"15min":2.5790000000000002},"min":{"1min":0.27100000000000002,"5min":0.22,"15min":0.22},"max":{"1min":17.347000000000001,"5min":46.408000000000001,"15min":46.408000000000001},"last":5.2889999999999997}]},{"osd":4,"last update":"Mon Feb 12 17:15:11 2024","interfaces":[{"interface":"back","average":{"1min":4.2880000000000003,"5min":3.3460000000000001,"15min":2.5169999999999999},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":8.7420000000000009,"5min":72.659000000000006,"15min":72.659000000000006},"last":5.2590000000000003},{"interface":"front","average":{"1min":2.9500000000000002,"5min":2.758,"15min":2.0419999999999998},"min":{"1min":0.313,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.087999999999999,"5min":40.930999999999997,"15min":40.930999999999997},"last":2.544}]},{"osd":5,"last update":"Mon Feb 12 17:15:33 2024","interfaces":[{"interface":"back","average":{"1min":5.7649999999999997,"5min":3.1389999999999998,"15min":2.577},"min":{"1min":0.438,"5min":0.33000000000000002,"15min":0.253},"max":{"1min":17.306999999999999,"5min":17.306999999999999,"15min":17.940999999999999},"last":5.2069999999999999},{"interface":"front","average":{"1min":4.2640000000000002,"5min":3.431,"15min":2.8500000000000001},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":11.586,"5min":72.620000000000005,"15min":72.620000000000005},"last":5.2990000000000004}]},{"osd":6,"last update":"Mon Feb 12 17:14:54 2024","interfaces":[{"interface":"back","average":{"1min":2.141,"5min":2.6920000000000002,"15min":1.927},"min":{"1min":0.31900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.9029999999999996,"5min":40.889000000000003,"15min":40.889000000000003},"last":2.6520000000000001},{"interface":"front","average":{"1min":2.6829999999999998,"5min":3.2730000000000001,"15min":2.319},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":8.1769999999999996,"5min":16.989999999999998,"15min":18.265000000000001},"last":5.3979999999999997}]},{"osd":7,"last update":"Mon Feb 12 17:15:11 2024","interfaces":[{"interface":"back","average":{"1min":3.7810000000000001,"5min":3.5169999999999999,"15min":2.9940000000000002},"min":{"1min":0.378,"5min":0.375,"15min":0.315},"max":{"1min":8.6690000000000005,"5min":72.712999999999994,"15min":72.712999999999994},"last":5.3179999999999996},{"interface":"front","average":{"1min":3.298,"5min":2.9420000000000002,"15min":2.778},"min":{"1min":0.439,"5min":0.38600000000000001,"15min":0.28299999999999997},"max":{"1min":8.2170000000000005,"5min":17.015000000000001,"15min":38.948},"last":5.4299999999999997}]}]},{"osd":4,"up_from":0,"seq":137438953802,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757574,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2414392,"kb_used_data":924976,"kb_used_omap":0,"kb_used_meta":1489408,"kb_avail":91330760,"statfs":{"total":95995035648,"available":93522698240,"internally_reserved":0,"allocated":947175424,"data_stored":945834719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1525153792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.696,"15min":1.7629999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":3.927,"5min":8.0350000000000001,"15min":46.365000000000002},"last":5.1040000000000001},{"interface":"front","average":{"1min":2.133,"5min":1.659,"15min":1.9350000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.20100000000000001},"max":{"1min":7.157,"5min":15.368,"15min":46.134},"last":4.1269999999999998}]},{"osd":1,"last update":"Mon Feb 12 17:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":9.484,"5min":9.484,"15min":9.484},"last":5.1909999999999998},{"interface":"front","average":{"1min":3.0649999999999999,"5min":3.0649999999999999,"15min":3.0649999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.6319999999999997,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.88100000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":2.6000000000000001,"15min":2.036},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":9.0269999999999992,"5min":57.390999999999998,"15min":57.390999999999998},"last":13.978},{"interface":"front","average":{"1min":2.7719999999999998,"5min":2.843,"15min":2.0219999999999998},"min":{"1min":0.32100000000000001,"5min":0.214,"15min":0.214},"max":{"1min":13.366,"5min":57.134999999999998,"15min":57.134999999999998},"last":3.9980000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":2.0030000000000001,"5min":3.996,"15min":2.6920000000000002},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":7.1630000000000003,"5min":57.170999999999999,"15min":57.170999999999999},"last":9.2100000000000009},{"interface":"front","average":{"1min":3.3530000000000002,"5min":3.4420000000000002,"15min":2.5579999999999998},"min":{"1min":0.27900000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.3829999999999991,"5min":57.484999999999999,"15min":57.484999999999999},"last":2.7349999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.169,"15min":1.556},"min":{"1min":0.29799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.9939999999999998,"5min":9.0429999999999993,"15min":12.656000000000001},"last":2.7440000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.4370000000000001,"15min":1.573},"min":{"1min":0.255,"5min":0.255,"15min":0.254},"max":{"1min":3.3250000000000002,"5min":10.619999999999999,"15min":20.550000000000001},"last":5.1390000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:15:03 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.194,"15min":1.613},"min":{"1min":0.28899999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.4109999999999996,"5min":14.675000000000001,"15min":14.675000000000001},"last":2.77},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.6399999999999999,"15min":1.7769999999999999},"min":{"1min":0.28299999999999997,"5min":0.245,"15min":0.245},"max":{"1min":3.5979999999999999,"5min":12.864000000000001,"15min":14.323},"last":5.1660000000000004}]},{"osd":7,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.5669999999999999,"15min":1.8180000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":5.8289999999999997,"5min":14.337999999999999,"15min":19.184999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.5449999999999999,"15min":1.8819999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":6.6950000000000003,"5min":14.637,"15min":20.274000000000001},"last":2.7949999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282722304,"data_stored":282722304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286720000,"data_stored":286720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1274286080,"data_stored":1274282008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1225981952,"data_stored":1225981952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1085542400,"data_stored":1085538328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":933888000,"data_stored":933888000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1442648064,"data_stored":1442643992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1367015424,"data_stored":1367015424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:15:49.511 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T17:15:49.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:49.614 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:49.644 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:49.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:49.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:49.848 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:49.897 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:49.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:50.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:50.022 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:15:50.022 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:15:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:50.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:50.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:15:50.212 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:50.247 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:15:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:50.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:15:50.351 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:50.422 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:49.099936+0000 mgr.y (mgr.24481) 984 : cluster [DBG] pgmap v1401: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 734 op/s; 15886/115008 objects degraded (13.813%); 15440/115008 objects misplaced (13.425%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:15:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:49.099936+0000 mgr.y (mgr.24481) 984 : cluster [DBG] pgmap v1401: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 734 op/s; 15886/115008 objects degraded (13.813%); 15440/115008 objects misplaced (13.425%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:15:50.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:49.099936+0000 mgr.y (mgr.24481) 984 : cluster [DBG] pgmap v1401: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 734 op/s; 15886/115008 objects degraded (13.813%); 15440/115008 objects misplaced (13.425%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T17:15:51.207 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:51.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:51.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:51.213 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:15:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:50] "GET /metrics HTTP/1.1" 200 176844 "" "Prometheus/2.33.4" 2024-02-12T17:15:51.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:51.353 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:51.386 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:51.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:51.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:51.526 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:51.551 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:51.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:51.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:51.726 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:52.446 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:52.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:52.493 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T17:15:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:52.553 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:52.555 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:52.629 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:52.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:15:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:15:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:15:52.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:51.100626+0000 mgr.y (mgr.24481) 985 : cluster [DBG] pgmap v1402: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 15886/115008 objects degraded (13.813%); 15440/115008 objects misplaced (13.425%); 1.9 MiB/s, 31 objects/s recovering 2024-02-12T17:15:52.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:51.459747+0000 osd.2 (osd.2) 233 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:52.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:51.459807+0000 osd.2 (osd.2) 234 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:52.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:51.754437+0000 mon.a (mon.0) 2224 : cluster [WRN] Health check update: Degraded data redundancy: 15886/115008 objects degraded (13.813%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:52.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:51.100626+0000 mgr.y (mgr.24481) 985 : cluster [DBG] pgmap v1402: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 15886/115008 objects degraded (13.813%); 15440/115008 objects misplaced (13.425%); 1.9 MiB/s, 31 objects/s recovering 2024-02-12T17:15:52.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:51.459747+0000 osd.2 (osd.2) 233 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:52.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:51.459807+0000 osd.2 (osd.2) 234 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:52.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:51.754437+0000 mon.a (mon.0) 2224 : cluster [WRN] Health check update: Degraded data redundancy: 15886/115008 objects degraded (13.813%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:52.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:51.100626+0000 mgr.y (mgr.24481) 985 : cluster [DBG] pgmap v1402: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 15886/115008 objects degraded (13.813%); 15440/115008 objects misplaced (13.425%); 1.9 MiB/s, 31 objects/s recovering 2024-02-12T17:15:52.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:51.459747+0000 osd.2 (osd.2) 233 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:52.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:51.459807+0000 osd.2 (osd.2) 234 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:52.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:51.754437+0000 mon.a (mon.0) 2224 : cluster [WRN] Health check update: Degraded data redundancy: 15886/115008 objects degraded (13.813%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:53.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:15:53.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:52.235325+0000 osd.6 (osd.6) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:53.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:52.235372+0000 osd.6 (osd.6) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:53.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:52.363509+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2024-02-12T17:15:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:52.235325+0000 osd.6 (osd.6) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:52.235372+0000 osd.6 (osd.6) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:53.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:52.363509+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2024-02-12T17:15:53.607 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:53.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:53.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:15:53.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:52.235325+0000 osd.6 (osd.6) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:53.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:52.235372+0000 osd.6 (osd.6) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:53.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:52.363509+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2024-02-12T17:15:53.800 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:54.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:53.106756+0000 mgr.y (mgr.24481) 986 : cluster [DBG] pgmap v1404: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 571 op/s; 15260/111396 objects degraded (13.699%); 15220/111396 objects misplaced (13.663%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:54.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:53.524940+0000 osd.4 (osd.4) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:54.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:53.524994+0000 osd.4 (osd.4) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:54.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:53.106756+0000 mgr.y (mgr.24481) 986 : cluster [DBG] pgmap v1404: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 571 op/s; 15260/111396 objects degraded (13.699%); 15220/111396 objects misplaced (13.663%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:54.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:53.524940+0000 osd.4 (osd.4) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:54.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:53.524994+0000 osd.4 (osd.4) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:54.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:53.106756+0000 mgr.y (mgr.24481) 986 : cluster [DBG] pgmap v1404: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 571 op/s; 15260/111396 objects degraded (13.699%); 15220/111396 objects misplaced (13.663%); 2.8 MiB/s, 45 objects/s recovering 2024-02-12T17:15:54.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:53.524940+0000 osd.4 (osd.4) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:15:54.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:53.524994+0000 osd.4 (osd.4) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:15:54.785 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:15:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:54.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:15:54.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:15:54.933 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:54.972 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:15:54.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:54.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:15:55.076 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:55.168 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:55.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:15:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:15:55.452 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:54.432903+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2024-02-12T17:15:55.452 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:54.871926+0000 mon.c (mon.2) 312 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:15:55.453 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:54.432903+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2024-02-12T17:15:55.453 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:54.871926+0000 mon.c (mon.2) 312 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:15:55.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:54.432903+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2024-02-12T17:15:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:54.871926+0000 mon.c (mon.2) 312 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:15:56.473 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:55.088952+0000 mgr.y (mgr.24481) 987 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:56.473 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:55.109155+0000 mgr.y (mgr.24481) 988 : cluster [DBG] pgmap v1406: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 648 op/s; 14365/105510 objects degraded (13.615%); 14002/105510 objects misplaced (13.271%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:56.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:55.088952+0000 mgr.y (mgr.24481) 987 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:56.474 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:55.109155+0000 mgr.y (mgr.24481) 988 : cluster [DBG] pgmap v1406: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 648 op/s; 14365/105510 objects degraded (13.615%); 14002/105510 objects misplaced (13.271%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:56.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:55.088952+0000 mgr.y (mgr.24481) 987 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:15:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:55.109155+0000 mgr.y (mgr.24481) 988 : cluster [DBG] pgmap v1406: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 648 op/s; 14365/105510 objects degraded (13.615%); 14002/105510 objects misplaced (13.271%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T17:15:57.190 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:15:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:57.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:15:57.231 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:57.298 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:57.397 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:57.498 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:56.757282+0000 mon.a (mon.0) 2227 : cluster [WRN] Health check update: Degraded data redundancy: 14365/105510 objects degraded (13.615%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:57.498 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:57.368888+0000 mon.a (mon.0) 2228 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:57.522 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:57.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:57.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:57.676 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:57.702 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:57.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:57.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:56.757282+0000 mon.a (mon.0) 2227 : cluster [WRN] Health check update: Degraded data redundancy: 14365/105510 objects degraded (13.615%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:57.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:57.368888+0000 mon.a (mon.0) 2228 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:57.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:15:57.879 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:57.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:56.757282+0000 mon.a (mon.0) 2227 : cluster [WRN] Health check update: Degraded data redundancy: 14365/105510 objects degraded (13.615%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:15:57.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:57.368888+0000 mon.a (mon.0) 2228 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:57.997 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:15:57.997 INFO:teuthology.orchestra.run.smithi033.stdout:{"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13760/101979 objects degraded (13.493%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1790,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":696,"num_osds":8,"num_up_osds":8,"osd_up_since":1707758025,"num_in_osds":7,"osd_in_since":1707758000,"num_remapped_pgs":18},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":100},{"state_name":"active+remapped+backfill_wait","count":8},{"state_name":"active+undersized+degraded+remapped+backfilling","count":4},{"state_name":"active+remapped+backfill_wait+backfill_toofull","count":2},{"state_name":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","count":2},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_wait","count":1}],"num_pgs":118,"num_pools":7,"num_objects":33993,"data_bytes":2215122620,"bytes_used":13186572288,"bytes_avail":658778677248,"bytes_total":671965249536,"degraded_objects":13760,"degraded_total":101979,"degraded_ratio":0.1349297404367566,"misplaced_objects":13811,"misplaced_total":101979,"misplaced_ratio":0.13542984339913119,"recovering_objects_per_sec":43,"recovering_bytes_per_sec":2879920,"recovering_keys_per_sec":0,"num_objects_recovered":352,"num_bytes_recovered":23068672,"num_keys_recovered":0,"read_bytes_sec":1278,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":542},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.33:8443/","prometheus":"http://172.21.15.33:9283/"}},"servicemap":{"epoch":15,"modified":"2024-02-12T17:14:45.050033+0000","services":{"rgw":{"daemons":{"summary":"","24397":{"start_epoch":12,"start_stamp":"2024-02-12T16:49:46.308902+0000","gid":24397,"addr":"172.21.15.33:0/1589842856","metadata":{"arch":"x86_64","ceph_release":"pacific","ceph_version":"ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)","ceph_version_short":"16.2.14-521-g0e714d9a","container_hostname":"smithi033","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi033","id":"foo.a","kernel_description":"#1 SMP Thu Jan 18 15:15:29 UTC 2024","kernel_version":"4.18.0-536.el8.x86_64","mem_swap_kb":"0","mem_total_kb":"32543856","num_handles":"1","os":"Linux","pid":"8","realm_id":"","realm_name":"","zone_id":"4799771a-d28e-4dc8-a5a0-179e1408b6e6","zone_name":"default","zonegroup_id":"0384a922-9dfb-46b6-aed7-e430c5071cb8","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{"627379c4-3690-4a16-a102-e69cb00adc70":{"message":"Global Recovery Event (7m)\n [=======================.....] (remaining: 82s)","progress":0.8474576473236084,"add_to_ceph_s":true}}} 2024-02-12T17:15:58.073 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:15:58.073 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:15:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:58.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:15:58.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:58.225 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:58.353 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:58.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:58.464 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:58.505 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:58.525 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:58.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:58.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:15:58.671 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:57.109911+0000 mgr.y (mgr.24481) 989 : cluster [DBG] pgmap v1407: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 543 op/s; 13760/101979 objects degraded (13.493%); 13811/101979 objects misplaced (13.543%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:15:58.678 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:57.979525+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? v1:172.21.15.33:0/2746758224' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:15:58.678 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:57.109911+0000 mgr.y (mgr.24481) 989 : cluster [DBG] pgmap v1407: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 543 op/s; 13760/101979 objects degraded (13.493%); 13811/101979 objects misplaced (13.543%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:15:58.678 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:57.979525+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? v1:172.21.15.33:0/2746758224' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:15:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:15:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:58.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:15:58.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:15:58.838 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:57.109911+0000 mgr.y (mgr.24481) 989 : cluster [DBG] pgmap v1407: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 543 op/s; 13760/101979 objects degraded (13.493%); 13811/101979 objects misplaced (13.543%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:15:58.838 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:57.979525+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? v1:172.21.15.33:0/2746758224' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:15:58.844 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:58.879 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:15:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:15:58.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:15:58.983 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:15:59.105 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:15:59.219 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:15:59.241 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:15:59.309 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:15:59.358 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:15:59.407 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:15:59.478 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:15:59.486 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:15:59.535 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:15:59.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:15:59.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:15:59.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:15:59.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:15:59.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:15:59.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:15:59.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:15:59.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:15:59.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:58.818867+0000 mon.c (mon.2) 313 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:15:59.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:58.823505+0000 mon.c (mon.2) 314 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:15:59.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:58.831041+0000 mon.a (mon.0) 2229 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:59.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:15:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:15:59.117931+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.33:0/2725106500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:59.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:58.818867+0000 mon.c (mon.2) 313 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:15:59.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:58.823505+0000 mon.c (mon.2) 314 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:15:59.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:58.831041+0000 mon.a (mon.0) 2229 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:59.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:15:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:15:59.117931+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.33:0/2725106500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:15:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:58.818867+0000 mon.c (mon.2) 313 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:15:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:58.823505+0000 mon.c (mon.2) 314 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:15:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:58.831041+0000 mon.a (mon.0) 2229 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:15:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:15:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:15:59.117931+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.33:0/2725106500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:00.883 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:15:59.111124+0000 mgr.y (mgr.24481) 990 : cluster [DBG] pgmap v1408: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 796 op/s; 12823/95877 objects degraded (13.374%); 12612/95877 objects misplaced (13.154%); 3.1 MiB/s, 50 objects/s recovering 2024-02-12T17:16:00.883 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:15:59.111124+0000 mgr.y (mgr.24481) 990 : cluster [DBG] pgmap v1408: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 796 op/s; 12823/95877 objects degraded (13.374%); 12612/95877 objects misplaced (13.154%); 3.1 MiB/s, 50 objects/s recovering 2024-02-12T17:16:00.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:15:59.111124+0000 mgr.y (mgr.24481) 990 : cluster [DBG] pgmap v1408: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 796 op/s; 12823/95877 objects degraded (13.374%); 12612/95877 objects misplaced (13.154%); 3.1 MiB/s, 50 objects/s recovering 2024-02-12T17:16:01.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:16:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:00] "GET /metrics HTTP/1.1" 200 176844 "" "Prometheus/2.33.4" 2024-02-12T17:16:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:01.112404+0000 mgr.y (mgr.24481) 991 : cluster [DBG] pgmap v1409: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 593 op/s; 12823/95877 objects degraded (13.374%); 12612/95877 objects misplaced (13.154%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T17:16:02.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:01.758507+0000 mon.a (mon.0) 2230 : cluster [WRN] Health check update: Degraded data redundancy: 12823/95877 objects degraded (13.374%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:02.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:01.112404+0000 mgr.y (mgr.24481) 991 : cluster [DBG] pgmap v1409: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 593 op/s; 12823/95877 objects degraded (13.374%); 12612/95877 objects misplaced (13.154%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T17:16:02.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:01.758507+0000 mon.a (mon.0) 2230 : cluster [WRN] Health check update: Degraded data redundancy: 12823/95877 objects degraded (13.374%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:16:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:16:02.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:01.112404+0000 mgr.y (mgr.24481) 991 : cluster [DBG] pgmap v1409: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 593 op/s; 12823/95877 objects degraded (13.374%); 12612/95877 objects misplaced (13.154%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T17:16:02.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:01.758507+0000 mon.a (mon.0) 2230 : cluster [WRN] Health check update: Degraded data redundancy: 12823/95877 objects degraded (13.374%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:03.176 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-02-12T17:16:03.176 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:16:03.181 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:03.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:03.288 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:03.362 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:03.932 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:03.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:04.043 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:16:04.105 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:04.133 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:16:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:04.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:16:04.239 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:04.319 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:04.507 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:04.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:04.613 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:04.756 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:04.886 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:04.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:03.113970+0000 mgr.y (mgr.24481) 992 : cluster [DBG] pgmap v1410: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 637 op/s; 12235/92292 objects degraded (13.257%); 12393/92292 objects misplaced (13.428%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:04.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:03.116649+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:04.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:03.116721+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:04.888 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:03.591961+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2024-02-12T17:16:04.888 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:03.113970+0000 mgr.y (mgr.24481) 992 : cluster [DBG] pgmap v1410: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 637 op/s; 12235/92292 objects degraded (13.257%); 12393/92292 objects misplaced (13.428%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:04.888 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:03.116649+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:04.889 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:03.116721+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:04.889 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:03.591961+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2024-02-12T17:16:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:04.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:04.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:03.113970+0000 mgr.y (mgr.24481) 992 : cluster [DBG] pgmap v1410: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 637 op/s; 12235/92292 objects degraded (13.257%); 12393/92292 objects misplaced (13.428%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:04.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:03.116649+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:03.116721+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:03.591961+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2024-02-12T17:16:04.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:05.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:05.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:05.178 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:05.192 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:05.214 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:05.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:05.235 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:16:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:16:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.36023.0:224241 11.b e697/651 11:d491e06e:::benchmark_data_smithi033_188107_object98551:head v 697'14077, mlcod=697'14075)", 2024-02-12T17:16:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:16:05.233404+0000", 2024-02-12T17:16:05.235 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.000151266, 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000185334, 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233404+0000", 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:16:05.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233404+0000", 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233402+0000", 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4049000000000001e-05 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233416+0000", 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0322e-05 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233436+0000", 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6999999999999997e-06 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233445+0000", 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9621000000000001e-05 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233534+0000", 2024-02-12T17:16:05.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3224999999999999e-05 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233557+0000", 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015319199999999999 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.36023.0:224237 11.9 e697/651)", 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:16:05.233459+0000", 2024-02-12T17:16:05.239 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.645e-05, 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020566000000000001, 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233459+0000", 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233459+0000", 2024-02-12T17:16:05.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999995 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233459+0000", 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8749999999999999e-06 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233463+0000", 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8760000000000004e-06 2024-02-12T17:16:05.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233468+0000", 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7979999999999999e-06 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233472+0000", 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3078000000000001e-05 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:05.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.36023.0:224244 11.1 e697/652)", 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:16:05.233552+0000", 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.4709999999999999e-06, 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016500799999999999, 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "queued for pg", 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233552+0000", 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:16:05.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233552+0000", 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233550+0000", 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4175999999999999e-05 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:16:05.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233564+0000", 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.13e-05 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233576+0000", 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1199999999999999e-06 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:05.233582+0000", 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9609999999999999e-05 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:16:05.245 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:16:05.246 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 3 2024-02-12T17:16:05.246 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:16:05.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:05.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:16:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:16:05.355 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:05.380 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:05.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:05.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:05.594 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:05.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:04.594140+0000 osd.5 (osd.5) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:05.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:04.594216+0000 osd.5 (osd.5) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:05.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:04.979058+0000 osd.0 (osd.0) 236 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:05.655 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:04.979102+0000 osd.0 (osd.0) 237 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:05.656 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:05.060246+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.33:0/2862164597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:05.656 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:05.061214+0000 mon.a (mon.0) 2232 : audit [INF] from='client.26101 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:05.674 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:16:05.681 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:05.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:05.787 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:05.816 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:05.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:05.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:06.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:04.594140+0000 osd.5 (osd.5) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:06.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:04.594216+0000 osd.5 (osd.5) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:06.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:04.979058+0000 osd.0 (osd.0) 236 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:06.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:04.979102+0000 osd.0 (osd.0) 237 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:06.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:05.060246+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.33:0/2862164597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:06.019 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:05.061214+0000 mon.a (mon.0) 2232 : audit [INF] from='client.26101 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:06.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:04.594140+0000 osd.5 (osd.5) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:06.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:04.594216+0000 osd.5 (osd.5) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:06.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:04.979058+0000 osd.0 (osd.0) 236 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:06.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:04.979102+0000 osd.0 (osd.0) 237 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:06.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:05.060246+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.33:0/2862164597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:06.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:05.061214+0000 mon.a (mon.0) 2232 : audit [INF] from='client.26101 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:06.025 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:06.068 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:06.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:06.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:16:06.262 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:06.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:16:06.724 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:16:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:06.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:16:06.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:06.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:05.099707+0000 mgr.y (mgr.24481) 993 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:06.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:05.114774+0000 mgr.y (mgr.24481) 994 : cluster [DBG] pgmap v1412: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 647 op/s; 11278/86094 objects degraded (13.100%); 11139/86094 objects misplaced (12.938%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:06.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:05.596926+0000 mon.a (mon.0) 2233 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:16:06.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:05.647584+0000 mon.a (mon.0) 2234 : audit [INF] from='client.26101 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:16:06.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:05.647637+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2024-02-12T17:16:06.916 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:05.099707+0000 mgr.y (mgr.24481) 993 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:06.916 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:05.114774+0000 mgr.y (mgr.24481) 994 : cluster [DBG] pgmap v1412: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 647 op/s; 11278/86094 objects degraded (13.100%); 11139/86094 objects misplaced (12.938%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:06.916 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:05.596926+0000 mon.a (mon.0) 2233 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:16:06.916 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:05.647584+0000 mon.a (mon.0) 2234 : audit [INF] from='client.26101 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:16:06.916 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:05.647637+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2024-02-12T17:16:06.917 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:06.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:05.099707+0000 mgr.y (mgr.24481) 993 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:05.114774+0000 mgr.y (mgr.24481) 994 : cluster [DBG] pgmap v1412: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 647 op/s; 11278/86094 objects degraded (13.100%); 11139/86094 objects misplaced (12.938%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:05.596926+0000 mon.a (mon.0) 2233 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:16:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:05.647584+0000 mon.a (mon.0) 2234 : audit [INF] from='client.26101 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:16:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:05.647637+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2024-02-12T17:16:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:06.759927+0000 mon.a (mon.0) 2236 : cluster [WRN] Health check update: Degraded data redundancy: 11278/86094 objects degraded (13.100%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:08.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:06.759927+0000 mon.a (mon.0) 2236 : cluster [WRN] Health check update: Degraded data redundancy: 11278/86094 objects degraded (13.100%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:08.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:06.759927+0000 mon.a (mon.0) 2236 : cluster [WRN] Health check update: Degraded data redundancy: 11278/86094 objects degraded (13.100%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:08.714 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:08.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:08.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:08.863 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:08.901 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:08.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:08.993 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:07.115431+0000 mgr.y (mgr.24481) 995 : cluster [DBG] pgmap v1414: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 556 op/s; 10727/82554 objects degraded (12.994%); 10953/82554 objects misplaced (13.268%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:09.007 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:07.115431+0000 mgr.y (mgr.24481) 995 : cluster [DBG] pgmap v1414: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 556 op/s; 10727/82554 objects degraded (12.994%); 10953/82554 objects misplaced (13.268%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:09.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:07.115431+0000 mgr.y (mgr.24481) 995 : cluster [DBG] pgmap v1414: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 556 op/s; 10727/82554 objects degraded (12.994%); 10953/82554 objects misplaced (13.268%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:09.052 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:09.081 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:09.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:09.185 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T17:16:09.186 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:09.268 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:09.655 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:09.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:09.762 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:09.814 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:09.841 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:09.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:09.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:10.033 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:10.187 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:10.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:10.295 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:10.322 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:10.348 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:10.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:10.408 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:10.431 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:10.452 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:10.503 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:10.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:10.607 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:10.714 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:16:10.733 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:10.741 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:16:10.898 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:10.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:10.975 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:09.116195+0000 mgr.y (mgr.24481) 996 : cluster [DBG] pgmap v1415: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 798 op/s; 9814/76719 objects degraded (12.792%); 9815/76719 objects misplaced (12.793%); 3.2 MiB/s, 50 objects/s recovering 2024-02-12T17:16:10.975 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:09.116195+0000 mgr.y (mgr.24481) 996 : cluster [DBG] pgmap v1415: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 798 op/s; 9814/76719 objects degraded (12.792%); 9815/76719 objects misplaced (12.793%); 3.2 MiB/s, 50 objects/s recovering 2024-02-12T17:16:10.977 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:16:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:10] "GET /metrics HTTP/1.1" 200 176858 "" "Prometheus/2.33.4" 2024-02-12T17:16:10.995 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:09.116195+0000 mgr.y (mgr.24481) 996 : cluster [DBG] pgmap v1415: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 798 op/s; 9814/76719 objects degraded (12.792%); 9815/76719 objects misplaced (12.793%); 3.2 MiB/s, 50 objects/s recovering 2024-02-12T17:16:11.006 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:11.085 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:11.153 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:11.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:11.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:11.294 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:11.320 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:11.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:11.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:11.454 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:11.481 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:11.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:11.587 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:16:11.704 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:10.692699+0000 mgr.y (mgr.24481) 997 : audit [DBG] from='client.26107 v1:172.21.15.33:0/2292703230' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:12.031 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T17:16:12.032 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:10.692699+0000 mgr.y (mgr.24481) 997 : audit [DBG] from='client.26107 v1:172.21.15.33:0/2292703230' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:12.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:10.692699+0000 mgr.y (mgr.24481) 997 : audit [DBG] from='client.26107 v1:172.21.15.33:0/2292703230' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:12.875 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:16:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:12.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:16:12.901 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1415,"stamp":"2024-02-12T17:16:09.115590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1663309500,"num_objects":25573,"num_object_clones":0,"num_object_copies":76719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9814,"num_objects_misplaced":9815,"num_objects_unfound":0,"num_objects_dirty":25573,"num_whiteouts":0,"num_read":1884,"num_read_kb":1636,"num_write":226995,"num_write_kb":8044010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22348,"num_bytes_recovered":1409309472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92557,"ondisk_log_size":92557,"up":354,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":368,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11686536,"kb_used_data":4190352,"kb_used_omap":3,"kb_used_meta":7496060,"kb_avail":644529528,"statfs":{"total":671965249536,"available":659998236672,"internally_reserved":0,"allocated":4290920448,"data_stored":4281140814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":7675965501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,9,5,6,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-418512896,"num_objects":-6386,"num_object_clones":0,"num_object_copies":-19158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3009,"num_objects_misplaced":-2797,"num_objects_unfound":0,"num_objects_dirty":-6386,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":26673152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003726"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":423,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.615654+0000","last_change":"2024-02-12T17:14:10.348720+0000","last_active":"2024-02-12T17:16:03.615654+0000","last_peered":"2024-02-12T17:16:03.615654+0000","last_clean":"2024-02-12T17:16:03.615654+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:16:03.615654+0000","last_undegraded":"2024-02-12T17:16:03.615654+0000","last_fullsized":"2024-02-12T17:16:03.615654+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":206,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.612572+0000","last_change":"2024-02-12T17:13:46.694822+0000","last_active":"2024-02-12T17:16:03.612572+0000","last_peered":"2024-02-12T17:16:03.612572+0000","last_clean":"2024-02-12T17:16:03.612572+0000","last_became_active":"2024-02-12T17:13:46.694448+0000","last_became_peered":"2024-02-12T17:13:46.694448+0000","last_unstale":"2024-02-12T17:16:03.612572+0000","last_undegraded":"2024-02-12T17:16:03.612572+0000","last_fullsized":"2024-02-12T17:16:03.612572+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:12:55.862345+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:12:55.862345+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":450,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613568+0000","last_change":"2024-02-12T17:13:46.787428+0000","last_active":"2024-02-12T17:16:03.613568+0000","last_peered":"2024-02-12T17:16:03.613568+0000","last_clean":"2024-02-12T17:16:03.613568+0000","last_became_active":"2024-02-12T17:13:46.786996+0000","last_became_peered":"2024-02-12T17:13:46.786996+0000","last_unstale":"2024-02-12T17:16:03.613568+0000","last_undegraded":"2024-02-12T17:16:03.613568+0000","last_fullsized":"2024-02-12T17:16:03.613568+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:13:12.016227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:12.016227+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":478,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.614105+0000","last_change":"2024-02-12T17:14:09.685503+0000","last_active":"2024-02-12T17:16:03.614105+0000","last_peered":"2024-02-12T17:16:03.614105+0000","last_clean":"2024-02-12T17:16:03.614105+0000","last_became_active":"2024-02-12T17:14:08.761132+0000","last_became_peered":"2024-02-12T17:14:08.761132+0000","last_unstale":"2024-02-12T17:16:03.614105+0000","last_undegraded":"2024-02-12T17:16:03.614105+0000","last_fullsized":"2024-02-12T17:16:03.614105+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:54.052006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.052006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":493,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.615149+0000","last_change":"2024-02-12T17:13:46.753228+0000","last_active":"2024-02-12T17:16:03.615149+0000","last_peered":"2024-02-12T17:16:03.615149+0000","last_clean":"2024-02-12T17:16:03.615149+0000","last_became_active":"2024-02-12T17:13:46.752755+0000","last_became_peered":"2024-02-12T17:13:46.752755+0000","last_unstale":"2024-02-12T17:16:03.615149+0000","last_undegraded":"2024-02-12T17:16:03.615149+0000","last_fullsized":"2024-02-12T17:16:03.615149+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":1540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":157,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.440045+0000","last_change":"2024-02-12T17:13:46.607814+0000","last_active":"2024-02-12T17:15:54.440045+0000","last_peered":"2024-02-12T17:15:54.440045+0000","last_clean":"2024-02-12T17:15:54.440045+0000","last_became_active":"2024-02-12T17:13:46.606367+0000","last_became_peered":"2024-02-12T17:13:46.606367+0000","last_unstale":"2024-02-12T17:15:54.440045+0000","last_undegraded":"2024-02-12T17:15:54.440045+0000","last_fullsized":"2024-02-12T17:15:54.440045+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":233,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613557+0000","last_change":"2024-02-12T17:13:46.567244+0000","last_active":"2024-02-12T17:16:03.613557+0000","last_peered":"2024-02-12T17:16:03.613557+0000","last_clean":"2024-02-12T17:16:03.613557+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:16:03.613557+0000","last_undegraded":"2024-02-12T17:16:03.613557+0000","last_fullsized":"2024-02-12T17:16:03.613557+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":67,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.443627+0000","last_change":"2024-02-12T17:14:08.993489+0000","last_active":"2024-02-12T17:15:54.443627+0000","last_peered":"2024-02-12T17:15:54.443627+0000","last_clean":"2024-02-12T17:15:54.443627+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:15:54.443627+0000","last_undegraded":"2024-02-12T17:15:54.443627+0000","last_fullsized":"2024-02-12T17:15:54.443627+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":421,"reported_epoch":697,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:03.611055+0000","last_change":"2024-02-12T17:14:09.723827+0000","last_active":"2024-02-12T17:16:03.611055+0000","last_peered":"2024-02-12T17:16:03.611055+0000","last_clean":"2024-02-12T17:14:05.085957+0000","last_became_active":"2024-02-12T17:14:09.723480+0000","last_became_peered":"2024-02-12T17:14:09.723480+0000","last_unstale":"2024-02-12T17:16:03.611055+0000","last_undegraded":"2024-02-12T17:16:03.611055+0000","last_fullsized":"2024-02-12T17:16:03.611055+0000","mapping_epoch":652,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":391,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613681+0000","last_change":"2024-02-12T17:14:08.755098+0000","last_active":"2024-02-12T17:16:03.613681+0000","last_peered":"2024-02-12T17:16:03.613681+0000","last_clean":"2024-02-12T17:16:03.613681+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:16:03.613681+0000","last_undegraded":"2024-02-12T17:16:03.613681+0000","last_fullsized":"2024-02-12T17:16:03.613681+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":414,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.615956+0000","last_change":"2024-02-12T17:12:12.141297+0000","last_active":"2024-02-12T17:16:03.615956+0000","last_peered":"2024-02-12T17:16:03.615956+0000","last_clean":"2024-02-12T17:16:03.615956+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:16:03.615956+0000","last_undegraded":"2024-02-12T17:16:03.615956+0000","last_fullsized":"2024-02-12T17:16:03.615956+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.141204+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":243,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613275+0000","last_change":"2024-02-12T17:14:08.743209+0000","last_active":"2024-02-12T17:16:03.613275+0000","last_peered":"2024-02-12T17:16:03.613275+0000","last_clean":"2024-02-12T17:16:03.613275+0000","last_became_active":"2024-02-12T17:14:08.742886+0000","last_became_peered":"2024-02-12T17:14:08.742886+0000","last_unstale":"2024-02-12T17:16:03.613275+0000","last_undegraded":"2024-02-12T17:16:03.613275+0000","last_fullsized":"2024-02-12T17:16:03.613275+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.783527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.783527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":692,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.611019+0000","last_change":"2024-02-12T17:13:21.591627+0000","last_active":"2024-02-12T17:16:03.611019+0000","last_peered":"2024-02-12T17:16:03.611019+0000","last_clean":"2024-02-12T17:16:03.611019+0000","last_became_active":"2024-02-12T17:13:21.591246+0000","last_became_peered":"2024-02-12T17:13:21.591246+0000","last_unstale":"2024-02-12T17:16:03.611019+0000","last_undegraded":"2024-02-12T17:16:03.611019+0000","last_fullsized":"2024-02-12T17:16:03.611019+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.028495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":436,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613402+0000","last_change":"2024-02-12T17:13:46.558817+0000","last_active":"2024-02-12T17:16:03.613402+0000","last_peered":"2024-02-12T17:16:03.613402+0000","last_clean":"2024-02-12T17:16:03.613402+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:16:03.613402+0000","last_undegraded":"2024-02-12T17:16:03.613402+0000","last_fullsized":"2024-02-12T17:16:03.613402+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":89,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.438075+0000","last_change":"2024-02-12T17:13:46.576676+0000","last_active":"2024-02-12T17:15:54.438075+0000","last_peered":"2024-02-12T17:15:54.438075+0000","last_clean":"2024-02-12T17:15:54.438075+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:15:54.438075+0000","last_undegraded":"2024-02-12T17:15:54.438075+0000","last_fullsized":"2024-02-12T17:15:54.438075+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":373,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.615791+0000","last_change":"2024-02-12T17:14:08.763317+0000","last_active":"2024-02-12T17:16:03.615791+0000","last_peered":"2024-02-12T17:16:03.615791+0000","last_clean":"2024-02-12T17:16:03.615791+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:16:03.615791+0000","last_undegraded":"2024-02-12T17:16:03.615791+0000","last_fullsized":"2024-02-12T17:16:03.615791+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":129,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.439101+0000","last_change":"2024-02-12T17:14:08.742170+0000","last_active":"2024-02-12T17:15:54.439101+0000","last_peered":"2024-02-12T17:15:54.439101+0000","last_clean":"2024-02-12T17:15:54.439101+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:15:54.439101+0000","last_undegraded":"2024-02-12T17:15:54.439101+0000","last_fullsized":"2024-02-12T17:15:54.439101+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":211,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613718+0000","last_change":"2024-02-12T17:13:46.784204+0000","last_active":"2024-02-12T17:16:03.613718+0000","last_peered":"2024-02-12T17:16:03.613718+0000","last_clean":"2024-02-12T17:16:03.613718+0000","last_became_active":"2024-02-12T17:13:46.783737+0000","last_became_peered":"2024-02-12T17:13:46.783737+0000","last_unstale":"2024-02-12T17:16:03.613718+0000","last_undegraded":"2024-02-12T17:16:03.613718+0000","last_fullsized":"2024-02-12T17:16:03.613718+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:15.817751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.817751+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":760,"reported_epoch":697,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:03.613085+0000","last_change":"2024-02-12T17:14:40.177468+0000","last_active":"2024-02-12T17:16:03.613085+0000","last_peered":"2024-02-12T17:16:03.613085+0000","last_clean":"2024-02-12T17:13:44.507959+0000","last_became_active":"2024-02-12T17:14:08.758564+0000","last_became_peered":"2024-02-12T17:14:08.758564+0000","last_unstale":"2024-02-12T17:16:03.613085+0000","last_undegraded":"2024-02-12T17:16:03.613085+0000","last_fullsized":"2024-02-12T17:16:03.613085+0000","mapping_epoch":651,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":55,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.442602+0000","last_change":"2024-02-12T17:14:08.762208+0000","last_active":"2024-02-12T17:15:54.442602+0000","last_peered":"2024-02-12T17:15:54.442602+0000","last_clean":"2024-02-12T17:15:54.442602+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:15:54.442602+0000","last_undegraded":"2024-02-12T17:15:54.442602+0000","last_fullsized":"2024-02-12T17:15:54.442602+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":201,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613349+0000","last_change":"2024-02-12T17:12:02.752352+0000","last_active":"2024-02-12T17:16:03.613349+0000","last_peered":"2024-02-12T17:16:03.613349+0000","last_clean":"2024-02-12T17:16:03.613349+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:16:03.613349+0000","last_undegraded":"2024-02-12T17:16:03.613349+0000","last_fullsized":"2024-02-12T17:16:03.613349+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:02.752280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:02.752280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":161,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.437948+0000","last_change":"2024-02-12T17:13:46.582722+0000","last_active":"2024-02-12T17:15:54.437948+0000","last_peered":"2024-02-12T17:15:54.437948+0000","last_clean":"2024-02-12T17:15:54.437948+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:15:54.437948+0000","last_undegraded":"2024-02-12T17:15:54.437948+0000","last_fullsized":"2024-02-12T17:15:54.437948+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":793,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.614453+0000","last_change":"2024-02-12T17:12:58.020311+0000","last_active":"2024-02-12T17:16:03.614453+0000","last_peered":"2024-02-12T17:16:03.614453+0000","last_clean":"2024-02-12T17:16:03.614453+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:16:03.614453+0000","last_undegraded":"2024-02-12T17:16:03.614453+0000","last_fullsized":"2024-02-12T17:16:03.614453+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:12:58.018712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.018712+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":55,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.451931+0000","last_change":"2024-02-12T17:14:08.872636+0000","last_active":"2024-02-12T17:15:54.451931+0000","last_peered":"2024-02-12T17:15:54.451931+0000","last_clean":"2024-02-12T17:15:54.451931+0000","last_became_active":"2024-02-12T17:14:08.872123+0000","last_became_peered":"2024-02-12T17:14:08.872123+0000","last_unstale":"2024-02-12T17:15:54.451931+0000","last_undegraded":"2024-02-12T17:15:54.451931+0000","last_fullsized":"2024-02-12T17:15:54.451931+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:06.904354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:06.904354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":788,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605913+0000","last_change":"2024-02-12T17:12:12.838640+0000","last_active":"2024-02-12T17:16:03.605913+0000","last_peered":"2024-02-12T17:16:03.605913+0000","last_clean":"2024-02-12T17:16:03.605913+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:16:03.605913+0000","last_undegraded":"2024-02-12T17:16:03.605913+0000","last_fullsized":"2024-02-12T17:16:03.605913+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:12.838563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.838563+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"655'24","reported_seq":791,"reported_epoch":698,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:05.658598+0000","last_change":"2024-02-12T17:14:09.729342+0000","last_active":"2024-02-12T17:16:05.658598+0000","last_peered":"2024-02-12T17:16:05.658598+0000","last_clean":"2024-02-12T17:14:07.616779+0000","last_became_active":"2024-02-12T17:14:09.728969+0000","last_became_peered":"2024-02-12T17:14:09.728969+0000","last_unstale":"2024-02-12T17:16:05.658598+0000","last_undegraded":"2024-02-12T17:16:05.658598+0000","last_fullsized":"2024-02-12T17:16:05.658598+0000","mapping_epoch":652,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":692,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605130+0000","last_change":"2024-02-12T17:12:54.656523+0000","last_active":"2024-02-12T17:16:03.605130+0000","last_peered":"2024-02-12T17:16:03.605130+0000","last_clean":"2024-02-12T17:16:03.605130+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:16:03.605130+0000","last_undegraded":"2024-02-12T17:16:03.605130+0000","last_fullsized":"2024-02-12T17:16:03.605130+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:54.656456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.656456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":427,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.614294+0000","last_change":"2024-02-12T17:13:46.577769+0000","last_active":"2024-02-12T17:16:03.614294+0000","last_peered":"2024-02-12T17:16:03.614294+0000","last_clean":"2024-02-12T17:16:03.614294+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:16:03.614294+0000","last_undegraded":"2024-02-12T17:16:03.614294+0000","last_fullsized":"2024-02-12T17:16:03.614294+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":714,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.611350+0000","last_change":"2024-02-12T17:13:21.591810+0000","last_active":"2024-02-12T17:16:03.611350+0000","last_peered":"2024-02-12T17:16:03.611350+0000","last_clean":"2024-02-12T17:16:03.611350+0000","last_became_active":"2024-02-12T17:13:21.591502+0000","last_became_peered":"2024-02-12T17:13:21.591502+0000","last_unstale":"2024-02-12T17:16:03.611350+0000","last_undegraded":"2024-02-12T17:16:03.611350+0000","last_fullsized":"2024-02-12T17:16:03.611350+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.882066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.882066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":235,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613100+0000","last_change":"2024-02-12T17:13:46.651066+0000","last_active":"2024-02-12T17:16:03.613100+0000","last_peered":"2024-02-12T17:16:03.613100+0000","last_clean":"2024-02-12T17:16:03.613100+0000","last_became_active":"2024-02-12T17:13:46.650776+0000","last_became_peered":"2024-02-12T17:13:46.650776+0000","last_unstale":"2024-02-12T17:16:03.613100+0000","last_undegraded":"2024-02-12T17:16:03.613100+0000","last_fullsized":"2024-02-12T17:16:03.613100+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:50.786900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.786900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":416,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.621539+0000","last_change":"2024-02-12T17:13:46.822111+0000","last_active":"2024-02-12T17:16:03.621539+0000","last_peered":"2024-02-12T17:16:03.621539+0000","last_clean":"2024-02-12T17:16:03.621539+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:16:03.621539+0000","last_undegraded":"2024-02-12T17:16:03.621539+0000","last_fullsized":"2024-02-12T17:16:03.621539+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":326,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.611417+0000","last_change":"2024-02-12T17:14:08.745944+0000","last_active":"2024-02-12T17:16:03.611417+0000","last_peered":"2024-02-12T17:16:03.611417+0000","last_clean":"2024-02-12T17:16:03.611417+0000","last_became_active":"2024-02-12T17:14:08.745729+0000","last_became_peered":"2024-02-12T17:14:08.745729+0000","last_unstale":"2024-02-12T17:16:03.611417+0000","last_undegraded":"2024-02-12T17:16:03.611417+0000","last_fullsized":"2024-02-12T17:16:03.611417+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:49.809844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:12:49.809844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":89,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.441300+0000","last_change":"2024-02-12T17:13:46.559519+0000","last_active":"2024-02-12T17:15:54.441300+0000","last_peered":"2024-02-12T17:15:54.441300+0000","last_clean":"2024-02-12T17:15:54.441300+0000","last_became_active":"2024-02-12T17:13:46.559095+0000","last_became_peered":"2024-02-12T17:13:46.559095+0000","last_unstale":"2024-02-12T17:15:54.441300+0000","last_undegraded":"2024-02-12T17:15:54.441300+0000","last_fullsized":"2024-02-12T17:15:54.441300+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":65,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.443711+0000","last_change":"2024-02-12T17:14:08.993503+0000","last_active":"2024-02-12T17:15:54.443711+0000","last_peered":"2024-02-12T17:15:54.443711+0000","last_clean":"2024-02-12T17:15:54.443711+0000","last_became_active":"2024-02-12T17:14:08.861886+0000","last_became_peered":"2024-02-12T17:14:08.861886+0000","last_unstale":"2024-02-12T17:15:54.443711+0000","last_undegraded":"2024-02-12T17:15:54.443711+0000","last_fullsized":"2024-02-12T17:15:54.443711+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"698'14293","reported_seq":21267,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-12T17:16:08.711667+0000","last_change":"2024-02-12T17:15:39.901144+0000","last_active":"2024-02-12T17:16:08.711667+0000","last_peered":"2024-02-12T17:16:08.711667+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.966441+0000","last_became_peered":"2024-02-12T17:14:08.966441+0000","last_unstale":"2024-02-12T17:16:08.711667+0000","last_undegraded":"2024-02-12T17:14:08.734833+0000","last_fullsized":"2024-02-12T17:14:08.712653+0000","mapping_epoch":651,"log_start":"626'7221","ondisk_log_start":"626'7221","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":7072,"ondisk_log_size":7072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100073472,"num_objects":1527,"num_object_clones":0,"num_object_copies":4581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1527,"num_objects_misplaced":1527,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14293,"num_write_kb":506240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3413,"num_bytes_recovered":223674368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[2,4],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4","objects":1527}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":415,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613445+0000","last_change":"2024-02-12T17:13:46.787305+0000","last_active":"2024-02-12T17:16:03.613445+0000","last_peered":"2024-02-12T17:16:03.613445+0000","last_clean":"2024-02-12T17:16:03.613445+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:16:03.613445+0000","last_undegraded":"2024-02-12T17:16:03.613445+0000","last_fullsized":"2024-02-12T17:16:03.613445+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":414,"reported_epoch":697,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:03.607077+0000","last_change":"2024-02-12T17:14:09.723027+0000","last_active":"2024-02-12T17:16:03.607077+0000","last_peered":"2024-02-12T17:16:03.607077+0000","last_clean":"2024-02-12T17:11:53.687795+0000","last_became_active":"2024-02-12T17:14:09.722681+0000","last_became_peered":"2024-02-12T17:14:09.722681+0000","last_unstale":"2024-02-12T17:16:03.607077+0000","last_undegraded":"2024-02-12T17:16:03.607077+0000","last_fullsized":"2024-02-12T17:16:03.607077+0000","mapping_epoch":652,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:12:09.882247+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:12:09.882247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[5,7,6],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":163,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.438398+0000","last_change":"2024-02-12T17:13:46.744739+0000","last_active":"2024-02-12T17:15:54.438398+0000","last_peered":"2024-02-12T17:15:54.438398+0000","last_clean":"2024-02-12T17:15:54.438398+0000","last_became_active":"2024-02-12T17:13:46.744003+0000","last_became_peered":"2024-02-12T17:13:46.744003+0000","last_unstale":"2024-02-12T17:15:54.438398+0000","last_undegraded":"2024-02-12T17:15:54.438398+0000","last_fullsized":"2024-02-12T17:15:54.438398+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:51.126056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:51.126056+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.442676+0000","last_change":"2024-02-12T17:14:08.769824+0000","last_active":"2024-02-12T17:15:54.442676+0000","last_peered":"2024-02-12T17:15:54.442676+0000","last_clean":"2024-02-12T17:15:54.442676+0000","last_became_active":"2024-02-12T17:14:08.769221+0000","last_became_peered":"2024-02-12T17:14:08.769221+0000","last_unstale":"2024-02-12T17:15:54.442676+0000","last_undegraded":"2024-02-12T17:15:54.442676+0000","last_fullsized":"2024-02-12T17:15:54.442676+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"698'14018","reported_seq":14159,"reported_epoch":698,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:16:08.715788+0000","last_change":"2024-02-12T17:14:16.786942+0000","last_active":"2024-02-12T17:16:08.715788+0000","last_peered":"2024-02-12T17:16:08.715788+0000","last_clean":"2024-02-12T17:13:45.521351+0000","last_became_active":"2024-02-12T17:13:46.743970+0000","last_became_peered":"2024-02-12T17:13:46.743970+0000","last_unstale":"2024-02-12T17:16:08.715788+0000","last_undegraded":"2024-02-12T17:16:08.715788+0000","last_fullsized":"2024-02-12T17:16:08.715788+0000","mapping_epoch":630,"log_start":"629'7968","ondisk_log_start":"629'7968","created":597,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'5769","last_scrub_stamp":"2024-02-12T17:13:06.847792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:13:06.847792+0000","log_size":6050,"ondisk_log_size":6050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101777432,"num_objects":1554,"num_object_clones":0,"num_object_copies":4662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1554,"num_objects_unfound":0,"num_objects_dirty":1554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14018,"num_write_kb":498241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":1554}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":702,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.442903+0000","last_change":"2024-02-12T17:14:07.897458+0000","last_active":"2024-02-12T17:15:54.442903+0000","last_peered":"2024-02-12T17:15:54.442903+0000","last_clean":"2024-02-12T17:15:54.442903+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:15:54.442903+0000","last_undegraded":"2024-02-12T17:15:54.442903+0000","last_fullsized":"2024-02-12T17:15:54.442903+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.780381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.780381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.440189+0000","last_change":"2024-02-12T17:13:46.791505+0000","last_active":"2024-02-12T17:15:54.440189+0000","last_peered":"2024-02-12T17:15:54.440189+0000","last_clean":"2024-02-12T17:15:54.440189+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:15:54.440189+0000","last_undegraded":"2024-02-12T17:15:54.440189+0000","last_fullsized":"2024-02-12T17:15:54.440189+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":169,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.438153+0000","last_change":"2024-02-12T17:14:09.246132+0000","last_active":"2024-02-12T17:15:54.438153+0000","last_peered":"2024-02-12T17:15:54.438153+0000","last_clean":"2024-02-12T17:15:54.438153+0000","last_became_active":"2024-02-12T17:14:08.752149+0000","last_became_peered":"2024-02-12T17:14:08.752149+0000","last_unstale":"2024-02-12T17:15:54.438153+0000","last_undegraded":"2024-02-12T17:15:54.438153+0000","last_fullsized":"2024-02-12T17:15:54.438153+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:13:01.543047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.543047+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"655'23","reported_seq":519,"reported_epoch":697,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:03.613329+0000","last_change":"2024-02-12T17:14:09.749094+0000","last_active":"2024-02-12T17:16:03.613329+0000","last_peered":"2024-02-12T17:16:03.613329+0000","last_clean":"2024-02-12T17:03:47.420086+0000","last_became_active":"2024-02-12T17:14:09.748790+0000","last_became_peered":"2024-02-12T17:14:09.748790+0000","last_unstale":"2024-02-12T17:16:03.613329+0000","last_undegraded":"2024-02-12T17:16:03.613329+0000","last_fullsized":"2024-02-12T17:16:03.613329+0000","mapping_epoch":652,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'21","last_scrub_stamp":"2024-02-12T17:12:57.741798+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:12:57.741798+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[3,6,5],"avail_no_missing":["3","0","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"698'14129","reported_seq":17450,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:16:06.646375+0000","last_change":"2024-02-12T17:14:31.380762+0000","last_active":"2024-02-12T17:16:06.646375+0000","last_peered":"2024-02-12T17:16:06.646375+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.877848+0000","last_became_peered":"2024-02-12T17:14:08.877848+0000","last_unstale":"2024-02-12T17:16:06.646375+0000","last_undegraded":"2024-02-12T17:14:09.686965+0000","last_fullsized":"2024-02-12T17:14:08.709420+0000","mapping_epoch":651,"log_start":"650'9163","ondisk_log_start":"650'9163","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4966,"ondisk_log_size":4966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104267776,"num_objects":1591,"num_object_clones":0,"num_object_copies":4773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1140,"num_objects_misplaced":1140,"num_objects_unfound":0,"num_objects_dirty":1591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14129,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1115,"num_bytes_recovered":72941568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4","objects":1591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":418,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.614783+0000","last_change":"2024-02-12T17:13:21.603543+0000","last_active":"2024-02-12T17:16:03.614783+0000","last_peered":"2024-02-12T17:16:03.614783+0000","last_clean":"2024-02-12T17:16:03.614783+0000","last_became_active":"2024-02-12T17:13:21.603226+0000","last_became_peered":"2024-02-12T17:13:21.603226+0000","last_unstale":"2024-02-12T17:16:03.614783+0000","last_undegraded":"2024-02-12T17:16:03.614783+0000","last_fullsized":"2024-02-12T17:16:03.614783+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.262134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.262134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":734,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605184+0000","last_change":"2024-02-12T17:13:46.566957+0000","last_active":"2024-02-12T17:16:03.605184+0000","last_peered":"2024-02-12T17:16:03.605184+0000","last_clean":"2024-02-12T17:16:03.605184+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:16:03.605184+0000","last_undegraded":"2024-02-12T17:16:03.605184+0000","last_fullsized":"2024-02-12T17:16:03.605184+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":211,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:16:02.674547+0000","last_change":"2024-02-12T17:13:46.607171+0000","last_active":"2024-02-12T17:16:02.674547+0000","last_peered":"2024-02-12T17:16:02.674547+0000","last_clean":"2024-02-12T17:16:02.674547+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:16:02.674547+0000","last_undegraded":"2024-02-12T17:16:02.674547+0000","last_fullsized":"2024-02-12T17:16:02.674547+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":648,"reported_epoch":696,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:15:54.438880+0000","last_change":"2024-02-12T17:14:09.723924+0000","last_active":"2024-02-12T17:15:54.438880+0000","last_peered":"2024-02-12T17:15:54.438880+0000","last_clean":"2024-02-12T17:14:05.074559+0000","last_became_active":"2024-02-12T17:14:09.723600+0000","last_became_peered":"2024-02-12T17:14:09.723600+0000","last_unstale":"2024-02-12T17:15:54.438880+0000","last_undegraded":"2024-02-12T17:15:54.438880+0000","last_fullsized":"2024-02-12T17:15:54.438880+0000","mapping_epoch":652,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:13:02.852563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:13:02.852563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[1,6,7],"avail_no_missing":["1","0","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"698'14290","reported_seq":14399,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:06.663887+0000","last_change":"2024-02-12T17:11:59.008876+0000","last_active":"2024-02-12T17:16:06.663887+0000","last_peered":"2024-02-12T17:16:06.663887+0000","last_clean":"2024-02-12T17:16:06.663887+0000","last_became_active":"2024-02-12T17:11:59.008587+0000","last_became_peered":"2024-02-12T17:11:59.008587+0000","last_unstale":"2024-02-12T17:16:06.663887+0000","last_undegraded":"2024-02-12T17:16:06.663887+0000","last_fullsized":"2024-02-12T17:16:06.663887+0000","mapping_epoch":597,"log_start":"650'9318","ondisk_log_start":"650'9318","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4972,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110362624,"num_objects":1684,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14290,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":703,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.611549+0000","last_change":"2024-02-12T17:13:00.766698+0000","last_active":"2024-02-12T17:16:03.611549+0000","last_peered":"2024-02-12T17:16:03.611549+0000","last_clean":"2024-02-12T17:16:03.611549+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:16:03.611549+0000","last_undegraded":"2024-02-12T17:16:03.611549+0000","last_fullsized":"2024-02-12T17:16:03.611549+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.766590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":335,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605778+0000","last_change":"2024-02-12T17:13:46.584941+0000","last_active":"2024-02-12T17:16:03.605778+0000","last_peered":"2024-02-12T17:16:03.605778+0000","last_clean":"2024-02-12T17:16:03.605778+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:16:03.605778+0000","last_undegraded":"2024-02-12T17:16:03.605778+0000","last_fullsized":"2024-02-12T17:16:03.605778+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":350,"num_read_kb":345,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":364,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.616038+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:16:03.616038+0000","last_peered":"2024-02-12T17:16:03.616038+0000","last_clean":"2024-02-12T17:16:03.616038+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:16:03.616038+0000","last_undegraded":"2024-02-12T17:16:03.616038+0000","last_fullsized":"2024-02-12T17:16:03.616038+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":162,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.439354+0000","last_change":"2024-02-12T17:14:09.321979+0000","last_active":"2024-02-12T17:15:54.439354+0000","last_peered":"2024-02-12T17:15:54.439354+0000","last_clean":"2024-02-12T17:15:54.439354+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:15:54.439354+0000","last_undegraded":"2024-02-12T17:15:54.439354+0000","last_fullsized":"2024-02-12T17:15:54.439354+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":627,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.606502+0000","last_change":"2024-02-12T17:13:10.412138+0000","last_active":"2024-02-12T17:16:03.606502+0000","last_peered":"2024-02-12T17:16:03.606502+0000","last_clean":"2024-02-12T17:16:03.606502+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:16:03.606502+0000","last_undegraded":"2024-02-12T17:16:03.606502+0000","last_fullsized":"2024-02-12T17:16:03.606502+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:13:10.412031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"698'13908","reported_seq":14016,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:06.686052+0000","last_change":"2024-02-12T17:11:59.007548+0000","last_active":"2024-02-12T17:16:06.686052+0000","last_peered":"2024-02-12T17:16:06.686052+0000","last_clean":"2024-02-12T17:16:06.686052+0000","last_became_active":"2024-02-12T17:11:59.007214+0000","last_became_peered":"2024-02-12T17:11:59.007214+0000","last_unstale":"2024-02-12T17:16:06.686052+0000","last_undegraded":"2024-02-12T17:16:06.686052+0000","last_fullsized":"2024-02-12T17:16:06.686052+0000","mapping_epoch":597,"log_start":"650'8955","ondisk_log_start":"650'8955","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4953,"ondisk_log_size":4953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106823680,"num_objects":1630,"num_object_clones":0,"num_object_copies":4890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13908,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":766,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.607422+0000","last_change":"2024-02-12T17:13:46.826853+0000","last_active":"2024-02-12T17:16:03.607422+0000","last_peered":"2024-02-12T17:16:03.607422+0000","last_clean":"2024-02-12T17:16:03.607422+0000","last_became_active":"2024-02-12T17:13:46.821418+0000","last_became_peered":"2024-02-12T17:13:46.821418+0000","last_unstale":"2024-02-12T17:16:03.607422+0000","last_undegraded":"2024-02-12T17:16:03.607422+0000","last_fullsized":"2024-02-12T17:16:03.607422+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.951964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.951964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":396,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613838+0000","last_change":"2024-02-12T17:12:59.006372+0000","last_active":"2024-02-12T17:16:03.613838+0000","last_peered":"2024-02-12T17:16:03.613838+0000","last_clean":"2024-02-12T17:16:03.613838+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:16:03.613838+0000","last_undegraded":"2024-02-12T17:16:03.613838+0000","last_fullsized":"2024-02-12T17:16:03.613838+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.006260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.006260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":674,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605691+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:16:03.605691+0000","last_peered":"2024-02-12T17:16:03.605691+0000","last_clean":"2024-02-12T17:16:03.605691+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:16:03.605691+0000","last_undegraded":"2024-02-12T17:16:03.605691+0000","last_fullsized":"2024-02-12T17:16:03.605691+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":743,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.610931+0000","last_change":"2024-02-12T17:14:08.743668+0000","last_active":"2024-02-12T17:16:03.610931+0000","last_peered":"2024-02-12T17:16:03.610931+0000","last_clean":"2024-02-12T17:16:03.610931+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:16:03.610931+0000","last_undegraded":"2024-02-12T17:16:03.610931+0000","last_fullsized":"2024-02-12T17:16:03.610931+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":308,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613542+0000","last_change":"2024-02-12T17:14:08.781462+0000","last_active":"2024-02-12T17:16:03.613542+0000","last_peered":"2024-02-12T17:16:03.613542+0000","last_clean":"2024-02-12T17:16:03.613542+0000","last_became_active":"2024-02-12T17:14:08.726463+0000","last_became_peered":"2024-02-12T17:14:08.726463+0000","last_unstale":"2024-02-12T17:16:03.613542+0000","last_undegraded":"2024-02-12T17:16:03.613542+0000","last_fullsized":"2024-02-12T17:16:03.613542+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:47.763264+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.763264+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":705,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.611415+0000","last_change":"2024-02-12T17:12:03.922963+0000","last_active":"2024-02-12T17:16:03.611415+0000","last_peered":"2024-02-12T17:16:03.611415+0000","last_clean":"2024-02-12T17:16:03.611415+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:16:03.611415+0000","last_undegraded":"2024-02-12T17:16:03.611415+0000","last_fullsized":"2024-02-12T17:16:03.611415+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:12:03.922893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.922893+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"698'14113","reported_seq":16484,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:16:06.663680+0000","last_change":"2024-02-12T17:14:10.177438+0000","last_active":"2024-02-12T17:16:06.663680+0000","last_peered":"2024-02-12T17:16:06.663680+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.873608+0000","last_became_peered":"2024-02-12T17:14:08.873608+0000","last_unstale":"2024-02-12T17:16:06.663680+0000","last_undegraded":"2024-02-12T17:14:08.744781+0000","last_fullsized":"2024-02-12T17:14:08.704327+0000","mapping_epoch":651,"log_start":"650'9142","ondisk_log_start":"650'9142","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4971,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":107151360,"num_objects":1635,"num_object_clones":0,"num_object_copies":4905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14113,"num_write_kb":503936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1123,"num_bytes_recovered":73596928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":89,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.440263+0000","last_change":"2024-02-12T17:13:46.608049+0000","last_active":"2024-02-12T17:15:54.440263+0000","last_peered":"2024-02-12T17:15:54.440263+0000","last_clean":"2024-02-12T17:15:54.440263+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:15:54.440263+0000","last_undegraded":"2024-02-12T17:15:54.440263+0000","last_fullsized":"2024-02-12T17:15:54.440263+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.450621+0000","last_change":"2024-02-12T17:14:08.859925+0000","last_active":"2024-02-12T17:15:54.450621+0000","last_peered":"2024-02-12T17:15:54.450621+0000","last_clean":"2024-02-12T17:15:54.450621+0000","last_became_active":"2024-02-12T17:14:08.859396+0000","last_became_peered":"2024-02-12T17:14:08.859396+0000","last_unstale":"2024-02-12T17:15:54.450621+0000","last_undegraded":"2024-02-12T17:15:54.450621+0000","last_fullsized":"2024-02-12T17:15:54.450621+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:03.904976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.904976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":743,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.610864+0000","last_change":"2024-02-12T17:14:05.666060+0000","last_active":"2024-02-12T17:16:03.610864+0000","last_peered":"2024-02-12T17:16:03.610864+0000","last_clean":"2024-02-12T17:16:03.610864+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:16:03.610864+0000","last_undegraded":"2024-02-12T17:16:03.610864+0000","last_fullsized":"2024-02-12T17:16:03.610864+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":69,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.443792+0000","last_change":"2024-02-12T17:14:09.046726+0000","last_active":"2024-02-12T17:15:54.443792+0000","last_peered":"2024-02-12T17:15:54.443792+0000","last_clean":"2024-02-12T17:15:54.443792+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:15:54.443792+0000","last_undegraded":"2024-02-12T17:15:54.443792+0000","last_fullsized":"2024-02-12T17:15:54.443792+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":746,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.607337+0000","last_change":"2024-02-12T17:13:46.827015+0000","last_active":"2024-02-12T17:16:03.607337+0000","last_peered":"2024-02-12T17:16:03.607337+0000","last_clean":"2024-02-12T17:16:03.607337+0000","last_became_active":"2024-02-12T17:13:46.821796+0000","last_became_peered":"2024-02-12T17:13:46.821796+0000","last_unstale":"2024-02-12T17:16:03.607337+0000","last_undegraded":"2024-02-12T17:16:03.607337+0000","last_fullsized":"2024-02-12T17:16:03.607337+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.698727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.698727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"698'14042","reported_seq":20139,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:16:08.701095+0000","last_change":"2024-02-12T17:14:46.835208+0000","last_active":"2024-02-12T17:16:08.701095+0000","last_peered":"2024-02-12T17:16:08.701095+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.806885+0000","last_became_peered":"2024-02-12T17:13:46.806885+0000","last_unstale":"2024-02-12T17:16:08.701095+0000","last_undegraded":"2024-02-12T17:13:46.572970+0000","last_fullsized":"2024-02-12T17:13:46.531119+0000","mapping_epoch":630,"log_start":"626'6974","ondisk_log_start":"626'6974","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":7068,"ondisk_log_size":7068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99221504,"num_objects":1514,"num_object_clones":0,"num_object_copies":4542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":855,"num_objects_misplaced":1514,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14042,"num_write_kb":497792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2984,"num_bytes_recovered":195559424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":1514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":120,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.438311+0000","last_change":"2024-02-12T17:13:46.579742+0000","last_active":"2024-02-12T17:15:54.438311+0000","last_peered":"2024-02-12T17:15:54.438311+0000","last_clean":"2024-02-12T17:15:54.438311+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:15:54.438311+0000","last_undegraded":"2024-02-12T17:15:54.438311+0000","last_fullsized":"2024-02-12T17:15:54.438311+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":757,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.611608+0000","last_change":"2024-02-12T17:14:08.746702+0000","last_active":"2024-02-12T17:16:03.611608+0000","last_peered":"2024-02-12T17:16:03.611608+0000","last_clean":"2024-02-12T17:16:03.611608+0000","last_became_active":"2024-02-12T17:14:08.746395+0000","last_became_peered":"2024-02-12T17:14:08.746395+0000","last_unstale":"2024-02-12T17:16:03.611608+0000","last_undegraded":"2024-02-12T17:16:03.611608+0000","last_fullsized":"2024-02-12T17:16:03.611608+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:45.764955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:45.764955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":73,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.451763+0000","last_change":"2024-02-12T17:14:11.148689+0000","last_active":"2024-02-12T17:15:54.451763+0000","last_peered":"2024-02-12T17:15:54.451763+0000","last_clean":"2024-02-12T17:15:54.451763+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:15:54.451763+0000","last_undegraded":"2024-02-12T17:15:54.451763+0000","last_fullsized":"2024-02-12T17:15:54.451763+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":199,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613288+0000","last_change":"2024-02-12T17:12:53.746937+0000","last_active":"2024-02-12T17:16:03.613288+0000","last_peered":"2024-02-12T17:16:03.613288+0000","last_clean":"2024-02-12T17:16:03.613288+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:16:03.613288+0000","last_undegraded":"2024-02-12T17:16:03.613288+0000","last_fullsized":"2024-02-12T17:16:03.613288+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.746858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.746858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"698'14112","reported_seq":14235,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:16:08.969912+0000","last_change":"2024-02-12T17:14:46.848102+0000","last_active":"2024-02-12T17:16:08.969912+0000","last_peered":"2024-02-12T17:16:08.969912+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.832453+0000","last_became_peered":"2024-02-12T17:13:46.832453+0000","last_unstale":"2024-02-12T17:16:08.969912+0000","last_undegraded":"2024-02-12T17:13:46.655125+0000","last_fullsized":"2024-02-12T17:13:46.642131+0000","mapping_epoch":631,"log_start":"637'8370","ondisk_log_start":"637'8370","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5742,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105512960,"num_objects":1610,"num_object_clones":0,"num_object_copies":4830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14112,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1610}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":405,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.614013+0000","last_change":"2024-02-12T17:13:21.589965+0000","last_active":"2024-02-12T17:16:03.614013+0000","last_peered":"2024-02-12T17:16:03.614013+0000","last_clean":"2024-02-12T17:16:03.614013+0000","last_became_active":"2024-02-12T17:13:21.589584+0000","last_became_peered":"2024-02-12T17:13:21.589584+0000","last_unstale":"2024-02-12T17:16:03.614013+0000","last_undegraded":"2024-02-12T17:16:03.614013+0000","last_fullsized":"2024-02-12T17:16:03.614013+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:01.059570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.059570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.438862+0000","last_change":"2024-02-12T17:14:18.716530+0000","last_active":"2024-02-12T17:15:54.438862+0000","last_peered":"2024-02-12T17:15:54.438862+0000","last_clean":"2024-02-12T17:15:54.438862+0000","last_became_active":"2024-02-12T17:14:18.716232+0000","last_became_peered":"2024-02-12T17:14:18.716232+0000","last_unstale":"2024-02-12T17:15:54.438862+0000","last_undegraded":"2024-02-12T17:15:54.438862+0000","last_fullsized":"2024-02-12T17:15:54.438862+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":464,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613685+0000","last_change":"2024-02-12T17:14:10.144923+0000","last_active":"2024-02-12T17:16:03.613685+0000","last_peered":"2024-02-12T17:16:03.613685+0000","last_clean":"2024-02-12T17:16:03.613685+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:16:03.613685+0000","last_undegraded":"2024-02-12T17:16:03.613685+0000","last_fullsized":"2024-02-12T17:16:03.613685+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":684,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.615869+0000","last_change":"2024-02-12T17:14:08.765556+0000","last_active":"2024-02-12T17:16:03.615869+0000","last_peered":"2024-02-12T17:16:03.615869+0000","last_clean":"2024-02-12T17:16:03.615869+0000","last_became_active":"2024-02-12T17:14:08.757310+0000","last_became_peered":"2024-02-12T17:14:08.757310+0000","last_unstale":"2024-02-12T17:16:03.615869+0000","last_undegraded":"2024-02-12T17:16:03.615869+0000","last_fullsized":"2024-02-12T17:16:03.615869+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:58.871569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.871569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"698'14235","reported_seq":18521,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:16:08.936993+0000","last_change":"2024-02-12T17:14:10.288517+0000","last_active":"2024-02-12T17:16:08.936993+0000","last_peered":"2024-02-12T17:16:08.936993+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.867083+0000","last_became_peered":"2024-02-12T17:14:08.867083+0000","last_unstale":"2024-02-12T17:16:08.936993+0000","last_undegraded":"2024-02-12T17:14:08.714899+0000","last_fullsized":"2024-02-12T17:14:08.701373+0000","mapping_epoch":651,"log_start":"640'8542","ondisk_log_start":"640'8542","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5693,"ondisk_log_size":5693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98107392,"num_objects":1497,"num_object_clones":0,"num_object_copies":4491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":984,"num_objects_misplaced":984,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14235,"num_write_kb":503424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":96141312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":1497}],"blocked_by":[],"up_primary":1,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":736,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.611257+0000","last_change":"2024-02-12T17:13:08.874063+0000","last_active":"2024-02-12T17:16:03.611257+0000","last_peered":"2024-02-12T17:16:03.611257+0000","last_clean":"2024-02-12T17:16:03.611257+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:16:03.611257+0000","last_undegraded":"2024-02-12T17:16:03.611257+0000","last_fullsized":"2024-02-12T17:16:03.611257+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:13:08.873961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:13:08.873961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":217,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613480+0000","last_change":"2024-02-12T17:14:14.843264+0000","last_active":"2024-02-12T17:16:03.613480+0000","last_peered":"2024-02-12T17:16:03.613480+0000","last_clean":"2024-02-12T17:16:03.613480+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:16:03.613480+0000","last_undegraded":"2024-02-12T17:16:03.613480+0000","last_fullsized":"2024-02-12T17:16:03.613480+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:56.688982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.688982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":882,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605318+0000","last_change":"2024-02-12T17:14:09.517724+0000","last_active":"2024-02-12T17:16:03.605318+0000","last_peered":"2024-02-12T17:16:03.605318+0000","last_clean":"2024-02-12T17:16:03.605318+0000","last_became_active":"2024-02-12T17:14:08.740691+0000","last_became_peered":"2024-02-12T17:14:08.740691+0000","last_unstale":"2024-02-12T17:16:03.605318+0000","last_undegraded":"2024-02-12T17:16:03.605318+0000","last_fullsized":"2024-02-12T17:16:03.605318+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:12:05.850456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.850456+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"655'32","reported_seq":645,"reported_epoch":697,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:03.613517+0000","last_change":"2024-02-12T17:14:09.746021+0000","last_active":"2024-02-12T17:16:03.613517+0000","last_peered":"2024-02-12T17:16:03.613517+0000","last_clean":"2024-02-12T17:14:07.617720+0000","last_became_active":"2024-02-12T17:14:09.745704+0000","last_became_peered":"2024-02-12T17:14:09.745704+0000","last_unstale":"2024-02-12T17:16:03.613517+0000","last_undegraded":"2024-02-12T17:16:03.613517+0000","last_fullsized":"2024-02-12T17:16:03.613517+0000","mapping_epoch":652,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3","0","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"698'14139","reported_seq":14264,"reported_epoch":698,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:08.662318+0000","last_change":"2024-02-12T17:14:09.843467+0000","last_active":"2024-02-12T17:16:08.662318+0000","last_peered":"2024-02-12T17:16:08.662318+0000","last_clean":"2024-02-12T17:14:07.683370+0000","last_became_active":"2024-02-12T17:14:09.782739+0000","last_became_peered":"2024-02-12T17:14:09.782739+0000","last_unstale":"2024-02-12T17:16:08.662318+0000","last_undegraded":"2024-02-12T17:16:08.662318+0000","last_fullsized":"2024-02-12T17:16:08.662318+0000","mapping_epoch":652,"log_start":"633'8051","ondisk_log_start":"633'8051","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6088,"ondisk_log_size":6088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99155968,"num_objects":1513,"num_object_clones":0,"num_object_copies":4539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1513,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":1513}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":407,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.614021+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:16:03.614021+0000","last_peered":"2024-02-12T17:16:03.614021+0000","last_clean":"2024-02-12T17:16:03.614021+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:16:03.614021+0000","last_undegraded":"2024-02-12T17:16:03.614021+0000","last_fullsized":"2024-02-12T17:16:03.614021+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"642'31","reported_seq":1107,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.657434+0000","last_change":"2024-02-12T17:13:53.773053+0000","last_active":"2024-02-12T17:16:05.657434+0000","last_peered":"2024-02-12T17:16:05.657434+0000","last_clean":"2024-02-12T17:16:05.657434+0000","last_became_active":"2024-02-12T17:13:53.772744+0000","last_became_peered":"2024-02-12T17:13:53.772744+0000","last_unstale":"2024-02-12T17:16:05.657434+0000","last_undegraded":"2024-02-12T17:16:05.657434+0000","last_fullsized":"2024-02-12T17:16:05.657434+0000","mapping_epoch":638,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'28","last_scrub_stamp":"2024-02-12T17:12:07.826227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:12:07.826227+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":411,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.610780+0000","last_change":"2024-02-12T17:14:10.756477+0000","last_active":"2024-02-12T17:16:03.610780+0000","last_peered":"2024-02-12T17:16:03.610780+0000","last_clean":"2024-02-12T17:16:03.610780+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:16:03.610780+0000","last_undegraded":"2024-02-12T17:16:03.610780+0000","last_fullsized":"2024-02-12T17:16:03.610780+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:16.873570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.873570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":271,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613295+0000","last_change":"2024-02-12T17:13:46.807564+0000","last_active":"2024-02-12T17:16:03.613295+0000","last_peered":"2024-02-12T17:16:03.613295+0000","last_clean":"2024-02-12T17:16:03.613295+0000","last_became_active":"2024-02-12T17:13:46.807214+0000","last_became_peered":"2024-02-12T17:13:46.807214+0000","last_unstale":"2024-02-12T17:16:03.613295+0000","last_undegraded":"2024-02-12T17:16:03.613295+0000","last_fullsized":"2024-02-12T17:16:03.613295+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:04.705646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.705646+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"698'14200","reported_seq":17457,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:16:06.677828+0000","last_change":"2024-02-12T17:14:09.819105+0000","last_active":"2024-02-12T17:16:06.677828+0000","last_peered":"2024-02-12T17:16:06.677828+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.810504+0000","last_became_peered":"2024-02-12T17:13:46.810504+0000","last_unstale":"2024-02-12T17:16:06.677828+0000","last_undegraded":"2024-02-12T17:13:46.690611+0000","last_fullsized":"2024-02-12T17:13:46.648648+0000","mapping_epoch":631,"log_start":"652'9271","ondisk_log_start":"652'9271","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4929,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109969408,"num_objects":1678,"num_object_clones":0,"num_object_copies":5034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14200,"num_write_kb":508096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1568,"num_bytes_recovered":102760448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1678}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":741,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.606056+0000","last_change":"2024-02-12T17:14:08.740896+0000","last_active":"2024-02-12T17:16:03.606056+0000","last_peered":"2024-02-12T17:16:03.606056+0000","last_clean":"2024-02-12T17:16:03.606056+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:16:03.606056+0000","last_undegraded":"2024-02-12T17:16:03.606056+0000","last_fullsized":"2024-02-12T17:16:03.606056+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"668'27","reported_seq":1132,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.657239+0000","last_change":"2024-02-12T17:14:24.235718+0000","last_active":"2024-02-12T17:16:05.657239+0000","last_peered":"2024-02-12T17:16:05.657239+0000","last_clean":"2024-02-12T17:16:05.657239+0000","last_became_active":"2024-02-12T17:14:24.235386+0000","last_became_peered":"2024-02-12T17:14:24.235386+0000","last_unstale":"2024-02-12T17:16:05.657239+0000","last_undegraded":"2024-02-12T17:16:05.657239+0000","last_fullsized":"2024-02-12T17:16:05.657239+0000","mapping_epoch":665,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'24","last_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.647212+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":107,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.438227+0000","last_change":"2024-02-12T17:14:05.106537+0000","last_active":"2024-02-12T17:15:54.438227+0000","last_peered":"2024-02-12T17:15:54.438227+0000","last_clean":"2024-02-12T17:15:54.438227+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:15:54.438227+0000","last_undegraded":"2024-02-12T17:15:54.438227+0000","last_fullsized":"2024-02-12T17:15:54.438227+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":155,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.439507+0000","last_change":"2024-02-12T17:13:46.819259+0000","last_active":"2024-02-12T17:15:54.439507+0000","last_peered":"2024-02-12T17:15:54.439507+0000","last_clean":"2024-02-12T17:15:54.439507+0000","last_became_active":"2024-02-12T17:13:46.818771+0000","last_became_peered":"2024-02-12T17:13:46.818771+0000","last_unstale":"2024-02-12T17:15:54.439507+0000","last_undegraded":"2024-02-12T17:15:54.439507+0000","last_fullsized":"2024-02-12T17:15:54.439507+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"698'13877","reported_seq":26807,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:08.673932+0000","last_change":"2024-02-12T17:14:32.710386+0000","last_active":"2024-02-12T17:16:08.673932+0000","last_peered":"2024-02-12T17:16:08.673932+0000","last_clean":"2024-02-12T17:16:08.673932+0000","last_became_active":"2024-02-12T17:14:32.696812+0000","last_became_peered":"2024-02-12T17:14:32.696812+0000","last_unstale":"2024-02-12T17:16:08.673932+0000","last_undegraded":"2024-02-12T17:16:08.673932+0000","last_fullsized":"2024-02-12T17:16:08.673932+0000","mapping_epoch":671,"log_start":"628'7800","ondisk_log_start":"628'7800","created":597,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6077,"ondisk_log_size":6077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101253120,"num_objects":1545,"num_object_clones":0,"num_object_copies":4635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13877,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6402,"num_bytes_recovered":419561472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":87,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.438790+0000","last_change":"2024-02-12T17:13:46.805110+0000","last_active":"2024-02-12T17:15:54.438790+0000","last_peered":"2024-02-12T17:15:54.438790+0000","last_clean":"2024-02-12T17:15:54.438790+0000","last_became_active":"2024-02-12T17:13:46.802275+0000","last_became_peered":"2024-02-12T17:13:46.802275+0000","last_unstale":"2024-02-12T17:15:54.438790+0000","last_undegraded":"2024-02-12T17:15:54.438790+0000","last_fullsized":"2024-02-12T17:15:54.438790+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":635,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.658422+0000","last_change":"2024-02-12T17:13:07.711701+0000","last_active":"2024-02-12T17:16:05.658422+0000","last_peered":"2024-02-12T17:16:05.658422+0000","last_clean":"2024-02-12T17:16:05.658422+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:16:05.658422+0000","last_undegraded":"2024-02-12T17:16:05.658422+0000","last_fullsized":"2024-02-12T17:16:05.658422+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:13:07.711601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:07.711601+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":107,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.437868+0000","last_change":"2024-02-12T17:14:00.999940+0000","last_active":"2024-02-12T17:15:54.437868+0000","last_peered":"2024-02-12T17:15:54.437868+0000","last_clean":"2024-02-12T17:15:54.437868+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:15:54.437868+0000","last_undegraded":"2024-02-12T17:15:54.437868+0000","last_fullsized":"2024-02-12T17:15:54.437868+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":501,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.614122+0000","last_change":"2024-02-12T17:13:46.806094+0000","last_active":"2024-02-12T17:16:03.614122+0000","last_peered":"2024-02-12T17:16:03.614122+0000","last_clean":"2024-02-12T17:16:03.614122+0000","last_became_active":"2024-02-12T17:13:46.805712+0000","last_became_peered":"2024-02-12T17:13:46.805712+0000","last_unstale":"2024-02-12T17:16:03.614122+0000","last_undegraded":"2024-02-12T17:16:03.614122+0000","last_fullsized":"2024-02-12T17:16:03.614122+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"698'14189","reported_seq":14297,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:06.660112+0000","last_change":"2024-02-12T17:11:59.496950+0000","last_active":"2024-02-12T17:16:06.660112+0000","last_peered":"2024-02-12T17:16:06.660112+0000","last_clean":"2024-02-12T17:16:06.660112+0000","last_became_active":"2024-02-12T17:11:59.496585+0000","last_became_peered":"2024-02-12T17:11:59.496585+0000","last_unstale":"2024-02-12T17:16:06.660112+0000","last_undegraded":"2024-02-12T17:16:06.660112+0000","last_fullsized":"2024-02-12T17:16:06.660112+0000","mapping_epoch":597,"log_start":"650'9248","ondisk_log_start":"650'9248","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4941,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109772800,"num_objects":1675,"num_object_clones":0,"num_object_copies":5025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14189,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":427,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.621462+0000","last_change":"2024-02-12T17:13:21.603687+0000","last_active":"2024-02-12T17:16:03.621462+0000","last_peered":"2024-02-12T17:16:03.621462+0000","last_clean":"2024-02-12T17:16:03.621462+0000","last_became_active":"2024-02-12T17:13:21.603332+0000","last_became_peered":"2024-02-12T17:13:21.603332+0000","last_unstale":"2024-02-12T17:16:03.621462+0000","last_undegraded":"2024-02-12T17:16:03.621462+0000","last_fullsized":"2024-02-12T17:16:03.621462+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":118,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.440120+0000","last_change":"2024-02-12T17:13:46.804446+0000","last_active":"2024-02-12T17:15:54.440120+0000","last_peered":"2024-02-12T17:15:54.440120+0000","last_clean":"2024-02-12T17:15:54.440120+0000","last_became_active":"2024-02-12T17:13:46.804033+0000","last_became_peered":"2024-02-12T17:13:46.804033+0000","last_unstale":"2024-02-12T17:15:54.440120+0000","last_undegraded":"2024-02-12T17:15:54.440120+0000","last_fullsized":"2024-02-12T17:15:54.440120+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.793569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.793569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":721,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.610995+0000","last_change":"2024-02-12T17:13:56.661004+0000","last_active":"2024-02-12T17:16:03.610995+0000","last_peered":"2024-02-12T17:16:03.610995+0000","last_clean":"2024-02-12T17:16:03.610995+0000","last_became_active":"2024-02-12T17:13:56.660707+0000","last_became_peered":"2024-02-12T17:13:56.660707+0000","last_unstale":"2024-02-12T17:16:03.610995+0000","last_undegraded":"2024-02-12T17:16:03.610995+0000","last_fullsized":"2024-02-12T17:16:03.610995+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.685601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.685601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":187,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.438716+0000","last_change":"2024-02-12T17:13:46.757888+0000","last_active":"2024-02-12T17:15:54.438716+0000","last_peered":"2024-02-12T17:15:54.438716+0000","last_clean":"2024-02-12T17:15:54.438716+0000","last_became_active":"2024-02-12T17:13:46.757450+0000","last_became_peered":"2024-02-12T17:13:46.757450+0000","last_unstale":"2024-02-12T17:15:54.438716+0000","last_undegraded":"2024-02-12T17:15:54.438716+0000","last_fullsized":"2024-02-12T17:15:54.438716+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:56.880282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.880282+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.5","version":"698'14062","reported_seq":14174,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:08.893527+0000","last_change":"2024-02-12T17:13:12.678687+0000","last_active":"2024-02-12T17:16:08.893527+0000","last_peered":"2024-02-12T17:16:08.893527+0000","last_clean":"2024-02-12T17:16:08.893527+0000","last_became_active":"2024-02-12T17:11:59.009543+0000","last_became_peered":"2024-02-12T17:11:59.009543+0000","last_unstale":"2024-02-12T17:16:08.893527+0000","last_undegraded":"2024-02-12T17:16:08.893527+0000","last_fullsized":"2024-02-12T17:16:08.893527+0000","mapping_epoch":597,"log_start":"638'8329","ondisk_log_start":"638'8329","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5733,"ondisk_log_size":5733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103022592,"num_objects":1572,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14062,"num_write_kb":500288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":721,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.610835+0000","last_change":"2024-02-12T17:12:08.013330+0000","last_active":"2024-02-12T17:16:03.610835+0000","last_peered":"2024-02-12T17:16:03.610835+0000","last_clean":"2024-02-12T17:16:03.610835+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:16:03.610835+0000","last_undegraded":"2024-02-12T17:16:03.610835+0000","last_fullsized":"2024-02-12T17:16:03.610835+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.013247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.013247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":617,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.452014+0000","last_change":"2024-02-12T17:12:04.711418+0000","last_active":"2024-02-12T17:15:54.452014+0000","last_peered":"2024-02-12T17:15:54.452014+0000","last_clean":"2024-02-12T17:15:54.452014+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:15:54.452014+0000","last_undegraded":"2024-02-12T17:15:54.452014+0000","last_fullsized":"2024-02-12T17:15:54.452014+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:04.711333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.711333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":476,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.606076+0000","last_change":"2024-02-12T17:14:08.740752+0000","last_active":"2024-02-12T17:16:03.606076+0000","last_peered":"2024-02-12T17:16:03.606076+0000","last_clean":"2024-02-12T17:16:03.606076+0000","last_became_active":"2024-02-12T17:14:08.740430+0000","last_became_peered":"2024-02-12T17:14:08.740430+0000","last_unstale":"2024-02-12T17:16:03.606076+0000","last_undegraded":"2024-02-12T17:16:03.606076+0000","last_fullsized":"2024-02-12T17:16:03.606076+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":253,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613326+0000","last_change":"2024-02-12T17:13:46.559877+0000","last_active":"2024-02-12T17:16:03.613326+0000","last_peered":"2024-02-12T17:16:03.613326+0000","last_clean":"2024-02-12T17:16:03.613326+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:16:03.613326+0000","last_undegraded":"2024-02-12T17:16:03.613326+0000","last_fullsized":"2024-02-12T17:16:03.613326+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"698'14315","reported_seq":14423,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:08.988258+0000","last_change":"2024-02-12T17:11:59.007574+0000","last_active":"2024-02-12T17:16:08.988258+0000","last_peered":"2024-02-12T17:16:08.988258+0000","last_clean":"2024-02-12T17:16:08.988258+0000","last_became_active":"2024-02-12T17:11:59.007300+0000","last_became_peered":"2024-02-12T17:11:59.007300+0000","last_unstale":"2024-02-12T17:16:08.988258+0000","last_undegraded":"2024-02-12T17:16:08.988258+0000","last_fullsized":"2024-02-12T17:16:08.988258+0000","mapping_epoch":597,"log_start":"639'8567","ondisk_log_start":"639'8567","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5748,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103481344,"num_objects":1579,"num_object_clones":0,"num_object_copies":4737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14315,"num_write_kb":508608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":55,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.443973+0000","last_change":"2024-02-12T17:14:08.859503+0000","last_active":"2024-02-12T17:15:54.443973+0000","last_peered":"2024-02-12T17:15:54.443973+0000","last_clean":"2024-02-12T17:15:54.443973+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:15:54.443973+0000","last_undegraded":"2024-02-12T17:15:54.443973+0000","last_fullsized":"2024-02-12T17:15:54.443973+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:03.705889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:13:03.705889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":618,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.613415+0000","last_change":"2024-02-12T17:14:08.762783+0000","last_active":"2024-02-12T17:16:03.613415+0000","last_peered":"2024-02-12T17:16:03.613415+0000","last_clean":"2024-02-12T17:16:03.613415+0000","last_became_active":"2024-02-12T17:14:08.760654+0000","last_became_peered":"2024-02-12T17:14:08.760654+0000","last_unstale":"2024-02-12T17:16:03.613415+0000","last_undegraded":"2024-02-12T17:16:03.613415+0000","last_fullsized":"2024-02-12T17:16:03.613415+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:09.044325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:09.044325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":396,"reported_epoch":697,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:16:03.615251+0000","last_change":"2024-02-12T17:14:16.753241+0000","last_active":"2024-02-12T17:16:03.615251+0000","last_peered":"2024-02-12T17:16:03.615251+0000","last_clean":"2024-02-12T17:13:44.508652+0000","last_became_active":"2024-02-12T17:13:46.563455+0000","last_became_peered":"2024-02-12T17:13:46.563455+0000","last_unstale":"2024-02-12T17:16:03.615251+0000","last_undegraded":"2024-02-12T17:16:03.615251+0000","last_fullsized":"2024-02-12T17:16:03.615251+0000","mapping_epoch":630,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":65,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-12T17:15:54.444127+0000","last_change":"2024-02-12T17:14:10.285684+0000","last_active":"2024-02-12T17:15:54.444127+0000","last_peered":"2024-02-12T17:15:54.444127+0000","last_clean":"2024-02-12T17:15:54.444127+0000","last_became_active":"2024-02-12T17:14:08.871051+0000","last_became_peered":"2024-02-12T17:14:08.871051+0000","last_unstale":"2024-02-12T17:15:54.444127+0000","last_undegraded":"2024-02-12T17:15:54.444127+0000","last_fullsized":"2024-02-12T17:15:54.444127+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:50.764686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.764686+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"698'14073","reported_seq":21088,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:08.391219+0000","last_change":"2024-02-12T17:14:39.687781+0000","last_active":"2024-02-12T17:16:08.391219+0000","last_peered":"2024-02-12T17:16:08.391219+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.886954+0000","last_became_peered":"2024-02-12T17:14:08.886954+0000","last_unstale":"2024-02-12T17:16:08.391219+0000","last_undegraded":"2024-02-12T17:14:08.719570+0000","last_fullsized":"2024-02-12T17:14:08.707803+0000","mapping_epoch":651,"log_start":"628'7678","ondisk_log_start":"628'7678","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6395,"ondisk_log_size":6395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":4731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1577,"num_objects_misplaced":1577,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14073,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3435,"num_bytes_recovered":225050624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,4],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5","objects":1577}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":704,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605043+0000","last_change":"2024-02-12T17:12:11.906816+0000","last_active":"2024-02-12T17:16:03.605043+0000","last_peered":"2024-02-12T17:16:03.605043+0000","last_clean":"2024-02-12T17:16:03.605043+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:16:03.605043+0000","last_undegraded":"2024-02-12T17:16:03.605043+0000","last_fullsized":"2024-02-12T17:16:03.605043+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:11.906737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:11.906737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":382,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.614909+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:16:03.614909+0000","last_peered":"2024-02-12T17:16:03.614909+0000","last_clean":"2024-02-12T17:16:03.614909+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:16:03.614909+0000","last_undegraded":"2024-02-12T17:16:03.614909+0000","last_fullsized":"2024-02-12T17:16:03.614909+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":822,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605042+0000","last_change":"2024-02-12T17:12:16.820857+0000","last_active":"2024-02-12T17:16:03.605042+0000","last_peered":"2024-02-12T17:16:03.605042+0000","last_clean":"2024-02-12T17:16:03.605042+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:16:03.605042+0000","last_undegraded":"2024-02-12T17:16:03.605042+0000","last_fullsized":"2024-02-12T17:16:03.605042+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:12:16.820782+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.820782+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"632'35","reported_seq":1196,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654372+0000","last_change":"2024-02-12T17:13:46.691596+0000","last_active":"2024-02-12T17:16:05.654372+0000","last_peered":"2024-02-12T17:16:05.654372+0000","last_clean":"2024-02-12T17:16:05.654372+0000","last_became_active":"2024-02-12T17:13:46.691213+0000","last_became_peered":"2024-02-12T17:13:46.691213+0000","last_unstale":"2024-02-12T17:16:05.654372+0000","last_undegraded":"2024-02-12T17:16:05.654372+0000","last_fullsized":"2024-02-12T17:16:05.654372+0000","mapping_epoch":631,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'29","last_scrub_stamp":"2024-02-12T17:12:47.788327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.788327+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1663303704,"num_objects":25381,"num_object_clones":0,"num_object_copies":76143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9814,"num_objects_misplaced":9809,"num_objects_unfound":0,"num_objects_dirty":25381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":225995,"num_write_kb":8043969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21509,"num_bytes_recovered":1409286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4814680064,"data_stored":4814667848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91398,"ondisk_log_size":91398,"up":48,"acting":40,"num_store_stats":8},{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":350,"num_read_kb":345,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":16216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1029,"ondisk_log_size":1029,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234305,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2349092,"kb_used_data":848732,"kb_used_omap":0,"kb_used_meta":1500352,"kb_avail":91396060,"statfs":{"total":95995035648,"available":93589565440,"internally_reserved":0,"allocated":869101568,"data_stored":867708857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1536360448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":2.6899999999999999,"5min":1.9910000000000001,"15min":2.077},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":15.856999999999999,"5min":15.856999999999999,"15min":16.759},"last":2.758},{"interface":"front","average":{"1min":2.79,"5min":1.837,"15min":1.915},"min":{"1min":0.27600000000000002,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":12.565,"5min":12.565,"15min":17.844000000000001},"last":2.7490000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:15:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4569999999999999,"5min":2.9220000000000002,"15min":2.8330000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.632999999999999,"5min":11.632999999999999,"15min":11.632999999999999},"last":0.746},{"interface":"front","average":{"1min":3.8100000000000001,"5min":2.7360000000000002,"15min":2.5569999999999999},"min":{"1min":0.29299999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":10.91,"5min":10.91,"15min":10.91},"last":3.9609999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":3.431,"5min":2.8359999999999999,"15min":3.1419999999999999},"min":{"1min":0.34599999999999997,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":23.611000000000001,"5min":42.162999999999997,"15min":218.35300000000001},"last":4.4379999999999997},{"interface":"front","average":{"1min":3.8090000000000002,"5min":2.8639999999999999,"15min":3.2269999999999999},"min":{"1min":0.47099999999999997,"5min":0.247,"15min":0.247},"max":{"1min":15.672000000000001,"5min":42.109999999999999,"15min":218.42099999999999},"last":2.7429999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":2.415,"15min":2.0939999999999999},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.255},"max":{"1min":7.5830000000000002,"5min":15.811999999999999,"15min":15.811999999999999},"last":11.741},{"interface":"front","average":{"1min":4.1710000000000003,"5min":2.847,"15min":2.5350000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":11.009,"5min":13.452,"15min":31.152000000000001},"last":12.019}]},{"osd":4,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.421,"15min":1.8540000000000001},"min":{"1min":0.247,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":3.4969999999999999,"5min":20.672999999999998,"15min":20.672999999999998},"last":2.7810000000000001},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.976,"15min":2.1509999999999998},"min":{"1min":0.29299999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.6769999999999996,"5min":38.390000000000001,"15min":38.390000000000001},"last":2.7240000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.8029999999999999,"15min":2.407},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":4.79,"5min":20.702000000000002,"15min":56.560000000000002},"last":2.7000000000000002},{"interface":"front","average":{"1min":2.5089999999999999,"5min":1.4690000000000001,"15min":2.1139999999999999},"min":{"1min":0.28499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":9.7170000000000005,"5min":10.669,"15min":53.963999999999999},"last":2.6930000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.4690000000000001,"15min":1.734},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":5.2539999999999996,"5min":21.620000000000001,"15min":21.620000000000001},"last":0.438},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.722,"15min":1.8759999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29199999999999998},"max":{"1min":3.7519999999999998,"5min":18.071000000000002,"15min":18.071000000000002},"last":2.7360000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528646,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712692,"kb_used_data":820332,"kb_used_omap":0,"kb_used_meta":892352,"kb_avail":92032460,"statfs":{"total":95995035648,"available":94241239040,"internally_reserved":0,"allocated":840019968,"data_stored":838611517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913768448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.7410000000000001,"15min":1.806},"min":{"1min":0.22,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.266,"5min":33.576999999999998,"15min":33.576999999999998},"last":3.3279999999999998},{"interface":"front","average":{"1min":2.194,"5min":2.0859999999999999,"15min":1.9239999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9740000000000002,"5min":33.569000000000003,"15min":33.569000000000003},"last":0.22700000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":2.1160000000000001,"15min":2.1000000000000001},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.6260000000000003,"5min":13.805999999999999,"15min":13.805999999999999},"last":5.7210000000000001},{"interface":"front","average":{"1min":1.9299999999999999,"5min":2.141,"15min":2.1760000000000002},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.6399999999999997,"5min":13.778,"15min":13.778},"last":5.694}]},{"osd":2,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8849999999999998,"5min":2.7290000000000001,"15min":2.6240000000000001},"min":{"1min":0.35399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.5760000000000005,"5min":33.594999999999999,"15min":56.798000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":4.3019999999999996,"5min":2.6680000000000001,"15min":2.3639999999999999},"min":{"1min":0.36099999999999999,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":14.045999999999999,"5min":33.526000000000003,"15min":56.780999999999999},"last":5.5720000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4489999999999998,"5min":3.1779999999999999,"15min":2.3999999999999999},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":12.336,"5min":58.420999999999999,"15min":58.420999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":3.7109999999999999,"5min":3.8290000000000002,"15min":2.8530000000000002},"min":{"1min":0.40400000000000003,"5min":0.254,"15min":0.254},"max":{"1min":11.737,"5min":58.709000000000003,"15min":58.709000000000003},"last":5.5940000000000003}]},{"osd":4,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.542,"15min":1.272},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.23499999999999999},"max":{"1min":5.0810000000000004,"5min":33.706000000000003,"15min":33.706000000000003},"last":0.314},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.8540000000000001,"15min":1.508},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":4.2460000000000004,"5min":34.895000000000003,"15min":34.895000000000003},"last":0.35699999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.6639999999999999,"15min":1.716},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.1970000000000001,"5min":14.56,"15min":34.869},"last":0.377},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.319,"15min":1.6799999999999999},"min":{"1min":0.36399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.1390000000000002,"5min":11.1,"15min":33.591000000000001},"last":0.36799999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.677,"15min":1.871},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":6.1310000000000002,"5min":35.122999999999998,"15min":35.122999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.7390000000000001,"15min":1.901},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.26500000000000001},"max":{"1min":11.081,"5min":34.045000000000002,"15min":34.045000000000002},"last":0.41899999999999998}]}]},{"osd":1,"up_from":630,"seq":2705829396509,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":550968,"kb_used_data":220464,"kb_used_omap":3,"kb_used_meta":330428,"kb_avail":93194184,"statfs":{"total":95995035648,"available":95430844416,"internally_reserved":0,"allocated":225755136,"data_stored":224335690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":338358333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8580000000000001,"5min":3.359,"15min":3.4430000000000001},"min":{"1min":0.33900000000000002,"5min":0.214,"15min":0.214},"max":{"1min":9.0109999999999992,"5min":14.67,"15min":14.67},"last":23.952999999999999},{"interface":"front","average":{"1min":4.609,"5min":3.6579999999999999,"15min":3.5},"min":{"1min":0.33800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":16.920999999999999,"5min":16.920999999999999,"15min":16.920999999999999},"last":13.003}]},{"osd":2,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":3.4220000000000002,"15min":3.528},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":7.5250000000000004,"5min":8.766,"15min":8.766},"last":12.856},{"interface":"front","average":{"1min":3.6619999999999999,"5min":3.5299999999999998,"15min":3.508},"min":{"1min":0.33100000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.303000000000001,"5min":14.303000000000001,"15min":14.303000000000001},"last":0.255}]},{"osd":3,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":5.3479999999999999,"5min":3.984,"15min":3.7559999999999998},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":18.577000000000002,"5min":18.577000000000002,"15min":18.577000000000002},"last":1.474},{"interface":"front","average":{"1min":4.7169999999999996,"5min":4.4009999999999998,"15min":4.3479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":17.527000000000001,"5min":17.527000000000001,"15min":17.527000000000001},"last":23.905999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.4359999999999999,"5min":2.5310000000000001,"15min":2.5470000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.3209999999999997,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":12.901999999999999},{"interface":"front","average":{"1min":2.6059999999999999,"5min":2.746,"15min":2.7690000000000001},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":8.8249999999999993,"5min":12.805999999999999,"15min":12.805999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.6309999999999998,"5min":3.0990000000000002,"15min":3.177},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.9120000000000008,"5min":12.769,"15min":12.769},"last":0.44700000000000001},{"interface":"front","average":{"1min":3.1459999999999999,"5min":2.319,"15min":2.181},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":11.15,"5min":11.15,"15min":11.15},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":3.0449999999999999,"5min":2.8730000000000002,"15min":2.8439999999999999},"min":{"1min":0.41599999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":11.115,"5min":12.720000000000001,"15min":12.720000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":2.4039999999999999,"5min":2.7829999999999999,"15min":2.8460000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.29,"5min":8.0890000000000004,"15min":8.0890000000000004},"last":12.929}]},{"osd":7,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.8460000000000001,"15min":2.9239999999999999},"min":{"1min":0.39000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":7.6669999999999998,"5min":12.829000000000001,"15min":12.829000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":3.4630000000000001,"5min":2.6059999999999999,"15min":2.4630000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":11.179,"5min":11.179,"15min":11.179},"last":3.1949999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738722,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1118468,"kb_used_data":214076,"kb_used_omap":0,"kb_used_meta":904384,"kb_avail":92626684,"statfs":{"total":95995035648,"available":94849724416,"internally_reserved":0,"allocated":219213824,"data_stored":217839256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":926089216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:15:49 2024","interfaces":[{"interface":"back","average":{"1min":4.2720000000000002,"5min":3.597,"15min":3.4849999999999999},"min":{"1min":0.25900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":14.048999999999999,"5min":14.048999999999999,"15min":14.048999999999999},"last":9.1929999999999996},{"interface":"front","average":{"1min":2.3919999999999999,"5min":2.7200000000000002,"15min":2.774},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":8.1120000000000001,"5min":13.787000000000001,"15min":13.787000000000001},"last":1.2130000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:15:31 2024","interfaces":[{"interface":"back","average":{"1min":3.6640000000000001,"5min":2.7759999999999998,"15min":2.0710000000000002},"min":{"1min":0.311,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":13.762,"5min":19.312000000000001,"15min":19.312000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":4.1500000000000004,"5min":3.0539999999999998,"15min":2.6019999999999999},"min":{"1min":0.39700000000000002,"5min":0.218,"15min":0.20799999999999999},"max":{"1min":22.959,"5min":22.959,"15min":26.039999999999999},"last":1.516}]},{"osd":3,"last update":"Mon Feb 12 17:15:45 2024","interfaces":[{"interface":"back","average":{"1min":9.375,"5min":3.629,"15min":3.1179999999999999},"min":{"1min":0.28599999999999998,"5min":0.186,"15min":0.186},"max":{"1min":61.177,"5min":61.177,"15min":87.311000000000007},"last":9.1069999999999993},{"interface":"front","average":{"1min":6.2800000000000002,"5min":3.6920000000000002,"15min":3.1400000000000001},"min":{"1min":0.45800000000000002,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":59.149999999999999,"5min":59.149999999999999,"15min":90.991},"last":1.242}]},{"osd":4,"last update":"Mon Feb 12 17:15:04 2024","interfaces":[{"interface":"back","average":{"1min":2.363,"5min":1.6000000000000001,"15min":1.5629999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.23599999999999999},"max":{"1min":13.861000000000001,"5min":15.141999999999999,"15min":16.888999999999999},"last":1.262},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.6030000000000002,"15min":2.1000000000000001},"min":{"1min":0.30499999999999999,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":13.503,"5min":26.082999999999998,"15min":26.082999999999998},"last":1.552}]},{"osd":5,"last update":"Mon Feb 12 17:15:39 2024","interfaces":[{"interface":"back","average":{"1min":2.1890000000000001,"5min":1.8740000000000001,"15min":2.2229999999999999},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":10.347,"5min":15.090999999999999,"15min":50.404000000000003},"last":9.1449999999999996},{"interface":"front","average":{"1min":2.0790000000000002,"5min":1.748,"15min":2.0030000000000001},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.74,"5min":15.164,"15min":49.979999999999997},"last":3.9649999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:15:04 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":3.4119999999999999,"15min":2.343},"min":{"1min":0.39900000000000002,"5min":0.35499999999999998,"15min":0.29999999999999999},"max":{"1min":13.48,"5min":48.780999999999999,"15min":48.780999999999999},"last":1.579},{"interface":"front","average":{"1min":2.2250000000000001,"5min":3.1280000000000001,"15min":2.3109999999999999},"min":{"1min":0.36399999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":13.962999999999999,"5min":49.067999999999998,"15min":49.067999999999998},"last":9.1699999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:15:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.04,"15min":2.04},"min":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":13.813000000000001,"5min":15.183999999999999,"15min":19.373000000000001},"last":3.992},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.3279999999999998,"15min":2.4430000000000001},"min":{"1min":0.38400000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":13.542,"5min":13.542,"15min":26.135000000000002},"last":1.6000000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411672,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2205020,"kb_used_data":706836,"kb_used_omap":0,"kb_used_meta":1498176,"kb_avail":91540132,"statfs":{"total":95995035648,"available":93737095168,"internally_reserved":0,"allocated":723800064,"data_stored":722416035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1534132224},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":3.2559999999999998,"5min":1.9690000000000001,"15min":1.8440000000000001},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17000000000000001},"max":{"1min":11.663,"5min":11.663,"15min":36.100999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":5.1219999999999999,"5min":2.4489999999999998,"15min":2.181},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":16.350000000000001,"5min":16.350000000000001,"15min":36.299999999999997},"last":1.077}]},{"osd":1,"last update":"Mon Feb 12 17:15:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5489999999999999,"5min":3.722,"15min":3.9180000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":9.3239999999999998,"5min":11.612,"15min":11.612},"last":0.754},{"interface":"front","average":{"1min":4.5579999999999998,"5min":5.6260000000000003,"15min":5.8040000000000003},"min":{"1min":0.42599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":11.141999999999999,"5min":22.52,"15min":22.52},"last":1.3600000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:16:00 2024","interfaces":[{"interface":"back","average":{"1min":5.5259999999999998,"5min":3.6930000000000001,"15min":2.9220000000000002},"min":{"1min":0.29099999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":30.303000000000001,"5min":30.303000000000001,"15min":40.759999999999998},"last":4.2960000000000003},{"interface":"front","average":{"1min":5.9139999999999997,"5min":3.262,"15min":2.5070000000000001},"min":{"1min":0.221,"5min":0.22,"15min":0.22},"max":{"1min":30.120999999999999,"5min":30.120999999999999,"15min":30.120999999999999},"last":4.3970000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:15:22 2024","interfaces":[{"interface":"back","average":{"1min":3.008,"5min":2.0129999999999999,"15min":1.98},"min":{"1min":0.27600000000000002,"5min":0.245,"15min":0.245},"max":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.8220000000000001,"5min":1.8939999999999999,"15min":2.0510000000000002},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.744,"5min":11.744,"15min":20.149000000000001},"last":1.0189999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":2.681,"5min":1.8700000000000001,"15min":2.0139999999999998},"min":{"1min":0.33900000000000002,"5min":0.217,"15min":0.217},"max":{"1min":11.581,"5min":11.581,"15min":32.57},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.881,"15min":2.1349999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.255},"max":{"1min":7.798,"5min":10.896000000000001,"15min":31.905999999999999},"last":1.2749999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:16:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.7829999999999999,"15min":1.8300000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.0919999999999996,"5min":10.714,"15min":20.966999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":2.5880000000000001,"5min":2.1949999999999998,"15min":1.8580000000000001},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.6130000000000004,"5min":11.834,"15min":20.193000000000001},"last":0.99099999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9590000000000001,"5min":2.121,"15min":2.4830000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":11.789,"5min":11.789,"15min":45.468000000000004},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.9990000000000001,"15min":2.4319999999999999},"min":{"1min":0.34499999999999997,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":7.8330000000000002,"5min":10.762,"15min":46.656999999999996},"last":1.335}]}]},{"osd":3,"up_from":26,"seq":111669150034,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1652224,"kb_used_data":758968,"kb_used_omap":0,"kb_used_meta":893248,"kb_avail":92092928,"statfs":{"total":95995035648,"available":94303158272,"internally_reserved":0,"allocated":777183232,"data_stored":775763492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914685952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":6.7190000000000003,"5min":3.738,"15min":2.8420000000000001},"min":{"1min":0.45500000000000002,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":58.451999999999998},"last":22.16},{"interface":"front","average":{"1min":4.351,"5min":3.7240000000000002,"15min":3.0409999999999999},"min":{"1min":0.40400000000000003,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":11.720000000000001,"5min":72.688000000000002,"15min":72.688000000000002},"last":15.439}]},{"osd":1,"last update":"Mon Feb 12 17:15:49 2024","interfaces":[{"interface":"back","average":{"1min":7.9610000000000003,"5min":5.1769999999999996,"15min":4.7130000000000001},"min":{"1min":0.27800000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":22.524000000000001,"5min":22.524000000000001,"15min":22.524000000000001},"last":5.4180000000000001},{"interface":"front","average":{"1min":5.6269999999999998,"5min":5.4889999999999999,"15min":5.4660000000000002},"min":{"1min":0.25600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":20.262,"5min":20.262,"15min":20.262},"last":15.706}]},{"osd":2,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":7.9420000000000002,"5min":3.5049999999999999,"15min":2.5659999999999998},"min":{"1min":0.245,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":38.034999999999997,"5min":40.954999999999998,"15min":40.954999999999998},"last":0.186},{"interface":"front","average":{"1min":6.7450000000000001,"5min":4.1289999999999996,"15min":2.9710000000000001},"min":{"1min":0.41399999999999998,"5min":0.22,"15min":0.22},"max":{"1min":22.138999999999999,"5min":46.408000000000001,"15min":46.408000000000001},"last":12.449999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:15:11 2024","interfaces":[{"interface":"back","average":{"1min":4.2880000000000003,"5min":3.3460000000000001,"15min":2.5169999999999999},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":8.7420000000000009,"5min":72.659000000000006,"15min":72.659000000000006},"last":15.494999999999999},{"interface":"front","average":{"1min":2.9500000000000002,"5min":2.758,"15min":2.0419999999999998},"min":{"1min":0.313,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":15.087999999999999,"5min":40.930999999999997,"15min":40.930999999999997},"last":0.25800000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:15:33 2024","interfaces":[{"interface":"back","average":{"1min":5.7649999999999997,"5min":3.1389999999999998,"15min":2.577},"min":{"1min":0.438,"5min":0.33000000000000002,"15min":0.253},"max":{"1min":17.306999999999999,"5min":17.306999999999999,"15min":17.940999999999999},"last":12.329000000000001},{"interface":"front","average":{"1min":4.2640000000000002,"5min":3.431,"15min":2.8500000000000001},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":11.586,"5min":72.620000000000005,"15min":72.620000000000005},"last":15.522}]},{"osd":6,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":5.2039999999999997,"5min":2.9510000000000001,"15min":2.153},"min":{"1min":0.31900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":22.489999999999998,"5min":40.889000000000003,"15min":40.889000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":5.1559999999999997,"5min":3.27,"15min":2.4649999999999999},"min":{"1min":0.39800000000000002,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":15.113,"5min":16.989999999999998,"15min":18.265000000000001},"last":12.414}]},{"osd":7,"last update":"Mon Feb 12 17:15:11 2024","interfaces":[{"interface":"back","average":{"1min":3.7810000000000001,"5min":3.5169999999999999,"15min":2.9940000000000002},"min":{"1min":0.378,"5min":0.375,"15min":0.315},"max":{"1min":8.6690000000000005,"5min":72.712999999999994,"15min":72.712999999999994},"last":15.467000000000001},{"interface":"front","average":{"1min":3.298,"5min":2.9420000000000002,"15min":2.778},"min":{"1min":0.439,"5min":0.38600000000000001,"15min":0.28299999999999997},"max":{"1min":8.2170000000000005,"5min":17.015000000000001,"15min":38.948},"last":12.372999999999999}]}]},{"osd":4,"up_from":0,"seq":137438953808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757580,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2098072,"kb_used_data":620944,"kb_used_omap":0,"kb_used_meta":1477120,"kb_avail":91647080,"statfs":{"total":95995035648,"available":93846609920,"internally_reserved":0,"allocated":635846656,"data_stored":634465967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1512570880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.696,"15min":1.7629999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":3.927,"5min":8.0350000000000001,"15min":46.365000000000002},"last":3.448},{"interface":"front","average":{"1min":2.133,"5min":1.659,"15min":1.9350000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.20100000000000001},"max":{"1min":7.157,"5min":15.368,"15min":46.134},"last":12.061}]},{"osd":1,"last update":"Mon Feb 12 17:15:52 2024","interfaces":[{"interface":"back","average":{"1min":2.512,"5min":1.9399999999999999,"15min":1.845},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.8710000000000004,"5min":9.484,"15min":9.484},"last":3.3719999999999999},{"interface":"front","average":{"1min":1.968,"5min":2.8450000000000002,"15min":2.9910000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":6.7210000000000001,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":10.412000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":2.6000000000000001,"15min":2.036},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":9.0269999999999992,"5min":57.390999999999998,"15min":57.390999999999998},"last":3.415},{"interface":"front","average":{"1min":2.7719999999999998,"5min":2.843,"15min":2.0219999999999998},"min":{"1min":0.32100000000000001,"5min":0.214,"15min":0.214},"max":{"1min":13.366,"5min":57.134999999999998,"15min":57.134999999999998},"last":2.1549999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":2.0030000000000001,"5min":3.996,"15min":2.6920000000000002},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":7.1630000000000003,"5min":57.170999999999999,"15min":57.170999999999999},"last":11.871},{"interface":"front","average":{"1min":3.3530000000000002,"5min":3.4420000000000002,"15min":2.5579999999999998},"min":{"1min":0.27900000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.3829999999999991,"5min":57.484999999999999,"15min":57.484999999999999},"last":8.0429999999999993}]},{"osd":4,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.169,"15min":1.556},"min":{"1min":0.29799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.9939999999999998,"5min":9.0429999999999993,"15min":12.656000000000001},"last":2.7639999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.4370000000000001,"15min":1.573},"min":{"1min":0.255,"5min":0.255,"15min":0.254},"max":{"1min":3.3250000000000002,"5min":10.619999999999999,"15min":20.550000000000001},"last":4.9580000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:16:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.5030000000000001,"5min":14.675000000000001,"15min":14.675000000000001},"last":2.5299999999999998},{"interface":"front","average":{"1min":1.651,"5min":1.802,"15min":1.5629999999999999},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":5.2720000000000002,"5min":12.864000000000001,"15min":13.916},"last":4.8140000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.5669999999999999,"15min":1.8180000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":5.8289999999999997,"5min":14.337999999999999,"15min":19.184999999999999},"last":6.0170000000000003},{"interface":"front","average":{"1min":1.454,"5min":1.5449999999999999,"15min":1.8819999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":6.6950000000000003,"5min":14.637,"15min":20.274000000000001},"last":2.9159999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214761472,"data_stored":214761472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":222167040,"data_stored":222167040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":714674176,"data_stored":714670104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":765657088,"data_stored":765657088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":600182784,"data_stored":600178712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":622460928,"data_stored":622460928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":817500160,"data_stored":817496088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":857276416,"data_stored":857276416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:16:12.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:12.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:11.117009+0000 mgr.y (mgr.24481) 998 : cluster [DBG] pgmap v1416: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 649 op/s; 9814/76719 objects degraded (12.792%); 9815/76719 objects misplaced (12.793%); 1.9 MiB/s, 31 objects/s recovering 2024-02-12T17:16:12.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:11.766085+0000 mon.a (mon.0) 2237 : cluster [WRN] Health check update: Degraded data redundancy: 9814/76719 objects degraded (12.792%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:12.991 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:16:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:16:13.020 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:13.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:11.117009+0000 mgr.y (mgr.24481) 998 : cluster [DBG] pgmap v1416: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 649 op/s; 9814/76719 objects degraded (12.792%); 9815/76719 objects misplaced (12.793%); 1.9 MiB/s, 31 objects/s recovering 2024-02-12T17:16:13.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:11.766085+0000 mon.a (mon.0) 2237 : cluster [WRN] Health check update: Degraded data redundancy: 9814/76719 objects degraded (12.792%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:13.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:11.117009+0000 mgr.y (mgr.24481) 998 : cluster [DBG] pgmap v1416: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 649 op/s; 9814/76719 objects degraded (12.792%); 9815/76719 objects misplaced (12.793%); 1.9 MiB/s, 31 objects/s recovering 2024-02-12T17:16:13.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:11.766085+0000 mon.a (mon.0) 2237 : cluster [WRN] Health check update: Degraded data redundancy: 9814/76719 objects degraded (12.792%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:13.046 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:13.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:13.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:13.237 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:14.006 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:14.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:14.114 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:14.212 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:14.696 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:14.699 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:14.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:14.804 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:14.858 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:14.887 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:14.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:13.118941+0000 mgr.y (mgr.24481) 999 : cluster [DBG] pgmap v1417: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 652 op/s; 9255/73056 objects degraded (12.668%); 9615/73056 objects misplaced (13.161%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:16:14.992 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:15.037 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:13.118941+0000 mgr.y (mgr.24481) 999 : cluster [DBG] pgmap v1417: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 652 op/s; 9255/73056 objects degraded (12.668%); 9615/73056 objects misplaced (13.161%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:16:15.038 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:13.118941+0000 mgr.y (mgr.24481) 999 : cluster [DBG] pgmap v1417: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 652 op/s; 9255/73056 objects degraded (12.668%); 9615/73056 objects misplaced (13.161%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T17:16:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:15.207 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:15.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:15.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:15.348 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:15.376 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:15.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:15.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:16:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:16:15.483 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:15.633 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:15.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:15.743 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:15.782 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:15.813 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:15.112469+0000 mgr.y (mgr.24481) 1000 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:15.813 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:15.119809+0000 mgr.y (mgr.24481) 1001 : cluster [DBG] pgmap v1418: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 632 op/s; 8361/67140 objects degraded (12.453%); 8466/67140 objects misplaced (12.609%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:15.815 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:15.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:15.907 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:16:15.923 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:16.025 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:16:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-02-12T17:16:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-02-12T17:16:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-02-12T17:16:16.026 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:16:16.037 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:16.194 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:15.112469+0000 mgr.y (mgr.24481) 1000 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:16.194 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:15.119809+0000 mgr.y (mgr.24481) 1001 : cluster [DBG] pgmap v1418: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 632 op/s; 8361/67140 objects degraded (12.453%); 8466/67140 objects misplaced (12.609%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:15.112469+0000 mgr.y (mgr.24481) 1000 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:16.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:15.119809+0000 mgr.y (mgr.24481) 1001 : cluster [DBG] pgmap v1418: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 632 op/s; 8361/67140 objects degraded (12.453%); 8466/67140 objects misplaced (12.609%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T17:16:17.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:16.767161+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check update: Degraded data redundancy: 8361/67140 objects degraded (12.453%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:17.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:16.767161+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check update: Degraded data redundancy: 8361/67140 objects degraded (12.453%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:17.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:16.767161+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check update: Degraded data redundancy: 8361/67140 objects degraded (12.453%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:17.651 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:17.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:17.759 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:17.848 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:17.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:18.043 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:18.108 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:18.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:18.134 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:16:18.174 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:16:18.210 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:16:18.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:18.242 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:18.275 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:18.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:18.290 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:16:18.317 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:16:18.364 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:16:18.383 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:18.407 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:16:18.424 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:18.449 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:18.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:18.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:18.456 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:16:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:16:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:16:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:16:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:16:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:16:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:16:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:16:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:16:18.557 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:18.621 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:18.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:17.124964+0000 mgr.y (mgr.24481) 1002 : cluster [DBG] pgmap v1419: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 893 B/s rd, 0 B/s wr, 658 op/s; 7820/63489 objects degraded (12.317%); 8289/63489 objects misplaced (13.056%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T17:16:18.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:17.546794+0000 mon.a (mon.0) 2239 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:16:18.742 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:18.066301+0000 mon.a (mon.0) 2240 : audit [DBG] from='client.? v1:172.21.15.33:0/2918106119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:18.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:17.124964+0000 mgr.y (mgr.24481) 1002 : cluster [DBG] pgmap v1419: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 893 B/s rd, 0 B/s wr, 658 op/s; 7820/63489 objects degraded (12.317%); 8289/63489 objects misplaced (13.056%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T17:16:18.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:17.546794+0000 mon.a (mon.0) 2239 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:16:18.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:18.066301+0000 mon.a (mon.0) 2240 : audit [DBG] from='client.? v1:172.21.15.33:0/2918106119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:18.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:17.124964+0000 mgr.y (mgr.24481) 1002 : cluster [DBG] pgmap v1419: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 893 B/s rd, 0 B/s wr, 658 op/s; 7820/63489 objects degraded (12.317%); 8289/63489 objects misplaced (13.056%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T17:16:18.783 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:17.546794+0000 mon.a (mon.0) 2239 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:16:18.784 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:18.066301+0000 mon.a (mon.0) 2240 : audit [DBG] from='client.? v1:172.21.15.33:0/2918106119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:20.193 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:20.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:20.221 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T17:16:20.300 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:20.332 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:20.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:20.464 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:20.497 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:20.524 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:20.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:20.630 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:20.716 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:19.129487+0000 mgr.y (mgr.24481) 1003 : cluster [DBG] pgmap v1420: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 693 op/s; 6952/57582 objects degraded (12.073%); 7139/57582 objects misplaced (12.398%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:16:20.716 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:19.129487+0000 mgr.y (mgr.24481) 1003 : cluster [DBG] pgmap v1420: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 693 op/s; 6952/57582 objects degraded (12.073%); 7139/57582 objects misplaced (12.398%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:16:20.728 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:20.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:19.129487+0000 mgr.y (mgr.24481) 1003 : cluster [DBG] pgmap v1420: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 693 op/s; 6952/57582 objects degraded (12.073%); 7139/57582 objects misplaced (12.398%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:16:20.801 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:20.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:20.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:20.940 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:20.967 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:20.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:21.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:16:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:20] "GET /metrics HTTP/1.1" 200 176850 "" "Prometheus/2.33.4" 2024-02-12T17:16:21.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:21.110 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:21.133 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:21.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:21.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:21.270 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:21.297 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:21.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:21.406 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:21.481 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:21.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:21.586 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:21.667 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:22.074 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:22.080 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:22.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:22.181 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:16:22.267 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:22.315 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:16:22.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:22.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:16:22.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:22.451 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:22.478 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:22.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:22.584 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:22.738 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:22.739 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:16:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:16:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:21.130035+0000 mgr.y (mgr.24481) 1004 : cluster [DBG] pgmap v1421: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 532 op/s; 6952/57582 objects degraded (12.073%); 7139/57582 objects misplaced (12.398%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:16:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:21.769717+0000 mon.a (mon.0) 2241 : cluster [WRN] Health check update: Degraded data redundancy: 6952/57582 objects degraded (12.073%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:22.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:22.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:22.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:21.130035+0000 mgr.y (mgr.24481) 1004 : cluster [DBG] pgmap v1421: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 532 op/s; 6952/57582 objects degraded (12.073%); 7139/57582 objects misplaced (12.398%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:16:22.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:21.769717+0000 mon.a (mon.0) 2241 : cluster [WRN] Health check update: Degraded data redundancy: 6952/57582 objects degraded (12.073%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:22.946 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:21.130035+0000 mgr.y (mgr.24481) 1004 : cluster [DBG] pgmap v1421: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 532 op/s; 6952/57582 objects degraded (12.073%); 7139/57582 objects misplaced (12.398%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T17:16:22.946 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:21.769717+0000 mon.a (mon.0) 2241 : cluster [WRN] Health check update: Degraded data redundancy: 6952/57582 objects degraded (12.073%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:22.953 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:23.297 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:23.342 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:23.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:23.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:23.490 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:23.521 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:23.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:23.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:23.656 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-02-12T17:16:23.722 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:23.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:22.888350+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? v1:172.21.15.33:0/341092984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:16:23.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T17:16:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:22.888350+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? v1:172.21.15.33:0/341092984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:16:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:22.888350+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? v1:172.21.15.33:0/341092984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T17:16:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:23.130756+0000 mgr.y (mgr.24481) 1005 : cluster [DBG] pgmap v1422: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 634 op/s; 6449/53892 objects degraded (11.967%); 6962/53892 objects misplaced (12.918%); 3.0 MiB/s, 47 objects/s recovering 2024-02-12T17:16:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:23.616373+0000 mon.a (mon.0) 2243 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:23.634621+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? v1:172.21.15.33:0/341092984' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:16:24.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:23.634668+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2024-02-12T17:16:25.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:23.130756+0000 mgr.y (mgr.24481) 1005 : cluster [DBG] pgmap v1422: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 634 op/s; 6449/53892 objects degraded (11.967%); 6962/53892 objects misplaced (12.918%); 3.0 MiB/s, 47 objects/s recovering 2024-02-12T17:16:25.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:23.616373+0000 mon.a (mon.0) 2243 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:25.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:23.634621+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? v1:172.21.15.33:0/341092984' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:16:25.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:23.634668+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2024-02-12T17:16:25.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:23.130756+0000 mgr.y (mgr.24481) 1005 : cluster [DBG] pgmap v1422: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 1.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 634 op/s; 6449/53892 objects degraded (11.967%); 6962/53892 objects misplaced (12.918%); 3.0 MiB/s, 47 objects/s recovering 2024-02-12T17:16:25.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:23.616373+0000 mon.a (mon.0) 2243 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:25.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:23.634621+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? v1:172.21.15.33:0/341092984' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T17:16:25.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:23.634668+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2024-02-12T17:16:25.493 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:16:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:16:26.181 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:26.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:26.290 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:26.390 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:26.431 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:26.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:26.536 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:26.567 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:26.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:26.702 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:26.773 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:26.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:26.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:25.123165+0000 mgr.y (mgr.24481) 1006 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:26.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:25.132676+0000 mgr.y (mgr.24481) 1007 : cluster [DBG] pgmap v1424: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 983 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 646 op/s; 5571/47757 objects degraded (11.665%); 5794/47757 objects misplaced (12.132%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:26.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:25.123165+0000 mgr.y (mgr.24481) 1006 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:26.782 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:25.132676+0000 mgr.y (mgr.24481) 1007 : cluster [DBG] pgmap v1424: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 983 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 646 op/s; 5571/47757 objects degraded (11.665%); 5794/47757 objects misplaced (12.132%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:26.878 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:26.907 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:25.123165+0000 mgr.y (mgr.24481) 1006 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:26.908 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:25.132676+0000 mgr.y (mgr.24481) 1007 : cluster [DBG] pgmap v1424: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 983 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 646 op/s; 5571/47757 objects degraded (11.665%); 5794/47757 objects misplaced (12.132%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:26.910 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:26.946 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:26.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:27.052 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:16:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:27.323 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:16:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:27.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:16:27.430 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:27.550 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:27.664 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:16:27.702 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:16:27.927 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T17:16:28.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:26.771293+0000 mon.a (mon.0) 2246 : cluster [WRN] Health check update: Degraded data redundancy: 5571/47757 objects degraded (11.665%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:28.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:26.783528+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e700: 8 total, 8 up, 7 in 2024-02-12T17:16:28.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:27.133382+0000 mgr.y (mgr.24481) 1008 : cluster [DBG] pgmap v1426: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 909 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 558 op/s; 5128/44202 objects degraded (11.601%); 5634/44202 objects misplaced (12.746%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:28.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:27.641543+0000 mgr.y (mgr.24481) 1009 : audit [DBG] from='client.36233 v1:172.21.15.33:0/906612366' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:28.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:26.771293+0000 mon.a (mon.0) 2246 : cluster [WRN] Health check update: Degraded data redundancy: 5571/47757 objects degraded (11.665%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:28.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:26.783528+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e700: 8 total, 8 up, 7 in 2024-02-12T17:16:28.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:27.133382+0000 mgr.y (mgr.24481) 1008 : cluster [DBG] pgmap v1426: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 909 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 558 op/s; 5128/44202 objects degraded (11.601%); 5634/44202 objects misplaced (12.746%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:28.031 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:27.641543+0000 mgr.y (mgr.24481) 1009 : audit [DBG] from='client.36233 v1:172.21.15.33:0/906612366' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:28.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:26.771293+0000 mon.a (mon.0) 2246 : cluster [WRN] Health check update: Degraded data redundancy: 5571/47757 objects degraded (11.665%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:28.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:26.783528+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e700: 8 total, 8 up, 7 in 2024-02-12T17:16:28.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:27.133382+0000 mgr.y (mgr.24481) 1008 : cluster [DBG] pgmap v1426: 118 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 909 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 558 op/s; 5128/44202 objects degraded (11.601%); 5634/44202 objects misplaced (12.746%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:28.243 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:27.641543+0000 mgr.y (mgr.24481) 1009 : audit [DBG] from='client.36233 v1:172.21.15.33:0/906612366' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:29.081 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:29.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:29.189 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:29.263 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:29.734 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:29.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:29.755 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:29.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:29.937 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:30.211 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:29.134642+0000 mgr.y (mgr.24481) 1010 : cluster [DBG] pgmap v1427: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 779 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 816 op/s; 4218/37980 objects degraded (11.106%); 4459/37980 objects misplaced (11.740%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T17:16:30.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:29.134642+0000 mgr.y (mgr.24481) 1010 : cluster [DBG] pgmap v1427: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 779 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 816 op/s; 4218/37980 objects degraded (11.106%); 4459/37980 objects misplaced (11.740%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T17:16:30.524 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:30.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:30.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:29.134642+0000 mgr.y (mgr.24481) 1010 : cluster [DBG] pgmap v1427: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 779 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 816 op/s; 4218/37980 objects degraded (11.106%); 4459/37980 objects misplaced (11.740%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T17:16:30.629 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:30.697 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1426,"stamp":"2024-02-12T17:16:27.132837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":952964796,"num_objects":14734,"num_object_clones":0,"num_object_copies":44202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5128,"num_objects_misplaced":5634,"num_objects_unfound":0,"num_objects_dirty":14734,"num_whiteouts":0,"num_read":1904,"num_read_kb":1656,"num_write":237834,"num_write_kb":8044010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23122,"num_bytes_recovered":1460034336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92396,"ondisk_log_size":92396,"up":354,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":368,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10205008,"kb_used_data":2712856,"kb_used_omap":3,"kb_used_meta":7491964,"kb_avail":646011056,"statfs":{"total":671965249536,"available":661515321344,"internally_reserved":0,"allocated":2777964544,"data_stored":2768154872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":7671771197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-292290560,"num_objects":-4460,"num_object_clones":0,"num_object_copies":-13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1824,"num_objects_misplaced":-1505,"num_objects_unfound":0,"num_objects_dirty":-4460,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4460,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":23592960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003937"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":424,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.694398+0000","last_change":"2024-02-12T17:14:10.348720+0000","last_active":"2024-02-12T17:16:05.694398+0000","last_peered":"2024-02-12T17:16:05.694398+0000","last_clean":"2024-02-12T17:16:05.694398+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:16:05.694398+0000","last_undegraded":"2024-02-12T17:16:05.694398+0000","last_fullsized":"2024-02-12T17:16:05.694398+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":207,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.663795+0000","last_change":"2024-02-12T17:13:46.694822+0000","last_active":"2024-02-12T17:16:05.663795+0000","last_peered":"2024-02-12T17:16:05.663795+0000","last_clean":"2024-02-12T17:16:05.663795+0000","last_became_active":"2024-02-12T17:13:46.694448+0000","last_became_peered":"2024-02-12T17:13:46.694448+0000","last_unstale":"2024-02-12T17:16:05.663795+0000","last_undegraded":"2024-02-12T17:16:05.663795+0000","last_fullsized":"2024-02-12T17:16:05.663795+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:12:55.862345+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:12:55.862345+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":451,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.696154+0000","last_change":"2024-02-12T17:13:46.787428+0000","last_active":"2024-02-12T17:16:05.696154+0000","last_peered":"2024-02-12T17:16:05.696154+0000","last_clean":"2024-02-12T17:16:05.696154+0000","last_became_active":"2024-02-12T17:13:46.786996+0000","last_became_peered":"2024-02-12T17:13:46.786996+0000","last_unstale":"2024-02-12T17:16:05.696154+0000","last_undegraded":"2024-02-12T17:16:05.696154+0000","last_fullsized":"2024-02-12T17:16:05.696154+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:13:12.016227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:12.016227+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":479,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697448+0000","last_change":"2024-02-12T17:14:09.685503+0000","last_active":"2024-02-12T17:16:05.697448+0000","last_peered":"2024-02-12T17:16:05.697448+0000","last_clean":"2024-02-12T17:16:05.697448+0000","last_became_active":"2024-02-12T17:14:08.761132+0000","last_became_peered":"2024-02-12T17:14:08.761132+0000","last_unstale":"2024-02-12T17:16:05.697448+0000","last_undegraded":"2024-02-12T17:16:05.697448+0000","last_fullsized":"2024-02-12T17:16:05.697448+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:54.052006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.052006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":494,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.694416+0000","last_change":"2024-02-12T17:13:46.753228+0000","last_active":"2024-02-12T17:16:05.694416+0000","last_peered":"2024-02-12T17:16:05.694416+0000","last_clean":"2024-02-12T17:16:05.694416+0000","last_became_active":"2024-02-12T17:13:46.752755+0000","last_became_peered":"2024-02-12T17:13:46.752755+0000","last_unstale":"2024-02-12T17:16:05.694416+0000","last_undegraded":"2024-02-12T17:16:05.694416+0000","last_fullsized":"2024-02-12T17:16:05.694416+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":1540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":159,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.699745+0000","last_change":"2024-02-12T17:13:46.607814+0000","last_active":"2024-02-12T17:16:05.699745+0000","last_peered":"2024-02-12T17:16:05.699745+0000","last_clean":"2024-02-12T17:16:05.699745+0000","last_became_active":"2024-02-12T17:13:46.606367+0000","last_became_peered":"2024-02-12T17:13:46.606367+0000","last_unstale":"2024-02-12T17:16:05.699745+0000","last_undegraded":"2024-02-12T17:16:05.699745+0000","last_fullsized":"2024-02-12T17:16:05.699745+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":234,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.664802+0000","last_change":"2024-02-12T17:13:46.567244+0000","last_active":"2024-02-12T17:16:05.664802+0000","last_peered":"2024-02-12T17:16:05.664802+0000","last_clean":"2024-02-12T17:16:05.664802+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:16:05.664802+0000","last_undegraded":"2024-02-12T17:16:05.664802+0000","last_fullsized":"2024-02-12T17:16:05.664802+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":69,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.682908+0000","last_change":"2024-02-12T17:14:08.993489+0000","last_active":"2024-02-12T17:16:05.682908+0000","last_peered":"2024-02-12T17:16:05.682908+0000","last_clean":"2024-02-12T17:16:05.682908+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:16:05.682908+0000","last_undegraded":"2024-02-12T17:16:05.682908+0000","last_fullsized":"2024-02-12T17:16:05.682908+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":422,"reported_epoch":698,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:05.654409+0000","last_change":"2024-02-12T17:14:09.723827+0000","last_active":"2024-02-12T17:16:05.654409+0000","last_peered":"2024-02-12T17:16:05.654409+0000","last_clean":"2024-02-12T17:14:05.085957+0000","last_became_active":"2024-02-12T17:14:09.723480+0000","last_became_peered":"2024-02-12T17:14:09.723480+0000","last_unstale":"2024-02-12T17:16:05.654409+0000","last_undegraded":"2024-02-12T17:16:05.654409+0000","last_fullsized":"2024-02-12T17:16:05.654409+0000","mapping_epoch":652,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":392,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.696227+0000","last_change":"2024-02-12T17:14:08.755098+0000","last_active":"2024-02-12T17:16:05.696227+0000","last_peered":"2024-02-12T17:16:05.696227+0000","last_clean":"2024-02-12T17:16:05.696227+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:16:05.696227+0000","last_undegraded":"2024-02-12T17:16:05.696227+0000","last_fullsized":"2024-02-12T17:16:05.696227+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":415,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.694588+0000","last_change":"2024-02-12T17:12:12.141297+0000","last_active":"2024-02-12T17:16:05.694588+0000","last_peered":"2024-02-12T17:16:05.694588+0000","last_clean":"2024-02-12T17:16:05.694588+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:16:05.694588+0000","last_undegraded":"2024-02-12T17:16:05.694588+0000","last_fullsized":"2024-02-12T17:16:05.694588+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.141204+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":244,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.665921+0000","last_change":"2024-02-12T17:14:08.743209+0000","last_active":"2024-02-12T17:16:05.665921+0000","last_peered":"2024-02-12T17:16:05.665921+0000","last_clean":"2024-02-12T17:16:05.665921+0000","last_became_active":"2024-02-12T17:14:08.742886+0000","last_became_peered":"2024-02-12T17:14:08.742886+0000","last_unstale":"2024-02-12T17:16:05.665921+0000","last_undegraded":"2024-02-12T17:16:05.665921+0000","last_fullsized":"2024-02-12T17:16:05.665921+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.783527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.783527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":693,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654478+0000","last_change":"2024-02-12T17:13:21.591627+0000","last_active":"2024-02-12T17:16:05.654478+0000","last_peered":"2024-02-12T17:16:05.654478+0000","last_clean":"2024-02-12T17:16:05.654478+0000","last_became_active":"2024-02-12T17:13:21.591246+0000","last_became_peered":"2024-02-12T17:13:21.591246+0000","last_unstale":"2024-02-12T17:16:05.654478+0000","last_undegraded":"2024-02-12T17:16:05.654478+0000","last_fullsized":"2024-02-12T17:16:05.654478+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.028495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":437,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.681910+0000","last_change":"2024-02-12T17:13:46.558817+0000","last_active":"2024-02-12T17:16:05.681910+0000","last_peered":"2024-02-12T17:16:05.681910+0000","last_clean":"2024-02-12T17:16:05.681910+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:16:05.681910+0000","last_undegraded":"2024-02-12T17:16:05.681910+0000","last_fullsized":"2024-02-12T17:16:05.681910+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":91,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697657+0000","last_change":"2024-02-12T17:13:46.576676+0000","last_active":"2024-02-12T17:16:05.697657+0000","last_peered":"2024-02-12T17:16:05.697657+0000","last_clean":"2024-02-12T17:16:05.697657+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:16:05.697657+0000","last_undegraded":"2024-02-12T17:16:05.697657+0000","last_fullsized":"2024-02-12T17:16:05.697657+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":374,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.726488+0000","last_change":"2024-02-12T17:14:08.763317+0000","last_active":"2024-02-12T17:16:05.726488+0000","last_peered":"2024-02-12T17:16:05.726488+0000","last_clean":"2024-02-12T17:16:05.726488+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:16:05.726488+0000","last_undegraded":"2024-02-12T17:16:05.726488+0000","last_fullsized":"2024-02-12T17:16:05.726488+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":131,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.695090+0000","last_change":"2024-02-12T17:14:08.742170+0000","last_active":"2024-02-12T17:16:05.695090+0000","last_peered":"2024-02-12T17:16:05.695090+0000","last_clean":"2024-02-12T17:16:05.695090+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:16:05.695090+0000","last_undegraded":"2024-02-12T17:16:05.695090+0000","last_fullsized":"2024-02-12T17:16:05.695090+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":212,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.665589+0000","last_change":"2024-02-12T17:13:46.784204+0000","last_active":"2024-02-12T17:16:05.665589+0000","last_peered":"2024-02-12T17:16:05.665589+0000","last_clean":"2024-02-12T17:16:05.665589+0000","last_became_active":"2024-02-12T17:13:46.783737+0000","last_became_peered":"2024-02-12T17:13:46.783737+0000","last_unstale":"2024-02-12T17:16:05.665589+0000","last_undegraded":"2024-02-12T17:16:05.665589+0000","last_fullsized":"2024-02-12T17:16:05.665589+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:15.817751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.817751+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":761,"reported_epoch":698,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:05.695585+0000","last_change":"2024-02-12T17:14:40.177468+0000","last_active":"2024-02-12T17:16:05.695585+0000","last_peered":"2024-02-12T17:16:05.695585+0000","last_clean":"2024-02-12T17:13:44.507959+0000","last_became_active":"2024-02-12T17:14:08.758564+0000","last_became_peered":"2024-02-12T17:14:08.758564+0000","last_unstale":"2024-02-12T17:16:05.695585+0000","last_undegraded":"2024-02-12T17:16:05.695585+0000","last_fullsized":"2024-02-12T17:16:05.695585+0000","mapping_epoch":651,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":57,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.670375+0000","last_change":"2024-02-12T17:14:08.762208+0000","last_active":"2024-02-12T17:16:05.670375+0000","last_peered":"2024-02-12T17:16:05.670375+0000","last_clean":"2024-02-12T17:16:05.670375+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:16:05.670375+0000","last_undegraded":"2024-02-12T17:16:05.670375+0000","last_fullsized":"2024-02-12T17:16:05.670375+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":202,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.664281+0000","last_change":"2024-02-12T17:12:02.752352+0000","last_active":"2024-02-12T17:16:05.664281+0000","last_peered":"2024-02-12T17:16:05.664281+0000","last_clean":"2024-02-12T17:16:05.664281+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:16:05.664281+0000","last_undegraded":"2024-02-12T17:16:05.664281+0000","last_fullsized":"2024-02-12T17:16:05.664281+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:02.752280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:02.752280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":163,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697527+0000","last_change":"2024-02-12T17:13:46.582722+0000","last_active":"2024-02-12T17:16:05.697527+0000","last_peered":"2024-02-12T17:16:05.697527+0000","last_clean":"2024-02-12T17:16:05.697527+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:16:05.697527+0000","last_undegraded":"2024-02-12T17:16:05.697527+0000","last_fullsized":"2024-02-12T17:16:05.697527+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":798,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:22.692520+0000","last_change":"2024-02-12T17:12:58.020311+0000","last_active":"2024-02-12T17:16:22.692520+0000","last_peered":"2024-02-12T17:16:22.692520+0000","last_clean":"2024-02-12T17:16:22.692520+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:16:22.692520+0000","last_undegraded":"2024-02-12T17:16:22.692520+0000","last_fullsized":"2024-02-12T17:16:22.692520+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:12:58.018712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.018712+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":57,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.672122+0000","last_change":"2024-02-12T17:14:08.872636+0000","last_active":"2024-02-12T17:16:05.672122+0000","last_peered":"2024-02-12T17:16:05.672122+0000","last_clean":"2024-02-12T17:16:05.672122+0000","last_became_active":"2024-02-12T17:14:08.872123+0000","last_became_peered":"2024-02-12T17:14:08.872123+0000","last_unstale":"2024-02-12T17:16:05.672122+0000","last_undegraded":"2024-02-12T17:16:05.672122+0000","last_fullsized":"2024-02-12T17:16:05.672122+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:06.904354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:06.904354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":788,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605913+0000","last_change":"2024-02-12T17:12:12.838640+0000","last_active":"2024-02-12T17:16:03.605913+0000","last_peered":"2024-02-12T17:16:03.605913+0000","last_clean":"2024-02-12T17:16:03.605913+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:16:03.605913+0000","last_undegraded":"2024-02-12T17:16:03.605913+0000","last_fullsized":"2024-02-12T17:16:03.605913+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:12.838563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.838563+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"655'24","reported_seq":794,"reported_epoch":698,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:17.690976+0000","last_change":"2024-02-12T17:14:09.729342+0000","last_active":"2024-02-12T17:16:17.690976+0000","last_peered":"2024-02-12T17:16:17.690976+0000","last_clean":"2024-02-12T17:14:07.616779+0000","last_became_active":"2024-02-12T17:14:09.728969+0000","last_became_peered":"2024-02-12T17:14:09.728969+0000","last_unstale":"2024-02-12T17:16:17.690976+0000","last_undegraded":"2024-02-12T17:16:17.690976+0000","last_fullsized":"2024-02-12T17:16:17.690976+0000","mapping_epoch":652,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":692,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605130+0000","last_change":"2024-02-12T17:12:54.656523+0000","last_active":"2024-02-12T17:16:03.605130+0000","last_peered":"2024-02-12T17:16:03.605130+0000","last_clean":"2024-02-12T17:16:03.605130+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:16:03.605130+0000","last_undegraded":"2024-02-12T17:16:03.605130+0000","last_fullsized":"2024-02-12T17:16:03.605130+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:54.656456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.656456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":428,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697830+0000","last_change":"2024-02-12T17:13:46.577769+0000","last_active":"2024-02-12T17:16:05.697830+0000","last_peered":"2024-02-12T17:16:05.697830+0000","last_clean":"2024-02-12T17:16:05.697830+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:16:05.697830+0000","last_undegraded":"2024-02-12T17:16:05.697830+0000","last_fullsized":"2024-02-12T17:16:05.697830+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":715,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.658069+0000","last_change":"2024-02-12T17:13:21.591810+0000","last_active":"2024-02-12T17:16:05.658069+0000","last_peered":"2024-02-12T17:16:05.658069+0000","last_clean":"2024-02-12T17:16:05.658069+0000","last_became_active":"2024-02-12T17:13:21.591502+0000","last_became_peered":"2024-02-12T17:13:21.591502+0000","last_unstale":"2024-02-12T17:16:05.658069+0000","last_undegraded":"2024-02-12T17:16:05.658069+0000","last_fullsized":"2024-02-12T17:16:05.658069+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.882066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.882066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":236,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.665804+0000","last_change":"2024-02-12T17:13:46.651066+0000","last_active":"2024-02-12T17:16:05.665804+0000","last_peered":"2024-02-12T17:16:05.665804+0000","last_clean":"2024-02-12T17:16:05.665804+0000","last_became_active":"2024-02-12T17:13:46.650776+0000","last_became_peered":"2024-02-12T17:13:46.650776+0000","last_unstale":"2024-02-12T17:16:05.665804+0000","last_undegraded":"2024-02-12T17:16:05.665804+0000","last_fullsized":"2024-02-12T17:16:05.665804+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:50.786900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.786900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":417,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.695408+0000","last_change":"2024-02-12T17:13:46.822111+0000","last_active":"2024-02-12T17:16:05.695408+0000","last_peered":"2024-02-12T17:16:05.695408+0000","last_clean":"2024-02-12T17:16:05.695408+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:16:05.695408+0000","last_undegraded":"2024-02-12T17:16:05.695408+0000","last_fullsized":"2024-02-12T17:16:05.695408+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":327,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654562+0000","last_change":"2024-02-12T17:14:08.745944+0000","last_active":"2024-02-12T17:16:05.654562+0000","last_peered":"2024-02-12T17:16:05.654562+0000","last_clean":"2024-02-12T17:16:05.654562+0000","last_became_active":"2024-02-12T17:14:08.745729+0000","last_became_peered":"2024-02-12T17:14:08.745729+0000","last_unstale":"2024-02-12T17:16:05.654562+0000","last_undegraded":"2024-02-12T17:16:05.654562+0000","last_fullsized":"2024-02-12T17:16:05.654562+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:49.809844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:12:49.809844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":91,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.699091+0000","last_change":"2024-02-12T17:13:46.559519+0000","last_active":"2024-02-12T17:16:05.699091+0000","last_peered":"2024-02-12T17:16:05.699091+0000","last_clean":"2024-02-12T17:16:05.699091+0000","last_became_active":"2024-02-12T17:13:46.559095+0000","last_became_peered":"2024-02-12T17:13:46.559095+0000","last_unstale":"2024-02-12T17:16:05.699091+0000","last_undegraded":"2024-02-12T17:16:05.699091+0000","last_fullsized":"2024-02-12T17:16:05.699091+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":67,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.682989+0000","last_change":"2024-02-12T17:14:08.993503+0000","last_active":"2024-02-12T17:16:05.682989+0000","last_peered":"2024-02-12T17:16:05.682989+0000","last_clean":"2024-02-12T17:16:05.682989+0000","last_became_active":"2024-02-12T17:14:08.861886+0000","last_became_peered":"2024-02-12T17:14:08.861886+0000","last_unstale":"2024-02-12T17:16:05.682989+0000","last_undegraded":"2024-02-12T17:16:05.682989+0000","last_fullsized":"2024-02-12T17:16:05.682989+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"699'14905","reported_seq":21884,"reported_epoch":699,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-12T17:16:23.650324+0000","last_change":"2024-02-12T17:16:09.902235+0000","last_active":"2024-02-12T17:16:23.650324+0000","last_peered":"2024-02-12T17:16:23.650324+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.966441+0000","last_became_peered":"2024-02-12T17:14:08.966441+0000","last_unstale":"2024-02-12T17:16:23.650324+0000","last_undegraded":"2024-02-12T17:14:08.734833+0000","last_fullsized":"2024-02-12T17:14:08.712653+0000","mapping_epoch":651,"log_start":"628'7921","ondisk_log_start":"628'7921","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6984,"ondisk_log_size":6984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59965440,"num_objects":915,"num_object_clones":0,"num_object_copies":2745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":915,"num_objects_misplaced":915,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14905,"num_write_kb":506240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3413,"num_bytes_recovered":223674368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[2,4],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4","objects":915}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":416,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.696075+0000","last_change":"2024-02-12T17:13:46.787305+0000","last_active":"2024-02-12T17:16:05.696075+0000","last_peered":"2024-02-12T17:16:05.696075+0000","last_clean":"2024-02-12T17:16:05.696075+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:16:05.696075+0000","last_undegraded":"2024-02-12T17:16:05.696075+0000","last_fullsized":"2024-02-12T17:16:05.696075+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":414,"reported_epoch":697,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:03.607077+0000","last_change":"2024-02-12T17:14:09.723027+0000","last_active":"2024-02-12T17:16:03.607077+0000","last_peered":"2024-02-12T17:16:03.607077+0000","last_clean":"2024-02-12T17:11:53.687795+0000","last_became_active":"2024-02-12T17:14:09.722681+0000","last_became_peered":"2024-02-12T17:14:09.722681+0000","last_unstale":"2024-02-12T17:16:03.607077+0000","last_undegraded":"2024-02-12T17:16:03.607077+0000","last_fullsized":"2024-02-12T17:16:03.607077+0000","mapping_epoch":652,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:12:09.882247+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:12:09.882247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[5,7,6],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":165,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.696182+0000","last_change":"2024-02-12T17:13:46.744739+0000","last_active":"2024-02-12T17:16:05.696182+0000","last_peered":"2024-02-12T17:16:05.696182+0000","last_clean":"2024-02-12T17:16:05.696182+0000","last_became_active":"2024-02-12T17:13:46.744003+0000","last_became_peered":"2024-02-12T17:13:46.744003+0000","last_unstale":"2024-02-12T17:16:05.696182+0000","last_undegraded":"2024-02-12T17:16:05.696182+0000","last_fullsized":"2024-02-12T17:16:05.696182+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:51.126056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:51.126056+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.670452+0000","last_change":"2024-02-12T17:14:08.769824+0000","last_active":"2024-02-12T17:16:05.670452+0000","last_peered":"2024-02-12T17:16:05.670452+0000","last_clean":"2024-02-12T17:16:05.670452+0000","last_became_active":"2024-02-12T17:14:08.769221+0000","last_became_peered":"2024-02-12T17:14:08.769221+0000","last_unstale":"2024-02-12T17:16:05.670452+0000","last_undegraded":"2024-02-12T17:16:05.670452+0000","last_fullsized":"2024-02-12T17:16:05.670452+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"699'14625","reported_seq":14768,"reported_epoch":699,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:16:23.691617+0000","last_change":"2024-02-12T17:14:16.786942+0000","last_active":"2024-02-12T17:16:23.691617+0000","last_peered":"2024-02-12T17:16:23.691617+0000","last_clean":"2024-02-12T17:13:45.521351+0000","last_became_active":"2024-02-12T17:13:46.743970+0000","last_became_peered":"2024-02-12T17:13:46.743970+0000","last_unstale":"2024-02-12T17:16:23.691617+0000","last_undegraded":"2024-02-12T17:16:23.691617+0000","last_fullsized":"2024-02-12T17:16:23.691617+0000","mapping_epoch":630,"log_start":"643'8568","ondisk_log_start":"643'8568","created":597,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'5769","last_scrub_stamp":"2024-02-12T17:13:06.847792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:13:06.847792+0000","log_size":6057,"ondisk_log_size":6057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61997080,"num_objects":947,"num_object_clones":0,"num_object_copies":2841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":947,"num_objects_unfound":0,"num_objects_dirty":947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14625,"num_write_kb":498241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":947}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":704,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.670779+0000","last_change":"2024-02-12T17:14:07.897458+0000","last_active":"2024-02-12T17:16:05.670779+0000","last_peered":"2024-02-12T17:16:05.670779+0000","last_clean":"2024-02-12T17:16:05.670779+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:16:05.670779+0000","last_undegraded":"2024-02-12T17:16:05.670779+0000","last_fullsized":"2024-02-12T17:16:05.670779+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.780381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.780381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":91,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.699896+0000","last_change":"2024-02-12T17:13:46.791505+0000","last_active":"2024-02-12T17:16:05.699896+0000","last_peered":"2024-02-12T17:16:05.699896+0000","last_clean":"2024-02-12T17:16:05.699896+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:16:05.699896+0000","last_undegraded":"2024-02-12T17:16:05.699896+0000","last_fullsized":"2024-02-12T17:16:05.699896+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":171,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697736+0000","last_change":"2024-02-12T17:14:09.246132+0000","last_active":"2024-02-12T17:16:05.697736+0000","last_peered":"2024-02-12T17:16:05.697736+0000","last_clean":"2024-02-12T17:16:05.697736+0000","last_became_active":"2024-02-12T17:14:08.752149+0000","last_became_peered":"2024-02-12T17:14:08.752149+0000","last_unstale":"2024-02-12T17:16:05.697736+0000","last_undegraded":"2024-02-12T17:16:05.697736+0000","last_fullsized":"2024-02-12T17:16:05.697736+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:13:01.543047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.543047+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"655'23","reported_seq":524,"reported_epoch":698,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:22.692495+0000","last_change":"2024-02-12T17:14:09.749094+0000","last_active":"2024-02-12T17:16:22.692495+0000","last_peered":"2024-02-12T17:16:22.692495+0000","last_clean":"2024-02-12T17:03:47.420086+0000","last_became_active":"2024-02-12T17:14:09.748790+0000","last_became_peered":"2024-02-12T17:14:09.748790+0000","last_unstale":"2024-02-12T17:16:22.692495+0000","last_undegraded":"2024-02-12T17:16:22.692495+0000","last_fullsized":"2024-02-12T17:16:22.692495+0000","mapping_epoch":652,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'21","last_scrub_stamp":"2024-02-12T17:12:57.741798+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:12:57.741798+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[3,6,5],"avail_no_missing":["3","0","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"699'14934","reported_seq":18844,"reported_epoch":699,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:16:26.720978+0000","last_change":"2024-02-12T17:14:31.380762+0000","last_active":"2024-02-12T17:16:26.720978+0000","last_peered":"2024-02-12T17:16:26.720978+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.877848+0000","last_became_peered":"2024-02-12T17:14:08.877848+0000","last_unstale":"2024-02-12T17:16:26.720978+0000","last_undegraded":"2024-02-12T17:14:09.686965+0000","last_fullsized":"2024-02-12T17:14:08.709420+0000","mapping_epoch":651,"log_start":"666'9963","ondisk_log_start":"666'9963","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4971,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51511296,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":426,"num_objects_misplaced":426,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14934,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":85786624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":419,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.694955+0000","last_change":"2024-02-12T17:13:21.603543+0000","last_active":"2024-02-12T17:16:05.694955+0000","last_peered":"2024-02-12T17:16:05.694955+0000","last_clean":"2024-02-12T17:16:05.694955+0000","last_became_active":"2024-02-12T17:13:21.603226+0000","last_became_peered":"2024-02-12T17:13:21.603226+0000","last_unstale":"2024-02-12T17:16:05.694955+0000","last_undegraded":"2024-02-12T17:16:05.694955+0000","last_fullsized":"2024-02-12T17:16:05.694955+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.262134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.262134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":734,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605184+0000","last_change":"2024-02-12T17:13:46.566957+0000","last_active":"2024-02-12T17:16:03.605184+0000","last_peered":"2024-02-12T17:16:03.605184+0000","last_clean":"2024-02-12T17:16:03.605184+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:16:03.605184+0000","last_undegraded":"2024-02-12T17:16:03.605184+0000","last_fullsized":"2024-02-12T17:16:03.605184+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":217,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:22.699024+0000","last_change":"2024-02-12T17:13:46.607171+0000","last_active":"2024-02-12T17:16:22.699024+0000","last_peered":"2024-02-12T17:16:22.699024+0000","last_clean":"2024-02-12T17:16:22.699024+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:16:22.699024+0000","last_undegraded":"2024-02-12T17:16:22.699024+0000","last_fullsized":"2024-02-12T17:16:22.699024+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":650,"reported_epoch":698,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:05.694953+0000","last_change":"2024-02-12T17:14:09.723924+0000","last_active":"2024-02-12T17:16:05.694953+0000","last_peered":"2024-02-12T17:16:05.694953+0000","last_clean":"2024-02-12T17:14:05.074559+0000","last_became_active":"2024-02-12T17:14:09.723600+0000","last_became_peered":"2024-02-12T17:14:09.723600+0000","last_unstale":"2024-02-12T17:16:05.694953+0000","last_undegraded":"2024-02-12T17:16:05.694953+0000","last_fullsized":"2024-02-12T17:16:05.694953+0000","mapping_epoch":652,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:13:02.852563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:13:02.852563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[1,6,7],"avail_no_missing":["1","0","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"699'15108","reported_seq":15218,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-02-12T17:16:26.734350+0000","last_change":"2024-02-12T17:11:59.008876+0000","last_active":"2024-02-12T17:16:26.734350+0000","last_peered":"2024-02-12T17:16:26.734350+0000","last_clean":"2024-02-12T17:16:26.734350+0000","last_became_active":"2024-02-12T17:11:59.008587+0000","last_became_peered":"2024-02-12T17:11:59.008587+0000","last_unstale":"2024-02-12T17:16:26.734350+0000","last_undegraded":"2024-02-12T17:16:26.734350+0000","last_fullsized":"2024-02-12T17:16:26.734350+0000","mapping_epoch":597,"log_start":"666'10118","ondisk_log_start":"666'10118","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4990,"ondisk_log_size":4990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":2598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15108,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":704,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.658266+0000","last_change":"2024-02-12T17:13:00.766698+0000","last_active":"2024-02-12T17:16:05.658266+0000","last_peered":"2024-02-12T17:16:05.658266+0000","last_clean":"2024-02-12T17:16:05.658266+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:16:05.658266+0000","last_undegraded":"2024-02-12T17:16:05.658266+0000","last_fullsized":"2024-02-12T17:16:05.658266+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.766590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":357,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-02-12T17:16:23.643917+0000","last_change":"2024-02-12T17:13:46.584941+0000","last_active":"2024-02-12T17:16:23.643917+0000","last_peered":"2024-02-12T17:16:23.643917+0000","last_clean":"2024-02-12T17:16:23.643917+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:16:23.643917+0000","last_undegraded":"2024-02-12T17:16:23.643917+0000","last_fullsized":"2024-02-12T17:16:23.643917+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":365,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.694663+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:16:05.694663+0000","last_peered":"2024-02-12T17:16:05.694663+0000","last_clean":"2024-02-12T17:16:05.694663+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:16:05.694663+0000","last_undegraded":"2024-02-12T17:16:05.694663+0000","last_fullsized":"2024-02-12T17:16:05.694663+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":164,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.698470+0000","last_change":"2024-02-12T17:14:09.321979+0000","last_active":"2024-02-12T17:16:05.698470+0000","last_peered":"2024-02-12T17:16:05.698470+0000","last_clean":"2024-02-12T17:16:05.698470+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:16:05.698470+0000","last_undegraded":"2024-02-12T17:16:05.698470+0000","last_fullsized":"2024-02-12T17:16:05.698470+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":627,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.606502+0000","last_change":"2024-02-12T17:13:10.412138+0000","last_active":"2024-02-12T17:16:03.606502+0000","last_peered":"2024-02-12T17:16:03.606502+0000","last_clean":"2024-02-12T17:16:03.606502+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:16:03.606502+0000","last_undegraded":"2024-02-12T17:16:03.606502+0000","last_fullsized":"2024-02-12T17:16:03.606502+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:13:10.412031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"699'14729","reported_seq":14839,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-02-12T17:16:26.729295+0000","last_change":"2024-02-12T17:11:59.007548+0000","last_active":"2024-02-12T17:16:26.729295+0000","last_peered":"2024-02-12T17:16:26.729295+0000","last_clean":"2024-02-12T17:16:26.729295+0000","last_became_active":"2024-02-12T17:11:59.007214+0000","last_became_peered":"2024-02-12T17:11:59.007214+0000","last_unstale":"2024-02-12T17:16:26.729295+0000","last_undegraded":"2024-02-12T17:16:26.729295+0000","last_fullsized":"2024-02-12T17:16:26.729295+0000","mapping_epoch":597,"log_start":"666'9755","ondisk_log_start":"666'9755","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4974,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53018624,"num_objects":809,"num_object_clones":0,"num_object_copies":2427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14729,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":766,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.607422+0000","last_change":"2024-02-12T17:13:46.826853+0000","last_active":"2024-02-12T17:16:03.607422+0000","last_peered":"2024-02-12T17:16:03.607422+0000","last_clean":"2024-02-12T17:16:03.607422+0000","last_became_active":"2024-02-12T17:13:46.821418+0000","last_became_peered":"2024-02-12T17:13:46.821418+0000","last_unstale":"2024-02-12T17:16:03.607422+0000","last_undegraded":"2024-02-12T17:16:03.607422+0000","last_fullsized":"2024-02-12T17:16:03.607422+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.951964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.951964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":397,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697208+0000","last_change":"2024-02-12T17:12:59.006372+0000","last_active":"2024-02-12T17:16:05.697208+0000","last_peered":"2024-02-12T17:16:05.697208+0000","last_clean":"2024-02-12T17:16:05.697208+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:16:05.697208+0000","last_undegraded":"2024-02-12T17:16:05.697208+0000","last_fullsized":"2024-02-12T17:16:05.697208+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.006260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.006260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":674,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605691+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:16:03.605691+0000","last_peered":"2024-02-12T17:16:03.605691+0000","last_clean":"2024-02-12T17:16:03.605691+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:16:03.605691+0000","last_undegraded":"2024-02-12T17:16:03.605691+0000","last_fullsized":"2024-02-12T17:16:03.605691+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":744,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654292+0000","last_change":"2024-02-12T17:14:08.743668+0000","last_active":"2024-02-12T17:16:05.654292+0000","last_peered":"2024-02-12T17:16:05.654292+0000","last_clean":"2024-02-12T17:16:05.654292+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:16:05.654292+0000","last_undegraded":"2024-02-12T17:16:05.654292+0000","last_fullsized":"2024-02-12T17:16:05.654292+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":309,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.664574+0000","last_change":"2024-02-12T17:14:08.781462+0000","last_active":"2024-02-12T17:16:05.664574+0000","last_peered":"2024-02-12T17:16:05.664574+0000","last_clean":"2024-02-12T17:16:05.664574+0000","last_became_active":"2024-02-12T17:14:08.726463+0000","last_became_peered":"2024-02-12T17:14:08.726463+0000","last_unstale":"2024-02-12T17:16:05.664574+0000","last_undegraded":"2024-02-12T17:16:05.664574+0000","last_fullsized":"2024-02-12T17:16:05.664574+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:47.763264+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.763264+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":706,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654593+0000","last_change":"2024-02-12T17:12:03.922963+0000","last_active":"2024-02-12T17:16:05.654593+0000","last_peered":"2024-02-12T17:16:05.654593+0000","last_clean":"2024-02-12T17:16:05.654593+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:16:05.654593+0000","last_undegraded":"2024-02-12T17:16:05.654593+0000","last_fullsized":"2024-02-12T17:16:05.654593+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:12:03.922893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.922893+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"699'14903","reported_seq":17662,"reported_epoch":699,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:16:26.752541+0000","last_change":"2024-02-12T17:14:10.177438+0000","last_active":"2024-02-12T17:16:26.752541+0000","last_peered":"2024-02-12T17:16:26.752541+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.873608+0000","last_became_peered":"2024-02-12T17:14:08.873608+0000","last_unstale":"2024-02-12T17:16:26.752541+0000","last_undegraded":"2024-02-12T17:14:08.744781+0000","last_fullsized":"2024-02-12T17:14:08.704327+0000","mapping_epoch":651,"log_start":"666'9942","ondisk_log_start":"666'9942","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4961,"ondisk_log_size":4961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55377920,"num_objects":845,"num_object_clones":0,"num_object_copies":2535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14903,"num_write_kb":503936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":86310912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":91,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.699976+0000","last_change":"2024-02-12T17:13:46.608049+0000","last_active":"2024-02-12T17:16:05.699976+0000","last_peered":"2024-02-12T17:16:05.699976+0000","last_clean":"2024-02-12T17:16:05.699976+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:16:05.699976+0000","last_undegraded":"2024-02-12T17:16:05.699976+0000","last_fullsized":"2024-02-12T17:16:05.699976+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.677474+0000","last_change":"2024-02-12T17:14:08.859925+0000","last_active":"2024-02-12T17:16:05.677474+0000","last_peered":"2024-02-12T17:16:05.677474+0000","last_clean":"2024-02-12T17:16:05.677474+0000","last_became_active":"2024-02-12T17:14:08.859396+0000","last_became_peered":"2024-02-12T17:14:08.859396+0000","last_unstale":"2024-02-12T17:16:05.677474+0000","last_undegraded":"2024-02-12T17:16:05.677474+0000","last_fullsized":"2024-02-12T17:16:05.677474+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:03.904976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.904976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":744,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654196+0000","last_change":"2024-02-12T17:14:05.666060+0000","last_active":"2024-02-12T17:16:05.654196+0000","last_peered":"2024-02-12T17:16:05.654196+0000","last_clean":"2024-02-12T17:16:05.654196+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:16:05.654196+0000","last_undegraded":"2024-02-12T17:16:05.654196+0000","last_fullsized":"2024-02-12T17:16:05.654196+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":71,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.683067+0000","last_change":"2024-02-12T17:14:09.046726+0000","last_active":"2024-02-12T17:16:05.683067+0000","last_peered":"2024-02-12T17:16:05.683067+0000","last_clean":"2024-02-12T17:16:05.683067+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:16:05.683067+0000","last_undegraded":"2024-02-12T17:16:05.683067+0000","last_fullsized":"2024-02-12T17:16:05.683067+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":746,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.607337+0000","last_change":"2024-02-12T17:13:46.827015+0000","last_active":"2024-02-12T17:16:03.607337+0000","last_peered":"2024-02-12T17:16:03.607337+0000","last_clean":"2024-02-12T17:16:03.607337+0000","last_became_active":"2024-02-12T17:13:46.821796+0000","last_became_peered":"2024-02-12T17:13:46.821796+0000","last_unstale":"2024-02-12T17:16:03.607337+0000","last_undegraded":"2024-02-12T17:16:03.607337+0000","last_fullsized":"2024-02-12T17:16:03.607337+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.698727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.698727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"699'14616","reported_seq":20714,"reported_epoch":699,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:16:23.680099+0000","last_change":"2024-02-12T17:14:46.835208+0000","last_active":"2024-02-12T17:16:23.680099+0000","last_peered":"2024-02-12T17:16:23.680099+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.806885+0000","last_became_peered":"2024-02-12T17:13:46.806885+0000","last_unstale":"2024-02-12T17:16:23.680099+0000","last_undegraded":"2024-02-12T17:13:46.572970+0000","last_fullsized":"2024-02-12T17:13:46.531119+0000","mapping_epoch":630,"log_start":"628'7574","ondisk_log_start":"628'7574","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":7042,"ondisk_log_size":7042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61603840,"num_objects":940,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":528,"num_objects_misplaced":940,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14616,"num_write_kb":497792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2984,"num_bytes_recovered":195559424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":122,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.696094+0000","last_change":"2024-02-12T17:13:46.579742+0000","last_active":"2024-02-12T17:16:05.696094+0000","last_peered":"2024-02-12T17:16:05.696094+0000","last_clean":"2024-02-12T17:16:05.696094+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:16:05.696094+0000","last_undegraded":"2024-02-12T17:16:05.696094+0000","last_fullsized":"2024-02-12T17:16:05.696094+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":758,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.658306+0000","last_change":"2024-02-12T17:14:08.746702+0000","last_active":"2024-02-12T17:16:05.658306+0000","last_peered":"2024-02-12T17:16:05.658306+0000","last_clean":"2024-02-12T17:16:05.658306+0000","last_became_active":"2024-02-12T17:14:08.746395+0000","last_became_peered":"2024-02-12T17:14:08.746395+0000","last_unstale":"2024-02-12T17:16:05.658306+0000","last_undegraded":"2024-02-12T17:16:05.658306+0000","last_fullsized":"2024-02-12T17:16:05.658306+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:45.764955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:45.764955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":75,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.671985+0000","last_change":"2024-02-12T17:14:11.148689+0000","last_active":"2024-02-12T17:16:05.671985+0000","last_peered":"2024-02-12T17:16:05.671985+0000","last_clean":"2024-02-12T17:16:05.671985+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:16:05.671985+0000","last_undegraded":"2024-02-12T17:16:05.671985+0000","last_fullsized":"2024-02-12T17:16:05.671985+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":200,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.664283+0000","last_change":"2024-02-12T17:12:53.746937+0000","last_active":"2024-02-12T17:16:05.664283+0000","last_peered":"2024-02-12T17:16:05.664283+0000","last_clean":"2024-02-12T17:16:05.664283+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:16:05.664283+0000","last_undegraded":"2024-02-12T17:16:05.664283+0000","last_fullsized":"2024-02-12T17:16:05.664283+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.746858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.746858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"699'14735","reported_seq":14859,"reported_epoch":699,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:16:23.889103+0000","last_change":"2024-02-12T17:14:46.848102+0000","last_active":"2024-02-12T17:16:23.889103+0000","last_peered":"2024-02-12T17:16:23.889103+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.832453+0000","last_became_peered":"2024-02-12T17:13:46.832453+0000","last_unstale":"2024-02-12T17:16:23.889103+0000","last_undegraded":"2024-02-12T17:13:46.655125+0000","last_fullsized":"2024-02-12T17:13:46.642131+0000","mapping_epoch":631,"log_start":"650'9070","ondisk_log_start":"650'9070","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5665,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64684032,"num_objects":987,"num_object_clones":0,"num_object_copies":2961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14735,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":987}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":406,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697289+0000","last_change":"2024-02-12T17:13:21.589965+0000","last_active":"2024-02-12T17:16:05.697289+0000","last_peered":"2024-02-12T17:16:05.697289+0000","last_clean":"2024-02-12T17:16:05.697289+0000","last_became_active":"2024-02-12T17:13:21.589584+0000","last_became_peered":"2024-02-12T17:13:21.589584+0000","last_unstale":"2024-02-12T17:16:05.697289+0000","last_undegraded":"2024-02-12T17:16:05.697289+0000","last_fullsized":"2024-02-12T17:16:05.697289+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:01.059570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.059570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.702963+0000","last_change":"2024-02-12T17:14:18.716530+0000","last_active":"2024-02-12T17:16:05.702963+0000","last_peered":"2024-02-12T17:16:05.702963+0000","last_clean":"2024-02-12T17:16:05.702963+0000","last_became_active":"2024-02-12T17:14:18.716232+0000","last_became_peered":"2024-02-12T17:14:18.716232+0000","last_unstale":"2024-02-12T17:16:05.702963+0000","last_undegraded":"2024-02-12T17:16:05.702963+0000","last_fullsized":"2024-02-12T17:16:05.702963+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":465,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.682195+0000","last_change":"2024-02-12T17:14:10.144923+0000","last_active":"2024-02-12T17:16:05.682195+0000","last_peered":"2024-02-12T17:16:05.682195+0000","last_clean":"2024-02-12T17:16:05.682195+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:16:05.682195+0000","last_undegraded":"2024-02-12T17:16:05.682195+0000","last_fullsized":"2024-02-12T17:16:05.682195+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":685,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.694505+0000","last_change":"2024-02-12T17:14:08.765556+0000","last_active":"2024-02-12T17:16:05.694505+0000","last_peered":"2024-02-12T17:16:05.694505+0000","last_clean":"2024-02-12T17:16:05.694505+0000","last_became_active":"2024-02-12T17:14:08.757310+0000","last_became_peered":"2024-02-12T17:14:08.757310+0000","last_unstale":"2024-02-12T17:16:05.694505+0000","last_undegraded":"2024-02-12T17:16:05.694505+0000","last_fullsized":"2024-02-12T17:16:05.694505+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:58.871569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.871569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"699'14820","reported_seq":19563,"reported_epoch":699,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:16:23.917219+0000","last_change":"2024-02-12T17:14:10.288517+0000","last_active":"2024-02-12T17:16:23.917219+0000","last_peered":"2024-02-12T17:16:23.917219+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.867083+0000","last_became_peered":"2024-02-12T17:14:08.867083+0000","last_unstale":"2024-02-12T17:16:23.917219+0000","last_undegraded":"2024-02-12T17:14:08.714899+0000","last_fullsized":"2024-02-12T17:14:08.701373+0000","mapping_epoch":651,"log_start":"650'9142","ondisk_log_start":"650'9142","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5678,"ondisk_log_size":5678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59768832,"num_objects":912,"num_object_clones":0,"num_object_copies":2736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":493,"num_objects_misplaced":493,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14820,"num_write_kb":503424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":106102784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":912}],"blocked_by":[],"up_primary":1,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":737,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.655915+0000","last_change":"2024-02-12T17:13:08.874063+0000","last_active":"2024-02-12T17:16:05.655915+0000","last_peered":"2024-02-12T17:16:05.655915+0000","last_clean":"2024-02-12T17:16:05.655915+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:16:05.655915+0000","last_undegraded":"2024-02-12T17:16:05.655915+0000","last_fullsized":"2024-02-12T17:16:05.655915+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:13:08.873961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:13:08.873961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":218,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.664665+0000","last_change":"2024-02-12T17:14:14.843264+0000","last_active":"2024-02-12T17:16:05.664665+0000","last_peered":"2024-02-12T17:16:05.664665+0000","last_clean":"2024-02-12T17:16:05.664665+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:16:05.664665+0000","last_undegraded":"2024-02-12T17:16:05.664665+0000","last_fullsized":"2024-02-12T17:16:05.664665+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:56.688982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.688982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":882,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605318+0000","last_change":"2024-02-12T17:14:09.517724+0000","last_active":"2024-02-12T17:16:03.605318+0000","last_peered":"2024-02-12T17:16:03.605318+0000","last_clean":"2024-02-12T17:16:03.605318+0000","last_became_active":"2024-02-12T17:14:08.740691+0000","last_became_peered":"2024-02-12T17:14:08.740691+0000","last_unstale":"2024-02-12T17:16:03.605318+0000","last_undegraded":"2024-02-12T17:16:03.605318+0000","last_fullsized":"2024-02-12T17:16:03.605318+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:12:05.850456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.850456+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"655'32","reported_seq":650,"reported_epoch":698,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:22.696289+0000","last_change":"2024-02-12T17:14:09.746021+0000","last_active":"2024-02-12T17:16:22.696289+0000","last_peered":"2024-02-12T17:16:22.696289+0000","last_clean":"2024-02-12T17:14:07.617720+0000","last_became_active":"2024-02-12T17:14:09.745704+0000","last_became_peered":"2024-02-12T17:14:09.745704+0000","last_unstale":"2024-02-12T17:16:22.696289+0000","last_undegraded":"2024-02-12T17:16:22.696289+0000","last_fullsized":"2024-02-12T17:16:22.696289+0000","mapping_epoch":652,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3","0","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"699'14735","reported_seq":14861,"reported_epoch":699,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:23.731660+0000","last_change":"2024-02-12T17:14:09.843467+0000","last_active":"2024-02-12T17:16:23.731660+0000","last_peered":"2024-02-12T17:16:23.731660+0000","last_clean":"2024-02-12T17:14:07.683370+0000","last_became_active":"2024-02-12T17:14:09.782739+0000","last_became_peered":"2024-02-12T17:14:09.782739+0000","last_unstale":"2024-02-12T17:16:23.731660+0000","last_undegraded":"2024-02-12T17:16:23.731660+0000","last_fullsized":"2024-02-12T17:16:23.731660+0000","mapping_epoch":652,"log_start":"644'8651","ondisk_log_start":"644'8651","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6084,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60096512,"num_objects":917,"num_object_clones":0,"num_object_copies":2751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":917,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14735,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":917}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":408,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697583+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:16:05.697583+0000","last_peered":"2024-02-12T17:16:05.697583+0000","last_clean":"2024-02-12T17:16:05.697583+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:16:05.697583+0000","last_undegraded":"2024-02-12T17:16:05.697583+0000","last_fullsized":"2024-02-12T17:16:05.697583+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"642'31","reported_seq":1110,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:17.691343+0000","last_change":"2024-02-12T17:13:53.773053+0000","last_active":"2024-02-12T17:16:17.691343+0000","last_peered":"2024-02-12T17:16:17.691343+0000","last_clean":"2024-02-12T17:16:17.691343+0000","last_became_active":"2024-02-12T17:13:53.772744+0000","last_became_peered":"2024-02-12T17:13:53.772744+0000","last_unstale":"2024-02-12T17:16:17.691343+0000","last_undegraded":"2024-02-12T17:16:17.691343+0000","last_fullsized":"2024-02-12T17:16:17.691343+0000","mapping_epoch":638,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'28","last_scrub_stamp":"2024-02-12T17:12:07.826227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:12:07.826227+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":412,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654099+0000","last_change":"2024-02-12T17:14:10.756477+0000","last_active":"2024-02-12T17:16:05.654099+0000","last_peered":"2024-02-12T17:16:05.654099+0000","last_clean":"2024-02-12T17:16:05.654099+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:16:05.654099+0000","last_undegraded":"2024-02-12T17:16:05.654099+0000","last_fullsized":"2024-02-12T17:16:05.654099+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:16.873570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.873570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":272,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.664434+0000","last_change":"2024-02-12T17:13:46.807564+0000","last_active":"2024-02-12T17:16:05.664434+0000","last_peered":"2024-02-12T17:16:05.664434+0000","last_clean":"2024-02-12T17:16:05.664434+0000","last_became_active":"2024-02-12T17:13:46.807214+0000","last_became_peered":"2024-02-12T17:13:46.807214+0000","last_unstale":"2024-02-12T17:16:05.664434+0000","last_undegraded":"2024-02-12T17:16:05.664434+0000","last_fullsized":"2024-02-12T17:16:05.664434+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:04.705646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.705646+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"699'15012","reported_seq":18734,"reported_epoch":699,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-12T17:16:26.741739+0000","last_change":"2024-02-12T17:14:09.819105+0000","last_active":"2024-02-12T17:16:26.741739+0000","last_peered":"2024-02-12T17:16:26.741739+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.810504+0000","last_became_peered":"2024-02-12T17:13:46.810504+0000","last_unstale":"2024-02-12T17:16:26.741739+0000","last_undegraded":"2024-02-12T17:13:46.690611+0000","last_fullsized":"2024-02-12T17:13:46.648648+0000","mapping_epoch":631,"log_start":"668'10071","ondisk_log_start":"668'10071","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4941,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":2598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15012,"num_write_kb":508096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1800,"num_bytes_recovered":117964800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":866}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":741,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.606056+0000","last_change":"2024-02-12T17:14:08.740896+0000","last_active":"2024-02-12T17:16:03.606056+0000","last_peered":"2024-02-12T17:16:03.606056+0000","last_clean":"2024-02-12T17:16:03.606056+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:16:03.606056+0000","last_undegraded":"2024-02-12T17:16:03.606056+0000","last_fullsized":"2024-02-12T17:16:03.606056+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"668'27","reported_seq":1135,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:17.690958+0000","last_change":"2024-02-12T17:14:24.235718+0000","last_active":"2024-02-12T17:16:17.690958+0000","last_peered":"2024-02-12T17:16:17.690958+0000","last_clean":"2024-02-12T17:16:17.690958+0000","last_became_active":"2024-02-12T17:14:24.235386+0000","last_became_peered":"2024-02-12T17:14:24.235386+0000","last_unstale":"2024-02-12T17:16:17.690958+0000","last_undegraded":"2024-02-12T17:16:17.690958+0000","last_fullsized":"2024-02-12T17:16:17.690958+0000","mapping_epoch":665,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'24","last_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.647212+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":109,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.696009+0000","last_change":"2024-02-12T17:14:05.106537+0000","last_active":"2024-02-12T17:16:05.696009+0000","last_peered":"2024-02-12T17:16:05.696009+0000","last_clean":"2024-02-12T17:16:05.696009+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:16:05.696009+0000","last_undegraded":"2024-02-12T17:16:05.696009+0000","last_fullsized":"2024-02-12T17:16:05.696009+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":157,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.705391+0000","last_change":"2024-02-12T17:13:46.819259+0000","last_active":"2024-02-12T17:16:05.705391+0000","last_peered":"2024-02-12T17:16:05.705391+0000","last_clean":"2024-02-12T17:16:05.705391+0000","last_became_active":"2024-02-12T17:13:46.818771+0000","last_became_peered":"2024-02-12T17:13:46.818771+0000","last_unstale":"2024-02-12T17:16:05.705391+0000","last_undegraded":"2024-02-12T17:16:05.705391+0000","last_fullsized":"2024-02-12T17:16:05.705391+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"698'14473","reported_seq":27404,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:23.603371+0000","last_change":"2024-02-12T17:14:32.710386+0000","last_active":"2024-02-12T17:16:23.603371+0000","last_peered":"2024-02-12T17:16:23.603371+0000","last_clean":"2024-02-12T17:16:23.603371+0000","last_became_active":"2024-02-12T17:14:32.696812+0000","last_became_peered":"2024-02-12T17:14:32.696812+0000","last_unstale":"2024-02-12T17:16:23.603371+0000","last_undegraded":"2024-02-12T17:16:23.603371+0000","last_fullsized":"2024-02-12T17:16:23.603371+0000","mapping_epoch":671,"log_start":"642'8400","ondisk_log_start":"642'8400","created":597,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6073,"ondisk_log_size":6073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62193664,"num_objects":949,"num_object_clones":0,"num_object_copies":2847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14473,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6402,"num_bytes_recovered":419561472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":89,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.702906+0000","last_change":"2024-02-12T17:13:46.805110+0000","last_active":"2024-02-12T17:16:05.702906+0000","last_peered":"2024-02-12T17:16:05.702906+0000","last_clean":"2024-02-12T17:16:05.702906+0000","last_became_active":"2024-02-12T17:13:46.802275+0000","last_became_peered":"2024-02-12T17:13:46.802275+0000","last_unstale":"2024-02-12T17:16:05.702906+0000","last_undegraded":"2024-02-12T17:16:05.702906+0000","last_fullsized":"2024-02-12T17:16:05.702906+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":639,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:22.692794+0000","last_change":"2024-02-12T17:13:07.711701+0000","last_active":"2024-02-12T17:16:22.692794+0000","last_peered":"2024-02-12T17:16:22.692794+0000","last_clean":"2024-02-12T17:16:22.692794+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:16:22.692794+0000","last_undegraded":"2024-02-12T17:16:22.692794+0000","last_fullsized":"2024-02-12T17:16:22.692794+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:13:07.711601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:07.711601+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":109,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697447+0000","last_change":"2024-02-12T17:14:00.999940+0000","last_active":"2024-02-12T17:16:05.697447+0000","last_peered":"2024-02-12T17:16:05.697447+0000","last_clean":"2024-02-12T17:16:05.697447+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:16:05.697447+0000","last_undegraded":"2024-02-12T17:16:05.697447+0000","last_fullsized":"2024-02-12T17:16:05.697447+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":502,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.697680+0000","last_change":"2024-02-12T17:13:46.806094+0000","last_active":"2024-02-12T17:16:05.697680+0000","last_peered":"2024-02-12T17:16:05.697680+0000","last_clean":"2024-02-12T17:16:05.697680+0000","last_became_active":"2024-02-12T17:13:46.805712+0000","last_became_peered":"2024-02-12T17:13:46.805712+0000","last_unstale":"2024-02-12T17:16:05.697680+0000","last_undegraded":"2024-02-12T17:16:05.697680+0000","last_fullsized":"2024-02-12T17:16:05.697680+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"699'14996","reported_seq":15106,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-02-12T17:16:26.766740+0000","last_change":"2024-02-12T17:11:59.496950+0000","last_active":"2024-02-12T17:16:26.766740+0000","last_peered":"2024-02-12T17:16:26.766740+0000","last_clean":"2024-02-12T17:16:26.766740+0000","last_became_active":"2024-02-12T17:11:59.496585+0000","last_became_peered":"2024-02-12T17:11:59.496585+0000","last_unstale":"2024-02-12T17:16:26.766740+0000","last_undegraded":"2024-02-12T17:16:26.766740+0000","last_fullsized":"2024-02-12T17:16:26.766740+0000","mapping_epoch":597,"log_start":"666'10048","ondisk_log_start":"666'10048","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":4948,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56885248,"num_objects":868,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14996,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":428,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.695326+0000","last_change":"2024-02-12T17:13:21.603687+0000","last_active":"2024-02-12T17:16:05.695326+0000","last_peered":"2024-02-12T17:16:05.695326+0000","last_clean":"2024-02-12T17:16:05.695326+0000","last_became_active":"2024-02-12T17:13:21.603332+0000","last_became_peered":"2024-02-12T17:13:21.603332+0000","last_unstale":"2024-02-12T17:16:05.695326+0000","last_undegraded":"2024-02-12T17:16:05.695326+0000","last_fullsized":"2024-02-12T17:16:05.695326+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":120,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.699825+0000","last_change":"2024-02-12T17:13:46.804446+0000","last_active":"2024-02-12T17:16:05.699825+0000","last_peered":"2024-02-12T17:16:05.699825+0000","last_clean":"2024-02-12T17:16:05.699825+0000","last_became_active":"2024-02-12T17:13:46.804033+0000","last_became_peered":"2024-02-12T17:13:46.804033+0000","last_unstale":"2024-02-12T17:16:05.699825+0000","last_undegraded":"2024-02-12T17:16:05.699825+0000","last_fullsized":"2024-02-12T17:16:05.699825+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.793569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.793569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":722,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654835+0000","last_change":"2024-02-12T17:13:56.661004+0000","last_active":"2024-02-12T17:16:05.654835+0000","last_peered":"2024-02-12T17:16:05.654835+0000","last_clean":"2024-02-12T17:16:05.654835+0000","last_became_active":"2024-02-12T17:13:56.660707+0000","last_became_peered":"2024-02-12T17:13:56.660707+0000","last_unstale":"2024-02-12T17:16:05.654835+0000","last_undegraded":"2024-02-12T17:16:05.654835+0000","last_fullsized":"2024-02-12T17:16:05.654835+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.685601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.685601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":189,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.694853+0000","last_change":"2024-02-12T17:13:46.757888+0000","last_active":"2024-02-12T17:16:05.694853+0000","last_peered":"2024-02-12T17:16:05.694853+0000","last_clean":"2024-02-12T17:16:05.694853+0000","last_became_active":"2024-02-12T17:13:46.757450+0000","last_became_peered":"2024-02-12T17:13:46.757450+0000","last_unstale":"2024-02-12T17:16:05.694853+0000","last_undegraded":"2024-02-12T17:16:05.694853+0000","last_fullsized":"2024-02-12T17:16:05.694853+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:56.880282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.880282+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.5","version":"699'14645","reported_seq":14758,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-02-12T17:16:23.931945+0000","last_change":"2024-02-12T17:13:12.678687+0000","last_active":"2024-02-12T17:16:23.931945+0000","last_peered":"2024-02-12T17:16:23.931945+0000","last_clean":"2024-02-12T17:16:23.931945+0000","last_became_active":"2024-02-12T17:11:59.009543+0000","last_became_peered":"2024-02-12T17:11:59.009543+0000","last_unstale":"2024-02-12T17:16:23.931945+0000","last_undegraded":"2024-02-12T17:16:23.931945+0000","last_fullsized":"2024-02-12T17:16:23.931945+0000","mapping_epoch":597,"log_start":"649'8929","ondisk_log_start":"649'8929","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5716,"ondisk_log_size":5716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64815104,"num_objects":989,"num_object_clones":0,"num_object_copies":2967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14645,"num_write_kb":500288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":722,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.654197+0000","last_change":"2024-02-12T17:12:08.013330+0000","last_active":"2024-02-12T17:16:05.654197+0000","last_peered":"2024-02-12T17:16:05.654197+0000","last_clean":"2024-02-12T17:16:05.654197+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:16:05.654197+0000","last_undegraded":"2024-02-12T17:16:05.654197+0000","last_fullsized":"2024-02-12T17:16:05.654197+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.013247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.013247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":619,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.672188+0000","last_change":"2024-02-12T17:12:04.711418+0000","last_active":"2024-02-12T17:16:05.672188+0000","last_peered":"2024-02-12T17:16:05.672188+0000","last_clean":"2024-02-12T17:16:05.672188+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:16:05.672188+0000","last_undegraded":"2024-02-12T17:16:05.672188+0000","last_fullsized":"2024-02-12T17:16:05.672188+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:04.711333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.711333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":476,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.606076+0000","last_change":"2024-02-12T17:14:08.740752+0000","last_active":"2024-02-12T17:16:03.606076+0000","last_peered":"2024-02-12T17:16:03.606076+0000","last_clean":"2024-02-12T17:16:03.606076+0000","last_became_active":"2024-02-12T17:14:08.740430+0000","last_became_peered":"2024-02-12T17:14:08.740430+0000","last_unstale":"2024-02-12T17:16:03.606076+0000","last_undegraded":"2024-02-12T17:16:03.606076+0000","last_fullsized":"2024-02-12T17:16:03.606076+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":254,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.664367+0000","last_change":"2024-02-12T17:13:46.559877+0000","last_active":"2024-02-12T17:16:05.664367+0000","last_peered":"2024-02-12T17:16:05.664367+0000","last_clean":"2024-02-12T17:16:05.664367+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:16:05.664367+0000","last_undegraded":"2024-02-12T17:16:05.664367+0000","last_fullsized":"2024-02-12T17:16:05.664367+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"699'14938","reported_seq":15048,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-02-12T17:16:23.853227+0000","last_change":"2024-02-12T17:11:59.007574+0000","last_active":"2024-02-12T17:16:23.853227+0000","last_peered":"2024-02-12T17:16:23.853227+0000","last_clean":"2024-02-12T17:16:23.853227+0000","last_became_active":"2024-02-12T17:11:59.007300+0000","last_became_peered":"2024-02-12T17:11:59.007300+0000","last_unstale":"2024-02-12T17:16:23.853227+0000","last_undegraded":"2024-02-12T17:16:23.853227+0000","last_fullsized":"2024-02-12T17:16:23.853227+0000","mapping_epoch":597,"log_start":"650'9267","ondisk_log_start":"650'9267","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5671,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62652416,"num_objects":956,"num_object_clones":0,"num_object_copies":2868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14938,"num_write_kb":508608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":57,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.674716+0000","last_change":"2024-02-12T17:14:08.859503+0000","last_active":"2024-02-12T17:16:05.674716+0000","last_peered":"2024-02-12T17:16:05.674716+0000","last_clean":"2024-02-12T17:16:05.674716+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:16:05.674716+0000","last_undegraded":"2024-02-12T17:16:05.674716+0000","last_fullsized":"2024-02-12T17:16:05.674716+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:03.705889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:13:03.705889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":619,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.696987+0000","last_change":"2024-02-12T17:14:08.762783+0000","last_active":"2024-02-12T17:16:05.696987+0000","last_peered":"2024-02-12T17:16:05.696987+0000","last_clean":"2024-02-12T17:16:05.696987+0000","last_became_active":"2024-02-12T17:14:08.760654+0000","last_became_peered":"2024-02-12T17:14:08.760654+0000","last_unstale":"2024-02-12T17:16:05.696987+0000","last_undegraded":"2024-02-12T17:16:05.696987+0000","last_fullsized":"2024-02-12T17:16:05.696987+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:09.044325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:09.044325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":397,"reported_epoch":698,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-02-12T17:16:05.694521+0000","last_change":"2024-02-12T17:14:16.753241+0000","last_active":"2024-02-12T17:16:05.694521+0000","last_peered":"2024-02-12T17:16:05.694521+0000","last_clean":"2024-02-12T17:13:44.508652+0000","last_became_active":"2024-02-12T17:13:46.563455+0000","last_became_peered":"2024-02-12T17:13:46.563455+0000","last_unstale":"2024-02-12T17:16:05.694521+0000","last_undegraded":"2024-02-12T17:16:05.694521+0000","last_fullsized":"2024-02-12T17:16:05.694521+0000","mapping_epoch":630,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":67,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.671793+0000","last_change":"2024-02-12T17:14:10.285684+0000","last_active":"2024-02-12T17:16:05.671793+0000","last_peered":"2024-02-12T17:16:05.671793+0000","last_clean":"2024-02-12T17:16:05.671793+0000","last_became_active":"2024-02-12T17:14:08.871051+0000","last_became_peered":"2024-02-12T17:14:08.871051+0000","last_unstale":"2024-02-12T17:16:05.671793+0000","last_undegraded":"2024-02-12T17:16:05.671793+0000","last_fullsized":"2024-02-12T17:16:05.671793+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:50.764686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.764686+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"698'14660","reported_seq":21675,"reported_epoch":698,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-02-12T17:16:23.366239+0000","last_change":"2024-02-12T17:14:39.687781+0000","last_active":"2024-02-12T17:16:23.366239+0000","last_peered":"2024-02-12T17:16:23.366239+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.886954+0000","last_became_peered":"2024-02-12T17:14:08.886954+0000","last_unstale":"2024-02-12T17:16:23.366239+0000","last_undegraded":"2024-02-12T17:14:08.719570+0000","last_fullsized":"2024-02-12T17:14:08.707803+0000","mapping_epoch":651,"log_start":"635'8178","ondisk_log_start":"635'8178","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6482,"ondisk_log_size":6482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64880640,"num_objects":990,"num_object_clones":0,"num_object_copies":2970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":990,"num_objects_misplaced":990,"num_objects_unfound":0,"num_objects_dirty":990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14660,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3435,"num_bytes_recovered":225050624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,4],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5","objects":990}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":704,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605043+0000","last_change":"2024-02-12T17:12:11.906816+0000","last_active":"2024-02-12T17:16:03.605043+0000","last_peered":"2024-02-12T17:16:03.605043+0000","last_clean":"2024-02-12T17:16:03.605043+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:16:03.605043+0000","last_undegraded":"2024-02-12T17:16:03.605043+0000","last_fullsized":"2024-02-12T17:16:03.605043+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:11.906737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:11.906737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":383,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:05.695079+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:16:05.695079+0000","last_peered":"2024-02-12T17:16:05.695079+0000","last_clean":"2024-02-12T17:16:05.695079+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:16:05.695079+0000","last_undegraded":"2024-02-12T17:16:05.695079+0000","last_fullsized":"2024-02-12T17:16:05.695079+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":822,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-12T17:16:03.605042+0000","last_change":"2024-02-12T17:12:16.820857+0000","last_active":"2024-02-12T17:16:03.605042+0000","last_peered":"2024-02-12T17:16:03.605042+0000","last_clean":"2024-02-12T17:16:03.605042+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:16:03.605042+0000","last_undegraded":"2024-02-12T17:16:03.605042+0000","last_fullsized":"2024-02-12T17:16:03.605042+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:12:16.820782+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.820782+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"632'35","reported_seq":1200,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-02-12T17:16:22.692553+0000","last_change":"2024-02-12T17:13:46.691596+0000","last_active":"2024-02-12T17:16:22.692553+0000","last_peered":"2024-02-12T17:16:22.692553+0000","last_clean":"2024-02-12T17:16:22.692553+0000","last_became_active":"2024-02-12T17:13:46.691213+0000","last_became_peered":"2024-02-12T17:13:46.691213+0000","last_unstale":"2024-02-12T17:16:22.692553+0000","last_undegraded":"2024-02-12T17:16:22.692553+0000","last_fullsized":"2024-02-12T17:16:22.692553+0000","mapping_epoch":631,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'29","last_scrub_stamp":"2024-02-12T17:12:47.788327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.788327+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":952959000,"num_objects":14542,"num_object_clones":0,"num_object_copies":43626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5128,"num_objects_misplaced":5628,"num_objects_unfound":0,"num_objects_dirty":14542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236834,"num_write_kb":8043969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22283,"num_bytes_recovered":1460011008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3062771712,"data_stored":3062759496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91237,"ondisk_log_size":91237,"up":48,"acting":40,"num_store_stats":8},{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":16216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1029,"ondisk_log_size":1029,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234308,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2097508,"kb_used_data":601244,"kb_used_omap":0,"kb_used_meta":1496256,"kb_avail":91647644,"statfs":{"total":95995035648,"available":93847187456,"internally_reserved":0,"allocated":615673856,"data_stored":614281145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1532166144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":2.5249999999999999,"5min":1.9099999999999999,"15min":2.024},"min":{"1min":0.36799999999999999,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":10.417,"5min":15.856999999999999,"15min":15.856999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.756,"5min":1.8180000000000001,"15min":1.7290000000000001},"min":{"1min":0.23599999999999999,"5min":0.221,"15min":0.22},"max":{"1min":7.1120000000000001,"5min":12.565,"15min":12.565},"last":0.98799999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:15:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4569999999999999,"5min":2.9220000000000002,"15min":2.8330000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.632999999999999,"5min":11.632999999999999,"15min":11.632999999999999},"last":3.0680000000000001},{"interface":"front","average":{"1min":3.8100000000000001,"5min":2.7360000000000002,"15min":2.5569999999999999},"min":{"1min":0.29299999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":10.91,"5min":10.91,"15min":10.91},"last":3.0880000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":2.8010000000000002,"15min":3.1349999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22800000000000001},"max":{"1min":9.5389999999999997,"5min":42.162999999999997,"15min":218.35300000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":2.3820000000000001,"5min":3,"15min":3.2480000000000002},"min":{"1min":0.375,"5min":0.25600000000000001,"15min":0.247},"max":{"1min":9.407,"5min":42.109999999999999,"15min":218.42099999999999},"last":2.5259999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":3.754,"5min":2.52,"15min":2.1930000000000001},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.255},"max":{"1min":11.741,"5min":15.811999999999999,"15min":15.811999999999999},"last":6.3579999999999997},{"interface":"front","average":{"1min":3.9079999999999999,"5min":3.0710000000000002,"15min":2.649},"min":{"1min":0.42299999999999999,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":12.375999999999999,"5min":13.452,"15min":31.152000000000001},"last":4.2759999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.4099999999999999,"15min":1.6759999999999999},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":2.7810000000000001,"5min":20.672999999999998,"15min":20.672999999999998},"last":0.309},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.8939999999999999,"15min":1.96},"min":{"1min":0.23300000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.7240000000000002,"5min":38.390000000000001,"15min":38.390000000000001},"last":0.317}]},{"osd":5,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.8169999999999999,"15min":2.169},"min":{"1min":0.28100000000000003,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":2.7000000000000002,"5min":20.702000000000002,"15min":56.560000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.4330000000000001,"15min":1.915},"min":{"1min":0.25800000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.6930000000000001,"5min":10.669,"15min":53.963999999999999},"last":0.36399999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.4570000000000001,"15min":1.532},"min":{"1min":0.30399999999999999,"5min":0.223,"15min":0.223},"max":{"1min":4.0030000000000001,"5min":21.620000000000001,"15min":21.620000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.671,"15min":1.724},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.7360000000000002,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.34799999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528649,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407476,"kb_used_data":515116,"kb_used_omap":0,"kb_used_meta":892352,"kb_avail":92337676,"statfs":{"total":95995035648,"available":94553780224,"internally_reserved":0,"allocated":527478784,"data_stored":526070333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913768448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.7410000000000001,"15min":1.806},"min":{"1min":0.22,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.266,"5min":33.576999999999998,"15min":33.576999999999998},"last":6.2069999999999999},{"interface":"front","average":{"1min":2.194,"5min":2.0859999999999999,"15min":1.9239999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9740000000000002,"5min":33.569000000000003,"15min":33.569000000000003},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":2.1160000000000001,"15min":2.1000000000000001},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.6260000000000003,"5min":13.805999999999999,"15min":13.805999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.9299999999999999,"5min":2.141,"15min":2.1760000000000002},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.6399999999999997,"5min":13.778,"15min":13.778},"last":4.7969999999999997}]},{"osd":2,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8849999999999998,"5min":2.7290000000000001,"15min":2.6240000000000001},"min":{"1min":0.35399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.5760000000000005,"5min":33.594999999999999,"15min":56.798000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":4.3019999999999996,"5min":2.6680000000000001,"15min":2.3639999999999999},"min":{"1min":0.36099999999999999,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":14.045999999999999,"5min":33.526000000000003,"15min":56.780999999999999},"last":7.226}]},{"osd":3,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4489999999999998,"5min":3.1779999999999999,"15min":2.3999999999999999},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":12.336,"5min":58.420999999999999,"15min":58.420999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":3.7109999999999999,"5min":3.8290000000000002,"15min":2.8530000000000002},"min":{"1min":0.40400000000000003,"5min":0.254,"15min":0.254},"max":{"1min":11.737,"5min":58.709000000000003,"15min":58.709000000000003},"last":1.1890000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.542,"15min":1.272},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.23499999999999999},"max":{"1min":5.0810000000000004,"5min":33.706000000000003,"15min":33.706000000000003},"last":3.1269999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.8540000000000001,"15min":1.508},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":4.2460000000000004,"5min":34.895000000000003,"15min":34.895000000000003},"last":0.26600000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.6639999999999999,"15min":1.716},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.1970000000000001,"5min":14.56,"15min":34.869},"last":2.573},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.319,"15min":1.6799999999999999},"min":{"1min":0.36399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.1390000000000002,"5min":11.1,"15min":33.591000000000001},"last":3.1579999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:16:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.7090000000000001,"15min":1.7629999999999999},"min":{"1min":0.26500000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":4.4550000000000001,"5min":35.122999999999998,"15min":35.122999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.421,"5min":1.869,"15min":1.6220000000000001},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.26500000000000001},"max":{"1min":5.1890000000000001,"5min":34.045000000000002,"15min":34.045000000000002},"last":3.089}]}]},{"osd":1,"up_from":630,"seq":2705829396513,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487824,"kb_used_data":149192,"kb_used_omap":3,"kb_used_meta":338620,"kb_avail":93257328,"statfs":{"total":95995035648,"available":95495503872,"internally_reserved":0,"allocated":152772608,"data_stored":151343166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":346746941},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8580000000000001,"5min":3.359,"15min":3.4430000000000001},"min":{"1min":0.33900000000000002,"5min":0.214,"15min":0.214},"max":{"1min":9.0109999999999992,"5min":14.67,"15min":14.67},"last":3.573},{"interface":"front","average":{"1min":4.609,"5min":3.6579999999999999,"15min":3.5},"min":{"1min":0.33800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":16.920999999999999,"5min":16.920999999999999,"15min":16.920999999999999},"last":3.613}]},{"osd":2,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":3.4220000000000002,"15min":3.528},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":7.5250000000000004,"5min":8.766,"15min":8.766},"last":3.8199999999999998},{"interface":"front","average":{"1min":3.6619999999999999,"5min":3.5299999999999998,"15min":3.508},"min":{"1min":0.33100000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.303000000000001,"5min":14.303000000000001,"15min":14.303000000000001},"last":3.9569999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":5.3479999999999999,"5min":3.984,"15min":3.7559999999999998},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":18.577000000000002,"5min":18.577000000000002,"15min":18.577000000000002},"last":11.061999999999999},{"interface":"front","average":{"1min":4.7169999999999996,"5min":4.4009999999999998,"15min":4.3479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":17.527000000000001,"5min":17.527000000000001,"15min":17.527000000000001},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.4359999999999999,"5min":2.5310000000000001,"15min":2.5470000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.3209999999999997,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.6059999999999999,"5min":2.746,"15min":2.7690000000000001},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":8.8249999999999993,"5min":12.805999999999999,"15min":12.805999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.6309999999999998,"5min":3.0990000000000002,"15min":3.177},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.9120000000000008,"5min":12.769,"15min":12.769},"last":0.68899999999999995},{"interface":"front","average":{"1min":3.1459999999999999,"5min":2.319,"15min":2.181},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":11.15,"5min":11.15,"15min":11.15},"last":5.7380000000000004}]},{"osd":6,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":3.0449999999999999,"5min":2.8730000000000002,"15min":2.8439999999999999},"min":{"1min":0.41599999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":11.115,"5min":12.720000000000001,"15min":12.720000000000001},"last":5.7830000000000004},{"interface":"front","average":{"1min":2.4039999999999999,"5min":2.7829999999999999,"15min":2.8460000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.29,"5min":8.0890000000000004,"15min":8.0890000000000004},"last":0.46800000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.8460000000000001,"15min":2.9239999999999999},"min":{"1min":0.39000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":7.6669999999999998,"5min":12.829000000000001,"15min":12.829000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":3.4630000000000001,"5min":2.6059999999999999,"15min":2.4630000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":11.179,"5min":11.179,"15min":11.179},"last":5.766}]}]},{"osd":0,"up_from":8,"seq":34359738726,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037864,"kb_used_data":145696,"kb_used_omap":0,"kb_used_meta":892096,"kb_avail":92707288,"statfs":{"total":95995035648,"available":94932262912,"internally_reserved":0,"allocated":149192704,"data_stored":147803142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913506304},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:15:49 2024","interfaces":[{"interface":"back","average":{"1min":4.2720000000000002,"5min":3.597,"15min":3.4849999999999999},"min":{"1min":0.25900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":14.048999999999999,"5min":14.048999999999999,"15min":14.048999999999999},"last":1.6479999999999999},{"interface":"front","average":{"1min":2.3919999999999999,"5min":2.7200000000000002,"15min":2.774},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":8.1120000000000001,"5min":13.787000000000001,"15min":13.787000000000001},"last":3.339}]},{"osd":2,"last update":"Mon Feb 12 17:15:31 2024","interfaces":[{"interface":"back","average":{"1min":3.6640000000000001,"5min":2.7759999999999998,"15min":2.0710000000000002},"min":{"1min":0.311,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":13.762,"5min":19.312000000000001,"15min":19.312000000000001},"last":9.2509999999999994},{"interface":"front","average":{"1min":4.1500000000000004,"5min":3.0539999999999998,"15min":2.6019999999999999},"min":{"1min":0.39700000000000002,"5min":0.218,"15min":0.20799999999999999},"max":{"1min":22.959,"5min":22.959,"15min":26.039999999999999},"last":8.4710000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:15:45 2024","interfaces":[{"interface":"back","average":{"1min":9.375,"5min":3.629,"15min":3.1179999999999999},"min":{"1min":0.28599999999999998,"5min":0.186,"15min":0.186},"max":{"1min":61.177,"5min":61.177,"15min":87.311000000000007},"last":0.46100000000000002},{"interface":"front","average":{"1min":6.2800000000000002,"5min":3.6920000000000002,"15min":3.1400000000000001},"min":{"1min":0.45800000000000002,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":59.149999999999999,"5min":59.149999999999999,"15min":90.991},"last":9.2810000000000006}]},{"osd":4,"last update":"Mon Feb 12 17:16:05 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.5529999999999999,"15min":1.476},"min":{"1min":0.28799999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":7.5830000000000002,"5min":15.141999999999999,"15min":15.141999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.641,"15min":2.0310000000000001},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":6.7220000000000004,"5min":13.503,"15min":26.082999999999998},"last":6.7149999999999999}]},{"osd":5,"last update":"Mon Feb 12 17:15:39 2024","interfaces":[{"interface":"back","average":{"1min":2.1890000000000001,"5min":1.8740000000000001,"15min":2.2229999999999999},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":10.347,"5min":15.090999999999999,"15min":50.404000000000003},"last":0.433},{"interface":"front","average":{"1min":2.0790000000000002,"5min":1.748,"15min":2.0030000000000001},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.74,"5min":15.164,"15min":49.979999999999997},"last":0.47199999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:16:05 2024","interfaces":[{"interface":"back","average":{"1min":3.0169999999999999,"5min":2.5600000000000001,"15min":2.379},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.8369999999999997,"5min":48.780999999999999,"15min":48.780999999999999},"last":6.6749999999999998},{"interface":"front","average":{"1min":3.3239999999999998,"5min":2.827,"15min":2.3370000000000002},"min":{"1min":0.38600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":10.390000000000001,"5min":49.067999999999998,"15min":49.067999999999998},"last":0.40000000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:15:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.04,"15min":2.04},"min":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":13.813000000000001,"5min":15.183999999999999,"15min":19.373000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":2.9889999999999999,"5min":2.3279999999999998,"15min":2.4430000000000001},"min":{"1min":0.38400000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":13.542,"5min":13.542,"15min":26.135000000000002},"last":6.7430000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411675,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1928540,"kb_used_data":434452,"kb_used_omap":0,"kb_used_meta":1494080,"kb_avail":91816612,"statfs":{"total":95995035648,"available":94020210688,"internally_reserved":0,"allocated":444878848,"data_stored":443494819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1529937920},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":3.2559999999999998,"5min":1.9690000000000001,"15min":1.8440000000000001},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17000000000000001},"max":{"1min":11.663,"5min":11.663,"15min":36.100999999999999},"last":3.722},{"interface":"front","average":{"1min":5.1219999999999999,"5min":2.4489999999999998,"15min":2.181},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":16.350000000000001,"5min":16.350000000000001,"15min":36.299999999999997},"last":9.1890000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:15:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5489999999999999,"5min":3.722,"15min":3.9180000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":9.3239999999999998,"5min":11.612,"15min":11.612},"last":0.54000000000000004},{"interface":"front","average":{"1min":4.5579999999999998,"5min":5.6260000000000003,"15min":5.8040000000000003},"min":{"1min":0.42599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":11.141999999999999,"5min":22.52,"15min":22.52},"last":4.0460000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:16:00 2024","interfaces":[{"interface":"back","average":{"1min":5.5259999999999998,"5min":3.6930000000000001,"15min":2.9220000000000002},"min":{"1min":0.29099999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":30.303000000000001,"5min":30.303000000000001,"15min":40.759999999999998},"last":2.2480000000000002},{"interface":"front","average":{"1min":5.9139999999999997,"5min":3.262,"15min":2.5070000000000001},"min":{"1min":0.221,"5min":0.22,"15min":0.22},"max":{"1min":30.120999999999999,"5min":30.120999999999999,"15min":30.120999999999999},"last":3.2229999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:15:22 2024","interfaces":[{"interface":"back","average":{"1min":3.008,"5min":2.0129999999999999,"15min":1.98},"min":{"1min":0.27600000000000002,"5min":0.245,"15min":0.245},"max":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":2.8220000000000001,"5min":1.8939999999999999,"15min":2.0510000000000002},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.744,"5min":11.744,"15min":20.149000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":2.681,"5min":1.8700000000000001,"15min":2.0139999999999998},"min":{"1min":0.33900000000000002,"5min":0.217,"15min":0.217},"max":{"1min":11.581,"5min":11.581,"15min":32.57},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.881,"15min":2.1349999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.255},"max":{"1min":7.798,"5min":10.896000000000001,"15min":31.905999999999999},"last":4.1189999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:16:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.7829999999999999,"15min":1.8300000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.0919999999999996,"5min":10.714,"15min":20.966999999999999},"last":4.0949999999999998},{"interface":"front","average":{"1min":2.5880000000000001,"5min":2.1949999999999998,"15min":1.8580000000000001},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.6130000000000004,"5min":11.834,"15min":20.193000000000001},"last":0.70699999999999996}]},{"osd":7,"last update":"Mon Feb 12 17:15:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9590000000000001,"5min":2.121,"15min":2.4830000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":11.789,"5min":11.789,"15min":45.468000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.9990000000000001,"15min":2.4319999999999999},"min":{"1min":0.34499999999999997,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":7.8330000000000002,"5min":10.762,"15min":46.656999999999996},"last":4.1550000000000002}]}]},{"osd":3,"up_from":26,"seq":111669150038,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1292236,"kb_used_data":398916,"kb_used_omap":0,"kb_used_meta":893248,"kb_avail":92452916,"statfs":{"total":95995035648,"available":94671785984,"internally_reserved":0,"allocated":408489984,"data_stored":407065196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914685952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":6.7190000000000003,"5min":3.738,"15min":2.8420000000000001},"min":{"1min":0.45500000000000002,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":58.451999999999998},"last":11.939},{"interface":"front","average":{"1min":4.351,"5min":3.7240000000000002,"15min":3.0409999999999999},"min":{"1min":0.40400000000000003,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":11.720000000000001,"5min":72.688000000000002,"15min":72.688000000000002},"last":8.734}]},{"osd":1,"last update":"Mon Feb 12 17:15:49 2024","interfaces":[{"interface":"back","average":{"1min":7.9610000000000003,"5min":5.1769999999999996,"15min":4.7130000000000001},"min":{"1min":0.27800000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":22.524000000000001,"5min":22.524000000000001,"15min":22.524000000000001},"last":1.6000000000000001},{"interface":"front","average":{"1min":5.6269999999999998,"5min":5.4889999999999999,"15min":5.4660000000000002},"min":{"1min":0.25600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":20.262,"5min":20.262,"15min":20.262},"last":8.7759999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":7.9420000000000002,"5min":3.5049999999999999,"15min":2.5659999999999998},"min":{"1min":0.245,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":38.034999999999997,"5min":40.954999999999998,"15min":40.954999999999998},"last":1.579},{"interface":"front","average":{"1min":6.7450000000000001,"5min":4.1289999999999996,"15min":2.9710000000000001},"min":{"1min":0.41399999999999998,"5min":0.22,"15min":0.22},"max":{"1min":22.138999999999999,"5min":46.408000000000001,"15min":46.408000000000001},"last":11.848000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:16:15 2024","interfaces":[{"interface":"back","average":{"1min":5.2839999999999998,"5min":3.746,"15min":2.6720000000000002},"min":{"1min":0.54600000000000004,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":20.478000000000002,"5min":72.659000000000006,"15min":72.659000000000006},"last":3.5449999999999999},{"interface":"front","average":{"1min":4.5279999999999996,"5min":3.0979999999999999,"15min":2.1099999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23999999999999999},"max":{"1min":22.440999999999999,"5min":40.930999999999997,"15min":40.930999999999997},"last":1.55}]},{"osd":5,"last update":"Mon Feb 12 17:15:33 2024","interfaces":[{"interface":"back","average":{"1min":5.7649999999999997,"5min":3.1389999999999998,"15min":2.577},"min":{"1min":0.438,"5min":0.33000000000000002,"15min":0.253},"max":{"1min":17.306999999999999,"5min":17.306999999999999,"15min":17.940999999999999},"last":4.0220000000000002},{"interface":"front","average":{"1min":4.2640000000000002,"5min":3.431,"15min":2.8500000000000001},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":11.586,"5min":72.620000000000005,"15min":72.620000000000005},"last":3.6160000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":5.2039999999999997,"5min":2.9510000000000001,"15min":2.153},"min":{"1min":0.31900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":22.489999999999998,"5min":40.889000000000003,"15min":40.889000000000003},"last":4.5090000000000003},{"interface":"front","average":{"1min":5.1559999999999997,"5min":3.27,"15min":2.4649999999999999},"min":{"1min":0.39800000000000002,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":15.113,"5min":16.989999999999998,"15min":18.265000000000001},"last":3.996}]},{"osd":7,"last update":"Mon Feb 12 17:16:15 2024","interfaces":[{"interface":"back","average":{"1min":5.0890000000000004,"5min":3.5609999999999999,"15min":3.0880000000000001},"min":{"1min":0.53500000000000003,"5min":0.375,"15min":0.315},"max":{"1min":20.515999999999998,"5min":72.712999999999994,"15min":72.712999999999994},"last":3.5830000000000002},{"interface":"front","average":{"1min":5.6120000000000001,"5min":3.0259999999999998,"15min":2.9089999999999998},"min":{"1min":0.44900000000000001,"5min":0.38600000000000001,"15min":0.28299999999999997},"max":{"1min":15.169,"5min":15.169,"15min":38.948},"last":4.0599999999999996}]}]},{"osd":4,"up_from":0,"seq":137438953811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757583,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1953560,"kb_used_data":468240,"kb_used_omap":0,"kb_used_meta":1485312,"kb_avail":91791592,"statfs":{"total":95995035648,"available":93994590208,"internally_reserved":0,"allocated":479477760,"data_stored":478097071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1520959488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.696,"15min":1.7629999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":3.927,"5min":8.0350000000000001,"15min":46.365000000000002},"last":6.4279999999999999},{"interface":"front","average":{"1min":2.133,"5min":1.659,"15min":1.9350000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.20100000000000001},"max":{"1min":7.157,"5min":15.368,"15min":46.134},"last":9.2539999999999996}]},{"osd":1,"last update":"Mon Feb 12 17:15:52 2024","interfaces":[{"interface":"back","average":{"1min":2.512,"5min":1.9399999999999999,"15min":1.845},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.8710000000000004,"5min":9.484,"15min":9.484},"last":6.3209999999999997},{"interface":"front","average":{"1min":1.968,"5min":2.8450000000000002,"15min":2.9910000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":6.7210000000000001,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":2.1659999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":2.6000000000000001,"15min":2.036},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":9.0269999999999992,"5min":57.390999999999998,"15min":57.390999999999998},"last":4.1580000000000004},{"interface":"front","average":{"1min":2.7719999999999998,"5min":2.843,"15min":2.0219999999999998},"min":{"1min":0.32100000000000001,"5min":0.214,"15min":0.214},"max":{"1min":13.366,"5min":57.134999999999998,"15min":57.134999999999998},"last":2.2130000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":2.0030000000000001,"5min":3.996,"15min":2.6920000000000002},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":7.1630000000000003,"5min":57.170999999999999,"15min":57.170999999999999},"last":4.0999999999999996},{"interface":"front","average":{"1min":3.3530000000000002,"5min":3.4420000000000002,"15min":2.5579999999999998},"min":{"1min":0.27900000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.3829999999999991,"5min":57.484999999999999,"15min":57.484999999999999},"last":2.2650000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.169,"15min":1.556},"min":{"1min":0.29799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.9939999999999998,"5min":9.0429999999999993,"15min":12.656000000000001},"last":3.3660000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.4370000000000001,"15min":1.573},"min":{"1min":0.255,"5min":0.255,"15min":0.254},"max":{"1min":3.3250000000000002,"5min":10.619999999999999,"15min":20.550000000000001},"last":4.1269999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:16:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.5030000000000001,"5min":14.675000000000001,"15min":14.675000000000001},"last":3.3900000000000001},{"interface":"front","average":{"1min":1.651,"5min":1.802,"15min":1.5629999999999999},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":5.2720000000000002,"5min":12.864000000000001,"15min":13.916},"last":4.1429999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.5669999999999999,"15min":1.8180000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":5.8289999999999997,"5min":14.337999999999999,"15min":19.184999999999999},"last":2.194},{"interface":"front","average":{"1min":1.454,"5min":1.5449999999999999,"15min":1.8819999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":6.6950000000000003,"5min":14.637,"15min":20.274000000000001},"last":3.6349999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":144179200,"data_stored":144179200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148832256,"data_stored":148832256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":435949568,"data_stored":435945496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":396886016,"data_stored":396886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":367529984,"data_stored":367525912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":465043456,"data_stored":465043456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":500502528,"data_stored":500498456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":603848704,"data_stored":603848704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:16:30.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T17:16:30.757 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:30.943 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:30.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:31.049 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:31.103 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:16:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:30] "GET /metrics HTTP/1.1" 200 176850 "" "Prometheus/2.33.4" 2024-02-12T17:16:31.153 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:32.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:31.138622+0000 mgr.y (mgr.24481) 1011 : cluster [DBG] pgmap v1428: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 779 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 663 op/s; 4218/37980 objects degraded (11.106%); 4459/37980 objects misplaced (11.740%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:16:32.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:31.774428+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check update: Degraded data redundancy: 4218/37980 objects degraded (11.106%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:32.493 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:31.138622+0000 mgr.y (mgr.24481) 1011 : cluster [DBG] pgmap v1428: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 779 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 663 op/s; 4218/37980 objects degraded (11.106%); 4459/37980 objects misplaced (11.740%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:16:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:31.774428+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check update: Degraded data redundancy: 4218/37980 objects degraded (11.106%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:31.138622+0000 mgr.y (mgr.24481) 1011 : cluster [DBG] pgmap v1428: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 779 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 663 op/s; 4218/37980 objects degraded (11.106%); 4459/37980 objects misplaced (11.740%); 2.0 MiB/s, 31 objects/s recovering 2024-02-12T17:16:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:31.774428+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check update: Degraded data redundancy: 4218/37980 objects degraded (11.106%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:32.529 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:32.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:32.635 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:32.678 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:32.708 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:32.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:32.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:32.815 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:32.853 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:16:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:16:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:32.882 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:32.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:32.986 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:33.016 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:33.041 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:33.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:33.049 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:16:33.049 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:16:33.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:33.183 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:33.210 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:33.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:33.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:33.402 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:33.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:32.551170+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:16:33.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:32.551170+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:16:33.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:32.551170+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:16:34.106 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:34.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:34.223 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:34.273 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:34.374 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:34.376 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:34.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:34.481 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:34.541 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:34.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:33.142305+0000 mgr.y (mgr.24481) 1012 : cluster [DBG] pgmap v1429: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 704 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 663 op/s; 3794/34389 objects degraded (11.033%); 4317/34389 objects misplaced (12.553%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:16:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:33.142305+0000 mgr.y (mgr.24481) 1012 : cluster [DBG] pgmap v1429: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 704 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 663 op/s; 3794/34389 objects degraded (11.033%); 4317/34389 objects misplaced (12.553%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:16:34.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:33.142305+0000 mgr.y (mgr.24481) 1012 : cluster [DBG] pgmap v1429: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 704 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 663 op/s; 3794/34389 objects degraded (11.033%); 4317/34389 objects misplaced (12.553%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T17:16:35.002 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T17:16:35.230 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:35.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:35.341 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:35.402 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:35.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:16:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:16:36.693 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:36.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:36.804 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:36.859 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:36.863 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:35.133620+0000 mgr.y (mgr.24481) 1013 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:36.863 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:35.145952+0000 mgr.y (mgr.24481) 1014 : cluster [DBG] pgmap v1430: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 577 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 649 op/s; 2914/28290 objects degraded (10.300%); 3181/28290 objects misplaced (11.244%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:36.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:36.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:36.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:35.133620+0000 mgr.y (mgr.24481) 1013 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:36.917 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:35.145952+0000 mgr.y (mgr.24481) 1014 : cluster [DBG] pgmap v1430: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 577 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 649 op/s; 2914/28290 objects degraded (10.300%); 3181/28290 objects misplaced (11.244%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:36.917 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:35.133620+0000 mgr.y (mgr.24481) 1013 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:36.917 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:35.145952+0000 mgr.y (mgr.24481) 1014 : cluster [DBG] pgmap v1430: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 577 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 649 op/s; 2914/28290 objects degraded (10.300%); 3181/28290 objects misplaced (11.244%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:36.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:36.993 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:37.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:37.119 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:37.208 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:37.482 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:37.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:37.589 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:37.666 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:37.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:36.781400+0000 mon.a (mon.0) 2250 : cluster [WRN] Health check update: Degraded data redundancy: 2914/28290 objects degraded (10.300%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:36.781400+0000 mon.a (mon.0) 2250 : cluster [WRN] Health check update: Degraded data redundancy: 2914/28290 objects degraded (10.300%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:37.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:36.781400+0000 mon.a (mon.0) 2250 : cluster [WRN] Health check update: Degraded data redundancy: 2914/28290 objects degraded (10.300%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:38.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:16:38.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:16:38.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:16:38.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:16:38.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:16:38.368 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:16:38.399 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:16:38.451 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:16:38.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T17:16:38.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T17:16:38.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T17:16:38.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T17:16:38.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T17:16:38.469 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T17:16:38.469 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T17:16:38.469 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T17:16:38.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:37.146649+0000 mgr.y (mgr.24481) 1015 : cluster [DBG] pgmap v1431: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 502 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 743 op/s; 2533/24684 objects degraded (10.262%); 3045/24684 objects misplaced (12.336%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:16:38.782 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:37.998621+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? v1:172.21.15.33:0/3642380487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:38.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:37.146649+0000 mgr.y (mgr.24481) 1015 : cluster [DBG] pgmap v1431: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 502 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 743 op/s; 2533/24684 objects degraded (10.262%); 3045/24684 objects misplaced (12.336%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:16:38.783 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:37.998621+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? v1:172.21.15.33:0/3642380487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:38.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:37.146649+0000 mgr.y (mgr.24481) 1015 : cluster [DBG] pgmap v1431: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 502 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 743 op/s; 2533/24684 objects degraded (10.262%); 3045/24684 objects misplaced (12.336%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T17:16:38.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:37.998621+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? v1:172.21.15.33:0/3642380487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:39.053 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:39.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:39.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:39.232 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:39.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:39.337 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:39.460 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:39.709 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:39.527002+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.33:0/3308835554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:39.709 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:39.527002+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.33:0/3308835554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:39.714 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-02-12T17:16:39.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:39.527002+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.33:0/3308835554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:16:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:39.147907+0000 mgr.y (mgr.24481) 1016 : cluster [DBG] pgmap v1432: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 377 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 709 op/s; 1709/18651 objects degraded (9.163%); 1920/18651 objects misplaced (10.294%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:16:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:39.624894+0000 mon.a (mon.0) 2252 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:39.647966+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? v1:172.21.15.33:0/3308835554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:16:40.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:39.648028+0000 mon.a (mon.0) 2254 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2024-02-12T17:16:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:39.147907+0000 mgr.y (mgr.24481) 1016 : cluster [DBG] pgmap v1432: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 377 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 709 op/s; 1709/18651 objects degraded (9.163%); 1920/18651 objects misplaced (10.294%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:16:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:39.624894+0000 mon.a (mon.0) 2252 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:39.647966+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? v1:172.21.15.33:0/3308835554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:16:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:39.648028+0000 mon.a (mon.0) 2254 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2024-02-12T17:16:41.029 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:40] "GET /metrics HTTP/1.1" 200 176860 "" "Prometheus/2.33.4" 2024-02-12T17:16:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:39.147907+0000 mgr.y (mgr.24481) 1016 : cluster [DBG] pgmap v1432: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 377 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 709 op/s; 1709/18651 objects degraded (9.163%); 1920/18651 objects misplaced (10.294%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T17:16:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:39.624894+0000 mon.a (mon.0) 2252 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:39.647966+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? v1:172.21.15.33:0/3308835554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T17:16:41.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:39.648028+0000 mon.a (mon.0) 2254 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2024-02-12T17:16:41.055 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:41.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:41.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:41.209 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:41.232 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:41.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:41.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:41.339 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:41.421 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:42.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:16:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:16:42.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:41.150916+0000 mgr.y (mgr.24481) 1017 : cluster [DBG] pgmap v1434: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 377 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 644 op/s; 1709/18651 objects degraded (9.163%); 1920/18651 objects misplaced (10.294%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:41.786910+0000 mon.a (mon.0) 2255 : cluster [WRN] Health check update: Degraded data redundancy: 1709/18651 objects degraded (9.163%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:42.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:41.801772+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2024-02-12T17:16:43.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:41.150916+0000 mgr.y (mgr.24481) 1017 : cluster [DBG] pgmap v1434: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 377 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 644 op/s; 1709/18651 objects degraded (9.163%); 1920/18651 objects misplaced (10.294%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:43.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:41.786910+0000 mon.a (mon.0) 2255 : cluster [WRN] Health check update: Degraded data redundancy: 1709/18651 objects degraded (9.163%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:43.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:41.801772+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2024-02-12T17:16:43.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:41.150916+0000 mgr.y (mgr.24481) 1017 : cluster [DBG] pgmap v1434: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 377 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 644 op/s; 1709/18651 objects degraded (9.163%); 1920/18651 objects misplaced (10.294%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T17:16:43.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:41.786910+0000 mon.a (mon.0) 2255 : cluster [WRN] Health check update: Degraded data redundancy: 1709/18651 objects degraded (9.163%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:43.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:41.801772+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2024-02-12T17:16:43.035 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:43.123 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:43.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:43.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:43.214 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:16:43.214 INFO:teuthology.orchestra.run.smithi033.stdout:{"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1709/18651 objects degraded (9.163%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1835,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":702,"num_osds":8,"num_up_osds":8,"osd_up_since":1707758025,"num_in_osds":7,"osd_in_since":1707758000,"num_remapped_pgs":18},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":100},{"state_name":"active+remapped+backfill_wait","count":8},{"state_name":"active+undersized+degraded+remapped+backfilling","count":4},{"state_name":"active+remapped+backfill_wait+backfill_toofull","count":2},{"state_name":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","count":2},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_wait","count":1}],"num_pgs":118,"num_pools":7,"num_objects":6217,"data_bytes":394794684,"bytes_used":9073807360,"bytes_avail":662891442176,"bytes_total":671965249536,"degraded_objects":1709,"degraded_total":18651,"degraded_ratio":0.091630475577717016,"misplaced_objects":1920,"misplaced_total":18651,"misplaced_ratio":0.10294354190123854,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":2716377,"recovering_keys_per_sec":0,"num_objects_recovered":415,"num_bytes_recovered":27197440,"num_keys_recovered":0,"read_bytes_sec":1022,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":643},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.33:8443/","prometheus":"http://172.21.15.33:9283/"}},"servicemap":{"epoch":15,"modified":"2024-02-12T17:14:45.050033+0000","services":{"rgw":{"daemons":{"summary":"","24397":{"start_epoch":12,"start_stamp":"2024-02-12T16:49:46.308902+0000","gid":24397,"addr":"172.21.15.33:0/1589842856","metadata":{"arch":"x86_64","ceph_release":"pacific","ceph_version":"ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)","ceph_version_short":"16.2.14-521-g0e714d9a","container_hostname":"smithi033","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi033","id":"foo.a","kernel_description":"#1 SMP Thu Jan 18 15:15:29 UTC 2024","kernel_version":"4.18.0-536.el8.x86_64","mem_swap_kb":"0","mem_total_kb":"32543856","num_handles":"1","os":"Linux","pid":"8","realm_id":"","realm_name":"","zone_id":"4799771a-d28e-4dc8-a5a0-179e1408b6e6","zone_name":"default","zonegroup_id":"0384a922-9dfb-46b6-aed7-e430c5071cb8","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{"627379c4-3690-4a16-a102-e69cb00adc70":{"message":"Global Recovery Event (8m)\n [=======================.....] (remaining: 90s)","progress":0.8474576473236084,"add_to_ceph_s":true}}} 2024-02-12T17:16:43.231 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:43.342 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:43.438 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:43.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:43.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:43.585 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:43.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:43.697 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:16:43.707 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:43.203136+0000 mon.a (mon.0) 2257 : audit [DBG] from='client.? v1:172.21.15.33:0/2729418648' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:16:43.721 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:43.806 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:43.979 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:43.980 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:43.203136+0000 mon.a (mon.0) 2257 : audit [DBG] from='client.? v1:172.21.15.33:0/2729418648' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:16:43.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:43.203136+0000 mon.a (mon.0) 2257 : audit [DBG] from='client.? v1:172.21.15.33:0/2729418648' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T17:16:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:43.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:44.095 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:16:44.148 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:44.870 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T17:16:44.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:43.151609+0000 mgr.y (mgr.24481) 1018 : cluster [DBG] pgmap v1436: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 297 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 560 op/s; 1417/14835 objects degraded (9.552%); 1817/14835 objects misplaced (12.248%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:44.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:43.703249+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2024-02-12T17:16:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:43.151609+0000 mgr.y (mgr.24481) 1018 : cluster [DBG] pgmap v1436: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 297 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 560 op/s; 1417/14835 objects degraded (9.552%); 1817/14835 objects misplaced (12.248%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:45.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:43.703249+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2024-02-12T17:16:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:43.151609+0000 mgr.y (mgr.24481) 1018 : cluster [DBG] pgmap v1436: 118 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 297 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 560 op/s; 1417/14835 objects degraded (9.552%); 1817/14835 objects misplaced (12.248%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:45.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:43.703249+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2024-02-12T17:16:45.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:16:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:16:45.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:44.703570+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2024-02-12T17:16:46.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:44.703570+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2024-02-12T17:16:46.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:45 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:44.703570+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2024-02-12T17:16:46.192 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-02-12T17:16:46.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:16:46.713 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:16:46.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:46.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:16:46.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:46.874 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:46.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:46.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:45.146529+0000 mgr.y (mgr.24481) 1019 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:46.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:45.152070+0000 mgr.y (mgr.24481) 1020 : cluster [DBG] pgmap v1439: 118 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 167 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 610 op/s; 610/8610 objects degraded (7.085%); 693/8610 objects misplaced (8.049%); 3.0 MiB/s, 48 objects/s recovering 2024-02-12T17:16:46.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:45.706520+0000 mon.a (mon.0) 2260 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:16:46.992 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:45.714208+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2024-02-12T17:16:47.013 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:47.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:45.146529+0000 mgr.y (mgr.24481) 1019 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:47.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:45.152070+0000 mgr.y (mgr.24481) 1020 : cluster [DBG] pgmap v1439: 118 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 167 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 610 op/s; 610/8610 objects degraded (7.085%); 693/8610 objects misplaced (8.049%); 3.0 MiB/s, 48 objects/s recovering 2024-02-12T17:16:47.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:45.706520+0000 mon.a (mon.0) 2260 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:16:47.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:45.714208+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2024-02-12T17:16:47.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:45.146529+0000 mgr.y (mgr.24481) 1019 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:47.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:45.152070+0000 mgr.y (mgr.24481) 1020 : cluster [DBG] pgmap v1439: 118 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 8 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 167 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 610 op/s; 610/8610 objects degraded (7.085%); 693/8610 objects misplaced (8.049%); 3.0 MiB/s, 48 objects/s recovering 2024-02-12T17:16:47.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:45.706520+0000 mon.a (mon.0) 2260 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:16:47.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:45.714208+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2024-02-12T17:16:47.083 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:47.120 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:47.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:47.229 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:16:47.288 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:47.392 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:16:47.393 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:47.400 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:47.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:16:47.454 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:47.502 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:47.528 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-02-12T17:16:47.528 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-02-12T17:16:47.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:250118 11.6 11:608397e9:::benchmark_data_smithi033_188107_object124428:head [delete] snapc 0=[] ondisk+write+known_if_redirected e706)", 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:16:47.514551+0000", 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.011642853999999999, 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011668771, 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:16:47.529 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 250118 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.514551+0000", 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.514551+0000", 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.514548+0000", 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9949999999999994e-06 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.514557+0000", 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6699000000000001e-05 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.514574+0000", 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6649999999999997e-06 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.514581+0000", 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028852859999999999 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.517466+0000", 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012888099999999999 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.517595+0000", 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015943700000000001 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,1,4", 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.517754+0000", 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050241340000000004 2024-02-12T17:16:47.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.522778+0000", 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036977899999999998 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523148+0000", 2024-02-12T17:16:47.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085970500000000002 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_repop(client.36023.0:250121 11.d e706/630 11:b2c7f29a:::benchmark_data_smithi033_188107_object124431:head v 706'15495, mlcod=706'15493)", 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:16:47.523006+0000", 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.003188017, 2024-02-12T17:16:47.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0032792640000000001, 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523006+0000", 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523006+0000", 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999981 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523004+0000", 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0601999999999999e-05 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523015+0000", 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2262999999999999e-05 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523027+0000", 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2060000000000004e-06 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523033+0000", 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6128999999999998e-05 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523079+0000", 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9621e-05 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.523109+0000", 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102905 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:250101 11.6 11:6c129479:::benchmark_data_smithi033_188107_object124411:head [delete] snapc 0=[] ondisk+write+known_if_redirected e706)", 2024-02-12T17:16:47.541 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:16:47.495649+0000", 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.030544931000000001, 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030686377000000001, 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 250101 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:16:47.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.495649+0000", 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.495649+0000", 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999971 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.495646+0000", 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5410000000000006e-06 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.495656+0000", 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6356000000000001e-05 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.495672+0000", 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5869999999999999e-05 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.495688+0000", 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2840000000000002e-05 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.495741+0000", 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000164912 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.495906+0000", 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110475 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,1,4", 2024-02-12T17:16:47.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.496016+0000", 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011741079 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.507757+0000", 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00072024399999999998 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.508478+0000", 2024-02-12T17:16:47.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089630639999999998 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.517441+0000", 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021791688 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.36023.0:250113 11.6 11:6a7df53c:::benchmark_data_smithi033_188107_object124423:head [delete] snapc 0=[] ondisk+write+known_if_redirected e706)", 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-02-12T17:16:47.503106+0000", 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.023088008, 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023308332000000001, 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for sub ops", 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.36023", 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "v1:172.21.15.33:0/2237287492", 2024-02-12T17:16:47.549 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 250113 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.503106+0000", 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.503106+0000", 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4294967295.9999967 2024-02-12T17:16:47.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.503103+0000", 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1968e-05 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.503115+0000", 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.467e-05 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.503129+0000", 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0239999999999997e-06 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.503137+0000", 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069224499999999997 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.503830+0000", 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000149992 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.503980+0000", 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017607000000000001 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for subops from 0,1,4", 2024-02-12T17:16:47.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.504156+0000", 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043564600000000004 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_commit_rec", 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.508512+0000", 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013184606999999999 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op_commit", 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-02-12T17:16:47.521697+0000", 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018590867000000001 2024-02-12T17:16:47.554 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.555 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-02-12T17:16:47.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-02-12T17:16:47.555 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-02-12T17:16:47.555 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 4 2024-02-12T17:16:47.555 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-02-12T17:16:47.577 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:47.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:46.722935+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e706: 8 total, 8 up, 7 in 2024-02-12T17:16:47.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:46.799316+0000 mon.a (mon.0) 2263 : cluster [WRN] Health check update: Degraded data redundancy: 610/8610 objects degraded (7.085%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:48.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:46.722935+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e706: 8 total, 8 up, 7 in 2024-02-12T17:16:48.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:46.799316+0000 mon.a (mon.0) 2263 : cluster [WRN] Health check update: Degraded data redundancy: 610/8610 objects degraded (7.085%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:48.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:46.722935+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e706: 8 total, 8 up, 7 in 2024-02-12T17:16:48.032 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:46.799316+0000 mon.a (mon.0) 2263 : cluster [WRN] Health check update: Degraded data redundancy: 610/8610 objects degraded (7.085%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:48.032 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:16:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw-foo-a[65833]: debug 2024-02-12T17:16:47.697+0000 7efdb04dc700 -1 RGWWatcher::handle_error cookie 94508044028160 err (110) Connection timed out 2024-02-12T17:16:48.735 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:47.152730+0000 mgr.y (mgr.24481) 1021 : cluster [DBG] pgmap v1442: 118 pgs: 2 active+remapped+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean; 133 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 657 op/s; 506/6960 objects degraded (7.270%); 648/6960 objects misplaced (9.310%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:48.736 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:47.152730+0000 mgr.y (mgr.24481) 1021 : cluster [DBG] pgmap v1442: 118 pgs: 2 active+remapped+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean; 133 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 657 op/s; 506/6960 objects degraded (7.270%); 648/6960 objects misplaced (9.310%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:48.951 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Removed 125687 objects 2024-02-12T17:16:48.952 INFO:tasks.radosbench.radosbench.0.smithi033.stdout:Clean up completed and total clean up time :186.521 2024-02-12T17:16:48.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:47.152730+0000 mgr.y (mgr.24481) 1021 : cluster [DBG] pgmap v1442: 118 pgs: 2 active+remapped+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean; 133 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 657 op/s; 506/6960 objects degraded (7.270%); 648/6960 objects misplaced (9.310%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T17:16:49.666 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:49.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:49.772 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:49.805 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:50.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T17:16:50.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:49.153568+0000 mgr.y (mgr.24481) 1022 : cluster [DBG] pgmap v1443: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 107 active+clean; 68 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 633 op/s; 104/3846 objects degraded (2.704%); 130/3846 objects misplaced (3.380%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:16:50.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:49.747156+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e707: 8 total, 8 up, 7 in 2024-02-12T17:16:50.907 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:49.153568+0000 mgr.y (mgr.24481) 1022 : cluster [DBG] pgmap v1443: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 107 active+clean; 68 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 633 op/s; 104/3846 objects degraded (2.704%); 130/3846 objects misplaced (3.380%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:16:50.907 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:49.747156+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e707: 8 total, 8 up, 7 in 2024-02-12T17:16:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:49.153568+0000 mgr.y (mgr.24481) 1022 : cluster [DBG] pgmap v1443: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 107 active+clean; 68 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 633 op/s; 104/3846 objects degraded (2.704%); 130/3846 objects misplaced (3.380%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T17:16:51.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:49.747156+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e707: 8 total, 8 up, 7 in 2024-02-12T17:16:51.248 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:16:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:50] "GET /metrics HTTP/1.1" 200 176840 "" "Prometheus/2.33.4" 2024-02-12T17:16:51.254 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:51.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:51.360 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:51.388 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:51.637 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:51.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:51.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:51.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:51.774 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:51.936 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:50.750815+0000 mon.a (mon.0) 2265 : cluster [DBG] osdmap e708: 8 total, 8 up, 7 in 2024-02-12T17:16:51.936 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:50.914340+0000 osd.7 (osd.7) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:51.936 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:50.914419+0000 osd.7 (osd.7) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:51.936 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:50.750815+0000 mon.a (mon.0) 2265 : cluster [DBG] osdmap e708: 8 total, 8 up, 7 in 2024-02-12T17:16:51.937 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:50.914340+0000 osd.7 (osd.7) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:51.937 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:50.914419+0000 osd.7 (osd.7) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:51.954 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-02-12T17:16:51.954 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-02-12T17:16:51.972 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:51.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:52.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:52.107 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:50.750815+0000 mon.a (mon.0) 2265 : cluster [DBG] osdmap e708: 8 total, 8 up, 7 in 2024-02-12T17:16:52.107 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:50.914340+0000 osd.7 (osd.7) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:52.107 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:50.914419+0000 osd.7 (osd.7) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:52.110 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:52.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:52.236 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:52.265 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:52.285 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:52.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:52.389 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:52.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:16:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:16:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:16:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:51.153957+0000 mgr.y (mgr.24481) 1023 : cluster [DBG] pgmap v1446: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 113 active+clean; 40 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 940 B/s rd, 0 B/s wr, 375 op/s; 10/2502 objects degraded (0.400%); 20/2502 objects misplaced (0.799%); 5.1 MiB/s, 83 objects/s recovering 2024-02-12T17:16:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:51.756795+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e709: 8 total, 8 up, 7 in 2024-02-12T17:16:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:51.800936+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check update: Degraded data redundancy: 10/2502 objects degraded (0.400%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:52.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:51.806590+0000 mon.a (mon.0) 2268 : cluster [DBG] osdmap e710: 8 total, 8 up, 7 in 2024-02-12T17:16:53.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:51.153957+0000 mgr.y (mgr.24481) 1023 : cluster [DBG] pgmap v1446: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 113 active+clean; 40 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 940 B/s rd, 0 B/s wr, 375 op/s; 10/2502 objects degraded (0.400%); 20/2502 objects misplaced (0.799%); 5.1 MiB/s, 83 objects/s recovering 2024-02-12T17:16:53.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:51.756795+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e709: 8 total, 8 up, 7 in 2024-02-12T17:16:53.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:51.800936+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check update: Degraded data redundancy: 10/2502 objects degraded (0.400%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:53.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:51.806590+0000 mon.a (mon.0) 2268 : cluster [DBG] osdmap e710: 8 total, 8 up, 7 in 2024-02-12T17:16:53.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:51.153957+0000 mgr.y (mgr.24481) 1023 : cluster [DBG] pgmap v1446: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 113 active+clean; 40 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 940 B/s rd, 0 B/s wr, 375 op/s; 10/2502 objects degraded (0.400%); 20/2502 objects misplaced (0.799%); 5.1 MiB/s, 83 objects/s recovering 2024-02-12T17:16:53.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:51.756795+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e709: 8 total, 8 up, 7 in 2024-02-12T17:16:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:51.800936+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check update: Degraded data redundancy: 10/2502 objects degraded (0.400%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-12T17:16:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:51.806590+0000 mon.a (mon.0) 2268 : cluster [DBG] osdmap e710: 8 total, 8 up, 7 in 2024-02-12T17:16:53.339 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:16:53.341 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:16:53.635 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:53.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:53.740 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:53.771 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:54.182 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:54.422 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:54.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:54.527 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:54.560 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:54.582 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:54.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:54.686 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:54.716 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:54.771 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:16:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:53.154449+0000 mgr.y (mgr.24481) 1024 : cluster [DBG] pgmap v1449: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 113 active+clean; 4.0 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 257 op/s; 10/771 objects degraded (1.297%); 20/771 objects misplaced (2.594%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T17:16:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:53.320763+0000 mgr.y (mgr.24481) 1025 : audit [DBG] from='client.36248 v1:172.21.15.33:0/2100699671' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:54.552667+0000 mon.c (mon.2) 315 : audit [INF] from='client.? v1:172.21.15.33:0/1745309129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:16:54.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:54.552941+0000 mon.a (mon.0) 2269 : audit [INF] from='client.36254 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:16:55.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:53.154449+0000 mgr.y (mgr.24481) 1024 : cluster [DBG] pgmap v1449: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 113 active+clean; 4.0 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 257 op/s; 10/771 objects degraded (1.297%); 20/771 objects misplaced (2.594%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T17:16:55.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:53.320763+0000 mgr.y (mgr.24481) 1025 : audit [DBG] from='client.36248 v1:172.21.15.33:0/2100699671' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:55.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:54.552667+0000 mon.c (mon.2) 315 : audit [INF] from='client.? v1:172.21.15.33:0/1745309129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:16:55.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:54.552941+0000 mon.a (mon.0) 2269 : audit [INF] from='client.36254 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:16:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:53.154449+0000 mgr.y (mgr.24481) 1024 : cluster [DBG] pgmap v1449: 118 pgs: 1 active+undersized+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 113 active+clean; 4.0 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 257 op/s; 10/771 objects degraded (1.297%); 20/771 objects misplaced (2.594%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T17:16:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:53.320763+0000 mgr.y (mgr.24481) 1025 : audit [DBG] from='client.36248 v1:172.21.15.33:0/2100699671' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:16:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:54.552667+0000 mon.c (mon.2) 315 : audit [INF] from='client.? v1:172.21.15.33:0/1745309129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:16:55.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:54.552941+0000 mon.a (mon.0) 2269 : audit [INF] from='client.36254 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:16:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:55.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:55.070 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1449,"stamp":"2024-02-12T17:16:53.154124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4200124,"num_objects":257,"num_object_clones":0,"num_object_copies":771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":1929,"num_read_kb":1681,"num_write":252311,"num_write_kb":8044010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24310,"num_bytes_recovered":1537432444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91159,"ondisk_log_size":91159,"up":354,"acting":352,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":364,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7991716,"kb_used_data":503724,"kb_used_omap":3,"kb_used_meta":7487868,"kb_avail":648224348,"statfs":{"total":671965249536,"available":663781732352,"internally_reserved":0,"allocated":515813376,"data_stored":505721446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":7667576893},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67174400,"num_objects":-1025,"num_object_clones":0,"num_object_copies":-3075,"num_objects_missing_on_primary":0,"num_objects_missing":-2,"num_objects_degraded":-94,"num_objects_misplaced":-110,"num_objects_unfound":0,"num_objects_dirty":-1025,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1025,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":6226012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001152"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":434,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951398+0000","last_change":"2024-02-12T17:14:10.348720+0000","last_active":"2024-02-12T17:16:50.951398+0000","last_peered":"2024-02-12T17:16:50.951398+0000","last_clean":"2024-02-12T17:16:50.951398+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:16:50.951398+0000","last_undegraded":"2024-02-12T17:16:50.951398+0000","last_fullsized":"2024-02-12T17:16:50.951398+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":214,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.718120+0000","last_change":"2024-02-12T17:13:46.694822+0000","last_active":"2024-02-12T17:16:45.718120+0000","last_peered":"2024-02-12T17:16:45.718120+0000","last_clean":"2024-02-12T17:16:45.718120+0000","last_became_active":"2024-02-12T17:13:46.694448+0000","last_became_peered":"2024-02-12T17:13:46.694448+0000","last_unstale":"2024-02-12T17:16:45.718120+0000","last_undegraded":"2024-02-12T17:16:45.718120+0000","last_fullsized":"2024-02-12T17:16:45.718120+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:12:55.862345+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:12:55.862345+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":461,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949105+0000","last_change":"2024-02-12T17:13:46.787428+0000","last_active":"2024-02-12T17:16:50.949105+0000","last_peered":"2024-02-12T17:16:50.949105+0000","last_clean":"2024-02-12T17:16:50.949105+0000","last_became_active":"2024-02-12T17:13:46.786996+0000","last_became_peered":"2024-02-12T17:13:46.786996+0000","last_unstale":"2024-02-12T17:16:50.949105+0000","last_undegraded":"2024-02-12T17:16:50.949105+0000","last_fullsized":"2024-02-12T17:16:50.949105+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:13:12.016227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:12.016227+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":489,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951260+0000","last_change":"2024-02-12T17:14:09.685503+0000","last_active":"2024-02-12T17:16:50.951260+0000","last_peered":"2024-02-12T17:16:50.951260+0000","last_clean":"2024-02-12T17:16:50.951260+0000","last_became_active":"2024-02-12T17:14:08.761132+0000","last_became_peered":"2024-02-12T17:14:08.761132+0000","last_unstale":"2024-02-12T17:16:50.951260+0000","last_undegraded":"2024-02-12T17:16:50.951260+0000","last_fullsized":"2024-02-12T17:16:50.951260+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:54.052006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.052006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":504,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949253+0000","last_change":"2024-02-12T17:13:46.753228+0000","last_active":"2024-02-12T17:16:50.949253+0000","last_peered":"2024-02-12T17:16:50.949253+0000","last_clean":"2024-02-12T17:16:50.949253+0000","last_became_active":"2024-02-12T17:13:46.752755+0000","last_became_peered":"2024-02-12T17:13:46.752755+0000","last_unstale":"2024-02-12T17:16:50.949253+0000","last_undegraded":"2024-02-12T17:16:50.949253+0000","last_fullsized":"2024-02-12T17:16:50.949253+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":1540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":166,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717213+0000","last_change":"2024-02-12T17:13:46.607814+0000","last_active":"2024-02-12T17:16:45.717213+0000","last_peered":"2024-02-12T17:16:45.717213+0000","last_clean":"2024-02-12T17:16:45.717213+0000","last_became_active":"2024-02-12T17:13:46.606367+0000","last_became_peered":"2024-02-12T17:13:46.606367+0000","last_unstale":"2024-02-12T17:16:45.717213+0000","last_undegraded":"2024-02-12T17:16:45.717213+0000","last_fullsized":"2024-02-12T17:16:45.717213+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":241,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.719159+0000","last_change":"2024-02-12T17:13:46.567244+0000","last_active":"2024-02-12T17:16:45.719159+0000","last_peered":"2024-02-12T17:16:45.719159+0000","last_clean":"2024-02-12T17:16:45.719159+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:16:45.719159+0000","last_undegraded":"2024-02-12T17:16:45.719159+0000","last_fullsized":"2024-02-12T17:16:45.719159+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":76,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.724400+0000","last_change":"2024-02-12T17:14:08.993489+0000","last_active":"2024-02-12T17:16:45.724400+0000","last_peered":"2024-02-12T17:16:45.724400+0000","last_clean":"2024-02-12T17:16:45.724400+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:16:45.724400+0000","last_undegraded":"2024-02-12T17:16:45.724400+0000","last_fullsized":"2024-02-12T17:16:45.724400+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":442,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.727669+0000","last_change":"2024-02-12T17:16:46.727669+0000","last_active":"2024-02-12T17:16:46.727669+0000","last_peered":"2024-02-12T17:16:46.727669+0000","last_clean":"2024-02-12T17:16:46.727669+0000","last_became_active":"2024-02-12T17:16:46.727369+0000","last_became_peered":"2024-02-12T17:16:46.727369+0000","last_unstale":"2024-02-12T17:16:46.727669+0000","last_undegraded":"2024-02-12T17:16:46.727669+0000","last_fullsized":"2024-02-12T17:16:46.727669+0000","mapping_epoch":705,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":402,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950732+0000","last_change":"2024-02-12T17:14:08.755098+0000","last_active":"2024-02-12T17:16:50.950732+0000","last_peered":"2024-02-12T17:16:50.950732+0000","last_clean":"2024-02-12T17:16:50.950732+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:16:50.950732+0000","last_undegraded":"2024-02-12T17:16:50.950732+0000","last_fullsized":"2024-02-12T17:16:50.950732+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:03.551629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:03.551629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":425,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951619+0000","last_change":"2024-02-12T17:12:12.141297+0000","last_active":"2024-02-12T17:16:50.951619+0000","last_peered":"2024-02-12T17:16:50.951619+0000","last_clean":"2024-02-12T17:16:50.951619+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:16:50.951619+0000","last_undegraded":"2024-02-12T17:16:50.951619+0000","last_fullsized":"2024-02-12T17:16:50.951619+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.141204+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":251,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.722288+0000","last_change":"2024-02-12T17:14:08.743209+0000","last_active":"2024-02-12T17:16:45.722288+0000","last_peered":"2024-02-12T17:16:45.722288+0000","last_clean":"2024-02-12T17:16:45.722288+0000","last_became_active":"2024-02-12T17:14:08.742886+0000","last_became_peered":"2024-02-12T17:14:08.742886+0000","last_unstale":"2024-02-12T17:16:45.722288+0000","last_undegraded":"2024-02-12T17:16:45.722288+0000","last_fullsized":"2024-02-12T17:16:45.722288+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.783527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.783527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":700,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.715442+0000","last_change":"2024-02-12T17:13:21.591627+0000","last_active":"2024-02-12T17:16:45.715442+0000","last_peered":"2024-02-12T17:16:45.715442+0000","last_clean":"2024-02-12T17:16:45.715442+0000","last_became_active":"2024-02-12T17:13:21.591246+0000","last_became_peered":"2024-02-12T17:13:21.591246+0000","last_unstale":"2024-02-12T17:16:45.715442+0000","last_undegraded":"2024-02-12T17:16:45.715442+0000","last_fullsized":"2024-02-12T17:16:45.715442+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.028495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":447,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949806+0000","last_change":"2024-02-12T17:13:46.558817+0000","last_active":"2024-02-12T17:16:50.949806+0000","last_peered":"2024-02-12T17:16:50.949806+0000","last_clean":"2024-02-12T17:16:50.949806+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:16:50.949806+0000","last_undegraded":"2024-02-12T17:16:50.949806+0000","last_fullsized":"2024-02-12T17:16:50.949806+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":98,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716550+0000","last_change":"2024-02-12T17:13:46.576676+0000","last_active":"2024-02-12T17:16:45.716550+0000","last_peered":"2024-02-12T17:16:45.716550+0000","last_clean":"2024-02-12T17:16:45.716550+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:16:45.716550+0000","last_undegraded":"2024-02-12T17:16:45.716550+0000","last_fullsized":"2024-02-12T17:16:45.716550+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":384,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951473+0000","last_change":"2024-02-12T17:14:08.763317+0000","last_active":"2024-02-12T17:16:50.951473+0000","last_peered":"2024-02-12T17:16:50.951473+0000","last_clean":"2024-02-12T17:16:50.951473+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:16:50.951473+0000","last_undegraded":"2024-02-12T17:16:50.951473+0000","last_fullsized":"2024-02-12T17:16:50.951473+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":138,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717924+0000","last_change":"2024-02-12T17:14:08.742170+0000","last_active":"2024-02-12T17:16:45.717924+0000","last_peered":"2024-02-12T17:16:45.717924+0000","last_clean":"2024-02-12T17:16:45.717924+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:16:45.717924+0000","last_undegraded":"2024-02-12T17:16:45.717924+0000","last_fullsized":"2024-02-12T17:16:45.717924+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":219,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.720021+0000","last_change":"2024-02-12T17:13:46.784204+0000","last_active":"2024-02-12T17:16:45.720021+0000","last_peered":"2024-02-12T17:16:45.720021+0000","last_clean":"2024-02-12T17:16:45.720021+0000","last_became_active":"2024-02-12T17:13:46.783737+0000","last_became_peered":"2024-02-12T17:13:46.783737+0000","last_unstale":"2024-02-12T17:16:45.720021+0000","last_undegraded":"2024-02-12T17:16:45.720021+0000","last_fullsized":"2024-02-12T17:16:45.720021+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:15.817751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.817751+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":774,"reported_epoch":708,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:16:50.950513+0000","last_change":"2024-02-12T17:16:45.076370+0000","last_active":"2024-02-12T17:16:50.950513+0000","last_peered":"2024-02-12T17:16:50.950513+0000","last_clean":"2024-02-12T17:13:44.507959+0000","last_became_active":"2024-02-12T17:14:08.758564+0000","last_became_peered":"2024-02-12T17:14:08.758564+0000","last_unstale":"2024-02-12T17:16:50.950513+0000","last_undegraded":"2024-02-12T17:16:50.950513+0000","last_fullsized":"2024-02-12T17:16:50.950513+0000","mapping_epoch":651,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":64,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.722703+0000","last_change":"2024-02-12T17:14:08.762208+0000","last_active":"2024-02-12T17:16:45.722703+0000","last_peered":"2024-02-12T17:16:45.722703+0000","last_clean":"2024-02-12T17:16:45.722703+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:16:45.722703+0000","last_undegraded":"2024-02-12T17:16:45.722703+0000","last_fullsized":"2024-02-12T17:16:45.722703+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:44.765042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.765042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":209,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.718554+0000","last_change":"2024-02-12T17:12:02.752352+0000","last_active":"2024-02-12T17:16:45.718554+0000","last_peered":"2024-02-12T17:16:45.718554+0000","last_clean":"2024-02-12T17:16:45.718554+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:16:45.718554+0000","last_undegraded":"2024-02-12T17:16:45.718554+0000","last_fullsized":"2024-02-12T17:16:45.718554+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:02.752280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:02.752280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":170,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716415+0000","last_change":"2024-02-12T17:13:46.582722+0000","last_active":"2024-02-12T17:16:45.716415+0000","last_peered":"2024-02-12T17:16:45.716415+0000","last_clean":"2024-02-12T17:16:45.716415+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:16:45.716415+0000","last_undegraded":"2024-02-12T17:16:45.716415+0000","last_fullsized":"2024-02-12T17:16:45.716415+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":813,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950346+0000","last_change":"2024-02-12T17:12:58.020311+0000","last_active":"2024-02-12T17:16:50.950346+0000","last_peered":"2024-02-12T17:16:50.950346+0000","last_clean":"2024-02-12T17:16:50.950346+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:16:50.950346+0000","last_undegraded":"2024-02-12T17:16:50.950346+0000","last_fullsized":"2024-02-12T17:16:50.950346+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:12:58.018712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.018712+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":64,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.725233+0000","last_change":"2024-02-12T17:14:08.872636+0000","last_active":"2024-02-12T17:16:45.725233+0000","last_peered":"2024-02-12T17:16:45.725233+0000","last_clean":"2024-02-12T17:16:45.725233+0000","last_became_active":"2024-02-12T17:14:08.872123+0000","last_became_peered":"2024-02-12T17:14:08.872123+0000","last_unstale":"2024-02-12T17:16:45.725233+0000","last_undegraded":"2024-02-12T17:16:45.725233+0000","last_fullsized":"2024-02-12T17:16:45.725233+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:06.904354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:06.904354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":796,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.714242+0000","last_change":"2024-02-12T17:12:12.838640+0000","last_active":"2024-02-12T17:16:45.714242+0000","last_peered":"2024-02-12T17:16:45.714242+0000","last_clean":"2024-02-12T17:16:45.714242+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:16:45.714242+0000","last_undegraded":"2024-02-12T17:16:45.714242+0000","last_fullsized":"2024-02-12T17:16:45.714242+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:12.838563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.838563+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"706'26","reported_seq":823,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:47.704695+0000","last_change":"2024-02-12T17:16:46.731065+0000","last_active":"2024-02-12T17:16:47.704695+0000","last_peered":"2024-02-12T17:16:47.704695+0000","last_clean":"2024-02-12T17:16:47.704695+0000","last_became_active":"2024-02-12T17:16:46.730642+0000","last_became_peered":"2024-02-12T17:16:46.730642+0000","last_unstale":"2024-02-12T17:16:47.704695+0000","last_undegraded":"2024-02-12T17:16:47.704695+0000","last_fullsized":"2024-02-12T17:16:47.704695+0000","mapping_epoch":705,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":700,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.713139+0000","last_change":"2024-02-12T17:12:54.656523+0000","last_active":"2024-02-12T17:16:45.713139+0000","last_peered":"2024-02-12T17:16:45.713139+0000","last_clean":"2024-02-12T17:16:45.713139+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:16:45.713139+0000","last_undegraded":"2024-02-12T17:16:45.713139+0000","last_fullsized":"2024-02-12T17:16:45.713139+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:54.656456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.656456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":438,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950267+0000","last_change":"2024-02-12T17:13:46.577769+0000","last_active":"2024-02-12T17:16:50.950267+0000","last_peered":"2024-02-12T17:16:50.950267+0000","last_clean":"2024-02-12T17:16:50.950267+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:16:50.950267+0000","last_undegraded":"2024-02-12T17:16:50.950267+0000","last_fullsized":"2024-02-12T17:16:50.950267+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":722,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716029+0000","last_change":"2024-02-12T17:13:21.591810+0000","last_active":"2024-02-12T17:16:45.716029+0000","last_peered":"2024-02-12T17:16:45.716029+0000","last_clean":"2024-02-12T17:16:45.716029+0000","last_became_active":"2024-02-12T17:13:21.591502+0000","last_became_peered":"2024-02-12T17:13:21.591502+0000","last_unstale":"2024-02-12T17:16:45.716029+0000","last_undegraded":"2024-02-12T17:16:45.716029+0000","last_fullsized":"2024-02-12T17:16:45.716029+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.882066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.882066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":243,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.722195+0000","last_change":"2024-02-12T17:13:46.651066+0000","last_active":"2024-02-12T17:16:45.722195+0000","last_peered":"2024-02-12T17:16:45.722195+0000","last_clean":"2024-02-12T17:16:45.722195+0000","last_became_active":"2024-02-12T17:13:46.650776+0000","last_became_peered":"2024-02-12T17:13:46.650776+0000","last_unstale":"2024-02-12T17:16:45.722195+0000","last_undegraded":"2024-02-12T17:16:45.722195+0000","last_fullsized":"2024-02-12T17:16:45.722195+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:50.786900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.786900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":427,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949462+0000","last_change":"2024-02-12T17:13:46.822111+0000","last_active":"2024-02-12T17:16:50.949462+0000","last_peered":"2024-02-12T17:16:50.949462+0000","last_clean":"2024-02-12T17:16:50.949462+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:16:50.949462+0000","last_undegraded":"2024-02-12T17:16:50.949462+0000","last_fullsized":"2024-02-12T17:16:50.949462+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":334,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.715746+0000","last_change":"2024-02-12T17:14:08.745944+0000","last_active":"2024-02-12T17:16:45.715746+0000","last_peered":"2024-02-12T17:16:45.715746+0000","last_clean":"2024-02-12T17:16:45.715746+0000","last_became_active":"2024-02-12T17:14:08.745729+0000","last_became_peered":"2024-02-12T17:14:08.745729+0000","last_unstale":"2024-02-12T17:16:45.715746+0000","last_undegraded":"2024-02-12T17:16:45.715746+0000","last_fullsized":"2024-02-12T17:16:45.715746+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:49.809844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:12:49.809844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":98,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717451+0000","last_change":"2024-02-12T17:13:46.559519+0000","last_active":"2024-02-12T17:16:45.717451+0000","last_peered":"2024-02-12T17:16:45.717451+0000","last_clean":"2024-02-12T17:16:45.717451+0000","last_became_active":"2024-02-12T17:13:46.559095+0000","last_became_peered":"2024-02-12T17:13:46.559095+0000","last_unstale":"2024-02-12T17:16:45.717451+0000","last_undegraded":"2024-02-12T17:16:45.717451+0000","last_fullsized":"2024-02-12T17:16:45.717451+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":74,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.724495+0000","last_change":"2024-02-12T17:14:08.993503+0000","last_active":"2024-02-12T17:16:45.724495+0000","last_peered":"2024-02-12T17:16:45.724495+0000","last_clean":"2024-02-12T17:16:45.724495+0000","last_became_active":"2024-02-12T17:14:08.861886+0000","last_became_peered":"2024-02-12T17:14:08.861886+0000","last_unstale":"2024-02-12T17:16:45.724495+0000","last_undegraded":"2024-02-12T17:16:45.724495+0000","last_fullsized":"2024-02-12T17:16:45.724495+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"706'15810","reported_seq":22800,"reported_epoch":706,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-12T17:16:48.744173+0000","last_change":"2024-02-12T17:16:44.669721+0000","last_active":"2024-02-12T17:16:48.744173+0000","last_peered":"2024-02-12T17:16:48.744173+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:14:08.966441+0000","last_became_peered":"2024-02-12T17:14:08.966441+0000","last_unstale":"2024-02-12T17:16:48.744173+0000","last_undegraded":"2024-02-12T17:14:08.734833+0000","last_fullsized":"2024-02-12T17:14:08.712653+0000","mapping_epoch":651,"log_start":"642'8721","ondisk_log_start":"642'8721","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":7089,"ondisk_log_size":7089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15810,"num_write_kb":506240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3413,"num_bytes_recovered":223674368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[2,4],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":426,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949254+0000","last_change":"2024-02-12T17:13:46.787305+0000","last_active":"2024-02-12T17:16:50.949254+0000","last_peered":"2024-02-12T17:16:50.949254+0000","last_clean":"2024-02-12T17:16:50.949254+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:16:50.949254+0000","last_undegraded":"2024-02-12T17:16:50.949254+0000","last_fullsized":"2024-02-12T17:16:50.949254+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":436,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.731039+0000","last_change":"2024-02-12T17:16:46.731039+0000","last_active":"2024-02-12T17:16:46.731039+0000","last_peered":"2024-02-12T17:16:46.731039+0000","last_clean":"2024-02-12T17:16:46.731039+0000","last_became_active":"2024-02-12T17:16:46.730672+0000","last_became_peered":"2024-02-12T17:16:46.730672+0000","last_unstale":"2024-02-12T17:16:46.731039+0000","last_undegraded":"2024-02-12T17:16:46.731039+0000","last_fullsized":"2024-02-12T17:16:46.731039+0000","mapping_epoch":705,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:12:09.882247+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:12:09.882247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":172,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717201+0000","last_change":"2024-02-12T17:13:46.744739+0000","last_active":"2024-02-12T17:16:45.717201+0000","last_peered":"2024-02-12T17:16:45.717201+0000","last_clean":"2024-02-12T17:16:45.717201+0000","last_became_active":"2024-02-12T17:13:46.744003+0000","last_became_peered":"2024-02-12T17:13:46.744003+0000","last_unstale":"2024-02-12T17:16:45.717201+0000","last_undegraded":"2024-02-12T17:16:45.717201+0000","last_fullsized":"2024-02-12T17:16:45.717201+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:51.126056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:51.126056+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":64,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.722858+0000","last_change":"2024-02-12T17:14:08.769824+0000","last_active":"2024-02-12T17:16:45.722858+0000","last_peered":"2024-02-12T17:16:45.722858+0000","last_clean":"2024-02-12T17:16:45.722858+0000","last_became_active":"2024-02-12T17:14:08.769221+0000","last_became_peered":"2024-02-12T17:14:08.769221+0000","last_unstale":"2024-02-12T17:16:45.722858+0000","last_undegraded":"2024-02-12T17:16:45.722858+0000","last_fullsized":"2024-02-12T17:16:45.722858+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"706'15564","reported_seq":15719,"reported_epoch":706,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:16:48.741273+0000","last_change":"2024-02-12T17:16:44.559067+0000","last_active":"2024-02-12T17:16:48.741273+0000","last_peered":"2024-02-12T17:16:48.741273+0000","last_clean":"2024-02-12T17:13:45.521351+0000","last_became_active":"2024-02-12T17:13:46.743970+0000","last_became_peered":"2024-02-12T17:13:46.743970+0000","last_unstale":"2024-02-12T17:16:48.741273+0000","last_undegraded":"2024-02-12T17:16:48.741273+0000","last_fullsized":"2024-02-12T17:16:48.741273+0000","mapping_epoch":630,"log_start":"661'9468","ondisk_log_start":"661'9468","created":597,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'5769","last_scrub_stamp":"2024-02-12T17:13:06.847792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:13:06.847792+0000","log_size":6096,"ondisk_log_size":6096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":458776,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15564,"num_write_kb":498241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":711,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.724255+0000","last_change":"2024-02-12T17:14:07.897458+0000","last_active":"2024-02-12T17:16:45.724255+0000","last_peered":"2024-02-12T17:16:45.724255+0000","last_clean":"2024-02-12T17:16:45.724255+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:16:45.724255+0000","last_undegraded":"2024-02-12T17:16:45.724255+0000","last_fullsized":"2024-02-12T17:16:45.724255+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.780381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.780381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":98,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717376+0000","last_change":"2024-02-12T17:13:46.791505+0000","last_active":"2024-02-12T17:16:45.717376+0000","last_peered":"2024-02-12T17:16:45.717376+0000","last_clean":"2024-02-12T17:16:45.717376+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:16:45.717376+0000","last_undegraded":"2024-02-12T17:16:45.717376+0000","last_fullsized":"2024-02-12T17:16:45.717376+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":178,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716629+0000","last_change":"2024-02-12T17:14:09.246132+0000","last_active":"2024-02-12T17:16:45.716629+0000","last_peered":"2024-02-12T17:16:45.716629+0000","last_clean":"2024-02-12T17:16:45.716629+0000","last_became_active":"2024-02-12T17:14:08.752149+0000","last_became_peered":"2024-02-12T17:14:08.752149+0000","last_unstale":"2024-02-12T17:16:45.716629+0000","last_undegraded":"2024-02-12T17:16:45.716629+0000","last_fullsized":"2024-02-12T17:16:45.716629+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:13:01.543047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.543047+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"706'24","reported_seq":549,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.732545+0000","last_change":"2024-02-12T17:16:46.730984+0000","last_active":"2024-02-12T17:16:46.732545+0000","last_peered":"2024-02-12T17:16:46.732545+0000","last_clean":"2024-02-12T17:16:46.732545+0000","last_became_active":"2024-02-12T17:16:46.730723+0000","last_became_peered":"2024-02-12T17:16:46.730723+0000","last_unstale":"2024-02-12T17:16:46.732545+0000","last_undegraded":"2024-02-12T17:16:46.732545+0000","last_fullsized":"2024-02-12T17:16:46.732545+0000","mapping_epoch":705,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'21","last_scrub_stamp":"2024-02-12T17:12:57.741798+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:12:57.741798+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"706'15720","reported_seq":20167,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950613+0000","last_change":"2024-02-12T17:16:46.776098+0000","last_active":"2024-02-12T17:16:50.950613+0000","last_peered":"2024-02-12T17:16:50.950613+0000","last_clean":"2024-02-12T17:16:50.950613+0000","last_became_active":"2024-02-12T17:16:46.731367+0000","last_became_peered":"2024-02-12T17:16:46.731367+0000","last_unstale":"2024-02-12T17:16:50.950613+0000","last_undegraded":"2024-02-12T17:16:50.950613+0000","last_fullsized":"2024-02-12T17:16:50.950613+0000","mapping_epoch":705,"log_start":"674'10663","ondisk_log_start":"674'10663","created":597,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5057,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15720,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1483,"num_bytes_recovered":97058816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":429,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950173+0000","last_change":"2024-02-12T17:13:21.603543+0000","last_active":"2024-02-12T17:16:50.950173+0000","last_peered":"2024-02-12T17:16:50.950173+0000","last_clean":"2024-02-12T17:16:50.950173+0000","last_became_active":"2024-02-12T17:13:21.603226+0000","last_became_peered":"2024-02-12T17:13:21.603226+0000","last_unstale":"2024-02-12T17:16:50.950173+0000","last_undegraded":"2024-02-12T17:16:50.950173+0000","last_fullsized":"2024-02-12T17:16:50.950173+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.262134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.262134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":742,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.713154+0000","last_change":"2024-02-12T17:13:46.566957+0000","last_active":"2024-02-12T17:16:45.713154+0000","last_peered":"2024-02-12T17:16:45.713154+0000","last_clean":"2024-02-12T17:16:45.713154+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:16:45.713154+0000","last_undegraded":"2024-02-12T17:16:45.713154+0000","last_fullsized":"2024-02-12T17:16:45.713154+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":229,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.728013+0000","last_change":"2024-02-12T17:13:46.607171+0000","last_active":"2024-02-12T17:16:46.728013+0000","last_peered":"2024-02-12T17:16:46.728013+0000","last_clean":"2024-02-12T17:16:46.728013+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:16:46.728013+0000","last_undegraded":"2024-02-12T17:16:46.728013+0000","last_fullsized":"2024-02-12T17:16:46.728013+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":667,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.731125+0000","last_change":"2024-02-12T17:16:46.731125+0000","last_active":"2024-02-12T17:16:46.731125+0000","last_peered":"2024-02-12T17:16:46.731125+0000","last_clean":"2024-02-12T17:16:46.731125+0000","last_became_active":"2024-02-12T17:16:46.730767+0000","last_became_peered":"2024-02-12T17:16:46.730767+0000","last_unstale":"2024-02-12T17:16:46.731125+0000","last_undegraded":"2024-02-12T17:16:46.731125+0000","last_fullsized":"2024-02-12T17:16:46.731125+0000","mapping_epoch":705,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:13:02.852563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:13:02.852563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"706'15974","reported_seq":16094,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949784+0000","last_change":"2024-02-12T17:11:59.008876+0000","last_active":"2024-02-12T17:16:50.949784+0000","last_peered":"2024-02-12T17:16:50.949784+0000","last_clean":"2024-02-12T17:16:50.949784+0000","last_became_active":"2024-02-12T17:11:59.008587+0000","last_became_peered":"2024-02-12T17:11:59.008587+0000","last_unstale":"2024-02-12T17:16:50.949784+0000","last_undegraded":"2024-02-12T17:16:50.949784+0000","last_fullsized":"2024-02-12T17:16:50.949784+0000","mapping_epoch":597,"log_start":"675'10918","ondisk_log_start":"675'10918","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5056,"ondisk_log_size":5056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15974,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":711,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716684+0000","last_change":"2024-02-12T17:13:00.766698+0000","last_active":"2024-02-12T17:16:45.716684+0000","last_peered":"2024-02-12T17:16:45.716684+0000","last_clean":"2024-02-12T17:16:45.716684+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:16:45.716684+0000","last_undegraded":"2024-02-12T17:16:45.716684+0000","last_fullsized":"2024-02-12T17:16:45.716684+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.766590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":389,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.370779+0000","last_change":"2024-02-12T17:13:46.584941+0000","last_active":"2024-02-12T17:16:48.370779+0000","last_peered":"2024-02-12T17:16:48.370779+0000","last_clean":"2024-02-12T17:16:48.370779+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:16:48.370779+0000","last_undegraded":"2024-02-12T17:16:48.370779+0000","last_fullsized":"2024-02-12T17:16:48.370779+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":395,"num_read_kb":390,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":375,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951706+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:16:50.951706+0000","last_peered":"2024-02-12T17:16:50.951706+0000","last_clean":"2024-02-12T17:16:50.951706+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:16:50.951706+0000","last_undegraded":"2024-02-12T17:16:50.951706+0000","last_fullsized":"2024-02-12T17:16:50.951706+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":171,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717856+0000","last_change":"2024-02-12T17:14:09.321979+0000","last_active":"2024-02-12T17:16:45.717856+0000","last_peered":"2024-02-12T17:16:45.717856+0000","last_clean":"2024-02-12T17:16:45.717856+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:16:45.717856+0000","last_undegraded":"2024-02-12T17:16:45.717856+0000","last_fullsized":"2024-02-12T17:16:45.717856+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'22","last_scrub_stamp":"2024-02-12T17:02:25.762172+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:02:25.762172+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":635,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.713834+0000","last_change":"2024-02-12T17:13:10.412138+0000","last_active":"2024-02-12T17:16:45.713834+0000","last_peered":"2024-02-12T17:16:45.713834+0000","last_clean":"2024-02-12T17:16:45.713834+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:16:45.713834+0000","last_undegraded":"2024-02-12T17:16:45.713834+0000","last_fullsized":"2024-02-12T17:16:45.713834+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:13:10.412031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"706'15538","reported_seq":15658,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951097+0000","last_change":"2024-02-12T17:11:59.007548+0000","last_active":"2024-02-12T17:16:50.951097+0000","last_peered":"2024-02-12T17:16:50.951097+0000","last_clean":"2024-02-12T17:16:50.951097+0000","last_became_active":"2024-02-12T17:11:59.007214+0000","last_became_peered":"2024-02-12T17:11:59.007214+0000","last_unstale":"2024-02-12T17:16:50.951097+0000","last_undegraded":"2024-02-12T17:16:50.951097+0000","last_fullsized":"2024-02-12T17:16:50.951097+0000","mapping_epoch":597,"log_start":"674'10455","ondisk_log_start":"674'10455","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5083,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15538,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":774,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.714856+0000","last_change":"2024-02-12T17:13:46.826853+0000","last_active":"2024-02-12T17:16:45.714856+0000","last_peered":"2024-02-12T17:16:45.714856+0000","last_clean":"2024-02-12T17:16:45.714856+0000","last_became_active":"2024-02-12T17:13:46.821418+0000","last_became_peered":"2024-02-12T17:13:46.821418+0000","last_unstale":"2024-02-12T17:16:45.714856+0000","last_undegraded":"2024-02-12T17:16:45.714856+0000","last_fullsized":"2024-02-12T17:16:45.714856+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.951964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.951964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":407,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951137+0000","last_change":"2024-02-12T17:12:59.006372+0000","last_active":"2024-02-12T17:16:50.951137+0000","last_peered":"2024-02-12T17:16:50.951137+0000","last_clean":"2024-02-12T17:16:50.951137+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:16:50.951137+0000","last_undegraded":"2024-02-12T17:16:50.951137+0000","last_fullsized":"2024-02-12T17:16:50.951137+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.006260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.006260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":682,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.713470+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:16:45.713470+0000","last_peered":"2024-02-12T17:16:45.713470+0000","last_clean":"2024-02-12T17:16:45.713470+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:16:45.713470+0000","last_undegraded":"2024-02-12T17:16:45.713470+0000","last_fullsized":"2024-02-12T17:16:45.713470+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":751,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716207+0000","last_change":"2024-02-12T17:14:08.743668+0000","last_active":"2024-02-12T17:16:45.716207+0000","last_peered":"2024-02-12T17:16:45.716207+0000","last_clean":"2024-02-12T17:16:45.716207+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:16:45.716207+0000","last_undegraded":"2024-02-12T17:16:45.716207+0000","last_fullsized":"2024-02-12T17:16:45.716207+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":316,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.719567+0000","last_change":"2024-02-12T17:14:08.781462+0000","last_active":"2024-02-12T17:16:45.719567+0000","last_peered":"2024-02-12T17:16:45.719567+0000","last_clean":"2024-02-12T17:16:45.719567+0000","last_became_active":"2024-02-12T17:14:08.726463+0000","last_became_peered":"2024-02-12T17:14:08.726463+0000","last_unstale":"2024-02-12T17:16:45.719567+0000","last_undegraded":"2024-02-12T17:16:45.719567+0000","last_fullsized":"2024-02-12T17:16:45.719567+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:47.763264+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.763264+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":713,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.715828+0000","last_change":"2024-02-12T17:12:03.922963+0000","last_active":"2024-02-12T17:16:45.715828+0000","last_peered":"2024-02-12T17:16:45.715828+0000","last_clean":"2024-02-12T17:16:45.715828+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:16:45.715828+0000","last_undegraded":"2024-02-12T17:16:45.715828+0000","last_fullsized":"2024-02-12T17:16:45.715828+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:12:03.922893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.922893+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"706'15748","reported_seq":18895,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951316+0000","last_change":"2024-02-12T17:16:46.779291+0000","last_active":"2024-02-12T17:16:50.951316+0000","last_peered":"2024-02-12T17:16:50.951316+0000","last_clean":"2024-02-12T17:16:50.951316+0000","last_became_active":"2024-02-12T17:16:46.739062+0000","last_became_peered":"2024-02-12T17:16:46.739062+0000","last_unstale":"2024-02-12T17:16:50.951316+0000","last_undegraded":"2024-02-12T17:16:50.951316+0000","last_fullsized":"2024-02-12T17:16:50.951316+0000","mapping_epoch":705,"log_start":"674'10742","ondisk_log_start":"674'10742","created":597,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5006,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15748,"num_write_kb":503936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1500,"num_bytes_recovered":98304000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":98,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717461+0000","last_change":"2024-02-12T17:13:46.608049+0000","last_active":"2024-02-12T17:16:45.717461+0000","last_peered":"2024-02-12T17:16:45.717461+0000","last_clean":"2024-02-12T17:16:45.717461+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:16:45.717461+0000","last_undegraded":"2024-02-12T17:16:45.717461+0000","last_fullsized":"2024-02-12T17:16:45.717461+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.726388+0000","last_change":"2024-02-12T17:14:08.859925+0000","last_active":"2024-02-12T17:16:45.726388+0000","last_peered":"2024-02-12T17:16:45.726388+0000","last_clean":"2024-02-12T17:16:45.726388+0000","last_became_active":"2024-02-12T17:14:08.859396+0000","last_became_peered":"2024-02-12T17:14:08.859396+0000","last_unstale":"2024-02-12T17:16:45.726388+0000","last_undegraded":"2024-02-12T17:16:45.726388+0000","last_fullsized":"2024-02-12T17:16:45.726388+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:03.904976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.904976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":751,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716139+0000","last_change":"2024-02-12T17:14:05.666060+0000","last_active":"2024-02-12T17:16:45.716139+0000","last_peered":"2024-02-12T17:16:45.716139+0000","last_clean":"2024-02-12T17:16:45.716139+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:16:45.716139+0000","last_undegraded":"2024-02-12T17:16:45.716139+0000","last_fullsized":"2024-02-12T17:16:45.716139+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":78,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.724589+0000","last_change":"2024-02-12T17:14:09.046726+0000","last_active":"2024-02-12T17:16:45.724589+0000","last_peered":"2024-02-12T17:16:45.724589+0000","last_clean":"2024-02-12T17:16:45.724589+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:16:45.724589+0000","last_undegraded":"2024-02-12T17:16:45.724589+0000","last_fullsized":"2024-02-12T17:16:45.724589+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'24","last_scrub_stamp":"2024-02-12T17:05:55.847927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:05:55.847927+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":754,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.714773+0000","last_change":"2024-02-12T17:13:46.827015+0000","last_active":"2024-02-12T17:16:45.714773+0000","last_peered":"2024-02-12T17:16:45.714773+0000","last_clean":"2024-02-12T17:16:45.714773+0000","last_became_active":"2024-02-12T17:13:46.821796+0000","last_became_peered":"2024-02-12T17:13:46.821796+0000","last_unstale":"2024-02-12T17:16:45.714773+0000","last_undegraded":"2024-02-12T17:16:45.714773+0000","last_fullsized":"2024-02-12T17:16:45.714773+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.698727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.698727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"706'15556","reported_seq":22031,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.943023+0000","last_change":"2024-02-12T17:16:45.797129+0000","last_active":"2024-02-12T17:16:48.943023+0000","last_peered":"2024-02-12T17:16:48.943023+0000","last_clean":"2024-02-12T17:16:48.943023+0000","last_became_active":"2024-02-12T17:16:45.721814+0000","last_became_peered":"2024-02-12T17:16:45.721814+0000","last_unstale":"2024-02-12T17:16:48.943023+0000","last_undegraded":"2024-02-12T17:16:48.943023+0000","last_fullsized":"2024-02-12T17:16:48.943023+0000","mapping_epoch":704,"log_start":"674'10426","ondisk_log_start":"674'10426","created":597,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5130,"ondisk_log_size":5130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":497792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3126,"num_bytes_recovered":204865536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":129,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717114+0000","last_change":"2024-02-12T17:13:46.579742+0000","last_active":"2024-02-12T17:16:45.717114+0000","last_peered":"2024-02-12T17:16:45.717114+0000","last_clean":"2024-02-12T17:16:45.717114+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:16:45.717114+0000","last_undegraded":"2024-02-12T17:16:45.717114+0000","last_fullsized":"2024-02-12T17:16:45.717114+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":765,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716755+0000","last_change":"2024-02-12T17:14:08.746702+0000","last_active":"2024-02-12T17:16:45.716755+0000","last_peered":"2024-02-12T17:16:45.716755+0000","last_clean":"2024-02-12T17:16:45.716755+0000","last_became_active":"2024-02-12T17:14:08.746395+0000","last_became_peered":"2024-02-12T17:14:08.746395+0000","last_unstale":"2024-02-12T17:16:45.716755+0000","last_undegraded":"2024-02-12T17:16:45.716755+0000","last_fullsized":"2024-02-12T17:16:45.716755+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:45.764955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:45.764955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":82,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.725119+0000","last_change":"2024-02-12T17:14:11.148689+0000","last_active":"2024-02-12T17:16:45.725119+0000","last_peered":"2024-02-12T17:16:45.725119+0000","last_clean":"2024-02-12T17:16:45.725119+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:16:45.725119+0000","last_undegraded":"2024-02-12T17:16:45.725119+0000","last_fullsized":"2024-02-12T17:16:45.725119+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":207,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.718544+0000","last_change":"2024-02-12T17:12:53.746937+0000","last_active":"2024-02-12T17:16:45.718544+0000","last_peered":"2024-02-12T17:16:45.718544+0000","last_clean":"2024-02-12T17:16:45.718544+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:16:45.718544+0000","last_undegraded":"2024-02-12T17:16:45.718544+0000","last_fullsized":"2024-02-12T17:16:45.718544+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.746858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.746858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"706'15722","reported_seq":15960,"reported_epoch":706,"state":"active+undersized+remapped+backfilling","last_fresh":"2024-02-12T17:16:48.946075+0000","last_change":"2024-02-12T17:16:48.945751+0000","last_active":"2024-02-12T17:16:48.946075+0000","last_peered":"2024-02-12T17:16:48.946075+0000","last_clean":"2024-02-12T17:11:57.982521+0000","last_became_active":"2024-02-12T17:13:46.832453+0000","last_became_peered":"2024-02-12T17:13:46.832453+0000","last_unstale":"2024-02-12T17:16:48.946075+0000","last_undegraded":"2024-02-12T17:16:48.946075+0000","last_fullsized":"2024-02-12T17:13:46.642131+0000","mapping_epoch":631,"log_start":"665'9870","ondisk_log_start":"665'9870","created":597,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5852,"ondisk_log_size":5852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15722,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2],"avail_no_missing":["7","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":416,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951179+0000","last_change":"2024-02-12T17:13:21.589965+0000","last_active":"2024-02-12T17:16:50.951179+0000","last_peered":"2024-02-12T17:16:50.951179+0000","last_clean":"2024-02-12T17:16:50.951179+0000","last_became_active":"2024-02-12T17:13:21.589584+0000","last_became_peered":"2024-02-12T17:13:21.589584+0000","last_unstale":"2024-02-12T17:16:50.951179+0000","last_undegraded":"2024-02-12T17:16:50.951179+0000","last_fullsized":"2024-02-12T17:16:50.951179+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:01.059570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.059570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":145,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716775+0000","last_change":"2024-02-12T17:14:18.716530+0000","last_active":"2024-02-12T17:16:45.716775+0000","last_peered":"2024-02-12T17:16:45.716775+0000","last_clean":"2024-02-12T17:16:45.716775+0000","last_became_active":"2024-02-12T17:14:18.716232+0000","last_became_peered":"2024-02-12T17:14:18.716232+0000","last_unstale":"2024-02-12T17:16:45.716775+0000","last_undegraded":"2024-02-12T17:16:45.716775+0000","last_fullsized":"2024-02-12T17:16:45.716775+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":475,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949909+0000","last_change":"2024-02-12T17:14:10.144923+0000","last_active":"2024-02-12T17:16:50.949909+0000","last_peered":"2024-02-12T17:16:50.949909+0000","last_clean":"2024-02-12T17:16:50.949909+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:16:50.949909+0000","last_undegraded":"2024-02-12T17:16:50.949909+0000","last_fullsized":"2024-02-12T17:16:50.949909+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'30","last_scrub_stamp":"2024-02-12T17:02:18.703128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:02:18.703128+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":695,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951532+0000","last_change":"2024-02-12T17:14:08.765556+0000","last_active":"2024-02-12T17:16:50.951532+0000","last_peered":"2024-02-12T17:16:50.951532+0000","last_clean":"2024-02-12T17:16:50.951532+0000","last_became_active":"2024-02-12T17:14:08.757310+0000","last_became_peered":"2024-02-12T17:14:08.757310+0000","last_unstale":"2024-02-12T17:16:50.951532+0000","last_undegraded":"2024-02-12T17:16:50.951532+0000","last_fullsized":"2024-02-12T17:16:50.951532+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:58.871569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.871569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"706'15732","reported_seq":21122,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.864626+0000","last_change":"2024-02-12T17:16:45.793491+0000","last_active":"2024-02-12T17:16:48.864626+0000","last_peered":"2024-02-12T17:16:48.864626+0000","last_clean":"2024-02-12T17:16:48.864626+0000","last_became_active":"2024-02-12T17:16:45.726761+0000","last_became_peered":"2024-02-12T17:16:45.726761+0000","last_unstale":"2024-02-12T17:16:48.864626+0000","last_undegraded":"2024-02-12T17:16:48.864626+0000","last_fullsized":"2024-02-12T17:16:48.864626+0000","mapping_epoch":704,"log_start":"674'10608","ondisk_log_start":"674'10608","created":597,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5124,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15732,"num_write_kb":503424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1829,"num_bytes_recovered":119865344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":744,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716536+0000","last_change":"2024-02-12T17:13:08.874063+0000","last_active":"2024-02-12T17:16:45.716536+0000","last_peered":"2024-02-12T17:16:45.716536+0000","last_clean":"2024-02-12T17:16:45.716536+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:16:45.716536+0000","last_undegraded":"2024-02-12T17:16:45.716536+0000","last_fullsized":"2024-02-12T17:16:45.716536+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:13:08.873961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:13:08.873961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.719004+0000","last_change":"2024-02-12T17:14:14.843264+0000","last_active":"2024-02-12T17:16:45.719004+0000","last_peered":"2024-02-12T17:16:45.719004+0000","last_clean":"2024-02-12T17:16:45.719004+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:16:45.719004+0000","last_undegraded":"2024-02-12T17:16:45.719004+0000","last_fullsized":"2024-02-12T17:16:45.719004+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:56.688982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.688982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":890,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.713052+0000","last_change":"2024-02-12T17:14:09.517724+0000","last_active":"2024-02-12T17:16:45.713052+0000","last_peered":"2024-02-12T17:16:45.713052+0000","last_clean":"2024-02-12T17:16:45.713052+0000","last_became_active":"2024-02-12T17:14:08.740691+0000","last_became_peered":"2024-02-12T17:14:08.740691+0000","last_unstale":"2024-02-12T17:16:45.713052+0000","last_undegraded":"2024-02-12T17:16:45.713052+0000","last_fullsized":"2024-02-12T17:16:45.713052+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:12:05.850456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.850456+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"706'33","reported_seq":681,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949891+0000","last_change":"2024-02-12T17:16:46.736263+0000","last_active":"2024-02-12T17:16:50.949891+0000","last_peered":"2024-02-12T17:16:50.949891+0000","last_clean":"2024-02-12T17:16:50.949891+0000","last_became_active":"2024-02-12T17:16:46.735883+0000","last_became_peered":"2024-02-12T17:16:46.735883+0000","last_unstale":"2024-02-12T17:16:50.949891+0000","last_undegraded":"2024-02-12T17:16:50.949891+0000","last_fullsized":"2024-02-12T17:16:50.949891+0000","mapping_epoch":705,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"706'15642","reported_seq":16022,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.744598+0000","last_change":"2024-02-12T17:16:46.776133+0000","last_active":"2024-02-12T17:16:48.744598+0000","last_peered":"2024-02-12T17:16:48.744598+0000","last_clean":"2024-02-12T17:16:48.744598+0000","last_became_active":"2024-02-12T17:16:46.735858+0000","last_became_peered":"2024-02-12T17:16:46.735858+0000","last_unstale":"2024-02-12T17:16:48.744598+0000","last_undegraded":"2024-02-12T17:16:48.744598+0000","last_fullsized":"2024-02-12T17:16:48.744598+0000","mapping_epoch":705,"log_start":"662'9551","ondisk_log_start":"662'9551","created":597,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6091,"ondisk_log_size":6091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15642,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":7667712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":418,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950146+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:16:50.950146+0000","last_peered":"2024-02-12T17:16:50.950146+0000","last_clean":"2024-02-12T17:16:50.950146+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:16:50.950146+0000","last_undegraded":"2024-02-12T17:16:50.950146+0000","last_fullsized":"2024-02-12T17:16:50.950146+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"642'31","reported_seq":1123,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.720298+0000","last_change":"2024-02-12T17:13:53.773053+0000","last_active":"2024-02-12T17:16:46.720298+0000","last_peered":"2024-02-12T17:16:46.720298+0000","last_clean":"2024-02-12T17:16:46.720298+0000","last_became_active":"2024-02-12T17:13:53.772744+0000","last_became_peered":"2024-02-12T17:13:53.772744+0000","last_unstale":"2024-02-12T17:16:46.720298+0000","last_undegraded":"2024-02-12T17:16:46.720298+0000","last_fullsized":"2024-02-12T17:16:46.720298+0000","mapping_epoch":638,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'28","last_scrub_stamp":"2024-02-12T17:12:07.826227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:12:07.826227+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":419,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716068+0000","last_change":"2024-02-12T17:14:10.756477+0000","last_active":"2024-02-12T17:16:45.716068+0000","last_peered":"2024-02-12T17:16:45.716068+0000","last_clean":"2024-02-12T17:16:45.716068+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:16:45.716068+0000","last_undegraded":"2024-02-12T17:16:45.716068+0000","last_fullsized":"2024-02-12T17:16:45.716068+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:16.873570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.873570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":279,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.718634+0000","last_change":"2024-02-12T17:13:46.807564+0000","last_active":"2024-02-12T17:16:45.718634+0000","last_peered":"2024-02-12T17:16:45.718634+0000","last_clean":"2024-02-12T17:16:45.718634+0000","last_became_active":"2024-02-12T17:13:46.807214+0000","last_became_peered":"2024-02-12T17:13:46.807214+0000","last_unstale":"2024-02-12T17:16:45.718634+0000","last_undegraded":"2024-02-12T17:16:45.718634+0000","last_fullsized":"2024-02-12T17:16:45.718634+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:04.705646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.705646+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"706'15878","reported_seq":19993,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.922599+0000","last_change":"2024-02-12T17:16:44.884171+0000","last_active":"2024-02-12T17:16:48.922599+0000","last_peered":"2024-02-12T17:16:48.922599+0000","last_clean":"2024-02-12T17:16:48.922599+0000","last_became_active":"2024-02-12T17:16:44.723441+0000","last_became_peered":"2024-02-12T17:16:44.723441+0000","last_unstale":"2024-02-12T17:16:48.922599+0000","last_undegraded":"2024-02-12T17:16:48.922599+0000","last_fullsized":"2024-02-12T17:16:48.922599+0000","mapping_epoch":703,"log_start":"674'10771","ondisk_log_start":"674'10771","created":597,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5107,"ondisk_log_size":5107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15878,"num_write_kb":508096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1986,"num_bytes_recovered":130023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":749,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.714408+0000","last_change":"2024-02-12T17:14:08.740896+0000","last_active":"2024-02-12T17:16:45.714408+0000","last_peered":"2024-02-12T17:16:45.714408+0000","last_clean":"2024-02-12T17:16:45.714408+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:16:45.714408+0000","last_undegraded":"2024-02-12T17:16:45.714408+0000","last_fullsized":"2024-02-12T17:16:45.714408+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"668'27","reported_seq":1148,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.720148+0000","last_change":"2024-02-12T17:14:24.235718+0000","last_active":"2024-02-12T17:16:46.720148+0000","last_peered":"2024-02-12T17:16:46.720148+0000","last_clean":"2024-02-12T17:16:46.720148+0000","last_became_active":"2024-02-12T17:14:24.235386+0000","last_became_peered":"2024-02-12T17:14:24.235386+0000","last_unstale":"2024-02-12T17:16:46.720148+0000","last_undegraded":"2024-02-12T17:16:46.720148+0000","last_fullsized":"2024-02-12T17:16:46.720148+0000","mapping_epoch":665,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'24","last_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.647212+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":116,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717031+0000","last_change":"2024-02-12T17:14:05.106537+0000","last_active":"2024-02-12T17:16:45.717031+0000","last_peered":"2024-02-12T17:16:45.717031+0000","last_clean":"2024-02-12T17:16:45.717031+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:16:45.717031+0000","last_undegraded":"2024-02-12T17:16:45.717031+0000","last_fullsized":"2024-02-12T17:16:45.717031+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":164,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.718010+0000","last_change":"2024-02-12T17:13:46.819259+0000","last_active":"2024-02-12T17:16:45.718010+0000","last_peered":"2024-02-12T17:16:45.718010+0000","last_clean":"2024-02-12T17:16:45.718010+0000","last_became_active":"2024-02-12T17:13:46.818771+0000","last_became_peered":"2024-02-12T17:13:46.818771+0000","last_unstale":"2024-02-12T17:16:45.718010+0000","last_undegraded":"2024-02-12T17:16:45.718010+0000","last_fullsized":"2024-02-12T17:16:45.718010+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"706'15407","reported_seq":28345,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.746750+0000","last_change":"2024-02-12T17:14:32.710386+0000","last_active":"2024-02-12T17:16:48.746750+0000","last_peered":"2024-02-12T17:16:48.746750+0000","last_clean":"2024-02-12T17:16:48.746750+0000","last_became_active":"2024-02-12T17:14:32.696812+0000","last_became_peered":"2024-02-12T17:14:32.696812+0000","last_unstale":"2024-02-12T17:16:48.746750+0000","last_undegraded":"2024-02-12T17:16:48.746750+0000","last_fullsized":"2024-02-12T17:16:48.746750+0000","mapping_epoch":671,"log_start":"658'9300","ondisk_log_start":"658'9300","created":597,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6107,"ondisk_log_size":6107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15407,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6402,"num_bytes_recovered":419561472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":96,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716692+0000","last_change":"2024-02-12T17:13:46.805110+0000","last_active":"2024-02-12T17:16:45.716692+0000","last_peered":"2024-02-12T17:16:45.716692+0000","last_clean":"2024-02-12T17:16:45.716692+0000","last_became_active":"2024-02-12T17:13:46.802275+0000","last_became_peered":"2024-02-12T17:13:46.802275+0000","last_unstale":"2024-02-12T17:16:45.716692+0000","last_undegraded":"2024-02-12T17:16:45.716692+0000","last_fullsized":"2024-02-12T17:16:45.716692+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":651,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.722341+0000","last_change":"2024-02-12T17:13:07.711701+0000","last_active":"2024-02-12T17:16:46.722341+0000","last_peered":"2024-02-12T17:16:46.722341+0000","last_clean":"2024-02-12T17:16:46.722341+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:16:46.722341+0000","last_undegraded":"2024-02-12T17:16:46.722341+0000","last_fullsized":"2024-02-12T17:16:46.722341+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:13:07.711601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:07.711601+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":116,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717638+0000","last_change":"2024-02-12T17:14:00.999940+0000","last_active":"2024-02-12T17:16:45.717638+0000","last_peered":"2024-02-12T17:16:45.717638+0000","last_clean":"2024-02-12T17:16:45.717638+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:16:45.717638+0000","last_undegraded":"2024-02-12T17:16:45.717638+0000","last_fullsized":"2024-02-12T17:16:45.717638+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":512,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.949399+0000","last_change":"2024-02-12T17:13:46.806094+0000","last_active":"2024-02-12T17:16:50.949399+0000","last_peered":"2024-02-12T17:16:50.949399+0000","last_clean":"2024-02-12T17:16:50.949399+0000","last_became_active":"2024-02-12T17:13:46.805712+0000","last_became_peered":"2024-02-12T17:13:46.805712+0000","last_unstale":"2024-02-12T17:16:50.949399+0000","last_undegraded":"2024-02-12T17:16:50.949399+0000","last_fullsized":"2024-02-12T17:16:50.949399+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"706'15864","reported_seq":15984,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950887+0000","last_change":"2024-02-12T17:11:59.496950+0000","last_active":"2024-02-12T17:16:50.950887+0000","last_peered":"2024-02-12T17:16:50.950887+0000","last_clean":"2024-02-12T17:16:50.950887+0000","last_became_active":"2024-02-12T17:11:59.496585+0000","last_became_peered":"2024-02-12T17:11:59.496585+0000","last_unstale":"2024-02-12T17:16:50.950887+0000","last_undegraded":"2024-02-12T17:16:50.950887+0000","last_fullsized":"2024-02-12T17:16:50.950887+0000","mapping_epoch":597,"log_start":"674'10848","ondisk_log_start":"674'10848","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5016,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15864,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":438,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950509+0000","last_change":"2024-02-12T17:13:21.603687+0000","last_active":"2024-02-12T17:16:50.950509+0000","last_peered":"2024-02-12T17:16:50.950509+0000","last_clean":"2024-02-12T17:16:50.950509+0000","last_became_active":"2024-02-12T17:13:21.603332+0000","last_became_peered":"2024-02-12T17:13:21.603332+0000","last_unstale":"2024-02-12T17:16:50.950509+0000","last_undegraded":"2024-02-12T17:16:50.950509+0000","last_fullsized":"2024-02-12T17:16:50.950509+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":127,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717294+0000","last_change":"2024-02-12T17:13:46.804446+0000","last_active":"2024-02-12T17:16:45.717294+0000","last_peered":"2024-02-12T17:16:45.717294+0000","last_clean":"2024-02-12T17:16:45.717294+0000","last_became_active":"2024-02-12T17:13:46.804033+0000","last_became_peered":"2024-02-12T17:13:46.804033+0000","last_unstale":"2024-02-12T17:16:45.717294+0000","last_undegraded":"2024-02-12T17:16:45.717294+0000","last_fullsized":"2024-02-12T17:16:45.717294+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.793569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.793569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":729,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.715773+0000","last_change":"2024-02-12T17:13:56.661004+0000","last_active":"2024-02-12T17:16:45.715773+0000","last_peered":"2024-02-12T17:16:45.715773+0000","last_clean":"2024-02-12T17:16:45.715773+0000","last_became_active":"2024-02-12T17:13:56.660707+0000","last_became_peered":"2024-02-12T17:13:56.660707+0000","last_unstale":"2024-02-12T17:16:45.715773+0000","last_undegraded":"2024-02-12T17:16:45.715773+0000","last_fullsized":"2024-02-12T17:16:45.715773+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.685601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.685601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":196,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.717639+0000","last_change":"2024-02-12T17:13:46.757888+0000","last_active":"2024-02-12T17:16:45.717639+0000","last_peered":"2024-02-12T17:16:45.717639+0000","last_clean":"2024-02-12T17:16:45.717639+0000","last_became_active":"2024-02-12T17:13:46.757450+0000","last_became_peered":"2024-02-12T17:13:46.757450+0000","last_unstale":"2024-02-12T17:16:45.717639+0000","last_undegraded":"2024-02-12T17:16:45.717639+0000","last_fullsized":"2024-02-12T17:16:45.717639+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:56.880282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.880282+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.5","version":"706'15634","reported_seq":15754,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.951011+0000","last_change":"2024-02-12T17:13:12.678687+0000","last_active":"2024-02-12T17:16:48.951011+0000","last_peered":"2024-02-12T17:16:48.951011+0000","last_clean":"2024-02-12T17:16:48.951011+0000","last_became_active":"2024-02-12T17:11:59.009543+0000","last_became_peered":"2024-02-12T17:11:59.009543+0000","last_unstale":"2024-02-12T17:16:48.951011+0000","last_undegraded":"2024-02-12T17:16:48.951011+0000","last_fullsized":"2024-02-12T17:16:48.951011+0000","mapping_epoch":597,"log_start":"666'9829","ondisk_log_start":"666'9829","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5805,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15634,"num_write_kb":500288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":729,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.716116+0000","last_change":"2024-02-12T17:12:08.013330+0000","last_active":"2024-02-12T17:16:45.716116+0000","last_peered":"2024-02-12T17:16:45.716116+0000","last_clean":"2024-02-12T17:16:45.716116+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:16:45.716116+0000","last_undegraded":"2024-02-12T17:16:45.716116+0000","last_fullsized":"2024-02-12T17:16:45.716116+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.013247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.013247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":626,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.725322+0000","last_change":"2024-02-12T17:12:04.711418+0000","last_active":"2024-02-12T17:16:45.725322+0000","last_peered":"2024-02-12T17:16:45.725322+0000","last_clean":"2024-02-12T17:16:45.725322+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:16:45.725322+0000","last_undegraded":"2024-02-12T17:16:45.725322+0000","last_fullsized":"2024-02-12T17:16:45.725322+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:04.711333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.711333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":484,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.713558+0000","last_change":"2024-02-12T17:14:08.740752+0000","last_active":"2024-02-12T17:16:45.713558+0000","last_peered":"2024-02-12T17:16:45.713558+0000","last_clean":"2024-02-12T17:16:45.713558+0000","last_became_active":"2024-02-12T17:14:08.740430+0000","last_became_peered":"2024-02-12T17:14:08.740430+0000","last_unstale":"2024-02-12T17:16:45.713558+0000","last_undegraded":"2024-02-12T17:16:45.713558+0000","last_fullsized":"2024-02-12T17:16:45.713558+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":261,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.718627+0000","last_change":"2024-02-12T17:13:46.559877+0000","last_active":"2024-02-12T17:16:45.718627+0000","last_peered":"2024-02-12T17:16:45.718627+0000","last_clean":"2024-02-12T17:16:45.718627+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:16:45.718627+0000","last_undegraded":"2024-02-12T17:16:45.718627+0000","last_fullsized":"2024-02-12T17:16:45.718627+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'35","last_scrub_stamp":"2024-02-12T17:07:47.261504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:47.261504+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"706'15894","reported_seq":16011,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.943058+0000","last_change":"2024-02-12T17:11:59.007574+0000","last_active":"2024-02-12T17:16:48.943058+0000","last_peered":"2024-02-12T17:16:48.943058+0000","last_clean":"2024-02-12T17:16:48.943058+0000","last_became_active":"2024-02-12T17:11:59.007300+0000","last_became_peered":"2024-02-12T17:11:59.007300+0000","last_unstale":"2024-02-12T17:16:48.943058+0000","last_undegraded":"2024-02-12T17:16:48.943058+0000","last_fullsized":"2024-02-12T17:16:48.943058+0000","mapping_epoch":597,"log_start":"668'10067","ondisk_log_start":"668'10067","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":5827,"ondisk_log_size":5827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15894,"num_write_kb":508608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":64,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.726323+0000","last_change":"2024-02-12T17:14:08.859503+0000","last_active":"2024-02-12T17:16:45.726323+0000","last_peered":"2024-02-12T17:16:45.726323+0000","last_clean":"2024-02-12T17:16:45.726323+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:16:45.726323+0000","last_undegraded":"2024-02-12T17:16:45.726323+0000","last_fullsized":"2024-02-12T17:16:45.726323+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:03.705889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:13:03.705889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":629,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.951018+0000","last_change":"2024-02-12T17:14:08.762783+0000","last_active":"2024-02-12T17:16:50.951018+0000","last_peered":"2024-02-12T17:16:50.951018+0000","last_clean":"2024-02-12T17:16:50.951018+0000","last_became_active":"2024-02-12T17:14:08.760654+0000","last_became_peered":"2024-02-12T17:14:08.760654+0000","last_unstale":"2024-02-12T17:16:50.951018+0000","last_undegraded":"2024-02-12T17:16:50.951018+0000","last_fullsized":"2024-02-12T17:16:50.951018+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:09.044325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:09.044325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":410,"reported_epoch":708,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:16:50.950073+0000","last_change":"2024-02-12T17:16:45.074788+0000","last_active":"2024-02-12T17:16:50.950073+0000","last_peered":"2024-02-12T17:16:50.950073+0000","last_clean":"2024-02-12T17:13:44.508652+0000","last_became_active":"2024-02-12T17:13:46.563455+0000","last_became_peered":"2024-02-12T17:13:46.563455+0000","last_unstale":"2024-02-12T17:16:50.950073+0000","last_undegraded":"2024-02-12T17:16:50.950073+0000","last_fullsized":"2024-02-12T17:16:50.950073+0000","mapping_epoch":630,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":74,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.724948+0000","last_change":"2024-02-12T17:14:10.285684+0000","last_active":"2024-02-12T17:16:45.724948+0000","last_peered":"2024-02-12T17:16:45.724948+0000","last_clean":"2024-02-12T17:16:45.724948+0000","last_became_active":"2024-02-12T17:14:08.871051+0000","last_became_peered":"2024-02-12T17:14:08.871051+0000","last_unstale":"2024-02-12T17:16:45.724948+0000","last_undegraded":"2024-02-12T17:16:45.724948+0000","last_fullsized":"2024-02-12T17:16:45.724948+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:50.764686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.764686+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"706'15628","reported_seq":23023,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:48.478568+0000","last_change":"2024-02-12T17:16:46.780468+0000","last_active":"2024-02-12T17:16:48.478568+0000","last_peered":"2024-02-12T17:16:48.478568+0000","last_clean":"2024-02-12T17:16:48.478568+0000","last_became_active":"2024-02-12T17:16:46.732970+0000","last_became_peered":"2024-02-12T17:16:46.732970+0000","last_unstale":"2024-02-12T17:16:48.478568+0000","last_undegraded":"2024-02-12T17:16:48.478568+0000","last_fullsized":"2024-02-12T17:16:48.478568+0000","mapping_epoch":705,"log_start":"650'9078","ondisk_log_start":"650'9078","created":597,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:11:57.982521+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.982521+0000","log_size":6550,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15628,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3555,"num_bytes_recovered":232914944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":712,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.713057+0000","last_change":"2024-02-12T17:12:11.906816+0000","last_active":"2024-02-12T17:16:45.713057+0000","last_peered":"2024-02-12T17:16:45.713057+0000","last_clean":"2024-02-12T17:16:45.713057+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:16:45.713057+0000","last_undegraded":"2024-02-12T17:16:45.713057+0000","last_fullsized":"2024-02-12T17:16:45.713057+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:11.906737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:11.906737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":393,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-02-12T17:16:50.950273+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:16:50.950273+0000","last_peered":"2024-02-12T17:16:50.950273+0000","last_clean":"2024-02-12T17:16:50.950273+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:16:50.950273+0000","last_undegraded":"2024-02-12T17:16:50.950273+0000","last_fullsized":"2024-02-12T17:16:50.950273+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":830,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-12T17:16:45.713008+0000","last_change":"2024-02-12T17:12:16.820857+0000","last_active":"2024-02-12T17:16:45.713008+0000","last_peered":"2024-02-12T17:16:45.713008+0000","last_clean":"2024-02-12T17:16:45.713008+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:16:45.713008+0000","last_undegraded":"2024-02-12T17:16:45.713008+0000","last_fullsized":"2024-02-12T17:16:45.713008+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:12:16.820782+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.820782+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"632'35","reported_seq":1212,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-12T17:16:46.721690+0000","last_change":"2024-02-12T17:13:46.691596+0000","last_active":"2024-02-12T17:16:46.721690+0000","last_peered":"2024-02-12T17:16:46.721690+0000","last_clean":"2024-02-12T17:16:46.721690+0000","last_became_active":"2024-02-12T17:13:46.691213+0000","last_became_peered":"2024-02-12T17:13:46.691213+0000","last_unstale":"2024-02-12T17:16:46.721690+0000","last_undegraded":"2024-02-12T17:16:46.721690+0000","last_fullsized":"2024-02-12T17:16:46.721690+0000","mapping_epoch":631,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'29","last_scrub_stamp":"2024-02-12T17:12:47.788327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.788327+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4194328,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":251311,"num_write_kb":8043969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23466,"num_bytes_recovered":1537409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":429207552,"data_stored":429195336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89996,"ondisk_log_size":89996,"up":48,"acting":46,"num_store_stats":8},{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":395,"num_read_kb":390,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":16216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1029,"ondisk_log_size":1029,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234313,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1698052,"kb_used_data":209980,"kb_used_omap":0,"kb_used_meta":1488064,"kb_avail":92047100,"statfs":{"total":95995035648,"available":94256230400,"internally_reserved":0,"allocated":215019520,"data_stored":213585884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1523777536},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":2.5249999999999999,"5min":1.9099999999999999,"15min":2.024},"min":{"1min":0.36799999999999999,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":10.417,"5min":15.856999999999999,"15min":15.856999999999999},"last":0.219},{"interface":"front","average":{"1min":1.756,"5min":1.8180000000000001,"15min":1.7290000000000001},"min":{"1min":0.23599999999999999,"5min":0.221,"15min":0.22},"max":{"1min":7.1120000000000001,"5min":12.565,"15min":12.565},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:15:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4569999999999999,"5min":2.9220000000000002,"15min":2.8330000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.632999999999999,"5min":11.632999999999999,"15min":11.632999999999999},"last":2.6269999999999998},{"interface":"front","average":{"1min":3.8100000000000001,"5min":2.7360000000000002,"15min":2.5569999999999999},"min":{"1min":0.29299999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":10.91,"5min":10.91,"15min":10.91},"last":2.617}]},{"osd":2,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":2.8010000000000002,"15min":3.1349999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22800000000000001},"max":{"1min":9.5389999999999997,"5min":42.162999999999997,"15min":218.35300000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.3820000000000001,"5min":3,"15min":3.2480000000000002},"min":{"1min":0.375,"5min":0.25600000000000001,"15min":0.247},"max":{"1min":9.407,"5min":42.109999999999999,"15min":218.42099999999999},"last":1.2569999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":3.754,"5min":2.52,"15min":2.1930000000000001},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.255},"max":{"1min":11.741,"5min":15.811999999999999,"15min":15.811999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":3.9079999999999999,"5min":3.0710000000000002,"15min":2.649},"min":{"1min":0.42299999999999999,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":12.375999999999999,"5min":13.452,"15min":31.152000000000001},"last":3.2450000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.4099999999999999,"15min":1.6759999999999999},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":2.7810000000000001,"5min":20.672999999999998,"15min":20.672999999999998},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.8939999999999999,"15min":1.96},"min":{"1min":0.23300000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.7240000000000002,"5min":38.390000000000001,"15min":38.390000000000001},"last":0.27100000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.8169999999999999,"15min":2.169},"min":{"1min":0.28100000000000003,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":2.7000000000000002,"5min":20.702000000000002,"15min":56.560000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.4330000000000001,"15min":1.915},"min":{"1min":0.25800000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.6930000000000001,"5min":10.669,"15min":53.963999999999999},"last":0.371}]},{"osd":6,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.4570000000000001,"15min":1.532},"min":{"1min":0.30399999999999999,"5min":0.223,"15min":0.223},"max":{"1min":4.0030000000000001,"5min":21.620000000000001,"15min":21.620000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.671,"15min":1.724},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.7360000000000002,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.39700000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528654,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925708,"kb_used_data":37444,"kb_used_omap":0,"kb_used_meta":888256,"kb_avail":92819444,"statfs":{"total":95995035648,"available":95047110656,"internally_reserved":0,"allocated":38342656,"data_stored":36901380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":909574144},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.7410000000000001,"15min":1.806},"min":{"1min":0.22,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.266,"5min":33.576999999999998,"15min":33.576999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.194,"5min":2.0859999999999999,"15min":1.9239999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9740000000000002,"5min":33.569000000000003,"15min":33.569000000000003},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":2.1160000000000001,"15min":2.1000000000000001},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.6260000000000003,"5min":13.805999999999999,"15min":13.805999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.9299999999999999,"5min":2.141,"15min":2.1760000000000002},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.6399999999999997,"5min":13.778,"15min":13.778},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8849999999999998,"5min":2.7290000000000001,"15min":2.6240000000000001},"min":{"1min":0.35399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.5760000000000005,"5min":33.594999999999999,"15min":56.798000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":4.3019999999999996,"5min":2.6680000000000001,"15min":2.3639999999999999},"min":{"1min":0.36099999999999999,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":14.045999999999999,"5min":33.526000000000003,"15min":56.780999999999999},"last":1.7789999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4489999999999998,"5min":3.1779999999999999,"15min":2.3999999999999999},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":12.336,"5min":58.420999999999999,"15min":58.420999999999999},"last":1.8839999999999999},{"interface":"front","average":{"1min":3.7109999999999999,"5min":3.8290000000000002,"15min":2.8530000000000002},"min":{"1min":0.40400000000000003,"5min":0.254,"15min":0.254},"max":{"1min":11.737,"5min":58.709000000000003,"15min":58.709000000000003},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.542,"15min":1.272},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.23499999999999999},"max":{"1min":5.0810000000000004,"5min":33.706000000000003,"15min":33.706000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.8540000000000001,"15min":1.508},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":4.2460000000000004,"5min":34.895000000000003,"15min":34.895000000000003},"last":0.46100000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.6639999999999999,"15min":1.716},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.1970000000000001,"5min":14.56,"15min":34.869},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.319,"15min":1.6799999999999999},"min":{"1min":0.36399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.1390000000000002,"5min":11.1,"15min":33.591000000000001},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:16:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.7090000000000001,"15min":1.7629999999999999},"min":{"1min":0.26500000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":4.4550000000000001,"5min":35.122999999999998,"15min":35.122999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.421,"5min":1.869,"15min":1.6220000000000001},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.26500000000000001},"max":{"1min":5.1890000000000001,"5min":34.045000000000002,"15min":34.045000000000002},"last":0.42199999999999999}]}]},{"osd":1,"up_from":630,"seq":2705829396518,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":337956,"kb_used_data":3420,"kb_used_omap":3,"kb_used_meta":334524,"kb_avail":93407196,"statfs":{"total":95995035648,"available":95648968704,"internally_reserved":0,"allocated":3502080,"data_stored":2036761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":342552637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8580000000000001,"5min":3.359,"15min":3.4430000000000001},"min":{"1min":0.33900000000000002,"5min":0.214,"15min":0.214},"max":{"1min":9.0109999999999992,"5min":14.67,"15min":14.67},"last":0.26100000000000001},{"interface":"front","average":{"1min":4.609,"5min":3.6579999999999999,"15min":3.5},"min":{"1min":0.33800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":16.920999999999999,"5min":16.920999999999999,"15min":16.920999999999999},"last":0.379}]},{"osd":2,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":3.4220000000000002,"15min":3.528},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":7.5250000000000004,"5min":8.766,"15min":8.766},"last":0.29199999999999998},{"interface":"front","average":{"1min":3.6619999999999999,"5min":3.5299999999999998,"15min":3.508},"min":{"1min":0.33100000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.303000000000001,"5min":14.303000000000001,"15min":14.303000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":5.3479999999999999,"5min":3.984,"15min":3.7559999999999998},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":18.577000000000002,"5min":18.577000000000002,"15min":18.577000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":4.7169999999999996,"5min":4.4009999999999998,"15min":4.3479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":17.527000000000001,"5min":17.527000000000001,"15min":17.527000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.4359999999999999,"5min":2.5310000000000001,"15min":2.5470000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.3209999999999997,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":0.371},{"interface":"front","average":{"1min":2.6059999999999999,"5min":2.746,"15min":2.7690000000000001},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":8.8249999999999993,"5min":12.805999999999999,"15min":12.805999999999999},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.6309999999999998,"5min":3.0990000000000002,"15min":3.177},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.9120000000000008,"5min":12.769,"15min":12.769},"last":0.441},{"interface":"front","average":{"1min":3.1459999999999999,"5min":2.319,"15min":2.181},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":11.15,"5min":11.15,"15min":11.15},"last":0.39200000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":3.0449999999999999,"5min":2.8730000000000002,"15min":2.8439999999999999},"min":{"1min":0.41599999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":11.115,"5min":12.720000000000001,"15min":12.720000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.4039999999999999,"5min":2.7829999999999999,"15min":2.8460000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.29,"5min":8.0890000000000004,"15min":8.0890000000000004},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.8460000000000001,"15min":2.9239999999999999},"min":{"1min":0.39000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":7.6669999999999998,"5min":12.829000000000001,"15min":12.829000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":3.4630000000000001,"5min":2.6059999999999999,"15min":2.4630000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":11.179,"5min":11.179,"15min":11.179},"last":0.46000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738731,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910596,"kb_used_data":10300,"kb_used_omap":0,"kb_used_meta":900288,"kb_avail":92834556,"statfs":{"total":95995035648,"available":95062585344,"internally_reserved":0,"allocated":10547200,"data_stored":9113661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":921894912},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:15:49 2024","interfaces":[{"interface":"back","average":{"1min":4.2720000000000002,"5min":3.597,"15min":3.4849999999999999},"min":{"1min":0.25900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":14.048999999999999,"5min":14.048999999999999,"15min":14.048999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":2.3919999999999999,"5min":2.7200000000000002,"15min":2.774},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":8.1120000000000001,"5min":13.787000000000001,"15min":13.787000000000001},"last":3.5750000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:16:36 2024","interfaces":[{"interface":"back","average":{"1min":3.6360000000000001,"5min":2.589,"15min":2.266},"min":{"1min":0.27000000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.509,"5min":19.312000000000001,"15min":19.312000000000001},"last":3.2519999999999998},{"interface":"front","average":{"1min":3.9769999999999999,"5min":2.9609999999999999,"15min":2.8029999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.20799999999999999},"max":{"1min":11.377000000000001,"5min":22.959,"15min":26.039999999999999},"last":0.66500000000000004}]},{"osd":3,"last update":"Mon Feb 12 17:15:45 2024","interfaces":[{"interface":"back","average":{"1min":9.375,"5min":3.629,"15min":3.1179999999999999},"min":{"1min":0.28599999999999998,"5min":0.186,"15min":0.186},"max":{"1min":61.177,"5min":61.177,"15min":87.311000000000007},"last":0.42499999999999999},{"interface":"front","average":{"1min":6.2800000000000002,"5min":3.6920000000000002,"15min":3.1400000000000001},"min":{"1min":0.45800000000000002,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":59.149999999999999,"5min":59.149999999999999,"15min":90.991},"last":15.188000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:16:05 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.5529999999999999,"15min":1.476},"min":{"1min":0.28799999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":7.5830000000000002,"5min":15.141999999999999,"15min":15.141999999999999},"last":3.2160000000000002},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.641,"15min":2.0310000000000001},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":6.7220000000000004,"5min":13.503,"15min":26.082999999999998},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Feb 12 17:16:42 2024","interfaces":[{"interface":"back","average":{"1min":2.8730000000000002,"5min":2.0289999999999999,"15min":2.2829999999999999},"min":{"1min":0.38200000000000001,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":13.981999999999999,"5min":15.090999999999999,"15min":50.404000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":2.4129999999999998,"5min":1.819,"15min":2.0750000000000002},"min":{"1min":0.31900000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.6070000000000002,"5min":15.164,"15min":49.979999999999997},"last":3.2759999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:16:05 2024","interfaces":[{"interface":"back","average":{"1min":3.0169999999999999,"5min":2.5600000000000001,"15min":2.379},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.8369999999999997,"5min":48.780999999999999,"15min":48.780999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":3.3239999999999998,"5min":2.827,"15min":2.3370000000000002},"min":{"1min":0.38600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":10.390000000000001,"5min":49.067999999999998,"15min":49.067999999999998},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:16:36 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.0289999999999999,"15min":2.056},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.6299999999999999,"5min":15.183999999999999,"15min":19.373000000000001},"last":3.3050000000000002},{"interface":"front","average":{"1min":2.9279999999999999,"5min":2.25,"15min":2.4159999999999999},"min":{"1min":0.36799999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":11.4,"5min":13.542,"15min":26.135000000000002},"last":0.67900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411680,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1512764,"kb_used_data":14516,"kb_used_omap":0,"kb_used_meta":1498176,"kb_avail":92232388,"statfs":{"total":95995035648,"available":94445965312,"internally_reserved":0,"allocated":14864384,"data_stored":13439430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1534132224},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":2.327,"5min":1.929,"15min":1.966},"min":{"1min":0.32500000000000001,"5min":0.17899999999999999,"15min":0.17000000000000001},"max":{"1min":7.1749999999999998,"5min":11.663,"15min":36.100999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":3.9209999999999998,"5min":2.5609999999999999,"15min":2.3860000000000001},"min":{"1min":0.307,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.271000000000001,"5min":16.350000000000001,"15min":36.299999999999997},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:15:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5489999999999999,"5min":3.722,"15min":3.9180000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":9.3239999999999998,"5min":11.612,"15min":11.612},"last":0.36899999999999999},{"interface":"front","average":{"1min":4.5579999999999998,"5min":5.6260000000000003,"15min":5.8040000000000003},"min":{"1min":0.42599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":11.141999999999999,"5min":22.52,"15min":22.52},"last":0.29999999999999999}]},{"osd":3,"last update":"Mon Feb 12 17:16:00 2024","interfaces":[{"interface":"back","average":{"1min":5.5259999999999998,"5min":3.6930000000000001,"15min":2.9220000000000002},"min":{"1min":0.29099999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":30.303000000000001,"5min":30.303000000000001,"15min":40.759999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":5.9139999999999997,"5min":3.262,"15min":2.5070000000000001},"min":{"1min":0.221,"5min":0.22,"15min":0.22},"max":{"1min":30.120999999999999,"5min":30.120999999999999,"15min":30.120999999999999},"last":0.38600000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:16:24 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.754,"15min":1.8759999999999999},"min":{"1min":0.42599999999999999,"5min":0.247,"15min":0.245},"max":{"1min":5.851,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.77,"15min":1.9370000000000001},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.0590000000000002,"5min":11.744,"15min":20.149000000000001},"last":0.44}]},{"osd":5,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.748,"15min":2.0030000000000001},"min":{"1min":0.46000000000000002,"5min":0.26000000000000001,"15min":0.217},"max":{"1min":7.1310000000000002,"5min":11.581,"15min":32.57},"last":0.46100000000000002},{"interface":"front","average":{"1min":2.2309999999999999,"5min":1.871,"15min":2.1070000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.255},"max":{"1min":6.2030000000000003,"5min":10.787000000000001,"15min":31.905999999999999},"last":0.40000000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:16:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.7829999999999999,"15min":1.8300000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.0919999999999996,"5min":10.714,"15min":20.966999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":2.5880000000000001,"5min":2.1949999999999998,"15min":1.8580000000000001},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.6130000000000004,"5min":11.834,"15min":20.193000000000001},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.871,"15min":2.4609999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.7119999999999997,"5min":11.789,"15min":45.468000000000004},"last":0.501},{"interface":"front","average":{"1min":2.3210000000000002,"5min":2.0169999999999999,"15min":2.4569999999999999},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":6.1150000000000002,"5min":10.762,"15min":46.656999999999996},"last":0.48899999999999999}]}]},{"osd":3,"up_from":26,"seq":111669150043,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899512,"kb_used_data":6256,"kb_used_omap":0,"kb_used_meta":893248,"kb_avail":92845640,"statfs":{"total":95995035648,"available":95073935360,"internally_reserved":0,"allocated":6406144,"data_stored":4934552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914685952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":6.7190000000000003,"5min":3.738,"15min":2.8420000000000001},"min":{"1min":0.45500000000000002,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":58.451999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":4.351,"5min":3.7240000000000002,"15min":3.0409999999999999},"min":{"1min":0.40400000000000003,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":11.720000000000001,"5min":72.688000000000002,"15min":72.688000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:16:51 2024","interfaces":[{"interface":"back","average":{"1min":3.8279999999999998,"5min":5.0460000000000003,"15min":4.6689999999999996},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":16.010000000000002,"5min":22.524000000000001,"15min":22.524000000000001},"last":0.253},{"interface":"front","average":{"1min":5.4749999999999996,"5min":5.4930000000000003,"15min":5.4669999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.710000000000001,"5min":20.710000000000001,"15min":20.710000000000001},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":7.9420000000000002,"5min":3.5049999999999999,"15min":2.5659999999999998},"min":{"1min":0.245,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":38.034999999999997,"5min":40.954999999999998,"15min":40.954999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":6.7450000000000001,"5min":4.1289999999999996,"15min":2.9710000000000001},"min":{"1min":0.41399999999999998,"5min":0.22,"15min":0.22},"max":{"1min":22.138999999999999,"5min":46.408000000000001,"15min":46.408000000000001},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Feb 12 17:16:15 2024","interfaces":[{"interface":"back","average":{"1min":5.2839999999999998,"5min":3.746,"15min":2.6720000000000002},"min":{"1min":0.54600000000000004,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":20.478000000000002,"5min":72.659000000000006,"15min":72.659000000000006},"last":0.56599999999999995},{"interface":"front","average":{"1min":4.5279999999999996,"5min":3.0979999999999999,"15min":2.1099999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23999999999999999},"max":{"1min":22.440999999999999,"5min":40.930999999999997,"15min":40.930999999999997},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:16:34 2024","interfaces":[{"interface":"back","average":{"1min":4.218,"5min":3.1120000000000001,"15min":2.7080000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.253},"max":{"1min":12.329000000000001,"5min":17.306999999999999,"15min":17.940999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":4.3310000000000004,"5min":3.665,"15min":2.9279999999999999},"min":{"1min":0.52000000000000002,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":15.522,"5min":72.620000000000005,"15min":72.620000000000005},"last":0.59499999999999997}]},{"osd":6,"last update":"Mon Feb 12 17:15:56 2024","interfaces":[{"interface":"back","average":{"1min":5.2039999999999997,"5min":2.9510000000000001,"15min":2.153},"min":{"1min":0.31900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":22.489999999999998,"5min":40.889000000000003,"15min":40.889000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":5.1559999999999997,"5min":3.27,"15min":2.4649999999999999},"min":{"1min":0.39800000000000002,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":15.113,"5min":16.989999999999998,"15min":18.265000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Feb 12 17:16:15 2024","interfaces":[{"interface":"back","average":{"1min":5.0890000000000004,"5min":3.5609999999999999,"15min":3.0880000000000001},"min":{"1min":0.53500000000000003,"5min":0.375,"15min":0.315},"max":{"1min":20.515999999999998,"5min":72.712999999999994,"15min":72.712999999999994},"last":0.51300000000000001},{"interface":"front","average":{"1min":5.6120000000000001,"5min":3.0259999999999998,"15min":2.9089999999999998},"min":{"1min":0.44900000000000001,"5min":0.38600000000000001,"15min":0.28299999999999997},"max":{"1min":15.169,"5min":15.169,"15min":38.948},"last":0.68300000000000005}]}]},{"osd":4,"up_from":0,"seq":137438953816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757588,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707128,"kb_used_data":221808,"kb_used_omap":0,"kb_used_meta":1485312,"kb_avail":92038024,"statfs":{"total":95995035648,"available":94246936576,"internally_reserved":0,"allocated":227131392,"data_stored":225709778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1520959488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":3.0950000000000002,"5min":1.9830000000000001,"15min":1.623},"min":{"1min":0.46500000000000002,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":6.8399999999999999,"5min":8.0350000000000001,"15min":10.971},"last":2.1469999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.915,"15min":1.8029999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20100000000000001},"max":{"1min":12.061,"5min":12.061,"15min":15.368},"last":1.972}]},{"osd":1,"last update":"Mon Feb 12 17:15:52 2024","interfaces":[{"interface":"back","average":{"1min":2.512,"5min":1.9399999999999999,"15min":1.845},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.8710000000000004,"5min":9.484,"15min":9.484},"last":8.0540000000000003},{"interface":"front","average":{"1min":1.968,"5min":2.8450000000000002,"15min":2.9910000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":6.7210000000000001,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":4.0110000000000001,"5min":2.871,"15min":2.0800000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23200000000000001},"max":{"1min":13.978,"5min":57.390999999999998,"15min":57.390999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":2.2559999999999998,"5min":2.7229999999999999,"15min":1.9870000000000001},"min":{"1min":0.53300000000000003,"5min":0.214,"15min":0.214},"max":{"1min":6.0259999999999998,"5min":57.134999999999998,"15min":57.134999999999998},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":3.9390000000000001,"5min":2.964,"15min":2.7389999999999999},"min":{"1min":0.36499999999999999,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":11.871,"5min":57.170999999999999,"15min":57.170999999999999},"last":0.88},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.984,"15min":2.6259999999999999},"min":{"1min":0.33600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":17.760999999999999,"5min":57.484999999999999,"15min":57.484999999999999},"last":13.166}]},{"osd":4,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.385},"min":{"1min":0.26000000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.3660000000000001,"5min":9.0429999999999993,"15min":12.656000000000001},"last":0.252},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6459999999999999,"15min":1.375},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.254},"max":{"1min":5.1390000000000002,"5min":10.619999999999999,"15min":11.068},"last":0.35499999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:16:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.5030000000000001,"5min":14.675000000000001,"15min":14.675000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.651,"5min":1.802,"15min":1.5629999999999999},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":5.2720000000000002,"5min":12.864000000000001,"15min":13.916},"last":1.3200000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.6499999999999999,"15min":1.599},"min":{"1min":0.31900000000000001,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":6.0170000000000003,"5min":14.337999999999999,"15min":14.337999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.6990000000000001,"15min":1.7629999999999999},"min":{"1min":0.316,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":3.9369999999999998,"5min":14.637,"15min":20.274000000000001},"last":0.30199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6950912,"data_stored":6946840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":917504,"data_stored":917504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12714008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":210894848,"data_stored":210894848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4657152,"data_stored":4653080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":193069056,"data_stored":193069056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:16:55.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:55.203 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:55.224 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:55.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:55.328 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:55.358 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:55.379 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:55.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:55.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:55.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:16:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:16:55.513 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:55.532 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:55.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:55.637 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:55.668 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:55.768 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:55.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:55.872 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:55.902 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:55.988 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:54.756220+0000 mon.a (mon.0) 2270 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:55.988 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:54.761539+0000 mon.a (mon.0) 2271 : audit [INF] from='client.36254 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:16:55.988 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:54.761580+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e711: 8 total, 8 up, 7 in 2024-02-12T17:16:55.988 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:54.756220+0000 mon.a (mon.0) 2270 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:55.988 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:54.761539+0000 mon.a (mon.0) 2271 : audit [INF] from='client.36254 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:16:55.988 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:54.761580+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e711: 8 total, 8 up, 7 in 2024-02-12T17:16:55.989 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:16:56.003 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:56.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:56.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:56.138 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:54.756220+0000 mon.a (mon.0) 2270 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T17:16:56.138 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:54.761539+0000 mon.a (mon.0) 2271 : audit [INF] from='client.36254 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:16:56.139 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:54.761580+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e711: 8 total, 8 up, 7 in 2024-02-12T17:16:56.141 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:56.163 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:56.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:56.267 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:56.299 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:56.318 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:56.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:56.422 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:56.449 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:56.468 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:56.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:56.572 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:56.598 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:56.617 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:56.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:56.721 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4 2024-02-12T17:16:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:56.767 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:16:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:56.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4' 2024-02-12T17:16:56.871 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:16:56.899 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:55.154803+0000 mgr.y (mgr.24481) 1026 : cluster [DBG] pgmap v1451: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 146 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:16:56.899 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:55.157338+0000 mgr.y (mgr.24481) 1027 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:56.899 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:55.692426+0000 osd.1 (osd.1) 8 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:16:56.899 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:55.695880+0000 osd.1 (osd.1) 9 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:16:56.899 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:56.253181+0000 osd.2 (osd.2) 235 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:56.899 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:56.253240+0000 osd.2 (osd.2) 236 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:56.899 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:55.154803+0000 mgr.y (mgr.24481) 1026 : cluster [DBG] pgmap v1451: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 146 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:16:56.900 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:55.157338+0000 mgr.y (mgr.24481) 1027 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:56.900 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:55.692426+0000 osd.1 (osd.1) 8 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:16:56.900 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:55.695880+0000 osd.1 (osd.1) 9 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:16:56.900 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:56.253181+0000 osd.2 (osd.2) 235 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:56.900 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:56.253240+0000 osd.2 (osd.2) 236 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:56.902 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:55.154803+0000 mgr.y (mgr.24481) 1026 : cluster [DBG] pgmap v1451: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 146 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:16:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:55.157338+0000 mgr.y (mgr.24481) 1027 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:16:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:55.692426+0000 osd.1 (osd.1) 8 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:16:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:55.695880+0000 osd.1 (osd.1) 9 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:16:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:56.253181+0000 osd.2 (osd.2) 235 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:16:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:56.253240+0000 osd.2 (osd.2) 236 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:16:57.604 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-02-12T17:16:57.621 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-02-12T17:16:57.638 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-02-12T17:16:57.656 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-02-12T17:16:57.674 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-02-12T17:16:57.692 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-02-12T17:16:57.711 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-02-12T17:16:57.728 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-02-12T17:16:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T17:16:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T17:16:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T17:16:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T17:16:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T17:16:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T17:16:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T17:16:57.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T17:16:57.778 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:16:58.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:56.740621+0000 osd.6 (osd.6) 182 : cluster [DBG] 4.a scrub starts 2024-02-12T17:16:58.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:56.742668+0000 osd.6 (osd.6) 183 : cluster [DBG] 4.a scrub ok 2024-02-12T17:16:58.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:56.781808+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e712: 8 total, 8 up, 7 in 2024-02-12T17:16:58.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:56.803163+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-02-12T17:16:58.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:57.155342+0000 mgr.y (mgr.24481) 1028 : cluster [DBG] pgmap v1453: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 119 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:16:58.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:57.585275+0000 mon.a (mon.0) 2275 : audit [DBG] from='client.? v1:172.21.15.33:0/2446028025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:56.740621+0000 osd.6 (osd.6) 182 : cluster [DBG] 4.a scrub starts 2024-02-12T17:16:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:56.742668+0000 osd.6 (osd.6) 183 : cluster [DBG] 4.a scrub ok 2024-02-12T17:16:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:56.781808+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e712: 8 total, 8 up, 7 in 2024-02-12T17:16:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:56.803163+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-02-12T17:16:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:57.155342+0000 mgr.y (mgr.24481) 1028 : cluster [DBG] pgmap v1453: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 119 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:16:58.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:57.585275+0000 mon.a (mon.0) 2275 : audit [DBG] from='client.? v1:172.21.15.33:0/2446028025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:58.078 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:16:58.161 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:16:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:58.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:16:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:56.740621+0000 osd.6 (osd.6) 182 : cluster [DBG] 4.a scrub starts 2024-02-12T17:16:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:56.742668+0000 osd.6 (osd.6) 183 : cluster [DBG] 4.a scrub ok 2024-02-12T17:16:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:56.781808+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e712: 8 total, 8 up, 7 in 2024-02-12T17:16:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:56.803163+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-02-12T17:16:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:57.155342+0000 mgr.y (mgr.24481) 1028 : cluster [DBG] pgmap v1453: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 119 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:16:58.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:57.585275+0000 mon.a (mon.0) 2275 : audit [DBG] from='client.? v1:172.21.15.33:0/2446028025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T17:16:58.269 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:58.301 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:58.323 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:58.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:58.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:58.459 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:58.480 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:58.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:58.584 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5 2024-02-12T17:16:58.614 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:58.634 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:16:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:58.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5' 2024-02-12T17:16:58.738 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3 2024-02-12T17:16:58.767 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:58.984 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:57.565134+0000 osd.0 (osd.0) 238 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:16:58.984 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:57.566899+0000 osd.0 (osd.0) 239 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:16:58.985 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:57.565134+0000 osd.0 (osd.0) 238 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:16:58.985 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:57.566899+0000 osd.0 (osd.0) 239 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:16:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:57.565134+0000 osd.0 (osd.0) 238 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:16:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:57.566899+0000 osd.0 (osd.0) 239 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:16:59.016 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:16:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:59.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3' 2024-02-12T17:16:59.120 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1 2024-02-12T17:16:59.148 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:59.297 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:16:59.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:59.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1' 2024-02-12T17:16:59.404 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:59.430 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:59.449 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:59.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:59.553 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:16:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:16:59.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:59.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:16:59.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:59.730 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:59.748 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:59.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:59.852 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6 2024-02-12T17:16:59.879 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:16:59.898 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:16:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:16:59.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6' 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:58.515943+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:58.558959+0000 osd.0 (osd.0) 241 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:58.847946+0000 mon.c (mon.2) 316 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:59.099621+0000 mon.a (mon.0) 2276 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:59.156021+0000 mgr.y (mgr.24481) 1029 : cluster [DBG] pgmap v1454: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 88 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:59.389227+0000 mon.c (mon.2) 317 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:59.389848+0000 mon.c (mon.2) 318 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:59.397623+0000 mon.a (mon.0) 2277 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:59.805612+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.33:0/2853113546' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:16:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:16:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:16:59.805890+0000 mon.a (mon.0) 2278 : audit [INF] from='client.36266 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:17:00.003 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:58.515943+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:58.558959+0000 osd.0 (osd.0) 241 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:58.847946+0000 mon.c (mon.2) 316 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:59.099621+0000 mon.a (mon.0) 2276 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:59.156021+0000 mgr.y (mgr.24481) 1029 : cluster [DBG] pgmap v1454: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 88 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:59.389227+0000 mon.c (mon.2) 317 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:59.389848+0000 mon.c (mon.2) 318 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:59.397623+0000 mon.a (mon.0) 2277 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:00.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:59.805612+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.33:0/2853113546' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:16:59.805890+0000 mon.a (mon.0) 2278 : audit [INF] from='client.36266 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:58.515943+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:58.558959+0000 osd.0 (osd.0) 241 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:58.847946+0000 mon.c (mon.2) 316 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:59.099621+0000 mon.a (mon.0) 2276 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:59.156021+0000 mgr.y (mgr.24481) 1029 : cluster [DBG] pgmap v1454: 118 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+remapped+backfill_toofull, 114 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 696 B/s rd, 0 B/s wr, 88 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:59.389227+0000 mon.c (mon.2) 317 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:59.389848+0000 mon.c (mon.2) 318 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:59.397623+0000 mon.a (mon.0) 2277 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:59.805612+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.33:0/2853113546' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:17:00.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:16:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:16:59.805890+0000 mon.a (mon.0) 2278 : audit [INF] from='client.36266 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T17:17:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:17:00.418 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_4' removed 2024-02-12T17:17:00.540 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:00.540 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:00.683 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:17:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:17:00.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0' 2024-02-12T17:17:00.787 DEBUG:teuthology.orchestra.run.smithi094:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7 2024-02-12T17:17:00.871 INFO:teuthology.orchestra.run.smithi094.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:17:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:17:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:17:00.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7' 2024-02-12T17:17:00.999 DEBUG:teuthology.orchestra.run.smithi033:> sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2 2024-02-12T17:17:01.025 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:00] "GET /metrics HTTP/1.1" 200 176840 "" "Prometheus/2.33.4" 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:59.492655+0000 osd.0 (osd.0) 242 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:59.494576+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:59.710232+0000 osd.1 (osd.1) 10 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:59.712503+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:59.772989+0000 osd.6 (osd.6) 184 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:16:59.774655+0000 osd.6 (osd.6) 185 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:00.284272+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub starts 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:00.287736+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.0 scrub ok 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:00.316526+0000 osd.3 (osd.3) 242 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:00.320120+0000 osd.3 (osd.3) 243 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:17:01.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:00.399933+0000 mon.a (mon.0) 2279 : audit [INF] from='client.36266 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:00.400021+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e713: 8 total, 8 up, 7 in 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:59.492655+0000 osd.0 (osd.0) 242 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:59.494576+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:59.710232+0000 osd.1 (osd.1) 10 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:59.712503+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:59.772989+0000 osd.6 (osd.6) 184 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:16:59.774655+0000 osd.6 (osd.6) 185 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:00.284272+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub starts 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:00.287736+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.0 scrub ok 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:00.316526+0000 osd.3 (osd.3) 242 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:00.320120+0000 osd.3 (osd.3) 243 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:00.399933+0000 mon.a (mon.0) 2279 : audit [INF] from='client.36266 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:17:01.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:00.400021+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e713: 8 total, 8 up, 7 in 2024-02-12T17:17:01.029 INFO:teuthology.orchestra.run.smithi033.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-12T17:17:01.179 INFO:teuthology.orchestra.run.smithi033.stderr:Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:17:01.181 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-12T17:17:01.196 INFO:tasks.thrashosds:joining thrashosds 2024-02-12T17:17:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-12T17:17:01.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi033 with status 125: 'sudo docker kill -s 1 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2' 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:59.492655+0000 osd.0 (osd.0) 242 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:59.494576+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:59.710232+0000 osd.1 (osd.1) 10 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:59.712503+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:59.772989+0000 osd.6 (osd.6) 184 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:16:59.774655+0000 osd.6 (osd.6) 185 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:00.284272+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub starts 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:00.287736+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.0 scrub ok 2024-02-12T17:17:01.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:00.316526+0000 osd.3 (osd.3) 242 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:17:01.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:00.320120+0000 osd.3 (osd.3) 243 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:17:01.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:00.399933+0000 mon.a (mon.0) 2279 : audit [INF] from='client.36266 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T17:17:01.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:00.400021+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e713: 8 total, 8 up, 7 in 2024-02-12T17:17:01.342 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:01.342 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:02.020 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:00.747318+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:17:02.020 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:00.753374+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:17:02.020 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:00.995075+0000 osd.5 (osd.5) 324 : cluster [DBG] 6.d scrub starts 2024-02-12T17:17:02.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.153722+0000 osd.5 (osd.5) 325 : cluster [DBG] 6.d scrub ok 2024-02-12T17:17:02.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.156373+0000 mgr.y (mgr.24481) 1030 : cluster [DBG] pgmap v1456: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 640 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:02.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.315219+0000 osd.3 (osd.3) 244 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:17:02.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.422042+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:17:02.021 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.807658+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-02-12T17:17:02.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:00.747318+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:17:02.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:00.753374+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:17:02.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:00.995075+0000 osd.5 (osd.5) 324 : cluster [DBG] 6.d scrub starts 2024-02-12T17:17:02.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.153722+0000 osd.5 (osd.5) 325 : cluster [DBG] 6.d scrub ok 2024-02-12T17:17:02.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.156373+0000 mgr.y (mgr.24481) 1030 : cluster [DBG] pgmap v1456: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 640 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:02.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.315219+0000 osd.3 (osd.3) 244 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:17:02.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.422042+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:17:02.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.807658+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-02-12T17:17:02.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T17:17:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:00.747318+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:17:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:00.753374+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:17:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:00.995075+0000 osd.5 (osd.5) 324 : cluster [DBG] 6.d scrub starts 2024-02-12T17:17:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.153722+0000 osd.5 (osd.5) 325 : cluster [DBG] 6.d scrub ok 2024-02-12T17:17:02.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.156373+0000 mgr.y (mgr.24481) 1030 : cluster [DBG] pgmap v1456: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 640 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:02.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.315219+0000 osd.3 (osd.3) 244 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:17:02.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.422042+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:17:02.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.807658+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-02-12T17:17:02.847 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:17:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:17:03.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.777809+0000 osd.6 (osd.6) 186 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:17:03.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.779285+0000 osd.6 (osd.6) 187 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:17:03.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.835252+0000 mon.a (mon.0) 2282 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:03.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:01.835298+0000 mon.a (mon.0) 2283 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-12T17:17:03.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:02.298609+0000 osd.7 (osd.7) 321 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:17:03.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:02.302380+0000 osd.7 (osd.7) 322 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:17:03.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:02.538133+0000 mon.a (mon.0) 2284 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:03.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.777809+0000 osd.6 (osd.6) 186 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:17:03.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.779285+0000 osd.6 (osd.6) 187 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:17:03.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.835252+0000 mon.a (mon.0) 2282 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:03.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:01.835298+0000 mon.a (mon.0) 2283 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-12T17:17:03.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:02.298609+0000 osd.7 (osd.7) 321 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:17:03.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:02.302380+0000 osd.7 (osd.7) 322 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:17:03.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:02.538133+0000 mon.a (mon.0) 2284 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.777809+0000 osd.6 (osd.6) 186 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:17:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.779285+0000 osd.6 (osd.6) 187 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:17:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.835252+0000 mon.a (mon.0) 2282 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:03.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:01.835298+0000 mon.a (mon.0) 2283 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-12T17:17:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:02.298609+0000 osd.7 (osd.7) 321 : cluster [DBG] 1.0 scrub starts 2024-02-12T17:17:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:02.302380+0000 osd.7 (osd.7) 322 : cluster [DBG] 1.0 scrub ok 2024-02-12T17:17:03.242 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:02.538133+0000 mon.a (mon.0) 2284 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:04.107 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:02.418852+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:17:04.108 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:02.421051+0000 osd.0 (osd.0) 245 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:17:04.108 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:02.762858+0000 osd.1 (osd.1) 14 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:17:04.108 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:02.766129+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:17:04.108 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:02.797384+0000 osd.6 (osd.6) 188 : cluster [DBG] 4.e scrub starts 2024-02-12T17:17:04.108 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:02.799495+0000 osd.6 (osd.6) 189 : cluster [DBG] 4.e scrub ok 2024-02-12T17:17:04.108 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:03.156791+0000 mgr.y (mgr.24481) 1031 : cluster [DBG] pgmap v1458: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 642 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:04.108 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:03.329775+0000 osd.7 (osd.7) 323 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:17:04.109 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:03.332763+0000 osd.7 (osd.7) 324 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:17:04.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:02.418852+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:17:04.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:02.421051+0000 osd.0 (osd.0) 245 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:17:04.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:02.762858+0000 osd.1 (osd.1) 14 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:17:04.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:02.766129+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:17:04.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:02.797384+0000 osd.6 (osd.6) 188 : cluster [DBG] 4.e scrub starts 2024-02-12T17:17:04.109 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:02.799495+0000 osd.6 (osd.6) 189 : cluster [DBG] 4.e scrub ok 2024-02-12T17:17:04.110 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:03.156791+0000 mgr.y (mgr.24481) 1031 : cluster [DBG] pgmap v1458: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 642 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:04.110 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:03.329775+0000 osd.7 (osd.7) 323 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:17:04.110 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:03.332763+0000 osd.7 (osd.7) 324 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:17:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:02.418852+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:17:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:02.421051+0000 osd.0 (osd.0) 245 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:17:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:02.762858+0000 osd.1 (osd.1) 14 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:17:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:02.766129+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:17:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:02.797384+0000 osd.6 (osd.6) 188 : cluster [DBG] 4.e scrub starts 2024-02-12T17:17:04.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:02.799495+0000 osd.6 (osd.6) 189 : cluster [DBG] 4.e scrub ok 2024-02-12T17:17:04.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:03.156791+0000 mgr.y (mgr.24481) 1031 : cluster [DBG] pgmap v1458: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 642 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:04.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:03.329775+0000 osd.7 (osd.7) 323 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:17:04.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:03.332763+0000 osd.7 (osd.7) 324 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:17:04.863 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:17:04.871 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:17:04.880 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:05.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.880899+0000 mon.c (mon.2) 320 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.881279+0000 mon.c (mon.2) 321 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.881398+0000 mon.a (mon.0) 2285 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.881506+0000 mon.c (mon.2) 322 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.881753+0000 mon.c (mon.2) 323 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.881977+0000 mon.a (mon.0) 2286 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.881979+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.882202+0000 mon.c (mon.2) 325 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.882465+0000 mon.a (mon.0) 2287 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.882702+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]: dispatch 2024-02-12T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.882884+0000 mon.a (mon.0) 2289 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-12T17:17:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:03.883067+0000 mon.a (mon.0) 2290 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-12T17:17:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:04.358626+0000 osd.3 (osd.3) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:04.358678+0000 osd.3 (osd.3) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:05.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:04.616925+0000 mgr.y (mgr.24481) 1032 : audit [DBG] from='client.16362 v1:172.21.15.33:0/3681630814' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:17:05.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.880899+0000 mon.c (mon.2) 320 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.881279+0000 mon.c (mon.2) 321 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.881398+0000 mon.a (mon.0) 2285 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.881506+0000 mon.c (mon.2) 322 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.881753+0000 mon.c (mon.2) 323 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.881977+0000 mon.a (mon.0) 2286 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.881979+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.882202+0000 mon.c (mon.2) 325 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.882465+0000 mon.a (mon.0) 2287 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.882702+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.882884+0000 mon.a (mon.0) 2289 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:03.883067+0000 mon.a (mon.0) 2290 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-12T17:17:05.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:04.358626+0000 osd.3 (osd.3) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:05.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:04.358678+0000 osd.3 (osd.3) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:05.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:04.616925+0000 mgr.y (mgr.24481) 1032 : audit [DBG] from='client.16362 v1:172.21.15.33:0/3681630814' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:17:05.169 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.880899+0000 mon.c (mon.2) 320 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.169 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.881279+0000 mon.c (mon.2) 321 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.881398+0000 mon.a (mon.0) 2285 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.881506+0000 mon.c (mon.2) 322 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.881753+0000 mon.c (mon.2) 323 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.881977+0000 mon.a (mon.0) 2286 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.881979+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.882202+0000 mon.c (mon.2) 325 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.882465+0000 mon.a (mon.0) 2287 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.882702+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]: dispatch 2024-02-12T17:17:05.170 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.882884+0000 mon.a (mon.0) 2289 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-12T17:17:05.171 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:03.883067+0000 mon.a (mon.0) 2290 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-12T17:17:05.171 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:04.358626+0000 osd.3 (osd.3) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:05.171 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:04.358678+0000 osd.3 (osd.3) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:05.171 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:04.616925+0000 mgr.y (mgr.24481) 1032 : audit [DBG] from='client.16362 v1:172.21.15.33:0/3681630814' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:17:05.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:04.763215+0000 osd.6 (osd.6) 190 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:04.765502+0000 osd.6 (osd.6) 191 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:04.800211+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:04.803679+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:04.859290+0000 mon.a (mon.0) 2291 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:04.859385+0000 mon.a (mon.0) 2292 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:04.859452+0000 mon.a (mon.0) 2293 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]': finished 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:04.859514+0000 mon.a (mon.0) 2294 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]': finished 2024-02-12T17:17:06.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:04.859584+0000 mon.a (mon.0) 2295 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-02-12T17:17:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:04.859652+0000 mon.a (mon.0) 2296 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]': finished 2024-02-12T17:17:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:04.859722+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-02-12T17:17:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:05.157377+0000 mgr.y (mgr.24481) 1033 : cluster [DBG] pgmap v1460: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:05.168401+0000 mgr.y (mgr.24481) 1034 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:05.348727+0000 osd.3 (osd.3) 248 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:17:06.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:05.352389+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:04.763215+0000 osd.6 (osd.6) 190 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:04.765502+0000 osd.6 (osd.6) 191 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:04.800211+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:04.803679+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:04.859290+0000 mon.a (mon.0) 2291 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:04.859385+0000 mon.a (mon.0) 2292 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:04.859452+0000 mon.a (mon.0) 2293 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]': finished 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:04.859514+0000 mon.a (mon.0) 2294 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]': finished 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:04.859584+0000 mon.a (mon.0) 2295 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-02-12T17:17:06.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:04.859652+0000 mon.a (mon.0) 2296 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]': finished 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:04.859722+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:05.157377+0000 mgr.y (mgr.24481) 1033 : cluster [DBG] pgmap v1460: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:05.168401+0000 mgr.y (mgr.24481) 1034 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:05.348727+0000 osd.3 (osd.3) 248 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:05.352389+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:04.763215+0000 osd.6 (osd.6) 190 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:04.765502+0000 osd.6 (osd.6) 191 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:04.800211+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:04.803679+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:04.859290+0000 mon.a (mon.0) 2291 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:04.859385+0000 mon.a (mon.0) 2292 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-12T17:17:06.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:04.859452+0000 mon.a (mon.0) 2293 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]': finished 2024-02-12T17:17:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:04.859514+0000 mon.a (mon.0) 2294 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]': finished 2024-02-12T17:17:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:04.859584+0000 mon.a (mon.0) 2295 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-02-12T17:17:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:04.859652+0000 mon.a (mon.0) 2296 : audit [INF] from='mgr.24481 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]': finished 2024-02-12T17:17:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:04.859722+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-02-12T17:17:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:05.157377+0000 mgr.y (mgr.24481) 1033 : cluster [DBG] pgmap v1460: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 2/576 objects misplaced (0.347%) 2024-02-12T17:17:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:05.168401+0000 mgr.y (mgr.24481) 1034 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:05.348727+0000 osd.3 (osd.3) 248 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:17:06.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:05.352389+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:17:06.517 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1458,"stamp":"2024-02-12T17:17:03.156518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":1938,"num_read_kb":1690,"num_write":1000,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":23420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1163,"ondisk_log_size":1163,"up":306,"acting":306,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":351,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7877324,"kb_used_data":385300,"kb_used_omap":3,"kb_used_meta":7491964,"kb_avail":648338740,"statfs":{"total":671965249536,"available":663898869760,"internally_reserved":0,"allocated":394547200,"data_stored":384235454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":7671771197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-251376,"num_write_kb":-8043969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-23466,"num_bytes_recovered":-1537409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.374307"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":438,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187131+0000","last_change":"2024-02-12T17:14:10.348720+0000","last_active":"2024-02-12T17:16:57.187131+0000","last_peered":"2024-02-12T17:16:57.187131+0000","last_clean":"2024-02-12T17:16:57.187131+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:16:57.187131+0000","last_undegraded":"2024-02-12T17:16:57.187131+0000","last_fullsized":"2024-02-12T17:16:57.187131+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":220,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.761427+0000","last_change":"2024-02-12T17:13:46.694822+0000","last_active":"2024-02-12T17:16:54.761427+0000","last_peered":"2024-02-12T17:16:54.761427+0000","last_clean":"2024-02-12T17:16:54.761427+0000","last_became_active":"2024-02-12T17:13:46.694448+0000","last_became_peered":"2024-02-12T17:13:46.694448+0000","last_unstale":"2024-02-12T17:16:54.761427+0000","last_undegraded":"2024-02-12T17:16:54.761427+0000","last_fullsized":"2024-02-12T17:16:54.761427+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:12:55.862345+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:12:55.862345+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":465,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189946+0000","last_change":"2024-02-12T17:13:46.787428+0000","last_active":"2024-02-12T17:16:57.189946+0000","last_peered":"2024-02-12T17:16:57.189946+0000","last_clean":"2024-02-12T17:16:57.189946+0000","last_became_active":"2024-02-12T17:13:46.786996+0000","last_became_peered":"2024-02-12T17:13:46.786996+0000","last_unstale":"2024-02-12T17:16:57.189946+0000","last_undegraded":"2024-02-12T17:16:57.189946+0000","last_fullsized":"2024-02-12T17:16:57.189946+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:13:12.016227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:12.016227+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":493,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187330+0000","last_change":"2024-02-12T17:14:09.685503+0000","last_active":"2024-02-12T17:16:57.187330+0000","last_peered":"2024-02-12T17:16:57.187330+0000","last_clean":"2024-02-12T17:16:57.187330+0000","last_became_active":"2024-02-12T17:14:08.761132+0000","last_became_peered":"2024-02-12T17:14:08.761132+0000","last_unstale":"2024-02-12T17:16:57.187330+0000","last_undegraded":"2024-02-12T17:16:57.187330+0000","last_fullsized":"2024-02-12T17:16:57.187330+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:54.052006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.052006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":508,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189390+0000","last_change":"2024-02-12T17:13:46.753228+0000","last_active":"2024-02-12T17:16:57.189390+0000","last_peered":"2024-02-12T17:16:57.189390+0000","last_clean":"2024-02-12T17:16:57.189390+0000","last_became_active":"2024-02-12T17:13:46.752755+0000","last_became_peered":"2024-02-12T17:13:46.752755+0000","last_unstale":"2024-02-12T17:16:57.189390+0000","last_undegraded":"2024-02-12T17:16:57.189390+0000","last_fullsized":"2024-02-12T17:16:57.189390+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":1540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":172,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766439+0000","last_change":"2024-02-12T17:13:46.607814+0000","last_active":"2024-02-12T17:16:54.766439+0000","last_peered":"2024-02-12T17:16:54.766439+0000","last_clean":"2024-02-12T17:16:54.766439+0000","last_became_active":"2024-02-12T17:13:46.606367+0000","last_became_peered":"2024-02-12T17:13:46.606367+0000","last_unstale":"2024-02-12T17:16:54.766439+0000","last_undegraded":"2024-02-12T17:16:54.766439+0000","last_fullsized":"2024-02-12T17:16:54.766439+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:03:17.746921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:03:17.746921+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":247,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.762420+0000","last_change":"2024-02-12T17:13:46.567244+0000","last_active":"2024-02-12T17:16:54.762420+0000","last_peered":"2024-02-12T17:16:54.762420+0000","last_clean":"2024-02-12T17:16:54.762420+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:16:54.762420+0000","last_undegraded":"2024-02-12T17:16:54.762420+0000","last_fullsized":"2024-02-12T17:16:54.762420+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:07:49.813601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.813601+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":82,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602583+0000","last_change":"2024-02-12T17:14:08.993489+0000","last_active":"2024-02-12T17:16:55.602583+0000","last_peered":"2024-02-12T17:16:55.602583+0000","last_clean":"2024-02-12T17:16:55.602583+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:16:55.602583+0000","last_undegraded":"2024-02-12T17:16:55.602583+0000","last_fullsized":"2024-02-12T17:16:55.602583+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:07:28.167508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:28.167508+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":448,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.283947+0000","last_change":"2024-02-12T17:16:46.727669+0000","last_active":"2024-02-12T17:16:55.283947+0000","last_peered":"2024-02-12T17:16:55.283947+0000","last_clean":"2024-02-12T17:16:55.283947+0000","last_became_active":"2024-02-12T17:16:46.727369+0000","last_became_peered":"2024-02-12T17:16:46.727369+0000","last_unstale":"2024-02-12T17:16:55.283947+0000","last_undegraded":"2024-02-12T17:16:55.283947+0000","last_fullsized":"2024-02-12T17:16:55.283947+0000","mapping_epoch":705,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":411,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-12T17:17:01.422130+0000","last_change":"2024-02-12T17:17:01.422130+0000","last_active":"2024-02-12T17:17:01.422130+0000","last_peered":"2024-02-12T17:17:01.422130+0000","last_clean":"2024-02-12T17:17:01.422130+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:17:01.422130+0000","last_undegraded":"2024-02-12T17:17:01.422130+0000","last_fullsized":"2024-02-12T17:17:01.422130+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:01.422046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:17:01.422046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":429,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187521+0000","last_change":"2024-02-12T17:12:12.141297+0000","last_active":"2024-02-12T17:16:57.187521+0000","last_peered":"2024-02-12T17:16:57.187521+0000","last_clean":"2024-02-12T17:16:57.187521+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:16:57.187521+0000","last_undegraded":"2024-02-12T17:16:57.187521+0000","last_fullsized":"2024-02-12T17:16:57.187521+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.141204+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":257,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.761852+0000","last_change":"2024-02-12T17:14:08.743209+0000","last_active":"2024-02-12T17:16:54.761852+0000","last_peered":"2024-02-12T17:16:54.761852+0000","last_clean":"2024-02-12T17:16:54.761852+0000","last_became_active":"2024-02-12T17:14:08.742886+0000","last_became_peered":"2024-02-12T17:14:08.742886+0000","last_unstale":"2024-02-12T17:16:54.761852+0000","last_undegraded":"2024-02-12T17:16:54.761852+0000","last_fullsized":"2024-02-12T17:16:54.761852+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.783527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.783527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":706,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.283946+0000","last_change":"2024-02-12T17:13:21.591627+0000","last_active":"2024-02-12T17:16:55.283946+0000","last_peered":"2024-02-12T17:16:55.283946+0000","last_clean":"2024-02-12T17:16:55.283946+0000","last_became_active":"2024-02-12T17:13:21.591246+0000","last_became_peered":"2024-02-12T17:13:21.591246+0000","last_unstale":"2024-02-12T17:16:55.283946+0000","last_undegraded":"2024-02-12T17:16:55.283946+0000","last_fullsized":"2024-02-12T17:16:55.283946+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.028495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":451,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187794+0000","last_change":"2024-02-12T17:13:46.558817+0000","last_active":"2024-02-12T17:16:57.187794+0000","last_peered":"2024-02-12T17:16:57.187794+0000","last_clean":"2024-02-12T17:16:57.187794+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:16:57.187794+0000","last_undegraded":"2024-02-12T17:16:57.187794+0000","last_fullsized":"2024-02-12T17:16:57.187794+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":104,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766301+0000","last_change":"2024-02-12T17:13:46.576676+0000","last_active":"2024-02-12T17:16:54.766301+0000","last_peered":"2024-02-12T17:16:54.766301+0000","last_clean":"2024-02-12T17:16:54.766301+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:16:54.766301+0000","last_undegraded":"2024-02-12T17:16:54.766301+0000","last_fullsized":"2024-02-12T17:16:54.766301+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":388,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187833+0000","last_change":"2024-02-12T17:14:08.763317+0000","last_active":"2024-02-12T17:16:57.187833+0000","last_peered":"2024-02-12T17:16:57.187833+0000","last_clean":"2024-02-12T17:16:57.187833+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:16:57.187833+0000","last_undegraded":"2024-02-12T17:16:57.187833+0000","last_fullsized":"2024-02-12T17:16:57.187833+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":144,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766749+0000","last_change":"2024-02-12T17:14:08.742170+0000","last_active":"2024-02-12T17:16:54.766749+0000","last_peered":"2024-02-12T17:16:54.766749+0000","last_clean":"2024-02-12T17:16:54.766749+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:16:54.766749+0000","last_undegraded":"2024-02-12T17:16:54.766749+0000","last_fullsized":"2024-02-12T17:16:54.766749+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:20.830126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.830126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":225,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.762068+0000","last_change":"2024-02-12T17:13:46.784204+0000","last_active":"2024-02-12T17:16:54.762068+0000","last_peered":"2024-02-12T17:16:54.762068+0000","last_clean":"2024-02-12T17:16:54.762068+0000","last_became_active":"2024-02-12T17:13:46.783737+0000","last_became_peered":"2024-02-12T17:13:46.783737+0000","last_unstale":"2024-02-12T17:16:54.762068+0000","last_undegraded":"2024-02-12T17:16:54.762068+0000","last_fullsized":"2024-02-12T17:16:54.762068+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:15.817751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.817751+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":778,"reported_epoch":712,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:16:57.189493+0000","last_change":"2024-02-12T17:16:45.076370+0000","last_active":"2024-02-12T17:16:57.189493+0000","last_peered":"2024-02-12T17:16:57.189493+0000","last_clean":"2024-02-12T17:13:44.507959+0000","last_became_active":"2024-02-12T17:14:08.758564+0000","last_became_peered":"2024-02-12T17:14:08.758564+0000","last_unstale":"2024-02-12T17:16:57.189493+0000","last_undegraded":"2024-02-12T17:16:57.189493+0000","last_fullsized":"2024-02-12T17:16:57.189493+0000","mapping_epoch":651,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":75,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:58.559051+0000","last_change":"2024-02-12T17:16:58.559051+0000","last_active":"2024-02-12T17:16:58.559051+0000","last_peered":"2024-02-12T17:16:58.559051+0000","last_clean":"2024-02-12T17:16:58.559051+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:16:58.559051+0000","last_undegraded":"2024-02-12T17:16:58.559051+0000","last_fullsized":"2024-02-12T17:16:58.559051+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:16:58.558963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:16:58.558963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":215,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.762067+0000","last_change":"2024-02-12T17:12:02.752352+0000","last_active":"2024-02-12T17:16:54.762067+0000","last_peered":"2024-02-12T17:16:54.762067+0000","last_clean":"2024-02-12T17:16:54.762067+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:16:54.762067+0000","last_undegraded":"2024-02-12T17:16:54.762067+0000","last_fullsized":"2024-02-12T17:16:54.762067+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:02.752280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:02.752280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":176,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766143+0000","last_change":"2024-02-12T17:13:46.582722+0000","last_active":"2024-02-12T17:16:54.766143+0000","last_peered":"2024-02-12T17:16:54.766143+0000","last_clean":"2024-02-12T17:16:54.766143+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:16:54.766143+0000","last_undegraded":"2024-02-12T17:16:54.766143+0000","last_fullsized":"2024-02-12T17:16:54.766143+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-12T17:06:45.758488+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.758488+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":819,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.700835+0000","last_change":"2024-02-12T17:12:58.020311+0000","last_active":"2024-02-12T17:16:57.700835+0000","last_peered":"2024-02-12T17:16:57.700835+0000","last_clean":"2024-02-12T17:16:57.700835+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:16:57.700835+0000","last_undegraded":"2024-02-12T17:16:57.700835+0000","last_fullsized":"2024-02-12T17:16:57.700835+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:12:58.018712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.018712+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":70,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602439+0000","last_change":"2024-02-12T17:14:08.872636+0000","last_active":"2024-02-12T17:16:55.602439+0000","last_peered":"2024-02-12T17:16:55.602439+0000","last_clean":"2024-02-12T17:16:55.602439+0000","last_became_active":"2024-02-12T17:14:08.872123+0000","last_became_peered":"2024-02-12T17:14:08.872123+0000","last_unstale":"2024-02-12T17:16:55.602439+0000","last_undegraded":"2024-02-12T17:16:55.602439+0000","last_fullsized":"2024-02-12T17:16:55.602439+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:06.904354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:06.904354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":802,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691863+0000","last_change":"2024-02-12T17:12:12.838640+0000","last_active":"2024-02-12T17:16:55.691863+0000","last_peered":"2024-02-12T17:16:55.691863+0000","last_clean":"2024-02-12T17:16:55.691863+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:16:55.691863+0000","last_undegraded":"2024-02-12T17:16:55.691863+0000","last_fullsized":"2024-02-12T17:16:55.691863+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:12.838563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.838563+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"706'26","reported_seq":831,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.188018+0000","last_change":"2024-02-12T17:16:46.731065+0000","last_active":"2024-02-12T17:16:57.188018+0000","last_peered":"2024-02-12T17:16:57.188018+0000","last_clean":"2024-02-12T17:16:57.188018+0000","last_became_active":"2024-02-12T17:16:46.730642+0000","last_became_peered":"2024-02-12T17:16:46.730642+0000","last_unstale":"2024-02-12T17:16:57.188018+0000","last_undegraded":"2024-02-12T17:16:57.188018+0000","last_fullsized":"2024-02-12T17:16:57.188018+0000","mapping_epoch":705,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":706,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691465+0000","last_change":"2024-02-12T17:12:54.656523+0000","last_active":"2024-02-12T17:16:55.691465+0000","last_peered":"2024-02-12T17:16:55.691465+0000","last_clean":"2024-02-12T17:16:55.691465+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:16:55.691465+0000","last_undegraded":"2024-02-12T17:16:55.691465+0000","last_fullsized":"2024-02-12T17:16:55.691465+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:54.656456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.656456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":442,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189739+0000","last_change":"2024-02-12T17:13:46.577769+0000","last_active":"2024-02-12T17:16:57.189739+0000","last_peered":"2024-02-12T17:16:57.189739+0000","last_clean":"2024-02-12T17:16:57.189739+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:16:57.189739+0000","last_undegraded":"2024-02-12T17:16:57.189739+0000","last_fullsized":"2024-02-12T17:16:57.189739+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'20","last_scrub_stamp":"2024-02-12T17:06:34.381767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:06:34.381767+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":728,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.284183+0000","last_change":"2024-02-12T17:13:21.591810+0000","last_active":"2024-02-12T17:16:55.284183+0000","last_peered":"2024-02-12T17:16:55.284183+0000","last_clean":"2024-02-12T17:16:55.284183+0000","last_became_active":"2024-02-12T17:13:21.591502+0000","last_became_peered":"2024-02-12T17:13:21.591502+0000","last_unstale":"2024-02-12T17:16:55.284183+0000","last_undegraded":"2024-02-12T17:16:55.284183+0000","last_fullsized":"2024-02-12T17:16:55.284183+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.882066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.882066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":249,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.761758+0000","last_change":"2024-02-12T17:13:46.651066+0000","last_active":"2024-02-12T17:16:54.761758+0000","last_peered":"2024-02-12T17:16:54.761758+0000","last_clean":"2024-02-12T17:16:54.761758+0000","last_became_active":"2024-02-12T17:13:46.650776+0000","last_became_peered":"2024-02-12T17:13:46.650776+0000","last_unstale":"2024-02-12T17:16:54.761758+0000","last_undegraded":"2024-02-12T17:16:54.761758+0000","last_fullsized":"2024-02-12T17:16:54.761758+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:50.786900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.786900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":431,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189255+0000","last_change":"2024-02-12T17:13:46.822111+0000","last_active":"2024-02-12T17:16:57.189255+0000","last_peered":"2024-02-12T17:16:57.189255+0000","last_clean":"2024-02-12T17:16:57.189255+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:16:57.189255+0000","last_undegraded":"2024-02-12T17:16:57.189255+0000","last_fullsized":"2024-02-12T17:16:57.189255+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":340,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.284192+0000","last_change":"2024-02-12T17:14:08.745944+0000","last_active":"2024-02-12T17:16:55.284192+0000","last_peered":"2024-02-12T17:16:55.284192+0000","last_clean":"2024-02-12T17:16:55.284192+0000","last_became_active":"2024-02-12T17:14:08.745729+0000","last_became_peered":"2024-02-12T17:14:08.745729+0000","last_unstale":"2024-02-12T17:16:55.284192+0000","last_undegraded":"2024-02-12T17:16:55.284192+0000","last_fullsized":"2024-02-12T17:16:55.284192+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:49.809844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:12:49.809844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":104,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766529+0000","last_change":"2024-02-12T17:13:46.559519+0000","last_active":"2024-02-12T17:16:54.766529+0000","last_peered":"2024-02-12T17:16:54.766529+0000","last_clean":"2024-02-12T17:16:54.766529+0000","last_became_active":"2024-02-12T17:13:46.559095+0000","last_became_peered":"2024-02-12T17:13:46.559095+0000","last_unstale":"2024-02-12T17:16:54.766529+0000","last_undegraded":"2024-02-12T17:16:54.766529+0000","last_fullsized":"2024-02-12T17:16:54.766529+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":80,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602545+0000","last_change":"2024-02-12T17:14:08.993503+0000","last_active":"2024-02-12T17:16:55.602545+0000","last_peered":"2024-02-12T17:16:55.602545+0000","last_clean":"2024-02-12T17:16:55.602545+0000","last_became_active":"2024-02-12T17:14:08.861886+0000","last_became_peered":"2024-02-12T17:14:08.861886+0000","last_unstale":"2024-02-12T17:16:55.602545+0000","last_undegraded":"2024-02-12T17:16:55.602545+0000","last_fullsized":"2024-02-12T17:16:55.602545+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'19","last_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:06:45.502635+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":430,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189921+0000","last_change":"2024-02-12T17:13:46.787305+0000","last_active":"2024-02-12T17:16:57.189921+0000","last_peered":"2024-02-12T17:16:57.189921+0000","last_clean":"2024-02-12T17:16:57.189921+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:16:57.189921+0000","last_undegraded":"2024-02-12T17:16:57.189921+0000","last_fullsized":"2024-02-12T17:16:57.189921+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":442,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602631+0000","last_change":"2024-02-12T17:16:46.731039+0000","last_active":"2024-02-12T17:16:55.602631+0000","last_peered":"2024-02-12T17:16:55.602631+0000","last_clean":"2024-02-12T17:16:55.602631+0000","last_became_active":"2024-02-12T17:16:46.730672+0000","last_became_peered":"2024-02-12T17:16:46.730672+0000","last_unstale":"2024-02-12T17:16:55.602631+0000","last_undegraded":"2024-02-12T17:16:55.602631+0000","last_fullsized":"2024-02-12T17:16:55.602631+0000","mapping_epoch":705,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:12:09.882247+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:12:09.882247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":178,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766214+0000","last_change":"2024-02-12T17:13:46.744739+0000","last_active":"2024-02-12T17:16:54.766214+0000","last_peered":"2024-02-12T17:16:54.766214+0000","last_clean":"2024-02-12T17:16:54.766214+0000","last_became_active":"2024-02-12T17:13:46.744003+0000","last_became_peered":"2024-02-12T17:13:46.744003+0000","last_unstale":"2024-02-12T17:16:54.766214+0000","last_undegraded":"2024-02-12T17:16:54.766214+0000","last_fullsized":"2024-02-12T17:16:54.766214+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:51.126056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:51.126056+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":70,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.601596+0000","last_change":"2024-02-12T17:14:08.769824+0000","last_active":"2024-02-12T17:16:55.601596+0000","last_peered":"2024-02-12T17:16:55.601596+0000","last_clean":"2024-02-12T17:16:55.601596+0000","last_became_active":"2024-02-12T17:14:08.769221+0000","last_became_peered":"2024-02-12T17:14:08.769221+0000","last_unstale":"2024-02-12T17:16:55.601596+0000","last_undegraded":"2024-02-12T17:16:55.601596+0000","last_fullsized":"2024-02-12T17:16:55.601596+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:39.761892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:07:39.761892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":717,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.601771+0000","last_change":"2024-02-12T17:14:07.897458+0000","last_active":"2024-02-12T17:16:55.601771+0000","last_peered":"2024-02-12T17:16:55.601771+0000","last_clean":"2024-02-12T17:16:55.601771+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:16:55.601771+0000","last_undegraded":"2024-02-12T17:16:55.601771+0000","last_fullsized":"2024-02-12T17:16:55.601771+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.780381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.780381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":104,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766653+0000","last_change":"2024-02-12T17:13:46.791505+0000","last_active":"2024-02-12T17:16:54.766653+0000","last_peered":"2024-02-12T17:16:54.766653+0000","last_clean":"2024-02-12T17:16:54.766653+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:16:54.766653+0000","last_undegraded":"2024-02-12T17:16:54.766653+0000","last_fullsized":"2024-02-12T17:16:54.766653+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":184,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766425+0000","last_change":"2024-02-12T17:14:09.246132+0000","last_active":"2024-02-12T17:16:54.766425+0000","last_peered":"2024-02-12T17:16:54.766425+0000","last_clean":"2024-02-12T17:16:54.766425+0000","last_became_active":"2024-02-12T17:14:08.752149+0000","last_became_peered":"2024-02-12T17:14:08.752149+0000","last_unstale":"2024-02-12T17:16:54.766425+0000","last_undegraded":"2024-02-12T17:16:54.766425+0000","last_fullsized":"2024-02-12T17:16:54.766425+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:13:01.543047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.543047+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"706'24","reported_seq":557,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.188187+0000","last_change":"2024-02-12T17:16:46.730984+0000","last_active":"2024-02-12T17:16:57.188187+0000","last_peered":"2024-02-12T17:16:57.188187+0000","last_clean":"2024-02-12T17:16:57.188187+0000","last_became_active":"2024-02-12T17:16:46.730723+0000","last_became_peered":"2024-02-12T17:16:46.730723+0000","last_unstale":"2024-02-12T17:16:57.188187+0000","last_undegraded":"2024-02-12T17:16:57.188187+0000","last_fullsized":"2024-02-12T17:16:57.188187+0000","mapping_epoch":705,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'21","last_scrub_stamp":"2024-02-12T17:12:57.741798+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:12:57.741798+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":433,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189672+0000","last_change":"2024-02-12T17:13:21.603543+0000","last_active":"2024-02-12T17:16:57.189672+0000","last_peered":"2024-02-12T17:16:57.189672+0000","last_clean":"2024-02-12T17:16:57.189672+0000","last_became_active":"2024-02-12T17:13:21.603226+0000","last_became_peered":"2024-02-12T17:13:21.603226+0000","last_unstale":"2024-02-12T17:16:57.189672+0000","last_undegraded":"2024-02-12T17:16:57.189672+0000","last_fullsized":"2024-02-12T17:16:57.189672+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.262134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.262134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":748,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691548+0000","last_change":"2024-02-12T17:13:46.566957+0000","last_active":"2024-02-12T17:16:55.691548+0000","last_peered":"2024-02-12T17:16:55.691548+0000","last_clean":"2024-02-12T17:16:55.691548+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:16:55.691548+0000","last_undegraded":"2024-02-12T17:16:55.691548+0000","last_fullsized":"2024-02-12T17:16:55.691548+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":237,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189142+0000","last_change":"2024-02-12T17:13:46.607171+0000","last_active":"2024-02-12T17:16:57.189142+0000","last_peered":"2024-02-12T17:16:57.189142+0000","last_clean":"2024-02-12T17:16:57.189142+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:16:57.189142+0000","last_undegraded":"2024-02-12T17:16:57.189142+0000","last_fullsized":"2024-02-12T17:16:57.189142+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'54","last_scrub_stamp":"2024-02-12T17:02:37.537690+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:02:37.537690+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":673,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602523+0000","last_change":"2024-02-12T17:16:46.731125+0000","last_active":"2024-02-12T17:16:55.602523+0000","last_peered":"2024-02-12T17:16:55.602523+0000","last_clean":"2024-02-12T17:16:55.602523+0000","last_became_active":"2024-02-12T17:16:46.730767+0000","last_became_peered":"2024-02-12T17:16:46.730767+0000","last_unstale":"2024-02-12T17:16:55.602523+0000","last_undegraded":"2024-02-12T17:16:55.602523+0000","last_fullsized":"2024-02-12T17:16:55.602523+0000","mapping_epoch":705,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:13:02.852563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:13:02.852563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":717,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.284293+0000","last_change":"2024-02-12T17:13:00.766698+0000","last_active":"2024-02-12T17:16:55.284293+0000","last_peered":"2024-02-12T17:16:55.284293+0000","last_clean":"2024-02-12T17:16:55.284293+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:16:55.284293+0000","last_undegraded":"2024-02-12T17:16:55.284293+0000","last_fullsized":"2024-02-12T17:16:55.284293+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.766590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":405,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:58.387047+0000","last_change":"2024-02-12T17:13:46.584941+0000","last_active":"2024-02-12T17:16:58.387047+0000","last_peered":"2024-02-12T17:16:58.387047+0000","last_clean":"2024-02-12T17:16:58.387047+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:16:58.387047+0000","last_undegraded":"2024-02-12T17:16:58.387047+0000","last_fullsized":"2024-02-12T17:16:58.387047+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":405,"num_read_kb":400,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":379,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187603+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:16:57.187603+0000","last_peered":"2024-02-12T17:16:57.187603+0000","last_clean":"2024-02-12T17:16:57.187603+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:16:57.187603+0000","last_undegraded":"2024-02-12T17:16:57.187603+0000","last_fullsized":"2024-02-12T17:16:57.187603+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":182,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.695986+0000","last_change":"2024-02-12T17:16:55.695958+0000","last_active":"2024-02-12T17:16:55.695986+0000","last_peered":"2024-02-12T17:16:55.695986+0000","last_clean":"2024-02-12T17:16:55.695986+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:16:55.695986+0000","last_undegraded":"2024-02-12T17:16:55.695986+0000","last_fullsized":"2024-02-12T17:16:55.695986+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'26","last_scrub_stamp":"2024-02-12T17:16:55.695884+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:16:55.695884+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":641,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691927+0000","last_change":"2024-02-12T17:13:10.412138+0000","last_active":"2024-02-12T17:16:55.691927+0000","last_peered":"2024-02-12T17:16:55.691927+0000","last_clean":"2024-02-12T17:16:55.691927+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:16:55.691927+0000","last_undegraded":"2024-02-12T17:16:55.691927+0000","last_fullsized":"2024-02-12T17:16:55.691927+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:13:10.412031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":780,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.692118+0000","last_change":"2024-02-12T17:13:46.826853+0000","last_active":"2024-02-12T17:16:55.692118+0000","last_peered":"2024-02-12T17:16:55.692118+0000","last_clean":"2024-02-12T17:16:55.692118+0000","last_became_active":"2024-02-12T17:13:46.821418+0000","last_became_peered":"2024-02-12T17:13:46.821418+0000","last_unstale":"2024-02-12T17:16:55.692118+0000","last_undegraded":"2024-02-12T17:16:55.692118+0000","last_fullsized":"2024-02-12T17:16:55.692118+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.951964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.951964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":411,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189045+0000","last_change":"2024-02-12T17:12:59.006372+0000","last_active":"2024-02-12T17:16:57.189045+0000","last_peered":"2024-02-12T17:16:57.189045+0000","last_clean":"2024-02-12T17:16:57.189045+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:16:57.189045+0000","last_undegraded":"2024-02-12T17:16:57.189045+0000","last_fullsized":"2024-02-12T17:16:57.189045+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.006260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.006260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":688,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691635+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:16:55.691635+0000","last_peered":"2024-02-12T17:16:55.691635+0000","last_clean":"2024-02-12T17:16:55.691635+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:16:55.691635+0000","last_undegraded":"2024-02-12T17:16:55.691635+0000","last_fullsized":"2024-02-12T17:16:55.691635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":757,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.283776+0000","last_change":"2024-02-12T17:14:08.743668+0000","last_active":"2024-02-12T17:16:55.283776+0000","last_peered":"2024-02-12T17:16:55.283776+0000","last_clean":"2024-02-12T17:16:55.283776+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:16:55.283776+0000","last_undegraded":"2024-02-12T17:16:55.283776+0000","last_fullsized":"2024-02-12T17:16:55.283776+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:49.876199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:06:49.876199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":322,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.762380+0000","last_change":"2024-02-12T17:14:08.781462+0000","last_active":"2024-02-12T17:16:54.762380+0000","last_peered":"2024-02-12T17:16:54.762380+0000","last_clean":"2024-02-12T17:16:54.762380+0000","last_became_active":"2024-02-12T17:14:08.726463+0000","last_became_peered":"2024-02-12T17:14:08.726463+0000","last_unstale":"2024-02-12T17:16:54.762380+0000","last_undegraded":"2024-02-12T17:16:54.762380+0000","last_fullsized":"2024-02-12T17:16:54.762380+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:47.763264+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.763264+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":719,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.284248+0000","last_change":"2024-02-12T17:12:03.922963+0000","last_active":"2024-02-12T17:16:55.284248+0000","last_peered":"2024-02-12T17:16:55.284248+0000","last_clean":"2024-02-12T17:16:55.284248+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:16:55.284248+0000","last_undegraded":"2024-02-12T17:16:55.284248+0000","last_fullsized":"2024-02-12T17:16:55.284248+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:12:03.922893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.922893+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":104,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766741+0000","last_change":"2024-02-12T17:13:46.608049+0000","last_active":"2024-02-12T17:16:54.766741+0000","last_peered":"2024-02-12T17:16:54.766741+0000","last_clean":"2024-02-12T17:16:54.766741+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:16:54.766741+0000","last_undegraded":"2024-02-12T17:16:54.766741+0000","last_fullsized":"2024-02-12T17:16:54.766741+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:02.044887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:02.044887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":70,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602276+0000","last_change":"2024-02-12T17:14:08.859925+0000","last_active":"2024-02-12T17:16:55.602276+0000","last_peered":"2024-02-12T17:16:55.602276+0000","last_clean":"2024-02-12T17:16:55.602276+0000","last_became_active":"2024-02-12T17:14:08.859396+0000","last_became_peered":"2024-02-12T17:14:08.859396+0000","last_unstale":"2024-02-12T17:16:55.602276+0000","last_undegraded":"2024-02-12T17:16:55.602276+0000","last_fullsized":"2024-02-12T17:16:55.602276+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:03.904976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.904976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":757,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.283726+0000","last_change":"2024-02-12T17:14:05.666060+0000","last_active":"2024-02-12T17:16:55.283726+0000","last_peered":"2024-02-12T17:16:55.283726+0000","last_clean":"2024-02-12T17:16:55.283726+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:16:55.283726+0000","last_undegraded":"2024-02-12T17:16:55.283726+0000","last_fullsized":"2024-02-12T17:16:55.283726+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:06:01.905929+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":89,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.566977+0000","last_change":"2024-02-12T17:16:57.566977+0000","last_active":"2024-02-12T17:16:57.566977+0000","last_peered":"2024-02-12T17:16:57.566977+0000","last_clean":"2024-02-12T17:16:57.566977+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:16:57.566977+0000","last_undegraded":"2024-02-12T17:16:57.566977+0000","last_fullsized":"2024-02-12T17:16:57.566977+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'36","last_scrub_stamp":"2024-02-12T17:16:57.566903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:16:57.566903+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":760,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.692104+0000","last_change":"2024-02-12T17:13:46.827015+0000","last_active":"2024-02-12T17:16:55.692104+0000","last_peered":"2024-02-12T17:16:55.692104+0000","last_clean":"2024-02-12T17:16:55.692104+0000","last_became_active":"2024-02-12T17:13:46.821796+0000","last_became_peered":"2024-02-12T17:13:46.821796+0000","last_unstale":"2024-02-12T17:16:55.692104+0000","last_undegraded":"2024-02-12T17:16:55.692104+0000","last_fullsized":"2024-02-12T17:16:55.692104+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.698727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.698727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":135,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766075+0000","last_change":"2024-02-12T17:13:46.579742+0000","last_active":"2024-02-12T17:16:54.766075+0000","last_peered":"2024-02-12T17:16:54.766075+0000","last_clean":"2024-02-12T17:16:54.766075+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:16:54.766075+0000","last_undegraded":"2024-02-12T17:16:54.766075+0000","last_fullsized":"2024-02-12T17:16:54.766075+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":771,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.284348+0000","last_change":"2024-02-12T17:14:08.746702+0000","last_active":"2024-02-12T17:16:55.284348+0000","last_peered":"2024-02-12T17:16:55.284348+0000","last_clean":"2024-02-12T17:16:55.284348+0000","last_became_active":"2024-02-12T17:14:08.746395+0000","last_became_peered":"2024-02-12T17:14:08.746395+0000","last_unstale":"2024-02-12T17:16:55.284348+0000","last_undegraded":"2024-02-12T17:16:55.284348+0000","last_fullsized":"2024-02-12T17:16:55.284348+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:45.764955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:45.764955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":88,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602320+0000","last_change":"2024-02-12T17:14:11.148689+0000","last_active":"2024-02-12T17:16:55.602320+0000","last_peered":"2024-02-12T17:16:55.602320+0000","last_clean":"2024-02-12T17:16:55.602320+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:16:55.602320+0000","last_undegraded":"2024-02-12T17:16:55.602320+0000","last_fullsized":"2024-02-12T17:16:55.602320+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":213,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.761872+0000","last_change":"2024-02-12T17:12:53.746937+0000","last_active":"2024-02-12T17:16:54.761872+0000","last_peered":"2024-02-12T17:16:54.761872+0000","last_clean":"2024-02-12T17:16:54.761872+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:16:54.761872+0000","last_undegraded":"2024-02-12T17:16:54.761872+0000","last_fullsized":"2024-02-12T17:16:54.761872+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.746858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.746858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":420,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189133+0000","last_change":"2024-02-12T17:13:21.589965+0000","last_active":"2024-02-12T17:16:57.189133+0000","last_peered":"2024-02-12T17:16:57.189133+0000","last_clean":"2024-02-12T17:16:57.189133+0000","last_became_active":"2024-02-12T17:13:21.589584+0000","last_became_peered":"2024-02-12T17:13:21.589584+0000","last_unstale":"2024-02-12T17:16:57.189133+0000","last_undegraded":"2024-02-12T17:16:57.189133+0000","last_fullsized":"2024-02-12T17:16:57.189133+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:01.059570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.059570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":151,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766851+0000","last_change":"2024-02-12T17:14:18.716530+0000","last_active":"2024-02-12T17:16:54.766851+0000","last_peered":"2024-02-12T17:16:54.766851+0000","last_clean":"2024-02-12T17:16:54.766851+0000","last_became_active":"2024-02-12T17:14:18.716232+0000","last_became_peered":"2024-02-12T17:14:18.716232+0000","last_unstale":"2024-02-12T17:16:54.766851+0000","last_undegraded":"2024-02-12T17:16:54.766851+0000","last_fullsized":"2024-02-12T17:16:54.766851+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":484,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:17:00.320217+0000","last_change":"2024-02-12T17:17:00.320191+0000","last_active":"2024-02-12T17:17:00.320217+0000","last_peered":"2024-02-12T17:17:00.320217+0000","last_clean":"2024-02-12T17:17:00.320217+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:17:00.320217+0000","last_undegraded":"2024-02-12T17:17:00.320217+0000","last_fullsized":"2024-02-12T17:17:00.320217+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:17:00.320127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:17:00.320127+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":699,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187422+0000","last_change":"2024-02-12T17:14:08.765556+0000","last_active":"2024-02-12T17:16:57.187422+0000","last_peered":"2024-02-12T17:16:57.187422+0000","last_clean":"2024-02-12T17:16:57.187422+0000","last_became_active":"2024-02-12T17:14:08.757310+0000","last_became_peered":"2024-02-12T17:14:08.757310+0000","last_unstale":"2024-02-12T17:16:57.187422+0000","last_undegraded":"2024-02-12T17:16:57.187422+0000","last_fullsized":"2024-02-12T17:16:57.187422+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:58.871569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.871569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":750,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.284290+0000","last_change":"2024-02-12T17:13:08.874063+0000","last_active":"2024-02-12T17:16:55.284290+0000","last_peered":"2024-02-12T17:16:55.284290+0000","last_clean":"2024-02-12T17:16:55.284290+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:16:55.284290+0000","last_undegraded":"2024-02-12T17:16:55.284290+0000","last_fullsized":"2024-02-12T17:16:55.284290+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:13:08.873961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:13:08.873961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":231,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.762250+0000","last_change":"2024-02-12T17:14:14.843264+0000","last_active":"2024-02-12T17:16:54.762250+0000","last_peered":"2024-02-12T17:16:54.762250+0000","last_clean":"2024-02-12T17:16:54.762250+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:16:54.762250+0000","last_undegraded":"2024-02-12T17:16:54.762250+0000","last_fullsized":"2024-02-12T17:16:54.762250+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:56.688982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.688982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":896,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691471+0000","last_change":"2024-02-12T17:14:09.517724+0000","last_active":"2024-02-12T17:16:55.691471+0000","last_peered":"2024-02-12T17:16:55.691471+0000","last_clean":"2024-02-12T17:16:55.691471+0000","last_became_active":"2024-02-12T17:14:08.740691+0000","last_became_peered":"2024-02-12T17:14:08.740691+0000","last_unstale":"2024-02-12T17:16:55.691471+0000","last_undegraded":"2024-02-12T17:16:55.691471+0000","last_fullsized":"2024-02-12T17:16:55.691471+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:12:05.850456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.850456+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"706'33","reported_seq":687,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.700822+0000","last_change":"2024-02-12T17:16:46.736263+0000","last_active":"2024-02-12T17:16:57.700822+0000","last_peered":"2024-02-12T17:16:57.700822+0000","last_clean":"2024-02-12T17:16:57.700822+0000","last_became_active":"2024-02-12T17:16:46.735883+0000","last_became_peered":"2024-02-12T17:16:46.735883+0000","last_unstale":"2024-02-12T17:16:57.700822+0000","last_undegraded":"2024-02-12T17:16:57.700822+0000","last_fullsized":"2024-02-12T17:16:57.700822+0000","mapping_epoch":705,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":422,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189523+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:16:57.189523+0000","last_peered":"2024-02-12T17:16:57.189523+0000","last_clean":"2024-02-12T17:16:57.189523+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:16:57.189523+0000","last_undegraded":"2024-02-12T17:16:57.189523+0000","last_fullsized":"2024-02-12T17:16:57.189523+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"642'31","reported_seq":1131,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187960+0000","last_change":"2024-02-12T17:13:53.773053+0000","last_active":"2024-02-12T17:16:57.187960+0000","last_peered":"2024-02-12T17:16:57.187960+0000","last_clean":"2024-02-12T17:16:57.187960+0000","last_became_active":"2024-02-12T17:13:53.772744+0000","last_became_peered":"2024-02-12T17:13:53.772744+0000","last_unstale":"2024-02-12T17:16:57.187960+0000","last_undegraded":"2024-02-12T17:16:57.187960+0000","last_fullsized":"2024-02-12T17:16:57.187960+0000","mapping_epoch":638,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'28","last_scrub_stamp":"2024-02-12T17:12:07.826227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:12:07.826227+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":425,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.283630+0000","last_change":"2024-02-12T17:14:10.756477+0000","last_active":"2024-02-12T17:16:55.283630+0000","last_peered":"2024-02-12T17:16:55.283630+0000","last_clean":"2024-02-12T17:16:55.283630+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:16:55.283630+0000","last_undegraded":"2024-02-12T17:16:55.283630+0000","last_fullsized":"2024-02-12T17:16:55.283630+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:16.873570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.873570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":285,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.762236+0000","last_change":"2024-02-12T17:13:46.807564+0000","last_active":"2024-02-12T17:16:54.762236+0000","last_peered":"2024-02-12T17:16:54.762236+0000","last_clean":"2024-02-12T17:16:54.762236+0000","last_became_active":"2024-02-12T17:13:46.807214+0000","last_became_peered":"2024-02-12T17:13:46.807214+0000","last_unstale":"2024-02-12T17:16:54.762236+0000","last_undegraded":"2024-02-12T17:16:54.762236+0000","last_fullsized":"2024-02-12T17:16:54.762236+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:12:04.705646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.705646+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":755,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691985+0000","last_change":"2024-02-12T17:14:08.740896+0000","last_active":"2024-02-12T17:16:55.691985+0000","last_peered":"2024-02-12T17:16:55.691985+0000","last_clean":"2024-02-12T17:16:55.691985+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:16:55.691985+0000","last_undegraded":"2024-02-12T17:16:55.691985+0000","last_fullsized":"2024-02-12T17:16:55.691985+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:41.672523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:02:41.672523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"668'27","reported_seq":1156,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187882+0000","last_change":"2024-02-12T17:14:24.235718+0000","last_active":"2024-02-12T17:16:57.187882+0000","last_peered":"2024-02-12T17:16:57.187882+0000","last_clean":"2024-02-12T17:16:57.187882+0000","last_became_active":"2024-02-12T17:14:24.235386+0000","last_became_peered":"2024-02-12T17:14:24.235386+0000","last_unstale":"2024-02-12T17:16:57.187882+0000","last_undegraded":"2024-02-12T17:16:57.187882+0000","last_fullsized":"2024-02-12T17:16:57.187882+0000","mapping_epoch":665,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'24","last_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.647212+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":122,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766065+0000","last_change":"2024-02-12T17:14:05.106537+0000","last_active":"2024-02-12T17:16:54.766065+0000","last_peered":"2024-02-12T17:16:54.766065+0000","last_clean":"2024-02-12T17:16:54.766065+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:16:54.766065+0000","last_undegraded":"2024-02-12T17:16:54.766065+0000","last_fullsized":"2024-02-12T17:16:54.766065+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:19.079155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:19.079155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":170,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766806+0000","last_change":"2024-02-12T17:13:46.819259+0000","last_active":"2024-02-12T17:16:54.766806+0000","last_peered":"2024-02-12T17:16:54.766806+0000","last_clean":"2024-02-12T17:16:54.766806+0000","last_became_active":"2024-02-12T17:13:46.818771+0000","last_became_peered":"2024-02-12T17:13:46.818771+0000","last_unstale":"2024-02-12T17:16:54.766806+0000","last_undegraded":"2024-02-12T17:16:54.766806+0000","last_fullsized":"2024-02-12T17:16:54.766806+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":102,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766773+0000","last_change":"2024-02-12T17:13:46.805110+0000","last_active":"2024-02-12T17:16:54.766773+0000","last_peered":"2024-02-12T17:16:54.766773+0000","last_clean":"2024-02-12T17:16:54.766773+0000","last_became_active":"2024-02-12T17:13:46.802275+0000","last_became_peered":"2024-02-12T17:13:46.802275+0000","last_unstale":"2024-02-12T17:16:54.766773+0000","last_undegraded":"2024-02-12T17:16:54.766773+0000","last_fullsized":"2024-02-12T17:16:54.766773+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":659,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189238+0000","last_change":"2024-02-12T17:13:07.711701+0000","last_active":"2024-02-12T17:16:57.189238+0000","last_peered":"2024-02-12T17:16:57.189238+0000","last_clean":"2024-02-12T17:16:57.189238+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:16:57.189238+0000","last_undegraded":"2024-02-12T17:16:57.189238+0000","last_fullsized":"2024-02-12T17:16:57.189238+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:13:07.711601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:07.711601+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":122,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766246+0000","last_change":"2024-02-12T17:14:00.999940+0000","last_active":"2024-02-12T17:16:54.766246+0000","last_peered":"2024-02-12T17:16:54.766246+0000","last_clean":"2024-02-12T17:16:54.766246+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:16:54.766246+0000","last_undegraded":"2024-02-12T17:16:54.766246+0000","last_fullsized":"2024-02-12T17:16:54.766246+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":516,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189624+0000","last_change":"2024-02-12T17:13:46.806094+0000","last_active":"2024-02-12T17:16:57.189624+0000","last_peered":"2024-02-12T17:16:57.189624+0000","last_clean":"2024-02-12T17:16:57.189624+0000","last_became_active":"2024-02-12T17:13:46.805712+0000","last_became_peered":"2024-02-12T17:13:46.805712+0000","last_unstale":"2024-02-12T17:16:57.189624+0000","last_undegraded":"2024-02-12T17:16:57.189624+0000","last_fullsized":"2024-02-12T17:16:57.189624+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":442,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189162+0000","last_change":"2024-02-12T17:13:21.603687+0000","last_active":"2024-02-12T17:16:57.189162+0000","last_peered":"2024-02-12T17:16:57.189162+0000","last_clean":"2024-02-12T17:16:57.189162+0000","last_became_active":"2024-02-12T17:13:21.603332+0000","last_became_peered":"2024-02-12T17:13:21.603332+0000","last_unstale":"2024-02-12T17:16:57.189162+0000","last_undegraded":"2024-02-12T17:16:57.189162+0000","last_fullsized":"2024-02-12T17:16:57.189162+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":133,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766571+0000","last_change":"2024-02-12T17:13:46.804446+0000","last_active":"2024-02-12T17:16:54.766571+0000","last_peered":"2024-02-12T17:16:54.766571+0000","last_clean":"2024-02-12T17:16:54.766571+0000","last_became_active":"2024-02-12T17:13:46.804033+0000","last_became_peered":"2024-02-12T17:13:46.804033+0000","last_unstale":"2024-02-12T17:16:54.766571+0000","last_undegraded":"2024-02-12T17:16:54.766571+0000","last_fullsized":"2024-02-12T17:16:54.766571+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.793569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.793569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":735,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.283935+0000","last_change":"2024-02-12T17:13:56.661004+0000","last_active":"2024-02-12T17:16:55.283935+0000","last_peered":"2024-02-12T17:16:55.283935+0000","last_clean":"2024-02-12T17:16:55.283935+0000","last_became_active":"2024-02-12T17:13:56.660707+0000","last_became_peered":"2024-02-12T17:13:56.660707+0000","last_unstale":"2024-02-12T17:16:55.283935+0000","last_undegraded":"2024-02-12T17:16:55.283935+0000","last_fullsized":"2024-02-12T17:16:55.283935+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.685601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.685601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":202,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:54.766608+0000","last_change":"2024-02-12T17:13:46.757888+0000","last_active":"2024-02-12T17:16:54.766608+0000","last_peered":"2024-02-12T17:16:54.766608+0000","last_clean":"2024-02-12T17:16:54.766608+0000","last_became_active":"2024-02-12T17:13:46.757450+0000","last_became_peered":"2024-02-12T17:13:46.757450+0000","last_unstale":"2024-02-12T17:16:54.766608+0000","last_undegraded":"2024-02-12T17:16:54.766608+0000","last_fullsized":"2024-02-12T17:16:54.766608+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:56.880282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.880282+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":735,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.283689+0000","last_change":"2024-02-12T17:12:08.013330+0000","last_active":"2024-02-12T17:16:55.283689+0000","last_peered":"2024-02-12T17:16:55.283689+0000","last_clean":"2024-02-12T17:16:55.283689+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:16:55.283689+0000","last_undegraded":"2024-02-12T17:16:55.283689+0000","last_fullsized":"2024-02-12T17:16:55.283689+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.013247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.013247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":632,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602468+0000","last_change":"2024-02-12T17:12:04.711418+0000","last_active":"2024-02-12T17:16:55.602468+0000","last_peered":"2024-02-12T17:16:55.602468+0000","last_clean":"2024-02-12T17:16:55.602468+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:16:55.602468+0000","last_undegraded":"2024-02-12T17:16:55.602468+0000","last_fullsized":"2024-02-12T17:16:55.602468+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:04.711333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.711333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":490,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691685+0000","last_change":"2024-02-12T17:14:08.740752+0000","last_active":"2024-02-12T17:16:55.691685+0000","last_peered":"2024-02-12T17:16:55.691685+0000","last_clean":"2024-02-12T17:16:55.691685+0000","last_became_active":"2024-02-12T17:14:08.740430+0000","last_became_peered":"2024-02-12T17:14:08.740430+0000","last_unstale":"2024-02-12T17:16:55.691685+0000","last_undegraded":"2024-02-12T17:16:55.691685+0000","last_fullsized":"2024-02-12T17:16:55.691685+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":272,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:56.742791+0000","last_change":"2024-02-12T17:16:56.742750+0000","last_active":"2024-02-12T17:16:56.742791+0000","last_peered":"2024-02-12T17:16:56.742791+0000","last_clean":"2024-02-12T17:16:56.742791+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:16:56.742791+0000","last_undegraded":"2024-02-12T17:16:56.742791+0000","last_fullsized":"2024-02-12T17:16:56.742791+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'51","last_scrub_stamp":"2024-02-12T17:16:56.742673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:16:56.742673+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":70,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602006+0000","last_change":"2024-02-12T17:14:08.859503+0000","last_active":"2024-02-12T17:16:55.602006+0000","last_peered":"2024-02-12T17:16:55.602006+0000","last_clean":"2024-02-12T17:16:55.602006+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:16:55.602006+0000","last_undegraded":"2024-02-12T17:16:55.602006+0000","last_fullsized":"2024-02-12T17:16:55.602006+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:03.705889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:13:03.705889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":633,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.187474+0000","last_change":"2024-02-12T17:14:08.762783+0000","last_active":"2024-02-12T17:16:57.187474+0000","last_peered":"2024-02-12T17:16:57.187474+0000","last_clean":"2024-02-12T17:16:57.187474+0000","last_became_active":"2024-02-12T17:14:08.760654+0000","last_became_peered":"2024-02-12T17:14:08.760654+0000","last_unstale":"2024-02-12T17:16:57.187474+0000","last_undegraded":"2024-02-12T17:16:57.187474+0000","last_fullsized":"2024-02-12T17:16:57.187474+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:09.044325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:09.044325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":414,"reported_epoch":712,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:16:57.189513+0000","last_change":"2024-02-12T17:16:45.074788+0000","last_active":"2024-02-12T17:16:57.189513+0000","last_peered":"2024-02-12T17:16:57.189513+0000","last_clean":"2024-02-12T17:13:44.508652+0000","last_became_active":"2024-02-12T17:13:46.563455+0000","last_became_peered":"2024-02-12T17:13:46.563455+0000","last_unstale":"2024-02-12T17:16:57.189513+0000","last_undegraded":"2024-02-12T17:16:57.189513+0000","last_fullsized":"2024-02-12T17:16:57.189513+0000","mapping_epoch":630,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":80,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.602141+0000","last_change":"2024-02-12T17:14:10.285684+0000","last_active":"2024-02-12T17:16:55.602141+0000","last_peered":"2024-02-12T17:16:55.602141+0000","last_clean":"2024-02-12T17:16:55.602141+0000","last_became_active":"2024-02-12T17:14:08.871051+0000","last_became_peered":"2024-02-12T17:14:08.871051+0000","last_unstale":"2024-02-12T17:16:55.602141+0000","last_undegraded":"2024-02-12T17:16:55.602141+0000","last_fullsized":"2024-02-12T17:16:55.602141+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:50.764686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.764686+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":718,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691460+0000","last_change":"2024-02-12T17:12:11.906816+0000","last_active":"2024-02-12T17:16:55.691460+0000","last_peered":"2024-02-12T17:16:55.691460+0000","last_clean":"2024-02-12T17:16:55.691460+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:16:55.691460+0000","last_undegraded":"2024-02-12T17:16:55.691460+0000","last_fullsized":"2024-02-12T17:16:55.691460+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:11.906737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:11.906737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":397,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.189761+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:16:57.189761+0000","last_peered":"2024-02-12T17:16:57.189761+0000","last_clean":"2024-02-12T17:16:57.189761+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:16:57.189761+0000","last_undegraded":"2024-02-12T17:16:57.189761+0000","last_fullsized":"2024-02-12T17:16:57.189761+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":836,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-12T17:16:55.691412+0000","last_change":"2024-02-12T17:12:16.820857+0000","last_active":"2024-02-12T17:16:55.691412+0000","last_peered":"2024-02-12T17:16:55.691412+0000","last_clean":"2024-02-12T17:16:55.691412+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:16:55.691412+0000","last_undegraded":"2024-02-12T17:16:55.691412+0000","last_fullsized":"2024-02-12T17:16:55.691412+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:12:16.820782+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.820782+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"632'35","reported_seq":1220,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-12T17:16:57.188821+0000","last_change":"2024-02-12T17:13:46.691596+0000","last_active":"2024-02-12T17:16:57.188821+0000","last_peered":"2024-02-12T17:16:57.188821+0000","last_clean":"2024-02-12T17:16:57.188821+0000","last_became_active":"2024-02-12T17:13:46.691213+0000","last_became_peered":"2024-02-12T17:13:46.691213+0000","last_unstale":"2024-02-12T17:16:57.188821+0000","last_undegraded":"2024-02-12T17:16:57.188821+0000","last_fullsized":"2024-02-12T17:16:57.188821+0000","mapping_epoch":631,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'29","last_scrub_stamp":"2024-02-12T17:12:47.788327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.788327+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":405,"num_read_kb":400,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":16216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1029,"ondisk_log_size":1029,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234315,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1666892,"kb_used_data":174724,"kb_used_omap":0,"kb_used_meta":1492160,"kb_avail":92078260,"statfs":{"total":95995035648,"available":94288138240,"internally_reserved":0,"allocated":178917376,"data_stored":177450985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1527971840},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":2.5249999999999999,"5min":1.9099999999999999,"15min":2.024},"min":{"1min":0.36799999999999999,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":10.417,"5min":15.856999999999999,"15min":15.856999999999999},"last":0.433},{"interface":"front","average":{"1min":1.756,"5min":1.8180000000000001,"15min":1.7290000000000001},"min":{"1min":0.23599999999999999,"5min":0.221,"15min":0.22},"max":{"1min":7.1120000000000001,"5min":12.565,"15min":12.565},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":2.6829999999999998,"15min":2.7530000000000001},"min":{"1min":0.26800000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":7.8949999999999996,"5min":11.632999999999999,"15min":11.632999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.853,"5min":2.613,"15min":2.516},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":7.8869999999999996,"5min":10.91,"15min":10.91},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":2.8010000000000002,"15min":3.1349999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22800000000000001},"max":{"1min":9.5389999999999997,"5min":42.162999999999997,"15min":218.35300000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.3820000000000001,"5min":3,"15min":3.2480000000000002},"min":{"1min":0.375,"5min":0.25600000000000001,"15min":0.247},"max":{"1min":9.407,"5min":42.109999999999999,"15min":218.42099999999999},"last":0.41199999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":3.754,"5min":2.52,"15min":2.1930000000000001},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.255},"max":{"1min":11.741,"5min":15.811999999999999,"15min":15.811999999999999},"last":0.504},{"interface":"front","average":{"1min":3.9079999999999999,"5min":3.0710000000000002,"15min":2.649},"min":{"1min":0.42299999999999999,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":12.375999999999999,"5min":13.452,"15min":31.152000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.4099999999999999,"15min":1.6759999999999999},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":2.7810000000000001,"5min":20.672999999999998,"15min":20.672999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.8939999999999999,"15min":1.96},"min":{"1min":0.23300000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.7240000000000002,"5min":38.390000000000001,"15min":38.390000000000001},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.8169999999999999,"15min":2.169},"min":{"1min":0.28100000000000003,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":2.7000000000000002,"5min":20.702000000000002,"15min":56.560000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.4330000000000001,"15min":1.915},"min":{"1min":0.25800000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.6930000000000001,"5min":10.669,"15min":53.963999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.4570000000000001,"15min":1.532},"min":{"1min":0.30399999999999999,"5min":0.223,"15min":0.223},"max":{"1min":4.0030000000000001,"5min":21.620000000000001,"15min":21.620000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.671,"15min":1.724},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.7360000000000002,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.59599999999999997}]}]},{"osd":6,"up_from":45,"seq":193273528656,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891728,"kb_used_data":3464,"kb_used_omap":0,"kb_used_meta":888256,"kb_avail":92853424,"statfs":{"total":95995035648,"available":95081906176,"internally_reserved":0,"allocated":3547136,"data_stored":2077177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":909574144},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.7410000000000001,"15min":1.806},"min":{"1min":0.22,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":5.266,"5min":33.576999999999998,"15min":33.576999999999998},"last":0.499},{"interface":"front","average":{"1min":2.194,"5min":2.0859999999999999,"15min":1.9239999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9740000000000002,"5min":33.569000000000003,"15min":33.569000000000003},"last":0.53200000000000003}]},{"osd":1,"last update":"Mon Feb 12 17:16:50 2024","interfaces":[{"interface":"back","average":{"1min":2.8889999999999998,"5min":2.2759999999999998,"15min":2.153},"min":{"1min":0.29799999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.611000000000001,"5min":20.611000000000001,"15min":20.611000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":2.7810000000000001,"5min":2.258,"15min":2.2149999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":20.622,"5min":20.622,"15min":20.622},"last":0.60599999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8849999999999998,"5min":2.7290000000000001,"15min":2.6240000000000001},"min":{"1min":0.35399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.5760000000000005,"5min":33.594999999999999,"15min":56.798000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":4.3019999999999996,"5min":2.6680000000000001,"15min":2.3639999999999999},"min":{"1min":0.36099999999999999,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":14.045999999999999,"5min":33.526000000000003,"15min":56.780999999999999},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4489999999999998,"5min":3.1779999999999999,"15min":2.3999999999999999},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":12.336,"5min":58.420999999999999,"15min":58.420999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":3.7109999999999999,"5min":3.8290000000000002,"15min":2.8530000000000002},"min":{"1min":0.40400000000000003,"5min":0.254,"15min":0.254},"max":{"1min":11.737,"5min":58.709000000000003,"15min":58.709000000000003},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.542,"15min":1.272},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.23499999999999999},"max":{"1min":5.0810000000000004,"5min":33.706000000000003,"15min":33.706000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.8540000000000001,"15min":1.508},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":4.2460000000000004,"5min":34.895000000000003,"15min":34.895000000000003},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.6639999999999999,"15min":1.716},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.1970000000000001,"5min":14.56,"15min":34.869},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.319,"15min":1.6799999999999999},"min":{"1min":0.36399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.1390000000000002,"5min":11.1,"15min":33.591000000000001},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Feb 12 17:16:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.7090000000000001,"15min":1.7629999999999999},"min":{"1min":0.26500000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":4.4550000000000001,"5min":35.122999999999998,"15min":35.122999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.421,"5min":1.869,"15min":1.6220000000000001},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.26500000000000001},"max":{"1min":5.1890000000000001,"5min":34.045000000000002,"15min":34.045000000000002},"last":0.65900000000000003}]}]},{"osd":1,"up_from":630,"seq":2705829396520,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338028,"kb_used_data":3492,"kb_used_omap":3,"kb_used_meta":334524,"kb_avail":93407124,"statfs":{"total":95995035648,"available":95648894976,"internally_reserved":0,"allocated":3575808,"data_stored":2077734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":342552637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":3.2170000000000001,"5min":3.306,"15min":3.4249999999999998},"min":{"1min":0.26100000000000001,"5min":0.214,"15min":0.214},"max":{"1min":23.952999999999999,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":8.1310000000000002,"5min":4.5999999999999996,"15min":3.8140000000000001},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":27.268999999999998,"5min":27.268999999999998,"15min":27.268999999999998},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":2.7959999999999998,"5min":3.2639999999999998,"15min":3.476},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.856,"5min":12.856,"15min":12.856},"last":0.32900000000000001},{"interface":"front","average":{"1min":5.0640000000000001,"5min":3.843,"15min":3.6120000000000001},"min":{"1min":0.255,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":24.236999999999998,"5min":24.236999999999998,"15min":24.236999999999998},"last":0.37}]},{"osd":3,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2800000000000002,"5min":4.5110000000000001,"15min":3.9319999999999999},"min":{"1min":0.29599999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":31.292999999999999,"5min":31.292999999999999,"15min":31.292999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":6.3230000000000004,"5min":4.8010000000000002,"15min":4.4809999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":23.905999999999999,"5min":23.905999999999999,"15min":23.905999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":2.3479999999999999,"15min":2.4860000000000002},"min":{"1min":0.34100000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":12.901999999999999,"5min":12.901999999999999,"15min":12.901999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":3.0499999999999998,"5min":2.7989999999999999,"15min":2.7869999999999999},"min":{"1min":0.40699999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":8.7840000000000007,"5min":12.805999999999999,"15min":12.805999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":3.0249999999999999,"5min":3.0600000000000001,"15min":3.1640000000000001},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.7159999999999993,"5min":12.769,"15min":12.769},"last":0.627},{"interface":"front","average":{"1min":4.1319999999999997,"5min":2.7229999999999999,"15min":2.3159999999999998},"min":{"1min":0.35999999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":23.300999999999998,"5min":23.300999999999998,"15min":23.300999999999998},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":4.1619999999999999,"5min":3.1389999999999998,"15min":2.9329999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":23.347000000000001,"5min":23.347000000000001,"15min":23.347000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.4890000000000001,"5min":2.5049999999999999,"15min":2.7530000000000001},"min":{"1min":0.41999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":12.929,"5min":12.929,"15min":12.929},"last":0.72399999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":3.1960000000000002,"5min":2.8929999999999998,"15min":2.9399999999999999},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":8.7539999999999996,"5min":12.829000000000001,"15min":12.829000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":4.2859999999999996,"5min":2.9849999999999999,"15min":2.589},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":23.376000000000001,"5min":23.376000000000001,"15min":23.376000000000001},"last":0.75800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738733,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903756,"kb_used_data":3460,"kb_used_omap":0,"kb_used_meta":900288,"kb_avail":92841396,"statfs":{"total":95995035648,"available":95069589504,"internally_reserved":0,"allocated":3543040,"data_stored":2076746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":921894912},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:16:49 2024","interfaces":[{"interface":"back","average":{"1min":3.6989999999999998,"5min":3.6509999999999998,"15min":3.5030000000000001},"min":{"1min":0.19500000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":11.32,"5min":14.048999999999999,"15min":14.048999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":2.532,"5min":2.6659999999999999,"15min":2.7559999999999998},"min":{"1min":0.255,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":11.012,"5min":13.787000000000001,"15min":13.787000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:16:36 2024","interfaces":[{"interface":"back","average":{"1min":3.6360000000000001,"5min":2.589,"15min":2.266},"min":{"1min":0.27000000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.509,"5min":19.312000000000001,"15min":19.312000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":3.9769999999999999,"5min":2.9609999999999999,"15min":2.8029999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.20799999999999999},"max":{"1min":11.377000000000001,"5min":22.959,"15min":26.039999999999999},"last":0.371}]},{"osd":3,"last update":"Mon Feb 12 17:16:49 2024","interfaces":[{"interface":"back","average":{"1min":4.5229999999999997,"5min":3.9089999999999998,"15min":3.3889999999999998},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.186},"max":{"1min":13.417,"5min":61.177,"15min":87.311000000000007},"last":0.35099999999999998},{"interface":"front","average":{"1min":6.7919999999999998,"5min":3.911,"15min":3.5550000000000002},"min":{"1min":0.433,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":21.504999999999999,"5min":59.149999999999999,"15min":90.991},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:16:05 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.5529999999999999,"15min":1.476},"min":{"1min":0.28799999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":7.5830000000000002,"5min":15.141999999999999,"15min":15.141999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.641,"15min":2.0310000000000001},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":6.7220000000000004,"5min":13.503,"15min":26.082999999999998},"last":0.625}]},{"osd":5,"last update":"Mon Feb 12 17:16:42 2024","interfaces":[{"interface":"back","average":{"1min":2.8730000000000002,"5min":2.0289999999999999,"15min":2.2829999999999999},"min":{"1min":0.38200000000000001,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":13.981999999999999,"5min":15.090999999999999,"15min":50.404000000000003},"last":0.68300000000000005},{"interface":"front","average":{"1min":2.4129999999999998,"5min":1.819,"15min":2.0750000000000002},"min":{"1min":0.31900000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.6070000000000002,"5min":15.164,"15min":49.979999999999997},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:16:05 2024","interfaces":[{"interface":"back","average":{"1min":3.0169999999999999,"5min":2.5600000000000001,"15min":2.379},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.8369999999999997,"5min":48.780999999999999,"15min":48.780999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":3.3239999999999998,"5min":2.827,"15min":2.3370000000000002},"min":{"1min":0.38600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":10.390000000000001,"5min":49.067999999999998,"15min":49.067999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:16:36 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.0289999999999999,"15min":2.056},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.6299999999999999,"5min":15.183999999999999,"15min":19.373000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":2.9279999999999999,"5min":2.25,"15min":2.4159999999999999},"min":{"1min":0.36799999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":11.4,"5min":13.542,"15min":26.135000000000002},"last":0.59999999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411682,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501632,"kb_used_data":3448,"kb_used_omap":0,"kb_used_meta":1498176,"kb_avail":92243520,"statfs":{"total":95995035648,"available":94457364480,"internally_reserved":0,"allocated":3530752,"data_stored":2077115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1534132224},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":2.327,"5min":1.929,"15min":1.966},"min":{"1min":0.32500000000000001,"5min":0.17899999999999999,"15min":0.17000000000000001},"max":{"1min":7.1749999999999998,"5min":11.663,"15min":36.100999999999999},"last":0.377},{"interface":"front","average":{"1min":3.9209999999999998,"5min":2.5609999999999999,"15min":2.3860000000000001},"min":{"1min":0.307,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.271000000000001,"5min":16.350000000000001,"15min":36.299999999999997},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:16:50 2024","interfaces":[{"interface":"back","average":{"1min":2.3999999999999999,"5min":3.399,"15min":3.8100000000000001},"min":{"1min":0.22900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":10.9,"5min":11.612,"15min":11.612},"last":0.29799999999999999},{"interface":"front","average":{"1min":4.0300000000000002,"5min":5.2539999999999996,"15min":5.6799999999999997},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":17.939,"5min":22.52,"15min":22.52},"last":0.33500000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:16:00 2024","interfaces":[{"interface":"back","average":{"1min":5.5259999999999998,"5min":3.6930000000000001,"15min":2.9220000000000002},"min":{"1min":0.29099999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":30.303000000000001,"5min":30.303000000000001,"15min":40.759999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":5.9139999999999997,"5min":3.262,"15min":2.5070000000000001},"min":{"1min":0.221,"5min":0.22,"15min":0.22},"max":{"1min":30.120999999999999,"5min":30.120999999999999,"15min":30.120999999999999},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:16:24 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.754,"15min":1.8759999999999999},"min":{"1min":0.42599999999999999,"5min":0.247,"15min":0.245},"max":{"1min":5.851,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.5},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.77,"15min":1.9370000000000001},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.0590000000000002,"5min":11.744,"15min":20.149000000000001},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.748,"15min":2.0030000000000001},"min":{"1min":0.46000000000000002,"5min":0.26000000000000001,"15min":0.217},"max":{"1min":7.1310000000000002,"5min":11.581,"15min":32.57},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.2309999999999999,"5min":1.871,"15min":2.1070000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.255},"max":{"1min":6.2030000000000003,"5min":10.787000000000001,"15min":31.905999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Feb 12 17:16:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.7829999999999999,"15min":1.8300000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.0919999999999996,"5min":10.714,"15min":20.966999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":2.5880000000000001,"5min":2.1949999999999998,"15min":1.8580000000000001},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.6130000000000004,"5min":11.834,"15min":20.193000000000001},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.871,"15min":2.4609999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.7119999999999997,"5min":11.789,"15min":45.468000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":2.3210000000000002,"5min":2.0169999999999999,"15min":2.4569999999999999},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":6.1150000000000002,"5min":10.762,"15min":46.656999999999996},"last":0.63700000000000001}]}]},{"osd":3,"up_from":26,"seq":111669150045,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896760,"kb_used_data":3504,"kb_used_omap":0,"kb_used_meta":893248,"kb_avail":92848392,"statfs":{"total":95995035648,"available":95076753408,"internally_reserved":0,"allocated":3588096,"data_stored":2085074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914685952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:57 2024","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.3860000000000001,"15min":3.0259999999999998},"min":{"1min":0.20999999999999999,"5min":0.189,"15min":0.17100000000000001},"max":{"1min":22.16,"5min":22.16,"15min":58.451999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":5.6280000000000001,"5min":3.6150000000000002,"15min":3.3849999999999998},"min":{"1min":0.34699999999999998,"5min":0.217,"15min":0.19500000000000001},"max":{"1min":20.681999999999999,"5min":42.984999999999999,"15min":72.688000000000002},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:16:51 2024","interfaces":[{"interface":"back","average":{"1min":3.8279999999999998,"5min":5.0460000000000003,"15min":4.6689999999999996},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":16.010000000000002,"5min":22.524000000000001,"15min":22.524000000000001},"last":0.311},{"interface":"front","average":{"1min":5.4749999999999996,"5min":5.4930000000000003,"15min":5.4669999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.710000000000001,"5min":20.710000000000001,"15min":20.710000000000001},"last":0.39400000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:16:57 2024","interfaces":[{"interface":"back","average":{"1min":5.3280000000000003,"5min":3.8319999999999999,"15min":2.8900000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":48.127000000000002,"5min":48.127000000000002,"15min":48.127000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":5.9539999999999997,"5min":3.9849999999999999,"15min":3.3330000000000002},"min":{"1min":0.34599999999999997,"5min":0.22,"15min":0.22},"max":{"1min":40.600999999999999,"5min":40.600999999999999,"15min":46.408000000000001},"last":0.29899999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:16:15 2024","interfaces":[{"interface":"back","average":{"1min":5.2839999999999998,"5min":3.746,"15min":2.6720000000000002},"min":{"1min":0.54600000000000004,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":20.478000000000002,"5min":72.659000000000006,"15min":72.659000000000006},"last":0.441},{"interface":"front","average":{"1min":4.5279999999999996,"5min":3.0979999999999999,"15min":2.1099999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23999999999999999},"max":{"1min":22.440999999999999,"5min":40.930999999999997,"15min":40.930999999999997},"last":0.35699999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:16:34 2024","interfaces":[{"interface":"back","average":{"1min":4.218,"5min":3.1120000000000001,"15min":2.7080000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.253},"max":{"1min":12.329000000000001,"5min":17.306999999999999,"15min":17.940999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":4.3310000000000004,"5min":3.665,"15min":2.9279999999999999},"min":{"1min":0.52000000000000002,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":15.522,"5min":72.620000000000005,"15min":72.620000000000005},"last":0.503}]},{"osd":6,"last update":"Mon Feb 12 17:16:57 2024","interfaces":[{"interface":"back","average":{"1min":2.1400000000000001,"5min":2.702,"15min":2.226},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":10.821,"5min":22.489999999999998,"15min":40.889000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":2.7999999999999998,"5min":3.089,"15min":2.5219999999999998},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":12.414,"5min":15.113,"15min":18.265000000000001},"last":0.47499999999999998}]},{"osd":7,"last update":"Mon Feb 12 17:16:15 2024","interfaces":[{"interface":"back","average":{"1min":5.0890000000000004,"5min":3.5609999999999999,"15min":3.0880000000000001},"min":{"1min":0.53500000000000003,"5min":0.375,"15min":0.315},"max":{"1min":20.515999999999998,"5min":72.712999999999994,"15min":72.712999999999994},"last":0.54800000000000004},{"interface":"front","average":{"1min":5.6120000000000001,"5min":3.0259999999999998,"15min":2.9089999999999998},"min":{"1min":0.44900000000000001,"5min":0.38600000000000001,"15min":0.28299999999999997},"max":{"1min":15.169,"5min":15.169,"15min":38.948},"last":0.51600000000000001}]}]},{"osd":4,"up_from":0,"seq":137438953818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757590,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1678528,"kb_used_data":193208,"kb_used_omap":0,"kb_used_meta":1485312,"kb_avail":92066624,"statfs":{"total":95995035648,"available":94276222976,"internally_reserved":0,"allocated":197844992,"data_stored":196390623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1520959488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":3.0950000000000002,"5min":1.9830000000000001,"15min":1.623},"min":{"1min":0.46500000000000002,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":6.8399999999999999,"5min":8.0350000000000001,"15min":10.971},"last":0.625},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.915,"15min":1.8029999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20100000000000001},"max":{"1min":12.061,"5min":12.061,"15min":15.368},"last":0.57899999999999996}]},{"osd":1,"last update":"Mon Feb 12 17:16:57 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.1200000000000001,"15min":1.905},"min":{"1min":0.312,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.0540000000000003,"5min":9.484,"15min":9.484},"last":0.67600000000000005},{"interface":"front","average":{"1min":3.552,"5min":2.9430000000000001,"15min":3.024},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":18.550000000000001,"5min":18.550000000000001,"15min":18.550000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":4.0110000000000001,"5min":2.871,"15min":2.0800000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23200000000000001},"max":{"1min":13.978,"5min":57.390999999999998,"15min":57.390999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":2.2559999999999998,"5min":2.7229999999999999,"15min":1.9870000000000001},"min":{"1min":0.53300000000000003,"5min":0.214,"15min":0.214},"max":{"1min":6.0259999999999998,"5min":57.134999999999998,"15min":57.134999999999998},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":3.9390000000000001,"5min":2.964,"15min":2.7389999999999999},"min":{"1min":0.36499999999999999,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":11.871,"5min":57.170999999999999,"15min":57.170999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.984,"15min":2.6259999999999999},"min":{"1min":0.33600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":17.760999999999999,"5min":57.484999999999999,"15min":57.484999999999999},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.385},"min":{"1min":0.26000000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.3660000000000001,"5min":9.0429999999999993,"15min":12.656000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6459999999999999,"15min":1.375},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.254},"max":{"1min":5.1390000000000002,"5min":10.619999999999999,"15min":11.068},"last":0.48599999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:16:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.5030000000000001,"5min":14.675000000000001,"15min":14.675000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.651,"5min":1.802,"15min":1.5629999999999999},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":5.2720000000000002,"5min":12.864000000000001,"15min":13.916},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.6499999999999999,"15min":1.599},"min":{"1min":0.31900000000000001,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":6.0170000000000003,"5min":14.337999999999999,"15min":14.337999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.6990000000000001,"15min":1.7629999999999999},"min":{"1min":0.316,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":3.9369999999999998,"5min":14.637,"15min":20.274000000000001},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:17:06.522 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T17:17:06.522 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T17:17:06.522 INFO:tasks.thrashosds.thrasher:Shrinking pool device_health_metrics 2024-02-12T17:17:06.522 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool device_health_metrics by 10 min 8 2024-02-12T17:17:06.523 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:17:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:05.455330+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:17:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:05.458232+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:17:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:05.752961+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:17:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:05.755875+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:17:07.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:05.863987+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-02-12T17:17:07.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:05.455330+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:17:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:05.458232+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:17:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:05.752961+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:17:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:05.755875+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:17:07.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:05.863987+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-02-12T17:17:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:05.455330+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:17:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:05.458232+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:17:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:05.752961+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:17:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:05.755875+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:17:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:05.863987+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-02-12T17:17:07.849 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:07.849 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:07.861 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:07.918 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:17:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:06.894500+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-02-12T17:17:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:06.898786+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 starting backfill to osd.3 from (0'0,0'0] MAX to 706'26 2024-02-12T17:17:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:06.985112+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.33:0/921123343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:06.985424+0000 mon.a (mon.0) 2300 : audit [INF] from='client.36278 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.157838+0000 mgr.y (mgr.24481) 1035 : cluster [DBG] pgmap v1463: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 957 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 82 B/s, 1 objects/s recovering 2024-02-12T17:17:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.344209+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:17:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.347675+0000 osd.3 (osd.3) 251 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:17:08.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:06.894500+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-02-12T17:17:08.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:06.898786+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 starting backfill to osd.3 from (0'0,0'0] MAX to 706'26 2024-02-12T17:17:08.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:06.985112+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.33:0/921123343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:08.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:06.985424+0000 mon.a (mon.0) 2300 : audit [INF] from='client.36278 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:08.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.157838+0000 mgr.y (mgr.24481) 1035 : cluster [DBG] pgmap v1463: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 957 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 82 B/s, 1 objects/s recovering 2024-02-12T17:17:08.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.344209+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:17:08.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.347675+0000 osd.3 (osd.3) 251 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:17:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:06.894500+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-02-12T17:17:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:06.898786+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 starting backfill to osd.3 from (0'0,0'0] MAX to 706'26 2024-02-12T17:17:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:06.985112+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.33:0/921123343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:06.985424+0000 mon.a (mon.0) 2300 : audit [INF] from='client.36278 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.157838+0000 mgr.y (mgr.24481) 1035 : cluster [DBG] pgmap v1463: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 957 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 82 B/s, 1 objects/s recovering 2024-02-12T17:17:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.344209+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:17:08.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.347675+0000 osd.3 (osd.3) 251 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:17:09.168 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.764976+0000 osd.6 (osd.6) 192 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:17:09.168 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.767083+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.773126+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.774074+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.903924+0000 mon.a (mon.0) 2301 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:07.911148+0000 mon.a (mon.0) 2302 : audit [INF] from='client.36278 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:07.911203+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.764976+0000 osd.6 (osd.6) 192 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.767083+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.773126+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.774074+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.903924+0000 mon.a (mon.0) 2301 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:07.911148+0000 mon.a (mon.0) 2302 : audit [INF] from='client.36278 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:17:09.169 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:07.911203+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-02-12T17:17:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.764976+0000 osd.6 (osd.6) 192 : cluster [DBG] 4.5 scrub starts 2024-02-12T17:17:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.767083+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.5 scrub ok 2024-02-12T17:17:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.773126+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:17:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.774074+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:17:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.903924+0000 mon.a (mon.0) 2301 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T17:17:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:07.911148+0000 mon.a (mon.0) 2302 : audit [INF] from='client.36278 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:17:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:07.911203+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-02-12T17:17:09.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T17:17:10.061 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:10.174 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:08.421421+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:17:10.174 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:08.424780+0000 osd.7 (osd.7) 326 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:17:10.174 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:08.916299+0000 mon.a (mon.0) 2304 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-02-12T17:17:10.174 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:09.158518+0000 mgr.y (mgr.24481) 1036 : cluster [DBG] pgmap v1466: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 2/576 objects misplaced (0.347%); 109 B/s, 2 objects/s recovering 2024-02-12T17:17:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:08.421421+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:17:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:08.424780+0000 osd.7 (osd.7) 326 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:17:10.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:08.916299+0000 mon.a (mon.0) 2304 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-02-12T17:17:10.175 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:09.158518+0000 mgr.y (mgr.24481) 1036 : cluster [DBG] pgmap v1466: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 2/576 objects misplaced (0.347%); 109 B/s, 2 objects/s recovering 2024-02-12T17:17:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:08.421421+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:17:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:08.424780+0000 osd.7 (osd.7) 326 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:17:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:08.916299+0000 mon.a (mon.0) 2304 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-02-12T17:17:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:09.158518+0000 mgr.y (mgr.24481) 1036 : cluster [DBG] pgmap v1466: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 2/576 objects misplaced (0.347%); 109 B/s, 2 objects/s recovering 2024-02-12T17:17:10.944 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T17:17:11.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:17:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:10] "GET /metrics HTTP/1.1" 200 174446 "" "Prometheus/2.33.4" 2024-02-12T17:17:11.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:10.772189+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:17:11.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:10.775078+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:17:11.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:10.772189+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:17:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:10.775078+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:17:11.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:10.772189+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.10 scrub starts 2024-02-12T17:17:11.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:10.775078+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.10 scrub ok 2024-02-12T17:17:12.490 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:17:12.492 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:17:12.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:11.158968+0000 mgr.y (mgr.24481) 1037 : cluster [DBG] pgmap v1467: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 966 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 124 B/s, 3 objects/s recovering 2024-02-12T17:17:12.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:11.813529+0000 mon.a (mon.0) 2305 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-02-12T17:17:12.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:11.158968+0000 mgr.y (mgr.24481) 1037 : cluster [DBG] pgmap v1467: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 966 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 124 B/s, 3 objects/s recovering 2024-02-12T17:17:12.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:11.813529+0000 mon.a (mon.0) 2305 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-02-12T17:17:12.512 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:11.158968+0000 mgr.y (mgr.24481) 1037 : cluster [DBG] pgmap v1467: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 966 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 124 B/s, 3 objects/s recovering 2024-02-12T17:17:12.512 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:11.813529+0000 mon.a (mon.0) 2305 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-02-12T17:17:12.551 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:12.551 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:12.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:17:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:17:13.166 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T17:17:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:11.457878+0000 osd.0 (osd.0) 248 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:17:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:11.461617+0000 osd.0 (osd.0) 249 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:17:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:12.476638+0000 mgr.y (mgr.24481) 1038 : audit [DBG] from='client.36284 v1:172.21.15.33:0/2519272628' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:17:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:12.833446+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:17:13.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:12.872554+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:17:13.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:11.457878+0000 osd.0 (osd.0) 248 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:17:13.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:11.461617+0000 osd.0 (osd.0) 249 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:17:13.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:12.476638+0000 mgr.y (mgr.24481) 1038 : audit [DBG] from='client.36284 v1:172.21.15.33:0/2519272628' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:17:13.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:12.833446+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:17:13.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:12.872554+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:17:13.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:11.457878+0000 osd.0 (osd.0) 248 : cluster [DBG] 4.3 scrub starts 2024-02-12T17:17:13.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:11.461617+0000 osd.0 (osd.0) 249 : cluster [DBG] 4.3 scrub ok 2024-02-12T17:17:13.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:12.476638+0000 mgr.y (mgr.24481) 1038 : audit [DBG] from='client.36284 v1:172.21.15.33:0/2519272628' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:17:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:12.833446+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:17:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:12.872554+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:17:13.974 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1468,"stamp":"2024-02-12T17:17:11.814757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":1948,"num_read_kb":1700,"num_write":1000,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":863,"num_bytes_recovered":24080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1165,"ondisk_log_size":1165,"up":306,"acting":306,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":310,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7849020,"kb_used_data":350764,"kb_used_omap":3,"kb_used_meta":7496060,"kb_avail":648367044,"statfs":{"total":671965249536,"available":663927853056,"internally_reserved":0,"allocated":359182336,"data_stored":349772623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":7675965501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.917277"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":441,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858561+0000","last_change":"2024-02-12T17:14:10.348720+0000","last_active":"2024-02-12T17:17:04.858561+0000","last_peered":"2024-02-12T17:17:04.858561+0000","last_clean":"2024-02-12T17:17:04.858561+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:17:04.858561+0000","last_undegraded":"2024-02-12T17:17:04.858561+0000","last_fullsized":"2024-02-12T17:17:04.858561+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'27","last_scrub_stamp":"2024-02-12T17:07:35.884885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:35.884885+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":226,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.895736+0000","last_change":"2024-02-12T17:13:46.694822+0000","last_active":"2024-02-12T17:17:06.895736+0000","last_peered":"2024-02-12T17:17:06.895736+0000","last_clean":"2024-02-12T17:17:06.895736+0000","last_became_active":"2024-02-12T17:13:46.694448+0000","last_became_peered":"2024-02-12T17:13:46.694448+0000","last_unstale":"2024-02-12T17:17:06.895736+0000","last_undegraded":"2024-02-12T17:17:06.895736+0000","last_fullsized":"2024-02-12T17:17:06.895736+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:12:55.862345+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:12:55.862345+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":468,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.860020+0000","last_change":"2024-02-12T17:13:46.787428+0000","last_active":"2024-02-12T17:17:04.860020+0000","last_peered":"2024-02-12T17:17:04.860020+0000","last_clean":"2024-02-12T17:17:04.860020+0000","last_became_active":"2024-02-12T17:13:46.786996+0000","last_became_peered":"2024-02-12T17:13:46.786996+0000","last_unstale":"2024-02-12T17:17:04.860020+0000","last_undegraded":"2024-02-12T17:17:04.860020+0000","last_fullsized":"2024-02-12T17:17:04.860020+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:13:12.016227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:12.016227+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":22,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898109+0000","last_change":"2024-02-12T17:17:06.561125+0000","last_active":"2024-02-12T17:17:06.898109+0000","last_peered":"2024-02-12T17:17:06.898109+0000","last_clean":"2024-02-12T17:17:06.898109+0000","last_became_active":"2024-02-12T17:17:05.871721+0000","last_became_peered":"2024-02-12T17:17:05.871721+0000","last_unstale":"2024-02-12T17:17:06.898109+0000","last_undegraded":"2024-02-12T17:17:06.898109+0000","last_fullsized":"2024-02-12T17:17:06.898109+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:54.052006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.052006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"526'24","reported_seq":532,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.385064+0000","last_change":"2024-02-12T17:17:06.385064+0000","last_active":"2024-02-12T17:17:06.385064+0000","last_peered":"2024-02-12T17:17:06.385064+0000","last_clean":"2024-02-12T17:17:06.385064+0000","last_became_active":"2024-02-12T17:17:05.870151+0000","last_became_peered":"2024-02-12T17:17:05.870151+0000","last_unstale":"2024-02-12T17:17:06.385064+0000","last_undegraded":"2024-02-12T17:17:06.385064+0000","last_fullsized":"2024-02-12T17:17:06.385064+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:05.116071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.116071+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":184,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910325+0000","last_change":"2024-02-12T17:17:00.753460+0000","last_active":"2024-02-12T17:17:07.910325+0000","last_peered":"2024-02-12T17:17:07.910325+0000","last_clean":"2024-02-12T17:17:07.910325+0000","last_became_active":"2024-02-12T17:13:46.606367+0000","last_became_peered":"2024-02-12T17:13:46.606367+0000","last_unstale":"2024-02-12T17:17:07.910325+0000","last_undegraded":"2024-02-12T17:17:07.910325+0000","last_fullsized":"2024-02-12T17:17:07.910325+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'39","last_scrub_stamp":"2024-02-12T17:17:00.753381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:17:00.753381+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"526'27","reported_seq":258,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.896425+0000","last_change":"2024-02-12T17:16:59.774727+0000","last_active":"2024-02-12T17:17:06.896425+0000","last_peered":"2024-02-12T17:17:06.896425+0000","last_clean":"2024-02-12T17:17:06.896425+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:17:06.896425+0000","last_undegraded":"2024-02-12T17:17:06.896425+0000","last_fullsized":"2024-02-12T17:17:06.896425+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:16:59.774659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:16:59.774659+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"526'30","reported_seq":94,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.916080+0000","last_change":"2024-02-12T17:17:02.421133+0000","last_active":"2024-02-12T17:17:07.916080+0000","last_peered":"2024-02-12T17:17:07.916080+0000","last_clean":"2024-02-12T17:17:07.916080+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:17:07.916080+0000","last_undegraded":"2024-02-12T17:17:07.916080+0000","last_fullsized":"2024-02-12T17:17:07.916080+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:17:02.421057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:17:02.421057+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":454,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939613+0000","last_change":"2024-02-12T17:16:46.727669+0000","last_active":"2024-02-12T17:17:06.939613+0000","last_peered":"2024-02-12T17:17:06.939613+0000","last_clean":"2024-02-12T17:17:06.939613+0000","last_became_active":"2024-02-12T17:16:46.727369+0000","last_became_peered":"2024-02-12T17:16:46.727369+0000","last_unstale":"2024-02-12T17:17:06.939613+0000","last_undegraded":"2024-02-12T17:17:06.939613+0000","last_fullsized":"2024-02-12T17:17:06.939613+0000","mapping_epoch":705,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":5,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":414,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.860102+0000","last_change":"2024-02-12T17:17:01.422130+0000","last_active":"2024-02-12T17:17:04.860102+0000","last_peered":"2024-02-12T17:17:04.860102+0000","last_clean":"2024-02-12T17:17:04.860102+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:17:04.860102+0000","last_undegraded":"2024-02-12T17:17:04.860102+0000","last_fullsized":"2024-02-12T17:17:04.860102+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:01.422046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:17:01.422046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":432,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858770+0000","last_change":"2024-02-12T17:12:12.141297+0000","last_active":"2024-02-12T17:17:04.858770+0000","last_peered":"2024-02-12T17:17:04.858770+0000","last_clean":"2024-02-12T17:17:04.858770+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:17:04.858770+0000","last_undegraded":"2024-02-12T17:17:04.858770+0000","last_fullsized":"2024-02-12T17:17:04.858770+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.141204+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":263,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.895970+0000","last_change":"2024-02-12T17:14:08.743209+0000","last_active":"2024-02-12T17:17:06.895970+0000","last_peered":"2024-02-12T17:17:06.895970+0000","last_clean":"2024-02-12T17:17:06.895970+0000","last_became_active":"2024-02-12T17:14:08.742886+0000","last_became_peered":"2024-02-12T17:14:08.742886+0000","last_unstale":"2024-02-12T17:17:06.895970+0000","last_undegraded":"2024-02-12T17:17:06.895970+0000","last_fullsized":"2024-02-12T17:17:06.895970+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.783527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.783527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":717,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:08.424863+0000","last_change":"2024-02-12T17:17:08.424863+0000","last_active":"2024-02-12T17:17:08.424863+0000","last_peered":"2024-02-12T17:17:08.424863+0000","last_clean":"2024-02-12T17:17:08.424863+0000","last_became_active":"2024-02-12T17:13:21.591246+0000","last_became_peered":"2024-02-12T17:13:21.591246+0000","last_unstale":"2024-02-12T17:17:08.424863+0000","last_undegraded":"2024-02-12T17:17:08.424863+0000","last_fullsized":"2024-02-12T17:17:08.424863+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:08.424783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:17:08.424783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":454,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858629+0000","last_change":"2024-02-12T17:13:46.558817+0000","last_active":"2024-02-12T17:17:04.858629+0000","last_peered":"2024-02-12T17:17:04.858629+0000","last_clean":"2024-02-12T17:17:04.858629+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:17:04.858629+0000","last_undegraded":"2024-02-12T17:17:04.858629+0000","last_fullsized":"2024-02-12T17:17:04.858629+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'10","last_scrub_stamp":"2024-02-12T17:06:35.412989+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:06:35.412989+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":111,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.911850+0000","last_change":"2024-02-12T17:13:46.576676+0000","last_active":"2024-02-12T17:17:07.911850+0000","last_peered":"2024-02-12T17:17:07.911850+0000","last_clean":"2024-02-12T17:17:07.911850+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:17:07.911850+0000","last_undegraded":"2024-02-12T17:17:07.911850+0000","last_fullsized":"2024-02-12T17:17:07.911850+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:03:15.810401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:03:15.810401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":391,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858667+0000","last_change":"2024-02-12T17:14:08.763317+0000","last_active":"2024-02-12T17:17:04.858667+0000","last_peered":"2024-02-12T17:17:04.858667+0000","last_clean":"2024-02-12T17:17:04.858667+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:17:04.858667+0000","last_undegraded":"2024-02-12T17:17:04.858667+0000","last_fullsized":"2024-02-12T17:17:04.858667+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:36.839346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:07:36.839346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":156,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.911014+0000","last_change":"2024-02-12T17:17:07.774145+0000","last_active":"2024-02-12T17:17:07.911014+0000","last_peered":"2024-02-12T17:17:07.911014+0000","last_clean":"2024-02-12T17:17:07.911014+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:17:07.911014+0000","last_undegraded":"2024-02-12T17:17:07.911014+0000","last_fullsized":"2024-02-12T17:17:07.911014+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:07.774077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:17:07.774077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":236,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.896042+0000","last_change":"2024-02-12T17:17:04.765622+0000","last_active":"2024-02-12T17:17:06.896042+0000","last_peered":"2024-02-12T17:17:06.896042+0000","last_clean":"2024-02-12T17:17:06.896042+0000","last_became_active":"2024-02-12T17:13:46.783737+0000","last_became_peered":"2024-02-12T17:13:46.783737+0000","last_unstale":"2024-02-12T17:17:06.896042+0000","last_undegraded":"2024-02-12T17:17:06.896042+0000","last_fullsized":"2024-02-12T17:17:06.896042+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:17:04.765513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:17:04.765513+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":781,"reported_epoch":715,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:17:04.858973+0000","last_change":"2024-02-12T17:16:45.076370+0000","last_active":"2024-02-12T17:17:04.858973+0000","last_peered":"2024-02-12T17:17:04.858973+0000","last_clean":"2024-02-12T17:13:44.507959+0000","last_became_active":"2024-02-12T17:14:08.758564+0000","last_became_peered":"2024-02-12T17:14:08.758564+0000","last_unstale":"2024-02-12T17:17:04.858973+0000","last_undegraded":"2024-02-12T17:17:04.858973+0000","last_fullsized":"2024-02-12T17:17:04.858973+0000","mapping_epoch":651,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"299'10","last_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:00:51.955721+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":82,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.916616+0000","last_change":"2024-02-12T17:16:58.559051+0000","last_active":"2024-02-12T17:17:07.916616+0000","last_peered":"2024-02-12T17:17:07.916616+0000","last_clean":"2024-02-12T17:17:07.916616+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:17:07.916616+0000","last_undegraded":"2024-02-12T17:17:07.916616+0000","last_fullsized":"2024-02-12T17:17:07.916616+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:16:58.558963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:16:58.558963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":226,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.896183+0000","last_change":"2024-02-12T17:17:01.779360+0000","last_active":"2024-02-12T17:17:06.896183+0000","last_peered":"2024-02-12T17:17:06.896183+0000","last_clean":"2024-02-12T17:17:06.896183+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:17:06.896183+0000","last_undegraded":"2024-02-12T17:17:06.896183+0000","last_fullsized":"2024-02-12T17:17:06.896183+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:01.779290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:17:01.779290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":188,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.911777+0000","last_change":"2024-02-12T17:17:04.803752+0000","last_active":"2024-02-12T17:17:07.911777+0000","last_peered":"2024-02-12T17:17:07.911777+0000","last_clean":"2024-02-12T17:17:07.911777+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:17:07.911777+0000","last_undegraded":"2024-02-12T17:17:07.911777+0000","last_fullsized":"2024-02-12T17:17:07.911777+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:17:04.803684+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:17:04.803684+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":823,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.860601+0000","last_change":"2024-02-12T17:12:58.020311+0000","last_active":"2024-02-12T17:17:04.860601+0000","last_peered":"2024-02-12T17:17:04.860601+0000","last_clean":"2024-02-12T17:17:04.860601+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:17:04.860601+0000","last_undegraded":"2024-02-12T17:17:04.860601+0000","last_fullsized":"2024-02-12T17:17:04.860601+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:12:58.018712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.018712+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":77,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.917846+0000","last_change":"2024-02-12T17:14:08.872636+0000","last_active":"2024-02-12T17:17:07.917846+0000","last_peered":"2024-02-12T17:17:07.917846+0000","last_clean":"2024-02-12T17:17:07.917846+0000","last_became_active":"2024-02-12T17:14:08.872123+0000","last_became_peered":"2024-02-12T17:14:08.872123+0000","last_unstale":"2024-02-12T17:17:07.917846+0000","last_undegraded":"2024-02-12T17:17:07.917846+0000","last_fullsized":"2024-02-12T17:17:07.917846+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:06.904354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:06.904354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":808,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898554+0000","last_change":"2024-02-12T17:12:12.838640+0000","last_active":"2024-02-12T17:17:06.898554+0000","last_peered":"2024-02-12T17:17:06.898554+0000","last_clean":"2024-02-12T17:17:06.898554+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:17:06.898554+0000","last_undegraded":"2024-02-12T17:17:06.898554+0000","last_fullsized":"2024-02-12T17:17:06.898554+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:12.838563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:12.838563+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"717'28","reported_seq":857,"reported_epoch":718,"state":"peering","last_fresh":"2024-02-12T17:17:07.912365+0000","last_change":"2024-02-12T17:17:07.912704+0000","last_active":"2024-02-12T17:17:07.912365+0000","last_peered":"2024-02-12T17:17:07.704436+0000","last_clean":"2024-02-12T17:17:07.704436+0000","last_became_active":"2024-02-12T17:17:06.906802+0000","last_became_peered":"2024-02-12T17:17:06.906802+0000","last_unstale":"2024-02-12T17:17:07.912365+0000","last_undegraded":"2024-02-12T17:17:07.912365+0000","last_fullsized":"2024-02-12T17:17:07.912365+0000","mapping_epoch":718,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:02:35.760747+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":712,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898229+0000","last_change":"2024-02-12T17:12:54.656523+0000","last_active":"2024-02-12T17:17:06.898229+0000","last_peered":"2024-02-12T17:17:06.898229+0000","last_clean":"2024-02-12T17:17:06.898229+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:17:06.898229+0000","last_undegraded":"2024-02-12T17:17:06.898229+0000","last_fullsized":"2024-02-12T17:17:06.898229+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:54.656456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:54.656456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":450,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:05.352497+0000","last_change":"2024-02-12T17:17:05.352472+0000","last_active":"2024-02-12T17:17:05.352497+0000","last_peered":"2024-02-12T17:17:05.352497+0000","last_clean":"2024-02-12T17:17:05.352497+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:17:05.352497+0000","last_undegraded":"2024-02-12T17:17:05.352497+0000","last_fullsized":"2024-02-12T17:17:05.352497+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:17:05.352395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:17:05.352395+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":734,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939652+0000","last_change":"2024-02-12T17:13:21.591810+0000","last_active":"2024-02-12T17:17:06.939652+0000","last_peered":"2024-02-12T17:17:06.939652+0000","last_clean":"2024-02-12T17:17:06.939652+0000","last_became_active":"2024-02-12T17:13:21.591502+0000","last_became_peered":"2024-02-12T17:13:21.591502+0000","last_unstale":"2024-02-12T17:17:06.939652+0000","last_undegraded":"2024-02-12T17:17:06.939652+0000","last_fullsized":"2024-02-12T17:17:06.939652+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.882066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.882066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":255,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.895899+0000","last_change":"2024-02-12T17:13:46.651066+0000","last_active":"2024-02-12T17:17:06.895899+0000","last_peered":"2024-02-12T17:17:06.895899+0000","last_clean":"2024-02-12T17:17:06.895899+0000","last_became_active":"2024-02-12T17:13:46.650776+0000","last_became_peered":"2024-02-12T17:13:46.650776+0000","last_unstale":"2024-02-12T17:17:06.895899+0000","last_undegraded":"2024-02-12T17:17:06.895899+0000","last_fullsized":"2024-02-12T17:17:06.895899+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:50.786900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.786900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":434,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.859534+0000","last_change":"2024-02-12T17:13:46.822111+0000","last_active":"2024-02-12T17:17:04.859534+0000","last_peered":"2024-02-12T17:17:04.859534+0000","last_clean":"2024-02-12T17:17:04.859534+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:17:04.859534+0000","last_undegraded":"2024-02-12T17:17:04.859534+0000","last_fullsized":"2024-02-12T17:17:04.859534+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:06:44.468722+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":346,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939674+0000","last_change":"2024-02-12T17:14:08.745944+0000","last_active":"2024-02-12T17:17:06.939674+0000","last_peered":"2024-02-12T17:17:06.939674+0000","last_clean":"2024-02-12T17:17:06.939674+0000","last_became_active":"2024-02-12T17:14:08.745729+0000","last_became_peered":"2024-02-12T17:14:08.745729+0000","last_unstale":"2024-02-12T17:17:06.939674+0000","last_undegraded":"2024-02-12T17:17:06.939674+0000","last_fullsized":"2024-02-12T17:17:06.939674+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:49.809844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:12:49.809844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":111,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910227+0000","last_change":"2024-02-12T17:13:46.559519+0000","last_active":"2024-02-12T17:17:07.910227+0000","last_peered":"2024-02-12T17:17:07.910227+0000","last_clean":"2024-02-12T17:17:07.910227+0000","last_became_active":"2024-02-12T17:13:46.559095+0000","last_became_peered":"2024-02-12T17:13:46.559095+0000","last_unstale":"2024-02-12T17:17:07.910227+0000","last_undegraded":"2024-02-12T17:17:07.910227+0000","last_fullsized":"2024-02-12T17:17:07.910227+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.571429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:02.744493+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.571429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":111,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.916182+0000","last_change":"2024-02-12T17:17:06.223215+0000","last_active":"2024-02-12T17:17:07.916182+0000","last_peered":"2024-02-12T17:17:07.916182+0000","last_clean":"2024-02-12T17:17:07.916182+0000","last_became_active":"2024-02-12T17:17:05.874272+0000","last_became_peered":"2024-02-12T17:17:05.874272+0000","last_unstale":"2024-02-12T17:17:07.916182+0000","last_undegraded":"2024-02-12T17:17:07.916182+0000","last_fullsized":"2024-02-12T17:17:07.916182+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:16:59.494581+0000","last_deep_scrub":"277'19","last_deep_scrub_stamp":"2024-02-12T17:06:45.502635+0000","last_clean_scrub_stamp":"2024-02-12T17:16:59.494581+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":433,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.859925+0000","last_change":"2024-02-12T17:13:46.787305+0000","last_active":"2024-02-12T17:17:04.859925+0000","last_peered":"2024-02-12T17:17:04.859925+0000","last_clean":"2024-02-12T17:17:04.859925+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:17:04.859925+0000","last_undegraded":"2024-02-12T17:17:04.859925+0000","last_fullsized":"2024-02-12T17:17:04.859925+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:18.845552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:18.845552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":449,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.914809+0000","last_change":"2024-02-12T17:16:46.731039+0000","last_active":"2024-02-12T17:17:07.914809+0000","last_peered":"2024-02-12T17:17:07.914809+0000","last_clean":"2024-02-12T17:17:07.914809+0000","last_became_active":"2024-02-12T17:16:46.730672+0000","last_became_peered":"2024-02-12T17:16:46.730672+0000","last_unstale":"2024-02-12T17:17:07.914809+0000","last_undegraded":"2024-02-12T17:17:07.914809+0000","last_fullsized":"2024-02-12T17:17:07.914809+0000","mapping_epoch":705,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:12:09.882247+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:12:09.882247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":250,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.303601+0000","last_change":"2024-02-12T17:17:06.303601+0000","last_active":"2024-02-12T17:17:06.303601+0000","last_peered":"2024-02-12T17:17:06.303601+0000","last_clean":"2024-02-12T17:17:06.303601+0000","last_became_active":"2024-02-12T17:17:05.872645+0000","last_became_peered":"2024-02-12T17:17:05.872645+0000","last_unstale":"2024-02-12T17:17:06.303601+0000","last_undegraded":"2024-02-12T17:17:06.303601+0000","last_fullsized":"2024-02-12T17:17:06.303601+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:12:51.126056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:51.126056+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":82,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.916705+0000","last_change":"2024-02-12T17:17:05.458342+0000","last_active":"2024-02-12T17:17:07.916705+0000","last_peered":"2024-02-12T17:17:07.916705+0000","last_clean":"2024-02-12T17:17:07.916705+0000","last_became_active":"2024-02-12T17:14:08.769221+0000","last_became_peered":"2024-02-12T17:14:08.769221+0000","last_unstale":"2024-02-12T17:17:07.916705+0000","last_undegraded":"2024-02-12T17:17:07.916705+0000","last_fullsized":"2024-02-12T17:17:07.916705+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:05.458242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.127831+0000","last_clean_scrub_stamp":"2024-02-12T17:17:05.458242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":724,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.917000+0000","last_change":"2024-02-12T17:14:07.897458+0000","last_active":"2024-02-12T17:17:07.917000+0000","last_peered":"2024-02-12T17:17:07.917000+0000","last_clean":"2024-02-12T17:17:07.917000+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:17:07.917000+0000","last_undegraded":"2024-02-12T17:17:07.917000+0000","last_fullsized":"2024-02-12T17:17:07.917000+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:10.780381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:10.780381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910428+0000","last_change":"2024-02-12T17:13:46.791505+0000","last_active":"2024-02-12T17:17:07.910428+0000","last_peered":"2024-02-12T17:17:07.910428+0000","last_clean":"2024-02-12T17:17:07.910428+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:17:07.910428+0000","last_undegraded":"2024-02-12T17:17:07.910428+0000","last_fullsized":"2024-02-12T17:17:07.910428+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.487301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.487301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":191,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.911893+0000","last_change":"2024-02-12T17:14:09.246132+0000","last_active":"2024-02-12T17:17:07.911893+0000","last_peered":"2024-02-12T17:17:07.911893+0000","last_clean":"2024-02-12T17:17:07.911893+0000","last_became_active":"2024-02-12T17:14:08.752149+0000","last_became_peered":"2024-02-12T17:14:08.752149+0000","last_unstale":"2024-02-12T17:17:07.911893+0000","last_undegraded":"2024-02-12T17:17:07.911893+0000","last_fullsized":"2024-02-12T17:17:07.911893+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:13:01.543047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.543047+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"706'24","reported_seq":566,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.917656+0000","last_change":"2024-02-12T17:16:46.730984+0000","last_active":"2024-02-12T17:17:07.917656+0000","last_peered":"2024-02-12T17:17:07.917656+0000","last_clean":"2024-02-12T17:17:07.917656+0000","last_became_active":"2024-02-12T17:16:46.730723+0000","last_became_peered":"2024-02-12T17:16:46.730723+0000","last_unstale":"2024-02-12T17:17:07.917656+0000","last_undegraded":"2024-02-12T17:17:07.917656+0000","last_fullsized":"2024-02-12T17:17:07.917656+0000","mapping_epoch":705,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'21","last_scrub_stamp":"2024-02-12T17:12:57.741798+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:12:57.741798+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":436,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.859695+0000","last_change":"2024-02-12T17:13:21.603543+0000","last_active":"2024-02-12T17:17:04.859695+0000","last_peered":"2024-02-12T17:17:04.859695+0000","last_clean":"2024-02-12T17:17:04.859695+0000","last_became_active":"2024-02-12T17:13:21.603226+0000","last_became_peered":"2024-02-12T17:13:21.603226+0000","last_unstale":"2024-02-12T17:17:04.859695+0000","last_undegraded":"2024-02-12T17:17:04.859695+0000","last_fullsized":"2024-02-12T17:17:04.859695+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:15.262134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:12:15.262134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":754,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898242+0000","last_change":"2024-02-12T17:13:46.566957+0000","last_active":"2024-02-12T17:17:06.898242+0000","last_peered":"2024-02-12T17:17:06.898242+0000","last_clean":"2024-02-12T17:17:06.898242+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:17:06.898242+0000","last_undegraded":"2024-02-12T17:17:06.898242+0000","last_fullsized":"2024-02-12T17:17:06.898242+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:22.791009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:22.791009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":251,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910301+0000","last_change":"2024-02-12T17:16:59.712636+0000","last_active":"2024-02-12T17:17:07.910301+0000","last_peered":"2024-02-12T17:17:07.910301+0000","last_clean":"2024-02-12T17:17:07.910301+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:17:07.910301+0000","last_undegraded":"2024-02-12T17:17:07.910301+0000","last_fullsized":"2024-02-12T17:17:07.910301+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'70","last_scrub_stamp":"2024-02-12T17:16:59.712512+0000","last_deep_scrub":"180'39","last_deep_scrub_stamp":"2024-02-12T16:56:43.083656+0000","last_clean_scrub_stamp":"2024-02-12T17:16:59.712512+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":680,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.915327+0000","last_change":"2024-02-12T17:16:46.731125+0000","last_active":"2024-02-12T17:17:07.915327+0000","last_peered":"2024-02-12T17:17:07.915327+0000","last_clean":"2024-02-12T17:17:07.915327+0000","last_became_active":"2024-02-12T17:16:46.730767+0000","last_became_peered":"2024-02-12T17:16:46.730767+0000","last_unstale":"2024-02-12T17:17:07.915327+0000","last_undegraded":"2024-02-12T17:17:07.915327+0000","last_fullsized":"2024-02-12T17:17:07.915327+0000","mapping_epoch":705,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:13:02.852563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:13:02.852563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":723,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939808+0000","last_change":"2024-02-12T17:13:00.766698+0000","last_active":"2024-02-12T17:17:06.939808+0000","last_peered":"2024-02-12T17:17:06.939808+0000","last_clean":"2024-02-12T17:17:06.939808+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:17:06.939808+0000","last_undegraded":"2024-02-12T17:17:06.939808+0000","last_fullsized":"2024-02-12T17:17:06.939808+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:00.766590+0000","last_clean_scrub_stamp":"2024-02-12T17:13:00.766590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":422,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-12T17:17:08.920168+0000","last_change":"2024-02-12T17:13:46.584941+0000","last_active":"2024-02-12T17:17:08.920168+0000","last_peered":"2024-02-12T17:17:08.920168+0000","last_clean":"2024-02-12T17:17:08.920168+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:17:08.920168+0000","last_undegraded":"2024-02-12T17:17:08.920168+0000","last_fullsized":"2024-02-12T17:17:08.920168+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:07:37.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:37.505002+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":415,"num_read_kb":410,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":382,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858764+0000","last_change":"2024-02-12T17:07:31.904869+0000","last_active":"2024-02-12T17:17:04.858764+0000","last_peered":"2024-02-12T17:17:04.858764+0000","last_clean":"2024-02-12T17:17:04.858764+0000","last_became_active":"2024-02-12T17:03:50.532676+0000","last_became_peered":"2024-02-12T17:03:50.532676+0000","last_unstale":"2024-02-12T17:17:04.858764+0000","last_undegraded":"2024-02-12T17:17:04.858764+0000","last_fullsized":"2024-02-12T17:17:04.858764+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:31.904783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:31.904783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":189,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910155+0000","last_change":"2024-02-12T17:16:55.695958+0000","last_active":"2024-02-12T17:17:07.910155+0000","last_peered":"2024-02-12T17:17:07.910155+0000","last_clean":"2024-02-12T17:17:07.910155+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:17:07.910155+0000","last_undegraded":"2024-02-12T17:17:07.910155+0000","last_fullsized":"2024-02-12T17:17:07.910155+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'26","last_scrub_stamp":"2024-02-12T17:16:55.695884+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:16:55.695884+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":647,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898529+0000","last_change":"2024-02-12T17:13:10.412138+0000","last_active":"2024-02-12T17:17:06.898529+0000","last_peered":"2024-02-12T17:17:06.898529+0000","last_clean":"2024-02-12T17:17:06.898529+0000","last_became_active":"2024-02-12T17:01:16.909132+0000","last_became_peered":"2024-02-12T17:01:16.909132+0000","last_unstale":"2024-02-12T17:17:06.898529+0000","last_undegraded":"2024-02-12T17:17:06.898529+0000","last_fullsized":"2024-02-12T17:17:06.898529+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:13:10.412031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":786,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898719+0000","last_change":"2024-02-12T17:13:46.826853+0000","last_active":"2024-02-12T17:17:06.898719+0000","last_peered":"2024-02-12T17:17:06.898719+0000","last_clean":"2024-02-12T17:17:06.898719+0000","last_became_active":"2024-02-12T17:13:46.821418+0000","last_became_peered":"2024-02-12T17:13:46.821418+0000","last_unstale":"2024-02-12T17:17:06.898719+0000","last_undegraded":"2024-02-12T17:17:06.898719+0000","last_fullsized":"2024-02-12T17:17:06.898719+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.951964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:00.237119+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.951964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":414,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858862+0000","last_change":"2024-02-12T17:12:59.006372+0000","last_active":"2024-02-12T17:17:04.858862+0000","last_peered":"2024-02-12T17:17:04.858862+0000","last_clean":"2024-02-12T17:17:04.858862+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:17:04.858862+0000","last_undegraded":"2024-02-12T17:17:04.858862+0000","last_fullsized":"2024-02-12T17:17:04.858862+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.006260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.006260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":694,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898486+0000","last_change":"2024-02-12T17:11:57.816604+0000","last_active":"2024-02-12T17:17:06.898486+0000","last_peered":"2024-02-12T17:17:06.898486+0000","last_clean":"2024-02-12T17:17:06.898486+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:17:06.898486+0000","last_undegraded":"2024-02-12T17:17:06.898486+0000","last_fullsized":"2024-02-12T17:17:06.898486+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:57.816535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:11:57.816535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":768,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939444+0000","last_change":"2024-02-12T17:17:02.302458+0000","last_active":"2024-02-12T17:17:06.939444+0000","last_peered":"2024-02-12T17:17:06.939444+0000","last_clean":"2024-02-12T17:17:06.939444+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:17:06.939444+0000","last_undegraded":"2024-02-12T17:17:06.939444+0000","last_fullsized":"2024-02-12T17:17:06.939444+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:02.302384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:47:33.612681+0000","last_clean_scrub_stamp":"2024-02-12T17:17:02.302384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":333,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.767283+0000","last_change":"2024-02-12T17:17:07.767237+0000","last_active":"2024-02-12T17:17:07.767283+0000","last_peered":"2024-02-12T17:17:07.767283+0000","last_clean":"2024-02-12T17:17:07.767283+0000","last_became_active":"2024-02-12T17:14:08.726463+0000","last_became_peered":"2024-02-12T17:14:08.726463+0000","last_unstale":"2024-02-12T17:17:07.767283+0000","last_undegraded":"2024-02-12T17:17:07.767283+0000","last_fullsized":"2024-02-12T17:17:07.767283+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:17:07.767093+0000","last_deep_scrub":"462'33","last_deep_scrub_stamp":"2024-02-12T17:06:53.886342+0000","last_clean_scrub_stamp":"2024-02-12T17:17:07.767093+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":730,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939959+0000","last_change":"2024-02-12T17:17:03.332914+0000","last_active":"2024-02-12T17:17:06.939959+0000","last_peered":"2024-02-12T17:17:06.939959+0000","last_clean":"2024-02-12T17:17:06.939959+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:17:06.939959+0000","last_undegraded":"2024-02-12T17:17:06.939959+0000","last_fullsized":"2024-02-12T17:17:06.939959+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:17:03.332780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:17:03.332780+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":116,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910446+0000","last_change":"2024-02-12T17:17:02.766208+0000","last_active":"2024-02-12T17:17:07.910446+0000","last_peered":"2024-02-12T17:17:07.910446+0000","last_clean":"2024-02-12T17:17:07.910446+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:17:07.910446+0000","last_undegraded":"2024-02-12T17:17:07.910446+0000","last_fullsized":"2024-02-12T17:17:07.910446+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:02.766134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:17:02.766134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":77,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.915298+0000","last_change":"2024-02-12T17:14:08.859925+0000","last_active":"2024-02-12T17:17:07.915298+0000","last_peered":"2024-02-12T17:17:07.915298+0000","last_clean":"2024-02-12T17:17:07.915298+0000","last_became_active":"2024-02-12T17:14:08.859396+0000","last_became_peered":"2024-02-12T17:14:08.859396+0000","last_unstale":"2024-02-12T17:17:07.915298+0000","last_undegraded":"2024-02-12T17:17:07.915298+0000","last_fullsized":"2024-02-12T17:17:07.915298+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:03.904976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:12:03.904976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":768,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939437+0000","last_change":"2024-02-12T17:17:00.287833+0000","last_active":"2024-02-12T17:17:06.939437+0000","last_peered":"2024-02-12T17:17:06.939437+0000","last_clean":"2024-02-12T17:17:06.939437+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:17:06.939437+0000","last_undegraded":"2024-02-12T17:17:06.939437+0000","last_fullsized":"2024-02-12T17:17:06.939437+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:17:00.287740+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:06:01.905929+0000","last_clean_scrub_stamp":"2024-02-12T17:17:00.287740+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":96,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.916279+0000","last_change":"2024-02-12T17:16:57.566977+0000","last_active":"2024-02-12T17:17:07.916279+0000","last_peered":"2024-02-12T17:17:07.916279+0000","last_clean":"2024-02-12T17:17:07.916279+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:17:07.916279+0000","last_undegraded":"2024-02-12T17:17:07.916279+0000","last_fullsized":"2024-02-12T17:17:07.916279+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'36","last_scrub_stamp":"2024-02-12T17:16:57.566903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:16:57.566903+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":766,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898667+0000","last_change":"2024-02-12T17:13:46.827015+0000","last_active":"2024-02-12T17:17:06.898667+0000","last_peered":"2024-02-12T17:17:06.898667+0000","last_clean":"2024-02-12T17:17:06.898667+0000","last_became_active":"2024-02-12T17:13:46.821796+0000","last_became_peered":"2024-02-12T17:13:46.821796+0000","last_unstale":"2024-02-12T17:17:06.898667+0000","last_undegraded":"2024-02-12T17:17:06.898667+0000","last_fullsized":"2024-02-12T17:17:06.898667+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:48.698727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:48.698727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":142,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910141+0000","last_change":"2024-02-12T17:13:46.579742+0000","last_active":"2024-02-12T17:17:07.910141+0000","last_peered":"2024-02-12T17:17:07.910141+0000","last_clean":"2024-02-12T17:17:07.910141+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:17:07.910141+0000","last_undegraded":"2024-02-12T17:17:07.910141+0000","last_fullsized":"2024-02-12T17:17:07.910141+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:48.211447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:07:48.211447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":777,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939888+0000","last_change":"2024-02-12T17:14:08.746702+0000","last_active":"2024-02-12T17:17:06.939888+0000","last_peered":"2024-02-12T17:17:06.939888+0000","last_clean":"2024-02-12T17:17:06.939888+0000","last_became_active":"2024-02-12T17:14:08.746395+0000","last_became_peered":"2024-02-12T17:14:08.746395+0000","last_unstale":"2024-02-12T17:17:06.939888+0000","last_undegraded":"2024-02-12T17:17:06.939888+0000","last_fullsized":"2024-02-12T17:17:06.939888+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:45.764955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:45.764955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"526'39","reported_seq":95,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.917756+0000","last_change":"2024-02-12T17:14:11.148689+0000","last_active":"2024-02-12T17:17:07.917756+0000","last_peered":"2024-02-12T17:17:07.917756+0000","last_clean":"2024-02-12T17:17:07.917756+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:17:07.917756+0000","last_undegraded":"2024-02-12T17:17:07.917756+0000","last_fullsized":"2024-02-12T17:17:07.917756+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'29","last_scrub_stamp":"2024-02-12T17:07:52.811091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:07:52.811091+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":219,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.896115+0000","last_change":"2024-02-12T17:12:53.746937+0000","last_active":"2024-02-12T17:17:06.896115+0000","last_peered":"2024-02-12T17:17:06.896115+0000","last_clean":"2024-02-12T17:17:06.896115+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:17:06.896115+0000","last_undegraded":"2024-02-12T17:17:06.896115+0000","last_fullsized":"2024-02-12T17:17:06.896115+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.746858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.746858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":423,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858919+0000","last_change":"2024-02-12T17:13:21.589965+0000","last_active":"2024-02-12T17:17:04.858919+0000","last_peered":"2024-02-12T17:17:04.858919+0000","last_clean":"2024-02-12T17:17:04.858919+0000","last_became_active":"2024-02-12T17:13:21.589584+0000","last_became_peered":"2024-02-12T17:13:21.589584+0000","last_unstale":"2024-02-12T17:17:04.858919+0000","last_undegraded":"2024-02-12T17:17:04.858919+0000","last_fullsized":"2024-02-12T17:17:04.858919+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:01.059570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:13:01.059570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":158,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910550+0000","last_change":"2024-02-12T17:14:18.716530+0000","last_active":"2024-02-12T17:17:07.910550+0000","last_peered":"2024-02-12T17:17:07.910550+0000","last_clean":"2024-02-12T17:17:07.910550+0000","last_became_active":"2024-02-12T17:14:18.716232+0000","last_became_peered":"2024-02-12T17:14:18.716232+0000","last_unstale":"2024-02-12T17:17:07.910550+0000","last_undegraded":"2024-02-12T17:17:07.910550+0000","last_fullsized":"2024-02-12T17:17:07.910550+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:11:58.772141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:11:58.772141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":487,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858808+0000","last_change":"2024-02-12T17:17:00.320191+0000","last_active":"2024-02-12T17:17:04.858808+0000","last_peered":"2024-02-12T17:17:04.858808+0000","last_clean":"2024-02-12T17:17:04.858808+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:17:04.858808+0000","last_undegraded":"2024-02-12T17:17:04.858808+0000","last_fullsized":"2024-02-12T17:17:04.858808+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:17:00.320127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:17:00.320127+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":702,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858662+0000","last_change":"2024-02-12T17:14:08.765556+0000","last_active":"2024-02-12T17:17:04.858662+0000","last_peered":"2024-02-12T17:17:04.858662+0000","last_clean":"2024-02-12T17:17:04.858662+0000","last_became_active":"2024-02-12T17:14:08.757310+0000","last_became_peered":"2024-02-12T17:14:08.757310+0000","last_unstale":"2024-02-12T17:17:04.858662+0000","last_undegraded":"2024-02-12T17:17:04.858662+0000","last_fullsized":"2024-02-12T17:17:04.858662+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:58.871569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:58.871569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"65'1","reported_seq":756,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939849+0000","last_change":"2024-02-12T17:13:08.874063+0000","last_active":"2024-02-12T17:17:06.939849+0000","last_peered":"2024-02-12T17:17:06.939849+0000","last_clean":"2024-02-12T17:17:06.939849+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:17:06.939849+0000","last_undegraded":"2024-02-12T17:17:06.939849+0000","last_fullsized":"2024-02-12T17:17:06.939849+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-12T17:13:08.873961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:13:08.873961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":237,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.896304+0000","last_change":"2024-02-12T17:14:14.843264+0000","last_active":"2024-02-12T17:17:06.896304+0000","last_peered":"2024-02-12T17:17:06.896304+0000","last_clean":"2024-02-12T17:17:06.896304+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:17:06.896304+0000","last_undegraded":"2024-02-12T17:17:06.896304+0000","last_fullsized":"2024-02-12T17:17:06.896304+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:56.688982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.688982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"526'42","reported_seq":902,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898111+0000","last_change":"2024-02-12T17:14:09.517724+0000","last_active":"2024-02-12T17:17:06.898111+0000","last_peered":"2024-02-12T17:17:06.898111+0000","last_clean":"2024-02-12T17:17:06.898111+0000","last_became_active":"2024-02-12T17:14:08.740691+0000","last_became_peered":"2024-02-12T17:14:08.740691+0000","last_unstale":"2024-02-12T17:17:06.898111+0000","last_undegraded":"2024-02-12T17:17:06.898111+0000","last_fullsized":"2024-02-12T17:17:06.898111+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:12:05.850456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:05.850456+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"706'33","reported_seq":691,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858710+0000","last_change":"2024-02-12T17:16:46.736263+0000","last_active":"2024-02-12T17:17:04.858710+0000","last_peered":"2024-02-12T17:17:04.858710+0000","last_clean":"2024-02-12T17:17:04.858710+0000","last_became_active":"2024-02-12T17:16:46.735883+0000","last_became_peered":"2024-02-12T17:16:46.735883+0000","last_unstale":"2024-02-12T17:17:04.858710+0000","last_undegraded":"2024-02-12T17:17:04.858710+0000","last_fullsized":"2024-02-12T17:17:04.858710+0000","mapping_epoch":705,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'11","last_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:02:29.064208+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":425,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.860230+0000","last_change":"2024-02-12T17:09:26.729032+0000","last_active":"2024-02-12T17:17:04.860230+0000","last_peered":"2024-02-12T17:17:04.860230+0000","last_clean":"2024-02-12T17:17:04.860230+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:17:04.860230+0000","last_undegraded":"2024-02-12T17:17:04.860230+0000","last_fullsized":"2024-02-12T17:17:04.860230+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:07:20.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:07:20.768688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"642'31","reported_seq":1139,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.703047+0000","last_change":"2024-02-12T17:13:53.773053+0000","last_active":"2024-02-12T17:17:07.703047+0000","last_peered":"2024-02-12T17:17:07.703047+0000","last_clean":"2024-02-12T17:17:07.703047+0000","last_became_active":"2024-02-12T17:13:53.772744+0000","last_became_peered":"2024-02-12T17:13:53.772744+0000","last_unstale":"2024-02-12T17:17:07.703047+0000","last_undegraded":"2024-02-12T17:17:07.703047+0000","last_fullsized":"2024-02-12T17:17:07.703047+0000","mapping_epoch":638,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'28","last_scrub_stamp":"2024-02-12T17:12:07.826227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:12:07.826227+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":431,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939342+0000","last_change":"2024-02-12T17:14:10.756477+0000","last_active":"2024-02-12T17:17:06.939342+0000","last_peered":"2024-02-12T17:17:06.939342+0000","last_clean":"2024-02-12T17:17:06.939342+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:17:06.939342+0000","last_undegraded":"2024-02-12T17:17:06.939342+0000","last_fullsized":"2024-02-12T17:17:06.939342+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:16.873570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.873570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":296,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.896250+0000","last_change":"2024-02-12T17:17:02.799575+0000","last_active":"2024-02-12T17:17:06.896250+0000","last_peered":"2024-02-12T17:17:06.896250+0000","last_clean":"2024-02-12T17:17:06.896250+0000","last_became_active":"2024-02-12T17:13:46.807214+0000","last_became_peered":"2024-02-12T17:13:46.807214+0000","last_unstale":"2024-02-12T17:17:06.896250+0000","last_undegraded":"2024-02-12T17:17:06.896250+0000","last_fullsized":"2024-02-12T17:17:06.896250+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:17:02.799501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:17:02.799501+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":766,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898760+0000","last_change":"2024-02-12T17:17:01.153820+0000","last_active":"2024-02-12T17:17:06.898760+0000","last_peered":"2024-02-12T17:17:06.898760+0000","last_clean":"2024-02-12T17:17:06.898760+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:17:06.898760+0000","last_undegraded":"2024-02-12T17:17:06.898760+0000","last_fullsized":"2024-02-12T17:17:06.898760+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:01.153726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:17:01.153726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"668'27","reported_seq":1164,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.703067+0000","last_change":"2024-02-12T17:14:24.235718+0000","last_active":"2024-02-12T17:17:07.703067+0000","last_peered":"2024-02-12T17:17:07.703067+0000","last_clean":"2024-02-12T17:17:07.703067+0000","last_became_active":"2024-02-12T17:14:24.235386+0000","last_became_peered":"2024-02-12T17:14:24.235386+0000","last_unstale":"2024-02-12T17:17:07.703067+0000","last_undegraded":"2024-02-12T17:17:07.703067+0000","last_fullsized":"2024-02-12T17:17:07.703067+0000","mapping_epoch":665,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'24","last_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.647212+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":134,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910053+0000","last_change":"2024-02-12T17:17:05.755943+0000","last_active":"2024-02-12T17:17:07.910053+0000","last_peered":"2024-02-12T17:17:07.910053+0000","last_clean":"2024-02-12T17:17:07.910053+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:17:07.910053+0000","last_undegraded":"2024-02-12T17:17:07.910053+0000","last_fullsized":"2024-02-12T17:17:07.910053+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:17:05.755880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:17:05.755880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":177,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910263+0000","last_change":"2024-02-12T17:13:46.819259+0000","last_active":"2024-02-12T17:17:07.910263+0000","last_peered":"2024-02-12T17:17:07.910263+0000","last_clean":"2024-02-12T17:17:07.910263+0000","last_became_active":"2024-02-12T17:13:46.818771+0000","last_became_peered":"2024-02-12T17:13:46.818771+0000","last_unstale":"2024-02-12T17:17:07.910263+0000","last_undegraded":"2024-02-12T17:17:07.910263+0000","last_fullsized":"2024-02-12T17:17:07.910263+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'34","last_scrub_stamp":"2024-02-12T17:07:40.934268+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2024-02-12T16:52:51.864004+0000","last_clean_scrub_stamp":"2024-02-12T17:07:40.934268+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":109,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910539+0000","last_change":"2024-02-12T17:13:46.805110+0000","last_active":"2024-02-12T17:17:07.910539+0000","last_peered":"2024-02-12T17:17:07.910539+0000","last_clean":"2024-02-12T17:17:07.910539+0000","last_became_active":"2024-02-12T17:13:46.802275+0000","last_became_peered":"2024-02-12T17:13:46.802275+0000","last_unstale":"2024-02-12T17:17:07.910539+0000","last_undegraded":"2024-02-12T17:17:07.910539+0000","last_fullsized":"2024-02-12T17:17:07.910539+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:00.901873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:12:00.901873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"433'14","reported_seq":667,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.703089+0000","last_change":"2024-02-12T17:13:07.711701+0000","last_active":"2024-02-12T17:17:07.703089+0000","last_peered":"2024-02-12T17:17:07.703089+0000","last_clean":"2024-02-12T17:17:07.703089+0000","last_became_active":"2024-02-12T17:05:40.431342+0000","last_became_peered":"2024-02-12T17:05:40.431342+0000","last_unstale":"2024-02-12T17:17:07.703089+0000","last_undegraded":"2024-02-12T17:17:07.703089+0000","last_fullsized":"2024-02-12T17:17:07.703089+0000","mapping_epoch":431,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-02-12T17:13:07.711601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:07.711601+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":129,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.911732+0000","last_change":"2024-02-12T17:14:00.999940+0000","last_active":"2024-02-12T17:17:07.911732+0000","last_peered":"2024-02-12T17:17:07.911732+0000","last_clean":"2024-02-12T17:17:07.911732+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:17:07.911732+0000","last_undegraded":"2024-02-12T17:17:07.911732+0000","last_fullsized":"2024-02-12T17:17:07.911732+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:07:49.837325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:07:49.837325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"526'30","reported_seq":519,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.860316+0000","last_change":"2024-02-12T17:13:46.806094+0000","last_active":"2024-02-12T17:17:04.860316+0000","last_peered":"2024-02-12T17:17:04.860316+0000","last_clean":"2024-02-12T17:17:04.860316+0000","last_became_active":"2024-02-12T17:13:46.805712+0000","last_became_peered":"2024-02-12T17:13:46.805712+0000","last_unstale":"2024-02-12T17:17:04.860316+0000","last_undegraded":"2024-02-12T17:17:04.860316+0000","last_fullsized":"2024-02-12T17:17:04.860316+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:11:59.132916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:11:59.132916+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":445,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.859376+0000","last_change":"2024-02-12T17:13:21.603687+0000","last_active":"2024-02-12T17:17:04.859376+0000","last_peered":"2024-02-12T17:17:04.859376+0000","last_clean":"2024-02-12T17:17:04.859376+0000","last_became_active":"2024-02-12T17:13:21.603332+0000","last_became_peered":"2024-02-12T17:13:21.603332+0000","last_unstale":"2024-02-12T17:17:04.859376+0000","last_undegraded":"2024-02-12T17:17:04.859376+0000","last_fullsized":"2024-02-12T17:17:04.859376+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:26.583654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:06:26.583654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":140,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910352+0000","last_change":"2024-02-12T17:13:46.804446+0000","last_active":"2024-02-12T17:17:07.910352+0000","last_peered":"2024-02-12T17:17:07.910352+0000","last_clean":"2024-02-12T17:17:07.910352+0000","last_became_active":"2024-02-12T17:13:46.804033+0000","last_became_peered":"2024-02-12T17:13:46.804033+0000","last_unstale":"2024-02-12T17:17:07.910352+0000","last_undegraded":"2024-02-12T17:17:07.910352+0000","last_fullsized":"2024-02-12T17:17:07.910352+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:53.793569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:12:53.793569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":741,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939579+0000","last_change":"2024-02-12T17:13:56.661004+0000","last_active":"2024-02-12T17:17:06.939579+0000","last_peered":"2024-02-12T17:17:06.939579+0000","last_clean":"2024-02-12T17:17:06.939579+0000","last_became_active":"2024-02-12T17:13:56.660707+0000","last_became_peered":"2024-02-12T17:13:56.660707+0000","last_unstale":"2024-02-12T17:17:06.939579+0000","last_undegraded":"2024-02-12T17:17:06.939579+0000","last_fullsized":"2024-02-12T17:17:06.939579+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:59.685601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:12:59.685601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":209,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.910922+0000","last_change":"2024-02-12T17:13:46.757888+0000","last_active":"2024-02-12T17:17:07.910922+0000","last_peered":"2024-02-12T17:17:07.910922+0000","last_clean":"2024-02-12T17:17:07.910922+0000","last_became_active":"2024-02-12T17:13:46.757450+0000","last_became_peered":"2024-02-12T17:13:46.757450+0000","last_unstale":"2024-02-12T17:17:07.910922+0000","last_undegraded":"2024-02-12T17:17:07.910922+0000","last_fullsized":"2024-02-12T17:17:07.910922+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:12:56.880282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:56.880282+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":741,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.939346+0000","last_change":"2024-02-12T17:12:08.013330+0000","last_active":"2024-02-12T17:17:06.939346+0000","last_peered":"2024-02-12T17:17:06.939346+0000","last_clean":"2024-02-12T17:17:06.939346+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:17:06.939346+0000","last_undegraded":"2024-02-12T17:17:06.939346+0000","last_fullsized":"2024-02-12T17:17:06.939346+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:08.013247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:12:08.013247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":639,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.917928+0000","last_change":"2024-02-12T17:12:04.711418+0000","last_active":"2024-02-12T17:17:07.917928+0000","last_peered":"2024-02-12T17:17:07.917928+0000","last_clean":"2024-02-12T17:17:07.917928+0000","last_became_active":"2024-02-12T17:01:16.911470+0000","last_became_peered":"2024-02-12T17:01:16.911470+0000","last_unstale":"2024-02-12T17:17:07.917928+0000","last_undegraded":"2024-02-12T17:17:07.917928+0000","last_fullsized":"2024-02-12T17:17:07.917928+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:04.711333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:58.524736+0000","last_clean_scrub_stamp":"2024-02-12T17:12:04.711333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":496,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898390+0000","last_change":"2024-02-12T17:14:08.740752+0000","last_active":"2024-02-12T17:17:06.898390+0000","last_peered":"2024-02-12T17:17:06.898390+0000","last_clean":"2024-02-12T17:17:06.898390+0000","last_became_active":"2024-02-12T17:14:08.740430+0000","last_became_peered":"2024-02-12T17:14:08.740430+0000","last_unstale":"2024-02-12T17:17:06.898390+0000","last_undegraded":"2024-02-12T17:17:06.898390+0000","last_fullsized":"2024-02-12T17:17:06.898390+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:38.925654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.794228+0000","last_clean_scrub_stamp":"2024-02-12T17:06:38.925654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":278,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.896212+0000","last_change":"2024-02-12T17:16:56.742750+0000","last_active":"2024-02-12T17:17:06.896212+0000","last_peered":"2024-02-12T17:17:06.896212+0000","last_clean":"2024-02-12T17:17:06.896212+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:17:06.896212+0000","last_undegraded":"2024-02-12T17:17:06.896212+0000","last_fullsized":"2024-02-12T17:17:06.896212+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'51","last_scrub_stamp":"2024-02-12T17:16:56.742673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:16:56.742673+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":77,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.915219+0000","last_change":"2024-02-12T17:14:08.859503+0000","last_active":"2024-02-12T17:17:07.915219+0000","last_peered":"2024-02-12T17:17:07.915219+0000","last_clean":"2024-02-12T17:17:07.915219+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:17:07.915219+0000","last_undegraded":"2024-02-12T17:17:07.915219+0000","last_fullsized":"2024-02-12T17:17:07.915219+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:03.705889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:13:03.705889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":636,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.858736+0000","last_change":"2024-02-12T17:14:08.762783+0000","last_active":"2024-02-12T17:17:04.858736+0000","last_peered":"2024-02-12T17:17:04.858736+0000","last_clean":"2024-02-12T17:17:04.858736+0000","last_became_active":"2024-02-12T17:14:08.760654+0000","last_became_peered":"2024-02-12T17:14:08.760654+0000","last_unstale":"2024-02-12T17:17:04.858736+0000","last_undegraded":"2024-02-12T17:17:04.858736+0000","last_fullsized":"2024-02-12T17:17:04.858736+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:13:09.044325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:13:09.044325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":417,"reported_epoch":715,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T17:17:04.859583+0000","last_change":"2024-02-12T17:16:45.074788+0000","last_active":"2024-02-12T17:17:04.859583+0000","last_peered":"2024-02-12T17:17:04.859583+0000","last_clean":"2024-02-12T17:13:44.508652+0000","last_became_active":"2024-02-12T17:13:46.563455+0000","last_became_peered":"2024-02-12T17:13:46.563455+0000","last_unstale":"2024-02-12T17:17:04.859583+0000","last_undegraded":"2024-02-12T17:17:04.859583+0000","last_fullsized":"2024-02-12T17:17:04.859583+0000","mapping_epoch":630,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:06:41.443673+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T16:53:56.761445+0000","last_clean_scrub_stamp":"2024-02-12T17:06:41.443673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":106,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.917478+0000","last_change":"2024-02-12T17:17:06.217345+0000","last_active":"2024-02-12T17:17:07.917478+0000","last_peered":"2024-02-12T17:17:07.917478+0000","last_clean":"2024-02-12T17:17:07.917478+0000","last_became_active":"2024-02-12T17:17:05.871903+0000","last_became_peered":"2024-02-12T17:17:05.871903+0000","last_unstale":"2024-02-12T17:17:07.917478+0000","last_undegraded":"2024-02-12T17:17:07.917478+0000","last_fullsized":"2024-02-12T17:17:07.917478+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:12:50.764686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:12:50.764686+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":724,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898155+0000","last_change":"2024-02-12T17:12:11.906816+0000","last_active":"2024-02-12T17:17:06.898155+0000","last_peered":"2024-02-12T17:17:06.898155+0000","last_clean":"2024-02-12T17:17:06.898155+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:17:06.898155+0000","last_undegraded":"2024-02-12T17:17:06.898155+0000","last_fullsized":"2024-02-12T17:17:06.898155+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:12:11.906737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:12:11.906737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":400,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-12T17:17:04.859835+0000","last_change":"2024-02-12T17:11:46.522455+0000","last_active":"2024-02-12T17:17:04.859835+0000","last_peered":"2024-02-12T17:17:04.859835+0000","last_clean":"2024-02-12T17:17:04.859835+0000","last_became_active":"2024-02-12T17:11:46.522149+0000","last_became_peered":"2024-02-12T17:11:46.522149+0000","last_unstale":"2024-02-12T17:17:04.859835+0000","last_undegraded":"2024-02-12T17:17:04.859835+0000","last_fullsized":"2024-02-12T17:17:04.859835+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":587,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:06:53.648837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:06:53.648837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":842,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:06.898034+0000","last_change":"2024-02-12T17:12:16.820857+0000","last_active":"2024-02-12T17:17:06.898034+0000","last_peered":"2024-02-12T17:17:06.898034+0000","last_clean":"2024-02-12T17:17:06.898034+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:17:06.898034+0000","last_undegraded":"2024-02-12T17:17:06.898034+0000","last_fullsized":"2024-02-12T17:17:06.898034+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:12:16.820782+0000","last_deep_scrub":"178'22","last_deep_scrub_stamp":"2024-02-12T16:56:39.138758+0000","last_clean_scrub_stamp":"2024-02-12T17:12:16.820782+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"632'35","reported_seq":1228,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-02-12T17:17:07.703076+0000","last_change":"2024-02-12T17:13:46.691596+0000","last_active":"2024-02-12T17:17:07.703076+0000","last_peered":"2024-02-12T17:17:07.703076+0000","last_clean":"2024-02-12T17:17:07.703076+0000","last_became_active":"2024-02-12T17:13:46.691213+0000","last_became_peered":"2024-02-12T17:13:46.691213+0000","last_unstale":"2024-02-12T17:17:07.703076+0000","last_undegraded":"2024-02-12T17:17:07.703076+0000","last_fullsized":"2024-02-12T17:17:07.703076+0000","mapping_epoch":631,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'29","last_scrub_stamp":"2024-02-12T17:12:47.788327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:12:47.788327+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":415,"num_read_kb":410,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":16876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1029,"ondisk_log_size":1029,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234317,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1649740,"kb_used_data":157212,"kb_used_omap":0,"kb_used_meta":1492160,"kb_avail":92095412,"statfs":{"total":95995035648,"available":94305701888,"internally_reserved":0,"allocated":160985088,"data_stored":159650081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1527971840},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":2.5249999999999999,"5min":1.9099999999999999,"15min":2.024},"min":{"1min":0.36799999999999999,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":10.417,"5min":15.856999999999999,"15min":15.856999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.756,"5min":1.8180000000000001,"15min":1.7290000000000001},"min":{"1min":0.23599999999999999,"5min":0.221,"15min":0.22},"max":{"1min":7.1120000000000001,"5min":12.565,"15min":12.565},"last":0.54800000000000004}]},{"osd":1,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":2.6829999999999998,"15min":2.7530000000000001},"min":{"1min":0.26800000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":7.8949999999999996,"5min":11.632999999999999,"15min":11.632999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.853,"5min":2.613,"15min":2.516},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":7.8869999999999996,"5min":10.91,"15min":10.91},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":2.8010000000000002,"15min":3.1349999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22800000000000001},"max":{"1min":9.5389999999999997,"5min":42.162999999999997,"15min":218.35300000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.3820000000000001,"5min":3,"15min":3.2480000000000002},"min":{"1min":0.375,"5min":0.25600000000000001,"15min":0.247},"max":{"1min":9.407,"5min":42.109999999999999,"15min":218.42099999999999},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":3.754,"5min":2.52,"15min":2.1930000000000001},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.255},"max":{"1min":11.741,"5min":15.811999999999999,"15min":15.811999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":3.9079999999999999,"5min":3.0710000000000002,"15min":2.649},"min":{"1min":0.42299999999999999,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":12.375999999999999,"5min":13.452,"15min":31.152000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.4099999999999999,"15min":1.6759999999999999},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":2.7810000000000001,"5min":20.672999999999998,"15min":20.672999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.8939999999999999,"15min":1.96},"min":{"1min":0.23300000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.7240000000000002,"5min":38.390000000000001,"15min":38.390000000000001},"last":0.53100000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.8169999999999999,"15min":2.169},"min":{"1min":0.28100000000000003,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":2.7000000000000002,"5min":20.702000000000002,"15min":56.560000000000002},"last":0.443},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.4330000000000001,"15min":1.915},"min":{"1min":0.25800000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.6930000000000001,"5min":10.669,"15min":53.963999999999999},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Feb 12 17:16:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.4570000000000001,"15min":1.532},"min":{"1min":0.30399999999999999,"5min":0.223,"15min":0.223},"max":{"1min":4.0030000000000001,"5min":21.620000000000001,"15min":21.620000000000001},"last":0.5},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.671,"15min":1.724},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.7360000000000002,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.627}]}]},{"osd":6,"up_from":45,"seq":193273528658,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895888,"kb_used_data":3168,"kb_used_omap":0,"kb_used_meta":892352,"kb_avail":92849264,"statfs":{"total":95995035648,"available":95077646336,"internally_reserved":0,"allocated":3244032,"data_stored":1905457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913768448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":2.8530000000000002,"5min":2.1709999999999998,"15min":1.9390000000000001},"min":{"1min":0.26900000000000002,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":11.115,"5min":33.576999999999998,"15min":33.576999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.944,"15min":1.8979999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.2569999999999997,"5min":33.569000000000003,"15min":33.569000000000003},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:16:50 2024","interfaces":[{"interface":"back","average":{"1min":2.8889999999999998,"5min":2.2759999999999998,"15min":2.153},"min":{"1min":0.29799999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.611000000000001,"5min":20.611000000000001,"15min":20.611000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":2.7810000000000001,"5min":2.258,"15min":2.2149999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":20.622,"5min":20.622,"15min":20.622},"last":0.63}]},{"osd":2,"last update":"Mon Feb 12 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":2.9089999999999998,"15min":2.698},"min":{"1min":0.33600000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.9079999999999995,"5min":33.594999999999999,"15min":56.798000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":2.653,"5min":2.96,"15min":2.4620000000000002},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":7.226,"5min":33.526000000000003,"15min":56.780999999999999},"last":0.439}]},{"osd":3,"last update":"Mon Feb 12 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":4.5250000000000004,"5min":3.1850000000000001,"15min":2.6360000000000001},"min":{"1min":0.36299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":25.867999999999999,"5min":33.548999999999999,"15min":58.420999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":2.8940000000000001,"5min":3.3380000000000001,"15min":2.927},"min":{"1min":0.33400000000000002,"5min":0.29299999999999998,"15min":0.254},"max":{"1min":14.435,"5min":33.503,"15min":58.709000000000003},"last":0.58599999999999997}]},{"osd":4,"last update":"Mon Feb 12 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.6739999999999999,"15min":1.306},"min":{"1min":0.29599999999999999,"5min":0.23599999999999999,"15min":0.23499999999999999},"max":{"1min":5.1369999999999996,"5min":33.706000000000003,"15min":33.706000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.897,"15min":1.4730000000000001},"min":{"1min":0.26600000000000001,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":3.9670000000000001,"5min":34.895000000000003,"15min":34.895000000000003},"last":0.622}]},{"osd":5,"last update":"Mon Feb 12 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.766,"15min":1.6930000000000001},"min":{"1min":0.33200000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.9390000000000001,"5min":14.56,"15min":34.869},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.093,"5min":1.3520000000000001,"15min":1.6819999999999999},"min":{"1min":0.30099999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.1100000000000003,"5min":11.1,"15min":33.591000000000001},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:16:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.7090000000000001,"15min":1.7629999999999999},"min":{"1min":0.26500000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":4.4550000000000001,"5min":35.122999999999998,"15min":35.122999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.421,"5min":1.869,"15min":1.6220000000000001},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.26500000000000001},"max":{"1min":5.1890000000000001,"5min":34.045000000000002,"15min":34.045000000000002},"last":0.60899999999999999}]}]},{"osd":1,"up_from":630,"seq":2705829396522,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338084,"kb_used_data":3104,"kb_used_omap":3,"kb_used_meta":334524,"kb_avail":93407068,"statfs":{"total":95995035648,"available":95648837632,"internally_reserved":0,"allocated":3178496,"data_stored":1865431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":342552637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":3.2170000000000001,"5min":3.306,"15min":3.4249999999999998},"min":{"1min":0.26100000000000001,"5min":0.214,"15min":0.214},"max":{"1min":23.952999999999999,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":8.1310000000000002,"5min":4.5999999999999996,"15min":3.8140000000000001},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":27.268999999999998,"5min":27.268999999999998,"15min":27.268999999999998},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":2.7959999999999998,"5min":3.2639999999999998,"15min":3.476},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.856,"5min":12.856,"15min":12.856},"last":0.48099999999999998},{"interface":"front","average":{"1min":5.0640000000000001,"5min":3.843,"15min":3.6120000000000001},"min":{"1min":0.255,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":24.236999999999998,"5min":24.236999999999998,"15min":24.236999999999998},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":6.2800000000000002,"5min":4.5110000000000001,"15min":3.9319999999999999},"min":{"1min":0.29599999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":31.292999999999999,"5min":31.292999999999999,"15min":31.292999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":6.3230000000000004,"5min":4.8010000000000002,"15min":4.4809999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":23.905999999999999,"5min":23.905999999999999,"15min":23.905999999999999},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":2.3479999999999999,"15min":2.4860000000000002},"min":{"1min":0.34100000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":12.901999999999999,"5min":12.901999999999999,"15min":12.901999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":3.0499999999999998,"5min":2.7989999999999999,"15min":2.7869999999999999},"min":{"1min":0.40699999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":8.7840000000000007,"5min":12.805999999999999,"15min":12.805999999999999},"last":0.76000000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":3.0249999999999999,"5min":3.0600000000000001,"15min":3.1640000000000001},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.7159999999999993,"5min":12.769,"15min":12.769},"last":0.80000000000000004},{"interface":"front","average":{"1min":4.1319999999999997,"5min":2.7229999999999999,"15min":2.3159999999999998},"min":{"1min":0.35999999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":23.300999999999998,"5min":23.300999999999998,"15min":23.300999999999998},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":4.1619999999999999,"5min":3.1389999999999998,"15min":2.9329999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":23.347000000000001,"5min":23.347000000000001,"15min":23.347000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.4890000000000001,"5min":2.5049999999999999,"15min":2.7530000000000001},"min":{"1min":0.41999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":12.929,"5min":12.929,"15min":12.929},"last":0.77000000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:16:53 2024","interfaces":[{"interface":"back","average":{"1min":3.1960000000000002,"5min":2.8929999999999998,"15min":2.9399999999999999},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":8.7539999999999996,"5min":12.829000000000001,"15min":12.829000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":4.2859999999999996,"5min":2.9849999999999999,"15min":2.589},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":23.376000000000001,"5min":23.376000000000001,"15min":23.376000000000001},"last":0.751}]}]},{"osd":0,"up_from":8,"seq":34359738735,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903472,"kb_used_data":3092,"kb_used_omap":0,"kb_used_meta":900288,"kb_avail":92841680,"statfs":{"total":95995035648,"available":95069880320,"internally_reserved":0,"allocated":3166208,"data_stored":1864993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":921894912},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:16:49 2024","interfaces":[{"interface":"back","average":{"1min":3.6989999999999998,"5min":3.6509999999999998,"15min":3.5030000000000001},"min":{"1min":0.19500000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":11.32,"5min":14.048999999999999,"15min":14.048999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.532,"5min":2.6659999999999999,"15min":2.7559999999999998},"min":{"1min":0.255,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":11.012,"5min":13.787000000000001,"15min":13.787000000000001},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:16:36 2024","interfaces":[{"interface":"back","average":{"1min":3.6360000000000001,"5min":2.589,"15min":2.266},"min":{"1min":0.27000000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":10.509,"5min":19.312000000000001,"15min":19.312000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":3.9769999999999999,"5min":2.9609999999999999,"15min":2.8029999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.20799999999999999},"max":{"1min":11.377000000000001,"5min":22.959,"15min":26.039999999999999},"last":0.437}]},{"osd":3,"last update":"Mon Feb 12 17:16:49 2024","interfaces":[{"interface":"back","average":{"1min":4.5229999999999997,"5min":3.9089999999999998,"15min":3.3889999999999998},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.186},"max":{"1min":13.417,"5min":61.177,"15min":87.311000000000007},"last":0.501},{"interface":"front","average":{"1min":6.7919999999999998,"5min":3.911,"15min":3.5550000000000002},"min":{"1min":0.433,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":21.504999999999999,"5min":59.149999999999999,"15min":90.991},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.3999999999999999,"15min":1.506},"min":{"1min":0.41899999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":6.4779999999999998,"5min":13.861000000000001,"15min":15.141999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.96,"5min":1.673,"15min":2.1099999999999999},"min":{"1min":0.307,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":11.339,"5min":13.503,"15min":26.082999999999998},"last":0.78200000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:16:42 2024","interfaces":[{"interface":"back","average":{"1min":2.8730000000000002,"5min":2.0289999999999999,"15min":2.2829999999999999},"min":{"1min":0.38200000000000001,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":13.981999999999999,"5min":15.090999999999999,"15min":50.404000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.4129999999999998,"5min":1.819,"15min":2.0750000000000002},"min":{"1min":0.31900000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.6070000000000002,"5min":15.164,"15min":49.979999999999997},"last":0.82199999999999995}]},{"osd":6,"last update":"Mon Feb 12 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2850000000000001,"5min":2.6309999999999998,"15min":2.4689999999999999},"min":{"1min":0.33700000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":14.686999999999999,"5min":48.780999999999999,"15min":48.780999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.8100000000000001,"15min":2.375},"min":{"1min":0.38300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.7859999999999996,"5min":49.067999999999998,"15min":49.067999999999998},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:16:36 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.0289999999999999,"15min":2.056},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.6299999999999999,"5min":15.183999999999999,"15min":19.373000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":2.9279999999999999,"5min":2.25,"15min":2.4159999999999999},"min":{"1min":0.36799999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":11.4,"5min":13.542,"15min":26.135000000000002},"last":0.746}]}]},{"osd":2,"up_from":18,"seq":77309411684,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501724,"kb_used_data":3096,"kb_used_omap":0,"kb_used_meta":1498176,"kb_avail":92243428,"statfs":{"total":95995035648,"available":94457270272,"internally_reserved":0,"allocated":3170304,"data_stored":1865802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1534132224},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":2.327,"5min":1.929,"15min":1.966},"min":{"1min":0.32500000000000001,"5min":0.17899999999999999,"15min":0.17000000000000001},"max":{"1min":7.1749999999999998,"5min":11.663,"15min":36.100999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":3.9209999999999998,"5min":2.5609999999999999,"15min":2.3860000000000001},"min":{"1min":0.307,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.271000000000001,"5min":16.350000000000001,"15min":36.299999999999997},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Feb 12 17:16:50 2024","interfaces":[{"interface":"back","average":{"1min":2.3999999999999999,"5min":3.399,"15min":3.8100000000000001},"min":{"1min":0.22900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":10.9,"5min":11.612,"15min":11.612},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.0300000000000002,"5min":5.2539999999999996,"15min":5.6799999999999997},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":17.939,"5min":22.52,"15min":22.52},"last":0.33500000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:17:00 2024","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":4.3159999999999998,"15min":3.2149999999999999},"min":{"1min":0.28199999999999997,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":37.704999999999998,"5min":37.704999999999998,"15min":40.759999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":3.54,"5min":3.4399999999999999,"15min":2.714},"min":{"1min":0.32100000000000001,"5min":0.22,"15min":0.22},"max":{"1min":30.285,"5min":30.285,"15min":30.285},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:16:24 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.754,"15min":1.8759999999999999},"min":{"1min":0.42599999999999999,"5min":0.247,"15min":0.245},"max":{"1min":5.851,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.77,"15min":1.9370000000000001},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.0590000000000002,"5min":11.744,"15min":20.149000000000001},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.748,"15min":2.0030000000000001},"min":{"1min":0.46000000000000002,"5min":0.26000000000000001,"15min":0.217},"max":{"1min":7.1310000000000002,"5min":11.581,"15min":32.57},"last":0.498},{"interface":"front","average":{"1min":2.2309999999999999,"5min":1.871,"15min":2.1070000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.255},"max":{"1min":6.2030000000000003,"5min":10.787000000000001,"15min":31.905999999999999},"last":0.45400000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:17:00 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0089999999999999,"15min":1.913},"min":{"1min":0.29199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.2889999999999997,"5min":10.714,"15min":20.966999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":2.1419999999999999,"5min":2.2200000000000002,"15min":1.9390000000000001},"min":{"1min":0.42199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":12.631,"5min":12.631,"15min":20.193000000000001},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:16:30 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.871,"15min":2.4609999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.7119999999999997,"5min":11.789,"15min":45.468000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":2.3210000000000002,"5min":2.0169999999999999,"15min":2.4569999999999999},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":6.1150000000000002,"5min":10.762,"15min":46.656999999999996},"last":0.433}]}]},{"osd":3,"up_from":26,"seq":111669150046,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896792,"kb_used_data":3536,"kb_used_omap":0,"kb_used_meta":893248,"kb_avail":92848360,"statfs":{"total":95995035648,"available":95076720640,"internally_reserved":0,"allocated":3620864,"data_stored":2104763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914685952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:57 2024","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.3860000000000001,"15min":3.0259999999999998},"min":{"1min":0.20999999999999999,"5min":0.189,"15min":0.17100000000000001},"max":{"1min":22.16,"5min":22.16,"15min":58.451999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":5.6280000000000001,"5min":3.6150000000000002,"15min":3.3849999999999998},"min":{"1min":0.34699999999999998,"5min":0.217,"15min":0.19500000000000001},"max":{"1min":20.681999999999999,"5min":42.984999999999999,"15min":72.688000000000002},"last":0.35799999999999998}]},{"osd":1,"last update":"Mon Feb 12 17:16:51 2024","interfaces":[{"interface":"back","average":{"1min":3.8279999999999998,"5min":5.0460000000000003,"15min":4.6689999999999996},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":16.010000000000002,"5min":22.524000000000001,"15min":22.524000000000001},"last":0.433},{"interface":"front","average":{"1min":5.4749999999999996,"5min":5.4930000000000003,"15min":5.4669999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.710000000000001,"5min":20.710000000000001,"15min":20.710000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:16:57 2024","interfaces":[{"interface":"back","average":{"1min":5.3280000000000003,"5min":3.8319999999999999,"15min":2.8900000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":48.127000000000002,"5min":48.127000000000002,"15min":48.127000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":5.9539999999999997,"5min":3.9849999999999999,"15min":3.3330000000000002},"min":{"1min":0.34599999999999997,"5min":0.22,"15min":0.22},"max":{"1min":40.600999999999999,"5min":40.600999999999999,"15min":46.408000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Mon Feb 12 17:16:15 2024","interfaces":[{"interface":"back","average":{"1min":5.2839999999999998,"5min":3.746,"15min":2.6720000000000002},"min":{"1min":0.54600000000000004,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":20.478000000000002,"5min":72.659000000000006,"15min":72.659000000000006},"last":0.64000000000000001},{"interface":"front","average":{"1min":4.5279999999999996,"5min":3.0979999999999999,"15min":2.1099999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23999999999999999},"max":{"1min":22.440999999999999,"5min":40.930999999999997,"15min":40.930999999999997},"last":0.76700000000000002}]},{"osd":5,"last update":"Mon Feb 12 17:16:34 2024","interfaces":[{"interface":"back","average":{"1min":4.218,"5min":3.1120000000000001,"15min":2.7080000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.253},"max":{"1min":12.329000000000001,"5min":17.306999999999999,"15min":17.940999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":4.3310000000000004,"5min":3.665,"15min":2.9279999999999999},"min":{"1min":0.52000000000000002,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":15.522,"5min":72.620000000000005,"15min":72.620000000000005},"last":0.61099999999999999}]},{"osd":6,"last update":"Mon Feb 12 17:16:57 2024","interfaces":[{"interface":"back","average":{"1min":2.1400000000000001,"5min":2.702,"15min":2.226},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":10.821,"5min":22.489999999999998,"15min":40.889000000000003},"last":0.80900000000000005},{"interface":"front","average":{"1min":2.7999999999999998,"5min":3.089,"15min":2.5219999999999998},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":12.414,"5min":15.113,"15min":18.265000000000001},"last":0.621}]},{"osd":7,"last update":"Mon Feb 12 17:16:15 2024","interfaces":[{"interface":"back","average":{"1min":5.0890000000000004,"5min":3.5609999999999999,"15min":3.0880000000000001},"min":{"1min":0.53500000000000003,"5min":0.375,"15min":0.315},"max":{"1min":20.515999999999998,"5min":72.712999999999994,"15min":72.712999999999994},"last":0.83999999999999997},{"interface":"front","average":{"1min":5.6120000000000001,"5min":3.0259999999999998,"15min":2.9089999999999998},"min":{"1min":0.44900000000000001,"5min":0.38600000000000001,"15min":0.28299999999999997},"max":{"1min":15.169,"5min":15.169,"15min":38.948},"last":0.71899999999999997}]}]},{"osd":4,"up_from":0,"seq":137438953820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":38,"seq":163208757592,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1663320,"kb_used_data":177556,"kb_used_omap":0,"kb_used_meta":1485312,"kb_avail":92081832,"statfs":{"total":95995035648,"available":94291795968,"internally_reserved":0,"allocated":181817344,"data_stored":180516096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1520959488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":3.0950000000000002,"5min":1.9830000000000001,"15min":1.623},"min":{"1min":0.46500000000000002,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":6.8399999999999999,"5min":8.0350000000000001,"15min":10.971},"last":0.54700000000000004},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.915,"15min":1.8029999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20100000000000001},"max":{"1min":12.061,"5min":12.061,"15min":15.368},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Feb 12 17:16:57 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.1200000000000001,"15min":1.905},"min":{"1min":0.312,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.0540000000000003,"5min":9.484,"15min":9.484},"last":0.45300000000000001},{"interface":"front","average":{"1min":3.552,"5min":2.9430000000000001,"15min":3.024},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":18.550000000000001,"5min":18.550000000000001,"15min":18.550000000000001},"last":0.436}]},{"osd":2,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":4.0110000000000001,"5min":2.871,"15min":2.0800000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23200000000000001},"max":{"1min":13.978,"5min":57.390999999999998,"15min":57.390999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.2559999999999998,"5min":2.7229999999999999,"15min":1.9870000000000001},"min":{"1min":0.53300000000000003,"5min":0.214,"15min":0.214},"max":{"1min":6.0259999999999998,"5min":57.134999999999998,"15min":57.134999999999998},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":3.9390000000000001,"5min":2.964,"15min":2.7389999999999999},"min":{"1min":0.36499999999999999,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":11.871,"5min":57.170999999999999,"15min":57.170999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.984,"15min":2.6259999999999999},"min":{"1min":0.33600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":17.760999999999999,"5min":57.484999999999999,"15min":57.484999999999999},"last":0.495}]},{"osd":4,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.385},"min":{"1min":0.26000000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.3660000000000001,"5min":9.0429999999999993,"15min":12.656000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6459999999999999,"15min":1.375},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.254},"max":{"1min":5.1390000000000002,"5min":10.619999999999999,"15min":11.068},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Feb 12 17:16:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.5030000000000001,"5min":14.675000000000001,"15min":14.675000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.651,"5min":1.802,"15min":1.5629999999999999},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":5.2720000000000002,"5min":12.864000000000001,"15min":13.916},"last":0.38700000000000001}]},{"osd":7,"last update":"Mon Feb 12 17:16:28 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.6499999999999999,"15min":1.599},"min":{"1min":0.31900000000000001,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":6.0170000000000003,"5min":14.337999999999999,"15min":14.337999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.6990000000000001,"15min":1.7629999999999999},"min":{"1min":0.316,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":3.9369999999999998,"5min":14.637,"15min":20.274000000000001},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:17:13.979 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-02-12T17:17:13.979 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:14.240 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:12.414018+0000 osd.0 (osd.0) 250 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:17:14.240 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:12.416993+0000 osd.0 (osd.0) 251 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:17:14.240 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:13.159431+0000 mgr.y (mgr.24481) 1039 : cluster [DBG] pgmap v1469: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 36 B/s, 1 objects/s recovering 2024-02-12T17:17:14.241 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:13.290740+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.f scrub starts 2024-02-12T17:17:14.241 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:13.465312+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.f scrub ok 2024-02-12T17:17:14.241 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:12.414018+0000 osd.0 (osd.0) 250 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:17:14.241 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:12.416993+0000 osd.0 (osd.0) 251 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:17:14.241 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:13.159431+0000 mgr.y (mgr.24481) 1039 : cluster [DBG] pgmap v1469: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 36 B/s, 1 objects/s recovering 2024-02-12T17:17:14.241 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:13.290740+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.f scrub starts 2024-02-12T17:17:14.241 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:13.465312+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.f scrub ok 2024-02-12T17:17:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:12.414018+0000 osd.0 (osd.0) 250 : cluster [DBG] 2.13 scrub starts 2024-02-12T17:17:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:12.416993+0000 osd.0 (osd.0) 251 : cluster [DBG] 2.13 scrub ok 2024-02-12T17:17:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:13.159431+0000 mgr.y (mgr.24481) 1039 : cluster [DBG] pgmap v1469: 102 pgs: 1 peering, 2 active+remapped+backfill_toofull, 99 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/576 objects misplaced (0.347%); 36 B/s, 1 objects/s recovering 2024-02-12T17:17:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:13.290740+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.f scrub starts 2024-02-12T17:17:14.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:13.465312+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.f scrub ok 2024-02-12T17:17:14.833 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:15.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:17:15.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:13.557051+0000 osd.5 (osd.5) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:13.557132+0000 osd.5 (osd.5) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:13.945347+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.c scrub starts 2024-02-12T17:17:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:13.946422+0000 osd.5 (osd.5) 330 : cluster [DBG] 2.c scrub ok 2024-02-12T17:17:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.125319+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-02-12T17:17:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.269785+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.269841+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.329140+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:17:15.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.332621+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:17:15.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.661509+0000 osd.1 (osd.1) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.492 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.661571+0000 osd.1 (osd.1) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:15.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:13.557051+0000 osd.5 (osd.5) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:13.557132+0000 osd.5 (osd.5) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:13.945347+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.c scrub starts 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:13.946422+0000 osd.5 (osd.5) 330 : cluster [DBG] 2.c scrub ok 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.125319+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.269785+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.269841+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.329140+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.332621+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.661509+0000 osd.1 (osd.1) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.661571+0000 osd.1 (osd.1) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:15.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:13.557051+0000 osd.5 (osd.5) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:13.557132+0000 osd.5 (osd.5) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:15.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:13.945347+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.c scrub starts 2024-02-12T17:17:15.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:13.946422+0000 osd.5 (osd.5) 330 : cluster [DBG] 2.c scrub ok 2024-02-12T17:17:15.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.125319+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-02-12T17:17:15.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.269785+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.269841+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:15.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.329140+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:17:15.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.332621+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:17:15.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.661509+0000 osd.1 (osd.1) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:15.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.661571+0000 osd.1 (osd.1) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.399733+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.402599+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.824189+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.11 deep-scrub starts 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.825430+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.11 deep-scrub ok 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.969858+0000 osd.5 (osd.5) 331 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:14.972769+0000 osd.5 (osd.5) 332 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:15.131251+0000 mon.a (mon.0) 2307 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:15.160061+0000 mgr.y (mgr.24481) 1040 : cluster [DBG] pgmap v1472: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/576 objects misplaced (0.347%); 36 B/s, 1 objects/s recovering 2024-02-12T17:17:16.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:15.170467+0000 mgr.y (mgr.24481) 1041 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.399733+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.402599+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.824189+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.11 deep-scrub starts 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.825430+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.11 deep-scrub ok 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.969858+0000 osd.5 (osd.5) 331 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:14.972769+0000 osd.5 (osd.5) 332 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:15.131251+0000 mon.a (mon.0) 2307 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:15.160061+0000 mgr.y (mgr.24481) 1040 : cluster [DBG] pgmap v1472: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/576 objects misplaced (0.347%); 36 B/s, 1 objects/s recovering 2024-02-12T17:17:16.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:15.170467+0000 mgr.y (mgr.24481) 1041 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.399733+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.402599+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.824189+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.11 deep-scrub starts 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.825430+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.11 deep-scrub ok 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.969858+0000 osd.5 (osd.5) 331 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:14.972769+0000 osd.5 (osd.5) 332 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:15.131251+0000 mon.a (mon.0) 2307 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:15.160061+0000 mgr.y (mgr.24481) 1040 : cluster [DBG] pgmap v1472: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/576 objects misplaced (0.347%); 36 B/s, 1 objects/s recovering 2024-02-12T17:17:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:15.170467+0000 mgr.y (mgr.24481) 1041 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:17.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:16.137462+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-02-12T17:17:17.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:16.323700+0000 osd.3 (osd.3) 256 : cluster [DBG] 2.e scrub starts 2024-02-12T17:17:17.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:16.326644+0000 osd.3 (osd.3) 257 : cluster [DBG] 2.e scrub ok 2024-02-12T17:17:17.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:16.813098+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-02-12T17:17:17.228 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:16.137462+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-02-12T17:17:17.229 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:16.323700+0000 osd.3 (osd.3) 256 : cluster [DBG] 2.e scrub starts 2024-02-12T17:17:17.229 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:16.326644+0000 osd.3 (osd.3) 257 : cluster [DBG] 2.e scrub ok 2024-02-12T17:17:17.229 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:16.813098+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-02-12T17:17:17.458 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:17.459 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:16.137462+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-02-12T17:17:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:16.323700+0000 osd.3 (osd.3) 256 : cluster [DBG] 2.e scrub starts 2024-02-12T17:17:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:16.326644+0000 osd.3 (osd.3) 257 : cluster [DBG] 2.e scrub ok 2024-02-12T17:17:17.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:16.813098+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:16.214265+0000 osd.0 (osd.0) 252 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:16.214327+0000 osd.0 (osd.0) 253 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:16.425731+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:16.466649+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:17.160716+0000 mgr.y (mgr.24481) 1042 : cluster [DBG] pgmap v1475: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:17.366778+0000 osd.3 (osd.3) 258 : cluster [DBG] 6.b scrub starts 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:17.504499+0000 osd.3 (osd.3) 259 : cluster [DBG] 6.b scrub ok 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:17.537338+0000 mon.a (mon.0) 2310 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:17.747918+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:17.748927+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:17.948010+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.33:0/1361720980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:17.948348+0000 mon.a (mon.0) 2311 : audit [INF] from='client.36290 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:16.214265+0000 osd.0 (osd.0) 252 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:16.214327+0000 osd.0 (osd.0) 253 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:16.425731+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:16.466649+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:17.160716+0000 mgr.y (mgr.24481) 1042 : cluster [DBG] pgmap v1475: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:17.366778+0000 osd.3 (osd.3) 258 : cluster [DBG] 6.b scrub starts 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:17.504499+0000 osd.3 (osd.3) 259 : cluster [DBG] 6.b scrub ok 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:17.537338+0000 mon.a (mon.0) 2310 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:17.747918+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:17.748927+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:17:18.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:17.948010+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.33:0/1361720980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:17:18.280 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:17.948348+0000 mon.a (mon.0) 2311 : audit [INF] from='client.36290 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:17:18.366 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:16.214265+0000 osd.0 (osd.0) 252 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:16.214327+0000 osd.0 (osd.0) 253 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:16.425731+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:16.466649+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:17.160716+0000 mgr.y (mgr.24481) 1042 : cluster [DBG] pgmap v1475: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:17.366778+0000 osd.3 (osd.3) 258 : cluster [DBG] 6.b scrub starts 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:17.504499+0000 osd.3 (osd.3) 259 : cluster [DBG] 6.b scrub ok 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:17.537338+0000 mon.a (mon.0) 2310 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:17.747918+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:17.748927+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:17:18.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:17.948010+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.33:0/1361720980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:17:18.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:17.948348+0000 mon.a (mon.0) 2311 : audit [INF] from='client.36290 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T17:17:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-02-12T17:17:18.979 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-12T17:17:18.980 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-02-12T17:17:18.980 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd in 4 2024-02-12T17:17:19.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:17.338821+0000 osd.7 (osd.7) 329 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:17:19.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:17.341821+0000 osd.7 (osd.7) 330 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:17:19.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:18.134778+0000 mon.a (mon.0) 2312 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:19.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:18.540518+0000 mon.a (mon.0) 2313 : audit [INF] from='client.36290 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:17:19.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:18.540584+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-02-12T17:17:19.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:17.338821+0000 osd.7 (osd.7) 329 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:17:19.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:17.341821+0000 osd.7 (osd.7) 330 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:17:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:18.134778+0000 mon.a (mon.0) 2312 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:18.540518+0000 mon.a (mon.0) 2313 : audit [INF] from='client.36290 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:17:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:18.540584+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-02-12T17:17:19.301 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T17:17:19.362 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:19.489 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:17.338821+0000 osd.7 (osd.7) 329 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:17:19.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:17.341821+0000 osd.7 (osd.7) 330 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:17:19.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:18.134778+0000 mon.a (mon.0) 2312 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:19.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:18.540518+0000 mon.a (mon.0) 2313 : audit [INF] from='client.36290 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T17:17:19.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:18.540584+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-02-12T17:17:20.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:18.417246+0000 osd.0 (osd.0) 256 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:17:20.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:18.420172+0000 osd.0 (osd.0) 257 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:17:20.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:19.161126+0000 mgr.y (mgr.24481) 1043 : cluster [DBG] pgmap v1477: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail 2024-02-12T17:17:20.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:19.295226+0000 osd.7 (osd.7) 331 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:17:20.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:19.298519+0000 osd.7 (osd.7) 332 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:17:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:18.417246+0000 osd.0 (osd.0) 256 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:17:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:18.420172+0000 osd.0 (osd.0) 257 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:17:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:19.161126+0000 mgr.y (mgr.24481) 1043 : cluster [DBG] pgmap v1477: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail 2024-02-12T17:17:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:19.295226+0000 osd.7 (osd.7) 331 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:17:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:19.298519+0000 osd.7 (osd.7) 332 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:17:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:18.417246+0000 osd.0 (osd.0) 256 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:17:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:18.420172+0000 osd.0 (osd.0) 257 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:17:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:19.161126+0000 mgr.y (mgr.24481) 1043 : cluster [DBG] pgmap v1477: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail 2024-02-12T17:17:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:19.295226+0000 osd.7 (osd.7) 331 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:17:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:19.298519+0000 osd.7 (osd.7) 332 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:17:21.149 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:17:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:20] "GET /metrics HTTP/1.1" 200 174457 "" "Prometheus/2.33.4" 2024-02-12T17:17:21.270 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:21.271 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:21.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:19.458469+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.b scrub starts 2024-02-12T17:17:21.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:19.461455+0000 osd.0 (osd.0) 259 : cluster [DBG] 5.b scrub ok 2024-02-12T17:17:21.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:20.287313+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.b scrub starts 2024-02-12T17:17:21.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:20.290445+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.b scrub ok 2024-02-12T17:17:21.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:19.458469+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.b scrub starts 2024-02-12T17:17:21.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:19.461455+0000 osd.0 (osd.0) 259 : cluster [DBG] 5.b scrub ok 2024-02-12T17:17:21.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:20.287313+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.b scrub starts 2024-02-12T17:17:21.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:20.290445+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.b scrub ok 2024-02-12T17:17:21.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:19.458469+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.b scrub starts 2024-02-12T17:17:21.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:19.461455+0000 osd.0 (osd.0) 259 : cluster [DBG] 5.b scrub ok 2024-02-12T17:17:21.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:20.287313+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.b scrub starts 2024-02-12T17:17:21.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:20.290445+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.b scrub ok 2024-02-12T17:17:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:20.424950+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.f scrub starts 2024-02-12T17:17:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:20.426654+0000 osd.0 (osd.0) 261 : cluster [DBG] 2.f scrub ok 2024-02-12T17:17:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:21.161576+0000 mgr.y (mgr.24481) 1044 : cluster [DBG] pgmap v1478: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:17:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:21.278190+0000 osd.7 (osd.7) 335 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:17:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:21.300664+0000 osd.7 (osd.7) 336 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:17:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:21.428219+0000 osd.4 (osd.4) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:21.428274+0000 osd.4 (osd.4) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:22.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:21.816699+0000 mon.a (mon.0) 2315 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:20.424950+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.f scrub starts 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:20.426654+0000 osd.0 (osd.0) 261 : cluster [DBG] 2.f scrub ok 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:21.161576+0000 mgr.y (mgr.24481) 1044 : cluster [DBG] pgmap v1478: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:21.278190+0000 osd.7 (osd.7) 335 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:21.300664+0000 osd.7 (osd.7) 336 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:21.428219+0000 osd.4 (osd.4) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:21.428274+0000 osd.4 (osd.4) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:21.816699+0000 mon.a (mon.0) 2315 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-02-12T17:17:22.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:20.424950+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.f scrub starts 2024-02-12T17:17:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:20.426654+0000 osd.0 (osd.0) 261 : cluster [DBG] 2.f scrub ok 2024-02-12T17:17:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:21.161576+0000 mgr.y (mgr.24481) 1044 : cluster [DBG] pgmap v1478: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:17:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:21.278190+0000 osd.7 (osd.7) 335 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:17:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:21.300664+0000 osd.7 (osd.7) 336 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:17:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:21.428219+0000 osd.4 (osd.4) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:17:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:21.428274+0000 osd.4 (osd.4) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:17:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:21.816699+0000 mon.a (mon.0) 2315 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-02-12T17:17:22.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:17:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:17:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:22.293511+0000 osd.7 (osd.7) 337 : cluster [DBG] 5.9 deep-scrub starts 2024-02-12T17:17:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:22.297764+0000 osd.7 (osd.7) 338 : cluster [DBG] 5.9 deep-scrub ok 2024-02-12T17:17:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:22.700904+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub starts 2024-02-12T17:17:23.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:22.704485+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.f scrub ok 2024-02-12T17:17:23.509 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:22.293511+0000 osd.7 (osd.7) 337 : cluster [DBG] 5.9 deep-scrub starts 2024-02-12T17:17:23.510 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:22.297764+0000 osd.7 (osd.7) 338 : cluster [DBG] 5.9 deep-scrub ok 2024-02-12T17:17:23.510 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:22.700904+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub starts 2024-02-12T17:17:23.510 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:22.704485+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.f scrub ok 2024-02-12T17:17:23.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:22.293511+0000 osd.7 (osd.7) 337 : cluster [DBG] 5.9 deep-scrub starts 2024-02-12T17:17:23.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:22.297764+0000 osd.7 (osd.7) 338 : cluster [DBG] 5.9 deep-scrub ok 2024-02-12T17:17:23.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:22.700904+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub starts 2024-02-12T17:17:23.510 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:22.704485+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.f scrub ok 2024-02-12T17:17:23.511 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:23.539 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T17:17:23.818 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:24.449 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:23.162040+0000 mgr.y (mgr.24481) 1045 : cluster [DBG] pgmap v1480: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 806 B/s rd, 0 op/s 2024-02-12T17:17:24.450 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:23.378226+0000 osd.3 (osd.3) 260 : cluster [DBG] 6.c scrub starts 2024-02-12T17:17:24.450 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:23.381689+0000 osd.3 (osd.3) 261 : cluster [DBG] 6.c scrub ok 2024-02-12T17:17:24.450 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:23.720035+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:17:24.450 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:23.723052+0000 osd.1 (osd.1) 33 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:23.162040+0000 mgr.y (mgr.24481) 1045 : cluster [DBG] pgmap v1480: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 806 B/s rd, 0 op/s 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:23.378226+0000 osd.3 (osd.3) 260 : cluster [DBG] 6.c scrub starts 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:23.381689+0000 osd.3 (osd.3) 261 : cluster [DBG] 6.c scrub ok 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:23.720035+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:23.723052+0000 osd.1 (osd.1) 33 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:17:24.451+0000 7fd96a9de700 -1 mon.a@0(leader).osd e726 definitely_dead 0 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:23.162040+0000 mgr.y (mgr.24481) 1045 : cluster [DBG] pgmap v1480: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 806 B/s rd, 0 op/s 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:23.378226+0000 osd.3 (osd.3) 260 : cluster [DBG] 6.c scrub starts 2024-02-12T17:17:24.454 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:23.381689+0000 osd.3 (osd.3) 261 : cluster [DBG] 6.c scrub ok 2024-02-12T17:17:24.455 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:23.720035+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:17:24.455 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:23.723052+0000 osd.1 (osd.1) 33 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:17:25.186 INFO:teuthology.orchestra.run.smithi033.stderr:marked in osd.4. 2024-02-12T17:17:25.357 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:24.452641+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? v1:172.21.15.33:0/2079395456' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:17:25.357 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:24.452962+0000 mon.a (mon.0) 2316 : audit [DBG] from='client.36296 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:17:25.357 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:24.696603+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.a scrub starts 2024-02-12T17:17:25.357 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:24.699604+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.a scrub ok 2024-02-12T17:17:25.357 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:24.452641+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? v1:172.21.15.33:0/2079395456' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:17:25.357 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:24.452962+0000 mon.a (mon.0) 2316 : audit [DBG] from='client.36296 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:17:25.357 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:24.696603+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.a scrub starts 2024-02-12T17:17:25.357 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:24.699604+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.a scrub ok 2024-02-12T17:17:25.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:17:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:17:25.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:24.452641+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? v1:172.21.15.33:0/2079395456' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:17:25.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:24.452962+0000 mon.a (mon.0) 2316 : audit [DBG] from='client.36296 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-12T17:17:25.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:24.696603+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.a scrub starts 2024-02-12T17:17:25.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:24.699604+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.a scrub ok 2024-02-12T17:17:25.817 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:25.817 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:26.060 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-02-12T17:17:26.060 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.control 2024-02-12T17:17:26.061 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.control pgp_num 22 2024-02-12T17:17:26.431 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:24.858653+0000 osd.6 (osd.6) 198 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:24.859469+0000 osd.6 (osd.6) 199 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.067171+0000 osd.5 (osd.5) 333 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.070635+0000 osd.5 (osd.5) 334 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.162613+0000 mgr.y (mgr.24481) 1046 : cluster [DBG] pgmap v1481: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:25.165581+0000 mon.a (mon.0) 2317 : audit [INF] from='client.36296 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.165635+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:25.182075+0000 mgr.y (mgr.24481) 1047 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.663053+0000 osd.1 (osd.1) 36 : cluster [DBG] 5.12 scrub starts 2024-02-12T17:17:26.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.665988+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.12 scrub ok 2024-02-12T17:17:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:24.858653+0000 osd.6 (osd.6) 198 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:17:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:24.859469+0000 osd.6 (osd.6) 199 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:17:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.067171+0000 osd.5 (osd.5) 333 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:17:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.070635+0000 osd.5 (osd.5) 334 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:17:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.162613+0000 mgr.y (mgr.24481) 1046 : cluster [DBG] pgmap v1481: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:25.165581+0000 mon.a (mon.0) 2317 : audit [INF] from='client.36296 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.165635+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:25.182075+0000 mgr.y (mgr.24481) 1047 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.663053+0000 osd.1 (osd.1) 36 : cluster [DBG] 5.12 scrub starts 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.665988+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.12 scrub ok 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:24.858653+0000 osd.6 (osd.6) 198 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:24.859469+0000 osd.6 (osd.6) 199 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.067171+0000 osd.5 (osd.5) 333 : cluster [DBG] 4.1 scrub starts 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.070635+0000 osd.5 (osd.5) 334 : cluster [DBG] 4.1 scrub ok 2024-02-12T17:17:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.162613+0000 mgr.y (mgr.24481) 1046 : cluster [DBG] pgmap v1481: 102 pgs: 1 active+remapped+backfill_toofull, 101 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:17:26.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:25.165581+0000 mon.a (mon.0) 2317 : audit [INF] from='client.36296 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-12T17:17:26.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.165635+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-12T17:17:26.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:25.182075+0000 mgr.y (mgr.24481) 1047 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:26.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.663053+0000 osd.1 (osd.1) 36 : cluster [DBG] 5.12 scrub starts 2024-02-12T17:17:26.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.665988+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.12 scrub ok 2024-02-12T17:17:26.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T17:17:26.933 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:27.187 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-02-12T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.527093+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.633148+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.841388+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:25.842420+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:26.177431+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-12T17:17:27.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:26.488824+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.33:0/3519195777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:27.191 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:26.489289+0000 mon.a (mon.0) 2320 : audit [INF] from='client.26194 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.527093+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:17:27.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.633148+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:17:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.841388+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:17:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:25.842420+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:17:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:26.177431+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-12T17:17:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:26.488824+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.33:0/3519195777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:27.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:26.489289+0000 mon.a (mon.0) 2320 : audit [INF] from='client.26194 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:27.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.527093+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:17:27.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.633148+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:17:27.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.841388+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.7 scrub starts 2024-02-12T17:17:27.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:25.842420+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.7 scrub ok 2024-02-12T17:17:27.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:26.177431+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-12T17:17:27.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:26.488824+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.33:0/3519195777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:27.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:26.489289+0000 mon.a (mon.0) 2320 : audit [INF] from='client.26194 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T17:17:28.390 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.163059+0000 mgr.y (mgr.24481) 1048 : cluster [DBG] pgmap v1484: 102 pgs: 1 active+recovery_wait+degraded, 1 active+recovering, 2 remapped+peering, 98 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/576 objects degraded (0.347%); 2/576 objects misplaced (0.347%); 73 B/s, 2 objects/s recovering 2024-02-12T17:17:28.390 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:27.173374+0000 mon.a (mon.0) 2321 : audit [INF] from='client.26194 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:17:28.390 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.173452+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-12T17:17:28.390 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.174910+0000 osd.7 (osd.7) 339 : cluster [DBG] 5.10 starting backfill to osd.4 from (0'0,0'0] MAX to 430'18 2024-02-12T17:17:28.390 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.175257+0000 osd.5 (osd.5) 335 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 668'27 2024-02-12T17:17:28.390 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.175385+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 642'31 2024-02-12T17:17:28.390 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.175708+0000 osd.3 (osd.3) 262 : cluster [INF] 2.d continuing backfill to osd.1 from (128'2,128'2] MIN to 128'2 2024-02-12T17:17:28.390 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.176133+0000 osd.0 (osd.0) 264 : cluster [DBG] 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'4 2024-02-12T17:17:28.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.176198+0000 osd.3 (osd.3) 263 : cluster [DBG] 5.0 starting backfill to osd.4 from (0'0,0'0] MAX to 706'33 2024-02-12T17:17:28.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.177118+0000 osd.7 (osd.7) 340 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 726'36 2024-02-12T17:17:28.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.177206+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.5 starting backfill to osd.4 from (0'0,0'0] MAX to 410'15 2024-02-12T17:17:28.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.817316+0000 mon.a (mon.0) 2323 : cluster [WRN] Health check failed: Degraded data redundancy: 2/576 objects degraded (0.347%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:17:28.391 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:27.817388+0000 mon.a (mon.0) 2324 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:17:28.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.163059+0000 mgr.y (mgr.24481) 1048 : cluster [DBG] pgmap v1484: 102 pgs: 1 active+recovery_wait+degraded, 1 active+recovering, 2 remapped+peering, 98 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/576 objects degraded (0.347%); 2/576 objects misplaced (0.347%); 73 B/s, 2 objects/s recovering 2024-02-12T17:17:28.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:27.173374+0000 mon.a (mon.0) 2321 : audit [INF] from='client.26194 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.173452+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.174910+0000 osd.7 (osd.7) 339 : cluster [DBG] 5.10 starting backfill to osd.4 from (0'0,0'0] MAX to 430'18 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.175257+0000 osd.5 (osd.5) 335 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 668'27 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.175385+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 642'31 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.175708+0000 osd.3 (osd.3) 262 : cluster [INF] 2.d continuing backfill to osd.1 from (128'2,128'2] MIN to 128'2 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.176133+0000 osd.0 (osd.0) 264 : cluster [DBG] 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'4 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.176198+0000 osd.3 (osd.3) 263 : cluster [DBG] 5.0 starting backfill to osd.4 from (0'0,0'0] MAX to 706'33 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.177118+0000 osd.7 (osd.7) 340 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 726'36 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.177206+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.5 starting backfill to osd.4 from (0'0,0'0] MAX to 410'15 2024-02-12T17:17:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.817316+0000 mon.a (mon.0) 2323 : cluster [WRN] Health check failed: Degraded data redundancy: 2/576 objects degraded (0.347%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:17:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:27.817388+0000 mon.a (mon.0) 2324 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:17:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.163059+0000 mgr.y (mgr.24481) 1048 : cluster [DBG] pgmap v1484: 102 pgs: 1 active+recovery_wait+degraded, 1 active+recovering, 2 remapped+peering, 98 active+clean; 5.7 KiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/576 objects degraded (0.347%); 2/576 objects misplaced (0.347%); 73 B/s, 2 objects/s recovering 2024-02-12T17:17:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:27.173374+0000 mon.a (mon.0) 2321 : audit [INF] from='client.26194 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T17:17:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.173452+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-12T17:17:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.174910+0000 osd.7 (osd.7) 339 : cluster [DBG] 5.10 starting backfill to osd.4 from (0'0,0'0] MAX to 430'18 2024-02-12T17:17:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.175257+0000 osd.5 (osd.5) 335 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 668'27 2024-02-12T17:17:28.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.175385+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 642'31 2024-02-12T17:17:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.175708+0000 osd.3 (osd.3) 262 : cluster [INF] 2.d continuing backfill to osd.1 from (128'2,128'2] MIN to 128'2 2024-02-12T17:17:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.176133+0000 osd.0 (osd.0) 264 : cluster [DBG] 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 91'4 2024-02-12T17:17:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.176198+0000 osd.3 (osd.3) 263 : cluster [DBG] 5.0 starting backfill to osd.4 from (0'0,0'0] MAX to 706'33 2024-02-12T17:17:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.177118+0000 osd.7 (osd.7) 340 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 726'36 2024-02-12T17:17:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.177206+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.5 starting backfill to osd.4 from (0'0,0'0] MAX to 410'15 2024-02-12T17:17:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.817316+0000 mon.a (mon.0) 2323 : cluster [WRN] Health check failed: Degraded data redundancy: 2/576 objects degraded (0.347%), 1 pg degraded (PG_DEGRADED) 2024-02-12T17:17:28.530 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:27.817388+0000 mon.a (mon.0) 2324 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:17:28.636 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:28.637 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:29.204 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 5 pgp_num to 22 2024-02-12T17:17:29.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:28.171289+0000 osd.7 (osd.7) 341 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:17:29.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:28.191053+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-12T17:17:29.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:28.290607+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:17:29.408 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:28.718136+0000 mon.a (mon.0) 2326 : audit [INF] from='client.? v1:172.21.15.33:0/1487275732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:29.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T17:17:29.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:28.171289+0000 osd.7 (osd.7) 341 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:17:29.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:28.191053+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-12T17:17:29.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:28.290607+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:17:29.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:28.718136+0000 mon.a (mon.0) 2326 : audit [INF] from='client.? v1:172.21.15.33:0/1487275732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:28.171289+0000 osd.7 (osd.7) 341 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:17:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:28.191053+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-12T17:17:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:28.290607+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:17:29.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:28.718136+0000 mon.a (mon.0) 2326 : audit [INF] from='client.? v1:172.21.15.33:0/1487275732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:29.699 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:30.000 INFO:tasks.thrashosds.thrasher:fixing pg num pool .rgw.root 2024-02-12T17:17:30.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set .rgw.root pgp_num 22 2024-02-12T17:17:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:28.786078+0000 osd.6 (osd.6) 202 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:17:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:28.786912+0000 osd.6 (osd.6) 203 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:17:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:29.163869+0000 mgr.y (mgr.24481) 1049 : cluster [DBG] pgmap v1487: 102 pgs: 4 active+remapped+backfill_toofull, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering, 2 remapped+peering, 93 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2/576 objects degraded (0.347%); 6/576 objects misplaced (1.042%); 281 B/s, 11 objects/s recovering 2024-02-12T17:17:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:29.189236+0000 mon.a (mon.0) 2327 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:29.195306+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? v1:172.21.15.33:0/1487275732' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:29.195385+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-12T17:17:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:28.786078+0000 osd.6 (osd.6) 202 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:17:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:28.786912+0000 osd.6 (osd.6) 203 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:17:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:29.163869+0000 mgr.y (mgr.24481) 1049 : cluster [DBG] pgmap v1487: 102 pgs: 4 active+remapped+backfill_toofull, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering, 2 remapped+peering, 93 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2/576 objects degraded (0.347%); 6/576 objects misplaced (1.042%); 281 B/s, 11 objects/s recovering 2024-02-12T17:17:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:29.189236+0000 mon.a (mon.0) 2327 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:29.195306+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? v1:172.21.15.33:0/1487275732' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:29.195385+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-12T17:17:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:28.786078+0000 osd.6 (osd.6) 202 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:17:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:28.786912+0000 osd.6 (osd.6) 203 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:17:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:29.163869+0000 mgr.y (mgr.24481) 1049 : cluster [DBG] pgmap v1487: 102 pgs: 4 active+remapped+backfill_toofull, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering, 2 remapped+peering, 93 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 2/576 objects degraded (0.347%); 6/576 objects misplaced (1.042%); 281 B/s, 11 objects/s recovering 2024-02-12T17:17:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:29.189236+0000 mon.a (mon.0) 2327 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:29.195306+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? v1:172.21.15.33:0/1487275732' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:29.195385+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-12T17:17:30.291 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:31.144 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:17:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:30] "GET /metrics HTTP/1.1" 200 174457 "" "Prometheus/2.33.4" 2024-02-12T17:17:31.257 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:30.229391+0000 osd.7 (osd.7) 343 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:17:31.258 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:30.441987+0000 osd.3 (osd.3) 264 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:17:31.258 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:30.445456+0000 osd.3 (osd.3) 265 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:17:31.258 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:30.602954+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d deep-scrub starts 2024-02-12T17:17:31.258 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:30.604578+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.d deep-scrub ok 2024-02-12T17:17:31.498 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:31.499 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:31.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:30.229391+0000 osd.7 (osd.7) 343 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:17:31.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:30.441987+0000 osd.3 (osd.3) 264 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:17:31.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:30.445456+0000 osd.3 (osd.3) 265 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:17:31.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:30.602954+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d deep-scrub starts 2024-02-12T17:17:31.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:30.604578+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.d deep-scrub ok 2024-02-12T17:17:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:30.229391+0000 osd.7 (osd.7) 343 : cluster [DBG] 6.3 scrub starts 2024-02-12T17:17:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:30.441987+0000 osd.3 (osd.3) 264 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:17:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:30.445456+0000 osd.3 (osd.3) 265 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:17:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:30.602954+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d deep-scrub starts 2024-02-12T17:17:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:30.604578+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.d deep-scrub ok 2024-02-12T17:17:32.221 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T17:17:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:30.407255+0000 osd.7 (osd.7) 344 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:17:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:31.164494+0000 mgr.y (mgr.24481) 1050 : cluster [DBG] pgmap v1489: 102 pgs: 5 active+remapped+backfill_toofull, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering, 1 remapped+peering, 93 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 616 B/s rd, 0 op/s; 2/576 objects degraded (0.347%); 6/576 objects misplaced (1.042%); 0 B/s, 13 objects/s recovering 2024-02-12T17:17:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:31.223947+0000 osd.7 (osd.7) 345 : cluster [DBG] 6.5 deep-scrub starts 2024-02-12T17:17:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:31.297067+0000 osd.7 (osd.7) 346 : cluster [DBG] 6.5 deep-scrub ok 2024-02-12T17:17:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:31.439778+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:17:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:31.442667+0000 osd.3 (osd.3) 267 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:17:32.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:31.817287+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-12T17:17:32.495 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:30.407255+0000 osd.7 (osd.7) 344 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:17:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:31.164494+0000 mgr.y (mgr.24481) 1050 : cluster [DBG] pgmap v1489: 102 pgs: 5 active+remapped+backfill_toofull, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering, 1 remapped+peering, 93 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 616 B/s rd, 0 op/s; 2/576 objects degraded (0.347%); 6/576 objects misplaced (1.042%); 0 B/s, 13 objects/s recovering 2024-02-12T17:17:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:31.223947+0000 osd.7 (osd.7) 345 : cluster [DBG] 6.5 deep-scrub starts 2024-02-12T17:17:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:31.297067+0000 osd.7 (osd.7) 346 : cluster [DBG] 6.5 deep-scrub ok 2024-02-12T17:17:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:31.439778+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:17:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:31.442667+0000 osd.3 (osd.3) 267 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:17:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:31.817287+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-12T17:17:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:30.407255+0000 osd.7 (osd.7) 344 : cluster [DBG] 6.3 scrub ok 2024-02-12T17:17:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:31.164494+0000 mgr.y (mgr.24481) 1050 : cluster [DBG] pgmap v1489: 102 pgs: 5 active+remapped+backfill_toofull, 1 peering, 1 active+recovery_wait+degraded, 1 active+recovering, 1 remapped+peering, 93 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 616 B/s rd, 0 op/s; 2/576 objects degraded (0.347%); 6/576 objects misplaced (1.042%); 0 B/s, 13 objects/s recovering 2024-02-12T17:17:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:31.223947+0000 osd.7 (osd.7) 345 : cluster [DBG] 6.5 deep-scrub starts 2024-02-12T17:17:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:31.297067+0000 osd.7 (osd.7) 346 : cluster [DBG] 6.5 deep-scrub ok 2024-02-12T17:17:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:31.439778+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:17:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:31.442667+0000 osd.3 (osd.3) 267 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:17:32.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:31.817287+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-12T17:17:32.765 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:17:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:17:33.214 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 2 pgp_num to 22 2024-02-12T17:17:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:32.217796+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.d scrub starts 2024-02-12T17:17:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:32.221243+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.d scrub ok 2024-02-12T17:17:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:32.268272+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.33:0/2191061884' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:32.268554+0000 mon.a (mon.0) 2331 : audit [INF] from='client.36308 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:32.473077+0000 osd.0 (osd.0) 266 : cluster [DBG] 4.b scrub starts 2024-02-12T17:17:33.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:32.476527+0000 osd.0 (osd.0) 267 : cluster [DBG] 4.b scrub ok 2024-02-12T17:17:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:32.487812+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:17:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:32.537479+0000 mon.a (mon.0) 2332 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:32.567461+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:17:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:32.682114+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:17:33.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:32.685202+0000 osd.1 (osd.1) 41 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:17:33.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:32.217796+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.d scrub starts 2024-02-12T17:17:33.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:32.221243+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.d scrub ok 2024-02-12T17:17:33.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:32.268272+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.33:0/2191061884' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:33.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:32.268554+0000 mon.a (mon.0) 2331 : audit [INF] from='client.36308 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:33.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:32.473077+0000 osd.0 (osd.0) 266 : cluster [DBG] 4.b scrub starts 2024-02-12T17:17:33.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:32.476527+0000 osd.0 (osd.0) 267 : cluster [DBG] 4.b scrub ok 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:32.487812+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:32.537479+0000 mon.a (mon.0) 2332 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:32.567461+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:32.682114+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:32.685202+0000 osd.1 (osd.1) 41 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:32.217796+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.d scrub starts 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:32.221243+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.d scrub ok 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:32.268272+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.33:0/2191061884' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:33.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:32.268554+0000 mon.a (mon.0) 2331 : audit [INF] from='client.36308 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:33.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:32.473077+0000 osd.0 (osd.0) 266 : cluster [DBG] 4.b scrub starts 2024-02-12T17:17:33.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:32.476527+0000 osd.0 (osd.0) 267 : cluster [DBG] 4.b scrub ok 2024-02-12T17:17:33.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:32.487812+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:17:33.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:32.537479+0000 mon.a (mon.0) 2332 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:33.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:32.567461+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:17:33.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:32.682114+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:17:33.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:32.685202+0000 osd.1 (osd.1) 41 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:17:34.042 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2024-02-12T17:17:34.043 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.meta pgp_num 22 2024-02-12T17:17:34.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.133667+0000 osd.5 (osd.5) 337 : cluster [DBG] 6.a scrub starts 2024-02-12T17:17:34.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.165141+0000 mgr.y (mgr.24481) 1051 : cluster [DBG] pgmap v1491: 102 pgs: 6 active+remapped+backfill_toofull, 1 peering, 95 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 512 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 0 B/s, 9 objects/s recovering 2024-02-12T17:17:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.196946+0000 osd.5 (osd.5) 338 : cluster [DBG] 6.a scrub ok 2024-02-12T17:17:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:33.209683+0000 mon.a (mon.0) 2333 : audit [INF] from='client.36308 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.209762+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-12T17:17:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.448424+0000 osd.3 (osd.3) 270 : cluster [DBG] 4.c deep-scrub starts 2024-02-12T17:17:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.454198+0000 osd.3 (osd.3) 271 : cluster [DBG] 4.c deep-scrub ok 2024-02-12T17:17:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.535742+0000 mon.a (mon.0) 2335 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/576 objects degraded (0.347%), 1 pg degraded) 2024-02-12T17:17:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.725184+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.e scrub starts 2024-02-12T17:17:34.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:33.773407+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.e scrub ok 2024-02-12T17:17:34.246 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:34.246 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:34.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.133667+0000 osd.5 (osd.5) 337 : cluster [DBG] 6.a scrub starts 2024-02-12T17:17:34.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.165141+0000 mgr.y (mgr.24481) 1051 : cluster [DBG] pgmap v1491: 102 pgs: 6 active+remapped+backfill_toofull, 1 peering, 95 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 512 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 0 B/s, 9 objects/s recovering 2024-02-12T17:17:34.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.196946+0000 osd.5 (osd.5) 338 : cluster [DBG] 6.a scrub ok 2024-02-12T17:17:34.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:33.209683+0000 mon.a (mon.0) 2333 : audit [INF] from='client.36308 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.209762+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-12T17:17:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.448424+0000 osd.3 (osd.3) 270 : cluster [DBG] 4.c deep-scrub starts 2024-02-12T17:17:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.454198+0000 osd.3 (osd.3) 271 : cluster [DBG] 4.c deep-scrub ok 2024-02-12T17:17:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.535742+0000 mon.a (mon.0) 2335 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/576 objects degraded (0.347%), 1 pg degraded) 2024-02-12T17:17:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.725184+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.e scrub starts 2024-02-12T17:17:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:33.773407+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.e scrub ok 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.133667+0000 osd.5 (osd.5) 337 : cluster [DBG] 6.a scrub starts 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.165141+0000 mgr.y (mgr.24481) 1051 : cluster [DBG] pgmap v1491: 102 pgs: 6 active+remapped+backfill_toofull, 1 peering, 95 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 512 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 0 B/s, 9 objects/s recovering 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.196946+0000 osd.5 (osd.5) 338 : cluster [DBG] 6.a scrub ok 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:33.209683+0000 mon.a (mon.0) 2333 : audit [INF] from='client.36308 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.209762+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.448424+0000 osd.3 (osd.3) 270 : cluster [DBG] 4.c deep-scrub starts 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.454198+0000 osd.3 (osd.3) 271 : cluster [DBG] 4.c deep-scrub ok 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.535742+0000 mon.a (mon.0) 2335 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/576 objects degraded (0.347%), 1 pg degraded) 2024-02-12T17:17:34.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.725184+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.e scrub starts 2024-02-12T17:17:34.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:33.773407+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.e scrub ok 2024-02-12T17:17:34.333 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:34.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T17:17:35.199 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:35.452 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:34.286736+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:17:35.452 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:34.288293+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:17:35.453 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:34.420348+0000 osd.0 (osd.0) 268 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:17:35.453 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:34.424244+0000 osd.0 (osd.0) 269 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:17:35.453 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:17:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:17:35.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:34.286736+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:17:35.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:34.288293+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:17:35.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:34.420348+0000 osd.0 (osd.0) 268 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:17:35.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:34.424244+0000 osd.0 (osd.0) 269 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:17:35.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:34.286736+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:17:35.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:34.288293+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:17:35.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:34.420348+0000 osd.0 (osd.0) 268 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:17:35.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:34.424244+0000 osd.0 (osd.0) 269 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:17:36.349 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:35.165758+0000 mgr.y (mgr.24481) 1052 : cluster [DBG] pgmap v1493: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4/576 objects misplaced (0.694%); 0 B/s, 4 objects/s recovering 2024-02-12T17:17:36.350 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:35.192477+0000 mgr.y (mgr.24481) 1053 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:36.350 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:35.397202+0000 osd.3 (osd.3) 272 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:17:36.350 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:35.400852+0000 osd.3 (osd.3) 273 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:17:36.350 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:35.729062+0000 osd.1 (osd.1) 44 : cluster [DBG] 5.7 deep-scrub starts 2024-02-12T17:17:36.350 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:35.731952+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.7 deep-scrub ok 2024-02-12T17:17:36.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:35.165758+0000 mgr.y (mgr.24481) 1052 : cluster [DBG] pgmap v1493: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4/576 objects misplaced (0.694%); 0 B/s, 4 objects/s recovering 2024-02-12T17:17:36.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:35.192477+0000 mgr.y (mgr.24481) 1053 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:36.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:35.397202+0000 osd.3 (osd.3) 272 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:17:36.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:35.400852+0000 osd.3 (osd.3) 273 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:17:36.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:35.729062+0000 osd.1 (osd.1) 44 : cluster [DBG] 5.7 deep-scrub starts 2024-02-12T17:17:36.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:35.731952+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.7 deep-scrub ok 2024-02-12T17:17:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:35.165758+0000 mgr.y (mgr.24481) 1052 : cluster [DBG] pgmap v1493: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4/576 objects misplaced (0.694%); 0 B/s, 4 objects/s recovering 2024-02-12T17:17:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:35.192477+0000 mgr.y (mgr.24481) 1053 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:35.397202+0000 osd.3 (osd.3) 272 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:17:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:35.400852+0000 osd.3 (osd.3) 273 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:17:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:35.729062+0000 osd.1 (osd.1) 44 : cluster [DBG] 5.7 deep-scrub starts 2024-02-12T17:17:36.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:35.731952+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.7 deep-scrub ok 2024-02-12T17:17:36.826 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 6 pgp_num to 22 2024-02-12T17:17:36.878 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:36.878 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:37.475 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:36.293607+0000 osd.4 (osd.4) 215 : cluster [DBG] 5.a scrub starts 2024-02-12T17:17:37.475 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:36.296518+0000 osd.4 (osd.4) 216 : cluster [DBG] 5.a scrub ok 2024-02-12T17:17:37.476 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:36.306524+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? v1:172.21.15.33:0/589113926' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:37.476 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:36.398484+0000 osd.3 (osd.3) 274 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:17:37.476 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:36.402012+0000 osd.3 (osd.3) 275 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:17:37.476 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:36.814519+0000 mon.a (mon.0) 2337 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:37.477 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:36.819463+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? v1:172.21.15.33:0/589113926' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:37.477 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:36.819512+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-12T17:17:37.495 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T17:17:37.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:36.293607+0000 osd.4 (osd.4) 215 : cluster [DBG] 5.a scrub starts 2024-02-12T17:17:37.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:36.296518+0000 osd.4 (osd.4) 216 : cluster [DBG] 5.a scrub ok 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:36.306524+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? v1:172.21.15.33:0/589113926' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:36.398484+0000 osd.3 (osd.3) 274 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:36.402012+0000 osd.3 (osd.3) 275 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:36.814519+0000 mon.a (mon.0) 2337 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:36.819463+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? v1:172.21.15.33:0/589113926' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:36.819512+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:36.293607+0000 osd.4 (osd.4) 215 : cluster [DBG] 5.a scrub starts 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:36.296518+0000 osd.4 (osd.4) 216 : cluster [DBG] 5.a scrub ok 2024-02-12T17:17:37.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:36.306524+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? v1:172.21.15.33:0/589113926' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-12T17:17:37.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:36.398484+0000 osd.3 (osd.3) 274 : cluster [DBG] 4.1d scrub starts 2024-02-12T17:17:37.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:36.402012+0000 osd.3 (osd.3) 275 : cluster [DBG] 4.1d scrub ok 2024-02-12T17:17:37.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:36.814519+0000 mon.a (mon.0) 2337 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:17:37.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:36.819463+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? v1:172.21.15.33:0/589113926' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "22"}]': finished 2024-02-12T17:17:37.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:36.819512+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-12T17:17:37.538 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-12T17:17:37.768 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:37.824 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:38.256 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:37.166481+0000 mgr.y (mgr.24481) 1054 : cluster [DBG] pgmap v1495: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 18 B/s, 0 objects/s recovering 2024-02-12T17:17:38.256 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:37.270254+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:17:38.256 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:37.273148+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:17:38.257 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:37.719761+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.c scrub starts 2024-02-12T17:17:38.257 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:37.720676+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.c scrub ok 2024-02-12T17:17:38.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:37.166481+0000 mgr.y (mgr.24481) 1054 : cluster [DBG] pgmap v1495: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 18 B/s, 0 objects/s recovering 2024-02-12T17:17:38.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:37.270254+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:17:38.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:37.273148+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:17:38.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:37.719761+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.c scrub starts 2024-02-12T17:17:38.339 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:37.720676+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.c scrub ok 2024-02-12T17:17:38.341 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:37.166481+0000 mgr.y (mgr.24481) 1054 : cluster [DBG] pgmap v1495: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 18 B/s, 0 objects/s recovering 2024-02-12T17:17:38.341 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:37.270254+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:17:38.341 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:37.273148+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:17:38.341 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:37.719761+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.c scrub starts 2024-02-12T17:17:38.341 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:37.720676+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.c scrub ok 2024-02-12T17:17:39.491 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:39.492 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:39.682 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a: {} 2024-02-12T17:17:39.692 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b: {} 2024-02-12T17:17:39.701 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c: {} 2024-02-12T17:17:39.702 INFO:teuthology.orchestra.run.smithi033.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-12T17:17:39.702 INFO:teuthology.orchestra.run.smithi033.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-12T17:17:39.702 INFO:teuthology.orchestra.run.smithi033.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-12T17:17:40.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T17:17:40.351 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:39.167217+0000 mgr.y (mgr.24481) 1055 : cluster [DBG] pgmap v1496: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 696 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 14 B/s, 0 objects/s recovering 2024-02-12T17:17:40.351 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:39.671407+0000 mon.a (mon.0) 2340 : audit [DBG] from='client.? v1:172.21.15.33:0/2144567552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T17:17:40.352 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:39.167217+0000 mgr.y (mgr.24481) 1055 : cluster [DBG] pgmap v1496: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 696 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 14 B/s, 0 objects/s recovering 2024-02-12T17:17:40.352 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:39.671407+0000 mon.a (mon.0) 2340 : audit [DBG] from='client.? v1:172.21.15.33:0/2144567552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T17:17:40.363 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:40.372 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T17:17:40.489 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:39.167217+0000 mgr.y (mgr.24481) 1055 : cluster [DBG] pgmap v1496: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 696 B/s rd, 0 op/s; 4/576 objects misplaced (0.694%); 14 B/s, 0 objects/s recovering 2024-02-12T17:17:40.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:39.671407+0000 mon.a (mon.0) 2340 : audit [DBG] from='client.? v1:172.21.15.33:0/2144567552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T17:17:40.658 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:41.134 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:17:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:40] "GET /metrics HTTP/1.1" 200 174467 "" "Prometheus/2.33.4" 2024-02-12T17:17:42.055 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:42.055 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:42.367 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:41.167698+0000 mgr.y (mgr.24481) 1056 : cluster [DBG] pgmap v1497: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:42.367 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:41.242167+0000 osd.5 (osd.5) 339 : cluster [DBG] 6.7 deep-scrub starts 2024-02-12T17:17:42.368 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:41.167698+0000 mgr.y (mgr.24481) 1056 : cluster [DBG] pgmap v1497: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:42.368 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:41.242167+0000 osd.5 (osd.5) 339 : cluster [DBG] 6.7 deep-scrub starts 2024-02-12T17:17:42.403 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:41.167698+0000 mgr.y (mgr.24481) 1056 : cluster [DBG] pgmap v1497: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:42.403 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:41.242167+0000 osd.5 (osd.5) 339 : cluster [DBG] 6.7 deep-scrub starts 2024-02-12T17:17:42.717 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:17:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:17:42.735 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T17:17:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:43.230 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:41.406165+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.7 deep-scrub ok 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:42.240979+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:42.243866+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:42.460843+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:42.479303+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:42.611963+0000 osd.1 (osd.1) 48 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:42.615519+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:42.818032+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.33:0/1338161746' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:43.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:42.818335+0000 mon.a (mon.0) 2341 : audit [INF] from='client.36332 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:41.406165+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.7 deep-scrub ok 2024-02-12T17:17:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:42.240979+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:17:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:42.243866+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:17:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:42.460843+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:17:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:42.479303+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:17:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:42.611963+0000 osd.1 (osd.1) 48 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:17:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:42.615519+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:42.818032+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.33:0/1338161746' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:42.818335+0000 mon.a (mon.0) 2341 : audit [INF] from='client.36332 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:41.406165+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.7 deep-scrub ok 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:42.240979+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:42.243866+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:42.460843+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:42.479303+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:42.611963+0000 osd.1 (osd.1) 48 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:42.615519+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:42.818032+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.33:0/1338161746' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:42.818335+0000 mon.a (mon.0) 2341 : audit [INF] from='client.36332 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:44.048 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-12T17:17:44.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:43.168203+0000 mgr.y (mgr.24481) 1057 : cluster [DBG] pgmap v1498: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:44.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:43.201802+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T17:17:44.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:43.205600+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T17:17:44.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:43.224903+0000 mon.a (mon.0) 2342 : audit [INF] from='client.36332 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:17:44.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:43.225021+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-12T17:17:44.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:43.615657+0000 mon.a (mon.0) 2344 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:44.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:43.168203+0000 mgr.y (mgr.24481) 1057 : cluster [DBG] pgmap v1498: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:43.201802+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T17:17:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:43.205600+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T17:17:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:43.224903+0000 mon.a (mon.0) 2342 : audit [INF] from='client.36332 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:17:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:43.225021+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-12T17:17:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:43.615657+0000 mon.a (mon.0) 2344 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:44.333 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:43.168203+0000 mgr.y (mgr.24481) 1057 : cluster [DBG] pgmap v1498: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:43.201802+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T17:17:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:43.205600+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T17:17:44.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:43.224903+0000 mon.a (mon.0) 2342 : audit [INF] from='client.36332 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:17:44.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:43.225021+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-12T17:17:44.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:43.615657+0000 mon.a (mon.0) 2344 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:44.507 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:44.507 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:45.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T17:17:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:45.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:17:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:17:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:45.169004+0000 mgr.y (mgr.24481) 1058 : cluster [DBG] pgmap v1500: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:45.202493+0000 mgr.y (mgr.24481) 1059 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:46.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:45.169004+0000 mgr.y (mgr.24481) 1058 : cluster [DBG] pgmap v1500: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:46.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:45.202493+0000 mgr.y (mgr.24481) 1059 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:46.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:45.169004+0000 mgr.y (mgr.24481) 1058 : cluster [DBG] pgmap v1500: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:46.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:45.202493+0000 mgr.y (mgr.24481) 1059 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:46.826 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-12T17:17:47.090 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:47.090 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:46.444265+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? v1:172.21.15.33:0/85386828' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:46.623151+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.d scrub starts 2024-02-12T17:17:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:46.625232+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.d scrub ok 2024-02-12T17:17:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:46.821080+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? v1:172.21.15.33:0/85386828' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:17:47.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:46.821148+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-12T17:17:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:46.444265+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? v1:172.21.15.33:0/85386828' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:46.623151+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.d scrub starts 2024-02-12T17:17:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:46.625232+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.d scrub ok 2024-02-12T17:17:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:46.821080+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? v1:172.21.15.33:0/85386828' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:17:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:46.821148+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-12T17:17:47.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:46.444265+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? v1:172.21.15.33:0/85386828' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T17:17:47.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:46.623151+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.d scrub starts 2024-02-12T17:17:47.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:46.625232+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.d scrub ok 2024-02-12T17:17:47.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:46.821080+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? v1:172.21.15.33:0/85386828' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-12T17:17:47.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:46.821148+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-12T17:17:47.697 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T17:17:47.728 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T17:17:47.984 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:48.002 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:47.169657+0000 mgr.y (mgr.24481) 1060 : cluster [DBG] pgmap v1502: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:47.539079+0000 mon.a (mon.0) 2348 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:48.923 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:47.169657+0000 mgr.y (mgr.24481) 1060 : cluster [DBG] pgmap v1502: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:48.924 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:47.539079+0000 mon.a (mon.0) 2348 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:47.169657+0000 mgr.y (mgr.24481) 1060 : cluster [DBG] pgmap v1502: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:48.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:47.539079+0000 mon.a (mon.0) 2348 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:17:49.673 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:49.673 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:50.345 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T17:17:50.549 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T17:17:50.615 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:50.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:49.170423+0000 mgr.y (mgr.24481) 1061 : cluster [DBG] pgmap v1503: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:50.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:50.017194+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? v1:172.21.15.33:0/503154343' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:17:50.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:49.170423+0000 mgr.y (mgr.24481) 1061 : cluster [DBG] pgmap v1503: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:50.879 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:50.017194+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? v1:172.21.15.33:0/503154343' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:17:50.879 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:49.170423+0000 mgr.y (mgr.24481) 1061 : cluster [DBG] pgmap v1503: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:50.880 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:50.017194+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? v1:172.21.15.33:0/503154343' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:17:51.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:17:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:50] "GET /metrics HTTP/1.1" 200 174460 "" "Prometheus/2.33.4" 2024-02-12T17:17:51.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-12T17:17:51.557 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:50.544120+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? v1:172.21.15.33:0/503154343' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:17:51.557 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:50.544201+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-12T17:17:51.557 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:50.544120+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? v1:172.21.15.33:0/503154343' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:17:51.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:50.544201+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-12T17:17:51.615 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:51.835 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:50.544120+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? v1:172.21.15.33:0/503154343' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:17:51.835 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:50.544201+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-12T17:17:52.659 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:52.660 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:52.679 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:17:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:17:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:17:52.679 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:51.170893+0000 mgr.y (mgr.24481) 1062 : cluster [DBG] pgmap v1505: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:52.679 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:51.823681+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-12T17:17:52.679 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:52.084501+0000 osd.5 (osd.5) 345 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:17:52.679 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:52.086361+0000 osd.5 (osd.5) 346 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:17:53.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:51.170893+0000 mgr.y (mgr.24481) 1062 : cluster [DBG] pgmap v1505: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:53.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:51.823681+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-12T17:17:53.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:52.084501+0000 osd.5 (osd.5) 345 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:17:53.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:52.086361+0000 osd.5 (osd.5) 346 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:17:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:51.170893+0000 mgr.y (mgr.24481) 1062 : cluster [DBG] pgmap v1505: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:51.823681+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-12T17:17:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:52.084501+0000 osd.5 (osd.5) 345 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:17:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:52.086361+0000 osd.5 (osd.5) 346 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:17:53.328 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T17:17:53.575 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-12T17:17:53.605 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T17:17:53.829 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:53.052974+0000 osd.5 (osd.5) 347 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:17:53.829 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:53.056748+0000 osd.5 (osd.5) 348 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:17:53.829 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:53.512447+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.33:0/2707774119' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:17:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:53.052974+0000 osd.5 (osd.5) 347 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:17:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:53.056748+0000 osd.5 (osd.5) 348 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:17:54.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:53.512447+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.33:0/2707774119' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:17:54.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:53.052974+0000 osd.5 (osd.5) 347 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:17:54.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:53.056748+0000 osd.5 (osd.5) 348 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:17:54.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:53.512447+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.33:0/2707774119' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T17:17:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:53.171388+0000 mgr.y (mgr.24481) 1063 : cluster [DBG] pgmap v1507: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:53.564267+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? v1:172.21.15.33:0/2707774119' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:17:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:53.564321+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-12T17:17:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:53.592604+0000 osd.3 (osd.3) 278 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:17:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:53.595964+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:17:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:54.565320+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-12T17:17:54.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:53.171388+0000 mgr.y (mgr.24481) 1063 : cluster [DBG] pgmap v1507: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:53.564267+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? v1:172.21.15.33:0/2707774119' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:53.564321+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:53.592604+0000 osd.3 (osd.3) 278 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:53.595964+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:54.565320+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:53.171388+0000 mgr.y (mgr.24481) 1063 : cluster [DBG] pgmap v1507: 102 pgs: 6 active+remapped+backfill_toofull, 96 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:53.564267+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? v1:172.21.15.33:0/2707774119' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:53.564321+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:53.592604+0000 osd.3 (osd.3) 278 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:17:54.959 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:53.595964+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:17:54.960 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:54.565320+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-12T17:17:54.982 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 3 1 2024-02-12T17:17:55.245 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:55.245 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:55.265 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:55.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:17:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:17:55.774 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:55.573618+0000 mon.a (mon.0) 2357 : cluster [WRN] Health check failed: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-02-12T17:17:55.774 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:55.573618+0000 mon.a (mon.0) 2357 : cluster [WRN] Health check failed: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-02-12T17:17:55.865 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:55.573618+0000 mon.a (mon.0) 2357 : cluster [WRN] Health check failed: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-02-12T17:17:55.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T17:17:56.770 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:54.712226+0000 osd.2 (osd.2) 237 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:17:56.770 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:54.714299+0000 osd.2 (osd.2) 238 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:17:56.770 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:55.171849+0000 mgr.y (mgr.24481) 1064 : cluster [DBG] pgmap v1510: 102 pgs: 14 peering, 6 active+remapped+backfill_toofull, 82 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:55.210525+0000 mgr.y (mgr.24481) 1065 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:55.618485+0000 osd.3 (osd.3) 280 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:55.621952+0000 osd.3 (osd.3) 281 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:54.712226+0000 osd.2 (osd.2) 237 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:54.714299+0000 osd.2 (osd.2) 238 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:55.171849+0000 mgr.y (mgr.24481) 1064 : cluster [DBG] pgmap v1510: 102 pgs: 14 peering, 6 active+remapped+backfill_toofull, 82 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:55.210525+0000 mgr.y (mgr.24481) 1065 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:55.618485+0000 osd.3 (osd.3) 280 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:17:56.771 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:55.621952+0000 osd.3 (osd.3) 281 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:17:56.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:54.712226+0000 osd.2 (osd.2) 237 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:17:56.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:54.714299+0000 osd.2 (osd.2) 238 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:17:56.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:55.171849+0000 mgr.y (mgr.24481) 1064 : cluster [DBG] pgmap v1510: 102 pgs: 14 peering, 6 active+remapped+backfill_toofull, 82 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:56.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:55.210525+0000 mgr.y (mgr.24481) 1065 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:17:56.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:55.618485+0000 osd.3 (osd.3) 280 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:17:56.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:55.621952+0000 osd.3 (osd.3) 281 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:17:57.023 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:57.839 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.3 to 1 (10000) 2024-02-12T17:17:57.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:56.635877+0000 osd.3 (osd.3) 282 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:17:57.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:56.658863+0000 osd.3 (osd.3) 283 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:17:57.890 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:57.177393+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? v1:172.21.15.33:0/651110009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:17:57.890 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:56.635877+0000 osd.3 (osd.3) 282 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:17:57.890 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:56.658863+0000 osd.3 (osd.3) 283 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:17:57.891 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:57.177393+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? v1:172.21.15.33:0/651110009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:17:57.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:56.635877+0000 osd.3 (osd.3) 282 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:17:57.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:56.658863+0000 osd.3 (osd.3) 283 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:17:57.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:57.177393+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? v1:172.21.15.33:0/651110009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:17:58.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:57.172330+0000 mgr.y (mgr.24481) 1066 : cluster [DBG] pgmap v1511: 102 pgs: 14 peering, 6 active+remapped+backfill_toofull, 82 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:58.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:57.832295+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? v1:172.21.15.33:0/651110009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:17:58.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:57.832428+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-12T17:17:58.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:58.066042+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:17:58.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:58.069212+0000 osd.5 (osd.5) 350 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:17:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:57.172330+0000 mgr.y (mgr.24481) 1066 : cluster [DBG] pgmap v1511: 102 pgs: 14 peering, 6 active+remapped+backfill_toofull, 82 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:57.832295+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? v1:172.21.15.33:0/651110009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:17:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:57.832428+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-12T17:17:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:58.066042+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:17:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:58.069212+0000 osd.5 (osd.5) 350 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:17:58.911 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-12T17:17:58.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:57.172330+0000 mgr.y (mgr.24481) 1066 : cluster [DBG] pgmap v1511: 102 pgs: 14 peering, 6 active+remapped+backfill_toofull, 82 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/579 objects misplaced (0.691%) 2024-02-12T17:17:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:57.832295+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? v1:172.21.15.33:0/651110009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:17:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:57.832428+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-12T17:17:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:58.066042+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:17:58.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:58.069212+0000 osd.5 (osd.5) 350 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:17:59.407 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:17:59.408 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:17:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:17:59.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:58.567743+0000 osd.3 (osd.3) 284 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:17:59.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:58.571334+0000 osd.3 (osd.3) 285 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:17:59.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:58.839148+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-12T17:17:59.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:17:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:17:59.402027+0000 mon.c (mon.2) 331 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:17:59.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:58.567743+0000 osd.3 (osd.3) 284 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:17:59.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:58.571334+0000 osd.3 (osd.3) 285 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:17:59.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:58.839148+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-12T17:17:59.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:17:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:17:59.402027+0000 mon.c (mon.2) 331 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:17:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:58.567743+0000 osd.3 (osd.3) 284 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:17:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:58.571334+0000 osd.3 (osd.3) 285 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:17:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:58.839148+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-12T17:17:59.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:17:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:17:59.402027+0000 mon.c (mon.2) 331 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:18:00.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:58.719070+0000 osd.2 (osd.2) 239 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:18:00.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:58.720118+0000 osd.2 (osd.2) 240 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:59.172712+0000 mgr.y (mgr.24481) 1067 : cluster [DBG] pgmap v1514: 102 pgs: 2 peering, 3 active+remapped+backfill_toofull, 97 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 913 B/s rd, 0 op/s; 1/579 objects misplaced (0.173%) 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:59.462075+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:59.465141+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:59.566452+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:59.569377+0000 osd.3 (osd.3) 287 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:59.829980+0000 mon.a (mon.0) 2362 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:59.830020+0000 mon.a (mon.0) 2363 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:17:59.986229+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.f scrub starts 2024-02-12T17:18:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:00.005110+0000 osd.5 (osd.5) 352 : cluster [DBG] 5.f scrub ok 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:58.719070+0000 osd.2 (osd.2) 239 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:58.720118+0000 osd.2 (osd.2) 240 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:59.172712+0000 mgr.y (mgr.24481) 1067 : cluster [DBG] pgmap v1514: 102 pgs: 2 peering, 3 active+remapped+backfill_toofull, 97 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 913 B/s rd, 0 op/s; 1/579 objects misplaced (0.173%) 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:59.462075+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:59.465141+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:59.566452+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:59.569377+0000 osd.3 (osd.3) 287 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:59.829980+0000 mon.a (mon.0) 2362 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:59.830020+0000 mon.a (mon.0) 2363 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2024-02-12T17:18:00.779 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:17:59.986229+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.f scrub starts 2024-02-12T17:18:00.780 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:00.005110+0000 osd.5 (osd.5) 352 : cluster [DBG] 5.f scrub ok 2024-02-12T17:18:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:58.719070+0000 osd.2 (osd.2) 239 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:18:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:58.720118+0000 osd.2 (osd.2) 240 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:18:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:59.172712+0000 mgr.y (mgr.24481) 1067 : cluster [DBG] pgmap v1514: 102 pgs: 2 peering, 3 active+remapped+backfill_toofull, 97 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 913 B/s rd, 0 op/s; 1/579 objects misplaced (0.173%) 2024-02-12T17:18:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:59.462075+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:18:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:59.465141+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:18:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:59.566452+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:18:00.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:59.569377+0000 osd.3 (osd.3) 287 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:18:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:59.829980+0000 mon.a (mon.0) 2362 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:59.830020+0000 mon.a (mon.0) 2363 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2024-02-12T17:18:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:17:59.986229+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.f scrub starts 2024-02-12T17:18:00.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:00.005110+0000 osd.5 (osd.5) 352 : cluster [DBG] 5.f scrub ok 2024-02-12T17:18:01.205 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:18:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:00] "GET /metrics HTTP/1.1" 200 174460 "" "Prometheus/2.33.4" 2024-02-12T17:18:01.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T17:18:02.647 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-12T17:18:02.657 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:02.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:01.173403+0000 mgr.y (mgr.24481) 1068 : cluster [DBG] pgmap v1515: 102 pgs: 2 peering, 3 active+remapped+backfill_toofull, 97 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/579 objects misplaced (0.173%); 0 B/s, 0 objects/s recovering 2024-02-12T17:18:02.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:01.711066+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.33:0/1714637945' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:18:02.933 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:01.711374+0000 mon.a (mon.0) 2364 : audit [INF] from='client.36362 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:18:02.934 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:01.173403+0000 mgr.y (mgr.24481) 1068 : cluster [DBG] pgmap v1515: 102 pgs: 2 peering, 3 active+remapped+backfill_toofull, 97 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/579 objects misplaced (0.173%); 0 B/s, 0 objects/s recovering 2024-02-12T17:18:02.934 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:01.711066+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.33:0/1714637945' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:18:02.934 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:01.711374+0000 mon.a (mon.0) 2364 : audit [INF] from='client.36362 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:18:02.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:18:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:18:02.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:01.173403+0000 mgr.y (mgr.24481) 1068 : cluster [DBG] pgmap v1515: 102 pgs: 2 peering, 3 active+remapped+backfill_toofull, 97 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/579 objects misplaced (0.173%); 0 B/s, 0 objects/s recovering 2024-02-12T17:18:02.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:01.711066+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.33:0/1714637945' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:18:02.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:01.711374+0000 mon.a (mon.0) 2364 : audit [INF] from='client.36362 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T17:18:03.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T17:18:03.957 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:02.550089+0000 osd.3 (osd.3) 288 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:18:03.957 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:02.553572+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:18:03.958 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:02.640528+0000 mon.a (mon.0) 2365 : audit [INF] from='client.36362 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:18:03.958 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:02.640609+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-12T17:18:03.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:02.550089+0000 osd.3 (osd.3) 288 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:18:03.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:02.553572+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:18:03.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:02.640528+0000 mon.a (mon.0) 2365 : audit [INF] from='client.36362 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:18:03.958 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:02.640609+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-12T17:18:03.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:02.550089+0000 osd.3 (osd.3) 288 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:18:03.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:02.553572+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:18:03.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:02.640528+0000 mon.a (mon.0) 2365 : audit [INF] from='client.36362 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T17:18:03.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:02.640609+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-12T17:18:04.914 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:03.173721+0000 mgr.y (mgr.24481) 1069 : cluster [DBG] pgmap v1517: 102 pgs: 2 peering, 2 active+remapped+backfill_toofull, 98 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%); 0 B/s, 0 objects/s recovering 2024-02-12T17:18:04.915 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:03.953439+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.e scrub starts 2024-02-12T17:18:04.915 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:03.955087+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.e scrub ok 2024-02-12T17:18:04.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:03.173721+0000 mgr.y (mgr.24481) 1069 : cluster [DBG] pgmap v1517: 102 pgs: 2 peering, 2 active+remapped+backfill_toofull, 98 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%); 0 B/s, 0 objects/s recovering 2024-02-12T17:18:04.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:03.953439+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.e scrub starts 2024-02-12T17:18:04.915 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:03.955087+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.e scrub ok 2024-02-12T17:18:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:03.173721+0000 mgr.y (mgr.24481) 1069 : cluster [DBG] pgmap v1517: 102 pgs: 2 peering, 2 active+remapped+backfill_toofull, 98 active+clean; 5.7 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%); 0 B/s, 0 objects/s recovering 2024-02-12T17:18:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:03.953439+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.e scrub starts 2024-02-12T17:18:04.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:03.955087+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.e scrub ok 2024-02-12T17:18:05.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:18:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:18:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:03.743081+0000 osd.2 (osd.2) 241 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:18:05.656 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:03.745098+0000 osd.2 (osd.2) 242 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:18:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:03.743081+0000 osd.2 (osd.2) 241 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:18:05.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:03.745098+0000 osd.2 (osd.2) 242 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:18:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:03.743081+0000 osd.2 (osd.2) 241 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:18:06.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:03.745098+0000 osd.2 (osd.2) 242 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:18:06.143 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:18:06.143 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:18:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:05.174276+0000 mgr.y (mgr.24481) 1070 : cluster [DBG] pgmap v1518: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 1/573 objects misplaced (0.175%) 2024-02-12T17:18:06.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:05.221231+0000 mgr.y (mgr.24481) 1071 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:07.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:05.174276+0000 mgr.y (mgr.24481) 1070 : cluster [DBG] pgmap v1518: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 1/573 objects misplaced (0.175%) 2024-02-12T17:18:07.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:05.221231+0000 mgr.y (mgr.24481) 1071 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:07.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:05.174276+0000 mgr.y (mgr.24481) 1070 : cluster [DBG] pgmap v1518: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 1/573 objects misplaced (0.175%) 2024-02-12T17:18:07.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:05.221231+0000 mgr.y (mgr.24481) 1071 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:07.148 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T17:18:07.465 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:07.509 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:07.945 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:06.823487+0000 mon.a (mon.0) 2367 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:07.945 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:06.828889+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-12T17:18:07.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:06.823487+0000 mon.a (mon.0) 2367 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:07.945 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:06.828889+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-12T17:18:07.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:06.823487+0000 mon.a (mon.0) 2367 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:07.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:06.828889+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-12T17:18:08.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:07.174677+0000 mgr.y (mgr.24481) 1072 : cluster [DBG] pgmap v1520: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:08.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:08.073782+0000 osd.5 (osd.5) 355 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:18:08.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:08.075544+0000 osd.5 (osd.5) 356 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:18:08.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:07.174677+0000 mgr.y (mgr.24481) 1072 : cluster [DBG] pgmap v1520: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:08.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:08.073782+0000 osd.5 (osd.5) 355 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:18:08.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:08.075544+0000 osd.5 (osd.5) 356 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:18:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:07.174677+0000 mgr.y (mgr.24481) 1072 : cluster [DBG] pgmap v1520: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:08.073782+0000 osd.5 (osd.5) 355 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:18:08.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:08.075544+0000 osd.5 (osd.5) 356 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:18:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:09.213869+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:18:09.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:09.215771+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:18:10.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:09.213869+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:18:10.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:09.215771+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:18:10.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:09.213869+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.1f scrub starts 2024-02-12T17:18:10.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:09.215771+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.1f scrub ok 2024-02-12T17:18:10.674 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T17:18:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:09.175384+0000 mgr.y (mgr.24481) 1073 : cluster [DBG] pgmap v1521: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:09.453429+0000 osd.0 (osd.0) 270 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:18:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:09.455282+0000 osd.0 (osd.0) 271 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:18:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:10.193804+0000 osd.6 (osd.6) 206 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:18:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:10.196090+0000 osd.6 (osd.6) 207 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:18:10.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:10.198471+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? v1:172.21.15.33:0/3130593025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:18:10.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:09.175384+0000 mgr.y (mgr.24481) 1073 : cluster [DBG] pgmap v1521: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:10.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:09.453429+0000 osd.0 (osd.0) 270 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:18:10.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:09.455282+0000 osd.0 (osd.0) 271 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:18:10.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:10.193804+0000 osd.6 (osd.6) 206 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:18:10.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:10.196090+0000 osd.6 (osd.6) 207 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:18:10.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:10.198471+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? v1:172.21.15.33:0/3130593025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:18:11.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:10] "GET /metrics HTTP/1.1" 200 174459 "" "Prometheus/2.33.4" 2024-02-12T17:18:11.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:09.175384+0000 mgr.y (mgr.24481) 1073 : cluster [DBG] pgmap v1521: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:11.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:09.453429+0000 osd.0 (osd.0) 270 : cluster [DBG] 4.16 scrub starts 2024-02-12T17:18:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:09.455282+0000 osd.0 (osd.0) 271 : cluster [DBG] 4.16 scrub ok 2024-02-12T17:18:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:10.193804+0000 osd.6 (osd.6) 206 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:18:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:10.196090+0000 osd.6 (osd.6) 207 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:18:11.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:10.198471+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? v1:172.21.15.33:0/3130593025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:18:11.164 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-02-12T17:18:11.164 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T17:18:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:10.669186+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? v1:172.21.15.33:0/3130593025' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:18:11.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:10.669253+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-12T17:18:12.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:10.669186+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? v1:172.21.15.33:0/3130593025' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:18:12.004 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:10.669253+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-12T17:18:12.004 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:10.669186+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? v1:172.21.15.33:0/3130593025' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:18:12.005 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:10.669253+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-12T17:18:12.930 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:11.175705+0000 mgr.y (mgr.24481) 1074 : cluster [DBG] pgmap v1523: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:12.931 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:11.829416+0000 mon.a (mon.0) 2372 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-12T17:18:12.931 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:11.901647+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:12.931 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:12.535435+0000 mon.c (mon.2) 333 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:18:12.931 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:12.536100+0000 mon.c (mon.2) 334 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:18:12.931 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:12.540847+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:12.931 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:11.175705+0000 mgr.y (mgr.24481) 1074 : cluster [DBG] pgmap v1523: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:12.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:11.829416+0000 mon.a (mon.0) 2372 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-12T17:18:12.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:11.901647+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:12.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:12.535435+0000 mon.c (mon.2) 333 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:18:12.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:12.536100+0000 mon.c (mon.2) 334 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:18:12.932 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:12.540847+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:12.949 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-12T17:18:12.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:18:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:18:12.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:11.175705+0000 mgr.y (mgr.24481) 1074 : cluster [DBG] pgmap v1523: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:12.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:11.829416+0000 mon.a (mon.0) 2372 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-12T17:18:12.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:11.901647+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:12.535435+0000 mon.c (mon.2) 333 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:18:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:12.536100+0000 mon.c (mon.2) 334 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:18:12.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:12.540847+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:13.228 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:13.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:13.108114+0000 osd.7 (osd.7) 351 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:13.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:13.108169+0000 osd.7 (osd.7) 352 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:13.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:13.108114+0000 osd.7 (osd.7) 351 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:13.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:13.108169+0000 osd.7 (osd.7) 352 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:13.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:13.108114+0000 osd.7 (osd.7) 351 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:13.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:13.108169+0000 osd.7 (osd.7) 352 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:13.176061+0000 mgr.y (mgr.24481) 1075 : cluster [DBG] pgmap v1525: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:14.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:13.685212+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-12T17:18:15.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:13.176061+0000 mgr.y (mgr.24481) 1075 : cluster [DBG] pgmap v1525: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:15.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:13.685212+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-12T17:18:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:13.176061+0000 mgr.y (mgr.24481) 1075 : cluster [DBG] pgmap v1525: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 806 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:14 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:13.685212+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-12T17:18:15.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:18:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:18:15.701 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-12T17:18:15.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:15.148048+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? v1:172.21.15.33:0/3724713582' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:18:16.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:15.148048+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? v1:172.21.15.33:0/3724713582' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:18:16.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:15.148048+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? v1:172.21.15.33:0/3724713582' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T17:18:16.480 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T17:18:16.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:15.176712+0000 mgr.y (mgr.24481) 1076 : cluster [DBG] pgmap v1527: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:16.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:15.225129+0000 mgr.y (mgr.24481) 1077 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:16.705 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:15.691815+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? v1:172.21.15.33:0/3724713582' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:18:16.706 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:15.691863+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-12T17:18:16.769 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:15.176712+0000 mgr.y (mgr.24481) 1076 : cluster [DBG] pgmap v1527: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:15.225129+0000 mgr.y (mgr.24481) 1077 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:15.691815+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? v1:172.21.15.33:0/3724713582' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:18:16.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:15.691863+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-12T17:18:16.999 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:15.176712+0000 mgr.y (mgr.24481) 1076 : cluster [DBG] pgmap v1527: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:16.999 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:15.225129+0000 mgr.y (mgr.24481) 1077 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:17.000 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:15.691815+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? v1:172.21.15.33:0/3724713582' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-12T17:18:17.000 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:15.691863+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-12T17:18:17.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:16.111401+0000 osd.2 (osd.2) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:17.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:16.111481+0000 osd.2 (osd.2) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:16.460184+0000 osd.0 (osd.0) 272 : cluster [DBG] 4.1e deep-scrub starts 2024-02-12T17:18:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:16.464542+0000 osd.0 (osd.0) 273 : cluster [DBG] 4.1e deep-scrub ok 2024-02-12T17:18:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:16.696351+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-12T17:18:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:16.751871+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:18:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:16.752917+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:18:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:16.833020+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-12T17:18:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:17.252599+0000 osd.7 (osd.7) 353 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:18:17.991 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:17.275874+0000 osd.7 (osd.7) 354 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:18:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:16.111401+0000 osd.2 (osd.2) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:16.111481+0000 osd.2 (osd.2) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:16.460184+0000 osd.0 (osd.0) 272 : cluster [DBG] 4.1e deep-scrub starts 2024-02-12T17:18:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:16.464542+0000 osd.0 (osd.0) 273 : cluster [DBG] 4.1e deep-scrub ok 2024-02-12T17:18:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:16.696351+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-12T17:18:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:16.751871+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:18:18.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:16.752917+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:18:18.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:16.833020+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-12T17:18:18.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:17.252599+0000 osd.7 (osd.7) 353 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:18:18.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:17.275874+0000 osd.7 (osd.7) 354 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:18:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:16.111401+0000 osd.2 (osd.2) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:16.111481+0000 osd.2 (osd.2) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:16.460184+0000 osd.0 (osd.0) 272 : cluster [DBG] 4.1e deep-scrub starts 2024-02-12T17:18:18.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:16.464542+0000 osd.0 (osd.0) 273 : cluster [DBG] 4.1e deep-scrub ok 2024-02-12T17:18:18.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:16.696351+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-12T17:18:18.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:16.751871+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:18:18.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:16.752917+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:18:18.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:16.833020+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-12T17:18:18.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:17.252599+0000 osd.7 (osd.7) 353 : cluster [DBG] 6.12 scrub starts 2024-02-12T17:18:18.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:17.275874+0000 osd.7 (osd.7) 354 : cluster [DBG] 6.12 scrub ok 2024-02-12T17:18:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T17:18:18.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:17.177231+0000 mgr.y (mgr.24481) 1078 : cluster [DBG] pgmap v1531: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:19.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:17.177231+0000 mgr.y (mgr.24481) 1078 : cluster [DBG] pgmap v1531: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:19.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:17.177231+0000 mgr.y (mgr.24481) 1078 : cluster [DBG] pgmap v1531: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:19.636 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-12T17:18:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:19.989 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:17.739148+0000 osd.3 (osd.3) 290 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:18:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:17.845512+0000 osd.3 (osd.3) 291 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:18:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:18.442269+0000 osd.3 (osd.3) 292 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:18.442348+0000 osd.3 (osd.3) 293 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:18.762338+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.33:0/221364731' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:18.762682+0000 mon.a (mon.0) 2381 : audit [INF] from='client.36380 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:18.841782+0000 mon.a (mon.0) 2382 : audit [INF] from='client.36380 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:18:19.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:18.841851+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-12T17:18:20.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:17.739148+0000 osd.3 (osd.3) 290 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:18:20.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:17.845512+0000 osd.3 (osd.3) 291 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:18:20.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:18.442269+0000 osd.3 (osd.3) 292 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:20.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:18.442348+0000 osd.3 (osd.3) 293 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:20.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:18.762338+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.33:0/221364731' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:20.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:18.762682+0000 mon.a (mon.0) 2381 : audit [INF] from='client.36380 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:20.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:18.841782+0000 mon.a (mon.0) 2382 : audit [INF] from='client.36380 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:18:20.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:18.841851+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-12T17:18:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:17.739148+0000 osd.3 (osd.3) 290 : cluster [DBG] 6.13 scrub starts 2024-02-12T17:18:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:17.845512+0000 osd.3 (osd.3) 291 : cluster [DBG] 6.13 scrub ok 2024-02-12T17:18:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:18.442269+0000 osd.3 (osd.3) 292 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:18.442348+0000 osd.3 (osd.3) 293 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:18.762338+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.33:0/221364731' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:18.762682+0000 mon.a (mon.0) 2381 : audit [INF] from='client.36380 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:18.841782+0000 mon.a (mon.0) 2382 : audit [INF] from='client.36380 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:18:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:18.841851+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-12T17:18:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:19.177702+0000 mgr.y (mgr.24481) 1079 : cluster [DBG] pgmap v1533: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:19.730560+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 deep-scrub starts 2024-02-12T17:18:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:19.733422+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.4 deep-scrub ok 2024-02-12T17:18:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:19.858399+0000 osd.4 (osd.4) 219 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:18:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:19.861834+0000 osd.4 (osd.4) 220 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:18:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:20.264896+0000 osd.7 (osd.7) 355 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:18:20.990 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:20.267940+0000 osd.7 (osd.7) 356 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:18:21.002 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:20] "GET /metrics HTTP/1.1" 200 174460 "" "Prometheus/2.33.4" 2024-02-12T17:18:21.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:19.177702+0000 mgr.y (mgr.24481) 1079 : cluster [DBG] pgmap v1533: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:21.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:19.730560+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 deep-scrub starts 2024-02-12T17:18:21.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:19.733422+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.4 deep-scrub ok 2024-02-12T17:18:21.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:19.858399+0000 osd.4 (osd.4) 219 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:18:21.002 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:19.861834+0000 osd.4 (osd.4) 220 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:20.264896+0000 osd.7 (osd.7) 355 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:20.267940+0000 osd.7 (osd.7) 356 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:19.177702+0000 mgr.y (mgr.24481) 1079 : cluster [DBG] pgmap v1533: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:19.730560+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 deep-scrub starts 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:19.733422+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.4 deep-scrub ok 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:19.858399+0000 osd.4 (osd.4) 219 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:19.861834+0000 osd.4 (osd.4) 220 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:20.264896+0000 osd.7 (osd.7) 355 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:18:21.003 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:20.267940+0000 osd.7 (osd.7) 356 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:18:22.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:20.639007+0000 osd.1 (osd.1) 56 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:22.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:20.639115+0000 osd.1 (osd.1) 57 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:22.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:20.827262+0000 osd.2 (osd.2) 245 : cluster [DBG] 5.8 deep-scrub starts 2024-02-12T17:18:22.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:20.828113+0000 osd.2 (osd.2) 246 : cluster [DBG] 5.8 deep-scrub ok 2024-02-12T17:18:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:20.845621+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-12T17:18:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:21.178124+0000 mgr.y (mgr.24481) 1080 : cluster [DBG] pgmap v1535: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:20.639007+0000 osd.1 (osd.1) 56 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:20.639115+0000 osd.1 (osd.1) 57 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:20.827262+0000 osd.2 (osd.2) 245 : cluster [DBG] 5.8 deep-scrub starts 2024-02-12T17:18:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:20.828113+0000 osd.2 (osd.2) 246 : cluster [DBG] 5.8 deep-scrub ok 2024-02-12T17:18:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:20.845621+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-12T17:18:22.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:21 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:21.178124+0000 mgr.y (mgr.24481) 1080 : cluster [DBG] pgmap v1535: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:20.639007+0000 osd.1 (osd.1) 56 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:20.639115+0000 osd.1 (osd.1) 57 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:20.827262+0000 osd.2 (osd.2) 245 : cluster [DBG] 5.8 deep-scrub starts 2024-02-12T17:18:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:20.828113+0000 osd.2 (osd.2) 246 : cluster [DBG] 5.8 deep-scrub ok 2024-02-12T17:18:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:20.845621+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-12T17:18:22.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:21 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:21.178124+0000 mgr.y (mgr.24481) 1080 : cluster [DBG] pgmap v1535: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:22.859 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-12T17:18:22.864 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:18:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:18:23.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:21.815671+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.c deep-scrub starts 2024-02-12T17:18:23.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:21.817031+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.c deep-scrub ok 2024-02-12T17:18:23.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:21.893645+0000 osd.4 (osd.4) 221 : cluster [DBG] 4.e scrub starts 2024-02-12T17:18:23.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:21.897201+0000 osd.4 (osd.4) 222 : cluster [DBG] 4.e scrub ok 2024-02-12T17:18:23.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:22.086857+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? v1:172.21.15.33:0/3024952808' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:23.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:21.815671+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.c deep-scrub starts 2024-02-12T17:18:23.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:21.817031+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.c deep-scrub ok 2024-02-12T17:18:23.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:21.893645+0000 osd.4 (osd.4) 221 : cluster [DBG] 4.e scrub starts 2024-02-12T17:18:23.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:21.897201+0000 osd.4 (osd.4) 222 : cluster [DBG] 4.e scrub ok 2024-02-12T17:18:23.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:22.086857+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? v1:172.21.15.33:0/3024952808' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:23.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:21.815671+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.c deep-scrub starts 2024-02-12T17:18:23.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:21.817031+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.c deep-scrub ok 2024-02-12T17:18:23.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:21.893645+0000 osd.4 (osd.4) 221 : cluster [DBG] 4.e scrub starts 2024-02-12T17:18:23.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:21.897201+0000 osd.4 (osd.4) 222 : cluster [DBG] 4.e scrub ok 2024-02-12T17:18:23.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:22.086857+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? v1:172.21.15.33:0/3024952808' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T17:18:23.702 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-12T17:18:23.997 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:24.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:22.854466+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? v1:172.21.15.33:0/3024952808' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:18:24.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:22.854522+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:22.861840+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.d scrub starts 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:23.035490+0000 osd.5 (osd.5) 358 : cluster [DBG] 6.d scrub ok 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:23.160468+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:23.160523+0000 osd.0 (osd.0) 275 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:23.178545+0000 mgr.y (mgr.24481) 1081 : cluster [DBG] pgmap v1537: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:22.854466+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? v1:172.21.15.33:0/3024952808' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:22.854522+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:22.861840+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.d scrub starts 2024-02-12T17:18:24.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:23.035490+0000 osd.5 (osd.5) 358 : cluster [DBG] 6.d scrub ok 2024-02-12T17:18:24.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:23.160468+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:24.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:23.160523+0000 osd.0 (osd.0) 275 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:24.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:23.178545+0000 mgr.y (mgr.24481) 1081 : cluster [DBG] pgmap v1537: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:22.854466+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? v1:172.21.15.33:0/3024952808' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-12T17:18:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:22.854522+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-12T17:18:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:22.861840+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.d scrub starts 2024-02-12T17:18:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:23.035490+0000 osd.5 (osd.5) 358 : cluster [DBG] 6.d scrub ok 2024-02-12T17:18:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:23.160468+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:23.160523+0000 osd.0 (osd.0) 275 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:24.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:23.178545+0000 mgr.y (mgr.24481) 1081 : cluster [DBG] pgmap v1537: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:25.233 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:23.679228+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:18:25.234 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:23.682187+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:18:25.234 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:23.797704+0000 osd.2 (osd.2) 249 : cluster [DBG] 6.f deep-scrub starts 2024-02-12T17:18:25.234 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:23.875292+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-12T17:18:25.234 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:24.021899+0000 osd.2 (osd.2) 250 : cluster [DBG] 6.f deep-scrub ok 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:23.679228+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:23.682187+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:23.797704+0000 osd.2 (osd.2) 249 : cluster [DBG] 6.f deep-scrub starts 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:23.875292+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:24.021899+0000 osd.2 (osd.2) 250 : cluster [DBG] 6.f deep-scrub ok 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:23.679228+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:23.682187+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:23.797704+0000 osd.2 (osd.2) 249 : cluster [DBG] 6.f deep-scrub starts 2024-02-12T17:18:25.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:23.875292+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-12T17:18:25.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:24.021899+0000 osd.2 (osd.2) 250 : cluster [DBG] 6.f deep-scrub ok 2024-02-12T17:18:25.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:18:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:18:25.983 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:25.179303+0000 mgr.y (mgr.24481) 1082 : cluster [DBG] pgmap v1539: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:25.983 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:25.232698+0000 mgr.y (mgr.24481) 1083 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:25.983 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:25.179303+0000 mgr.y (mgr.24481) 1082 : cluster [DBG] pgmap v1539: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:25.984 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:25.232698+0000 mgr.y (mgr.24481) 1083 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:25.179303+0000 mgr.y (mgr.24481) 1082 : cluster [DBG] pgmap v1539: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:26.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:25.232698+0000 mgr.y (mgr.24481) 1083 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:26.893 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.6 to 1 (10000) 2024-02-12T17:18:27.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:26.414642+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? v1:172.21.15.33:0/1966545992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:27.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:26.414642+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? v1:172.21.15.33:0/1966545992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:27.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:26.414642+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? v1:172.21.15.33:0/1966545992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:27.759 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-12T17:18:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:26.358704+0000 osd.0 (osd.0) 276 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:18:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:26.360568+0000 osd.0 (osd.0) 277 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:18:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:26.682302+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:18:28.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:26.685753+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:18:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:26.729040+0000 osd.3 (osd.3) 294 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:18:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:26.732584+0000 osd.3 (osd.3) 295 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:18:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:26.889259+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? v1:172.21.15.33:0/1966545992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:18:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:26.889340+0000 mon.a (mon.0) 2391 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-12T17:18:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:27.105895+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:27.105982+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:27.179732+0000 mgr.y (mgr.24481) 1084 : cluster [DBG] pgmap v1541: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:26.358704+0000 osd.0 (osd.0) 276 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:26.360568+0000 osd.0 (osd.0) 277 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:26.682302+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:26.685753+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:26.729040+0000 osd.3 (osd.3) 294 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:26.732584+0000 osd.3 (osd.3) 295 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:26.889259+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? v1:172.21.15.33:0/1966545992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:26.889340+0000 mon.a (mon.0) 2391 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-12T17:18:28.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:27.105895+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:28.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:27.105982+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:28.030 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:27.179732+0000 mgr.y (mgr.24481) 1084 : cluster [DBG] pgmap v1541: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:28.047 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:26.358704+0000 osd.0 (osd.0) 276 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:18:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:26.360568+0000 osd.0 (osd.0) 277 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:18:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:26.682302+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.7 scrub starts 2024-02-12T17:18:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:26.685753+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.7 scrub ok 2024-02-12T17:18:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:26.729040+0000 osd.3 (osd.3) 294 : cluster [DBG] 4.17 scrub starts 2024-02-12T17:18:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:26.732584+0000 osd.3 (osd.3) 295 : cluster [DBG] 4.17 scrub ok 2024-02-12T17:18:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:26.889259+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? v1:172.21.15.33:0/1966545992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:18:28.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:26.889340+0000 mon.a (mon.0) 2391 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-12T17:18:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:27.105895+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:27.105982+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:28.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:27.179732+0000 mgr.y (mgr.24481) 1084 : cluster [DBG] pgmap v1541: 102 pgs: 2 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/576 objects misplaced (0.174%) 2024-02-12T17:18:29.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:27.893788+0000 mon.a (mon.0) 2392 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-12T17:18:29.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:28.056323+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:18:29.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:28.057072+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:18:29.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:27.893788+0000 mon.a (mon.0) 2392 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-12T17:18:29.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:28.056323+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:18:29.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:28.057072+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:18:29.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:27.893788+0000 mon.a (mon.0) 2392 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-12T17:18:29.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:28.056323+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:18:29.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:28.057072+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:18:29.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:28.680736+0000 osd.1 (osd.1) 62 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:18:29.906 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:28.684206+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:18:29.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:28.895706+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-12T17:18:29.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:29.180321+0000 mgr.y (mgr.24481) 1085 : cluster [DBG] pgmap v1544: 102 pgs: 1 peering, 1 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 965 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:29.907 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:29.896704+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-12T17:18:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:28.680736+0000 osd.1 (osd.1) 62 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:18:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:28.684206+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:18:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:28.895706+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-12T17:18:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:29.180321+0000 mgr.y (mgr.24481) 1085 : cluster [DBG] pgmap v1544: 102 pgs: 1 peering, 1 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 965 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:30.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:29.896704+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-12T17:18:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:28.680736+0000 osd.1 (osd.1) 62 : cluster [DBG] 3.0 scrub starts 2024-02-12T17:18:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:28.684206+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.0 scrub ok 2024-02-12T17:18:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:28.895706+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-12T17:18:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:29.180321+0000 mgr.y (mgr.24481) 1085 : cluster [DBG] pgmap v1544: 102 pgs: 1 peering, 1 active+remapped+backfill_toofull, 100 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 965 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:30.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:29.896704+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-12T17:18:30.918 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-12T17:18:31.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:29.897763+0000 mon.a (mon.0) 2395 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:31.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:30.018223+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.33:0/2082959085' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:31.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:30.018526+0000 mon.a (mon.0) 2396 : audit [INF] from='client.36398 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:31.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:30.201666+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.0 deep-scrub starts 2024-02-12T17:18:31.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:30.205308+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.0 deep-scrub ok 2024-02-12T17:18:31.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:30] "GET /metrics HTTP/1.1" 200 174460 "" "Prometheus/2.33.4" 2024-02-12T17:18:31.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:29.897763+0000 mon.a (mon.0) 2395 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:31.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:30.018223+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.33:0/2082959085' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:31.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:30.018526+0000 mon.a (mon.0) 2396 : audit [INF] from='client.36398 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:31.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:30.201666+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.0 deep-scrub starts 2024-02-12T17:18:31.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:30.205308+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.0 deep-scrub ok 2024-02-12T17:18:31.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:29.897763+0000 mon.a (mon.0) 2395 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T17:18:31.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:30.018223+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.33:0/2082959085' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:31.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:30.018526+0000 mon.a (mon.0) 2396 : audit [INF] from='client.36398 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T17:18:31.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:30.201666+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.0 deep-scrub starts 2024-02-12T17:18:31.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:30.205308+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.0 deep-scrub ok 2024-02-12T17:18:31.720 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T17:18:32.016 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:30.816317+0000 osd.3 (osd.3) 296 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:30.819784+0000 osd.3 (osd.3) 297 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:30.911997+0000 mon.a (mon.0) 2397 : audit [INF] from='client.36398 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:30.912104+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:31.180709+0000 mgr.y (mgr.24481) 1086 : cluster [DBG] pgmap v1547: 102 pgs: 1 peering, 101 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:31.833999+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:30.816317+0000 osd.3 (osd.3) 296 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:30.819784+0000 osd.3 (osd.3) 297 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:30.911997+0000 mon.a (mon.0) 2397 : audit [INF] from='client.36398 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:18:32.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:30.912104+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-12T17:18:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:31.180709+0000 mgr.y (mgr.24481) 1086 : cluster [DBG] pgmap v1547: 102 pgs: 1 peering, 101 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:31.833999+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-12T17:18:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:30.816317+0000 osd.3 (osd.3) 296 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:18:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:30.819784+0000 osd.3 (osd.3) 297 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:18:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:30.911997+0000 mon.a (mon.0) 2397 : audit [INF] from='client.36398 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-12T17:18:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:30.912104+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-12T17:18:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:31.180709+0000 mgr.y (mgr.24481) 1086 : cluster [DBG] pgmap v1547: 102 pgs: 1 peering, 101 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:32.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:31.833999+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-12T17:18:32.922 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:18:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:18:33.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:31.913405+0000 mon.a (mon.0) 2400 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:18:33.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:31.913429+0000 mon.a (mon.0) 2401 : cluster [INF] Cluster is now healthy 2024-02-12T17:18:33.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:32.538830+0000 mon.a (mon.0) 2402 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:33.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:31.913405+0000 mon.a (mon.0) 2400 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:18:33.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:31.913429+0000 mon.a (mon.0) 2401 : cluster [INF] Cluster is now healthy 2024-02-12T17:18:33.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:32.538830+0000 mon.a (mon.0) 2402 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:33.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:31.913405+0000 mon.a (mon.0) 2400 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T17:18:33.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:31.913429+0000 mon.a (mon.0) 2401 : cluster [INF] Cluster is now healthy 2024-02-12T17:18:33.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:32.538830+0000 mon.a (mon.0) 2402 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:34.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:32.705154+0000 osd.1 (osd.1) 64 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:18:34.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:32.748031+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:18:34.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:33.181078+0000 mgr.y (mgr.24481) 1087 : cluster [DBG] pgmap v1549: 102 pgs: 1 peering, 101 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:34.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:32.705154+0000 osd.1 (osd.1) 64 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:18:34.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:32.748031+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:18:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:33.181078+0000 mgr.y (mgr.24481) 1087 : cluster [DBG] pgmap v1549: 102 pgs: 1 peering, 101 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:32.705154+0000 osd.1 (osd.1) 64 : cluster [DBG] 6.4 scrub starts 2024-02-12T17:18:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:32.748031+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub ok 2024-02-12T17:18:34.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:33 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:33.181078+0000 mgr.y (mgr.24481) 1087 : cluster [DBG] pgmap v1549: 102 pgs: 1 peering, 101 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-12T17:18:34.937 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T17:18:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:33.412217+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub starts 2024-02-12T17:18:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:33.413415+0000 osd.0 (osd.0) 279 : cluster [DBG] 5.5 scrub ok 2024-02-12T17:18:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:33.869599+0000 osd.3 (osd.3) 298 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:18:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:33.873236+0000 osd.3 (osd.3) 299 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:18:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:34.041702+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? v1:172.21.15.33:0/3446868615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:18:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:34.078121+0000 osd.6 (osd.6) 212 : cluster [DBG] 4.a scrub starts 2024-02-12T17:18:35.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:34.080618+0000 osd.6 (osd.6) 213 : cluster [DBG] 4.a scrub ok 2024-02-12T17:18:35.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:33.412217+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub starts 2024-02-12T17:18:35.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:33.413415+0000 osd.0 (osd.0) 279 : cluster [DBG] 5.5 scrub ok 2024-02-12T17:18:35.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:33.869599+0000 osd.3 (osd.3) 298 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:18:35.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:33.873236+0000 osd.3 (osd.3) 299 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:18:35.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:34.041702+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? v1:172.21.15.33:0/3446868615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:18:35.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:34.078121+0000 osd.6 (osd.6) 212 : cluster [DBG] 4.a scrub starts 2024-02-12T17:18:35.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:34.080618+0000 osd.6 (osd.6) 213 : cluster [DBG] 4.a scrub ok 2024-02-12T17:18:35.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:33.412217+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub starts 2024-02-12T17:18:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:33.413415+0000 osd.0 (osd.0) 279 : cluster [DBG] 5.5 scrub ok 2024-02-12T17:18:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:33.869599+0000 osd.3 (osd.3) 298 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:18:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:33.873236+0000 osd.3 (osd.3) 299 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:18:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:34.041702+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? v1:172.21.15.33:0/3446868615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:18:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:34.078121+0000 osd.6 (osd.6) 212 : cluster [DBG] 4.a scrub starts 2024-02-12T17:18:35.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:34.080618+0000 osd.6 (osd.6) 213 : cluster [DBG] 4.a scrub ok 2024-02-12T17:18:35.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:18:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:18:35.826 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-12T17:18:36.111 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:34.641449+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:18:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:34.687601+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:18:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:34.931466+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? v1:172.21.15.33:0/3446868615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:18:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:34.931556+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-12T17:18:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:35.181446+0000 mgr.y (mgr.24481) 1088 : cluster [DBG] pgmap v1551: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-12T17:18:36.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:35.242609+0000 mgr.y (mgr.24481) 1089 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:36.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:34.641449+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:18:36.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:34.687601+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:18:36.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:34.931466+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? v1:172.21.15.33:0/3446868615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:18:36.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:34.931556+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-12T17:18:36.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:35.181446+0000 mgr.y (mgr.24481) 1088 : cluster [DBG] pgmap v1551: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-12T17:18:36.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:35.242609+0000 mgr.y (mgr.24481) 1089 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:36.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:34.641449+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1 scrub starts 2024-02-12T17:18:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:34.687601+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1 scrub ok 2024-02-12T17:18:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:34.931466+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? v1:172.21.15.33:0/3446868615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:18:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:34.931556+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-12T17:18:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:35.181446+0000 mgr.y (mgr.24481) 1088 : cluster [DBG] pgmap v1551: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-12T17:18:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:35.242609+0000 mgr.y (mgr.24481) 1089 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:36.149056+0000 osd.7 (osd.7) 359 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T17:18:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:36.152133+0000 osd.7 (osd.7) 360 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T17:18:37.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:36.840789+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-12T17:18:37.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:36.149056+0000 osd.7 (osd.7) 359 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T17:18:37.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:36.152133+0000 osd.7 (osd.7) 360 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T17:18:37.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:36.840789+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-12T17:18:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:36.149056+0000 osd.7 (osd.7) 359 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T17:18:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:36.152133+0000 osd.7 (osd.7) 360 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T17:18:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:36.840789+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-12T17:18:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:36.715412+0000 osd.1 (osd.1) 68 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:18:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:36.718878+0000 osd.1 (osd.1) 69 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:18:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:37.180804+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:18:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:37.181941+0000 mgr.y (mgr.24481) 1090 : cluster [DBG] pgmap v1553: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:37.317302+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:18:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:37.682104+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e deep-scrub starts 2024-02-12T17:18:38.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:37.683272+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e deep-scrub ok 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:36.715412+0000 osd.1 (osd.1) 68 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:36.718878+0000 osd.1 (osd.1) 69 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:37.180804+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:37.181941+0000 mgr.y (mgr.24481) 1090 : cluster [DBG] pgmap v1553: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:37.317302+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:37.682104+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e deep-scrub starts 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:37.683272+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e deep-scrub ok 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:36.715412+0000 osd.1 (osd.1) 68 : cluster [DBG] 4.12 scrub starts 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:36.718878+0000 osd.1 (osd.1) 69 : cluster [DBG] 4.12 scrub ok 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:37.180804+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.15 scrub starts 2024-02-12T17:18:38.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:37.181941+0000 mgr.y (mgr.24481) 1090 : cluster [DBG] pgmap v1553: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:37.317302+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.15 scrub ok 2024-02-12T17:18:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:37.682104+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e deep-scrub starts 2024-02-12T17:18:38.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:37.683272+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e deep-scrub ok 2024-02-12T17:18:38.958 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-12T17:18:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:37.442271+0000 osd.0 (osd.0) 280 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:18:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:37.487346+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:18:39.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:38.279462+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? v1:172.21.15.33:0/1377150736' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:18:39.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:37.442271+0000 osd.0 (osd.0) 280 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:18:39.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:37.487346+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:18:39.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:38.279462+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? v1:172.21.15.33:0/1377150736' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:18:39.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:37.442271+0000 osd.0 (osd.0) 280 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:18:39.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:37.487346+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:18:39.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:38.279462+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? v1:172.21.15.33:0/1377150736' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T17:18:39.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-12T17:18:39.900 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:38.853271+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.c scrub starts 2024-02-12T17:18:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:38.898384+0000 osd.3 (osd.3) 301 : cluster [DBG] 6.c scrub ok 2024-02-12T17:18:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:38.953230+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? v1:172.21.15.33:0/1377150736' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:18:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:38.953332+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-12T17:18:40.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:39.182517+0000 mgr.y (mgr.24481) 1091 : cluster [DBG] pgmap v1555: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:38.853271+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.c scrub starts 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:38.898384+0000 osd.3 (osd.3) 301 : cluster [DBG] 6.c scrub ok 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:38.953230+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? v1:172.21.15.33:0/1377150736' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:38.953332+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:39.182517+0000 mgr.y (mgr.24481) 1091 : cluster [DBG] pgmap v1555: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:38.853271+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.c scrub starts 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:38.898384+0000 osd.3 (osd.3) 301 : cluster [DBG] 6.c scrub ok 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:38.953230+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? v1:172.21.15.33:0/1377150736' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:38.953332+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-12T17:18:40.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:39.182517+0000 mgr.y (mgr.24481) 1091 : cluster [DBG] pgmap v1555: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.132368+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:18:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.135832+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:18:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.706057+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 deep-scrub starts 2024-02-12T17:18:41.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.706981+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 deep-scrub ok 2024-02-12T17:18:41.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:40] "GET /metrics HTTP/1.1" 200 174462 "" "Prometheus/2.33.4" 2024-02-12T17:18:41.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.132368+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:18:41.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.135832+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:18:41.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.706057+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 deep-scrub starts 2024-02-12T17:18:41.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.706981+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 deep-scrub ok 2024-02-12T17:18:41.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.132368+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.9 scrub starts 2024-02-12T17:18:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.135832+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.9 scrub ok 2024-02-12T17:18:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.706057+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 deep-scrub starts 2024-02-12T17:18:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.706981+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 deep-scrub ok 2024-02-12T17:18:41.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.663790+0000 osd.5 (osd.5) 359 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:41.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.663841+0000 osd.5 (osd.5) 360 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:41.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.812052+0000 osd.3 (osd.3) 302 : cluster [DBG] 6.b scrub starts 2024-02-12T17:18:41.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.958627+0000 osd.3 (osd.3) 303 : cluster [DBG] 6.b scrub ok 2024-02-12T17:18:41.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:40.969441+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-12T17:18:41.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:41.183001+0000 mgr.y (mgr.24481) 1092 : cluster [DBG] pgmap v1557: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:41.986 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:41.709702+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:18:41.987 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:41.710753+0000 osd.2 (osd.2) 256 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:18:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.663790+0000 osd.5 (osd.5) 359 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.663841+0000 osd.5 (osd.5) 360 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.812052+0000 osd.3 (osd.3) 302 : cluster [DBG] 6.b scrub starts 2024-02-12T17:18:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.958627+0000 osd.3 (osd.3) 303 : cluster [DBG] 6.b scrub ok 2024-02-12T17:18:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:40.969441+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-12T17:18:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:41.183001+0000 mgr.y (mgr.24481) 1092 : cluster [DBG] pgmap v1557: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:41.709702+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:18:42.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:41.710753+0000 osd.2 (osd.2) 256 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:18:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.663790+0000 osd.5 (osd.5) 359 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.663841+0000 osd.5 (osd.5) 360 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.812052+0000 osd.3 (osd.3) 302 : cluster [DBG] 6.b scrub starts 2024-02-12T17:18:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.958627+0000 osd.3 (osd.3) 303 : cluster [DBG] 6.b scrub ok 2024-02-12T17:18:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:40.969441+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-12T17:18:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:41.183001+0000 mgr.y (mgr.24481) 1092 : cluster [DBG] pgmap v1557: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:42.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:41.709702+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.1 scrub starts 2024-02-12T17:18:42.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:41 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:41.710753+0000 osd.2 (osd.2) 256 : cluster [DBG] 2.1 scrub ok 2024-02-12T17:18:42.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:18:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:18:42.999 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.1 to 1 (10000) 2024-02-12T17:18:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:42.155573+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? v1:172.21.15.33:0/4185940795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:18:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:42.400936+0000 osd.4 (osd.4) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:43.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:42.401018+0000 osd.4 (osd.4) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:43.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:42.155573+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? v1:172.21.15.33:0/4185940795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:18:43.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:42.400936+0000 osd.4 (osd.4) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:43.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:42.401018+0000 osd.4 (osd.4) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:43.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:42.155573+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? v1:172.21.15.33:0/4185940795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:18:43.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:42.400936+0000 osd.4 (osd.4) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:18:43.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:42.401018+0000 osd.4 (osd.4) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:18:43.870 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-12T17:18:44.157 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:44.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:42.458868+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:18:44.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:42.569501+0000 osd.0 (osd.0) 283 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:18:44.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:42.995226+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? v1:172.21.15.33:0/4185940795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:18:44.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:42.995305+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-12T17:18:44.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:43.107724+0000 osd.7 (osd.7) 365 : cluster [DBG] 2.b scrub starts 2024-02-12T17:18:44.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:43.110636+0000 osd.7 (osd.7) 366 : cluster [DBG] 2.b scrub ok 2024-02-12T17:18:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:43.183418+0000 mgr.y (mgr.24481) 1093 : cluster [DBG] pgmap v1559: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:43.642669+0000 mon.a (mon.0) 2414 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:43.766428+0000 osd.4 (osd.4) 225 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:18:44.278 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:43.769406+0000 osd.4 (osd.4) 226 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:18:44.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:42.458868+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:18:44.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:42.569501+0000 osd.0 (osd.0) 283 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:18:44.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:42.995226+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? v1:172.21.15.33:0/4185940795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:18:44.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:42.995305+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-12T17:18:44.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:43.107724+0000 osd.7 (osd.7) 365 : cluster [DBG] 2.b scrub starts 2024-02-12T17:18:44.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:43.110636+0000 osd.7 (osd.7) 366 : cluster [DBG] 2.b scrub ok 2024-02-12T17:18:44.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:43.183418+0000 mgr.y (mgr.24481) 1093 : cluster [DBG] pgmap v1559: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:44.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:43.642669+0000 mon.a (mon.0) 2414 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:44.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:43.766428+0000 osd.4 (osd.4) 225 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:18:44.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:43.769406+0000 osd.4 (osd.4) 226 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:18:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:42.458868+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.9 scrub starts 2024-02-12T17:18:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:42.569501+0000 osd.0 (osd.0) 283 : cluster [DBG] 6.9 scrub ok 2024-02-12T17:18:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:42.995226+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? v1:172.21.15.33:0/4185940795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:18:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:42.995305+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-12T17:18:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:43.107724+0000 osd.7 (osd.7) 365 : cluster [DBG] 2.b scrub starts 2024-02-12T17:18:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:43.110636+0000 osd.7 (osd.7) 366 : cluster [DBG] 2.b scrub ok 2024-02-12T17:18:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:43.183418+0000 mgr.y (mgr.24481) 1093 : cluster [DBG] pgmap v1559: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:18:44.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:43.642669+0000 mon.a (mon.0) 2414 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:18:44.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:43.766428+0000 osd.4 (osd.4) 225 : cluster [DBG] 5.1 scrub starts 2024-02-12T17:18:44.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:43.769406+0000 osd.4 (osd.4) 226 : cluster [DBG] 5.1 scrub ok 2024-02-12T17:18:45.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:18:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:18:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:44.690010+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:18:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:44.692947+0000 osd.5 (osd.5) 362 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:18:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:45.025813+0000 osd.6 (osd.6) 214 : cluster [DBG] 2.7 deep-scrub starts 2024-02-12T17:18:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:45.026758+0000 osd.6 (osd.6) 215 : cluster [DBG] 2.7 deep-scrub ok 2024-02-12T17:18:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:45.083069+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:18:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:45.086039+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:18:46.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:45.184198+0000 mgr.y (mgr.24481) 1094 : cluster [DBG] pgmap v1560: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-12T17:18:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:45.246633+0000 mgr.y (mgr.24481) 1095 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:45.704135+0000 osd.4 (osd.4) 227 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:18:46.491 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:45.706211+0000 osd.4 (osd.4) 228 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:18:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:44.690010+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:18:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:44.692947+0000 osd.5 (osd.5) 362 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:18:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:45.025813+0000 osd.6 (osd.6) 214 : cluster [DBG] 2.7 deep-scrub starts 2024-02-12T17:18:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:45.026758+0000 osd.6 (osd.6) 215 : cluster [DBG] 2.7 deep-scrub ok 2024-02-12T17:18:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:45.083069+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:18:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:45.086039+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:18:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:45.184198+0000 mgr.y (mgr.24481) 1094 : cluster [DBG] pgmap v1560: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-12T17:18:46.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:45.246633+0000 mgr.y (mgr.24481) 1095 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:45.704135+0000 osd.4 (osd.4) 227 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:45.706211+0000 osd.4 (osd.4) 228 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:44.690010+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.2 scrub starts 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:44.692947+0000 osd.5 (osd.5) 362 : cluster [DBG] 2.2 scrub ok 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:45.025813+0000 osd.6 (osd.6) 214 : cluster [DBG] 2.7 deep-scrub starts 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:45.026758+0000 osd.6 (osd.6) 215 : cluster [DBG] 2.7 deep-scrub ok 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:45.083069+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.5 scrub starts 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:45.086039+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.5 scrub ok 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:45.184198+0000 mgr.y (mgr.24481) 1094 : cluster [DBG] pgmap v1560: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:45.246633+0000 mgr.y (mgr.24481) 1095 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:45.704135+0000 osd.4 (osd.4) 227 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:45.706211+0000 osd.4 (osd.4) 228 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:18:46.845 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-12T17:18:47.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.100550+0000 osd.7 (osd.7) 369 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:18:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.103536+0000 osd.7 (osd.7) 370 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:18:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:46.246464+0000 mon.a (mon.0) 2415 : audit [INF] from='client.? v1:172.21.15.33:0/1298706840' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:18:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.449171+0000 osd.0 (osd.0) 284 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:18:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.489724+0000 osd.0 (osd.0) 285 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:18:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.778271+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.d scrub starts 2024-02-12T17:18:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.780057+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.d scrub ok 2024-02-12T17:18:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:46.840416+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? v1:172.21.15.33:0/1298706840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:18:47.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.840498+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.100550+0000 osd.7 (osd.7) 369 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.103536+0000 osd.7 (osd.7) 370 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:46.246464+0000 mon.a (mon.0) 2415 : audit [INF] from='client.? v1:172.21.15.33:0/1298706840' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.449171+0000 osd.0 (osd.0) 284 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.489724+0000 osd.0 (osd.0) 285 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.778271+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.d scrub starts 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.780057+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.d scrub ok 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:46.840416+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? v1:172.21.15.33:0/1298706840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:18:47.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.840498+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-12T17:18:47.587 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-12T17:18:47.587 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-12T17:18:47.587 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-12T17:18:47.587 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-12T17:18:47.588 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-12T17:18:47.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T17:18:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.100550+0000 osd.7 (osd.7) 369 : cluster [DBG] 2.8 scrub starts 2024-02-12T17:18:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.103536+0000 osd.7 (osd.7) 370 : cluster [DBG] 2.8 scrub ok 2024-02-12T17:18:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:46.246464+0000 mon.a (mon.0) 2415 : audit [INF] from='client.? v1:172.21.15.33:0/1298706840' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T17:18:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.449171+0000 osd.0 (osd.0) 284 : cluster [DBG] 6.0 scrub starts 2024-02-12T17:18:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.489724+0000 osd.0 (osd.0) 285 : cluster [DBG] 6.0 scrub ok 2024-02-12T17:18:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.778271+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.d scrub starts 2024-02-12T17:18:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.780057+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.d scrub ok 2024-02-12T17:18:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:46.840416+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? v1:172.21.15.33:0/1298706840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-12T17:18:47.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.840498+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-12T17:18:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:48.388 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.852186+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:18:48.389 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:46.855659+0000 osd.3 (osd.3) 305 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:18:48.389 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:47.184714+0000 mgr.y (mgr.24481) 1096 : cluster [DBG] pgmap v1562: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-02-12T17:18:48.389 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:47.481966+0000 osd.0 (osd.0) 286 : cluster [DBG] 4.3 deep-scrub starts 2024-02-12T17:18:48.389 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:47.489908+0000 osd.0 (osd.0) 287 : cluster [DBG] 4.3 deep-scrub ok 2024-02-12T17:18:48.389 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.852186+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:18:48.389 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:46.855659+0000 osd.3 (osd.3) 305 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:18:48.389 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:47.184714+0000 mgr.y (mgr.24481) 1096 : cluster [DBG] pgmap v1562: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-02-12T17:18:48.390 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:47.481966+0000 osd.0 (osd.0) 286 : cluster [DBG] 4.3 deep-scrub starts 2024-02-12T17:18:48.390 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:47.489908+0000 osd.0 (osd.0) 287 : cluster [DBG] 4.3 deep-scrub ok 2024-02-12T17:18:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.852186+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.10 scrub starts 2024-02-12T17:18:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:46.855659+0000 osd.3 (osd.3) 305 : cluster [DBG] 4.10 scrub ok 2024-02-12T17:18:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:47.184714+0000 mgr.y (mgr.24481) 1096 : cluster [DBG] pgmap v1562: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-02-12T17:18:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:47.481966+0000 osd.0 (osd.0) 286 : cluster [DBG] 4.3 deep-scrub starts 2024-02-12T17:18:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:47.489908+0000 osd.0 (osd.0) 287 : cluster [DBG] 4.3 deep-scrub ok 2024-02-12T17:18:49.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:47.718330+0000 osd.5 (osd.5) 363 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:18:49.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:47.722682+0000 osd.5 (osd.5) 364 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:18:49.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:47.780625+0000 osd.2 (osd.2) 257 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T17:18:49.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:47.811981+0000 osd.2 (osd.2) 258 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T17:18:49.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:48.698947+0000 osd.4 (osd.4) 229 : cluster [DBG] 2.f scrub starts 2024-02-12T17:18:49.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:48.702414+0000 osd.4 (osd.4) 230 : cluster [DBG] 2.f scrub ok 2024-02-12T17:18:49.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:47.718330+0000 osd.5 (osd.5) 363 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:18:49.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:47.722682+0000 osd.5 (osd.5) 364 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:18:49.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:47.780625+0000 osd.2 (osd.2) 257 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T17:18:49.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:47.811981+0000 osd.2 (osd.2) 258 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T17:18:49.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:48.698947+0000 osd.4 (osd.4) 229 : cluster [DBG] 2.f scrub starts 2024-02-12T17:18:49.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:48.702414+0000 osd.4 (osd.4) 230 : cluster [DBG] 2.f scrub ok 2024-02-12T17:18:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:47.718330+0000 osd.5 (osd.5) 363 : cluster [DBG] 4.8 scrub starts 2024-02-12T17:18:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:47.722682+0000 osd.5 (osd.5) 364 : cluster [DBG] 4.8 scrub ok 2024-02-12T17:18:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:47.780625+0000 osd.2 (osd.2) 257 : cluster [DBG] 5.b deep-scrub starts 2024-02-12T17:18:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:47.811981+0000 osd.2 (osd.2) 258 : cluster [DBG] 5.b deep-scrub ok 2024-02-12T17:18:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:48.698947+0000 osd.4 (osd.4) 229 : cluster [DBG] 2.f scrub starts 2024-02-12T17:18:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:48.702414+0000 osd.4 (osd.4) 230 : cluster [DBG] 2.f scrub ok 2024-02-12T17:18:49.939 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:18:49.939 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":766,"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","created":"2024-02-12T16:43:50.282468+0000","modified":"2024-02-12T17:18:46.835149+0000","last_up_change":"2024-02-12T17:13:45.508528+0000","last_in_change":"2024-02-12T17:17:25.159490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T16:47:33.056945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-12T16:49:37.647343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"2.16","ready_epoch":661,"last_epoch_started":660,"last_epoch_clean":660,"source_version":"0'0","target_version":"0'0"},"last_change":"733","last_force_op_resend":"0","last_force_op_resend_prenautilus":"661","last_force_op_resend_preluminous":"659","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-12T16:49:38.076200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"527","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-12T16:49:39.589565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-12T16:49:41.623856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"5.10","ready_epoch":429,"last_epoch_started":428,"last_epoch_clean":428,"source_version":"0'0","target_version":"380'16"},"last_change":"731","last_force_op_resend":"0","last_force_op_resend_prenautilus":"429","last_force_op_resend_preluminous":"431","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-12T16:49:43.693806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"6.16","ready_epoch":396,"last_epoch_started":395,"last_epoch_clean":395,"source_version":"0'0","target_version":"0'0"},"last_change":"734","last_force_op_resend":"0","last_force_op_resend_prenautilus":"396","last_force_op_resend_preluminous":"394","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"5dd172eb-ee9d-4700-9b80-60f91c5f2cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":756,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6801","nonce":2545411201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6802","nonce":2545411201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6804","nonce":2545411201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6803","nonce":2545411201}]},"public_addr":"172.21.15.33:6801/2545411201","cluster_addr":"172.21.15.33:6802/2545411201","heartbeat_back_addr":"172.21.15.33:6804/2545411201","heartbeat_front_addr":"172.21.15.33:6803/2545411201","state":["exists","up"]},{"osd":1,"uuid":"71231bc4-1716-431d-bd88-f6a3cad0bd24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":532,"up_from":630,"up_thru":730,"down_at":533,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6805","nonce":2528921223}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6806","nonce":2528921223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6808","nonce":2528921223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6807","nonce":2528921223}]},"public_addr":"172.21.15.33:6805/2528921223","cluster_addr":"172.21.15.33:6806/2528921223","heartbeat_back_addr":"172.21.15.33:6808/2528921223","heartbeat_front_addr":"172.21.15.33:6807/2528921223","state":["exists","up"]},{"osd":2,"uuid":"8a9c7127-d17c-4c44-b666-4211006de47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":739,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6809","nonce":3126580659}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6810","nonce":3126580659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6812","nonce":3126580659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6811","nonce":3126580659}]},"public_addr":"172.21.15.33:6809/3126580659","cluster_addr":"172.21.15.33:6810/3126580659","heartbeat_back_addr":"172.21.15.33:6812/3126580659","heartbeat_front_addr":"172.21.15.33:6811/3126580659","state":["exists","up"]},{"osd":3,"uuid":"3cc40671-c8f7-412c-a0f1-987f48a016b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":741,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6813","nonce":3941314957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6814","nonce":3941314957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6816","nonce":3941314957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6815","nonce":3941314957}]},"public_addr":"172.21.15.33:6813/3941314957","cluster_addr":"172.21.15.33:6814/3941314957","heartbeat_back_addr":"172.21.15.33:6816/3941314957","heartbeat_front_addr":"172.21.15.33:6815/3941314957","state":["exists","up"]},{"osd":4,"uuid":"0498129c-861c-4f7e-b8ba-219949da3130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":748,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":544126779}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":544126779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":544126779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":544126779}]},"public_addr":"172.21.15.94:6800/544126779","cluster_addr":"172.21.15.94:6801/544126779","heartbeat_back_addr":"172.21.15.94:6803/544126779","heartbeat_front_addr":"172.21.15.94:6802/544126779","state":["exists","up"]},{"osd":5,"uuid":"05718a1e-1164-4b54-a493-23c1cd3adb50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":741,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":3229884005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3229884005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3229884005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":3229884005}]},"public_addr":"172.21.15.94:6804/3229884005","cluster_addr":"172.21.15.94:6805/3229884005","heartbeat_back_addr":"172.21.15.94:6807/3229884005","heartbeat_front_addr":"172.21.15.94:6806/3229884005","state":["exists","up"]},{"osd":6,"uuid":"dc48640f-8f27-405c-a9a4-cf1e3e6c8821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":727,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":507305558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":507305558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":507305558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":507305558}]},"public_addr":"172.21.15.94:6808/507305558","cluster_addr":"172.21.15.94:6809/507305558","heartbeat_back_addr":"172.21.15.94:6811/507305558","heartbeat_front_addr":"172.21.15.94:6810/507305558","state":["exists","up"]},{"osd":7,"uuid":"7d55aa3f-06b6-4fc3-97fd-62b535f78b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":757,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3401977873}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":3401977873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":3401977873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":3401977873}]},"public_addr":"172.21.15.94:6812/3401977873","cluster_addr":"172.21.15.94:6813/3401977873","heartbeat_back_addr":"172.21.15.94:6815/3401977873","heartbeat_front_addr":"172.21.15.94:6814/3401977873","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:23.160529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:20.639117+0000","dead_epoch":532},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:16.111485+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:18.442352+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:42.401022+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:40.663843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:27.105985+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:13.108172+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.7","osds":[1,4,5]}],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.16","mappings":[{"from":3,"to":2}]},{"pgid":"4.1a","mappings":[{"from":3,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"5.15","mappings":[{"from":0,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/2829464018":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/1835128904":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2277321750":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2116602769":"2024-02-13T16:50:02.257936+0000","172.21.15.33:6800/3967118778":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/2298361247":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/218227980":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/2070582473":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/4291575956":"2024-02-13T16:44:46.313346+0000","172.21.15.33:6800/3157562110":"2024-02-13T16:44:22.727241+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T17:18:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:49.160693+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:18:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:49.163661+0000 osd.7 (osd.7) 372 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:18:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:49.185274+0000 mgr.y (mgr.24481) 1097 : cluster [DBG] pgmap v1563: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:18:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:49.722412+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.e scrub starts 2024-02-12T17:18:50.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:49.810247+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.e scrub ok 2024-02-12T17:18:50.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:49.937074+0000 mon.c (mon.2) 337 : audit [DBG] from='client.? v1:172.21.15.33:0/1616444573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T17:18:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:49.160693+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:18:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:49.163661+0000 osd.7 (osd.7) 372 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:18:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:49.185274+0000 mgr.y (mgr.24481) 1097 : cluster [DBG] pgmap v1563: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:18:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:49.722412+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.e scrub starts 2024-02-12T17:18:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:49.810247+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.e scrub ok 2024-02-12T17:18:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:49.937074+0000 mon.c (mon.2) 337 : audit [DBG] from='client.? v1:172.21.15.33:0/1616444573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T17:18:50.731 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-12T17:18:50.731 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T17:18:50.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:49.160693+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.8 scrub starts 2024-02-12T17:18:50.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:49.163661+0000 osd.7 (osd.7) 372 : cluster [DBG] 6.8 scrub ok 2024-02-12T17:18:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:49.185274+0000 mgr.y (mgr.24481) 1097 : cluster [DBG] pgmap v1563: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:18:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:49.722412+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.e scrub starts 2024-02-12T17:18:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:49.810247+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.e scrub ok 2024-02-12T17:18:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:49.937074+0000 mon.c (mon.2) 337 : audit [DBG] from='client.? v1:172.21.15.33:0/1616444573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T17:18:51.019 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:51.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:18:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:50] "GET /metrics HTTP/1.1" 200 174463 "" "Prometheus/2.33.4" 2024-02-12T17:18:51.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:49.812218+0000 osd.2 (osd.2) 259 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:18:51.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:49.829470+0000 osd.2 (osd.2) 260 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:18:51.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:50.772035+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.c scrub starts 2024-02-12T17:18:51.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:50.773350+0000 osd.1 (osd.1) 73 : cluster [DBG] 5.c scrub ok 2024-02-12T17:18:51.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:49.812218+0000 osd.2 (osd.2) 259 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:18:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:49.829470+0000 osd.2 (osd.2) 260 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:18:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:50.772035+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.c scrub starts 2024-02-12T17:18:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:50.773350+0000 osd.1 (osd.1) 73 : cluster [DBG] 5.c scrub ok 2024-02-12T17:18:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:49.812218+0000 osd.2 (osd.2) 259 : cluster [DBG] 6.14 scrub starts 2024-02-12T17:18:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:49.829470+0000 osd.2 (osd.2) 260 : cluster [DBG] 6.14 scrub ok 2024-02-12T17:18:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:50.772035+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.c scrub starts 2024-02-12T17:18:51.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:50.773350+0000 osd.1 (osd.1) 73 : cluster [DBG] 5.c scrub ok 2024-02-12T17:18:52.510 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:51.014787+0000 osd.6 (osd.6) 216 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:18:52.510 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:51.015810+0000 osd.6 (osd.6) 217 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:18:52.511 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:51.185692+0000 mgr.y (mgr.24481) 1098 : cluster [DBG] pgmap v1564: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:18:52.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:51.014787+0000 osd.6 (osd.6) 216 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:18:52.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:51.015810+0000 osd.6 (osd.6) 217 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:18:52.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:51.185692+0000 mgr.y (mgr.24481) 1098 : cluster [DBG] pgmap v1564: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:18:52.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:51.014787+0000 osd.6 (osd.6) 216 : cluster [DBG] 5.2 scrub starts 2024-02-12T17:18:52.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:51.015810+0000 osd.6 (osd.6) 217 : cluster [DBG] 5.2 scrub ok 2024-02-12T17:18:52.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:51.185692+0000 mgr.y (mgr.24481) 1098 : cluster [DBG] pgmap v1564: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:18:52.990 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:18:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:18:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:18:53.113 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:18:53.114 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":766,"fsid":"cf6ad642-c9c5-11ee-95b9-87774f69a715","created":"2024-02-12T16:43:50.282468+0000","modified":"2024-02-12T17:18:46.835149+0000","last_up_change":"2024-02-12T17:13:45.508528+0000","last_in_change":"2024-02-12T17:17:25.159490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T16:47:33.056945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-12T16:49:37.647343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"2.16","ready_epoch":661,"last_epoch_started":660,"last_epoch_clean":660,"source_version":"0'0","target_version":"0'0"},"last_change":"733","last_force_op_resend":"0","last_force_op_resend_prenautilus":"661","last_force_op_resend_preluminous":"659","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-12T16:49:38.076200+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"527","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-12T16:49:39.589565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-12T16:49:41.623856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"5.10","ready_epoch":429,"last_epoch_started":428,"last_epoch_clean":428,"source_version":"0'0","target_version":"380'16"},"last_change":"731","last_force_op_resend":"0","last_force_op_resend_prenautilus":"429","last_force_op_resend_preluminous":"431","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-12T16:49:43.693806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"6.16","ready_epoch":396,"last_epoch_started":395,"last_epoch_clean":395,"source_version":"0'0","target_version":"0'0"},"last_change":"734","last_force_op_resend":"0","last_force_op_resend_prenautilus":"396","last_force_op_resend_preluminous":"394","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"5dd172eb-ee9d-4700-9b80-60f91c5f2cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":756,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6801","nonce":2545411201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6802","nonce":2545411201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6804","nonce":2545411201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6803","nonce":2545411201}]},"public_addr":"172.21.15.33:6801/2545411201","cluster_addr":"172.21.15.33:6802/2545411201","heartbeat_back_addr":"172.21.15.33:6804/2545411201","heartbeat_front_addr":"172.21.15.33:6803/2545411201","state":["exists","up"]},{"osd":1,"uuid":"71231bc4-1716-431d-bd88-f6a3cad0bd24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":532,"up_from":630,"up_thru":730,"down_at":533,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6805","nonce":2528921223}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6806","nonce":2528921223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6808","nonce":2528921223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6807","nonce":2528921223}]},"public_addr":"172.21.15.33:6805/2528921223","cluster_addr":"172.21.15.33:6806/2528921223","heartbeat_back_addr":"172.21.15.33:6808/2528921223","heartbeat_front_addr":"172.21.15.33:6807/2528921223","state":["exists","up"]},{"osd":2,"uuid":"8a9c7127-d17c-4c44-b666-4211006de47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":739,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6809","nonce":3126580659}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6810","nonce":3126580659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6812","nonce":3126580659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6811","nonce":3126580659}]},"public_addr":"172.21.15.33:6809/3126580659","cluster_addr":"172.21.15.33:6810/3126580659","heartbeat_back_addr":"172.21.15.33:6812/3126580659","heartbeat_front_addr":"172.21.15.33:6811/3126580659","state":["exists","up"]},{"osd":3,"uuid":"3cc40671-c8f7-412c-a0f1-987f48a016b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":741,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6813","nonce":3941314957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6814","nonce":3941314957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6816","nonce":3941314957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.33:6815","nonce":3941314957}]},"public_addr":"172.21.15.33:6813/3941314957","cluster_addr":"172.21.15.33:6814/3941314957","heartbeat_back_addr":"172.21.15.33:6816/3941314957","heartbeat_front_addr":"172.21.15.33:6815/3941314957","state":["exists","up"]},{"osd":4,"uuid":"0498129c-861c-4f7e-b8ba-219949da3130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":748,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":544126779}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":544126779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":544126779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":544126779}]},"public_addr":"172.21.15.94:6800/544126779","cluster_addr":"172.21.15.94:6801/544126779","heartbeat_back_addr":"172.21.15.94:6803/544126779","heartbeat_front_addr":"172.21.15.94:6802/544126779","state":["exists","up"]},{"osd":5,"uuid":"05718a1e-1164-4b54-a493-23c1cd3adb50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":741,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":3229884005}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":3229884005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":3229884005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":3229884005}]},"public_addr":"172.21.15.94:6804/3229884005","cluster_addr":"172.21.15.94:6805/3229884005","heartbeat_back_addr":"172.21.15.94:6807/3229884005","heartbeat_front_addr":"172.21.15.94:6806/3229884005","state":["exists","up"]},{"osd":6,"uuid":"dc48640f-8f27-405c-a9a4-cf1e3e6c8821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":727,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":507305558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":507305558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":507305558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":507305558}]},"public_addr":"172.21.15.94:6808/507305558","cluster_addr":"172.21.15.94:6809/507305558","heartbeat_back_addr":"172.21.15.94:6811/507305558","heartbeat_front_addr":"172.21.15.94:6810/507305558","state":["exists","up"]},{"osd":7,"uuid":"7d55aa3f-06b6-4fc3-97fd-62b535f78b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":757,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3401977873}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":3401977873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":3401977873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":3401977873}]},"public_addr":"172.21.15.94:6812/3401977873","cluster_addr":"172.21.15.94:6813/3401977873","heartbeat_back_addr":"172.21.15.94:6815/3401977873","heartbeat_front_addr":"172.21.15.94:6814/3401977873","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:23.160529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:20.639117+0000","dead_epoch":532},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:16.111485+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:18.442352+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:42.401022+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:40.663843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:27.105985+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T17:18:13.108172+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.7","osds":[1,4,5]}],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.16","mappings":[{"from":3,"to":2}]},{"pgid":"4.1a","mappings":[{"from":3,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"5.15","mappings":[{"from":0,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/2829464018":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/1835128904":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2277321750":"2024-02-13T16:50:02.257936+0000","172.21.15.33:0/2116602769":"2024-02-13T16:50:02.257936+0000","172.21.15.33:6800/3967118778":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/2298361247":"2024-02-13T16:44:46.313346+0000","172.21.15.33:0/218227980":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/2070582473":"2024-02-13T16:44:22.727241+0000","172.21.15.33:0/4291575956":"2024-02-13T16:44:46.313346+0000","172.21.15.33:6800/3157562110":"2024-02-13T16:44:22.727241+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T17:18:53.455 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:52.746805+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub starts 2024-02-12T17:18:53.455 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:52.749779+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.a scrub ok 2024-02-12T17:18:53.455 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:53.111362+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? v1:172.21.15.33:0/953797793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T17:18:53.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:52.746805+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub starts 2024-02-12T17:18:53.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:52.749779+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.a scrub ok 2024-02-12T17:18:53.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:53.111362+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? v1:172.21.15.33:0/953797793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T17:18:53.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:52.746805+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub starts 2024-02-12T17:18:53.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:52.749779+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.a scrub ok 2024-02-12T17:18:53.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:53.111362+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? v1:172.21.15.33:0/953797793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T17:18:53.841 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-12T17:18:54.122 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:54.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:52.826927+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.a scrub starts 2024-02-12T17:18:54.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:52.828184+0000 osd.2 (osd.2) 262 : cluster [DBG] 5.a scrub ok 2024-02-12T17:18:54.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:53.186081+0000 mgr.y (mgr.24481) 1099 : cluster [DBG] pgmap v1565: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:18:54.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:53.542326+0000 osd.0 (osd.0) 288 : cluster [DBG] 2.13 deep-scrub starts 2024-02-12T17:18:54.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:53.545216+0000 osd.0 (osd.0) 289 : cluster [DBG] 2.13 deep-scrub ok 2024-02-12T17:18:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:52.826927+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.a scrub starts 2024-02-12T17:18:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:52.828184+0000 osd.2 (osd.2) 262 : cluster [DBG] 5.a scrub ok 2024-02-12T17:18:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:53.186081+0000 mgr.y (mgr.24481) 1099 : cluster [DBG] pgmap v1565: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:18:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:53.542326+0000 osd.0 (osd.0) 288 : cluster [DBG] 2.13 deep-scrub starts 2024-02-12T17:18:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:54 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:53.545216+0000 osd.0 (osd.0) 289 : cluster [DBG] 2.13 deep-scrub ok 2024-02-12T17:18:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:52.826927+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.a scrub starts 2024-02-12T17:18:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:52.828184+0000 osd.2 (osd.2) 262 : cluster [DBG] 5.a scrub ok 2024-02-12T17:18:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:53.186081+0000 mgr.y (mgr.24481) 1099 : cluster [DBG] pgmap v1565: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:18:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:53.542326+0000 osd.0 (osd.0) 288 : cluster [DBG] 2.13 deep-scrub starts 2024-02-12T17:18:54.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:53.545216+0000 osd.0 (osd.0) 289 : cluster [DBG] 2.13 deep-scrub ok 2024-02-12T17:18:55.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:53.848539+0000 osd.2 (osd.2) 263 : cluster [DBG] 5.d scrub starts 2024-02-12T17:18:55.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:53.850261+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub ok 2024-02-12T17:18:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:54.055484+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:18:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:54.056446+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:18:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:54.677290+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 deep-scrub starts 2024-02-12T17:18:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:54.684298+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 deep-scrub ok 2024-02-12T17:18:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:53.848539+0000 osd.2 (osd.2) 263 : cluster [DBG] 5.d scrub starts 2024-02-12T17:18:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:53.850261+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub ok 2024-02-12T17:18:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:54.055484+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:18:55.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:54.056446+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:18:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:54.677290+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 deep-scrub starts 2024-02-12T17:18:55.529 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:54.684298+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 deep-scrub ok 2024-02-12T17:18:55.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:18:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:18:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:53.848539+0000 osd.2 (osd.2) 263 : cluster [DBG] 5.d scrub starts 2024-02-12T17:18:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:53.850261+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub ok 2024-02-12T17:18:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:54.055484+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.11 scrub starts 2024-02-12T17:18:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:54.056446+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.11 scrub ok 2024-02-12T17:18:55.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:54.677290+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 deep-scrub starts 2024-02-12T17:18:55.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:54.684298+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 deep-scrub ok 2024-02-12T17:18:56.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:55.186679+0000 mgr.y (mgr.24481) 1100 : cluster [DBG] pgmap v1566: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:18:56.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:18:55.250662+0000 mgr.y (mgr.24481) 1101 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:56.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:55.186679+0000 mgr.y (mgr.24481) 1100 : cluster [DBG] pgmap v1566: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:18:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:56 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:18:55.250662+0000 mgr.y (mgr.24481) 1101 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:55.186679+0000 mgr.y (mgr.24481) 1100 : cluster [DBG] pgmap v1566: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:18:56.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:18:55.250662+0000 mgr.y (mgr.24481) 1101 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:18:56.953 INFO:teuthology.orchestra.run.smithi033.stdout:34359738757 2024-02-12T17:18:56.953 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-12T17:18:57.233 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:18:57.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:56.233872+0000 osd.7 (osd.7) 373 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:18:57.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:56.309772+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:18:57.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:56.233872+0000 osd.7 (osd.7) 373 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:18:57.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:56.309772+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:18:57.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:56.233872+0000 osd.7 (osd.7) 373 : cluster [DBG] 6.5 scrub starts 2024-02-12T17:18:57.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:56.309772+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.5 scrub ok 2024-02-12T17:18:58.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:56.817673+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:56.915606+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.187147+0000 mgr.y (mgr.24481) 1102 : cluster [DBG] pgmap v1567: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-02-12T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.265918+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.3 deep-scrub starts 2024-02-12T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.546844+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.549944+0000 osd.0 (osd.0) 291 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.803637+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:18:58.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.804642+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:18:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:56.817673+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:18:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:56.915606+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:18:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.187147+0000 mgr.y (mgr.24481) 1102 : cluster [DBG] pgmap v1567: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-02-12T17:18:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.265918+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.3 deep-scrub starts 2024-02-12T17:18:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.546844+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:18:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.549944+0000 osd.0 (osd.0) 291 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:18:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.803637+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:18:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:58 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.804642+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:18:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:56.817673+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.11 scrub starts 2024-02-12T17:18:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:56.915606+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.11 scrub ok 2024-02-12T17:18:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.187147+0000 mgr.y (mgr.24481) 1102 : cluster [DBG] pgmap v1567: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-02-12T17:18:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.265918+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.3 deep-scrub starts 2024-02-12T17:18:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.546844+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.1 scrub starts 2024-02-12T17:18:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.549944+0000 osd.0 (osd.0) 291 : cluster [DBG] 3.1 scrub ok 2024-02-12T17:18:58.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.803637+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.4 scrub starts 2024-02-12T17:18:58.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.804642+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.4 scrub ok 2024-02-12T17:18:59.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.453699+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.3 deep-scrub ok 2024-02-12T17:18:59.722 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.863508+0000 osd.3 (osd.3) 308 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:18:59.723 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:18:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:57.866973+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:18:59.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.453699+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.3 deep-scrub ok 2024-02-12T17:18:59.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.863508+0000 osd.3 (osd.3) 308 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:18:59.723 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:18:59 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:57.866973+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:18:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.453699+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.3 deep-scrub ok 2024-02-12T17:18:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.863508+0000 osd.3 (osd.3) 308 : cluster [DBG] 4.19 scrub starts 2024-02-12T17:18:59.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:18:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:57.866973+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.19 scrub ok 2024-02-12T17:19:00.044 INFO:teuthology.orchestra.run.smithi033.stdout:2705829396544 2024-02-12T17:19:00.045 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-12T17:19:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:00.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:58.907263+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:19:00.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:59.106841+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:19:00.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:59.187625+0000 mgr.y (mgr.24481) 1103 : cluster [DBG] pgmap v1568: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:00.616 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:59.767221+0000 osd.4 (osd.4) 235 : cluster [DBG] 4.d scrub starts 2024-02-12T17:19:00.617 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:59.770670+0000 osd.4 (osd.4) 236 : cluster [DBG] 4.d scrub ok 2024-02-12T17:19:00.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:58.907263+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:19:00.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:59.106841+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:19:00.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:59.187625+0000 mgr.y (mgr.24481) 1103 : cluster [DBG] pgmap v1568: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:00.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:59.767221+0000 osd.4 (osd.4) 235 : cluster [DBG] 4.d scrub starts 2024-02-12T17:19:00.617 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:59.770670+0000 osd.4 (osd.4) 236 : cluster [DBG] 4.d scrub ok 2024-02-12T17:19:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:58.907263+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.7 scrub starts 2024-02-12T17:19:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:59.106841+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.7 scrub ok 2024-02-12T17:19:00.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:59.187625+0000 mgr.y (mgr.24481) 1103 : cluster [DBG] pgmap v1568: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:59.767221+0000 osd.4 (osd.4) 235 : cluster [DBG] 4.d scrub starts 2024-02-12T17:19:00.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:00 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:59.770670+0000 osd.4 (osd.4) 236 : cluster [DBG] 4.d scrub ok 2024-02-12T17:19:01.276 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:00] "GET /metrics HTTP/1.1" 200 174463 "" "Prometheus/2.33.4" 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:59.875614+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:59.879235+0000 osd.3 (osd.3) 311 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:59.942118+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:18:59.945185+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:00.041892+0000 osd.6 (osd.6) 220 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:00.042818+0000 osd.6 (osd.6) 221 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:00.489485+0000 osd.0 (osd.0) 292 : cluster [DBG] 4.b scrub starts 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:00.493210+0000 osd.0 (osd.0) 293 : cluster [DBG] 4.b scrub ok 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:00.741994+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.c scrub starts 2024-02-12T17:19:01.628 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:00.745485+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.c scrub ok 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:00.828492+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.12 scrub starts 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:00.831417+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.12 scrub ok 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:59.875614+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:59.879235+0000 osd.3 (osd.3) 311 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:59.942118+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:18:59.945185+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:00.041892+0000 osd.6 (osd.6) 220 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:00.042818+0000 osd.6 (osd.6) 221 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:00.489485+0000 osd.0 (osd.0) 292 : cluster [DBG] 4.b scrub starts 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:00.493210+0000 osd.0 (osd.0) 293 : cluster [DBG] 4.b scrub ok 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:00.741994+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.c scrub starts 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:00.745485+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.c scrub ok 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:00.828492+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.12 scrub starts 2024-02-12T17:19:01.629 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:01 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:00.831417+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.12 scrub ok 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:59.875614+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.11 scrub starts 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:59.879235+0000 osd.3 (osd.3) 311 : cluster [DBG] 4.11 scrub ok 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:59.942118+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.3 scrub starts 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:18:59.945185+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.3 scrub ok 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:00.041892+0000 osd.6 (osd.6) 220 : cluster [DBG] 5.11 scrub starts 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:00.042818+0000 osd.6 (osd.6) 221 : cluster [DBG] 5.11 scrub ok 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:00.489485+0000 osd.0 (osd.0) 292 : cluster [DBG] 4.b scrub starts 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:00.493210+0000 osd.0 (osd.0) 293 : cluster [DBG] 4.b scrub ok 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:00.741994+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.c scrub starts 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:00.745485+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.c scrub ok 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:00.828492+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.12 scrub starts 2024-02-12T17:19:01.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:01 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:00.831417+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.12 scrub ok 2024-02-12T17:19:02.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:01.188097+0000 mgr.y (mgr.24481) 1104 : cluster [DBG] pgmap v1569: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:02.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:01.519503+0000 osd.0 (osd.0) 294 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:19:02.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:01.522992+0000 osd.0 (osd.0) 295 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:19:02.712 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:01.188097+0000 mgr.y (mgr.24481) 1104 : cluster [DBG] pgmap v1569: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:02.713 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:01.519503+0000 osd.0 (osd.0) 294 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:19:02.713 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:01.522992+0000 osd.0 (osd.0) 295 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:19:02.739 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:19:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:01.188097+0000 mgr.y (mgr.24481) 1104 : cluster [DBG] pgmap v1569: 102 pgs: 1 active+clean+scrubbing, 101 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:01.519503+0000 osd.0 (osd.0) 294 : cluster [DBG] 5.3 scrub starts 2024-02-12T17:19:02.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:02 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:01.522992+0000 osd.0 (osd.0) 295 : cluster [DBG] 5.3 scrub ok 2024-02-12T17:19:03.032 INFO:teuthology.orchestra.run.smithi033.stdout:77309411707 2024-02-12T17:19:03.033 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-12T17:19:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:03.592 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:01.961550+0000 osd.5 (osd.5) 369 : cluster [DBG] 6.a deep-scrub starts 2024-02-12T17:19:03.592 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:02.041285+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.a deep-scrub ok 2024-02-12T17:19:03.592 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:02.729434+0000 osd.4 (osd.4) 239 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:19:03.592 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:02.732995+0000 osd.4 (osd.4) 240 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:19:03.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:01.961550+0000 osd.5 (osd.5) 369 : cluster [DBG] 6.a deep-scrub starts 2024-02-12T17:19:03.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:02.041285+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.a deep-scrub ok 2024-02-12T17:19:03.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:02.729434+0000 osd.4 (osd.4) 239 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:19:03.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:03 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:02.732995+0000 osd.4 (osd.4) 240 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:19:03.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:01.961550+0000 osd.5 (osd.5) 369 : cluster [DBG] 6.a deep-scrub starts 2024-02-12T17:19:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:02.041285+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.a deep-scrub ok 2024-02-12T17:19:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:02.729434+0000 osd.4 (osd.4) 239 : cluster [DBG] 4.1a scrub starts 2024-02-12T17:19:03.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:03 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:02.732995+0000 osd.4 (osd.4) 240 : cluster [DBG] 4.1a scrub ok 2024-02-12T17:19:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:03.188523+0000 mgr.y (mgr.24481) 1105 : cluster [DBG] pgmap v1570: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:03.324896+0000 osd.7 (osd.7) 377 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:19:04.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:04 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:03.327872+0000 osd.7 (osd.7) 378 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:19:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:03.188523+0000 mgr.y (mgr.24481) 1105 : cluster [DBG] pgmap v1570: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:03.324896+0000 osd.7 (osd.7) 377 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:19:04.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:03.327872+0000 osd.7 (osd.7) 378 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:19:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:03.188523+0000 mgr.y (mgr.24481) 1105 : cluster [DBG] pgmap v1570: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:04.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:03.324896+0000 osd.7 (osd.7) 377 : cluster [DBG] 5.10 scrub starts 2024-02-12T17:19:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:04 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:03.327872+0000 osd.7 (osd.7) 378 : cluster [DBG] 5.10 scrub ok 2024-02-12T17:19:05.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:03.755554+0000 osd.3 (osd.3) 312 : cluster [DBG] 4.1d deep-scrub starts 2024-02-12T17:19:05.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:03.760778+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.1d deep-scrub ok 2024-02-12T17:19:05.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:03.788469+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:19:05.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:03.789381+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:19:05.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:04.728643+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:19:05.375 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:04.732106+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:19:05.376 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:03.755554+0000 osd.3 (osd.3) 312 : cluster [DBG] 4.1d deep-scrub starts 2024-02-12T17:19:05.376 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:03.760778+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.1d deep-scrub ok 2024-02-12T17:19:05.376 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:03.788469+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:19:05.376 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:03.789381+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:19:05.376 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:04.728643+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:19:05.376 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:05 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:04.732106+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:19:05.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:19:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:19:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:03.755554+0000 osd.3 (osd.3) 312 : cluster [DBG] 4.1d deep-scrub starts 2024-02-12T17:19:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:03.760778+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.1d deep-scrub ok 2024-02-12T17:19:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:03.788469+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.6 scrub starts 2024-02-12T17:19:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:03.789381+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.6 scrub ok 2024-02-12T17:19:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:04.728643+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1b scrub starts 2024-02-12T17:19:05.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:05 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:04.732106+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1b scrub ok 2024-02-12T17:19:06.043 INFO:teuthology.orchestra.run.smithi033.stdout:111669150070 2024-02-12T17:19:06.043 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-12T17:19:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:06.592 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:05.189188+0000 mgr.y (mgr.24481) 1106 : cluster [DBG] pgmap v1571: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:06.592 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:05.258838+0000 mgr.y (mgr.24481) 1107 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:06.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:05.189188+0000 mgr.y (mgr.24481) 1106 : cluster [DBG] pgmap v1571: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:06.592 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:06 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:05.258838+0000 mgr.y (mgr.24481) 1107 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:05.189188+0000 mgr.y (mgr.24481) 1106 : cluster [DBG] pgmap v1571: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:06.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:06 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:05.258838+0000 mgr.y (mgr.24481) 1107 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:08.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:06.973356+0000 osd.5 (osd.5) 371 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:19:08.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:06.975251+0000 osd.5 (osd.5) 372 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:19:08.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:07.189742+0000 mgr.y (mgr.24481) 1108 : cluster [DBG] pgmap v1572: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:08.011 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:06.973356+0000 osd.5 (osd.5) 371 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:19:08.011 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:06.975251+0000 osd.5 (osd.5) 372 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:19:08.011 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:07 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:07.189742+0000 mgr.y (mgr.24481) 1108 : cluster [DBG] pgmap v1572: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:08.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:06.973356+0000 osd.5 (osd.5) 371 : cluster [DBG] 4.15 scrub starts 2024-02-12T17:19:08.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:06.975251+0000 osd.5 (osd.5) 372 : cluster [DBG] 4.15 scrub ok 2024-02-12T17:19:08.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:07 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:07.189742+0000 mgr.y (mgr.24481) 1108 : cluster [DBG] pgmap v1572: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:08.984 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:07.760544+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T17:19:08.984 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:07.764479+0000 osd.1 (osd.1) 81 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T17:19:08.984 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:07.760544+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T17:19:08.985 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:07.764479+0000 osd.1 (osd.1) 81 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T17:19:09.005 INFO:teuthology.orchestra.run.smithi033.stdout:137438953844 2024-02-12T17:19:09.005 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-12T17:19:09.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:07.760544+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.f deep-scrub starts 2024-02-12T17:19:09.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:08 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:07.764479+0000 osd.1 (osd.1) 81 : cluster [DBG] 4.f deep-scrub ok 2024-02-12T17:19:09.289 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:10.227 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:08.784695+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:19:10.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:08.787805+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:19:10.228 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:09.190196+0000 mgr.y (mgr.24481) 1109 : cluster [DBG] pgmap v1573: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:10.228 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:08.784695+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:19:10.228 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:08.787805+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:19:10.229 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:09 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:09.190196+0000 mgr.y (mgr.24481) 1109 : cluster [DBG] pgmap v1573: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:08.784695+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.6 scrub starts 2024-02-12T17:19:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:08.787805+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.6 scrub ok 2024-02-12T17:19:10.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:09 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:09.190196+0000 mgr.y (mgr.24481) 1109 : cluster [DBG] pgmap v1573: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:11.150 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:10] "GET /metrics HTTP/1.1" 200 174461 "" "Prometheus/2.33.4" 2024-02-12T17:19:11.151 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:09.746866+0000 osd.1 (osd.1) 84 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:19:11.151 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:09.750119+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:19:11.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:09.746866+0000 osd.1 (osd.1) 84 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:19:11.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:10 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:09.750119+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:19:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:09.746866+0000 osd.1 (osd.1) 84 : cluster [DBG] 5.7 scrub starts 2024-02-12T17:19:11.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:10 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:09.750119+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.7 scrub ok 2024-02-12T17:19:11.948 INFO:teuthology.orchestra.run.smithi033.stdout:163208757617 2024-02-12T17:19:11.949 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-12T17:19:12.232 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:12.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:11 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:11.190662+0000 mgr.y (mgr.24481) 1110 : cluster [DBG] pgmap v1574: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:12.277 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:11.190662+0000 mgr.y (mgr.24481) 1110 : cluster [DBG] pgmap v1574: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:12.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:11.190662+0000 mgr.y (mgr.24481) 1110 : cluster [DBG] pgmap v1574: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:12.868 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:19:12.868 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:11.828460+0000 osd.6 (osd.6) 222 : cluster [DBG] 4.1f deep-scrub starts 2024-02-12T17:19:12.868 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:11.833190+0000 osd.6 (osd.6) 223 : cluster [DBG] 4.1f deep-scrub ok 2024-02-12T17:19:12.869 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:12 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:12.545121+0000 mon.c (mon.2) 339 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:19:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:11.828460+0000 osd.6 (osd.6) 222 : cluster [DBG] 4.1f deep-scrub starts 2024-02-12T17:19:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:11.833190+0000 osd.6 (osd.6) 223 : cluster [DBG] 4.1f deep-scrub ok 2024-02-12T17:19:13.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:12.545121+0000 mon.c (mon.2) 339 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:19:13.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:11.828460+0000 osd.6 (osd.6) 222 : cluster [DBG] 4.1f deep-scrub starts 2024-02-12T17:19:13.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:11.833190+0000 osd.6 (osd.6) 223 : cluster [DBG] 4.1f deep-scrub ok 2024-02-12T17:19:13.028 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:12 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:12.545121+0000 mon.c (mon.2) 339 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:12.781422+0000 osd.4 (osd.4) 243 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:12.784942+0000 osd.4 (osd.4) 244 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:12.866316+0000 osd.6 (osd.6) 224 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:12.868460+0000 osd.6 (osd.6) 225 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:12.927619+0000 mon.c (mon.2) 340 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:12.928318+0000 mon.c (mon.2) 341 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:12.932791+0000 mon.a (mon.0) 2418 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:13.191135+0000 mgr.y (mgr.24481) 1111 : cluster [DBG] pgmap v1575: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:12.781422+0000 osd.4 (osd.4) 243 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:12.784942+0000 osd.4 (osd.4) 244 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:19:14.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:12.866316+0000 osd.6 (osd.6) 224 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:19:14.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:12.868460+0000 osd.6 (osd.6) 225 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:19:14.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:12.927619+0000 mon.c (mon.2) 340 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:19:14.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:12.928318+0000 mon.c (mon.2) 341 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:19:14.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:12.932791+0000 mon.a (mon.0) 2418 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:19:14.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:13 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:13.191135+0000 mgr.y (mgr.24481) 1111 : cluster [DBG] pgmap v1575: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:12.781422+0000 osd.4 (osd.4) 243 : cluster [DBG] 4.9 scrub starts 2024-02-12T17:19:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:12.784942+0000 osd.4 (osd.4) 244 : cluster [DBG] 4.9 scrub ok 2024-02-12T17:19:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:12.866316+0000 osd.6 (osd.6) 224 : cluster [DBG] 4.5 deep-scrub starts 2024-02-12T17:19:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:12.868460+0000 osd.6 (osd.6) 225 : cluster [DBG] 4.5 deep-scrub ok 2024-02-12T17:19:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:12.927619+0000 mon.c (mon.2) 340 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T17:19:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:12.928318+0000 mon.c (mon.2) 341 : audit [INF] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T17:19:14.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:12.932791+0000 mon.a (mon.0) 2418 : audit [INF] from='mgr.24481 ' entity='mgr.y' 2024-02-12T17:19:14.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:13 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:13.191135+0000 mgr.y (mgr.24481) 1111 : cluster [DBG] pgmap v1575: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:14.867 INFO:teuthology.orchestra.run.smithi033.stdout:193273528684 2024-02-12T17:19:14.867 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-12T17:19:15.152 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:15.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:19:15 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:19:16.499 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:15.191736+0000 mgr.y (mgr.24481) 1112 : cluster [DBG] pgmap v1576: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:16.499 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:15.263364+0000 mgr.y (mgr.24481) 1113 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:16.499 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:15.749177+0000 osd.1 (osd.1) 86 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:19:16.499 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:15.752756+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:19:16.500 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:15.191736+0000 mgr.y (mgr.24481) 1112 : cluster [DBG] pgmap v1576: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:16.500 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:15.263364+0000 mgr.y (mgr.24481) 1113 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:16.500 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:15.749177+0000 osd.1 (osd.1) 86 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:19:16.500 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:15.752756+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:19:16.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:15.191736+0000 mgr.y (mgr.24481) 1112 : cluster [DBG] pgmap v1576: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:16.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:15.263364+0000 mgr.y (mgr.24481) 1113 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:16.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:15.749177+0000 osd.1 (osd.1) 86 : cluster [DBG] 4.2 scrub starts 2024-02-12T17:19:16.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:16 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:15.752756+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub ok 2024-02-12T17:19:17.524 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:15.742771+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:19:17.525 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:15.745827+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:19:17.525 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:15.742771+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:19:17.525 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:17 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:15.745827+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:19:17.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:15.742771+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.2 scrub starts 2024-02-12T17:19:17.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:17 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:15.745827+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.2 scrub ok 2024-02-12T17:19:17.863 INFO:teuthology.orchestra.run.smithi033.stdout:231928234343 2024-02-12T17:19:17.863 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-12T17:19:18.146 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:18.249 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:17.192140+0000 mgr.y (mgr.24481) 1114 : cluster [DBG] pgmap v1577: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:18.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:18 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:17.192140+0000 mgr.y (mgr.24481) 1114 : cluster [DBG] pgmap v1577: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:18.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:18 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:17.192140+0000 mgr.y (mgr.24481) 1114 : cluster [DBG] pgmap v1577: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:17.794704+0000 osd.3 (osd.3) 316 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:19:19.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:17.798191+0000 osd.3 (osd.3) 317 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:19:19.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:17.794704+0000 osd.3 (osd.3) 316 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:19:19.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:19 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:17.798191+0000 osd.3 (osd.3) 317 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:19:19.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:17.794704+0000 osd.3 (osd.3) 316 : cluster [DBG] 5.14 scrub starts 2024-02-12T17:19:19.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:19 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:17.798191+0000 osd.3 (osd.3) 317 : cluster [DBG] 5.14 scrub ok 2024-02-12T17:19:20.184 INFO:teuthology.orchestra.run.smithi033.stdout:34359738761 2024-02-12T17:19:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:19.192670+0000 mgr.y (mgr.24481) 1115 : cluster [DBG] pgmap v1578: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:20.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:20.183986+0000 mon.c (mon.2) 342 : audit [DBG] from='client.? v1:172.21.15.33:0/2891685621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T17:19:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:19.192670+0000 mgr.y (mgr.24481) 1115 : cluster [DBG] pgmap v1578: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:20.183986+0000 mon.c (mon.2) 342 : audit [DBG] from='client.? v1:172.21.15.33:0/2891685621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T17:19:20.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:19.192670+0000 mgr.y (mgr.24481) 1115 : cluster [DBG] pgmap v1578: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:20.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:20.183986+0000 mon.c (mon.2) 342 : audit [DBG] from='client.? v1:172.21.15.33:0/2891685621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T17:19:20.953 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738757 got 34359738761 for osd.0 2024-02-12T17:19:20.953 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-12T17:19:21.236 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:21.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:20 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:20] "GET /metrics HTTP/1.1" 200 174455 "" "Prometheus/2.33.4" 2024-02-12T17:19:22.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:20.951707+0000 osd.5 (osd.5) 373 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:19:22.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:20.953217+0000 osd.5 (osd.5) 374 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:19:22.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:21.193075+0000 mgr.y (mgr.24481) 1116 : cluster [DBG] pgmap v1579: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:22.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:21.288222+0000 osd.7 (osd.7) 379 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:19:22.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:21.291086+0000 osd.7 (osd.7) 380 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:19:22.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:21.777369+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:19:22.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:21.821326+0000 osd.4 (osd.4) 246 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:19:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:20.951707+0000 osd.5 (osd.5) 373 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:19:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:20.953217+0000 osd.5 (osd.5) 374 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:19:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:21.193075+0000 mgr.y (mgr.24481) 1116 : cluster [DBG] pgmap v1579: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:21.288222+0000 osd.7 (osd.7) 379 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:19:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:21.291086+0000 osd.7 (osd.7) 380 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:19:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:21.777369+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:19:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:22 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:21.821326+0000 osd.4 (osd.4) 246 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:19:22.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:19:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:20.951707+0000 osd.5 (osd.5) 373 : cluster [DBG] 5.15 scrub starts 2024-02-12T17:19:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:20.953217+0000 osd.5 (osd.5) 374 : cluster [DBG] 5.15 scrub ok 2024-02-12T17:19:22.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:21.193075+0000 mgr.y (mgr.24481) 1116 : cluster [DBG] pgmap v1579: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:21.288222+0000 osd.7 (osd.7) 379 : cluster [DBG] 5.4 scrub starts 2024-02-12T17:19:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:21.291086+0000 osd.7 (osd.7) 380 : cluster [DBG] 5.4 scrub ok 2024-02-12T17:19:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:21.777369+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.2 scrub starts 2024-02-12T17:19:22.741 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:22 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:21.821326+0000 osd.4 (osd.4) 246 : cluster [DBG] 6.2 scrub ok 2024-02-12T17:19:23.270 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:21.933662+0000 osd.5 (osd.5) 375 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:19:23.270 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:21.936571+0000 osd.5 (osd.5) 376 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:19:23.416 INFO:teuthology.orchestra.run.smithi033.stdout:2705829396549 2024-02-12T17:19:23.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:21.933662+0000 osd.5 (osd.5) 375 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:19:23.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:23 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:21.936571+0000 osd.5 (osd.5) 376 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:19:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:21.933662+0000 osd.5 (osd.5) 375 : cluster [DBG] 2.12 scrub starts 2024-02-12T17:19:23.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:23 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:21.936571+0000 osd.5 (osd.5) 376 : cluster [DBG] 2.12 scrub ok 2024-02-12T17:19:24.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2705829396544 got 2705829396549 for osd.1 2024-02-12T17:19:24.139 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-12T17:19:24.427 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:24.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:23.193565+0000 mgr.y (mgr.24481) 1117 : cluster [DBG] pgmap v1580: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:24.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:23.415552+0000 mon.c (mon.2) 343 : audit [DBG] from='client.? v1:172.21.15.33:0/361811647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T17:19:24.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:23.672294+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:19:24.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:23.695647+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:19:24.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:23.193565+0000 mgr.y (mgr.24481) 1117 : cluster [DBG] pgmap v1580: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:24.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:23.415552+0000 mon.c (mon.2) 343 : audit [DBG] from='client.? v1:172.21.15.33:0/361811647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T17:19:24.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:23.672294+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:19:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:23.695647+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:19:24.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:23.193565+0000 mgr.y (mgr.24481) 1117 : cluster [DBG] pgmap v1580: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:24.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:23.415552+0000 mon.c (mon.2) 343 : audit [DBG] from='client.? v1:172.21.15.33:0/361811647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T17:19:24.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:23.672294+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.6 scrub starts 2024-02-12T17:19:24.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:24 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:23.695647+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.6 scrub ok 2024-02-12T17:19:25.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:19:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:19:25.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:23.732539+0000 osd.2 (osd.2) 267 : cluster [DBG] 2.c scrub starts 2024-02-12T17:19:25.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:25 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:23.733493+0000 osd.2 (osd.2) 268 : cluster [DBG] 2.c scrub ok 2024-02-12T17:19:25.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:23.732539+0000 osd.2 (osd.2) 267 : cluster [DBG] 2.c scrub starts 2024-02-12T17:19:25.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:23.733493+0000 osd.2 (osd.2) 268 : cluster [DBG] 2.c scrub ok 2024-02-12T17:19:25.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:23.732539+0000 osd.2 (osd.2) 267 : cluster [DBG] 2.c scrub starts 2024-02-12T17:19:25.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:25 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:23.733493+0000 osd.2 (osd.2) 268 : cluster [DBG] 2.c scrub ok 2024-02-12T17:19:26.525 INFO:teuthology.orchestra.run.smithi033.stdout:77309411712 2024-02-12T17:19:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:24.757055+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:19:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:24.758041+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:19:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:25.194139+0000 mgr.y (mgr.24481) 1118 : cluster [DBG] pgmap v1581: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:25.264834+0000 mgr.y (mgr.24481) 1119 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:25.384528+0000 osd.0 (osd.0) 296 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:19:26.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:25.386624+0000 osd.0 (osd.0) 297 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:19:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:24.757055+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:19:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:24.758041+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:19:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:25.194139+0000 mgr.y (mgr.24481) 1118 : cluster [DBG] pgmap v1581: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:25.264834+0000 mgr.y (mgr.24481) 1119 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:25.384528+0000 osd.0 (osd.0) 296 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:19:26.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:26 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:25.386624+0000 osd.0 (osd.0) 297 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:19:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:24.757055+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.8 scrub starts 2024-02-12T17:19:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:24.758041+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.8 scrub ok 2024-02-12T17:19:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:25.194139+0000 mgr.y (mgr.24481) 1118 : cluster [DBG] pgmap v1581: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:25.264834+0000 mgr.y (mgr.24481) 1119 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:25.384528+0000 osd.0 (osd.0) 296 : cluster [DBG] 4.1e scrub starts 2024-02-12T17:19:26.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:25.386624+0000 osd.0 (osd.0) 297 : cluster [DBG] 4.1e scrub ok 2024-02-12T17:19:27.599 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:25.940851+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.f scrub starts 2024-02-12T17:19:27.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:25.962347+0000 osd.5 (osd.5) 378 : cluster [DBG] 5.f scrub ok 2024-02-12T17:19:27.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:26.524428+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.33:0/1222937650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T17:19:27.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:26.828347+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:19:27.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:26.831811+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:19:27.600 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:25.940851+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.f scrub starts 2024-02-12T17:19:27.600 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:25.962347+0000 osd.5 (osd.5) 378 : cluster [DBG] 5.f scrub ok 2024-02-12T17:19:27.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:26.524428+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.33:0/1222937650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T17:19:27.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:26.828347+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:19:27.601 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:27 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:26.831811+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:19:27.623 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411707 got 77309411712 for osd.2 2024-02-12T17:19:27.623 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-12T17:19:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:25.940851+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.f scrub starts 2024-02-12T17:19:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:25.962347+0000 osd.5 (osd.5) 378 : cluster [DBG] 5.f scrub ok 2024-02-12T17:19:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:26.524428+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.33:0/1222937650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T17:19:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:26.828347+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.13 scrub starts 2024-02-12T17:19:27.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:26.831811+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.13 scrub ok 2024-02-12T17:19:27.909 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:28.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:27.194800+0000 mgr.y (mgr.24481) 1120 : cluster [DBG] pgmap v1582: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:28.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:27.591665+0000 osd.3 (osd.3) 320 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:19:28.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:27.594770+0000 osd.3 (osd.3) 321 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:19:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:27.840150+0000 osd.4 (osd.4) 249 : cluster [DBG] 5.e scrub starts 2024-02-12T17:19:28.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:27.843643+0000 osd.4 (osd.4) 250 : cluster [DBG] 5.e scrub ok 2024-02-12T17:19:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:27.194800+0000 mgr.y (mgr.24481) 1120 : cluster [DBG] pgmap v1582: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:27.591665+0000 osd.3 (osd.3) 320 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:19:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:27.594770+0000 osd.3 (osd.3) 321 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:19:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:27.840150+0000 osd.4 (osd.4) 249 : cluster [DBG] 5.e scrub starts 2024-02-12T17:19:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:27.843643+0000 osd.4 (osd.4) 250 : cluster [DBG] 5.e scrub ok 2024-02-12T17:19:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:27.194800+0000 mgr.y (mgr.24481) 1120 : cluster [DBG] pgmap v1582: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:27.591665+0000 osd.3 (osd.3) 320 : cluster [DBG] 5.13 scrub starts 2024-02-12T17:19:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:27.594770+0000 osd.3 (osd.3) 321 : cluster [DBG] 5.13 scrub ok 2024-02-12T17:19:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:27.840150+0000 osd.4 (osd.4) 249 : cluster [DBG] 5.e scrub starts 2024-02-12T17:19:28.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:27.843643+0000 osd.4 (osd.4) 250 : cluster [DBG] 5.e scrub ok 2024-02-12T17:19:29.599 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:27.957163+0000 osd.2 (osd.2) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:29.599 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:27.957214+0000 osd.2 (osd.2) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:29.599 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:28.628591+0000 osd.3 (osd.3) 322 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:19:29.599 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:28.632061+0000 osd.3 (osd.3) 323 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:19:29.600 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:28.854567+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-12T17:19:29.600 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:28.859083+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:19:29.600 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:28.860133+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:19:29.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:27.957163+0000 osd.2 (osd.2) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:29.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:27.957214+0000 osd.2 (osd.2) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:29.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:28.628591+0000 osd.3 (osd.3) 322 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:19:29.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:28.632061+0000 osd.3 (osd.3) 323 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:19:29.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:28.854567+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-12T17:19:29.601 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:28.859083+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:19:29.601 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:29 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:28.860133+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:19:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:27.957163+0000 osd.2 (osd.2) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:27.957214+0000 osd.2 (osd.2) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:28.628591+0000 osd.3 (osd.3) 322 : cluster [DBG] 5.0 scrub starts 2024-02-12T17:19:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:28.632061+0000 osd.3 (osd.3) 323 : cluster [DBG] 5.0 scrub ok 2024-02-12T17:19:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:28.854567+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-12T17:19:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:28.859083+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.15 scrub starts 2024-02-12T17:19:29.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:28.860133+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.15 scrub ok 2024-02-12T17:19:30.025 INFO:teuthology.orchestra.run.smithi033.stdout:111669150075 2024-02-12T17:19:30.456 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:29.195157+0000 mgr.y (mgr.24481) 1121 : cluster [DBG] pgmap v1584: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:19:30.457 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:30.025280+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? v1:172.21.15.33:0/2820943200' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T17:19:30.457 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:29.195157+0000 mgr.y (mgr.24481) 1121 : cluster [DBG] pgmap v1584: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:19:30.457 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:30.025280+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? v1:172.21.15.33:0/2820943200' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T17:19:30.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:29.195157+0000 mgr.y (mgr.24481) 1121 : cluster [DBG] pgmap v1584: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:19:30.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:30.025280+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? v1:172.21.15.33:0/2820943200' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T17:19:30.882 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150070 got 111669150075 for osd.3 2024-02-12T17:19:30.882 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-12T17:19:31.169 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:31.277 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:30 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:30] "GET /metrics HTTP/1.1" 200 174455 "" "Prometheus/2.33.4" 2024-02-12T17:19:31.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:30.346152+0000 osd.0 (osd.0) 298 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T17:19:31.681 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:30.348694+0000 osd.0 (osd.0) 299 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T17:19:31.681 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:30.346152+0000 osd.0 (osd.0) 298 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T17:19:31.681 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:31 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:30.348694+0000 osd.0 (osd.0) 299 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T17:19:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:30.346152+0000 osd.0 (osd.0) 298 : cluster [DBG] 4.16 deep-scrub starts 2024-02-12T17:19:31.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:30.348694+0000 osd.0 (osd.0) 299 : cluster [DBG] 4.16 deep-scrub ok 2024-02-12T17:19:32.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:31.195555+0000 mgr.y (mgr.24481) 1122 : cluster [DBG] pgmap v1585: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:19:32.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:31.850245+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-12T17:19:32.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:31.195555+0000 mgr.y (mgr.24481) 1122 : cluster [DBG] pgmap v1585: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:19:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:32 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:31.850245+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-12T17:19:32.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:19:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:31.195555+0000 mgr.y (mgr.24481) 1122 : cluster [DBG] pgmap v1585: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-12T17:19:32.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:31.850245+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-12T17:19:33.347 INFO:teuthology.orchestra.run.smithi033.stdout:137438953849 2024-02-12T17:19:34.060 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953844 got 137438953849 for osd.4 2024-02-12T17:19:34.060 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-12T17:19:34.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:33.195908+0000 mgr.y (mgr.24481) 1123 : cluster [DBG] pgmap v1587: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:19:34.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:33.346619+0000 mon.c (mon.2) 346 : audit [DBG] from='client.? v1:172.21.15.33:0/108906482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T17:19:34.304 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:33.802831+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:19:34.305 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:33.805872+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:19:34.344 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:34.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:33.195908+0000 mgr.y (mgr.24481) 1123 : cluster [DBG] pgmap v1587: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:19:34.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:33.346619+0000 mon.c (mon.2) 346 : audit [DBG] from='client.? v1:172.21.15.33:0/108906482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T17:19:34.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:33.802831+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:19:34.619 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:34 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:33.805872+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:19:34.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:33.195908+0000 mgr.y (mgr.24481) 1123 : cluster [DBG] pgmap v1587: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:19:34.756 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:33.346619+0000 mon.c (mon.2) 346 : audit [DBG] from='client.? v1:172.21.15.33:0/108906482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T17:19:34.756 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:33.802831+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.9 scrub starts 2024-02-12T17:19:34.756 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:33.805872+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.9 scrub ok 2024-02-12T17:19:35.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:33.785180+0000 osd.2 (osd.2) 273 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:19:35.713 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:33.787055+0000 osd.2 (osd.2) 274 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:19:35.713 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:33.785180+0000 osd.2 (osd.2) 273 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:19:35.713 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:35 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:33.787055+0000 osd.2 (osd.2) 274 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:19:35.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:19:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:19:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:33.785180+0000 osd.2 (osd.2) 273 : cluster [DBG] 4.1c scrub starts 2024-02-12T17:19:35.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:33.787055+0000 osd.2 (osd.2) 274 : cluster [DBG] 4.1c scrub ok 2024-02-12T17:19:36.415 INFO:teuthology.orchestra.run.smithi033.stdout:163208757622 2024-02-12T17:19:36.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:34.466415+0000 osd.7 (osd.7) 381 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:34.466471+0000 osd.7 (osd.7) 382 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:35.196470+0000 mgr.y (mgr.24481) 1124 : cluster [DBG] pgmap v1588: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:35.275791+0000 mgr.y (mgr.24481) 1125 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:36.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:35.315319+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-12T17:19:36.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:34.466415+0000 osd.7 (osd.7) 381 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:36.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:34.466471+0000 osd.7 (osd.7) 382 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:36.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:35.196470+0000 mgr.y (mgr.24481) 1124 : cluster [DBG] pgmap v1588: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:36.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:35.275791+0000 mgr.y (mgr.24481) 1125 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:36.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:35.315319+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-12T17:19:36.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:34.466415+0000 osd.7 (osd.7) 381 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:36.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:34.466471+0000 osd.7 (osd.7) 382 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:36.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:35.196470+0000 mgr.y (mgr.24481) 1124 : cluster [DBG] pgmap v1588: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:36.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:35.275791+0000 mgr.y (mgr.24481) 1125 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:36.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:36 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:35.315319+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-12T17:19:37.219 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757617 got 163208757622 for osd.5 2024-02-12T17:19:37.219 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-12T17:19:37.503 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:37.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:36.414820+0000 mon.a (mon.0) 2422 : audit [DBG] from='client.? v1:172.21.15.33:0/806785660' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T17:19:37.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:36.852088+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-12T17:19:37.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:36.414820+0000 mon.a (mon.0) 2422 : audit [DBG] from='client.? v1:172.21.15.33:0/806785660' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T17:19:37.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:37 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:36.852088+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-12T17:19:37.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:36.414820+0000 mon.a (mon.0) 2422 : audit [DBG] from='client.? v1:172.21.15.33:0/806785660' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T17:19:37.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:36.852088+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-12T17:19:38.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:37.196916+0000 mgr.y (mgr.24481) 1126 : cluster [DBG] pgmap v1591: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:38.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:37.196916+0000 mgr.y (mgr.24481) 1126 : cluster [DBG] pgmap v1591: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:38.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:38 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:37.196916+0000 mgr.y (mgr.24481) 1126 : cluster [DBG] pgmap v1591: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-12T17:19:39.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:37.791861+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:19:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:37.795810+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:19:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:37.791861+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:19:39.528 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:39 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:37.795810+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:19:39.699 INFO:teuthology.orchestra.run.smithi033.stdout:193273528689 2024-02-12T17:19:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:37.791861+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.0 scrub starts 2024-02-12T17:19:39.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:37.795810+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub ok 2024-02-12T17:19:40.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528684 got 193273528689 for osd.6 2024-02-12T17:19:40.525 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-12T17:19:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:39.197515+0000 mgr.y (mgr.24481) 1127 : cluster [DBG] pgmap v1592: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 696 B/s rd, 0 op/s 2024-02-12T17:19:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:39.425466+0000 osd.0 (osd.0) 300 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:19:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:39.427729+0000 osd.0 (osd.0) 301 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:19:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:39.698777+0000 mon.c (mon.2) 347 : audit [DBG] from='client.? v1:172.21.15.33:0/3019519044' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T17:19:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:39.931749+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.e scrub starts 2024-02-12T17:19:40.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:39.935379+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.e scrub ok 2024-02-12T17:19:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:39.197515+0000 mgr.y (mgr.24481) 1127 : cluster [DBG] pgmap v1592: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 696 B/s rd, 0 op/s 2024-02-12T17:19:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:39.425466+0000 osd.0 (osd.0) 300 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:19:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:39.427729+0000 osd.0 (osd.0) 301 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:19:40.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:39.698777+0000 mon.c (mon.2) 347 : audit [DBG] from='client.? v1:172.21.15.33:0/3019519044' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T17:19:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:39.931749+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.e scrub starts 2024-02-12T17:19:40.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:39.935379+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.e scrub ok 2024-02-12T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:39.197515+0000 mgr.y (mgr.24481) 1127 : cluster [DBG] pgmap v1592: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 696 B/s rd, 0 op/s 2024-02-12T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:39.425466+0000 osd.0 (osd.0) 300 : cluster [DBG] 4.6 scrub starts 2024-02-12T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:39.427729+0000 osd.0 (osd.0) 301 : cluster [DBG] 4.6 scrub ok 2024-02-12T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:39.698777+0000 mon.c (mon.2) 347 : audit [DBG] from='client.? v1:172.21.15.33:0/3019519044' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:39.931749+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.e scrub starts 2024-02-12T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:39.935379+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.e scrub ok 2024-02-12T17:19:40.811 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:41.145 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:40 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:40] "GET /metrics HTTP/1.1" 200 174464 "" "Prometheus/2.33.4" 2024-02-12T17:19:42.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:40.995470+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:19:42.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:40.996493+0000 osd.6 (osd.6) 227 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:19:42.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:41.198039+0000 mgr.y (mgr.24481) 1128 : cluster [DBG] pgmap v1593: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:42.328 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:41.429821+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:19:42.329 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:41.469105+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:19:42.740 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:19:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:40.995470+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:19:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:40.996493+0000 osd.6 (osd.6) 227 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:19:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:41.198039+0000 mgr.y (mgr.24481) 1128 : cluster [DBG] pgmap v1593: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:41.429821+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:19:42.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:41.469105+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:19:42.759 INFO:teuthology.orchestra.run.smithi033.stdout:231928234348 2024-02-12T17:19:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:40.995470+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.14 scrub starts 2024-02-12T17:19:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:40.996493+0000 osd.6 (osd.6) 227 : cluster [DBG] 2.14 scrub ok 2024-02-12T17:19:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:41.198039+0000 mgr.y (mgr.24481) 1128 : cluster [DBG] pgmap v1593: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:41.429821+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.10 scrub starts 2024-02-12T17:19:42.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:42 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:41.469105+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.10 scrub ok 2024-02-12T17:19:43.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:42.758495+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.33:0/816802912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T17:19:43.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:43 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:42.758495+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.33:0/816802912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T17:19:43.582 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234343 got 231928234348 for osd.7 2024-02-12T17:19:43.582 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T17:19:43.583 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:19:43.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:42.758495+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.33:0/816802912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T17:19:43.868 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:44.527 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:43.198588+0000 mgr.y (mgr.24481) 1129 : cluster [DBG] pgmap v1594: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:19:44.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:44 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:43.198588+0000 mgr.y (mgr.24481) 1129 : cluster [DBG] pgmap v1594: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:19:44.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:43.198588+0000 mgr.y (mgr.24481) 1129 : cluster [DBG] pgmap v1594: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-12T17:19:45.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:19:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:19:45.884 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:19:45.887 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:19:46.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:45.199159+0000 mgr.y (mgr.24481) 1130 : cluster [DBG] pgmap v1595: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T17:19:46.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:45.277564+0000 mgr.y (mgr.24481) 1131 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:46.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:45.295132+0000 osd.0 (osd.0) 304 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:46.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:45.295184+0000 osd.0 (osd.0) 305 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:46.727 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:45.347229+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-12T17:19:46.728 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:45.199159+0000 mgr.y (mgr.24481) 1130 : cluster [DBG] pgmap v1595: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T17:19:46.728 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:45.277564+0000 mgr.y (mgr.24481) 1131 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:46.728 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:45.295132+0000 osd.0 (osd.0) 304 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:46.728 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:45.295184+0000 osd.0 (osd.0) 305 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:46.728 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:46 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:45.347229+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-12T17:19:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:45.199159+0000 mgr.y (mgr.24481) 1130 : cluster [DBG] pgmap v1595: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-12T17:19:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:45.277564+0000 mgr.y (mgr.24481) 1131 : audit [DBG] from='client.24457 v1:172.21.15.94:0/2103276977' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-12T17:19:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:45.295132+0000 osd.0 (osd.0) 304 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:45.295184+0000 osd.0 (osd.0) 305 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:46.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:45.347229+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-12T17:19:46.749 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2024-02-12T17:19:45.344980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":2101,"num_read_kb":1853,"num_write":1000,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":22367,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1217,"ondisk_log_size":1217,"up":306,"acting":306,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":306,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8437832,"kb_used_data":29704,"kb_used_omap":3,"kb_used_meta":8408060,"kb_avail":741523384,"statfs":{"total":767960285184,"available":759319945216,"internally_reserved":0,"allocated":30416896,"data_stored":17680742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":8609853501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.494793"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":505,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829799+0000","last_change":"2024-02-12T17:18:57.867053+0000","last_active":"2024-02-12T17:19:35.829799+0000","last_peered":"2024-02-12T17:19:35.829799+0000","last_clean":"2024-02-12T17:19:35.829799+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:19:35.829799+0000","last_undegraded":"2024-02-12T17:19:35.829799+0000","last_fullsized":"2024-02-12T17:19:35.829799+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'39","last_scrub_stamp":"2024-02-12T17:18:57.866977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:18:57.866977+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"526'25","reported_seq":70,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314223+0000","last_change":"2024-02-12T17:18:45.706297+0000","last_active":"2024-02-12T17:19:35.314223+0000","last_peered":"2024-02-12T17:19:35.314223+0000","last_clean":"2024-02-12T17:19:35.314223+0000","last_became_active":"2024-02-12T17:17:26.207842+0000","last_became_peered":"2024-02-12T17:17:26.207842+0000","last_unstale":"2024-02-12T17:19:35.314223+0000","last_undegraded":"2024-02-12T17:19:35.314223+0000","last_fullsized":"2024-02-12T17:19:35.314223+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:18:45.706216+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:18:45.706216+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"526'9","reported_seq":517,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314450+0000","last_change":"2024-02-12T17:19:04.732212+0000","last_active":"2024-02-12T17:19:35.314450+0000","last_peered":"2024-02-12T17:19:35.314450+0000","last_clean":"2024-02-12T17:19:35.314450+0000","last_became_active":"2024-02-12T17:18:16.703769+0000","last_became_peered":"2024-02-12T17:18:16.703769+0000","last_unstale":"2024-02-12T17:19:35.314450+0000","last_undegraded":"2024-02-12T17:19:35.314450+0000","last_fullsized":"2024-02-12T17:19:35.314450+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:19:04.732111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:04.732111+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"526'21","reported_seq":69,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314300+0000","last_change":"2024-02-12T17:19:02.733076+0000","last_active":"2024-02-12T17:19:35.314300+0000","last_peered":"2024-02-12T17:19:35.314300+0000","last_clean":"2024-02-12T17:19:35.314300+0000","last_became_active":"2024-02-12T17:18:16.704045+0000","last_became_peered":"2024-02-12T17:18:16.704045+0000","last_unstale":"2024-02-12T17:19:35.314300+0000","last_undegraded":"2024-02-12T17:19:35.314300+0000","last_fullsized":"2024-02-12T17:19:35.314300+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:19:02.732999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:02.732999+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"766'29","reported_seq":622,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830403+0000","last_change":"2024-02-12T17:19:03.760844+0000","last_active":"2024-02-12T17:19:35.830403+0000","last_peered":"2024-02-12T17:19:35.830403+0000","last_clean":"2024-02-12T17:19:35.830403+0000","last_became_active":"2024-02-12T17:17:26.187638+0000","last_became_peered":"2024-02-12T17:17:26.187638+0000","last_unstale":"2024-02-12T17:19:35.830403+0000","last_undegraded":"2024-02-12T17:19:35.830403+0000","last_fullsized":"2024-02-12T17:19:35.830403+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'29","last_scrub_stamp":"2024-02-12T17:19:03.760782+0000","last_deep_scrub":"766'29","last_deep_scrub_stamp":"2024-02-12T17:19:03.760782+0000","last_clean_scrub_stamp":"2024-02-12T17:19:03.760782+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"526'39","reported_seq":128,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579481+0000","last_change":"2024-02-12T17:19:33.787184+0000","last_active":"2024-02-12T17:19:35.579481+0000","last_peered":"2024-02-12T17:19:35.579481+0000","last_clean":"2024-02-12T17:19:35.579481+0000","last_became_active":"2024-02-12T17:17:54.582173+0000","last_became_peered":"2024-02-12T17:17:54.582173+0000","last_unstale":"2024-02-12T17:19:35.579481+0000","last_undegraded":"2024-02-12T17:19:35.579481+0000","last_fullsized":"2024-02-12T17:19:35.579481+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'39","last_scrub_stamp":"2024-02-12T17:19:33.787066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:33.787066+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"766'32","reported_seq":325,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830024+0000","last_change":"2024-02-12T17:19:11.833278+0000","last_active":"2024-02-12T17:19:35.830024+0000","last_peered":"2024-02-12T17:19:35.830024+0000","last_clean":"2024-02-12T17:19:35.830024+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:19:35.830024+0000","last_undegraded":"2024-02-12T17:19:35.830024+0000","last_fullsized":"2024-02-12T17:19:35.830024+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'32","last_scrub_stamp":"2024-02-12T17:19:11.833198+0000","last_deep_scrub":"766'32","last_deep_scrub_stamp":"2024-02-12T17:19:11.833198+0000","last_clean_scrub_stamp":"2024-02-12T17:19:11.833198+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"748'35","reported_seq":159,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314959+0000","last_change":"2024-02-12T17:19:25.386701+0000","last_active":"2024-02-12T17:19:35.314959+0000","last_peered":"2024-02-12T17:19:35.314959+0000","last_clean":"2024-02-12T17:19:35.314959+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:19:35.314959+0000","last_undegraded":"2024-02-12T17:19:35.314959+0000","last_fullsized":"2024-02-12T17:19:35.314959+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"748'35","last_scrub_stamp":"2024-02-12T17:19:25.386629+0000","last_deep_scrub":"748'35","last_deep_scrub_stamp":"2024-02-12T17:18:16.464548+0000","last_clean_scrub_stamp":"2024-02-12T17:19:25.386629+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":542,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829687+0000","last_change":"2024-02-12T17:19:03.327974+0000","last_active":"2024-02-12T17:19:35.829687+0000","last_peered":"2024-02-12T17:19:35.829687+0000","last_clean":"2024-02-12T17:19:35.829687+0000","last_became_active":"2024-02-12T17:17:58.837339+0000","last_became_peered":"2024-02-12T17:17:58.837339+0000","last_unstale":"2024-02-12T17:19:35.829687+0000","last_undegraded":"2024-02-12T17:19:35.829687+0000","last_fullsized":"2024-02-12T17:19:35.829687+0000","mapping_epoch":741,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":5,"last_scrub":"430'18","last_scrub_stamp":"2024-02-12T17:19:03.327877+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:19:03.327877+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":473,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830213+0000","last_change":"2024-02-12T17:18:17.845608+0000","last_active":"2024-02-12T17:19:35.830213+0000","last_peered":"2024-02-12T17:19:35.830213+0000","last_clean":"2024-02-12T17:19:35.830213+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:19:35.830213+0000","last_undegraded":"2024-02-12T17:19:35.830213+0000","last_fullsized":"2024-02-12T17:19:35.830213+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:17.845516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:17.845516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":496,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830098+0000","last_change":"2024-02-12T17:18:59.879333+0000","last_active":"2024-02-12T17:19:35.830098+0000","last_peered":"2024-02-12T17:19:35.830098+0000","last_clean":"2024-02-12T17:19:35.830098+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:19:35.830098+0000","last_undegraded":"2024-02-12T17:19:35.830098+0000","last_fullsized":"2024-02-12T17:19:35.830098+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:18:59.879239+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:18:59.879239+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":334,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830550+0000","last_change":"2024-02-12T17:19:00.042896+0000","last_active":"2024-02-12T17:19:35.830550+0000","last_peered":"2024-02-12T17:19:35.830550+0000","last_clean":"2024-02-12T17:19:35.830550+0000","last_became_active":"2024-02-12T17:17:26.177455+0000","last_became_peered":"2024-02-12T17:17:26.177455+0000","last_unstale":"2024-02-12T17:19:35.830550+0000","last_undegraded":"2024-02-12T17:19:35.830550+0000","last_fullsized":"2024-02-12T17:19:35.830550+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:00.042822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:19:00.042822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":782,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830155+0000","last_change":"2024-02-12T17:18:17.275982+0000","last_active":"2024-02-12T17:19:35.830155+0000","last_peered":"2024-02-12T17:19:35.830155+0000","last_clean":"2024-02-12T17:19:35.830155+0000","last_became_active":"2024-02-12T17:17:26.202355+0000","last_became_peered":"2024-02-12T17:17:26.202355+0000","last_unstale":"2024-02-12T17:19:35.830155+0000","last_undegraded":"2024-02-12T17:19:35.830155+0000","last_fullsized":"2024-02-12T17:19:35.830155+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:17.275878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:18:17.275878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":518,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829813+0000","last_change":"2024-02-12T17:18:46.855768+0000","last_active":"2024-02-12T17:19:35.829813+0000","last_peered":"2024-02-12T17:19:35.829813+0000","last_clean":"2024-02-12T17:19:35.829813+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:19:35.829813+0000","last_undegraded":"2024-02-12T17:19:35.829813+0000","last_fullsized":"2024-02-12T17:19:35.829813+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'14","last_scrub_stamp":"2024-02-12T17:18:46.855664+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:18:46.855664+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":172,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664462+0000","last_change":"2024-02-12T17:19:00.831493+0000","last_active":"2024-02-12T17:19:35.664462+0000","last_peered":"2024-02-12T17:19:35.664462+0000","last_clean":"2024-02-12T17:19:35.664462+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:19:35.664462+0000","last_undegraded":"2024-02-12T17:19:35.664462+0000","last_fullsized":"2024-02-12T17:19:35.664462+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:00.831422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:19:00.831422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":455,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829906+0000","last_change":"2024-02-12T17:18:56.915716+0000","last_active":"2024-02-12T17:19:35.829906+0000","last_peered":"2024-02-12T17:19:35.829906+0000","last_clean":"2024-02-12T17:19:35.829906+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:19:35.829906+0000","last_undegraded":"2024-02-12T17:19:35.829906+0000","last_fullsized":"2024-02-12T17:19:35.829906+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:56.915610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:18:56.915610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":217,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664979+0000","last_change":"2024-02-12T17:19:28.860204+0000","last_active":"2024-02-12T17:19:35.664979+0000","last_peered":"2024-02-12T17:19:35.664979+0000","last_clean":"2024-02-12T17:19:35.664979+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:19:35.664979+0000","last_undegraded":"2024-02-12T17:19:35.664979+0000","last_fullsized":"2024-02-12T17:19:35.664979+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:28.860138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:19:28.860138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"526'27","reported_seq":72,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314825+0000","last_change":"2024-02-12T17:19:26.831904+0000","last_active":"2024-02-12T17:19:35.314825+0000","last_peered":"2024-02-12T17:19:35.314825+0000","last_clean":"2024-02-12T17:19:35.314825+0000","last_became_active":"2024-02-12T17:17:26.208368+0000","last_became_peered":"2024-02-12T17:17:26.208368+0000","last_unstale":"2024-02-12T17:19:35.314825+0000","last_undegraded":"2024-02-12T17:19:35.314825+0000","last_fullsized":"2024-02-12T17:19:35.314825+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:19:26.831816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:26.831816+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":860,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829991+0000","last_change":"2024-02-12T17:19:27.594876+0000","last_active":"2024-02-12T17:19:35.829991+0000","last_peered":"2024-02-12T17:19:35.829991+0000","last_clean":"2024-02-12T17:19:35.829991+0000","last_became_active":"2024-02-12T17:17:16.819318+0000","last_became_peered":"2024-02-12T17:17:16.819318+0000","last_unstale":"2024-02-12T17:19:35.829991+0000","last_undegraded":"2024-02-12T17:19:35.829991+0000","last_fullsized":"2024-02-12T17:19:35.829991+0000","mapping_epoch":723,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":5,"last_scrub":"419'14","last_scrub_stamp":"2024-02-12T17:19:27.594775+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:19:27.594775+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":142,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:41.469235+0000","last_change":"2024-02-12T17:19:41.469235+0000","last_active":"2024-02-12T17:19:41.469235+0000","last_peered":"2024-02-12T17:19:41.469235+0000","last_clean":"2024-02-12T17:19:41.469235+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:19:41.469235+0000","last_undegraded":"2024-02-12T17:19:41.469235+0000","last_fullsized":"2024-02-12T17:19:41.469235+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:41.469115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:19:41.469115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":288,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:40.996578+0000","last_change":"2024-02-12T17:19:40.996578+0000","last_active":"2024-02-12T17:19:40.996578+0000","last_peered":"2024-02-12T17:19:40.996578+0000","last_clean":"2024-02-12T17:19:40.996578+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:19:40.996578+0000","last_undegraded":"2024-02-12T17:19:40.996578+0000","last_fullsized":"2024-02-12T17:19:40.996578+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:40.996497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:19:40.996497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":244,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664419+0000","last_change":"2024-02-12T17:18:36.718952+0000","last_active":"2024-02-12T17:19:35.664419+0000","last_peered":"2024-02-12T17:19:35.664419+0000","last_clean":"2024-02-12T17:19:35.664419+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:19:35.664419+0000","last_undegraded":"2024-02-12T17:19:35.664419+0000","last_fullsized":"2024-02-12T17:19:35.664419+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:18:36.718882+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:18:36.718882+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":918,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529856+0000","last_change":"2024-02-12T17:19:17.798278+0000","last_active":"2024-02-12T17:19:37.529856+0000","last_peered":"2024-02-12T17:19:37.529856+0000","last_clean":"2024-02-12T17:19:37.529856+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:19:37.529856+0000","last_undegraded":"2024-02-12T17:19:37.529856+0000","last_fullsized":"2024-02-12T17:19:37.529856+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:19:17.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:19:17.798196+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":146,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314742+0000","last_change":"2024-02-12T17:18:53.545289+0000","last_active":"2024-02-12T17:19:35.314742+0000","last_peered":"2024-02-12T17:19:35.314742+0000","last_clean":"2024-02-12T17:19:35.314742+0000","last_became_active":"2024-02-12T17:17:26.195255+0000","last_became_peered":"2024-02-12T17:17:26.195255+0000","last_unstale":"2024-02-12T17:19:35.314742+0000","last_undegraded":"2024-02-12T17:19:35.314742+0000","last_fullsized":"2024-02-12T17:19:35.314742+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:53.545220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:53.545220+0000","last_clean_scrub_stamp":"2024-02-12T17:18:53.545220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"526'21","reported_seq":869,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829949+0000","last_change":"2024-02-12T17:19:06.975338+0000","last_active":"2024-02-12T17:19:35.829949+0000","last_peered":"2024-02-12T17:19:35.829949+0000","last_clean":"2024-02-12T17:19:35.829949+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:19:35.829949+0000","last_undegraded":"2024-02-12T17:19:35.829949+0000","last_fullsized":"2024-02-12T17:19:35.829949+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:19:06.975258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:06.975258+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"720'29","reported_seq":957,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.827622+0000","last_change":"2024-02-12T17:19:20.953303+0000","last_active":"2024-02-12T17:19:37.827622+0000","last_peered":"2024-02-12T17:19:37.827622+0000","last_clean":"2024-02-12T17:19:37.827622+0000","last_became_active":"2024-02-12T17:17:08.923702+0000","last_became_peered":"2024-02-12T17:17:08.923702+0000","last_unstale":"2024-02-12T17:19:37.827622+0000","last_undegraded":"2024-02-12T17:19:37.827622+0000","last_fullsized":"2024-02-12T17:19:37.827622+0000","mapping_epoch":718,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"720'29","last_scrub_stamp":"2024-02-12T17:19:20.953221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:19:20.953221+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":773,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829885+0000","last_change":"2024-02-12T17:19:21.936639+0000","last_active":"2024-02-12T17:19:35.829885+0000","last_peered":"2024-02-12T17:19:35.829885+0000","last_clean":"2024-02-12T17:19:35.829885+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:19:35.829885+0000","last_undegraded":"2024-02-12T17:19:35.829885+0000","last_fullsized":"2024-02-12T17:19:35.829885+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:21.936575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:19:21.936575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":509,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830278+0000","last_change":"2024-02-12T17:18:30.819866+0000","last_active":"2024-02-12T17:19:35.830278+0000","last_peered":"2024-02-12T17:19:35.830278+0000","last_clean":"2024-02-12T17:19:35.830278+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:19:35.830278+0000","last_undegraded":"2024-02-12T17:19:35.830278+0000","last_fullsized":"2024-02-12T17:19:35.830278+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:18:30.819789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:18:30.819789+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":804,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830536+0000","last_change":"2024-02-12T17:18:37.317396+0000","last_active":"2024-02-12T17:19:35.830536+0000","last_peered":"2024-02-12T17:19:35.830536+0000","last_clean":"2024-02-12T17:19:35.830536+0000","last_became_active":"2024-02-12T17:17:26.177631+0000","last_became_peered":"2024-02-12T17:17:26.177631+0000","last_unstale":"2024-02-12T17:19:35.830536+0000","last_undegraded":"2024-02-12T17:19:35.830536+0000","last_fullsized":"2024-02-12T17:19:35.830536+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:37.317307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:18:37.317307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":326,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830342+0000","last_change":"2024-02-12T17:18:54.056532+0000","last_active":"2024-02-12T17:19:35.830342+0000","last_peered":"2024-02-12T17:19:35.830342+0000","last_clean":"2024-02-12T17:19:35.830342+0000","last_became_active":"2024-02-12T17:17:26.203398+0000","last_became_peered":"2024-02-12T17:17:26.203398+0000","last_unstale":"2024-02-12T17:19:35.830342+0000","last_undegraded":"2024-02-12T17:19:35.830342+0000","last_fullsized":"2024-02-12T17:19:35.830342+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:54.056451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:17:14.825440+0000","last_clean_scrub_stamp":"2024-02-12T17:18:54.056451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":498,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830577+0000","last_change":"2024-02-12T17:18:26.732656+0000","last_active":"2024-02-12T17:19:35.830577+0000","last_peered":"2024-02-12T17:19:35.830577+0000","last_clean":"2024-02-12T17:19:35.830577+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:19:35.830577+0000","last_undegraded":"2024-02-12T17:19:35.830577+0000","last_fullsized":"2024-02-12T17:19:35.830577+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:18:26.732589+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:18:26.732589+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":318,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579192+0000","last_change":"2024-02-12T17:18:49.829558+0000","last_active":"2024-02-12T17:19:35.579192+0000","last_peered":"2024-02-12T17:19:35.579192+0000","last_clean":"2024-02-12T17:19:35.579192+0000","last_became_active":"2024-02-12T17:17:54.578042+0000","last_became_peered":"2024-02-12T17:17:54.578042+0000","last_unstale":"2024-02-12T17:19:35.579192+0000","last_undegraded":"2024-02-12T17:19:35.579192+0000","last_fullsized":"2024-02-12T17:19:35.579192+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:49.829475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:18:49.829475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":436,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.578999+0000","last_change":"2024-02-12T17:18:40.707055+0000","last_active":"2024-02-12T17:19:35.578999+0000","last_peered":"2024-02-12T17:19:35.578999+0000","last_clean":"2024-02-12T17:19:35.578999+0000","last_became_active":"2024-02-12T17:17:54.640595+0000","last_became_peered":"2024-02-12T17:17:54.640595+0000","last_unstale":"2024-02-12T17:19:35.578999+0000","last_undegraded":"2024-02-12T17:19:35.578999+0000","last_fullsized":"2024-02-12T17:19:35.578999+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:40.706985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:40.706985+0000","last_clean_scrub_stamp":"2024-02-12T17:18:40.706985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":171,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.315045+0000","last_change":"2024-02-12T17:19:30.348783+0000","last_active":"2024-02-12T17:19:35.315045+0000","last_peered":"2024-02-12T17:19:35.315045+0000","last_clean":"2024-02-12T17:19:35.315045+0000","last_became_active":"2024-02-12T17:17:05.874272+0000","last_became_peered":"2024-02-12T17:17:05.874272+0000","last_unstale":"2024-02-12T17:19:35.315045+0000","last_undegraded":"2024-02-12T17:19:35.315045+0000","last_fullsized":"2024-02-12T17:19:35.315045+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:19:30.348700+0000","last_deep_scrub":"526'25","last_deep_scrub_stamp":"2024-02-12T17:19:30.348700+0000","last_clean_scrub_stamp":"2024-02-12T17:19:30.348700+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":497,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830104+0000","last_change":"2024-02-12T17:18:40.958756+0000","last_active":"2024-02-12T17:19:35.830104+0000","last_peered":"2024-02-12T17:19:35.830104+0000","last_clean":"2024-02-12T17:19:35.830104+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:19:35.830104+0000","last_undegraded":"2024-02-12T17:19:35.830104+0000","last_fullsized":"2024-02-12T17:19:35.830104+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:40.958634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:40.958634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":529,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314938+0000","last_change":"2024-02-12T17:18:48.702512+0000","last_active":"2024-02-12T17:19:35.314938+0000","last_peered":"2024-02-12T17:19:35.314938+0000","last_clean":"2024-02-12T17:19:35.314938+0000","last_became_active":"2024-02-12T17:17:29.199721+0000","last_became_peered":"2024-02-12T17:17:29.199721+0000","last_unstale":"2024-02-12T17:19:35.314938+0000","last_undegraded":"2024-02-12T17:19:35.314938+0000","last_fullsized":"2024-02-12T17:19:35.314938+0000","mapping_epoch":730,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:18:48.702420+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:18:48.702420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"526'24","reported_seq":299,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314209+0000","last_change":"2024-02-12T17:19:12.785046+0000","last_active":"2024-02-12T17:19:35.314209+0000","last_peered":"2024-02-12T17:19:35.314209+0000","last_clean":"2024-02-12T17:19:35.314209+0000","last_became_active":"2024-02-12T17:18:16.700504+0000","last_became_peered":"2024-02-12T17:18:16.700504+0000","last_unstale":"2024-02-12T17:19:35.314209+0000","last_undegraded":"2024-02-12T17:19:35.314209+0000","last_fullsized":"2024-02-12T17:19:35.314209+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:19:12.784951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:12.784951+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":445,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.578978+0000","last_change":"2024-02-12T17:19:24.758158+0000","last_active":"2024-02-12T17:19:35.578978+0000","last_peered":"2024-02-12T17:19:35.578978+0000","last_clean":"2024-02-12T17:19:35.578978+0000","last_became_active":"2024-02-12T17:17:54.642052+0000","last_became_peered":"2024-02-12T17:17:54.642052+0000","last_unstale":"2024-02-12T17:19:35.578978+0000","last_undegraded":"2024-02-12T17:19:35.578978+0000","last_fullsized":"2024-02-12T17:19:35.578978+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:24.758048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:20.828117+0000","last_clean_scrub_stamp":"2024-02-12T17:19:24.758048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":784,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314149+0000","last_change":"2024-02-12T17:18:46.489852+0000","last_active":"2024-02-12T17:19:35.314149+0000","last_peered":"2024-02-12T17:19:35.314149+0000","last_clean":"2024-02-12T17:19:35.314149+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:19:35.314149+0000","last_undegraded":"2024-02-12T17:19:35.314149+0000","last_fullsized":"2024-02-12T17:19:35.314149+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:46.489732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:46.489732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664677+0000","last_change":"2024-02-12T17:18:57.804716+0000","last_active":"2024-02-12T17:19:35.664677+0000","last_peered":"2024-02-12T17:19:35.664677+0000","last_clean":"2024-02-12T17:19:35.664677+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:19:35.664677+0000","last_undegraded":"2024-02-12T17:19:35.664677+0000","last_fullsized":"2024-02-12T17:19:35.664677+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:57.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:57.804647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":273,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664531+0000","last_change":"2024-02-12T17:19:15.752836+0000","last_active":"2024-02-12T17:19:35.664531+0000","last_peered":"2024-02-12T17:19:35.664531+0000","last_clean":"2024-02-12T17:19:35.664531+0000","last_became_active":"2024-02-12T17:17:26.196542+0000","last_became_peered":"2024-02-12T17:17:26.196542+0000","last_unstale":"2024-02-12T17:19:35.664531+0000","last_undegraded":"2024-02-12T17:19:35.664531+0000","last_fullsized":"2024-02-12T17:19:35.664531+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:19:15.752760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:15.752760+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"706'24","reported_seq":657,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.730224+0000","last_change":"2024-02-12T17:19:01.523075+0000","last_active":"2024-02-12T17:19:37.730224+0000","last_peered":"2024-02-12T17:19:37.730224+0000","last_clean":"2024-02-12T17:19:37.730224+0000","last_became_active":"2024-02-12T17:16:46.730723+0000","last_became_peered":"2024-02-12T17:16:46.730723+0000","last_unstale":"2024-02-12T17:19:37.730224+0000","last_undegraded":"2024-02-12T17:19:37.730224+0000","last_fullsized":"2024-02-12T17:19:37.730224+0000","mapping_epoch":705,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"706'24","last_scrub_stamp":"2024-02-12T17:19:01.522997+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:19:01.522997+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":509,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830396+0000","last_change":"2024-02-12T17:19:23.695735+0000","last_active":"2024-02-12T17:19:35.830396+0000","last_peered":"2024-02-12T17:19:35.830396+0000","last_clean":"2024-02-12T17:19:35.830396+0000","last_became_active":"2024-02-12T17:17:26.187553+0000","last_became_peered":"2024-02-12T17:17:26.187553+0000","last_unstale":"2024-02-12T17:19:35.830396+0000","last_undegraded":"2024-02-12T17:19:35.830396+0000","last_fullsized":"2024-02-12T17:19:35.830396+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:23.695651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:19:23.695651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":815,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829998+0000","last_change":"2024-02-12T17:18:44.693029+0000","last_active":"2024-02-12T17:19:35.829998+0000","last_peered":"2024-02-12T17:19:35.829998+0000","last_clean":"2024-02-12T17:19:35.829998+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:19:35.829998+0000","last_undegraded":"2024-02-12T17:19:35.829998+0000","last_fullsized":"2024-02-12T17:19:35.829998+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:44.692951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:44.692951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":338,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.794539+0000","last_change":"2024-02-12T17:18:19.733557+0000","last_active":"2024-02-12T17:19:37.794539+0000","last_peered":"2024-02-12T17:19:37.794539+0000","last_clean":"2024-02-12T17:19:37.794539+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:19:37.794539+0000","last_undegraded":"2024-02-12T17:19:37.794539+0000","last_fullsized":"2024-02-12T17:19:37.794539+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'70","last_scrub_stamp":"2024-02-12T17:18:19.733432+0000","last_deep_scrub":"632'70","last_deep_scrub_stamp":"2024-02-12T17:18:19.733432+0000","last_clean_scrub_stamp":"2024-02-12T17:18:19.733432+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":766,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314766+0000","last_change":"2024-02-12T17:18:33.413491+0000","last_active":"2024-02-12T17:19:35.314766+0000","last_peered":"2024-02-12T17:19:35.314766+0000","last_clean":"2024-02-12T17:19:35.314766+0000","last_became_active":"2024-02-12T17:18:28.897328+0000","last_became_peered":"2024-02-12T17:18:28.897328+0000","last_unstale":"2024-02-12T17:19:35.314766+0000","last_undegraded":"2024-02-12T17:19:35.314766+0000","last_fullsized":"2024-02-12T17:19:35.314766+0000","mapping_epoch":756,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:18:33.413420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:18:33.413420+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":784,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830712+0000","last_change":"2024-02-12T17:18:56.309880+0000","last_active":"2024-02-12T17:19:35.830712+0000","last_peered":"2024-02-12T17:19:35.830712+0000","last_clean":"2024-02-12T17:19:35.830712+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:19:35.830712+0000","last_undegraded":"2024-02-12T17:19:35.830712+0000","last_fullsized":"2024-02-12T17:19:35.830712+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:56.309777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:17:31.297073+0000","last_clean_scrub_stamp":"2024-02-12T17:18:56.309777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":638,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:43.664619+0000","last_change":"2024-02-12T17:18:28.684274+0000","last_active":"2024-02-12T17:19:43.664619+0000","last_peered":"2024-02-12T17:19:43.664619+0000","last_clean":"2024-02-12T17:19:43.664619+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:19:43.664619+0000","last_undegraded":"2024-02-12T17:19:43.664619+0000","last_fullsized":"2024-02-12T17:19:43.664619+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:18:28.684209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:28.684209+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":570,"num_read_kb":565,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":345,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579162+0000","last_change":"2024-02-12T17:18:41.710877+0000","last_active":"2024-02-12T17:19:35.579162+0000","last_peered":"2024-02-12T17:19:35.579162+0000","last_clean":"2024-02-12T17:19:35.579162+0000","last_became_active":"2024-02-12T17:17:54.640095+0000","last_became_peered":"2024-02-12T17:17:54.640095+0000","last_unstale":"2024-02-12T17:19:35.579162+0000","last_undegraded":"2024-02-12T17:19:35.579162+0000","last_fullsized":"2024-02-12T17:19:35.579162+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:41.710761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:41.710761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":245,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664837+0000","last_change":"2024-02-12T17:18:26.685828+0000","last_active":"2024-02-12T17:19:35.664837+0000","last_peered":"2024-02-12T17:19:35.664837+0000","last_clean":"2024-02-12T17:19:35.664837+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:19:35.664837+0000","last_undegraded":"2024-02-12T17:19:35.664837+0000","last_fullsized":"2024-02-12T17:19:35.664837+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'26","last_scrub_stamp":"2024-02-12T17:18:26.685757+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:18:26.685757+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":337,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579583+0000","last_change":"2024-02-12T17:19:03.789449+0000","last_active":"2024-02-12T17:19:35.579583+0000","last_peered":"2024-02-12T17:19:35.579583+0000","last_clean":"2024-02-12T17:19:35.579583+0000","last_became_active":"2024-02-12T17:17:54.579441+0000","last_became_peered":"2024-02-12T17:17:54.579441+0000","last_unstale":"2024-02-12T17:19:35.579583+0000","last_undegraded":"2024-02-12T17:19:35.579583+0000","last_fullsized":"2024-02-12T17:19:35.579583+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:03.789385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:19:03.789385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":856,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830563+0000","last_change":"2024-02-12T17:18:59.106926+0000","last_active":"2024-02-12T17:19:35.830563+0000","last_peered":"2024-02-12T17:19:35.830563+0000","last_clean":"2024-02-12T17:19:35.830563+0000","last_became_active":"2024-02-12T17:17:26.190887+0000","last_became_peered":"2024-02-12T17:17:26.190887+0000","last_unstale":"2024-02-12T17:19:35.830563+0000","last_undegraded":"2024-02-12T17:19:35.830563+0000","last_fullsized":"2024-02-12T17:19:35.830563+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:59.106845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:17:41.406170+0000","last_clean_scrub_stamp":"2024-02-12T17:18:59.106845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":478,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830015+0000","last_change":"2024-02-12T17:19:15.745904+0000","last_active":"2024-02-12T17:19:35.830015+0000","last_peered":"2024-02-12T17:19:35.830015+0000","last_clean":"2024-02-12T17:19:35.830015+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:19:35.830015+0000","last_undegraded":"2024-02-12T17:19:35.830015+0000","last_fullsized":"2024-02-12T17:19:35.830015+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:15.745831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:19:15.745831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":755,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829470+0000","last_change":"2024-02-12T17:18:59.945263+0000","last_active":"2024-02-12T17:19:35.829470+0000","last_peered":"2024-02-12T17:19:35.829470+0000","last_clean":"2024-02-12T17:19:35.829470+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:19:35.829470+0000","last_undegraded":"2024-02-12T17:19:35.829470+0000","last_fullsized":"2024-02-12T17:19:35.829470+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:59.945190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:59.945190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":824,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829525+0000","last_change":"2024-02-12T17:18:36.152282+0000","last_active":"2024-02-12T17:19:35.829525+0000","last_peered":"2024-02-12T17:19:35.829525+0000","last_clean":"2024-02-12T17:19:35.829525+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:19:35.829525+0000","last_undegraded":"2024-02-12T17:19:35.829525+0000","last_fullsized":"2024-02-12T17:19:35.829525+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:36.152143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:36.152143+0000","last_clean_scrub_stamp":"2024-02-12T17:18:36.152143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"526'45","reported_seq":420,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830861+0000","last_change":"2024-02-12T17:19:12.868579+0000","last_active":"2024-02-12T17:19:35.830861+0000","last_peered":"2024-02-12T17:19:35.830861+0000","last_clean":"2024-02-12T17:19:35.830861+0000","last_became_active":"2024-02-12T17:17:26.177646+0000","last_became_peered":"2024-02-12T17:17:26.177646+0000","last_unstale":"2024-02-12T17:19:35.830861+0000","last_undegraded":"2024-02-12T17:19:35.830861+0000","last_fullsized":"2024-02-12T17:19:35.830861+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:19:12.868467+0000","last_deep_scrub":"526'45","last_deep_scrub_stamp":"2024-02-12T17:19:12.868467+0000","last_clean_scrub_stamp":"2024-02-12T17:19:12.868467+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":1078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":791,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830448+0000","last_change":"2024-02-12T17:19:21.291190+0000","last_active":"2024-02-12T17:19:35.830448+0000","last_peered":"2024-02-12T17:19:35.830448+0000","last_clean":"2024-02-12T17:19:35.830448+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:19:35.830448+0000","last_undegraded":"2024-02-12T17:19:35.830448+0000","last_fullsized":"2024-02-12T17:19:35.830448+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:19:21.291091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:19:21.291091+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":172,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664748+0000","last_change":"2024-02-12T17:18:32.748113+0000","last_active":"2024-02-12T17:19:35.664748+0000","last_peered":"2024-02-12T17:19:35.664748+0000","last_clean":"2024-02-12T17:19:35.664748+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:19:35.664748+0000","last_undegraded":"2024-02-12T17:19:35.664748+0000","last_fullsized":"2024-02-12T17:19:35.664748+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:32.748035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:32.748035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":146,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314411+0000","last_change":"2024-02-12T17:18:57.550083+0000","last_active":"2024-02-12T17:19:35.314411+0000","last_peered":"2024-02-12T17:19:35.314411+0000","last_clean":"2024-02-12T17:19:35.314411+0000","last_became_active":"2024-02-12T17:17:26.194140+0000","last_became_peered":"2024-02-12T17:17:26.194140+0000","last_unstale":"2024-02-12T17:19:35.314411+0000","last_undegraded":"2024-02-12T17:19:35.314411+0000","last_fullsized":"2024-02-12T17:19:35.314411+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:57.549951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:18:57.549951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":824,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829436+0000","last_change":"2024-02-12T17:18:30.205400+0000","last_active":"2024-02-12T17:19:35.829436+0000","last_peered":"2024-02-12T17:19:35.829436+0000","last_clean":"2024-02-12T17:19:35.829436+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:19:35.829436+0000","last_undegraded":"2024-02-12T17:19:35.829436+0000","last_fullsized":"2024-02-12T17:19:35.829436+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:18:30.205314+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:18:30.205314+0000","last_clean_scrub_stamp":"2024-02-12T17:18:30.205314+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":156,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:39.427871+0000","last_change":"2024-02-12T17:19:39.427871+0000","last_active":"2024-02-12T17:19:39.427871+0000","last_peered":"2024-02-12T17:19:39.427871+0000","last_clean":"2024-02-12T17:19:39.427871+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:19:39.427871+0000","last_undegraded":"2024-02-12T17:19:39.427871+0000","last_fullsized":"2024-02-12T17:19:39.427871+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'36","last_scrub_stamp":"2024-02-12T17:19:39.427737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:39.427737+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":734,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664923+0000","last_change":"2024-02-12T17:19:09.750191+0000","last_active":"2024-02-12T17:19:35.664923+0000","last_peered":"2024-02-12T17:19:35.664923+0000","last_clean":"2024-02-12T17:19:35.664923+0000","last_became_active":"2024-02-12T17:17:26.196430+0000","last_became_peered":"2024-02-12T17:17:26.196430+0000","last_unstale":"2024-02-12T17:19:35.664923+0000","last_undegraded":"2024-02-12T17:19:35.664923+0000","last_fullsized":"2024-02-12T17:19:35.664923+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:09.750123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:17:35.731956+0000","last_clean_scrub_stamp":"2024-02-12T17:19:09.750123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":203,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664307+0000","last_change":"2024-02-12T17:18:34.687748+0000","last_active":"2024-02-12T17:19:35.664307+0000","last_peered":"2024-02-12T17:19:35.664307+0000","last_clean":"2024-02-12T17:19:35.664307+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:19:35.664307+0000","last_undegraded":"2024-02-12T17:19:35.664307+0000","last_fullsized":"2024-02-12T17:19:35.664307+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:34.687610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:18:34.687610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":847,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830687+0000","last_change":"2024-02-12T17:18:45.086113+0000","last_active":"2024-02-12T17:19:35.830687+0000","last_peered":"2024-02-12T17:19:35.830687+0000","last_clean":"2024-02-12T17:19:35.830687+0000","last_became_active":"2024-02-12T17:17:26.184374+0000","last_became_peered":"2024-02-12T17:17:26.184374+0000","last_unstale":"2024-02-12T17:19:35.830687+0000","last_undegraded":"2024-02-12T17:19:35.830687+0000","last_fullsized":"2024-02-12T17:19:35.830687+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:45.086043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:45.086043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"766'47","reported_seq":163,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314701+0000","last_change":"2024-02-12T17:18:47.489994+0000","last_active":"2024-02-12T17:19:35.314701+0000","last_peered":"2024-02-12T17:19:35.314701+0000","last_clean":"2024-02-12T17:19:35.314701+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:19:35.314701+0000","last_undegraded":"2024-02-12T17:19:35.314701+0000","last_fullsized":"2024-02-12T17:19:35.314701+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'47","last_scrub_stamp":"2024-02-12T17:18:47.489913+0000","last_deep_scrub":"766'47","last_deep_scrub_stamp":"2024-02-12T17:18:47.489913+0000","last_clean_scrub_stamp":"2024-02-12T17:18:47.489913+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":281,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830493+0000","last_change":"2024-02-12T17:18:51.015924+0000","last_active":"2024-02-12T17:19:35.830493+0000","last_peered":"2024-02-12T17:19:35.830493+0000","last_clean":"2024-02-12T17:19:35.830493+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:19:35.830493+0000","last_undegraded":"2024-02-12T17:19:35.830493+0000","last_fullsized":"2024-02-12T17:19:35.830493+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:51.015818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:18:51.015818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":472,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314503+0000","last_change":"2024-02-12T17:19:21.821426+0000","last_active":"2024-02-12T17:19:35.314503+0000","last_peered":"2024-02-12T17:19:35.314503+0000","last_clean":"2024-02-12T17:19:35.314503+0000","last_became_active":"2024-02-12T17:18:16.699753+0000","last_became_peered":"2024-02-12T17:18:16.699753+0000","last_unstale":"2024-02-12T17:19:35.314503+0000","last_undegraded":"2024-02-12T17:19:35.314503+0000","last_fullsized":"2024-02-12T17:19:35.314503+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:21.821331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:19:21.821331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":228,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664839+0000","last_change":"2024-02-12T17:19:08.787891+0000","last_active":"2024-02-12T17:19:35.664839+0000","last_peered":"2024-02-12T17:19:35.664839+0000","last_clean":"2024-02-12T17:19:35.664839+0000","last_became_active":"2024-02-12T17:17:26.195505+0000","last_became_peered":"2024-02-12T17:17:26.195505+0000","last_unstale":"2024-02-12T17:19:35.664839+0000","last_undegraded":"2024-02-12T17:19:35.664839+0000","last_fullsized":"2024-02-12T17:19:35.664839+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:08.787809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:19:08.787809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":551,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.795899+0000","last_change":"2024-02-12T17:19:37.795899+0000","last_active":"2024-02-12T17:19:37.795899+0000","last_peered":"2024-02-12T17:19:37.795899+0000","last_clean":"2024-02-12T17:19:37.795899+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:19:37.795899+0000","last_undegraded":"2024-02-12T17:19:37.795899+0000","last_fullsized":"2024-02-12T17:19:37.795899+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:19:37.795814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:37.795814+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":60,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314472+0000","last_change":"2024-02-12T17:18:43.769492+0000","last_active":"2024-02-12T17:19:35.314472+0000","last_peered":"2024-02-12T17:19:35.314472+0000","last_clean":"2024-02-12T17:19:35.314472+0000","last_became_active":"2024-02-12T17:17:26.183165+0000","last_became_peered":"2024-02-12T17:17:26.183165+0000","last_unstale":"2024-02-12T17:19:35.314472+0000","last_undegraded":"2024-02-12T17:19:35.314472+0000","last_fullsized":"2024-02-12T17:19:35.314472+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:43.769411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:18:43.769411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"766'2","reported_seq":818,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830285+0000","last_change":"2024-02-12T17:18:57.453772+0000","last_active":"2024-02-12T17:19:35.830285+0000","last_peered":"2024-02-12T17:19:35.830285+0000","last_clean":"2024-02-12T17:19:35.830285+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:19:35.830285+0000","last_undegraded":"2024-02-12T17:19:35.830285+0000","last_fullsized":"2024-02-12T17:19:35.830285+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'2","last_scrub_stamp":"2024-02-12T17:18:57.453704+0000","last_deep_scrub":"766'2","last_deep_scrub_stamp":"2024-02-12T17:18:57.453704+0000","last_clean_scrub_stamp":"2024-02-12T17:18:57.453704+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":299,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829857+0000","last_change":"2024-02-12T17:18:45.026850+0000","last_active":"2024-02-12T17:19:35.829857+0000","last_peered":"2024-02-12T17:19:35.829857+0000","last_clean":"2024-02-12T17:19:35.829857+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:19:35.829857+0000","last_undegraded":"2024-02-12T17:19:35.829857+0000","last_fullsized":"2024-02-12T17:19:35.829857+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:45.026763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:45.026763+0000","last_clean_scrub_stamp":"2024-02-12T17:18:45.026763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"766'49","reported_seq":961,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314241+0000","last_change":"2024-02-12T17:18:54.684382+0000","last_active":"2024-02-12T17:19:35.314241+0000","last_peered":"2024-02-12T17:19:35.314241+0000","last_clean":"2024-02-12T17:19:35.314241+0000","last_became_active":"2024-02-12T17:18:16.697461+0000","last_became_peered":"2024-02-12T17:18:16.697461+0000","last_unstale":"2024-02-12T17:19:35.314241+0000","last_undegraded":"2024-02-12T17:19:35.314241+0000","last_fullsized":"2024-02-12T17:19:35.314241+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'49","last_scrub_stamp":"2024-02-12T17:18:54.684303+0000","last_deep_scrub":"766'49","last_deep_scrub_stamp":"2024-02-12T17:18:54.684303+0000","last_clean_scrub_stamp":"2024-02-12T17:18:54.684303+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"743'36","reported_seq":816,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529390+0000","last_change":"2024-02-12T17:19:28.632137+0000","last_active":"2024-02-12T17:19:37.529390+0000","last_peered":"2024-02-12T17:19:37.529390+0000","last_clean":"2024-02-12T17:19:37.529390+0000","last_became_active":"2024-02-12T17:17:58.841255+0000","last_became_peered":"2024-02-12T17:17:58.841255+0000","last_unstale":"2024-02-12T17:19:37.529390+0000","last_undegraded":"2024-02-12T17:19:37.529390+0000","last_fullsized":"2024-02-12T17:19:37.529390+0000","mapping_epoch":741,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'36","last_scrub_stamp":"2024-02-12T17:19:28.632065+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:19:28.632065+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":489,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830158+0000","last_change":"2024-02-12T17:18:38.898475+0000","last_active":"2024-02-12T17:19:35.830158+0000","last_peered":"2024-02-12T17:19:35.830158+0000","last_clean":"2024-02-12T17:19:35.830158+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:19:35.830158+0000","last_undegraded":"2024-02-12T17:19:35.830158+0000","last_fullsized":"2024-02-12T17:19:35.830158+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:18:38.898389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:38.898389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"743'34","reported_seq":1261,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.827832+0000","last_change":"2024-02-12T17:19:25.962456+0000","last_active":"2024-02-12T17:19:37.827832+0000","last_peered":"2024-02-12T17:19:37.827832+0000","last_clean":"2024-02-12T17:19:37.827832+0000","last_became_active":"2024-02-12T17:17:58.837743+0000","last_became_peered":"2024-02-12T17:17:58.837743+0000","last_unstale":"2024-02-12T17:19:37.827832+0000","last_undegraded":"2024-02-12T17:19:37.827832+0000","last_fullsized":"2024-02-12T17:19:37.827832+0000","mapping_epoch":741,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'34","last_scrub_stamp":"2024-02-12T17:19:25.962353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:19:25.962353+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":492,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829236+0000","last_change":"2024-02-12T17:18:46.103660+0000","last_active":"2024-02-12T17:19:35.829236+0000","last_peered":"2024-02-12T17:19:35.829236+0000","last_clean":"2024-02-12T17:19:35.829236+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:19:35.829236+0000","last_undegraded":"2024-02-12T17:19:35.829236+0000","last_fullsized":"2024-02-12T17:19:35.829236+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:46.103544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:18:46.103544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"526'27","reported_seq":348,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:39.935460+0000","last_change":"2024-02-12T17:19:39.935460+0000","last_active":"2024-02-12T17:19:39.935460+0000","last_peered":"2024-02-12T17:19:39.935460+0000","last_clean":"2024-02-12T17:19:39.935460+0000","last_became_active":"2024-02-12T17:18:16.699402+0000","last_became_peered":"2024-02-12T17:18:16.699402+0000","last_unstale":"2024-02-12T17:19:39.935460+0000","last_undegraded":"2024-02-12T17:19:39.935460+0000","last_fullsized":"2024-02-12T17:19:39.935460+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:19:39.935383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:39.935383+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":822,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830256+0000","last_change":"2024-02-12T17:18:23.035607+0000","last_active":"2024-02-12T17:19:35.830256+0000","last_peered":"2024-02-12T17:19:35.830256+0000","last_clean":"2024-02-12T17:19:35.830256+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:19:35.830256+0000","last_undegraded":"2024-02-12T17:19:35.830256+0000","last_fullsized":"2024-02-12T17:19:35.830256+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:23.035499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:23.035499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"750'31","reported_seq":1267,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.951410+0000","last_change":"2024-02-12T17:19:27.843776+0000","last_active":"2024-02-12T17:19:37.951410+0000","last_peered":"2024-02-12T17:19:37.951410+0000","last_clean":"2024-02-12T17:19:37.951410+0000","last_became_active":"2024-02-12T17:18:16.703985+0000","last_became_peered":"2024-02-12T17:18:16.703985+0000","last_unstale":"2024-02-12T17:19:37.951410+0000","last_undegraded":"2024-02-12T17:19:37.951410+0000","last_fullsized":"2024-02-12T17:19:37.951410+0000","mapping_epoch":748,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"750'31","last_scrub_stamp":"2024-02-12T17:19:27.843652+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:19:27.843652+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":195,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664237+0000","last_change":"2024-02-12T17:19:33.805935+0000","last_active":"2024-02-12T17:19:35.664237+0000","last_peered":"2024-02-12T17:19:35.664237+0000","last_clean":"2024-02-12T17:19:35.664237+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:19:35.664237+0000","last_undegraded":"2024-02-12T17:19:35.664237+0000","last_fullsized":"2024-02-12T17:19:35.664237+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:33.805876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:19:33.805876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":263,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664895+0000","last_change":"2024-02-12T17:19:07.764613+0000","last_active":"2024-02-12T17:19:35.664895+0000","last_peered":"2024-02-12T17:19:35.664895+0000","last_clean":"2024-02-12T17:19:35.664895+0000","last_became_active":"2024-02-12T17:17:26.196636+0000","last_became_peered":"2024-02-12T17:17:26.196636+0000","last_unstale":"2024-02-12T17:19:35.664895+0000","last_undegraded":"2024-02-12T17:19:35.664895+0000","last_fullsized":"2024-02-12T17:19:35.664895+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:19:07.764488+0000","last_deep_scrub":"526'46","last_deep_scrub_stamp":"2024-02-12T17:19:07.764488+0000","last_clean_scrub_stamp":"2024-02-12T17:19:07.764488+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":155,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314822+0000","last_change":"2024-02-12T17:18:49.810353+0000","last_active":"2024-02-12T17:19:35.314822+0000","last_peered":"2024-02-12T17:19:35.314822+0000","last_clean":"2024-02-12T17:19:35.314822+0000","last_became_active":"2024-02-12T17:18:16.701099+0000","last_became_peered":"2024-02-12T17:18:16.701099+0000","last_unstale":"2024-02-12T17:19:35.314822+0000","last_undegraded":"2024-02-12T17:19:35.314822+0000","last_fullsized":"2024-02-12T17:19:35.314822+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:49.810252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:18:49.810252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"740'15","reported_seq":502,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.845421+0000","last_change":"2024-02-12T17:18:53.850343+0000","last_active":"2024-02-12T17:19:37.845421+0000","last_peered":"2024-02-12T17:19:37.845421+0000","last_clean":"2024-02-12T17:19:37.845421+0000","last_became_active":"2024-02-12T17:17:54.579125+0000","last_became_peered":"2024-02-12T17:17:54.579125+0000","last_unstale":"2024-02-12T17:19:37.845421+0000","last_undegraded":"2024-02-12T17:19:37.845421+0000","last_fullsized":"2024-02-12T17:19:37.845421+0000","mapping_epoch":739,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'15","last_scrub_stamp":"2024-02-12T17:18:53.850266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:18:53.850266+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664241+0000","last_change":"2024-02-12T17:18:52.749854+0000","last_active":"2024-02-12T17:19:35.664241+0000","last_peered":"2024-02-12T17:19:35.664241+0000","last_clean":"2024-02-12T17:19:35.664241+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:19:35.664241+0000","last_undegraded":"2024-02-12T17:19:35.664241+0000","last_fullsized":"2024-02-12T17:19:35.664241+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:52.749783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:52.749783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"733'35","reported_seq":595,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314566+0000","last_change":"2024-02-12T17:19:00.745588+0000","last_active":"2024-02-12T17:19:35.314566+0000","last_peered":"2024-02-12T17:19:35.314566+0000","last_clean":"2024-02-12T17:19:35.314566+0000","last_became_active":"2024-02-12T17:18:16.699885+0000","last_became_peered":"2024-02-12T17:18:16.699885+0000","last_unstale":"2024-02-12T17:19:35.314566+0000","last_undegraded":"2024-02-12T17:19:35.314566+0000","last_fullsized":"2024-02-12T17:19:35.314566+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'35","last_scrub_stamp":"2024-02-12T17:19:00.745489+0000","last_deep_scrub":"733'35","last_deep_scrub_stamp":"2024-02-12T17:17:33.454203+0000","last_clean_scrub_stamp":"2024-02-12T17:19:00.745489+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":509,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579541+0000","last_change":"2024-02-12T17:18:24.022023+0000","last_active":"2024-02-12T17:19:35.579541+0000","last_peered":"2024-02-12T17:19:35.579541+0000","last_clean":"2024-02-12T17:19:35.579541+0000","last_became_active":"2024-02-12T17:17:54.582218+0000","last_became_peered":"2024-02-12T17:17:54.582218+0000","last_unstale":"2024-02-12T17:19:35.579541+0000","last_undegraded":"2024-02-12T17:19:35.579541+0000","last_fullsized":"2024-02-12T17:19:35.579541+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:24.021907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:24.021907+0000","last_clean_scrub_stamp":"2024-02-12T17:18:24.021907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":210,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664627+0000","last_change":"2024-02-12T17:18:50.773491+0000","last_active":"2024-02-12T17:19:35.664627+0000","last_peered":"2024-02-12T17:19:35.664627+0000","last_clean":"2024-02-12T17:19:35.664627+0000","last_became_active":"2024-02-12T17:17:26.202178+0000","last_became_peered":"2024-02-12T17:17:26.202178+0000","last_unstale":"2024-02-12T17:19:35.664627+0000","last_undegraded":"2024-02-12T17:19:35.664627+0000","last_fullsized":"2024-02-12T17:19:35.664627+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:50.773366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:18:50.773366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":811,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829959+0000","last_change":"2024-02-12T17:18:43.110721+0000","last_active":"2024-02-12T17:19:35.829959+0000","last_peered":"2024-02-12T17:19:35.829959+0000","last_clean":"2024-02-12T17:19:35.829959+0000","last_became_active":"2024-02-12T17:17:26.181989+0000","last_became_peered":"2024-02-12T17:17:26.181989+0000","last_unstale":"2024-02-12T17:19:35.829959+0000","last_undegraded":"2024-02-12T17:19:35.829959+0000","last_fullsized":"2024-02-12T17:19:35.829959+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:43.110640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:43.110640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"526'45","reported_seq":255,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314640+0000","last_change":"2024-02-12T17:18:59.770756+0000","last_active":"2024-02-12T17:19:35.314640+0000","last_peered":"2024-02-12T17:19:35.314640+0000","last_clean":"2024-02-12T17:19:35.314640+0000","last_became_active":"2024-02-12T17:18:16.701108+0000","last_became_peered":"2024-02-12T17:18:16.701108+0000","last_unstale":"2024-02-12T17:19:35.314640+0000","last_undegraded":"2024-02-12T17:19:35.314640+0000","last_fullsized":"2024-02-12T17:19:35.314640+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:18:59.770675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:18:59.770675+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":804,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829338+0000","last_change":"2024-02-12T17:18:49.163759+0000","last_active":"2024-02-12T17:19:35.829338+0000","last_peered":"2024-02-12T17:19:35.829338+0000","last_clean":"2024-02-12T17:19:35.829338+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:19:35.829338+0000","last_undegraded":"2024-02-12T17:19:35.829338+0000","last_fullsized":"2024-02-12T17:19:35.829338+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:49.163665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:18:49.163665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":337,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579352+0000","last_change":"2024-02-12T17:18:47.812083+0000","last_active":"2024-02-12T17:19:35.579352+0000","last_peered":"2024-02-12T17:19:35.579352+0000","last_clean":"2024-02-12T17:19:35.579352+0000","last_became_active":"2024-02-12T17:17:54.640651+0000","last_became_peered":"2024-02-12T17:17:54.640651+0000","last_unstale":"2024-02-12T17:19:35.579352+0000","last_undegraded":"2024-02-12T17:19:35.579352+0000","last_fullsized":"2024-02-12T17:19:35.579352+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:47.811986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:47.811986+0000","last_clean_scrub_stamp":"2024-02-12T17:18:47.811986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":473,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579316+0000","last_change":"2024-02-12T17:19:23.733560+0000","last_active":"2024-02-12T17:19:35.579316+0000","last_peered":"2024-02-12T17:19:35.579316+0000","last_clean":"2024-02-12T17:19:35.579316+0000","last_became_active":"2024-02-12T17:17:54.641860+0000","last_became_peered":"2024-02-12T17:17:54.641860+0000","last_unstale":"2024-02-12T17:19:35.579316+0000","last_undegraded":"2024-02-12T17:19:35.579316+0000","last_fullsized":"2024-02-12T17:19:35.579316+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:23.733497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:21.817041+0000","last_clean_scrub_stamp":"2024-02-12T17:19:23.733497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"526'51","reported_seq":335,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830628+0000","last_change":"2024-02-12T17:18:34.080719+0000","last_active":"2024-02-12T17:19:35.830628+0000","last_peered":"2024-02-12T17:19:35.830628+0000","last_clean":"2024-02-12T17:19:35.830628+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:19:35.830628+0000","last_undegraded":"2024-02-12T17:19:35.830628+0000","last_fullsized":"2024-02-12T17:19:35.830628+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'51","last_scrub_stamp":"2024-02-12T17:18:34.080625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:18:34.080625+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":137,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314348+0000","last_change":"2024-02-12T17:18:42.569595+0000","last_active":"2024-02-12T17:19:35.314348+0000","last_peered":"2024-02-12T17:19:35.314348+0000","last_clean":"2024-02-12T17:19:35.314348+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:19:35.314348+0000","last_undegraded":"2024-02-12T17:19:35.314348+0000","last_fullsized":"2024-02-12T17:19:35.314348+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:42.569505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:18:42.569505+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":610,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579055+0000","last_change":"2024-02-12T17:18:52.828275+0000","last_active":"2024-02-12T17:19:35.579055+0000","last_peered":"2024-02-12T17:19:35.579055+0000","last_clean":"2024-02-12T17:19:35.579055+0000","last_became_active":"2024-02-12T17:17:54.582224+0000","last_became_peered":"2024-02-12T17:17:54.582224+0000","last_unstale":"2024-02-12T17:19:35.579055+0000","last_undegraded":"2024-02-12T17:19:35.579055+0000","last_fullsized":"2024-02-12T17:19:35.579055+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:52.828190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:18:52.828190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":506,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664878+0000","last_change":"2024-02-12T17:18:46.780145+0000","last_active":"2024-02-12T17:19:35.664878+0000","last_peered":"2024-02-12T17:19:35.664878+0000","last_clean":"2024-02-12T17:19:35.664878+0000","last_became_active":"2024-02-12T17:17:29.201863+0000","last_became_peered":"2024-02-12T17:17:29.201863+0000","last_unstale":"2024-02-12T17:19:35.664878+0000","last_undegraded":"2024-02-12T17:19:35.664878+0000","last_fullsized":"2024-02-12T17:19:35.664878+0000","mapping_epoch":730,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:18:46.780062+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T17:17:30.604582+0000","last_clean_scrub_stamp":"2024-02-12T17:18:46.780062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":185,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314625+0000","last_change":"2024-02-12T17:19:00.493338+0000","last_active":"2024-02-12T17:19:35.314625+0000","last_peered":"2024-02-12T17:19:35.314625+0000","last_clean":"2024-02-12T17:19:35.314625+0000","last_became_active":"2024-02-12T17:17:26.199624+0000","last_became_peered":"2024-02-12T17:17:26.199624+0000","last_unstale":"2024-02-12T17:19:35.314625+0000","last_undegraded":"2024-02-12T17:19:35.314625+0000","last_fullsized":"2024-02-12T17:19:35.314625+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:19:00.493228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:00.493228+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":785,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829757+0000","last_change":"2024-02-12T17:19:02.041382+0000","last_active":"2024-02-12T17:19:35.829757+0000","last_peered":"2024-02-12T17:19:35.829757+0000","last_clean":"2024-02-12T17:19:35.829757+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:19:35.829757+0000","last_undegraded":"2024-02-12T17:19:35.829757+0000","last_fullsized":"2024-02-12T17:19:35.829757+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:02.041289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:19:02.041289+0000","last_clean_scrub_stamp":"2024-02-12T17:19:02.041289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":346,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.579478+0000","last_change":"2024-02-12T17:18:37.683393+0000","last_active":"2024-02-12T17:19:35.579478+0000","last_peered":"2024-02-12T17:19:35.579478+0000","last_clean":"2024-02-12T17:19:35.579478+0000","last_became_active":"2024-02-12T17:17:54.581523+0000","last_became_peered":"2024-02-12T17:17:54.581523+0000","last_unstale":"2024-02-12T17:19:35.579478+0000","last_undegraded":"2024-02-12T17:19:35.579478+0000","last_fullsized":"2024-02-12T17:19:35.579478+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:37.683281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:37.683281+0000","last_clean_scrub_stamp":"2024-02-12T17:18:37.683281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"526'46","reported_seq":903,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829206+0000","last_change":"2024-02-12T17:18:47.722776+0000","last_active":"2024-02-12T17:19:35.829206+0000","last_peered":"2024-02-12T17:19:35.829206+0000","last_clean":"2024-02-12T17:19:35.829206+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:19:35.829206+0000","last_undegraded":"2024-02-12T17:19:35.829206+0000","last_fullsized":"2024-02-12T17:19:35.829206+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:18:47.722690+0000","last_deep_scrub":"526'46","last_deep_scrub_stamp":"2024-02-12T17:17:43.205604+0000","last_clean_scrub_stamp":"2024-02-12T17:18:47.722690+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"760'39","reported_seq":1355,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.730400+0000","last_change":"2024-02-12T17:18:40.135936+0000","last_active":"2024-02-12T17:19:37.730400+0000","last_peered":"2024-02-12T17:19:37.730400+0000","last_clean":"2024-02-12T17:19:37.730400+0000","last_became_active":"2024-02-12T17:18:29.902894+0000","last_became_peered":"2024-02-12T17:18:29.902894+0000","last_unstale":"2024-02-12T17:19:37.730400+0000","last_undegraded":"2024-02-12T17:19:37.730400+0000","last_fullsized":"2024-02-12T17:19:37.730400+0000","mapping_epoch":757,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"760'39","last_scrub_stamp":"2024-02-12T17:18:40.135838+0000","last_deep_scrub":"726'36","last_deep_scrub_stamp":"2024-02-12T17:17:22.297773+0000","last_clean_scrub_stamp":"2024-02-12T17:18:40.135838+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":570,"num_read_kb":565,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1509,"num_read_kb":1266,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":17690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1064,"ondisk_log_size":1064,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234349,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1495880,"kb_used_data":3712,"kb_used_omap":0,"kb_used_meta":1492160,"kb_avail":92249272,"statfs":{"total":95995035648,"available":94463254528,"internally_reserved":0,"allocated":3801088,"data_stored":2209767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1527971840},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":1.3380000000000001,"15min":1.698},"min":{"1min":0.39400000000000002,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":0.69499999999999995,"5min":15.856999999999999,"15min":15.856999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.499,"5min":1.3859999999999999,"15min":1.569},"min":{"1min":0.38600000000000001,"5min":0.23599999999999999,"15min":0.22},"max":{"1min":0.65700000000000003,"5min":12.565,"15min":12.565},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":1.7749999999999999,"15min":2.4510000000000001},"min":{"1min":0.311,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.88600000000000001,"5min":11.632999999999999,"15min":11.632999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.849,"15min":2.2610000000000001},"min":{"1min":0.42899999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.86799999999999999,"5min":10.91,"15min":10.91},"last":0.76800000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.512,"15min":2.879},"min":{"1min":0.44,"5min":0.24399999999999999,"15min":0.22800000000000001},"max":{"1min":0.67900000000000005,"5min":23.611000000000001,"15min":218.35300000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.742,"15min":3.0939999999999999},"min":{"1min":0.46300000000000002,"5min":0.372,"15min":0.247},"max":{"1min":0.67100000000000004,"5min":15.672000000000001,"15min":218.42099999999999},"last":0.72099999999999997}]},{"osd":3,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.907,"15min":1.9710000000000001},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.255},"max":{"1min":0.746,"5min":11.741,"15min":15.811999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":2.1680000000000001,"15min":2.4049999999999998},"min":{"1min":0.38400000000000001,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":0.69799999999999995,"5min":12.375999999999999,"15min":13.699},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57999999999999996,"15min":1.2549999999999999},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23300000000000001},"max":{"1min":0.79700000000000004,"5min":3.4969999999999999,"15min":20.672999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.74399999999999999,"15min":1.5960000000000001},"min":{"1min":0.317,"5min":0.23300000000000001,"15min":0.19600000000000001},"max":{"1min":0.65900000000000003,"5min":6.6769999999999996,"15min":38.390000000000001},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.69599999999999995,"15min":1.6850000000000001},"min":{"1min":0.35199999999999998,"5min":0.23999999999999999,"15min":0.22500000000000001},"max":{"1min":0.82699999999999996,"5min":4.79,"15min":56.560000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.99399999999999999,"15min":1.6160000000000001},"min":{"1min":0.376,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.73999999999999999,"5min":9.7170000000000005,"15min":53.963999999999999},"last":0.80100000000000005}]},{"osd":6,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.71899999999999997,"15min":1.2030000000000001},"min":{"1min":0.374,"5min":0.25700000000000001,"15min":0.223},"max":{"1min":0.73199999999999998,"5min":5.2539999999999996,"15min":21.620000000000001},"last":0.628},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":1.3819999999999999},"min":{"1min":0.44400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.85099999999999998,"5min":3.7519999999999998,"15min":18.071000000000002},"last":0.73699999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528690,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896060,"kb_used_data":3700,"kb_used_omap":0,"kb_used_meta":892352,"kb_avail":92849092,"statfs":{"total":95995035648,"available":95077470208,"internally_reserved":0,"allocated":3788800,"data_stored":2209855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913768448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.4590000000000001,"15min":1.7949999999999999},"min":{"1min":0.34499999999999997,"5min":0.22,"15min":0.20399999999999999},"max":{"1min":0.70599999999999996,"5min":11.115,"15min":33.576999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.1120000000000001,"15min":1.7549999999999999},"min":{"1min":0.34799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.73699999999999999,"5min":6.1100000000000003,"15min":33.569000000000003},"last":0.54700000000000004}]},{"osd":1,"last update":"Mon Feb 12 17:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.661,"15min":1.948},"min":{"1min":0.41699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.749,"5min":20.611000000000001,"15min":20.611000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":1.6080000000000001,"15min":1.998},"min":{"1min":0.441,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":20.622,"15min":20.622},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":1.78,"15min":2.3740000000000001},"min":{"1min":0.312,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.71099999999999997,"5min":12.311,"15min":33.594999999999999},"last":0.501},{"interface":"front","average":{"1min":0.57399999999999995,"5min":2.0139999999999998,"15min":2.1040000000000001},"min":{"1min":0.36499999999999999,"5min":0.27600000000000002,"15min":0.24099999999999999},"max":{"1min":0.72499999999999998,"5min":14.045999999999999,"15min":33.526000000000003},"last":1.218}]},{"osd":3,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":2.1589999999999998,"15min":2.5619999999999998},"min":{"1min":0.38100000000000001,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":0.68600000000000005,"5min":25.867999999999999,"15min":58.420999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":2.254,"15min":2.7879999999999998},"min":{"1min":0.38400000000000001,"5min":0.33200000000000002,"15min":0.254},"max":{"1min":0.77400000000000002,"5min":14.69,"15min":58.709000000000003},"last":1.1399999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.78800000000000003,"15min":1.1859999999999999},"min":{"1min":0.28100000000000003,"5min":0.26100000000000001,"15min":0.23499999999999999},"max":{"1min":0.66600000000000004,"5min":11.012,"15min":33.706000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.76800000000000002,"15min":1.389},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.222},"max":{"1min":0.69699999999999995,"5min":10.948,"15min":34.895000000000003},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.91200000000000003,"15min":1.4350000000000001},"min":{"1min":0.30599999999999999,"5min":0.27600000000000002,"15min":0.23799999999999999},"max":{"1min":0.70899999999999996,"5min":10.77,"15min":34.869},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.93700000000000006,"15min":1.3839999999999999},"min":{"1min":0.33100000000000002,"5min":0.30099999999999999,"15min":0.22900000000000001},"max":{"1min":0.72399999999999998,"5min":11.1,"15min":33.591000000000001},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Feb 12 17:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.76300000000000001,"15min":1.468},"min":{"1min":0.32500000000000001,"5min":0.26500000000000001,"15min":0.255},"max":{"1min":0.76600000000000001,"5min":6.1310000000000002,"15min":35.122999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.90900000000000003,"15min":1.2889999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.26500000000000001},"max":{"1min":0.77200000000000002,"5min":11.081,"15min":34.045000000000002},"last":0.61499999999999999}]}]},{"osd":1,"up_from":630,"seq":2705829396554,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338264,"kb_used_data":3728,"kb_used_omap":3,"kb_used_meta":334524,"kb_avail":93406888,"statfs":{"total":95995035648,"available":95648653312,"internally_reserved":0,"allocated":3817472,"data_stored":2210412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":342552637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":2.0630000000000002,"15min":3.0110000000000001},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.63900000000000001,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":3.423,"15min":3.4209999999999998},"min":{"1min":0.254,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65700000000000003,"5min":27.268999999999998,"15min":27.268999999999998},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":1.992,"15min":3.052},"min":{"1min":0.26600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54800000000000004,"5min":12.856,"15min":12.856},"last":0.438},{"interface":"front","average":{"1min":0.373,"5min":2.5990000000000002,"15min":3.1970000000000001},"min":{"1min":0.25700000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":24.236999999999998,"15min":24.236999999999998},"last":0.29499999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":3.218,"15min":3.5009999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":31.292999999999999,"15min":31.292999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":3.2970000000000002,"15min":3.98},"min":{"1min":0.33100000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":23.905999999999999,"15min":23.905999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":1.5760000000000001,"15min":2.2280000000000002},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84699999999999998,"5min":12.901999999999999,"15min":12.901999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.9350000000000001,"15min":2.4990000000000001},"min":{"1min":0.38900000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.76700000000000002,"5min":12.805999999999999,"15min":12.805999999999999},"last":0.628}]},{"osd":5,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":2.0289999999999999,"15min":2.8199999999999998},"min":{"1min":0.437,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81200000000000006,"5min":12.769,"15min":12.769},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.133,"15min":2.1190000000000002},"min":{"1min":0.45300000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90900000000000003,"5min":23.300999999999998,"15min":23.300999999999998},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":2.2679999999999998,"15min":2.6419999999999999},"min":{"1min":0.42099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.879,"5min":23.347000000000001,"15min":23.347000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.629,"15min":2.4609999999999999},"min":{"1min":0.44500000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80900000000000005,"5min":12.929,"15min":12.929},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":1.9710000000000001,"15min":2.633},"min":{"1min":0.40699999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.88200000000000001,"5min":12.829000000000001,"15min":12.829000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":2.298,"15min":2.3599999999999999},"min":{"1min":0.48199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.94099999999999995,"5min":23.376000000000001,"15min":23.376000000000001},"last":0.59899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738767,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908104,"kb_used_data":3712,"kb_used_omap":0,"kb_used_meta":904384,"kb_avail":92837048,"statfs":{"total":95995035648,"available":95065137152,"internally_reserved":0,"allocated":3801088,"data_stored":2209820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":926089216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:18:50 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":2.448,"15min":3.1019999999999999},"min":{"1min":0.24199999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.626,"5min":14.048999999999999,"15min":14.048999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":1.7569999999999999,"15min":2.4529999999999998},"min":{"1min":0.27800000000000002,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53400000000000003,"5min":13.787000000000001,"15min":13.787000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:18:43 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":2.3370000000000002,"15min":2.319},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.19400000000000001},"max":{"1min":0.53500000000000003,"5min":18.204000000000001,"15min":19.312000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":2.3719999999999999,"15min":2.7650000000000001},"min":{"1min":0.245,"5min":0.217,"15min":0.20799999999999999},"max":{"1min":0.64500000000000002,"5min":22.959,"15min":26.039999999999999},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:18:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":3.7040000000000002,"15min":3.262},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.186},"max":{"1min":0.59899999999999998,"5min":61.177,"15min":87.311000000000007},"last":0.307},{"interface":"front","average":{"1min":0.44,"5min":3.778,"15min":3.4390000000000001},"min":{"1min":0.25,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":0.64100000000000001,"5min":59.149999999999999,"15min":90.991},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Feb 12 17:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":1.377,"15min":1.387},"min":{"1min":0.41399999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.89300000000000002,"5min":13.861000000000001,"15min":15.141999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.4910000000000001,"15min":1.964},"min":{"1min":0.438,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":0.90100000000000002,"5min":13.503,"15min":26.082999999999998},"last":0.70399999999999996}]},{"osd":5,"last update":"Mon Feb 12 17:18:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6679999999999999,"15min":1.9850000000000001},"min":{"1min":0.38300000000000001,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":0.85999999999999999,"5min":13.981999999999999,"15min":17.754000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.6819999999999999,"15min":1.857},"min":{"1min":0.41599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.81999999999999995,"5min":13.837,"15min":19.751999999999999},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":1.702,"15min":2.3570000000000002},"min":{"1min":0.44700000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84399999999999997,"5min":14.686999999999999,"15min":48.780999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.698,"15min":2.1840000000000002},"min":{"1min":0.498,"5min":0.36399999999999999,"15min":0.31900000000000001},"max":{"1min":0.875,"5min":13.962999999999999,"15min":49.067999999999998},"last":0.65500000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:18:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.9079999999999999,"15min":2.0270000000000001},"min":{"1min":0.52200000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.88100000000000001,"5min":13.813000000000001,"15min":19.373000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.9350000000000001,"15min":2.2949999999999999},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82799999999999996,"5min":13.542,"15min":26.135000000000002},"last":0.66300000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411716,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1505988,"kb_used_data":3708,"kb_used_omap":0,"kb_used_meta":1502272,"kb_avail":92239164,"statfs":{"total":95995035648,"available":94452903936,"internally_reserved":0,"allocated":3796992,"data_stored":2209690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1538326528},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":1.5109999999999999,"15min":1.8240000000000001},"min":{"1min":0.249,"5min":0.218,"15min":0.17000000000000001},"max":{"1min":0.76200000000000001,"5min":11.663,"15min":21.445},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":2.452,"15min":2.2789999999999999},"min":{"1min":0.251,"5min":0.224,"15min":0.19900000000000001},"max":{"1min":0.58399999999999996,"5min":26.852,"15min":26.852},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:18:53 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":1.9550000000000001,"15min":3.3290000000000002},"min":{"1min":0.27100000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.63800000000000001,"5min":11.612,"15min":11.612},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":3.056,"15min":4.9480000000000004},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.83699999999999997,"5min":22.52,"15min":22.52},"last":0.29299999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":4.0199999999999996,"15min":3.0870000000000002},"min":{"1min":0.34899999999999998,"5min":0.27100000000000002,"15min":0.20200000000000001},"max":{"1min":0.81799999999999995,"5min":37.704999999999998,"15min":40.759999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":2.9670000000000001,"15min":2.5990000000000002},"min":{"1min":0.32000000000000001,"5min":0.221,"15min":0.22},"max":{"1min":0.69699999999999995,"5min":30.285,"15min":30.285},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:19:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":1.409,"15min":1.756},"min":{"1min":0.503,"5min":0.27600000000000002,"15min":0.245},"max":{"1min":0.91700000000000004,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.751},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.5089999999999999,"15min":1.766},"min":{"1min":0.495,"5min":0.25700000000000001,"15min":0.25600000000000001},"max":{"1min":0.95599999999999996,"5min":12.597,"15min":20.149000000000001},"last":0.76200000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.403,"15min":1.77},"min":{"1min":0.47299999999999998,"5min":0.33900000000000002,"15min":0.217},"max":{"1min":0.95899999999999996,"5min":11.581,"15min":21.542999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":1.2929999999999999,"15min":1.8300000000000001},"min":{"1min":0.44900000000000001,"5min":0.28699999999999998,"15min":0.255},"max":{"1min":1.0129999999999999,"5min":8.3070000000000004,"15min":20.937000000000001},"last":0.72699999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.6279999999999999,"15min":1.8220000000000001},"min":{"1min":0.45400000000000001,"5min":0.29199999999999998,"15min":0.27500000000000002},"max":{"1min":0.86799999999999999,"5min":10.714,"15min":20.966999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.774,"15min":1.8009999999999999},"min":{"1min":0.47599999999999998,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.97799999999999998,"5min":12.631,"15min":20.193000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.5549999999999999,"15min":2.012},"min":{"1min":0.42799999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.94599999999999995,"5min":12.67,"15min":20.221},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.29,"15min":1.9219999999999999},"min":{"1min":0.47999999999999998,"5min":0.33000000000000002,"15min":0.30199999999999999},"max":{"1min":0.95699999999999996,"5min":8.3230000000000004,"15min":20.995999999999999},"last":0.70299999999999996}]}]},{"osd":3,"up_from":26,"seq":111669150078,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896980,"kb_used_data":3724,"kb_used_omap":0,"kb_used_meta":893248,"kb_avail":92848172,"statfs":{"total":95995035648,"available":95076528128,"internally_reserved":0,"allocated":3813376,"data_stored":2210970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914685952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":2.8879999999999999,"15min":2.8010000000000002},"min":{"1min":0.24199999999999999,"5min":0.20999999999999999,"15min":0.17100000000000001},"max":{"1min":0.49199999999999999,"5min":22.16,"15min":22.16},"last":0.316},{"interface":"front","average":{"1min":0.40999999999999998,"5min":3.3690000000000002,"15min":3.1520000000000001},"min":{"1min":0.28499999999999998,"5min":0.23599999999999999,"15min":0.19500000000000001},"max":{"1min":0.58899999999999997,"5min":42.984999999999999,"15min":72.688000000000002},"last":0.38}]},{"osd":1,"last update":"Mon Feb 12 17:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":3.403,"15min":4.1210000000000004},"min":{"1min":0.25700000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69599999999999995,"5min":22.524000000000001,"15min":22.524000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":3.476,"15min":4.7949999999999999},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.626,"5min":20.710000000000001,"15min":20.710000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":3.4649999999999999,"15min":2.7970000000000002},"min":{"1min":0.245,"5min":0.186,"15min":0.186},"max":{"1min":0.65200000000000002,"5min":48.127000000000002,"15min":48.127000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":3.4159999999999999,"15min":3.274},"min":{"1min":0.29199999999999998,"5min":0.27100000000000002,"15min":0.22},"max":{"1min":0.51900000000000002,"5min":40.600999999999999,"15min":46.408000000000001},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:19:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":2.4790000000000001,"15min":2.5819999999999999},"min":{"1min":0.432,"5min":0.309,"15min":0.29599999999999999},"max":{"1min":0.80400000000000005,"5min":20.478000000000002,"15min":72.659000000000006},"last":0.5},{"interface":"front","average":{"1min":0.63900000000000001,"5min":2.0150000000000001,"15min":2.0169999999999999},"min":{"1min":0.42299999999999999,"5min":0.25800000000000001,"15min":0.23999999999999999},"max":{"1min":0.88200000000000001,"5min":22.440999999999999,"15min":40.930999999999997},"last":0.48099999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":2.7559999999999998,"15min":2.5899999999999999},"min":{"1min":0.42499999999999999,"5min":0.314,"15min":0.253},"max":{"1min":0.871,"5min":17.306999999999999,"15min":17.940999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":2.5409999999999999,"15min":2.7999999999999998},"min":{"1min":0.39700000000000002,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":0.77900000000000003,"5min":15.522,"15min":72.620000000000005},"last":0.53800000000000003}]},{"osd":6,"last update":"Mon Feb 12 17:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":2.145,"15min":2.125},"min":{"1min":0.45500000000000002,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":0.85599999999999998,"5min":22.489999999999998,"15min":40.889000000000003},"last":0.442},{"interface":"front","average":{"1min":0.67300000000000004,"5min":2.379,"15min":2.4300000000000002},"min":{"1min":0.53400000000000003,"5min":0.36499999999999999,"15min":0.32600000000000001},"max":{"1min":0.93100000000000005,"5min":15.113,"15min":18.265000000000001},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Feb 12 17:19:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":2.3730000000000002,"15min":2.8999999999999999},"min":{"1min":0.45000000000000001,"5min":0.378,"15min":0.315},"max":{"1min":0.89100000000000001,"5min":20.515999999999998,"15min":72.712999999999994},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":2.3809999999999998,"15min":2.7090000000000001},"min":{"1min":0.53500000000000003,"5min":0.39900000000000002,"15min":0.28299999999999997},"max":{"1min":0.89800000000000002,"5min":15.169,"15min":38.948},"last":0.60199999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953852,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907532,"kb_used_data":3716,"kb_used_omap":0,"kb_used_meta":903808,"kb_avail":92837620,"statfs":{"total":95995035648,"available":95065722880,"internally_reserved":0,"allocated":3805184,"data_stored":2210489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":925499392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":1.0920000000000001,"15min":1.5900000000000001},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":0.62,"5min":5.7380000000000004,"15min":25.829000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.456,"15min":1.742},"min":{"1min":0.40400000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.68700000000000006,"5min":11.659000000000001,"15min":22.286000000000001},"last":0.42299999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:18:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.544,"15min":1.7969999999999999},"min":{"1min":0.40400000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.78100000000000003,"5min":11.319000000000001,"15min":11.319000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.5660000000000001,"15min":2.5800000000000001},"min":{"1min":0.48799999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71799999999999997,"5min":11.483000000000001,"15min":11.483000000000001},"last":0.61199999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":1.4650000000000001,"15min":3.1030000000000002},"min":{"1min":0.40400000000000003,"5min":0.23400000000000001,"15min":0.23300000000000001},"max":{"1min":0.70399999999999996,"5min":11.395,"15min":191.05600000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.622,"5min":1.5589999999999999,"15min":3.0030000000000001},"min":{"1min":0.50800000000000001,"5min":0.29699999999999999,"15min":0.20599999999999999},"max":{"1min":0.76800000000000002,"5min":11.298,"15min":191.083},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Feb 12 17:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":2.0920000000000001,"15min":3.161},"min":{"1min":0.42499999999999999,"5min":0.26500000000000001,"15min":0.25600000000000001},"max":{"1min":0.83999999999999997,"5min":19.638999999999999,"15min":248.05500000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56000000000000005,"5min":1.75,"15min":2.9900000000000002},"min":{"1min":0.38700000000000001,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.69099999999999995,"5min":10.191000000000001,"15min":249.20599999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Feb 12 17:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.75900000000000001,"15min":1.26},"min":{"1min":0.28199999999999997,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":0.63200000000000001,"5min":5.085,"15min":26.510000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.82799999999999996,"15min":1.3740000000000001},"min":{"1min":0.30399999999999999,"5min":0.24099999999999999,"15min":0.219},"max":{"1min":0.61799999999999999,"5min":6.032,"15min":23.986999999999998},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Feb 12 17:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.78800000000000003,"15min":0.96899999999999997},"min":{"1min":0.25900000000000001,"5min":0.255,"15min":0.246},"max":{"1min":0.748,"5min":5.4169999999999998,"15min":11.361000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.86799999999999999,"15min":1.155},"min":{"1min":0.33000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64700000000000002,"5min":11.332000000000001,"15min":11.332000000000001},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Feb 12 17:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.78400000000000003,"15min":1.3140000000000001},"min":{"1min":0.27100000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73899999999999999,"5min":7.6609999999999996,"15min":20.696000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":1.0209999999999999,"15min":1.2549999999999999},"min":{"1min":0.317,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.075,"5min":6.8689999999999998,"15min":10.99},"last":0.73499999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757624,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1489024,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":1485312,"kb_avail":92256128,"statfs":{"total":95995035648,"available":94470275072,"internally_reserved":0,"allocated":3792896,"data_stored":2209739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1520959488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":1.3360000000000001,"15min":1.53},"min":{"1min":0.218,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.70699999999999996,"5min":6.8399999999999999,"15min":10.971},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.5,"5min":1.468,"15min":1.6160000000000001},"min":{"1min":0.23100000000000001,"5min":0.22900000000000001,"15min":0.20100000000000001},"max":{"1min":0.64300000000000002,"5min":12.061,"15min":15.368},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Feb 12 17:19:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":1.613,"15min":1.736},"min":{"1min":0.26900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.748,"5min":9.484,"15min":9.484},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.9239999999999999,"15min":2.6840000000000002},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71899999999999997,"5min":18.550000000000001,"15min":18.550000000000001},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.708,"15min":1.9810000000000001},"min":{"1min":0.312,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":0.72399999999999998,"5min":13.978,"15min":57.390999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":1.3700000000000001,"15min":1.7789999999999999},"min":{"1min":0.37,"5min":0.32100000000000001,"15min":0.214},"max":{"1min":0.67600000000000005,"5min":13.366,"15min":57.134999999999998},"last":0.64300000000000002}]},{"osd":3,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.655,"15min":2.6619999999999999},"min":{"1min":0.36199999999999999,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":0.745,"5min":11.871,"15min":57.170999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.9079999999999999,"15min":2.492},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.23799999999999999},"max":{"1min":0.76200000000000001,"5min":17.760999999999999,"15min":57.484999999999999},"last":0.56699999999999995}]},{"osd":4,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.79800000000000004,"15min":1.085},"min":{"1min":0.31,"5min":0.252,"15min":0.23599999999999999},"max":{"1min":0.69499999999999995,"5min":5.9939999999999998,"15min":12.656000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.80600000000000005,"15min":1.1479999999999999},"min":{"1min":0.33900000000000002,"5min":0.255,"15min":0.254},"max":{"1min":0.69599999999999995,"5min":5.1390000000000002,"15min":11.068},"last":0.496}]},{"osd":6,"last update":"Mon Feb 12 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.83299999999999996,"15min":1.1619999999999999},"min":{"1min":0.30299999999999999,"5min":0.28899999999999998,"15min":0.23899999999999999},"max":{"1min":0.67900000000000005,"5min":6.4109999999999996,"15min":14.675000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.91900000000000004,"15min":1.2969999999999999},"min":{"1min":0.39600000000000002,"5min":0.27900000000000003,"15min":0.245},"max":{"1min":0.77400000000000002,"5min":5.2720000000000002,"15min":12.864000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.82699999999999996,"15min":1.24},"min":{"1min":0.36099999999999999,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.64200000000000002,"5min":6.2960000000000003,"15min":14.337999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.98299999999999998,"15min":1.3680000000000001},"min":{"1min":0.39300000000000002,"5min":0.30199999999999999,"15min":0.27800000000000002},"max":{"1min":0.70299999999999996,"5min":6.6950000000000003,"15min":20.274000000000001},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:19:46.755 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T17:19:47.045 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:47.559 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:45.869463+0000 mgr.y (mgr.24481) 1132 : audit [DBG] from='client.36536 v1:172.21.15.33:0/3278353380' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:19:47.560 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:46.854463+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-12T17:19:47.560 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:45.869463+0000 mgr.y (mgr.24481) 1132 : audit [DBG] from='client.36536 v1:172.21.15.33:0/3278353380' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:19:47.560 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:47 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:46.854463+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-12T17:19:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:45.869463+0000 mgr.y (mgr.24481) 1132 : audit [DBG] from='client.36536 v1:172.21.15.33:0/3278353380' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:19:47.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:46.854463+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-12T17:19:48.691 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:47.199629+0000 mgr.y (mgr.24481) 1133 : cluster [DBG] pgmap v1598: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:48.691 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:47.650973+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:19:48.691 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:47.654738+0000 osd.3 (osd.3) 327 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:19:48.691 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:47.199629+0000 mgr.y (mgr.24481) 1133 : cluster [DBG] pgmap v1598: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:48.691 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:47.650973+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:19:48.692 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:48 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:47.654738+0000 osd.3 (osd.3) 327 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:19:48.739 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:47.199629+0000 mgr.y (mgr.24481) 1133 : cluster [DBG] pgmap v1598: 102 pgs: 102 active+clean; 5.7 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-12T17:19:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:47.650973+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.14 scrub starts 2024-02-12T17:19:48.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:47.654738+0000 osd.3 (osd.3) 327 : cluster [DBG] 4.14 scrub ok 2024-02-12T17:19:49.127 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-02-12T17:19:49.130 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-02-12T17:19:49.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:48.119760+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:49.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:48.119801+0000 osd.3 (osd.3) 329 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:49.513 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:48.797266+0000 osd.5 (osd.5) 379 : cluster [DBG] 6.d scrub starts 2024-02-12T17:19:49.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:48.866543+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-12T17:19:49.514 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:48.965343+0000 osd.5 (osd.5) 380 : cluster [DBG] 6.d scrub ok 2024-02-12T17:19:49.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:48.119760+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:49.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:48.119801+0000 osd.3 (osd.3) 329 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:49.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:48.797266+0000 osd.5 (osd.5) 379 : cluster [DBG] 6.d scrub starts 2024-02-12T17:19:49.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:48.866543+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-12T17:19:49.514 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:49 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:48.965343+0000 osd.5 (osd.5) 380 : cluster [DBG] 6.d scrub ok 2024-02-12T17:19:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:48.119760+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:48.119801+0000 osd.3 (osd.3) 329 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:48.797266+0000 osd.5 (osd.5) 379 : cluster [DBG] 6.d scrub starts 2024-02-12T17:19:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:48.866543+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-12T17:19:49.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:48.965343+0000 osd.5 (osd.5) 380 : cluster [DBG] 6.d scrub ok 2024-02-12T17:19:49.796 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":1599,"stamp":"2024-02-12T17:19:48.864371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":2311,"num_read_kb":2063,"num_write":1140,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":22367,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1357,"ondisk_log_size":1357,"up":306,"acting":306,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":306,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8442036,"kb_used_data":29812,"kb_used_omap":3,"kb_used_meta":8412156,"kb_avail":741519180,"statfs":{"total":767960285184,"available":759315640320,"internally_reserved":0,"allocated":30527488,"data_stored":17739375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":8614047805},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":215,"num_write":140,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.666694"},"pg_stats":[{"pgid":"4.19","version":"526'39","reported_seq":506,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529230+0000","last_change":"2024-02-12T17:18:57.867053+0000","last_active":"2024-02-12T17:19:37.529230+0000","last_peered":"2024-02-12T17:19:37.529230+0000","last_clean":"2024-02-12T17:19:37.529230+0000","last_became_active":"2024-02-12T17:14:08.757736+0000","last_became_peered":"2024-02-12T17:14:08.757736+0000","last_unstale":"2024-02-12T17:19:37.529230+0000","last_undegraded":"2024-02-12T17:19:37.529230+0000","last_fullsized":"2024-02-12T17:19:37.529230+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'39","last_scrub_stamp":"2024-02-12T17:18:57.866977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:18:57.866977+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"772'31","reported_seq":81,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.036086+0000","last_change":"2024-02-12T17:18:45.706297+0000","last_active":"2024-02-12T17:19:48.036086+0000","last_peered":"2024-02-12T17:19:48.036086+0000","last_clean":"2024-02-12T17:19:48.036086+0000","last_became_active":"2024-02-12T17:17:26.207842+0000","last_became_peered":"2024-02-12T17:17:26.207842+0000","last_unstale":"2024-02-12T17:19:48.036086+0000","last_undegraded":"2024-02-12T17:19:48.036086+0000","last_fullsized":"2024-02-12T17:19:48.036086+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:18:45.706216+0000","last_deep_scrub":"315'19","last_deep_scrub_stamp":"2024-02-12T17:01:45.154164+0000","last_clean_scrub_stamp":"2024-02-12T17:18:45.706216+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":43,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"772'11","reported_seq":522,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.116099+0000","last_change":"2024-02-12T17:19:04.732212+0000","last_active":"2024-02-12T17:19:48.116099+0000","last_peered":"2024-02-12T17:19:48.116099+0000","last_clean":"2024-02-12T17:19:48.116099+0000","last_became_active":"2024-02-12T17:18:16.703769+0000","last_became_peered":"2024-02-12T17:18:16.703769+0000","last_unstale":"2024-02-12T17:19:48.116099+0000","last_undegraded":"2024-02-12T17:19:48.116099+0000","last_fullsized":"2024-02-12T17:19:48.116099+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9","last_scrub_stamp":"2024-02-12T17:19:04.732111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:04.732111+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"772'27","reported_seq":80,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.136454+0000","last_change":"2024-02-12T17:19:02.733076+0000","last_active":"2024-02-12T17:19:48.136454+0000","last_peered":"2024-02-12T17:19:48.136454+0000","last_clean":"2024-02-12T17:19:48.136454+0000","last_became_active":"2024-02-12T17:18:16.704045+0000","last_became_peered":"2024-02-12T17:18:16.704045+0000","last_unstale":"2024-02-12T17:19:48.136454+0000","last_undegraded":"2024-02-12T17:19:48.136454+0000","last_fullsized":"2024-02-12T17:19:48.136454+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:19:02.732999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:02.732999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"766'29","reported_seq":623,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.530034+0000","last_change":"2024-02-12T17:19:03.760844+0000","last_active":"2024-02-12T17:19:37.530034+0000","last_peered":"2024-02-12T17:19:37.530034+0000","last_clean":"2024-02-12T17:19:37.530034+0000","last_became_active":"2024-02-12T17:17:26.187638+0000","last_became_peered":"2024-02-12T17:17:26.187638+0000","last_unstale":"2024-02-12T17:19:37.530034+0000","last_undegraded":"2024-02-12T17:19:37.530034+0000","last_fullsized":"2024-02-12T17:19:37.530034+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'29","last_scrub_stamp":"2024-02-12T17:19:03.760782+0000","last_deep_scrub":"766'29","last_deep_scrub_stamp":"2024-02-12T17:19:03.760782+0000","last_clean_scrub_stamp":"2024-02-12T17:19:03.760782+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"772'51","reported_seq":148,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.168580+0000","last_change":"2024-02-12T17:19:33.787184+0000","last_active":"2024-02-12T17:19:48.168580+0000","last_peered":"2024-02-12T17:19:48.168580+0000","last_clean":"2024-02-12T17:19:48.168580+0000","last_became_active":"2024-02-12T17:17:54.582173+0000","last_became_peered":"2024-02-12T17:17:54.582173+0000","last_unstale":"2024-02-12T17:19:48.168580+0000","last_undegraded":"2024-02-12T17:19:48.168580+0000","last_fullsized":"2024-02-12T17:19:48.168580+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'39","last_scrub_stamp":"2024-02-12T17:19:33.787066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:33.787066+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":70,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"772'40","reported_seq":339,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.224825+0000","last_change":"2024-02-12T17:19:11.833278+0000","last_active":"2024-02-12T17:19:48.224825+0000","last_peered":"2024-02-12T17:19:48.224825+0000","last_clean":"2024-02-12T17:19:48.224825+0000","last_became_active":"2024-02-12T17:13:46.566858+0000","last_became_peered":"2024-02-12T17:13:46.566858+0000","last_unstale":"2024-02-12T17:19:48.224825+0000","last_undegraded":"2024-02-12T17:19:48.224825+0000","last_fullsized":"2024-02-12T17:19:48.224825+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'32","last_scrub_stamp":"2024-02-12T17:19:11.833198+0000","last_deep_scrub":"766'32","last_deep_scrub_stamp":"2024-02-12T17:19:11.833198+0000","last_clean_scrub_stamp":"2024-02-12T17:19:11.833198+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":55,"num_read_kb":48,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"748'35","reported_seq":159,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314959+0000","last_change":"2024-02-12T17:19:25.386701+0000","last_active":"2024-02-12T17:19:35.314959+0000","last_peered":"2024-02-12T17:19:35.314959+0000","last_clean":"2024-02-12T17:19:35.314959+0000","last_became_active":"2024-02-12T17:14:08.861298+0000","last_became_peered":"2024-02-12T17:14:08.861298+0000","last_unstale":"2024-02-12T17:19:35.314959+0000","last_undegraded":"2024-02-12T17:19:35.314959+0000","last_fullsized":"2024-02-12T17:19:35.314959+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"748'35","last_scrub_stamp":"2024-02-12T17:19:25.386629+0000","last_deep_scrub":"748'35","last_deep_scrub_stamp":"2024-02-12T17:18:16.464548+0000","last_clean_scrub_stamp":"2024-02-12T17:19:25.386629+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"430'18","reported_seq":542,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829687+0000","last_change":"2024-02-12T17:19:03.327974+0000","last_active":"2024-02-12T17:19:35.829687+0000","last_peered":"2024-02-12T17:19:35.829687+0000","last_clean":"2024-02-12T17:19:35.829687+0000","last_became_active":"2024-02-12T17:17:58.837339+0000","last_became_peered":"2024-02-12T17:17:58.837339+0000","last_unstale":"2024-02-12T17:19:35.829687+0000","last_undegraded":"2024-02-12T17:19:35.829687+0000","last_fullsized":"2024-02-12T17:19:35.829687+0000","mapping_epoch":741,"log_start":"380'16","ondisk_log_start":"380'16","created":431,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":5,"last_scrub":"430'18","last_scrub_stamp":"2024-02-12T17:19:03.327877+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:19:03.327877+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":474,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529784+0000","last_change":"2024-02-12T17:18:17.845608+0000","last_active":"2024-02-12T17:19:37.529784+0000","last_peered":"2024-02-12T17:19:37.529784+0000","last_clean":"2024-02-12T17:19:37.529784+0000","last_became_active":"2024-02-12T17:14:08.754786+0000","last_became_peered":"2024-02-12T17:14:08.754786+0000","last_unstale":"2024-02-12T17:19:37.529784+0000","last_undegraded":"2024-02-12T17:19:37.529784+0000","last_fullsized":"2024-02-12T17:19:37.529784+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:17.845516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:17.845516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"602'32","reported_seq":497,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529515+0000","last_change":"2024-02-12T17:18:59.879333+0000","last_active":"2024-02-12T17:19:37.529515+0000","last_peered":"2024-02-12T17:19:37.529515+0000","last_clean":"2024-02-12T17:19:37.529515+0000","last_became_active":"2024-02-12T17:09:21.506264+0000","last_became_peered":"2024-02-12T17:09:21.506264+0000","last_unstale":"2024-02-12T17:19:37.529515+0000","last_undegraded":"2024-02-12T17:19:37.529515+0000","last_fullsized":"2024-02-12T17:19:37.529515+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'32","last_scrub_stamp":"2024-02-12T17:18:59.879239+0000","last_deep_scrub":"602'32","last_deep_scrub_stamp":"2024-02-12T17:12:12.141204+0000","last_clean_scrub_stamp":"2024-02-12T17:18:59.879239+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":336,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.072307+0000","last_change":"2024-02-12T17:19:00.042896+0000","last_active":"2024-02-12T17:19:46.072307+0000","last_peered":"2024-02-12T17:19:46.072307+0000","last_clean":"2024-02-12T17:19:46.072307+0000","last_became_active":"2024-02-12T17:17:26.177455+0000","last_became_peered":"2024-02-12T17:17:26.177455+0000","last_unstale":"2024-02-12T17:19:46.072307+0000","last_undegraded":"2024-02-12T17:19:46.072307+0000","last_fullsized":"2024-02-12T17:19:46.072307+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:00.042822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:19:00.042822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":782,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830155+0000","last_change":"2024-02-12T17:18:17.275982+0000","last_active":"2024-02-12T17:19:35.830155+0000","last_peered":"2024-02-12T17:19:35.830155+0000","last_clean":"2024-02-12T17:19:35.830155+0000","last_became_active":"2024-02-12T17:17:26.202355+0000","last_became_peered":"2024-02-12T17:17:26.202355+0000","last_unstale":"2024-02-12T17:19:35.830155+0000","last_undegraded":"2024-02-12T17:19:35.830155+0000","last_fullsized":"2024-02-12T17:19:35.830155+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:17.275878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:12:10.028495+0000","last_clean_scrub_stamp":"2024-02-12T17:18:17.275878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"526'14","reported_seq":519,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529339+0000","last_change":"2024-02-12T17:18:46.855768+0000","last_active":"2024-02-12T17:19:37.529339+0000","last_peered":"2024-02-12T17:19:37.529339+0000","last_clean":"2024-02-12T17:19:37.529339+0000","last_became_active":"2024-02-12T17:13:46.558100+0000","last_became_peered":"2024-02-12T17:13:46.558100+0000","last_unstale":"2024-02-12T17:19:37.529339+0000","last_undegraded":"2024-02-12T17:19:37.529339+0000","last_fullsized":"2024-02-12T17:19:37.529339+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'14","last_scrub_stamp":"2024-02-12T17:18:46.855664+0000","last_deep_scrub":"301'10","last_deep_scrub_stamp":"2024-02-12T17:00:58.938805+0000","last_clean_scrub_stamp":"2024-02-12T17:18:46.855664+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":172,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664462+0000","last_change":"2024-02-12T17:19:00.831493+0000","last_active":"2024-02-12T17:19:35.664462+0000","last_peered":"2024-02-12T17:19:35.664462+0000","last_clean":"2024-02-12T17:19:35.664462+0000","last_became_active":"2024-02-12T17:13:46.576185+0000","last_became_peered":"2024-02-12T17:13:46.576185+0000","last_unstale":"2024-02-12T17:19:35.664462+0000","last_undegraded":"2024-02-12T17:19:35.664462+0000","last_fullsized":"2024-02-12T17:19:35.664462+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:00.831422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:19:00.831422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":456,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529209+0000","last_change":"2024-02-12T17:18:56.915716+0000","last_active":"2024-02-12T17:19:37.529209+0000","last_peered":"2024-02-12T17:19:37.529209+0000","last_clean":"2024-02-12T17:19:37.529209+0000","last_became_active":"2024-02-12T17:14:08.758148+0000","last_became_peered":"2024-02-12T17:14:08.758148+0000","last_unstale":"2024-02-12T17:19:37.529209+0000","last_undegraded":"2024-02-12T17:19:37.529209+0000","last_fullsized":"2024-02-12T17:19:37.529209+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:56.915610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:42.040828+0000","last_clean_scrub_stamp":"2024-02-12T17:18:56.915610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":217,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664979+0000","last_change":"2024-02-12T17:19:28.860204+0000","last_active":"2024-02-12T17:19:35.664979+0000","last_peered":"2024-02-12T17:19:35.664979+0000","last_clean":"2024-02-12T17:19:35.664979+0000","last_became_active":"2024-02-12T17:14:08.741893+0000","last_became_peered":"2024-02-12T17:14:08.741893+0000","last_unstale":"2024-02-12T17:19:35.664979+0000","last_undegraded":"2024-02-12T17:19:35.664979+0000","last_fullsized":"2024-02-12T17:19:35.664979+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:28.860138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:19:28.860138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"772'35","reported_seq":86,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.196648+0000","last_change":"2024-02-12T17:19:26.831904+0000","last_active":"2024-02-12T17:19:48.196648+0000","last_peered":"2024-02-12T17:19:48.196648+0000","last_clean":"2024-02-12T17:19:48.196648+0000","last_became_active":"2024-02-12T17:17:26.208368+0000","last_became_peered":"2024-02-12T17:17:26.208368+0000","last_unstale":"2024-02-12T17:19:48.196648+0000","last_undegraded":"2024-02-12T17:19:48.196648+0000","last_fullsized":"2024-02-12T17:19:48.196648+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:19:26.831816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:26.831816+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"419'14","reported_seq":861,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529610+0000","last_change":"2024-02-12T17:19:27.594876+0000","last_active":"2024-02-12T17:19:37.529610+0000","last_peered":"2024-02-12T17:19:37.529610+0000","last_clean":"2024-02-12T17:19:37.529610+0000","last_became_active":"2024-02-12T17:17:16.819318+0000","last_became_peered":"2024-02-12T17:17:16.819318+0000","last_unstale":"2024-02-12T17:19:37.529610+0000","last_undegraded":"2024-02-12T17:19:37.529610+0000","last_fullsized":"2024-02-12T17:19:37.529610+0000","mapping_epoch":723,"log_start":"348'12","ondisk_log_start":"348'12","created":431,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":5,"last_scrub":"419'14","last_scrub_stamp":"2024-02-12T17:19:27.594775+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:19:27.594775+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":142,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:41.469235+0000","last_change":"2024-02-12T17:19:41.469235+0000","last_active":"2024-02-12T17:19:41.469235+0000","last_peered":"2024-02-12T17:19:41.469235+0000","last_clean":"2024-02-12T17:19:41.469235+0000","last_became_active":"2024-02-12T17:14:08.761880+0000","last_became_peered":"2024-02-12T17:14:08.761880+0000","last_unstale":"2024-02-12T17:19:41.469235+0000","last_undegraded":"2024-02-12T17:19:41.469235+0000","last_fullsized":"2024-02-12T17:19:41.469235+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:41.469115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:28.886321+0000","last_clean_scrub_stamp":"2024-02-12T17:19:41.469115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":290,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.071752+0000","last_change":"2024-02-12T17:19:40.996578+0000","last_active":"2024-02-12T17:19:46.071752+0000","last_peered":"2024-02-12T17:19:46.071752+0000","last_clean":"2024-02-12T17:19:46.071752+0000","last_became_active":"2024-02-12T17:09:21.525003+0000","last_became_peered":"2024-02-12T17:09:21.525003+0000","last_unstale":"2024-02-12T17:19:46.071752+0000","last_undegraded":"2024-02-12T17:19:46.071752+0000","last_fullsized":"2024-02-12T17:19:46.071752+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:40.996497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:19:40.996497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"526'25","reported_seq":244,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664419+0000","last_change":"2024-02-12T17:18:36.718952+0000","last_active":"2024-02-12T17:19:35.664419+0000","last_peered":"2024-02-12T17:19:35.664419+0000","last_clean":"2024-02-12T17:19:35.664419+0000","last_became_active":"2024-02-12T17:13:46.578343+0000","last_became_peered":"2024-02-12T17:13:46.578343+0000","last_unstale":"2024-02-12T17:19:35.664419+0000","last_undegraded":"2024-02-12T17:19:35.664419+0000","last_fullsized":"2024-02-12T17:19:35.664419+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:18:36.718882+0000","last_deep_scrub":"80'13","last_deep_scrub_stamp":"2024-02-12T16:51:01.759802+0000","last_clean_scrub_stamp":"2024-02-12T17:18:36.718882+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"443'16","reported_seq":920,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:42.730834+0000","last_change":"2024-02-12T17:19:17.798278+0000","last_active":"2024-02-12T17:19:42.730834+0000","last_peered":"2024-02-12T17:19:42.730834+0000","last_clean":"2024-02-12T17:19:42.730834+0000","last_became_active":"2024-02-12T17:05:55.414378+0000","last_became_peered":"2024-02-12T17:05:55.414378+0000","last_unstale":"2024-02-12T17:19:42.730834+0000","last_undegraded":"2024-02-12T17:19:42.730834+0000","last_fullsized":"2024-02-12T17:19:42.730834+0000","mapping_epoch":441,"log_start":"406'12","ondisk_log_start":"406'12","created":431,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"443'16","last_scrub_stamp":"2024-02-12T17:19:17.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:19:17.798196+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":146,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314742+0000","last_change":"2024-02-12T17:18:53.545289+0000","last_active":"2024-02-12T17:19:35.314742+0000","last_peered":"2024-02-12T17:19:35.314742+0000","last_clean":"2024-02-12T17:19:35.314742+0000","last_became_active":"2024-02-12T17:17:26.195255+0000","last_became_peered":"2024-02-12T17:17:26.195255+0000","last_unstale":"2024-02-12T17:19:35.314742+0000","last_undegraded":"2024-02-12T17:19:35.314742+0000","last_fullsized":"2024-02-12T17:19:35.314742+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:53.545220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:53.545220+0000","last_clean_scrub_stamp":"2024-02-12T17:18:53.545220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"772'27","reported_seq":880,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.206705+0000","last_change":"2024-02-12T17:19:06.975338+0000","last_active":"2024-02-12T17:19:48.206705+0000","last_peered":"2024-02-12T17:19:48.206705+0000","last_clean":"2024-02-12T17:19:48.206705+0000","last_became_active":"2024-02-12T17:03:50.573738+0000","last_became_peered":"2024-02-12T17:03:50.573738+0000","last_unstale":"2024-02-12T17:19:48.206705+0000","last_undegraded":"2024-02-12T17:19:48.206705+0000","last_fullsized":"2024-02-12T17:19:48.206705+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:19:06.975258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:06.975258+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":44,"num_read_kb":38,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"720'29","reported_seq":960,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:47.152895+0000","last_change":"2024-02-12T17:19:20.953303+0000","last_active":"2024-02-12T17:19:47.152895+0000","last_peered":"2024-02-12T17:19:47.152895+0000","last_clean":"2024-02-12T17:19:47.152895+0000","last_became_active":"2024-02-12T17:17:08.923702+0000","last_became_peered":"2024-02-12T17:17:08.923702+0000","last_unstale":"2024-02-12T17:19:47.152895+0000","last_undegraded":"2024-02-12T17:19:47.152895+0000","last_fullsized":"2024-02-12T17:19:47.152895+0000","mapping_epoch":718,"log_start":"406'13","ondisk_log_start":"406'13","created":431,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"720'29","last_scrub_stamp":"2024-02-12T17:19:20.953221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:19:20.953221+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":775,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:45.341464+0000","last_change":"2024-02-12T17:19:21.936639+0000","last_active":"2024-02-12T17:19:45.341464+0000","last_peered":"2024-02-12T17:19:45.341464+0000","last_clean":"2024-02-12T17:19:45.341464+0000","last_became_active":"2024-02-12T17:03:50.577686+0000","last_became_peered":"2024-02-12T17:03:50.577686+0000","last_unstale":"2024-02-12T17:19:45.341464+0000","last_undegraded":"2024-02-12T17:19:45.341464+0000","last_fullsized":"2024-02-12T17:19:45.341464+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:21.936575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:19:21.936575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"526'30","reported_seq":510,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529798+0000","last_change":"2024-02-12T17:18:30.819866+0000","last_active":"2024-02-12T17:19:37.529798+0000","last_peered":"2024-02-12T17:19:37.529798+0000","last_clean":"2024-02-12T17:19:37.529798+0000","last_became_active":"2024-02-12T17:13:46.577420+0000","last_became_peered":"2024-02-12T17:13:46.577420+0000","last_unstale":"2024-02-12T17:19:37.529798+0000","last_undegraded":"2024-02-12T17:19:37.529798+0000","last_fullsized":"2024-02-12T17:19:37.529798+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:18:30.819789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:18:30.819789+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":804,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830536+0000","last_change":"2024-02-12T17:18:37.317396+0000","last_active":"2024-02-12T17:19:35.830536+0000","last_peered":"2024-02-12T17:19:35.830536+0000","last_clean":"2024-02-12T17:19:35.830536+0000","last_became_active":"2024-02-12T17:17:26.177631+0000","last_became_peered":"2024-02-12T17:17:26.177631+0000","last_unstale":"2024-02-12T17:19:35.830536+0000","last_undegraded":"2024-02-12T17:19:35.830536+0000","last_fullsized":"2024-02-12T17:19:35.830536+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:37.317307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:58.187738+0000","last_clean_scrub_stamp":"2024-02-12T17:18:37.317307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":328,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.072191+0000","last_change":"2024-02-12T17:18:54.056532+0000","last_active":"2024-02-12T17:19:46.072191+0000","last_peered":"2024-02-12T17:19:46.072191+0000","last_clean":"2024-02-12T17:19:46.072191+0000","last_became_active":"2024-02-12T17:17:26.203398+0000","last_became_peered":"2024-02-12T17:17:26.203398+0000","last_unstale":"2024-02-12T17:19:46.072191+0000","last_undegraded":"2024-02-12T17:19:46.072191+0000","last_fullsized":"2024-02-12T17:19:46.072191+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:54.056451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:17:14.825440+0000","last_clean_scrub_stamp":"2024-02-12T17:18:54.056451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"526'21","reported_seq":499,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.530177+0000","last_change":"2024-02-12T17:18:26.732656+0000","last_active":"2024-02-12T17:19:37.530177+0000","last_peered":"2024-02-12T17:19:37.530177+0000","last_clean":"2024-02-12T17:19:37.530177+0000","last_became_active":"2024-02-12T17:13:46.821811+0000","last_became_peered":"2024-02-12T17:13:46.821811+0000","last_unstale":"2024-02-12T17:19:37.530177+0000","last_undegraded":"2024-02-12T17:19:37.530177+0000","last_fullsized":"2024-02-12T17:19:37.530177+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:18:26.732589+0000","last_deep_scrub":"460'15","last_deep_scrub_stamp":"2024-02-12T17:06:44.468722+0000","last_clean_scrub_stamp":"2024-02-12T17:18:26.732589+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":320,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.414038+0000","last_change":"2024-02-12T17:18:49.829558+0000","last_active":"2024-02-12T17:19:46.414038+0000","last_peered":"2024-02-12T17:19:46.414038+0000","last_clean":"2024-02-12T17:19:46.414038+0000","last_became_active":"2024-02-12T17:17:54.578042+0000","last_became_peered":"2024-02-12T17:17:54.578042+0000","last_unstale":"2024-02-12T17:19:46.414038+0000","last_undegraded":"2024-02-12T17:19:46.414038+0000","last_fullsized":"2024-02-12T17:19:46.414038+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:49.829475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:38.699602+0000","last_clean_scrub_stamp":"2024-02-12T17:18:49.829475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":438,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.413891+0000","last_change":"2024-02-12T17:18:40.707055+0000","last_active":"2024-02-12T17:19:46.413891+0000","last_peered":"2024-02-12T17:19:46.413891+0000","last_clean":"2024-02-12T17:19:46.413891+0000","last_became_active":"2024-02-12T17:17:54.640595+0000","last_became_peered":"2024-02-12T17:17:54.640595+0000","last_unstale":"2024-02-12T17:19:46.413891+0000","last_undegraded":"2024-02-12T17:19:46.413891+0000","last_fullsized":"2024-02-12T17:19:46.413891+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:40.706985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:40.706985+0000","last_clean_scrub_stamp":"2024-02-12T17:18:40.706985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"526'25","reported_seq":171,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.315045+0000","last_change":"2024-02-12T17:19:30.348783+0000","last_active":"2024-02-12T17:19:35.315045+0000","last_peered":"2024-02-12T17:19:35.315045+0000","last_clean":"2024-02-12T17:19:35.315045+0000","last_became_active":"2024-02-12T17:17:05.874272+0000","last_became_peered":"2024-02-12T17:17:05.874272+0000","last_unstale":"2024-02-12T17:19:35.315045+0000","last_undegraded":"2024-02-12T17:19:35.315045+0000","last_fullsized":"2024-02-12T17:19:35.315045+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'25","last_scrub_stamp":"2024-02-12T17:19:30.348700+0000","last_deep_scrub":"526'25","last_deep_scrub_stamp":"2024-02-12T17:19:30.348700+0000","last_clean_scrub_stamp":"2024-02-12T17:19:30.348700+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":498,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529694+0000","last_change":"2024-02-12T17:18:40.958756+0000","last_active":"2024-02-12T17:19:37.529694+0000","last_peered":"2024-02-12T17:19:37.529694+0000","last_clean":"2024-02-12T17:19:37.529694+0000","last_became_active":"2024-02-12T17:13:46.786586+0000","last_became_peered":"2024-02-12T17:13:46.786586+0000","last_unstale":"2024-02-12T17:19:37.529694+0000","last_undegraded":"2024-02-12T17:19:37.529694+0000","last_fullsized":"2024-02-12T17:19:37.529694+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:40.958634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:40.958634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"91'4","reported_seq":531,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.132583+0000","last_change":"2024-02-12T17:18:48.702512+0000","last_active":"2024-02-12T17:19:46.132583+0000","last_peered":"2024-02-12T17:19:46.132583+0000","last_clean":"2024-02-12T17:19:46.132583+0000","last_became_active":"2024-02-12T17:17:29.199721+0000","last_became_peered":"2024-02-12T17:17:29.199721+0000","last_unstale":"2024-02-12T17:19:46.132583+0000","last_undegraded":"2024-02-12T17:19:46.132583+0000","last_fullsized":"2024-02-12T17:19:46.132583+0000","mapping_epoch":730,"log_start":"91'4","ondisk_log_start":"91'4","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'4","last_scrub_stamp":"2024-02-12T17:18:48.702420+0000","last_deep_scrub":"91'4","last_deep_scrub_stamp":"2024-02-12T16:51:53.786683+0000","last_clean_scrub_stamp":"2024-02-12T17:18:48.702420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"772'30","reported_seq":310,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.054496+0000","last_change":"2024-02-12T17:19:12.785046+0000","last_active":"2024-02-12T17:19:48.054496+0000","last_peered":"2024-02-12T17:19:48.054496+0000","last_clean":"2024-02-12T17:19:48.054496+0000","last_became_active":"2024-02-12T17:18:16.700504+0000","last_became_peered":"2024-02-12T17:18:16.700504+0000","last_unstale":"2024-02-12T17:19:48.054496+0000","last_undegraded":"2024-02-12T17:19:48.054496+0000","last_fullsized":"2024-02-12T17:19:48.054496+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'24","last_scrub_stamp":"2024-02-12T17:19:12.784951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:12.784951+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":50,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":447,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.413891+0000","last_change":"2024-02-12T17:19:24.758158+0000","last_active":"2024-02-12T17:19:46.413891+0000","last_peered":"2024-02-12T17:19:46.413891+0000","last_clean":"2024-02-12T17:19:46.413891+0000","last_became_active":"2024-02-12T17:17:54.642052+0000","last_became_peered":"2024-02-12T17:17:54.642052+0000","last_unstale":"2024-02-12T17:19:46.413891+0000","last_undegraded":"2024-02-12T17:19:46.413891+0000","last_fullsized":"2024-02-12T17:19:46.413891+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:24.758048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:20.828117+0000","last_clean_scrub_stamp":"2024-02-12T17:19:24.758048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":784,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314149+0000","last_change":"2024-02-12T17:18:46.489852+0000","last_active":"2024-02-12T17:19:35.314149+0000","last_peered":"2024-02-12T17:19:35.314149+0000","last_clean":"2024-02-12T17:19:35.314149+0000","last_became_active":"2024-02-12T17:03:50.533083+0000","last_became_peered":"2024-02-12T17:03:50.533083+0000","last_unstale":"2024-02-12T17:19:35.314149+0000","last_undegraded":"2024-02-12T17:19:35.314149+0000","last_fullsized":"2024-02-12T17:19:35.314149+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:46.489732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:46.489732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664677+0000","last_change":"2024-02-12T17:18:57.804716+0000","last_active":"2024-02-12T17:19:35.664677+0000","last_peered":"2024-02-12T17:19:35.664677+0000","last_clean":"2024-02-12T17:19:35.664677+0000","last_became_active":"2024-02-12T17:13:46.789265+0000","last_became_peered":"2024-02-12T17:13:46.789265+0000","last_unstale":"2024-02-12T17:19:35.664677+0000","last_undegraded":"2024-02-12T17:19:35.664677+0000","last_fullsized":"2024-02-12T17:19:35.664677+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:57.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:57.804647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"526'30","reported_seq":273,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664531+0000","last_change":"2024-02-12T17:19:15.752836+0000","last_active":"2024-02-12T17:19:35.664531+0000","last_peered":"2024-02-12T17:19:35.664531+0000","last_clean":"2024-02-12T17:19:35.664531+0000","last_became_active":"2024-02-12T17:17:26.196542+0000","last_became_peered":"2024-02-12T17:17:26.196542+0000","last_unstale":"2024-02-12T17:19:35.664531+0000","last_undegraded":"2024-02-12T17:19:35.664531+0000","last_fullsized":"2024-02-12T17:19:35.664531+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'30","last_scrub_stamp":"2024-02-12T17:19:15.752760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:15.752760+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"706'24","reported_seq":657,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.730224+0000","last_change":"2024-02-12T17:19:01.523075+0000","last_active":"2024-02-12T17:19:37.730224+0000","last_peered":"2024-02-12T17:19:37.730224+0000","last_clean":"2024-02-12T17:19:37.730224+0000","last_became_active":"2024-02-12T17:16:46.730723+0000","last_became_peered":"2024-02-12T17:16:46.730723+0000","last_unstale":"2024-02-12T17:19:37.730224+0000","last_undegraded":"2024-02-12T17:19:37.730224+0000","last_fullsized":"2024-02-12T17:19:37.730224+0000","mapping_epoch":705,"log_start":"348'12","ondisk_log_start":"348'12","created":61,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"706'24","last_scrub_stamp":"2024-02-12T17:19:01.522997+0000","last_deep_scrub":"299'10","last_deep_scrub_stamp":"2024-02-12T17:00:51.955721+0000","last_clean_scrub_stamp":"2024-02-12T17:19:01.522997+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":510,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.530031+0000","last_change":"2024-02-12T17:19:23.695735+0000","last_active":"2024-02-12T17:19:37.530031+0000","last_peered":"2024-02-12T17:19:37.530031+0000","last_clean":"2024-02-12T17:19:37.530031+0000","last_became_active":"2024-02-12T17:17:26.187553+0000","last_became_peered":"2024-02-12T17:17:26.187553+0000","last_unstale":"2024-02-12T17:19:37.530031+0000","last_undegraded":"2024-02-12T17:19:37.530031+0000","last_fullsized":"2024-02-12T17:19:37.530031+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:23.695651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.530269+0000","last_clean_scrub_stamp":"2024-02-12T17:19:23.695651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":817,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:45.341603+0000","last_change":"2024-02-12T17:18:44.693029+0000","last_active":"2024-02-12T17:19:45.341603+0000","last_peered":"2024-02-12T17:19:45.341603+0000","last_clean":"2024-02-12T17:19:45.341603+0000","last_became_active":"2024-02-12T17:13:46.566669+0000","last_became_peered":"2024-02-12T17:13:46.566669+0000","last_unstale":"2024-02-12T17:19:45.341603+0000","last_undegraded":"2024-02-12T17:19:45.341603+0000","last_fullsized":"2024-02-12T17:19:45.341603+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:44.692951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:44.692951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"632'70","reported_seq":338,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.794539+0000","last_change":"2024-02-12T17:18:19.733557+0000","last_active":"2024-02-12T17:19:37.794539+0000","last_peered":"2024-02-12T17:19:37.794539+0000","last_clean":"2024-02-12T17:19:37.794539+0000","last_became_active":"2024-02-12T17:13:46.605833+0000","last_became_peered":"2024-02-12T17:13:46.605833+0000","last_unstale":"2024-02-12T17:19:37.794539+0000","last_undegraded":"2024-02-12T17:19:37.794539+0000","last_fullsized":"2024-02-12T17:19:37.794539+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'70","last_scrub_stamp":"2024-02-12T17:18:19.733432+0000","last_deep_scrub":"632'70","last_deep_scrub_stamp":"2024-02-12T17:18:19.733432+0000","last_clean_scrub_stamp":"2024-02-12T17:18:19.733432+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"410'15","reported_seq":766,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314766+0000","last_change":"2024-02-12T17:18:33.413491+0000","last_active":"2024-02-12T17:19:35.314766+0000","last_peered":"2024-02-12T17:19:35.314766+0000","last_clean":"2024-02-12T17:19:35.314766+0000","last_became_active":"2024-02-12T17:18:28.897328+0000","last_became_peered":"2024-02-12T17:18:28.897328+0000","last_unstale":"2024-02-12T17:19:35.314766+0000","last_undegraded":"2024-02-12T17:19:35.314766+0000","last_fullsized":"2024-02-12T17:19:35.314766+0000","mapping_epoch":756,"log_start":"406'13","ondisk_log_start":"406'13","created":61,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-02-12T17:18:33.413420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:02:35.760747+0000","last_clean_scrub_stamp":"2024-02-12T17:18:33.413420+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":784,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830712+0000","last_change":"2024-02-12T17:18:56.309880+0000","last_active":"2024-02-12T17:19:35.830712+0000","last_peered":"2024-02-12T17:19:35.830712+0000","last_clean":"2024-02-12T17:19:35.830712+0000","last_became_active":"2024-02-12T17:09:21.502437+0000","last_became_peered":"2024-02-12T17:09:21.502437+0000","last_unstale":"2024-02-12T17:19:35.830712+0000","last_undegraded":"2024-02-12T17:19:35.830712+0000","last_fullsized":"2024-02-12T17:19:35.830712+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:56.309777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:17:31.297073+0000","last_clean_scrub_stamp":"2024-02-12T17:18:56.309777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":638,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:43.664619+0000","last_change":"2024-02-12T17:18:28.684274+0000","last_active":"2024-02-12T17:19:43.664619+0000","last_peered":"2024-02-12T17:19:43.664619+0000","last_clean":"2024-02-12T17:19:43.664619+0000","last_became_active":"2024-02-12T17:13:46.584224+0000","last_became_peered":"2024-02-12T17:13:46.584224+0000","last_unstale":"2024-02-12T17:19:43.664619+0000","last_undegraded":"2024-02-12T17:19:43.664619+0000","last_fullsized":"2024-02-12T17:19:43.664619+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-12T17:18:28.684209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:28.684209+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":570,"num_read_kb":565,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":347,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.413924+0000","last_change":"2024-02-12T17:18:41.710877+0000","last_active":"2024-02-12T17:19:46.413924+0000","last_peered":"2024-02-12T17:19:46.413924+0000","last_clean":"2024-02-12T17:19:46.413924+0000","last_became_active":"2024-02-12T17:17:54.640095+0000","last_became_peered":"2024-02-12T17:17:54.640095+0000","last_unstale":"2024-02-12T17:19:46.413924+0000","last_undegraded":"2024-02-12T17:19:46.413924+0000","last_fullsized":"2024-02-12T17:19:46.413924+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:41.710761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:41.710761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"526'26","reported_seq":245,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664837+0000","last_change":"2024-02-12T17:18:26.685828+0000","last_active":"2024-02-12T17:19:35.664837+0000","last_peered":"2024-02-12T17:19:35.664837+0000","last_clean":"2024-02-12T17:19:35.664837+0000","last_became_active":"2024-02-12T17:14:08.752255+0000","last_became_peered":"2024-02-12T17:14:08.752255+0000","last_unstale":"2024-02-12T17:19:35.664837+0000","last_undegraded":"2024-02-12T17:19:35.664837+0000","last_fullsized":"2024-02-12T17:19:35.664837+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'26","last_scrub_stamp":"2024-02-12T17:18:26.685757+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-12T16:53:04.923375+0000","last_clean_scrub_stamp":"2024-02-12T17:18:26.685757+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":339,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.414258+0000","last_change":"2024-02-12T17:19:03.789449+0000","last_active":"2024-02-12T17:19:46.414258+0000","last_peered":"2024-02-12T17:19:46.414258+0000","last_clean":"2024-02-12T17:19:46.414258+0000","last_became_active":"2024-02-12T17:17:54.579441+0000","last_became_peered":"2024-02-12T17:17:54.579441+0000","last_unstale":"2024-02-12T17:19:46.414258+0000","last_undegraded":"2024-02-12T17:19:46.414258+0000","last_fullsized":"2024-02-12T17:19:46.414258+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:03.789385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:13:10.412031+0000","last_clean_scrub_stamp":"2024-02-12T17:19:03.789385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":858,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:45.342170+0000","last_change":"2024-02-12T17:18:59.106926+0000","last_active":"2024-02-12T17:19:45.342170+0000","last_peered":"2024-02-12T17:19:45.342170+0000","last_clean":"2024-02-12T17:19:45.342170+0000","last_became_active":"2024-02-12T17:17:26.190887+0000","last_became_peered":"2024-02-12T17:17:26.190887+0000","last_unstale":"2024-02-12T17:19:45.342170+0000","last_undegraded":"2024-02-12T17:19:45.342170+0000","last_fullsized":"2024-02-12T17:19:45.342170+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:59.106845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:17:41.406170+0000","last_clean_scrub_stamp":"2024-02-12T17:18:59.106845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":479,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529621+0000","last_change":"2024-02-12T17:19:15.745904+0000","last_active":"2024-02-12T17:19:37.529621+0000","last_peered":"2024-02-12T17:19:37.529621+0000","last_clean":"2024-02-12T17:19:37.529621+0000","last_became_active":"2024-02-12T17:09:21.601886+0000","last_became_peered":"2024-02-12T17:09:21.601886+0000","last_unstale":"2024-02-12T17:19:37.529621+0000","last_undegraded":"2024-02-12T17:19:37.529621+0000","last_fullsized":"2024-02-12T17:19:37.529621+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:15.745831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:55.967335+0000","last_clean_scrub_stamp":"2024-02-12T17:19:15.745831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":757,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:45.341705+0000","last_change":"2024-02-12T17:18:59.945263+0000","last_active":"2024-02-12T17:19:45.341705+0000","last_peered":"2024-02-12T17:19:45.341705+0000","last_clean":"2024-02-12T17:19:45.341705+0000","last_became_active":"2024-02-12T16:49:39.105839+0000","last_became_peered":"2024-02-12T16:49:39.105839+0000","last_unstale":"2024-02-12T17:19:45.341705+0000","last_undegraded":"2024-02-12T17:19:45.341705+0000","last_fullsized":"2024-02-12T17:19:45.341705+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:59.945190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:59.945190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":824,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829525+0000","last_change":"2024-02-12T17:18:36.152282+0000","last_active":"2024-02-12T17:19:35.829525+0000","last_peered":"2024-02-12T17:19:35.829525+0000","last_clean":"2024-02-12T17:19:35.829525+0000","last_became_active":"2024-02-12T17:14:08.743405+0000","last_became_peered":"2024-02-12T17:14:08.743405+0000","last_unstale":"2024-02-12T17:19:35.829525+0000","last_undegraded":"2024-02-12T17:19:35.829525+0000","last_fullsized":"2024-02-12T17:19:35.829525+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:36.152143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:36.152143+0000","last_clean_scrub_stamp":"2024-02-12T17:18:36.152143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"772'57","reported_seq":440,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.212863+0000","last_change":"2024-02-12T17:19:12.868579+0000","last_active":"2024-02-12T17:19:48.212863+0000","last_peered":"2024-02-12T17:19:48.212863+0000","last_clean":"2024-02-12T17:19:48.212863+0000","last_became_active":"2024-02-12T17:17:26.177646+0000","last_became_peered":"2024-02-12T17:17:26.177646+0000","last_unstale":"2024-02-12T17:19:48.212863+0000","last_undegraded":"2024-02-12T17:19:48.212863+0000","last_fullsized":"2024-02-12T17:19:48.212863+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:19:12.868467+0000","last_deep_scrub":"526'45","last_deep_scrub_stamp":"2024-02-12T17:19:12.868467+0000","last_clean_scrub_stamp":"2024-02-12T17:19:12.868467+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":77,"num_read_kb":67,"num_write":49,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":1078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"414'14","reported_seq":791,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830448+0000","last_change":"2024-02-12T17:19:21.291190+0000","last_active":"2024-02-12T17:19:35.830448+0000","last_peered":"2024-02-12T17:19:35.830448+0000","last_clean":"2024-02-12T17:19:35.830448+0000","last_became_active":"2024-02-12T17:05:40.432616+0000","last_became_peered":"2024-02-12T17:05:40.432616+0000","last_unstale":"2024-02-12T17:19:35.830448+0000","last_undegraded":"2024-02-12T17:19:35.830448+0000","last_fullsized":"2024-02-12T17:19:35.830448+0000","mapping_epoch":61,"log_start":"406'12","ondisk_log_start":"406'12","created":61,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-02-12T17:19:21.291091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:19:21.291091+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":172,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664748+0000","last_change":"2024-02-12T17:18:32.748113+0000","last_active":"2024-02-12T17:19:35.664748+0000","last_peered":"2024-02-12T17:19:35.664748+0000","last_clean":"2024-02-12T17:19:35.664748+0000","last_became_active":"2024-02-12T17:13:46.606846+0000","last_became_peered":"2024-02-12T17:13:46.606846+0000","last_unstale":"2024-02-12T17:19:35.664748+0000","last_undegraded":"2024-02-12T17:19:35.664748+0000","last_fullsized":"2024-02-12T17:19:35.664748+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:32.748035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:32.748035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":146,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314411+0000","last_change":"2024-02-12T17:18:57.550083+0000","last_active":"2024-02-12T17:19:35.314411+0000","last_peered":"2024-02-12T17:19:35.314411+0000","last_clean":"2024-02-12T17:19:35.314411+0000","last_became_active":"2024-02-12T17:17:26.194140+0000","last_became_peered":"2024-02-12T17:17:26.194140+0000","last_unstale":"2024-02-12T17:19:35.314411+0000","last_undegraded":"2024-02-12T17:19:35.314411+0000","last_fullsized":"2024-02-12T17:19:35.314411+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:57.549951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:54.941776+0000","last_clean_scrub_stamp":"2024-02-12T17:18:57.549951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"444'2","reported_seq":824,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829436+0000","last_change":"2024-02-12T17:18:30.205400+0000","last_active":"2024-02-12T17:19:35.829436+0000","last_peered":"2024-02-12T17:19:35.829436+0000","last_clean":"2024-02-12T17:19:35.829436+0000","last_became_active":"2024-02-12T17:13:46.576443+0000","last_became_peered":"2024-02-12T17:13:46.576443+0000","last_unstale":"2024-02-12T17:19:35.829436+0000","last_undegraded":"2024-02-12T17:19:35.829436+0000","last_fullsized":"2024-02-12T17:19:35.829436+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'2","last_scrub_stamp":"2024-02-12T17:18:30.205314+0000","last_deep_scrub":"444'2","last_deep_scrub_stamp":"2024-02-12T17:18:30.205314+0000","last_clean_scrub_stamp":"2024-02-12T17:18:30.205314+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"526'36","reported_seq":156,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:39.427871+0000","last_change":"2024-02-12T17:19:39.427871+0000","last_active":"2024-02-12T17:19:39.427871+0000","last_peered":"2024-02-12T17:19:39.427871+0000","last_clean":"2024-02-12T17:19:39.427871+0000","last_became_active":"2024-02-12T17:14:08.860528+0000","last_became_peered":"2024-02-12T17:14:08.860528+0000","last_unstale":"2024-02-12T17:19:39.427871+0000","last_undegraded":"2024-02-12T17:19:39.427871+0000","last_fullsized":"2024-02-12T17:19:39.427871+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'36","last_scrub_stamp":"2024-02-12T17:19:39.427737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:39.427737+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":734,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664923+0000","last_change":"2024-02-12T17:19:09.750191+0000","last_active":"2024-02-12T17:19:35.664923+0000","last_peered":"2024-02-12T17:19:35.664923+0000","last_clean":"2024-02-12T17:19:35.664923+0000","last_became_active":"2024-02-12T17:17:26.196430+0000","last_became_peered":"2024-02-12T17:17:26.196430+0000","last_unstale":"2024-02-12T17:19:35.664923+0000","last_undegraded":"2024-02-12T17:19:35.664923+0000","last_fullsized":"2024-02-12T17:19:35.664923+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:09.750123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:17:35.731956+0000","last_clean_scrub_stamp":"2024-02-12T17:19:09.750123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":203,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664307+0000","last_change":"2024-02-12T17:18:34.687748+0000","last_active":"2024-02-12T17:19:35.664307+0000","last_peered":"2024-02-12T17:19:35.664307+0000","last_clean":"2024-02-12T17:19:35.664307+0000","last_became_active":"2024-02-12T17:13:46.579399+0000","last_became_peered":"2024-02-12T17:13:46.579399+0000","last_unstale":"2024-02-12T17:19:35.664307+0000","last_undegraded":"2024-02-12T17:19:35.664307+0000","last_fullsized":"2024-02-12T17:19:35.664307+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:34.687610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:32.884670+0000","last_clean_scrub_stamp":"2024-02-12T17:18:34.687610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":847,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830687+0000","last_change":"2024-02-12T17:18:45.086113+0000","last_active":"2024-02-12T17:19:35.830687+0000","last_peered":"2024-02-12T17:19:35.830687+0000","last_clean":"2024-02-12T17:19:35.830687+0000","last_became_active":"2024-02-12T17:17:26.184374+0000","last_became_peered":"2024-02-12T17:17:26.184374+0000","last_unstale":"2024-02-12T17:19:35.830687+0000","last_undegraded":"2024-02-12T17:19:35.830687+0000","last_fullsized":"2024-02-12T17:19:35.830687+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:45.086043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:45.086043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"766'47","reported_seq":163,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314701+0000","last_change":"2024-02-12T17:18:47.489994+0000","last_active":"2024-02-12T17:19:35.314701+0000","last_peered":"2024-02-12T17:19:35.314701+0000","last_clean":"2024-02-12T17:19:35.314701+0000","last_became_active":"2024-02-12T17:14:08.871911+0000","last_became_peered":"2024-02-12T17:14:08.871911+0000","last_unstale":"2024-02-12T17:19:35.314701+0000","last_undegraded":"2024-02-12T17:19:35.314701+0000","last_fullsized":"2024-02-12T17:19:35.314701+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'47","last_scrub_stamp":"2024-02-12T17:18:47.489913+0000","last_deep_scrub":"766'47","last_deep_scrub_stamp":"2024-02-12T17:18:47.489913+0000","last_clean_scrub_stamp":"2024-02-12T17:18:47.489913+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":283,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.071734+0000","last_change":"2024-02-12T17:18:51.015924+0000","last_active":"2024-02-12T17:19:46.071734+0000","last_peered":"2024-02-12T17:19:46.071734+0000","last_clean":"2024-02-12T17:19:46.071734+0000","last_became_active":"2024-02-12T17:09:21.558759+0000","last_became_peered":"2024-02-12T17:09:21.558759+0000","last_unstale":"2024-02-12T17:19:46.071734+0000","last_undegraded":"2024-02-12T17:19:46.071734+0000","last_fullsized":"2024-02-12T17:19:46.071734+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:51.015818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:18:51.015818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":474,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.131946+0000","last_change":"2024-02-12T17:19:21.821426+0000","last_active":"2024-02-12T17:19:46.131946+0000","last_peered":"2024-02-12T17:19:46.131946+0000","last_clean":"2024-02-12T17:19:46.131946+0000","last_became_active":"2024-02-12T17:18:16.699753+0000","last_became_peered":"2024-02-12T17:18:16.699753+0000","last_unstale":"2024-02-12T17:19:46.131946+0000","last_undegraded":"2024-02-12T17:19:46.131946+0000","last_fullsized":"2024-02-12T17:19:46.131946+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:21.821331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:32.780114+0000","last_clean_scrub_stamp":"2024-02-12T17:19:21.821331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":228,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664839+0000","last_change":"2024-02-12T17:19:08.787891+0000","last_active":"2024-02-12T17:19:35.664839+0000","last_peered":"2024-02-12T17:19:35.664839+0000","last_clean":"2024-02-12T17:19:35.664839+0000","last_became_active":"2024-02-12T17:17:26.195505+0000","last_became_peered":"2024-02-12T17:17:26.195505+0000","last_unstale":"2024-02-12T17:19:35.664839+0000","last_undegraded":"2024-02-12T17:19:35.664839+0000","last_fullsized":"2024-02-12T17:19:35.664839+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:08.787809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:00:52.143796+0000","last_clean_scrub_stamp":"2024-02-12T17:19:08.787809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"526'42","reported_seq":552,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.795928+0000","last_change":"2024-02-12T17:19:37.795899+0000","last_active":"2024-02-12T17:19:37.795928+0000","last_peered":"2024-02-12T17:19:37.795928+0000","last_clean":"2024-02-12T17:19:37.795928+0000","last_became_active":"2024-02-12T17:14:08.754589+0000","last_became_peered":"2024-02-12T17:14:08.754589+0000","last_unstale":"2024-02-12T17:19:37.795928+0000","last_undegraded":"2024-02-12T17:19:37.795928+0000","last_fullsized":"2024-02-12T17:19:37.795928+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'42","last_scrub_stamp":"2024-02-12T17:19:37.795814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:37.795814+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":62,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.132071+0000","last_change":"2024-02-12T17:18:43.769492+0000","last_active":"2024-02-12T17:19:46.132071+0000","last_peered":"2024-02-12T17:19:46.132071+0000","last_clean":"2024-02-12T17:19:46.132071+0000","last_became_active":"2024-02-12T17:17:26.183165+0000","last_became_peered":"2024-02-12T17:17:26.183165+0000","last_unstale":"2024-02-12T17:19:46.132071+0000","last_undegraded":"2024-02-12T17:19:46.132071+0000","last_fullsized":"2024-02-12T17:19:46.132071+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:43.769411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:18:43.769411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"766'2","reported_seq":818,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.830285+0000","last_change":"2024-02-12T17:18:57.453772+0000","last_active":"2024-02-12T17:19:35.830285+0000","last_peered":"2024-02-12T17:19:35.830285+0000","last_clean":"2024-02-12T17:19:35.830285+0000","last_became_active":"2024-02-12T17:09:21.493648+0000","last_became_peered":"2024-02-12T17:09:21.493648+0000","last_unstale":"2024-02-12T17:19:35.830285+0000","last_undegraded":"2024-02-12T17:19:35.830285+0000","last_fullsized":"2024-02-12T17:19:35.830285+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'2","last_scrub_stamp":"2024-02-12T17:18:57.453704+0000","last_deep_scrub":"766'2","last_deep_scrub_stamp":"2024-02-12T17:18:57.453704+0000","last_clean_scrub_stamp":"2024-02-12T17:18:57.453704+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":301,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.072477+0000","last_change":"2024-02-12T17:18:45.026850+0000","last_active":"2024-02-12T17:19:46.072477+0000","last_peered":"2024-02-12T17:19:46.072477+0000","last_clean":"2024-02-12T17:19:46.072477+0000","last_became_active":"2024-02-12T17:14:14.842980+0000","last_became_peered":"2024-02-12T17:14:14.842980+0000","last_unstale":"2024-02-12T17:19:46.072477+0000","last_undegraded":"2024-02-12T17:19:46.072477+0000","last_fullsized":"2024-02-12T17:19:46.072477+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:45.026763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:45.026763+0000","last_clean_scrub_stamp":"2024-02-12T17:18:45.026763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"772'63","reported_seq":984,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.220807+0000","last_change":"2024-02-12T17:18:54.684382+0000","last_active":"2024-02-12T17:19:48.220807+0000","last_peered":"2024-02-12T17:19:48.220807+0000","last_clean":"2024-02-12T17:19:48.220807+0000","last_became_active":"2024-02-12T17:18:16.697461+0000","last_became_peered":"2024-02-12T17:18:16.697461+0000","last_unstale":"2024-02-12T17:19:48.220807+0000","last_undegraded":"2024-02-12T17:19:48.220807+0000","last_fullsized":"2024-02-12T17:19:48.220807+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'49","last_scrub_stamp":"2024-02-12T17:18:54.684303+0000","last_deep_scrub":"766'49","last_deep_scrub_stamp":"2024-02-12T17:18:54.684303+0000","last_clean_scrub_stamp":"2024-02-12T17:18:54.684303+0000","log_size":63,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":83,"num_read_kb":76,"num_write":56,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"743'36","reported_seq":818,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:42.730857+0000","last_change":"2024-02-12T17:19:28.632137+0000","last_active":"2024-02-12T17:19:42.730857+0000","last_peered":"2024-02-12T17:19:42.730857+0000","last_clean":"2024-02-12T17:19:42.730857+0000","last_became_active":"2024-02-12T17:17:58.841255+0000","last_became_peered":"2024-02-12T17:17:58.841255+0000","last_unstale":"2024-02-12T17:19:42.730857+0000","last_undegraded":"2024-02-12T17:19:42.730857+0000","last_fullsized":"2024-02-12T17:19:42.730857+0000","mapping_epoch":741,"log_start":"380'16","ondisk_log_start":"380'16","created":61,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'36","last_scrub_stamp":"2024-02-12T17:19:28.632065+0000","last_deep_scrub":"328'11","last_deep_scrub_stamp":"2024-02-12T17:02:29.064208+0000","last_clean_scrub_stamp":"2024-02-12T17:19:28.632065+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"87'2","reported_seq":490,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.529730+0000","last_change":"2024-02-12T17:18:38.898475+0000","last_active":"2024-02-12T17:19:37.529730+0000","last_peered":"2024-02-12T17:19:37.529730+0000","last_clean":"2024-02-12T17:19:37.529730+0000","last_became_active":"2024-02-12T17:09:26.728651+0000","last_became_peered":"2024-02-12T17:09:26.728651+0000","last_unstale":"2024-02-12T17:19:37.529730+0000","last_undegraded":"2024-02-12T17:19:37.529730+0000","last_fullsized":"2024-02-12T17:19:37.529730+0000","mapping_epoch":518,"log_start":"87'2","ondisk_log_start":"87'2","created":63,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-02-12T17:18:38.898389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:38.898389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"743'34","reported_seq":1264,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:47.153030+0000","last_change":"2024-02-12T17:19:25.962456+0000","last_active":"2024-02-12T17:19:47.153030+0000","last_peered":"2024-02-12T17:19:47.153030+0000","last_clean":"2024-02-12T17:19:47.153030+0000","last_became_active":"2024-02-12T17:17:58.837743+0000","last_became_peered":"2024-02-12T17:17:58.837743+0000","last_unstale":"2024-02-12T17:19:47.153030+0000","last_undegraded":"2024-02-12T17:19:47.153030+0000","last_fullsized":"2024-02-12T17:19:47.153030+0000","mapping_epoch":741,"log_start":"199'9","ondisk_log_start":"199'9","created":61,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'34","last_scrub_stamp":"2024-02-12T17:19:25.962353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:05.201749+0000","last_clean_scrub_stamp":"2024-02-12T17:19:25.962353+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":492,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829236+0000","last_change":"2024-02-12T17:18:46.103660+0000","last_active":"2024-02-12T17:19:35.829236+0000","last_peered":"2024-02-12T17:19:35.829236+0000","last_clean":"2024-02-12T17:19:35.829236+0000","last_became_active":"2024-02-12T17:14:10.756266+0000","last_became_peered":"2024-02-12T17:14:10.756266+0000","last_unstale":"2024-02-12T17:19:35.829236+0000","last_undegraded":"2024-02-12T17:19:35.829236+0000","last_fullsized":"2024-02-12T17:19:35.829236+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:46.103544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:57:33.806968+0000","last_clean_scrub_stamp":"2024-02-12T17:18:46.103544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"772'35","reported_seq":362,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.216838+0000","last_change":"2024-02-12T17:19:39.935460+0000","last_active":"2024-02-12T17:19:48.216838+0000","last_peered":"2024-02-12T17:19:48.216838+0000","last_clean":"2024-02-12T17:19:48.216838+0000","last_became_active":"2024-02-12T17:18:16.699402+0000","last_became_peered":"2024-02-12T17:18:16.699402+0000","last_unstale":"2024-02-12T17:19:48.216838+0000","last_undegraded":"2024-02-12T17:19:48.216838+0000","last_fullsized":"2024-02-12T17:19:48.216838+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'27","last_scrub_stamp":"2024-02-12T17:19:39.935383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:39.935383+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":824,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:45.342055+0000","last_change":"2024-02-12T17:18:23.035607+0000","last_active":"2024-02-12T17:19:45.342055+0000","last_peered":"2024-02-12T17:19:45.342055+0000","last_clean":"2024-02-12T17:19:45.342055+0000","last_became_active":"2024-02-12T17:14:08.740586+0000","last_became_peered":"2024-02-12T17:14:08.740586+0000","last_unstale":"2024-02-12T17:19:45.342055+0000","last_undegraded":"2024-02-12T17:19:45.342055+0000","last_fullsized":"2024-02-12T17:19:45.342055+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:23.035499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:44.186028+0000","last_clean_scrub_stamp":"2024-02-12T17:18:23.035499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"750'31","reported_seq":1270,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.854873+0000","last_change":"2024-02-12T17:19:27.843776+0000","last_active":"2024-02-12T17:19:46.854873+0000","last_peered":"2024-02-12T17:19:46.854873+0000","last_clean":"2024-02-12T17:19:46.854873+0000","last_became_active":"2024-02-12T17:18:16.703985+0000","last_became_peered":"2024-02-12T17:18:16.703985+0000","last_unstale":"2024-02-12T17:19:46.854873+0000","last_undegraded":"2024-02-12T17:19:46.854873+0000","last_fullsized":"2024-02-12T17:19:46.854873+0000","mapping_epoch":748,"log_start":"98'10","ondisk_log_start":"98'10","created":61,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"750'31","last_scrub_stamp":"2024-02-12T17:19:27.843652+0000","last_deep_scrub":"596'24","last_deep_scrub_stamp":"2024-02-12T17:12:59.647212+0000","last_clean_scrub_stamp":"2024-02-12T17:19:27.843652+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":195,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664237+0000","last_change":"2024-02-12T17:19:33.805935+0000","last_active":"2024-02-12T17:19:35.664237+0000","last_peered":"2024-02-12T17:19:35.664237+0000","last_clean":"2024-02-12T17:19:35.664237+0000","last_became_active":"2024-02-12T17:14:05.105295+0000","last_became_peered":"2024-02-12T17:14:05.105295+0000","last_unstale":"2024-02-12T17:19:35.664237+0000","last_undegraded":"2024-02-12T17:19:35.664237+0000","last_fullsized":"2024-02-12T17:19:35.664237+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:33.805876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:19:33.805876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"526'46","reported_seq":263,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664895+0000","last_change":"2024-02-12T17:19:07.764613+0000","last_active":"2024-02-12T17:19:35.664895+0000","last_peered":"2024-02-12T17:19:35.664895+0000","last_clean":"2024-02-12T17:19:35.664895+0000","last_became_active":"2024-02-12T17:17:26.196636+0000","last_became_peered":"2024-02-12T17:17:26.196636+0000","last_unstale":"2024-02-12T17:19:35.664895+0000","last_undegraded":"2024-02-12T17:19:35.664895+0000","last_fullsized":"2024-02-12T17:19:35.664895+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:19:07.764488+0000","last_deep_scrub":"526'46","last_deep_scrub_stamp":"2024-02-12T17:19:07.764488+0000","last_clean_scrub_stamp":"2024-02-12T17:19:07.764488+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":157,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.132381+0000","last_change":"2024-02-12T17:18:49.810353+0000","last_active":"2024-02-12T17:19:46.132381+0000","last_peered":"2024-02-12T17:19:46.132381+0000","last_clean":"2024-02-12T17:19:46.132381+0000","last_became_active":"2024-02-12T17:18:16.701099+0000","last_became_peered":"2024-02-12T17:18:16.701099+0000","last_unstale":"2024-02-12T17:19:46.132381+0000","last_undegraded":"2024-02-12T17:19:46.132381+0000","last_fullsized":"2024-02-12T17:19:46.132381+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:49.810252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:05:57.077971+0000","last_clean_scrub_stamp":"2024-02-12T17:18:49.810252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"740'15","reported_seq":505,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.856920+0000","last_change":"2024-02-12T17:18:53.850343+0000","last_active":"2024-02-12T17:19:46.856920+0000","last_peered":"2024-02-12T17:19:46.856920+0000","last_clean":"2024-02-12T17:19:46.856920+0000","last_became_active":"2024-02-12T17:17:54.579125+0000","last_became_peered":"2024-02-12T17:17:54.579125+0000","last_unstale":"2024-02-12T17:19:46.856920+0000","last_undegraded":"2024-02-12T17:19:46.856920+0000","last_fullsized":"2024-02-12T17:19:46.856920+0000","mapping_epoch":739,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'15","last_scrub_stamp":"2024-02-12T17:18:53.850266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:18:53.850266+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664241+0000","last_change":"2024-02-12T17:18:52.749854+0000","last_active":"2024-02-12T17:19:35.664241+0000","last_peered":"2024-02-12T17:19:35.664241+0000","last_clean":"2024-02-12T17:19:35.664241+0000","last_became_active":"2024-02-12T17:14:00.999636+0000","last_became_peered":"2024-02-12T17:14:00.999636+0000","last_unstale":"2024-02-12T17:19:35.664241+0000","last_undegraded":"2024-02-12T17:19:35.664241+0000","last_fullsized":"2024-02-12T17:19:35.664241+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:52.749783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:52.749783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"772'45","reported_seq":612,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.176727+0000","last_change":"2024-02-12T17:19:00.745588+0000","last_active":"2024-02-12T17:19:48.176727+0000","last_peered":"2024-02-12T17:19:48.176727+0000","last_clean":"2024-02-12T17:19:48.176727+0000","last_became_active":"2024-02-12T17:18:16.699885+0000","last_became_peered":"2024-02-12T17:18:16.699885+0000","last_unstale":"2024-02-12T17:19:48.176727+0000","last_undegraded":"2024-02-12T17:19:48.176727+0000","last_fullsized":"2024-02-12T17:19:48.176727+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'35","last_scrub_stamp":"2024-02-12T17:19:00.745489+0000","last_deep_scrub":"733'35","last_deep_scrub_stamp":"2024-02-12T17:17:33.454203+0000","last_clean_scrub_stamp":"2024-02-12T17:19:00.745489+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":58,"num_read_kb":53,"num_write":40,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":511,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.414213+0000","last_change":"2024-02-12T17:18:24.022023+0000","last_active":"2024-02-12T17:19:46.414213+0000","last_peered":"2024-02-12T17:19:46.414213+0000","last_clean":"2024-02-12T17:19:46.414213+0000","last_became_active":"2024-02-12T17:17:54.582218+0000","last_became_peered":"2024-02-12T17:17:54.582218+0000","last_unstale":"2024-02-12T17:19:46.414213+0000","last_undegraded":"2024-02-12T17:19:46.414213+0000","last_fullsized":"2024-02-12T17:19:46.414213+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:24.021907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:24.021907+0000","last_clean_scrub_stamp":"2024-02-12T17:18:24.021907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":210,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664627+0000","last_change":"2024-02-12T17:18:50.773491+0000","last_active":"2024-02-12T17:19:35.664627+0000","last_peered":"2024-02-12T17:19:35.664627+0000","last_clean":"2024-02-12T17:19:35.664627+0000","last_became_active":"2024-02-12T17:17:26.202178+0000","last_became_peered":"2024-02-12T17:17:26.202178+0000","last_unstale":"2024-02-12T17:19:35.664627+0000","last_undegraded":"2024-02-12T17:19:35.664627+0000","last_fullsized":"2024-02-12T17:19:35.664627+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:50.773366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:56:33.052447+0000","last_clean_scrub_stamp":"2024-02-12T17:18:50.773366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":811,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829959+0000","last_change":"2024-02-12T17:18:43.110721+0000","last_active":"2024-02-12T17:19:35.829959+0000","last_peered":"2024-02-12T17:19:35.829959+0000","last_clean":"2024-02-12T17:19:35.829959+0000","last_became_active":"2024-02-12T17:17:26.181989+0000","last_became_peered":"2024-02-12T17:17:26.181989+0000","last_unstale":"2024-02-12T17:19:35.829959+0000","last_undegraded":"2024-02-12T17:19:35.829959+0000","last_fullsized":"2024-02-12T17:19:35.829959+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:43.110640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:38.079400+0000","last_clean_scrub_stamp":"2024-02-12T17:18:43.110640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"772'59","reported_seq":278,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.214899+0000","last_change":"2024-02-12T17:18:59.770756+0000","last_active":"2024-02-12T17:19:48.214899+0000","last_peered":"2024-02-12T17:19:48.214899+0000","last_clean":"2024-02-12T17:19:48.214899+0000","last_became_active":"2024-02-12T17:18:16.701108+0000","last_became_peered":"2024-02-12T17:18:16.701108+0000","last_unstale":"2024-02-12T17:19:48.214899+0000","last_undegraded":"2024-02-12T17:19:48.214899+0000","last_fullsized":"2024-02-12T17:19:48.214899+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'45","last_scrub_stamp":"2024-02-12T17:18:59.770675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:18:59.770675+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":90,"num_read_kb":80,"num_write":60,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":804,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.829338+0000","last_change":"2024-02-12T17:18:49.163759+0000","last_active":"2024-02-12T17:19:35.829338+0000","last_peered":"2024-02-12T17:19:35.829338+0000","last_clean":"2024-02-12T17:19:35.829338+0000","last_became_active":"2024-02-12T17:04:50.393489+0000","last_became_peered":"2024-02-12T17:04:50.393489+0000","last_unstale":"2024-02-12T17:19:35.829338+0000","last_undegraded":"2024-02-12T17:19:35.829338+0000","last_fullsized":"2024-02-12T17:19:35.829338+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:49.163665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:54.135009+0000","last_clean_scrub_stamp":"2024-02-12T17:18:49.163665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":339,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.414298+0000","last_change":"2024-02-12T17:18:47.812083+0000","last_active":"2024-02-12T17:19:46.414298+0000","last_peered":"2024-02-12T17:19:46.414298+0000","last_clean":"2024-02-12T17:19:46.414298+0000","last_became_active":"2024-02-12T17:17:54.640651+0000","last_became_peered":"2024-02-12T17:17:54.640651+0000","last_unstale":"2024-02-12T17:19:46.414298+0000","last_undegraded":"2024-02-12T17:19:46.414298+0000","last_fullsized":"2024-02-12T17:19:46.414298+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:47.811986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:47.811986+0000","last_clean_scrub_stamp":"2024-02-12T17:18:47.811986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":475,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.414175+0000","last_change":"2024-02-12T17:19:23.733560+0000","last_active":"2024-02-12T17:19:46.414175+0000","last_peered":"2024-02-12T17:19:46.414175+0000","last_clean":"2024-02-12T17:19:46.414175+0000","last_became_active":"2024-02-12T17:17:54.641860+0000","last_became_peered":"2024-02-12T17:17:54.641860+0000","last_unstale":"2024-02-12T17:19:46.414175+0000","last_undegraded":"2024-02-12T17:19:46.414175+0000","last_fullsized":"2024-02-12T17:19:46.414175+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:23.733497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:21.817041+0000","last_clean_scrub_stamp":"2024-02-12T17:19:23.733497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"772'67","reported_seq":361,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.234888+0000","last_change":"2024-02-12T17:18:34.080719+0000","last_active":"2024-02-12T17:19:48.234888+0000","last_peered":"2024-02-12T17:19:48.234888+0000","last_clean":"2024-02-12T17:19:48.234888+0000","last_became_active":"2024-02-12T17:13:46.559481+0000","last_became_peered":"2024-02-12T17:13:46.559481+0000","last_unstale":"2024-02-12T17:19:48.234888+0000","last_undegraded":"2024-02-12T17:19:48.234888+0000","last_fullsized":"2024-02-12T17:19:48.234888+0000","mapping_epoch":630,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'51","last_scrub_stamp":"2024-02-12T17:18:34.080625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:18:34.080625+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":68,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":137,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314348+0000","last_change":"2024-02-12T17:18:42.569595+0000","last_active":"2024-02-12T17:19:35.314348+0000","last_peered":"2024-02-12T17:19:35.314348+0000","last_clean":"2024-02-12T17:19:35.314348+0000","last_became_active":"2024-02-12T17:14:08.859215+0000","last_became_peered":"2024-02-12T17:14:08.859215+0000","last_unstale":"2024-02-12T17:19:35.314348+0000","last_undegraded":"2024-02-12T17:19:35.314348+0000","last_fullsized":"2024-02-12T17:19:35.314348+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:42.569505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:06:43.014737+0000","last_clean_scrub_stamp":"2024-02-12T17:18:42.569505+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":612,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.413885+0000","last_change":"2024-02-12T17:18:52.828275+0000","last_active":"2024-02-12T17:19:46.413885+0000","last_peered":"2024-02-12T17:19:46.413885+0000","last_clean":"2024-02-12T17:19:46.413885+0000","last_became_active":"2024-02-12T17:17:54.582224+0000","last_became_peered":"2024-02-12T17:17:54.582224+0000","last_unstale":"2024-02-12T17:19:46.413885+0000","last_undegraded":"2024-02-12T17:19:46.413885+0000","last_fullsized":"2024-02-12T17:19:46.413885+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:52.828190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:42.168508+0000","last_clean_scrub_stamp":"2024-02-12T17:18:52.828190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"128'2","reported_seq":506,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.664878+0000","last_change":"2024-02-12T17:18:46.780145+0000","last_active":"2024-02-12T17:19:35.664878+0000","last_peered":"2024-02-12T17:19:35.664878+0000","last_clean":"2024-02-12T17:19:35.664878+0000","last_became_active":"2024-02-12T17:17:29.201863+0000","last_became_peered":"2024-02-12T17:17:29.201863+0000","last_unstale":"2024-02-12T17:19:35.664878+0000","last_undegraded":"2024-02-12T17:19:35.664878+0000","last_fullsized":"2024-02-12T17:19:35.664878+0000","mapping_epoch":730,"log_start":"128'2","ondisk_log_start":"128'2","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-02-12T17:18:46.780062+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-02-12T17:17:30.604582+0000","last_clean_scrub_stamp":"2024-02-12T17:18:46.780062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"526'21","reported_seq":185,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-02-12T17:19:35.314625+0000","last_change":"2024-02-12T17:19:00.493338+0000","last_active":"2024-02-12T17:19:35.314625+0000","last_peered":"2024-02-12T17:19:35.314625+0000","last_clean":"2024-02-12T17:19:35.314625+0000","last_became_active":"2024-02-12T17:17:26.199624+0000","last_became_peered":"2024-02-12T17:17:26.199624+0000","last_unstale":"2024-02-12T17:19:35.314625+0000","last_undegraded":"2024-02-12T17:19:35.314625+0000","last_fullsized":"2024-02-12T17:19:35.314625+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'21","last_scrub_stamp":"2024-02-12T17:19:00.493228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:40.156223+0000","last_clean_scrub_stamp":"2024-02-12T17:19:00.493228+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":787,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:45.341381+0000","last_change":"2024-02-12T17:19:02.041382+0000","last_active":"2024-02-12T17:19:45.341381+0000","last_peered":"2024-02-12T17:19:45.341381+0000","last_clean":"2024-02-12T17:19:45.341381+0000","last_became_active":"2024-02-12T17:09:21.531501+0000","last_became_peered":"2024-02-12T17:09:21.531501+0000","last_unstale":"2024-02-12T17:19:45.341381+0000","last_undegraded":"2024-02-12T17:19:45.341381+0000","last_fullsized":"2024-02-12T17:19:45.341381+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:19:02.041289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:19:02.041289+0000","last_clean_scrub_stamp":"2024-02-12T17:19:02.041289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":348,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-02-12T17:19:46.414138+0000","last_change":"2024-02-12T17:18:37.683393+0000","last_active":"2024-02-12T17:19:46.414138+0000","last_peered":"2024-02-12T17:19:46.414138+0000","last_clean":"2024-02-12T17:19:46.414138+0000","last_became_active":"2024-02-12T17:17:54.581523+0000","last_became_peered":"2024-02-12T17:17:54.581523+0000","last_unstale":"2024-02-12T17:19:46.414138+0000","last_undegraded":"2024-02-12T17:19:46.414138+0000","last_fullsized":"2024-02-12T17:19:46.414138+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T17:18:37.683281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T17:18:37.683281+0000","last_clean_scrub_stamp":"2024-02-12T17:18:37.683281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"772'58","reported_seq":923,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-02-12T17:19:48.174660+0000","last_change":"2024-02-12T17:18:47.722776+0000","last_active":"2024-02-12T17:19:48.174660+0000","last_peered":"2024-02-12T17:19:48.174660+0000","last_clean":"2024-02-12T17:19:48.174660+0000","last_became_active":"2024-02-12T17:09:21.532001+0000","last_became_peered":"2024-02-12T17:09:21.532001+0000","last_unstale":"2024-02-12T17:19:48.174660+0000","last_undegraded":"2024-02-12T17:19:48.174660+0000","last_fullsized":"2024-02-12T17:19:48.174660+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'46","last_scrub_stamp":"2024-02-12T17:18:47.722690+0000","last_deep_scrub":"526'46","last_deep_scrub_stamp":"2024-02-12T17:17:43.205604+0000","last_clean_scrub_stamp":"2024-02-12T17:18:47.722690+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":80,"num_read_kb":71,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"760'39","reported_seq":1355,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-02-12T17:19:37.730400+0000","last_change":"2024-02-12T17:18:40.135936+0000","last_active":"2024-02-12T17:19:37.730400+0000","last_peered":"2024-02-12T17:19:37.730400+0000","last_clean":"2024-02-12T17:19:37.730400+0000","last_became_active":"2024-02-12T17:18:29.902894+0000","last_became_peered":"2024-02-12T17:18:29.902894+0000","last_unstale":"2024-02-12T17:19:37.730400+0000","last_undegraded":"2024-02-12T17:19:37.730400+0000","last_fullsized":"2024-02-12T17:19:37.730400+0000","mapping_epoch":757,"log_start":"65'8","ondisk_log_start":"65'8","created":61,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"760'39","last_scrub_stamp":"2024-02-12T17:18:40.135838+0000","last_deep_scrub":"726'36","last_deep_scrub_stamp":"2024-02-12T17:17:22.297773+0000","last_clean_scrub_stamp":"2024-02-12T17:18:40.135838+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":570,"num_read_kb":565,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1719,"num_read_kb":1476,"num_write":1129,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":17690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},"log_size":1204,"ondisk_log_size":1204,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":22,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":66,"acting":66,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234349,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1495880,"kb_used_data":3712,"kb_used_omap":0,"kb_used_meta":1492160,"kb_avail":92249272,"statfs":{"total":95995035648,"available":94463254528,"internally_reserved":0,"allocated":3801088,"data_stored":2209767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1527971840},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":1.3380000000000001,"15min":1.698},"min":{"1min":0.39400000000000002,"5min":0.16900000000000001,"15min":0.16700000000000001},"max":{"1min":0.69499999999999995,"5min":15.856999999999999,"15min":15.856999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.499,"5min":1.3859999999999999,"15min":1.569},"min":{"1min":0.38600000000000001,"5min":0.23599999999999999,"15min":0.22},"max":{"1min":0.65700000000000003,"5min":12.565,"15min":12.565},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":1.7749999999999999,"15min":2.4510000000000001},"min":{"1min":0.311,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.88600000000000001,"5min":11.632999999999999,"15min":11.632999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.849,"15min":2.2610000000000001},"min":{"1min":0.42899999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.86799999999999999,"5min":10.91,"15min":10.91},"last":0.76800000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.512,"15min":2.879},"min":{"1min":0.44,"5min":0.24399999999999999,"15min":0.22800000000000001},"max":{"1min":0.67900000000000005,"5min":23.611000000000001,"15min":218.35300000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.742,"15min":3.0939999999999999},"min":{"1min":0.46300000000000002,"5min":0.372,"15min":0.247},"max":{"1min":0.67100000000000004,"5min":15.672000000000001,"15min":218.42099999999999},"last":0.72099999999999997}]},{"osd":3,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.907,"15min":1.9710000000000001},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.255},"max":{"1min":0.746,"5min":11.741,"15min":15.811999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":2.1680000000000001,"15min":2.4049999999999998},"min":{"1min":0.38400000000000001,"5min":0.28999999999999998,"15min":0.28399999999999997},"max":{"1min":0.69799999999999995,"5min":12.375999999999999,"15min":13.699},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57999999999999996,"15min":1.2549999999999999},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23300000000000001},"max":{"1min":0.79700000000000004,"5min":3.4969999999999999,"15min":20.672999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.74399999999999999,"15min":1.5960000000000001},"min":{"1min":0.317,"5min":0.23300000000000001,"15min":0.19600000000000001},"max":{"1min":0.65900000000000003,"5min":6.6769999999999996,"15min":38.390000000000001},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.69599999999999995,"15min":1.6850000000000001},"min":{"1min":0.35199999999999998,"5min":0.23999999999999999,"15min":0.22500000000000001},"max":{"1min":0.82699999999999996,"5min":4.79,"15min":56.560000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.99399999999999999,"15min":1.6160000000000001},"min":{"1min":0.376,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.73999999999999999,"5min":9.7170000000000005,"15min":53.963999999999999},"last":0.80100000000000005}]},{"osd":6,"last update":"Mon Feb 12 17:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.71899999999999997,"15min":1.2030000000000001},"min":{"1min":0.374,"5min":0.25700000000000001,"15min":0.223},"max":{"1min":0.73199999999999998,"5min":5.2539999999999996,"15min":21.620000000000001},"last":0.628},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":1.3819999999999999},"min":{"1min":0.44400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.85099999999999998,"5min":3.7519999999999998,"15min":18.071000000000002},"last":0.73699999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528691,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896084,"kb_used_data":3724,"kb_used_omap":0,"kb_used_meta":892352,"kb_avail":92849068,"statfs":{"total":95995035648,"available":95077445632,"internally_reserved":0,"allocated":3813376,"data_stored":2222879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913768448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.4590000000000001,"15min":1.7949999999999999},"min":{"1min":0.34499999999999997,"5min":0.22,"15min":0.20399999999999999},"max":{"1min":0.70599999999999996,"5min":11.115,"15min":33.576999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.1120000000000001,"15min":1.7549999999999999},"min":{"1min":0.34799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.73699999999999999,"5min":6.1100000000000003,"15min":33.569000000000003},"last":0.498}]},{"osd":1,"last update":"Mon Feb 12 17:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.661,"15min":1.948},"min":{"1min":0.41699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.749,"5min":20.611000000000001,"15min":20.611000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":1.6080000000000001,"15min":1.998},"min":{"1min":0.441,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":20.622,"15min":20.622},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":1.78,"15min":2.3740000000000001},"min":{"1min":0.312,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.71099999999999997,"5min":12.311,"15min":33.594999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":2.0139999999999998,"15min":2.1040000000000001},"min":{"1min":0.36499999999999999,"5min":0.27600000000000002,"15min":0.24099999999999999},"max":{"1min":0.72499999999999998,"5min":14.045999999999999,"15min":33.526000000000003},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":2.1589999999999998,"15min":2.5619999999999998},"min":{"1min":0.38100000000000001,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":0.68600000000000005,"5min":25.867999999999999,"15min":58.420999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":2.254,"15min":2.7879999999999998},"min":{"1min":0.38400000000000001,"5min":0.33200000000000002,"15min":0.254},"max":{"1min":0.77400000000000002,"5min":14.69,"15min":58.709000000000003},"last":0.40500000000000003}]},{"osd":4,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.78800000000000003,"15min":1.1859999999999999},"min":{"1min":0.28100000000000003,"5min":0.26100000000000001,"15min":0.23499999999999999},"max":{"1min":0.66600000000000004,"5min":11.012,"15min":33.706000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.76800000000000002,"15min":1.389},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.222},"max":{"1min":0.69699999999999995,"5min":10.948,"15min":34.895000000000003},"last":0.47699999999999998}]},{"osd":5,"last update":"Mon Feb 12 17:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.91200000000000003,"15min":1.4350000000000001},"min":{"1min":0.30599999999999999,"5min":0.27600000000000002,"15min":0.23799999999999999},"max":{"1min":0.70899999999999996,"5min":10.77,"15min":34.869},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.93700000000000006,"15min":1.3839999999999999},"min":{"1min":0.33100000000000002,"5min":0.30099999999999999,"15min":0.22900000000000001},"max":{"1min":0.72399999999999998,"5min":11.1,"15min":33.591000000000001},"last":0.39500000000000002}]},{"osd":7,"last update":"Mon Feb 12 17:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.76300000000000001,"15min":1.468},"min":{"1min":0.32500000000000001,"5min":0.26500000000000001,"15min":0.255},"max":{"1min":0.76600000000000001,"5min":6.1310000000000002,"15min":35.122999999999998},"last":0.433},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.90900000000000003,"15min":1.2889999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.26500000000000001},"max":{"1min":0.77200000000000002,"5min":11.081,"15min":34.045000000000002},"last":0.57399999999999995}]}]},{"osd":1,"up_from":630,"seq":2705829396554,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338264,"kb_used_data":3728,"kb_used_omap":3,"kb_used_meta":334524,"kb_avail":93406888,"statfs":{"total":95995035648,"available":95648653312,"internally_reserved":0,"allocated":3817472,"data_stored":2210412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4035,"internal_metadata":342552637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":2.0630000000000002,"15min":3.0110000000000001},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.63900000000000001,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":3.423,"15min":3.4209999999999998},"min":{"1min":0.254,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65700000000000003,"5min":27.268999999999998,"15min":27.268999999999998},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":1.992,"15min":3.052},"min":{"1min":0.26600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54800000000000004,"5min":12.856,"15min":12.856},"last":0.438},{"interface":"front","average":{"1min":0.373,"5min":2.5990000000000002,"15min":3.1970000000000001},"min":{"1min":0.25700000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":24.236999999999998,"15min":24.236999999999998},"last":0.29499999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":3.218,"15min":3.5009999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":31.292999999999999,"15min":31.292999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":3.2970000000000002,"15min":3.98},"min":{"1min":0.33100000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":23.905999999999999,"15min":23.905999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":1.5760000000000001,"15min":2.2280000000000002},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84699999999999998,"5min":12.901999999999999,"15min":12.901999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.9350000000000001,"15min":2.4990000000000001},"min":{"1min":0.38900000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.76700000000000002,"5min":12.805999999999999,"15min":12.805999999999999},"last":0.628}]},{"osd":5,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":2.0289999999999999,"15min":2.8199999999999998},"min":{"1min":0.437,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81200000000000006,"5min":12.769,"15min":12.769},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.133,"15min":2.1190000000000002},"min":{"1min":0.45300000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90900000000000003,"5min":23.300999999999998,"15min":23.300999999999998},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":2.2679999999999998,"15min":2.6419999999999999},"min":{"1min":0.42099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.879,"5min":23.347000000000001,"15min":23.347000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.629,"15min":2.4609999999999999},"min":{"1min":0.44500000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80900000000000005,"5min":12.929,"15min":12.929},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Feb 12 17:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":1.9710000000000001,"15min":2.633},"min":{"1min":0.40699999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.88200000000000001,"5min":12.829000000000001,"15min":12.829000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":2.298,"15min":2.3599999999999999},"min":{"1min":0.48199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.94099999999999995,"5min":23.376000000000001,"15min":23.376000000000001},"last":0.59899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738767,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908104,"kb_used_data":3712,"kb_used_omap":0,"kb_used_meta":904384,"kb_avail":92837048,"statfs":{"total":95995035648,"available":95065137152,"internally_reserved":0,"allocated":3801088,"data_stored":2209820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":926089216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 17:18:50 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":2.448,"15min":3.1019999999999999},"min":{"1min":0.24199999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.626,"5min":14.048999999999999,"15min":14.048999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":1.7569999999999999,"15min":2.4529999999999998},"min":{"1min":0.27800000000000002,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53400000000000003,"5min":13.787000000000001,"15min":13.787000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Feb 12 17:18:43 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":2.3370000000000002,"15min":2.319},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.19400000000000001},"max":{"1min":0.53500000000000003,"5min":18.204000000000001,"15min":19.312000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":2.3719999999999999,"15min":2.7650000000000001},"min":{"1min":0.245,"5min":0.217,"15min":0.20799999999999999},"max":{"1min":0.64500000000000002,"5min":22.959,"15min":26.039999999999999},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Feb 12 17:18:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":3.7040000000000002,"15min":3.262},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.186},"max":{"1min":0.59899999999999998,"5min":61.177,"15min":87.311000000000007},"last":0.307},{"interface":"front","average":{"1min":0.44,"5min":3.778,"15min":3.4390000000000001},"min":{"1min":0.25,"5min":0.22900000000000001,"15min":0.221},"max":{"1min":0.64100000000000001,"5min":59.149999999999999,"15min":90.991},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Feb 12 17:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":1.377,"15min":1.387},"min":{"1min":0.41399999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.89300000000000002,"5min":13.861000000000001,"15min":15.141999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.4910000000000001,"15min":1.964},"min":{"1min":0.438,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":0.90100000000000002,"5min":13.503,"15min":26.082999999999998},"last":0.70399999999999996}]},{"osd":5,"last update":"Mon Feb 12 17:18:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6679999999999999,"15min":1.9850000000000001},"min":{"1min":0.38300000000000001,"5min":0.28599999999999998,"15min":0.28000000000000003},"max":{"1min":0.85999999999999999,"5min":13.981999999999999,"15min":17.754000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.6819999999999999,"15min":1.857},"min":{"1min":0.41599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.81999999999999995,"5min":13.837,"15min":19.751999999999999},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":1.702,"15min":2.3570000000000002},"min":{"1min":0.44700000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84399999999999997,"5min":14.686999999999999,"15min":48.780999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.698,"15min":2.1840000000000002},"min":{"1min":0.498,"5min":0.36399999999999999,"15min":0.31900000000000001},"max":{"1min":0.875,"5min":13.962999999999999,"15min":49.067999999999998},"last":0.65500000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:18:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.9079999999999999,"15min":2.0270000000000001},"min":{"1min":0.52200000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.88100000000000001,"5min":13.813000000000001,"15min":19.373000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.9350000000000001,"15min":2.2949999999999999},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82799999999999996,"5min":13.542,"15min":26.135000000000002},"last":0.66300000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411717,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506012,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":1502272,"kb_avail":92239140,"statfs":{"total":95995035648,"available":94452879360,"internally_reserved":0,"allocated":3821568,"data_stored":2222714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1538326528},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":1.5109999999999999,"15min":1.8240000000000001},"min":{"1min":0.249,"5min":0.218,"15min":0.17000000000000001},"max":{"1min":0.76200000000000001,"5min":11.663,"15min":21.445},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":2.452,"15min":2.2789999999999999},"min":{"1min":0.251,"5min":0.224,"15min":0.19900000000000001},"max":{"1min":0.58399999999999996,"5min":26.852,"15min":26.852},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Feb 12 17:18:53 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":1.9550000000000001,"15min":3.3290000000000002},"min":{"1min":0.27100000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.63800000000000001,"5min":11.612,"15min":11.612},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":3.056,"15min":4.9480000000000004},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.83699999999999997,"5min":22.52,"15min":22.52},"last":0.29299999999999998}]},{"osd":3,"last update":"Mon Feb 12 17:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":4.0199999999999996,"15min":3.0870000000000002},"min":{"1min":0.34899999999999998,"5min":0.27100000000000002,"15min":0.20200000000000001},"max":{"1min":0.81799999999999995,"5min":37.704999999999998,"15min":40.759999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":2.9670000000000001,"15min":2.5990000000000002},"min":{"1min":0.32000000000000001,"5min":0.221,"15min":0.22},"max":{"1min":0.69699999999999995,"5min":30.285,"15min":30.285},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Feb 12 17:19:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":1.409,"15min":1.756},"min":{"1min":0.503,"5min":0.27600000000000002,"15min":0.245},"max":{"1min":0.91700000000000004,"5min":12.282999999999999,"15min":21.515000000000001},"last":0.751},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.5089999999999999,"15min":1.766},"min":{"1min":0.495,"5min":0.25700000000000001,"15min":0.25600000000000001},"max":{"1min":0.95599999999999996,"5min":12.597,"15min":20.149000000000001},"last":0.76200000000000001}]},{"osd":5,"last update":"Mon Feb 12 17:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.403,"15min":1.77},"min":{"1min":0.47299999999999998,"5min":0.33900000000000002,"15min":0.217},"max":{"1min":0.95899999999999996,"5min":11.581,"15min":21.542999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":1.2929999999999999,"15min":1.8300000000000001},"min":{"1min":0.44900000000000001,"5min":0.28699999999999998,"15min":0.255},"max":{"1min":1.0129999999999999,"5min":8.3070000000000004,"15min":20.937000000000001},"last":0.72699999999999998}]},{"osd":6,"last update":"Mon Feb 12 17:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.6279999999999999,"15min":1.8220000000000001},"min":{"1min":0.45400000000000001,"5min":0.29199999999999998,"15min":0.27500000000000002},"max":{"1min":0.86799999999999999,"5min":10.714,"15min":20.966999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.774,"15min":1.8009999999999999},"min":{"1min":0.47599999999999998,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.97799999999999998,"5min":12.631,"15min":20.193000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.5549999999999999,"15min":2.012},"min":{"1min":0.42799999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.94599999999999995,"5min":12.67,"15min":20.221},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.29,"15min":1.9219999999999999},"min":{"1min":0.47999999999999998,"5min":0.33000000000000002,"15min":0.30199999999999999},"max":{"1min":0.95699999999999996,"5min":8.3230000000000004,"15min":20.995999999999999},"last":0.70299999999999996}]}]},{"osd":3,"up_from":26,"seq":111669150079,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896992,"kb_used_data":3736,"kb_used_omap":0,"kb_used_meta":893248,"kb_avail":92848160,"statfs":{"total":95995035648,"available":95076515840,"internally_reserved":0,"allocated":3825664,"data_stored":2217507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914685952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":2.8879999999999999,"15min":2.8010000000000002},"min":{"1min":0.24199999999999999,"5min":0.20999999999999999,"15min":0.17100000000000001},"max":{"1min":0.49199999999999999,"5min":22.16,"15min":22.16},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":3.3690000000000002,"15min":3.1520000000000001},"min":{"1min":0.28499999999999998,"5min":0.23599999999999999,"15min":0.19500000000000001},"max":{"1min":0.58899999999999997,"5min":42.984999999999999,"15min":72.688000000000002},"last":0.436}]},{"osd":1,"last update":"Mon Feb 12 17:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":3.403,"15min":4.1210000000000004},"min":{"1min":0.25700000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69599999999999995,"5min":22.524000000000001,"15min":22.524000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":3.476,"15min":4.7949999999999999},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.626,"5min":20.710000000000001,"15min":20.710000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Mon Feb 12 17:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":3.4649999999999999,"15min":2.7970000000000002},"min":{"1min":0.245,"5min":0.186,"15min":0.186},"max":{"1min":0.65200000000000002,"5min":48.127000000000002,"15min":48.127000000000002},"last":0.503},{"interface":"front","average":{"1min":0.41099999999999998,"5min":3.4159999999999999,"15min":3.274},"min":{"1min":0.29199999999999998,"5min":0.27100000000000002,"15min":0.22},"max":{"1min":0.51900000000000002,"5min":40.600999999999999,"15min":46.408000000000001},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 12 17:19:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":2.4790000000000001,"15min":2.5819999999999999},"min":{"1min":0.432,"5min":0.309,"15min":0.29599999999999999},"max":{"1min":0.80400000000000005,"5min":20.478000000000002,"15min":72.659000000000006},"last":0.626},{"interface":"front","average":{"1min":0.63900000000000001,"5min":2.0150000000000001,"15min":2.0169999999999999},"min":{"1min":0.42299999999999999,"5min":0.25800000000000001,"15min":0.23999999999999999},"max":{"1min":0.88200000000000001,"5min":22.440999999999999,"15min":40.930999999999997},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Feb 12 17:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":2.3580000000000001,"15min":2.444},"min":{"1min":0.45200000000000001,"5min":0.314,"15min":0.253},"max":{"1min":0.83599999999999997,"5min":17.306999999999999,"15min":17.940999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":2.097,"15min":2.6880000000000002},"min":{"1min":0.44400000000000001,"5min":0.35099999999999998,"15min":0.29799999999999999},"max":{"1min":0.90600000000000003,"5min":15.522,"15min":72.620000000000005},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Feb 12 17:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":2.145,"15min":2.125},"min":{"1min":0.45500000000000002,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":0.85599999999999998,"5min":22.489999999999998,"15min":40.889000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":2.379,"15min":2.4300000000000002},"min":{"1min":0.53400000000000003,"5min":0.36499999999999999,"15min":0.32600000000000001},"max":{"1min":0.93100000000000005,"5min":15.113,"15min":18.265000000000001},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Feb 12 17:19:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":2.3730000000000002,"15min":2.8999999999999999},"min":{"1min":0.45000000000000001,"5min":0.378,"15min":0.315},"max":{"1min":0.89100000000000001,"5min":20.515999999999998,"15min":72.712999999999994},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":2.3809999999999998,"15min":2.7090000000000001},"min":{"1min":0.53500000000000003,"5min":0.39900000000000002,"15min":0.28299999999999997},"max":{"1min":0.89800000000000002,"5min":15.169,"15min":38.948},"last":0.55700000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953853,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907556,"kb_used_data":3740,"kb_used_omap":0,"kb_used_meta":903808,"kb_avail":92837596,"statfs":{"total":95995035648,"available":95065698304,"internally_reserved":0,"allocated":3829760,"data_stored":2223513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":925499392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":1.0920000000000001,"15min":1.5900000000000001},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":0.62,"5min":5.7380000000000004,"15min":25.829000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.456,"15min":1.742},"min":{"1min":0.40400000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.68700000000000006,"5min":11.659000000000001,"15min":22.286000000000001},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Feb 12 17:18:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.544,"15min":1.7969999999999999},"min":{"1min":0.40400000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.78100000000000003,"5min":11.319000000000001,"15min":11.319000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.5660000000000001,"15min":2.5800000000000001},"min":{"1min":0.48799999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71799999999999997,"5min":11.483000000000001,"15min":11.483000000000001},"last":0.436}]},{"osd":2,"last update":"Mon Feb 12 17:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":1.4650000000000001,"15min":3.1030000000000002},"min":{"1min":0.40400000000000003,"5min":0.23400000000000001,"15min":0.23300000000000001},"max":{"1min":0.70399999999999996,"5min":11.395,"15min":191.05600000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.622,"5min":1.5589999999999999,"15min":3.0030000000000001},"min":{"1min":0.50800000000000001,"5min":0.29699999999999999,"15min":0.20599999999999999},"max":{"1min":0.76800000000000002,"5min":11.298,"15min":191.083},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Feb 12 17:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":2.0920000000000001,"15min":3.161},"min":{"1min":0.42499999999999999,"5min":0.26500000000000001,"15min":0.25600000000000001},"max":{"1min":0.83999999999999997,"5min":19.638999999999999,"15min":248.05500000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":1.75,"15min":2.9900000000000002},"min":{"1min":0.38700000000000001,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":0.69099999999999995,"5min":10.191000000000001,"15min":249.20599999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Mon Feb 12 17:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.75900000000000001,"15min":1.26},"min":{"1min":0.28199999999999997,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":0.63200000000000001,"5min":5.085,"15min":26.510000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.82799999999999996,"15min":1.3740000000000001},"min":{"1min":0.30399999999999999,"5min":0.24099999999999999,"15min":0.219},"max":{"1min":0.61799999999999999,"5min":6.032,"15min":23.986999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 12 17:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.78800000000000003,"15min":0.96899999999999997},"min":{"1min":0.25900000000000001,"5min":0.255,"15min":0.246},"max":{"1min":0.748,"5min":5.4169999999999998,"15min":11.361000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.86799999999999999,"15min":1.155},"min":{"1min":0.33000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64700000000000002,"5min":11.332000000000001,"15min":11.332000000000001},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Feb 12 17:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.78400000000000003,"15min":1.3140000000000001},"min":{"1min":0.27100000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73899999999999999,"5min":7.6609999999999996,"15min":20.696000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":1.0209999999999999,"15min":1.2549999999999999},"min":{"1min":0.317,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.075,"5min":6.8689999999999998,"15min":10.99},"last":0.64100000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757625,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1493144,"kb_used_data":3728,"kb_used_omap":0,"kb_used_meta":1489408,"kb_avail":92252008,"statfs":{"total":95995035648,"available":94466056192,"internally_reserved":0,"allocated":3817472,"data_stored":2222763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1525153792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":1.3360000000000001,"15min":1.53},"min":{"1min":0.218,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.70699999999999996,"5min":6.8399999999999999,"15min":10.971},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.5,"5min":1.468,"15min":1.6160000000000001},"min":{"1min":0.23100000000000001,"5min":0.22900000000000001,"15min":0.20100000000000001},"max":{"1min":0.64300000000000002,"5min":12.061,"15min":15.368},"last":0.77800000000000002}]},{"osd":1,"last update":"Mon Feb 12 17:19:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":1.613,"15min":1.736},"min":{"1min":0.26900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.748,"5min":9.484,"15min":9.484},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.9239999999999999,"15min":2.6840000000000002},"min":{"1min":0.26200000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71899999999999997,"5min":18.550000000000001,"15min":18.550000000000001},"last":0.79900000000000004}]},{"osd":2,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.708,"15min":1.9810000000000001},"min":{"1min":0.312,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":0.72399999999999998,"5min":13.978,"15min":57.390999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.53400000000000003,"5min":1.3700000000000001,"15min":1.7789999999999999},"min":{"1min":0.37,"5min":0.32100000000000001,"15min":0.214},"max":{"1min":0.67600000000000005,"5min":13.366,"15min":57.134999999999998},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.655,"15min":2.6619999999999999},"min":{"1min":0.36199999999999999,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":0.745,"5min":11.871,"15min":57.170999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.9079999999999999,"15min":2.492},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.23799999999999999},"max":{"1min":0.76200000000000001,"5min":17.760999999999999,"15min":57.484999999999999},"last":0.751}]},{"osd":4,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.79800000000000004,"15min":1.085},"min":{"1min":0.31,"5min":0.252,"15min":0.23599999999999999},"max":{"1min":0.69499999999999995,"5min":5.9939999999999998,"15min":12.656000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.80600000000000005,"15min":1.1479999999999999},"min":{"1min":0.33900000000000002,"5min":0.255,"15min":0.254},"max":{"1min":0.69599999999999995,"5min":5.1390000000000002,"15min":11.068},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Feb 12 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.83299999999999996,"15min":1.1619999999999999},"min":{"1min":0.30299999999999999,"5min":0.28899999999999998,"15min":0.23899999999999999},"max":{"1min":0.67900000000000005,"5min":6.4109999999999996,"15min":14.675000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.91900000000000004,"15min":1.2969999999999999},"min":{"1min":0.39600000000000002,"5min":0.27900000000000003,"15min":0.245},"max":{"1min":0.77400000000000002,"5min":5.2720000000000002,"15min":12.864000000000001},"last":0.67400000000000004}]},{"osd":7,"last update":"Mon Feb 12 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.82699999999999996,"15min":1.24},"min":{"1min":0.36099999999999999,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.64200000000000002,"5min":6.2960000000000003,"15min":14.337999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.98299999999999998,"15min":1.3680000000000001},"min":{"1min":0.39300000000000002,"5min":0.30199999999999999,"15min":0.27800000000000002},"max":{"1min":0.70299999999999996,"5min":6.6950000000000003,"15min":20.274000000000001},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3063,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T17:19:49.802 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T17:19:49.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-12T17:19:50.083 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:49.088816+0000 osd.6 (osd.6) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:49.088860+0000 osd.6 (osd.6) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:49.114246+0000 mgr.y (mgr.24481) 1134 : audit [DBG] from='client.36542 v1:172.21.15.33:0/1651421089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:49.199989+0000 mgr.y (mgr.24481) 1135 : cluster [DBG] pgmap v1600: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:49.681959+0000 osd.1 (osd.1) 92 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:49.684137+0000 osd.1 (osd.1) 93 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:49.868954+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:49.979347+0000 osd.1 (osd.1) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:50.740 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:49.979421+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:49.088816+0000 osd.6 (osd.6) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:49.088860+0000 osd.6 (osd.6) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:49.114246+0000 mgr.y (mgr.24481) 1134 : audit [DBG] from='client.36542 v1:172.21.15.33:0/1651421089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:49.199989+0000 mgr.y (mgr.24481) 1135 : cluster [DBG] pgmap v1600: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:49.681959+0000 osd.1 (osd.1) 92 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:49.684137+0000 osd.1 (osd.1) 93 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:49.868954+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:49.979347+0000 osd.1 (osd.1) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:49.979421+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:49.088816+0000 osd.6 (osd.6) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:50.777 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:49.088860+0000 osd.6 (osd.6) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:49.114246+0000 mgr.y (mgr.24481) 1134 : audit [DBG] from='client.36542 v1:172.21.15.33:0/1651421089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T17:19:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:49.199989+0000 mgr.y (mgr.24481) 1135 : cluster [DBG] pgmap v1600: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-02-12T17:19:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:49.681959+0000 osd.1 (osd.1) 92 : cluster [DBG] 4.4 scrub starts 2024-02-12T17:19:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:49.684137+0000 osd.1 (osd.1) 93 : cluster [DBG] 4.4 scrub ok 2024-02-12T17:19:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:49.868954+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-12T17:19:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:49.979347+0000 osd.1 (osd.1) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-12T17:19:50.778 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:49.979421+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-12T17:19:51.276 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:50 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:50] "GET /metrics HTTP/1.1" 200 174453 "" "Prometheus/2.33.4" 2024-02-12T17:19:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:50.746988+0000 osd.2 (osd.2) 275 : cluster [DBG] 6.f scrub starts 2024-02-12T17:19:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:50.872342+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-12T17:19:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:50.939994+0000 osd.2 (osd.2) 276 : cluster [DBG] 6.f scrub ok 2024-02-12T17:19:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:51.124778+0000 osd.6 (osd.6) 230 : cluster [DBG] 4.a scrub starts 2024-02-12T17:19:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:51.126941+0000 osd.6 (osd.6) 231 : cluster [DBG] 4.a scrub ok 2024-02-12T17:19:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:51.200389+0000 mgr.y (mgr.24481) 1136 : cluster [DBG] pgmap v1603: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-02-12T17:19:52.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:51.858954+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-12T17:19:52.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:50.746988+0000 osd.2 (osd.2) 275 : cluster [DBG] 6.f scrub starts 2024-02-12T17:19:52.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:50.872342+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-12T17:19:52.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:50.939994+0000 osd.2 (osd.2) 276 : cluster [DBG] 6.f scrub ok 2024-02-12T17:19:52.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:51.124778+0000 osd.6 (osd.6) 230 : cluster [DBG] 4.a scrub starts 2024-02-12T17:19:52.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:51.126941+0000 osd.6 (osd.6) 231 : cluster [DBG] 4.a scrub ok 2024-02-12T17:19:52.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:51.200389+0000 mgr.y (mgr.24481) 1136 : cluster [DBG] pgmap v1603: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-02-12T17:19:52.029 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:51 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:51.858954+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-12T17:19:52.239 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:50.746988+0000 osd.2 (osd.2) 275 : cluster [DBG] 6.f scrub starts 2024-02-12T17:19:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:50.872342+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-12T17:19:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:50.939994+0000 osd.2 (osd.2) 276 : cluster [DBG] 6.f scrub ok 2024-02-12T17:19:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:51.124778+0000 osd.6 (osd.6) 230 : cluster [DBG] 4.a scrub starts 2024-02-12T17:19:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:51.126941+0000 osd.6 (osd.6) 231 : cluster [DBG] 4.a scrub ok 2024-02-12T17:19:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:51.200389+0000 mgr.y (mgr.24481) 1136 : cluster [DBG] pgmap v1603: 102 pgs: 102 active+clean; 5.7 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-02-12T17:19:52.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:51.858954+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-12T17:19:52.527 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:52] ENGINE Bus STOPPING 2024-02-12T17:19:52.527 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-12T17:19:52.527 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:52] ENGINE Bus STOPPED 2024-02-12T17:19:52.527 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:52] ENGINE Bus STARTING 2024-02-12T17:19:52.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:52] ENGINE Serving on http://:::9283 2024-02-12T17:19:52.528 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:52] ENGINE Bus STARTED 2024-02-12T17:19:52.844 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-12T17:19:52.915 INFO:tasks.cephadm:Teardown begin 2024-02-12T17:19:52.916 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T17:19:52.917 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x[42318]: ::ffff:172.21.15.94 - - [12/Feb/2024:17:19:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-12T17:19:52.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T17:19:53.055 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-12T17:19:53.055 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-12T17:19:53.076 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-12T17:19:53.094 INFO:tasks.cephadm:Stopping all daemons... 2024-02-12T17:19:53.095 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-12T17:19:53.095 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.a 2024-02-12T17:19:53.144 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:51.781894+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.1 deep-scrub starts 2024-02-12T17:19:53.144 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:51.782958+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.1 deep-scrub ok 2024-02-12T17:19:53.144 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:51.868172+0000 osd.4 (osd.4) 253 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:19:53.144 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: cluster 2024-02-12T17:19:51.870063+0000 osd.4 (osd.4) 254 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:19:53.144 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:52.182904+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.33:0/883561928' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-12T17:19:53.145 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:52.183176+0000 mon.a (mon.0) 2430 : audit [INF] from='client.36548 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-12T17:19:53.145 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:52.188372+0000 mon.a (mon.0) 2431 : audit [INF] from='client.36548 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-12T17:19:53.145 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: audit 2024-02-12T17:19:52.228909+0000 mon.c (mon.2) 350 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:19:53.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:51.781894+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.1 deep-scrub starts 2024-02-12T17:19:53.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:51.782958+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.1 deep-scrub ok 2024-02-12T17:19:53.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:51.868172+0000 osd.4 (osd.4) 253 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:19:53.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: cluster 2024-02-12T17:19:51.870063+0000 osd.4 (osd.4) 254 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:19:53.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:52.182904+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.33:0/883561928' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-12T17:19:53.146 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:52.183176+0000 mon.a (mon.0) 2430 : audit [INF] from='client.36548 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-12T17:19:53.146 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:52.188372+0000 mon.a (mon.0) 2431 : audit [INF] from='client.36548 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-12T17:19:53.146 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:52 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: audit 2024-02-12T17:19:52.228909+0000 mon.c (mon.2) 350 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:19:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:51.781894+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.1 deep-scrub starts 2024-02-12T17:19:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:51.782958+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.1 deep-scrub ok 2024-02-12T17:19:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:51.868172+0000 osd.4 (osd.4) 253 : cluster [DBG] 4.18 scrub starts 2024-02-12T17:19:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: cluster 2024-02-12T17:19:51.870063+0000 osd.4 (osd.4) 254 : cluster [DBG] 4.18 scrub ok 2024-02-12T17:19:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:52.182904+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.33:0/883561928' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-12T17:19:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:52.183176+0000 mon.a (mon.0) 2430 : audit [INF] from='client.36548 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-12T17:19:53.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:52.188372+0000 mon.a (mon.0) 2431 : audit [INF] from='client.36548 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-12T17:19:53.241 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: audit 2024-02-12T17:19:52.228909+0000 mon.c (mon.2) 350 : audit [DBG] from='mgr.24481 v1:172.21.15.33:0/1674112449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T17:19:53.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:53 smithi033 systemd[1]: Stopping Ceph mon.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:19:53.469 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:53 smithi033 bash[225379]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon.a" found: no such container 2024-02-12T17:19:53.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:19:53.467+0000 7fd9711eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T17:19:53.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:53 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a[39521]: debug 2024-02-12T17:19:53.467+0000 7fd9711eb700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-12T17:19:54.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Feb 12 17:19:54 smithi033 bash[225418]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-a 2024-02-12T17:19:54.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.a.service' 2024-02-12T17:19:54.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:19:54.688 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-12T17:19:54.688 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-12T17:19:54.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.c 2024-02-12T17:19:55.007 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:54 smithi033 systemd[1]: Stopping Ceph mon.c for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:19:55.007 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:54 smithi033 bash[225536]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon.c" found: no such container 2024-02-12T17:19:55.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T17:19:55.006+0000 7f823eeaa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T17:19:55.279 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c[46083]: debug 2024-02-12T17:19:55.006+0000 7f823eeaa700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-12T17:19:55.279 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:55] ENGINE Bus STOPPING 2024-02-12T17:19:55.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-12T17:19:55.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:55] ENGINE Bus STOPPED 2024-02-12T17:19:55.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:55] ENGINE Bus STARTING 2024-02-12T17:19:55.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:55] ENGINE Serving on http://:::9283 2024-02-12T17:19:55.280 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:55 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: [12/Feb/2024:17:19:55] ENGINE Bus STARTED 2024-02-12T17:19:55.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:19:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug there is no tcmu-runner data available 2024-02-12T17:19:56.027 INFO:journalctl@ceph.mon.c.smithi033.stdout:Feb 12 17:19:55 smithi033 bash[225576]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-c 2024-02-12T17:19:56.157 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.c.service' 2024-02-12T17:19:56.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:19:56.196 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-12T17:19:56.196 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-12T17:19:56.196 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.b 2024-02-12T17:19:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:56 smithi094 systemd[1]: Stopping Ceph mon.b for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:19:56.490 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:56 smithi094 bash[136191]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon.b" found: no such container 2024-02-12T17:19:56.830 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T17:19:56.504+0000 7f95c6379700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T17:19:56.830 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b[40856]: debug 2024-02-12T17:19:56.504+0000 7f95c6379700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-12T17:19:57.240 INFO:journalctl@ceph.mon.b.smithi094.stdout:Feb 12 17:19:57 smithi094 bash[136242]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mon-b 2024-02-12T17:19:57.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mon.b.service' 2024-02-12T17:19:57.472 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:19:57.472 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-12T17:19:57.472 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-12T17:19:57.473 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.y 2024-02-12T17:19:57.775 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:57 smithi033 systemd[1]: Stopping Ceph mgr.y for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:19:57.775 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:57 smithi033 bash[225706]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr.y" found: no such container 2024-02-12T17:19:58.026 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T17:19:57.773+0000 7f12668fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:19:58.027 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:57 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y[39796]: debug 2024-02-12T17:19:57.773+0000 7f12668fe700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-12T17:19:58.526 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Feb 12 17:19:58 smithi033 bash[225746]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-y 2024-02-12T17:19:58.711 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.y.service' 2024-02-12T17:19:58.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:19:58.750 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-12T17:19:58.750 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-12T17:19:58.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.x 2024-02-12T17:19:59.240 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:58 smithi094 systemd[1]: Stopping Ceph mgr.x for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:19:59.240 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:58 smithi094 bash[136372]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr.x" found: no such container 2024-02-12T17:19:59.739 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Feb 12 17:19:59 smithi094 bash[136412]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-mgr-x 2024-02-12T17:19:59.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@mgr.x.service' 2024-02-12T17:19:59.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:19:59.979 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-12T17:19:59.979 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-12T17:19:59.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.0 2024-02-12T17:20:00.277 INFO:journalctl@ceph.osd.0.smithi033.stdout:Feb 12 17:20:00 smithi033 systemd[1]: Stopping Ceph osd.0 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:20:00.278 INFO:journalctl@ceph.osd.0.smithi033.stdout:Feb 12 17:20:00 smithi033 bash[225866]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-12T17:20:00.777 INFO:journalctl@ceph.osd.0.smithi033.stdout:Feb 12 17:20:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-0[49559]: debug 2024-02-12T17:20:00.316+0000 7f5ec5a0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:20:00.777 INFO:journalctl@ceph.osd.0.smithi033.stdout:Feb 12 17:20:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-0[49559]: debug 2024-02-12T17:20:00.316+0000 7f5ec5a0c700 -1 osd.0 776 *** Got signal Terminated *** 2024-02-12T17:20:00.777 INFO:journalctl@ceph.osd.0.smithi033.stdout:Feb 12 17:20:00 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-0[49559]: debug 2024-02-12T17:20:00.316+0000 7f5ec5a0c700 -1 osd.0 776 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:20:06.277 INFO:journalctl@ceph.osd.0.smithi033.stdout:Feb 12 17:20:05 smithi033 bash[225907]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-0 2024-02-12T17:20:07.874 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.0.service' 2024-02-12T17:20:07.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:20:07.927 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-12T17:20:07.928 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-12T17:20:07.928 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.1 2024-02-12T17:20:08.208 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:20:07 smithi033 systemd[1]: Stopping Ceph osd.1 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:20:08.208 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:20:07 smithi033 bash[226147]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-12T17:20:08.526 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:20:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[195731]: debug 2024-02-12T17:20:08.205+0000 7f2cb1527700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:20:08.526 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:20:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[195731]: debug 2024-02-12T17:20:08.205+0000 7f2cb1527700 -1 osd.1 776 *** Got signal Terminated *** 2024-02-12T17:20:08.527 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:20:08 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1[195731]: debug 2024-02-12T17:20:08.205+0000 7f2cb1527700 -1 osd.1 776 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:20:14.027 INFO:journalctl@ceph.osd.1.smithi033.stdout:Feb 12 17:20:13 smithi033 bash[226185]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-1 2024-02-12T17:20:15.657 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.1.service' 2024-02-12T17:20:15.705 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:20:15.705 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-12T17:20:15.706 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-12T17:20:15.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.2 2024-02-12T17:20:15.997 INFO:journalctl@ceph.osd.2.smithi033.stdout:Feb 12 17:20:15 smithi033 systemd[1]: Stopping Ceph osd.2 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:20:15.997 INFO:journalctl@ceph.osd.2.smithi033.stdout:Feb 12 17:20:15 smithi033 bash[226420]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-12T17:20:16.276 INFO:journalctl@ceph.osd.2.smithi033.stdout:Feb 12 17:20:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-2[58733]: debug 2024-02-12T17:20:15.994+0000 7fbd6d848700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:20:16.277 INFO:journalctl@ceph.osd.2.smithi033.stdout:Feb 12 17:20:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-2[58733]: debug 2024-02-12T17:20:15.994+0000 7fbd6d848700 -1 osd.2 776 *** Got signal Terminated *** 2024-02-12T17:20:16.277 INFO:journalctl@ceph.osd.2.smithi033.stdout:Feb 12 17:20:15 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-2[58733]: debug 2024-02-12T17:20:15.994+0000 7fbd6d848700 -1 osd.2 776 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:20:21.757 INFO:journalctl@ceph.osd.2.smithi033.stdout:Feb 12 17:20:21 smithi033 bash[226461]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-2 2024-02-12T17:20:23.633 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.2.service' 2024-02-12T17:20:23.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:20:23.679 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-12T17:20:23.679 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-12T17:20:23.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.3 2024-02-12T17:20:24.027 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 17:20:23 smithi033 systemd[1]: Stopping Ceph osd.3 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:20:24.027 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 17:20:23 smithi033 bash[226704]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-12T17:20:24.527 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 17:20:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-3[63188]: debug 2024-02-12T17:20:24.029+0000 7efd8f90c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:20:24.527 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 17:20:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-3[63188]: debug 2024-02-12T17:20:24.029+0000 7efd8f90c700 -1 osd.3 776 *** Got signal Terminated *** 2024-02-12T17:20:24.527 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 17:20:24 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-3[63188]: debug 2024-02-12T17:20:24.029+0000 7efd8f90c700 -1 osd.3 776 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:20:26.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:26 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:26.700+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:27.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:27.116+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:27.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:27 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:27.678+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:28.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:28.094+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:28.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:28.630+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:28.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:28 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:28.681+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:29.027 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 17:20:28 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-3[63188]: debug 2024-02-12T17:20:28.609+0000 7efd88881700 -1 osd.3 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:04.233149+0000 front 2024-02-12T17:20:04.233175+0000 (oldest deadline 2024-02-12T17:20:28.333309+0000) 2024-02-12T17:20:29.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:29.115+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:29.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:29.641+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:29.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:29 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:29.647+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:30.027 INFO:journalctl@ceph.osd.3.smithi033.stdout:Feb 12 17:20:29 smithi033 bash[226743]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-3 2024-02-12T17:20:30.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:30.112+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:30.977 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:30.617+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:30.977 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:30.637+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:31.239 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:31.119+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:31.240 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:30 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:30.975+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:04.833231+0000 front 2024-02-12T17:20:04.833370+0000 (oldest deadline 2024-02-12T17:20:30.733122+0000) 2024-02-12T17:20:31.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.3.service' 2024-02-12T17:20:31.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:20:31.889 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-12T17:20:31.889 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-12T17:20:31.889 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.4 2024-02-12T17:20:31.922 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:31.675+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:31.922 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:31 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:31.642+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:32.203 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:32.085+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:32.204 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:31 smithi094 systemd[1]: Stopping Ceph osd.4 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:20:32.204 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:31 smithi094 bash[136548]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-12T17:20:32.204 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:32.015+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:04.833231+0000 front 2024-02-12T17:20:04.833370+0000 (oldest deadline 2024-02-12T17:20:30.733122+0000) 2024-02-12T17:20:32.204 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:32.201+0000 7f2bb5b5f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:20:32.489 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:32.201+0000 7f2bb5b5f700 -1 osd.4 776 *** Got signal Terminated *** 2024-02-12T17:20:32.489 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:32.201+0000 7f2bb5b5f700 -1 osd.4 776 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:20:32.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:32.650+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:32.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:32 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:32.720+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:33.489 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:33.014+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:04.833231+0000 front 2024-02-12T17:20:04.833370+0000 (oldest deadline 2024-02-12T17:20:30.733122+0000) 2024-02-12T17:20:33.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:33.131+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:33.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:33.636+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:33.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:33.692+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:34.489 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:34.003+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:04.833231+0000 front 2024-02-12T17:20:04.833370+0000 (oldest deadline 2024-02-12T17:20:30.733122+0000) 2024-02-12T17:20:34.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:34.132+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:34.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:34.613+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:34.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:34.663+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:34.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:34 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:34.663+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:35.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:35.131+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:35.490 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:35.031+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:04.833231+0000 front 2024-02-12T17:20:04.833370+0000 (oldest deadline 2024-02-12T17:20:30.733122+0000) 2024-02-12T17:20:35.490 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:35.031+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.733686+0000 front 2024-02-12T17:20:10.733836+0000 (oldest deadline 2024-02-12T17:20:34.233566+0000) 2024-02-12T17:20:35.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:35.582+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:35.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:35.645+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:35.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:35 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:35.645+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:36.490 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:36.050+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:04.833231+0000 front 2024-02-12T17:20:04.833370+0000 (oldest deadline 2024-02-12T17:20:30.733122+0000) 2024-02-12T17:20:36.490 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:36.050+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.733686+0000 front 2024-02-12T17:20:10.733836+0000 (oldest deadline 2024-02-12T17:20:34.233566+0000) 2024-02-12T17:20:36.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:36.155+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:36.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:36.155+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:36.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:36.615+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:36.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:36.615+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:36.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:36.642+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:36.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:36 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:36.642+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:37.490 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:37.037+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:04.833231+0000 front 2024-02-12T17:20:04.833370+0000 (oldest deadline 2024-02-12T17:20:30.733122+0000) 2024-02-12T17:20:37.490 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4[45023]: debug 2024-02-12T17:20:37.037+0000 7f2baf2d5700 -1 osd.4 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.733686+0000 front 2024-02-12T17:20:10.733836+0000 (oldest deadline 2024-02-12T17:20:34.233566+0000) 2024-02-12T17:20:37.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:37.184+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:37.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:37.184+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:37.989 INFO:journalctl@ceph.osd.4.smithi094.stdout:Feb 12 17:20:37 smithi094 bash[136589]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-4 2024-02-12T17:20:37.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:37.590+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:37.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:37.590+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:37.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:37.603+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:37.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:37 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:37.603+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:38.449 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:38.172+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:38.449 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:38.172+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:38.740 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:38.568+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:38.740 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:38.568+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:38.740 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:38.596+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:38.740 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:38 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:38.596+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:39.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:39.207+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:39.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:39.207+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:39.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.4.service' 2024-02-12T17:20:39.610 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:20:39.610 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-12T17:20:39.610 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-12T17:20:39.610 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.5 2024-02-12T17:20:39.959 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:39.576+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:39.959 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:39.576+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:39.959 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:39.576+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:39.960 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:39.576+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:39.960 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:39 smithi094 systemd[1]: Stopping Ceph osd.5 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:20:39.960 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:39 smithi094 bash[136827]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-12T17:20:40.234 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:39.957+0000 7f941ff9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:20:40.234 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:39.957+0000 7f941ff9c700 -1 osd.5 776 *** Got signal Terminated *** 2024-02-12T17:20:40.234 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:39 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:39.957+0000 7f941ff9c700 -1 osd.5 776 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:20:40.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:40.232+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:40.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:40.232+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:40.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:40.575+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:40.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:40.575+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:40.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:40.577+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:40.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:40 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:40.577+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:41.539 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:41.258+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:41.539 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:41.258+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:41.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:41.604+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:41.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:41.604+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:41.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:41.538+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:41.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:41 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:41.538+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:42.568 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:42.264+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:42.568 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:42.264+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:42.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:42.566+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:42.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:42.566+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:42.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:42.604+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:42.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:42 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:42.604+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:43.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:43.219+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:43.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:43.219+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:43.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:43.219+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:43.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:43.583+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:43.990 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:43.583+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:43.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:43.561+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:43.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:43 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:43.561+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:44.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:44.216+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:44.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:44.216+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:44.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:44.216+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:44.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:44.609+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.453682+0000 front 2024-02-12T17:20:03.453781+0000 (oldest deadline 2024-02-12T17:20:25.753620+0000) 2024-02-12T17:20:44.989 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5[49660]: debug 2024-02-12T17:20:44.609+0000 7f941af15700 -1 osd.5 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.454713+0000 front 2024-02-12T17:20:10.454800+0000 (oldest deadline 2024-02-12T17:20:35.754674+0000) 2024-02-12T17:20:44.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:44.514+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:44.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:44.514+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:44.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:44 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:44.514+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:45.436 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:45.183+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:45.437 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:45.183+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:45.437 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:45.183+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:45.437 INFO:journalctl@ceph.osd.5.smithi094.stdout:Feb 12 17:20:45 smithi094 bash[136867]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-5 2024-02-12T17:20:45.991 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:45.489+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:45.991 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:45.489+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:45.991 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:45 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:45.489+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:46.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:46.208+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:46.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:46.208+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:46.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:46.208+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:46.860 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:46.517+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:46.860 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:46.517+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:46.860 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:46 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:46.517+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:47.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.5.service' 2024-02-12T17:20:47.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:20:47.193 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-12T17:20:47.193 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-12T17:20:47.194 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.6 2024-02-12T17:20:47.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:47.252+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:47.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:47.252+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:47.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:47.252+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:47.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:47 smithi094 systemd[1]: Stopping Ceph osd.6 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:20:47.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:47 smithi094 bash[137103]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-12T17:20:47.990 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:47.510+0000 7f1b9c428700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:20:47.990 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:47.510+0000 7f1b9c428700 -1 osd.6 776 *** Got signal Terminated *** 2024-02-12T17:20:47.990 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:47.510+0000 7f1b9c428700 -1 osd.6 776 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:20:47.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:47.494+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:47.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:47.494+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:47.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:47 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:47.494+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:48.489 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:48.224+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:48.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:48.224+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:48.490 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:48.224+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:48.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:48.536+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:48.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:48.536+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:48.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:48 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:48.536+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:49.740 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:49.270+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:49.740 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:49.270+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:49.740 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:49.270+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:49.740 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:49.510+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:49.740 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:49.510+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:49.741 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:49 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:49.510+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:50.551 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:50.250+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:50.552 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:50.250+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:50.552 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:50.250+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:50.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:50.550+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:50.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:50.550+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:50.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:50 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:50.550+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:51.740 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:51.296+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:51.740 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:51.296+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:51.740 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:51.296+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:51.740 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:51.523+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:51.740 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:51.523+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:51.740 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:51 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:51.523+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:52.520 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:52.263+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:01.415515+0000 front 2024-02-12T17:20:01.415534+0000 (oldest deadline 2024-02-12T17:20:26.115501+0000) 2024-02-12T17:20:52.520 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:52.263+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:10.216556+0000 front 2024-02-12T17:20:10.216702+0000 (oldest deadline 2024-02-12T17:20:36.116495+0000) 2024-02-12T17:20:52.520 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6[54079]: debug 2024-02-12T17:20:52.263+0000 7f1b993a5700 -1 osd.6 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:16.617458+0000 front 2024-02-12T17:20:16.617538+0000 (oldest deadline 2024-02-12T17:20:42.517499+0000) 2024-02-12T17:20:52.949 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:52.518+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:52.949 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:52.518+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:52.949 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:52 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:52.518+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:53.240 INFO:journalctl@ceph.osd.6.smithi094.stdout:Feb 12 17:20:53 smithi094 bash[137144]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-6 2024-02-12T17:20:53.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:53.566+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:53.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:53.566+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:53.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:53.566+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:53.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:53 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:53.566+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6815 osd.3 since back 2024-02-12T17:20:26.703201+0000 front 2024-02-12T17:20:26.703167+0000 (oldest deadline 2024-02-12T17:20:52.603094+0000) 2024-02-12T17:20:54.699 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.6.service' 2024-02-12T17:20:54.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:20:54.739 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-12T17:20:54.739 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-12T17:20:54.739 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.7 2024-02-12T17:20:54.833 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:54.564+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:54.833 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:54.564+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:54.833 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:54.564+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:54.833 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:54 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:54.564+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6815 osd.3 since back 2024-02-12T17:20:26.703201+0000 front 2024-02-12T17:20:26.703167+0000 (oldest deadline 2024-02-12T17:20:52.603094+0000) 2024-02-12T17:20:54.833 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:54 smithi094 systemd[1]: Stopping Ceph osd.7 for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:20:55.240 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:54 smithi094 bash[137381]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-12T17:20:55.240 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:55.064+0000 7f504c903700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:20:55.240 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:55.064+0000 7f504c903700 -1 osd.7 776 *** Got signal Terminated *** 2024-02-12T17:20:55.240 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:55.064+0000 7f504c903700 -1 osd.7 776 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T17:20:55.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:55.557+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:55.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:55.557+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:55.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:55.557+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:55.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:55 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:55.557+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6815 osd.3 since back 2024-02-12T17:20:26.703201+0000 front 2024-02-12T17:20:26.703167+0000 (oldest deadline 2024-02-12T17:20:52.603094+0000) 2024-02-12T17:20:56.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:56.560+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:56.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:56.560+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:56.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:56.560+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:56.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:56 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:56.560+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6815 osd.3 since back 2024-02-12T17:20:26.703201+0000 front 2024-02-12T17:20:26.703167+0000 (oldest deadline 2024-02-12T17:20:52.603094+0000) 2024-02-12T17:20:57.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:57.594+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:57.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:57.594+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:57.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:57.594+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:57.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:57 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:57.594+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6815 osd.3 since back 2024-02-12T17:20:26.703201+0000 front 2024-02-12T17:20:26.703167+0000 (oldest deadline 2024-02-12T17:20:52.603094+0000) 2024-02-12T17:20:58.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:58.615+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:58.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:58.615+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:58.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:58.615+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:58.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:58.615+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6815 osd.3 since back 2024-02-12T17:20:26.703201+0000 front 2024-02-12T17:20:26.703167+0000 (oldest deadline 2024-02-12T17:20:52.603094+0000) 2024-02-12T17:20:58.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:58 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:58.615+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.94:6802 osd.4 since back 2024-02-12T17:20:33.104014+0000 front 2024-02-12T17:20:33.104047+0000 (oldest deadline 2024-02-12T17:20:58.304433+0000) 2024-02-12T17:20:59.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:59.584+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6803 osd.0 since back 2024-02-12T17:20:03.899702+0000 front 2024-02-12T17:20:03.899795+0000 (oldest deadline 2024-02-12T17:20:28.600050+0000) 2024-02-12T17:20:59.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:59.584+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6807 osd.1 since back 2024-02-12T17:20:08.600701+0000 front 2024-02-12T17:20:08.600833+0000 (oldest deadline 2024-02-12T17:20:33.900741+0000) 2024-02-12T17:20:59.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:59.584+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6811 osd.2 since back 2024-02-12T17:20:19.801717+0000 front 2024-02-12T17:20:19.801802+0000 (oldest deadline 2024-02-12T17:20:43.901891+0000) 2024-02-12T17:20:59.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:59.584+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.33:6815 osd.3 since back 2024-02-12T17:20:26.703201+0000 front 2024-02-12T17:20:26.703167+0000 (oldest deadline 2024-02-12T17:20:52.603094+0000) 2024-02-12T17:20:59.990 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:20:59 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7[58712]: debug 2024-02-12T17:20:59.584+0000 7f5045878700 -1 osd.7 776 heartbeat_check: no reply from 172.21.15.94:6802 osd.4 since back 2024-02-12T17:20:33.104014+0000 front 2024-02-12T17:20:33.104047+0000 (oldest deadline 2024-02-12T17:20:58.304433+0000) 2024-02-12T17:21:00.989 INFO:journalctl@ceph.osd.7.smithi094.stdout:Feb 12 17:21:00 smithi094 bash[137421]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-osd-7 2024-02-12T17:21:02.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@osd.7.service' 2024-02-12T17:21:02.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:21:02.341 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-12T17:21:02.342 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-12T17:21:02.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@rgw.foo.a 2024-02-12T17:21:02.639 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:21:02 smithi033 systemd[1]: Stopping Ceph rgw.foo.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:21:02.639 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:21:02 smithi033 bash[226983]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw.foo.a" found: no such container 2024-02-12T17:21:02.639 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:21:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw-foo-a[65833]: debug 2024-02-12T17:21:02.636+0000 7efdb3ce3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T17:21:02.639 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:21:02 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw-foo-a[65833]: debug 2024-02-12T17:21:02.636+0000 7efdcadad880 -1 shutting down 2024-02-12T17:21:12.027 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 17:21:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=warn ts=2024-02-12T17:21:11.624Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi033.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.33:8443: connect: connection refused" 2024-02-12T17:21:12.027 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 17:21:11 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=warn ts=2024-02-12T17:21:11.625Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi094.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.94:8443: connect: connection refused" 2024-02-12T17:21:13.027 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:21:12 smithi033 bash[227023]: time="2024-02-12T17:21:12Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-12T17:21:13.527 INFO:journalctl@ceph.rgw.foo.a.smithi033.stdout:Feb 12 17:21:13 smithi033 bash[227023]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-rgw-foo-a 2024-02-12T17:21:13.733 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@rgw.foo.a.service' 2024-02-12T17:21:13.774 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:21:13.774 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-12T17:21:13.774 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-12T17:21:13.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@prometheus.a 2024-02-12T17:21:14.077 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:13 smithi094 systemd[1]: Stopping Ceph prometheus.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:21:14.077 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:13 smithi094 bash[137661]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-12T17:21:14.347 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-12T17:21:14.347 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-12T17:21:14.347 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-12T17:21:14.347 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-12T17:21:14.347 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-12T17:21:14.347 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-12T17:21:14.347 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-12T17:21:14.348 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-12T17:21:14.348 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.076Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-12T17:21:14.348 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.078Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-12T17:21:14.348 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.078Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-12T17:21:14.348 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a[64668]: ts=2024-02-12T17:21:14.078Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-12T17:21:14.740 INFO:journalctl@ceph.prometheus.a.smithi094.stdout:Feb 12 17:21:14 smithi094 bash[137701]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-prometheus-a 2024-02-12T17:21:14.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@prometheus.a.service' 2024-02-12T17:21:14.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T17:21:14.858 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-12T17:21:14.858 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-12T17:21:16.406 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 17:21:16 smithi033 systemd[1]: Stopping Ceph alertmanager.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:21:16.407 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 17:21:16 smithi033 bash[227447]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-12T17:21:16.709 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 17:21:16 smithi033 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a[72425]: level=info ts=2024-02-12T17:21:16.405Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-12T17:21:17.027 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 17:21:16 smithi033 bash[227487]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-alertmanager-a 2024-02-12T17:21:17.777 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 17:21:17 smithi033 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-12T17:21:17.777 INFO:journalctl@ceph.alertmanager.a.smithi033.stdout:Feb 12 17:21:17 smithi033 systemd[1]: Stopped Ceph alertmanager.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T17:21:17.778 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 17:21:17 smithi033 systemd[1]: Stopping Ceph node-exporter.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:21:17.778 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 17:21:17 smithi033 bash[227590]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter.a" found: no such container 2024-02-12T17:21:18.527 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 17:21:18 smithi033 bash[227630]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-a 2024-02-12T17:21:18.851 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 17:21:18 smithi033 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-12T17:21:18.852 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 17:21:18 smithi033 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-12T17:21:18.852 INFO:journalctl@ceph.node-exporter.a.smithi033.stdout:Feb 12 17:21:18 smithi033 systemd[1]: Stopped Ceph node-exporter.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T17:21:19.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-12T17:21:20.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:20 smithi094 systemd[1]: Stopping Ceph iscsi.iscsi.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:21:20.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:20 smithi094 bash[138065]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi.iscsi.a" found: no such container 2024-02-12T17:21:20.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:20 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a[62513]: debug Shutdown received 2024-02-12T17:21:30.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:30 smithi094 bash[138103]: time="2024-02-12T17:21:30Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-12T17:21:31.111 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:30 smithi094 bash[138103]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a 2024-02-12T17:21:31.410 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:31 smithi094 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-12T17:21:31.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:31 smithi094 bash[138176]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-iscsi-iscsi-a-tcmu 2024-02-12T17:21:31.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:31 smithi094 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-12T17:21:31.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi094.stdout:Feb 12 17:21:31 smithi094 systemd[1]: Stopped Ceph iscsi.iscsi.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T17:21:31.990 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 17:21:31 smithi094 systemd[1]: Stopping Ceph node-exporter.b for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:21:31.990 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 17:21:31 smithi094 bash[138252]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter.b" found: no such container 2024-02-12T17:21:32.741 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 17:21:32 smithi094 bash[138294]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-node-exporter-b 2024-02-12T17:21:32.741 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 17:21:32 smithi094 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-12T17:21:33.240 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 17:21:32 smithi094 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-12T17:21:33.240 INFO:journalctl@ceph.node-exporter.b.smithi094.stdout:Feb 12 17:21:32 smithi094 systemd[1]: Stopped Ceph node-exporter.b for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T17:21:33.241 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 17:21:33 smithi094 systemd[1]: Stopping Ceph grafana.a for cf6ad642-c9c5-11ee-95b9-87774f69a715... 2024-02-12T17:21:33.241 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 17:21:33 smithi094 bash[138430]: Error: no container with name or ID "ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana.a" found: no such container 2024-02-12T17:21:33.595 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 17:21:33 smithi094 ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a[65039]: t=2024-02-12T17:21:33+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-12T17:21:33.955 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 17:21:33 smithi094 bash[138468]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715-grafana-a 2024-02-12T17:21:34.223 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 17:21:33 smithi094 systemd[1]: ceph-cf6ad642-c9c5-11ee-95b9-87774f69a715@grafana.a.service: Succeeded. 2024-02-12T17:21:34.223 INFO:journalctl@ceph.grafana.a.smithi094.stdout:Feb 12 17:21:33 smithi094 systemd[1]: Stopped Ceph grafana.a for cf6ad642-c9c5-11ee-95b9-87774f69a715. 2024-02-12T17:21:34.362 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T17:21:34.402 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T17:21:34.433 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-12T17:21:34.436 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705/remote/ubuntu@smithi033.front.sepia.ceph.com/crash 2024-02-12T17:21:34.437 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/crash -- . 2024-02-12T17:21:34.497 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-12T17:21:34.497 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2024-02-12T17:21:34.499 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705/remote/ubuntu@smithi094.front.sepia.ceph.com/crash 2024-02-12T17:21:34.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/crash -- . 2024-02-12T17:21:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:tar: /var/lib/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-12T17:21:34.556 INFO:teuthology.orchestra.run.smithi094.stderr:tar: Error is not recoverable: exiting now 2024-02-12T17:21:34.556 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-12T17:21:34.557 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | head -n 1 2024-02-12T17:21:34.610 INFO:teuthology.orchestra.run.smithi033.stdout:2024-02-12T16:46:05.953472+0000 mon.a (mon.0) 162 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-12T17:21:34.613 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-12T17:21:34.613 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[SEC\]' /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | head -n 1 2024-02-12T17:21:34.647 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]' /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | head -n 1 2024-02-12T17:21:34.722 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[WRN\]' /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | head -n 1 2024-02-12T17:21:34.793 INFO:teuthology.orchestra.run.smithi033.stdout:2024-02-12T16:46:05.953472+0000 mon.a (mon.0) 162 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-12T17:21:34.795 INFO:tasks.cephadm:Compressing logs... 2024-02-12T17:21:34.795 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-12T17:21:34.838 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-12T17:21:34.869 INFO:teuthology.orchestra.run.smithi033.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-12T17:21:34.869 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-12T17:21:34.883 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-12T17:21:34.883 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.2.log 2024-02-12T17:21:34.883 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mgr.y.log 2024-02-12T17:21:34.883 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-volume.log: /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-12T17:21:34.883 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-12T17:21:34.883 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.3.log 2024-02-12T17:21:34.883 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.0.log 2024-02-12T17:21:34.883 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-12T17:21:34.884 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/objectstore_tool.7.log 2024-02-12T17:21:34.884 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.a.log 2024-02-12T17:21:34.884 INFO:teuthology.orchestra.run.smithi033.stderr: 61.8% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-12T17:21:34.884 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.c.log 2024-02-12T17:21:34.884 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log 2024-02-12T17:21:34.884 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.a.log: /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.1.log 2024-02-12T17:21:34.885 INFO:teuthology.orchestra.run.smithi094.stderr:find: ‘/var/log/rbd-target-api’gzip -5 : No such file or directory--verbose 2024-02-12T17:21:34.885 INFO:teuthology.orchestra.run.smithi094.stderr:-- /var/log/ceph/cephadm.log 2024-02-12T17:21:34.889 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.6.log 2024-02-12T17:21:34.889 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-12T17:21:34.889 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-12T17:21:34.889 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.6.log: /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-12T17:21:34.889 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.7.log 2024-02-12T17:21:34.892 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.audit.log: /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mgr.x.log 2024-02-12T17:21:34.892 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.b.log 2024-02-12T17:21:34.892 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log 2024-02-12T17:21:34.892 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.4.log 2024-02-12T17:21:34.892 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.5.log 2024-02-12T17:21:34.895 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/tcmu-runner.log 2024-02-12T17:21:34.895 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.5.log: /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/tcmu-runner.log: 67.5% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/tcmu-runner.log.gz 2024-02-12T17:21:34.895 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log: /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.1.log: 88.8% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-12T17:21:34.896 INFO:teuthology.orchestra.run.smithi033.stderr: 92.4% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-12T17:21:34.912 INFO:teuthology.orchestra.run.smithi094.stderr: 81.1% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-12T17:21:34.941 INFO:teuthology.orchestra.run.smithi094.stderr: 92.1% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mgr.x.log.gz 2024-02-12T17:21:34.957 INFO:teuthology.orchestra.run.smithi094.stderr: 92.1% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-12T17:21:34.959 INFO:teuthology.orchestra.run.smithi094.stderr: 90.0% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-12T17:21:34.964 INFO:teuthology.orchestra.run.smithi094.stderr: 86.4% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-12T17:21:34.977 INFO:teuthology.orchestra.run.smithi033.stderr: 94.0% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-12T17:21:35.003 INFO:teuthology.orchestra.run.smithi094.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-12T17:21:35.132 INFO:teuthology.orchestra.run.smithi033.stderr: 92.1% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-12T17:21:36.282 INFO:teuthology.orchestra.run.smithi033.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-12T17:21:36.500 INFO:teuthology.orchestra.run.smithi033.stderr: 95.1% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/objectstore_tool.7.log.gz 2024-02-12T17:21:36.753 INFO:teuthology.orchestra.run.smithi033.stderr: 89.2% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mgr.y.log.gz 2024-02-12T17:21:37.054 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.b.log.gz 2024-02-12T17:21:37.832 INFO:teuthology.orchestra.run.smithi033.stderr: 90.9% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.c.log.gz 2024-02-12T17:21:39.297 INFO:teuthology.orchestra.run.smithi033.stderr: 90.1% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-mon.a.log.gz 2024-02-12T17:22:50.690 INFO:teuthology.orchestra.run.smithi033.stderr: 93.5% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.1.log.gz 2024-02-12T17:23:10.260 INFO:teuthology.orchestra.run.smithi094.stderr: 93.6% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.4.log.gz 2024-02-12T17:23:51.171 INFO:teuthology.orchestra.run.smithi094.stderr: 92.6% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.6.log.gz 2024-02-12T17:24:34.318 INFO:teuthology.orchestra.run.smithi094.stderr: 93.3% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.7.log.gz 2024-02-12T17:24:49.028 INFO:teuthology.orchestra.run.smithi033.stderr: 93.0% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.3.log.gz 2024-02-12T17:24:57.346 INFO:teuthology.orchestra.run.smithi094.stderr: 92.8% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.5.log.gz 2024-02-12T17:24:57.348 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-02-12T17:24:57.349 INFO:teuthology.orchestra.run.smithi094.stderr:real 3m22.492s 2024-02-12T17:24:57.349 INFO:teuthology.orchestra.run.smithi094.stderr:user 5m3.776s 2024-02-12T17:24:57.349 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m12.227s 2024-02-12T17:25:00.804 INFO:teuthology.orchestra.run.smithi033.stderr: 92.4% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.2.log.gz 2024-02-12T17:25:02.132 INFO:teuthology.orchestra.run.smithi033.stderr: 92.6% -- replaced with /var/log/ceph/cf6ad642-c9c5-11ee-95b9-87774f69a715/ceph-osd.0.log.gz 2024-02-12T17:25:02.134 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-02-12T17:25:02.134 INFO:teuthology.orchestra.run.smithi033.stderr:real 3m27.280s 2024-02-12T17:25:02.134 INFO:teuthology.orchestra.run.smithi033.stderr:user 5m38.952s 2024-02-12T17:25:02.134 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m12.252s 2024-02-12T17:25:02.134 INFO:tasks.cephadm:Archiving logs... 2024-02-12T17:25:02.135 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705/remote/ubuntu@smithi033.front.sepia.ceph.com/log 2024-02-12T17:25:02.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-12T17:25:53.579 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705/remote/ubuntu@smithi094.front.sepia.ceph.com/log 2024-02-12T17:25:53.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-12T17:26:40.397 INFO:tasks.cephadm:Removing cluster... 2024-02-12T17:26:40.397 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 --force 2024-02-12T17:26:41.460 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf6ad642-c9c5-11ee-95b9-87774f69a715 --force 2024-02-12T17:26:42.366 INFO:tasks.cephadm:Removing cephadm ... 2024-02-12T17:26:42.367 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-12T17:26:42.390 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-12T17:26:42.408 INFO:tasks.cephadm:Teardown complete 2024-02-12T17:26:42.408 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-12T17:26:42.418 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-12T17:26:42.418 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-12T17:26:42.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-12T17:26:42.526 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-12T17:26:42.535 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-12T17:26:42.535 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T17:26:42.538 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T17:26:42.555 INFO:teuthology.orchestra.run.smithi094.stderr:bash: ntpq: command not found 2024-02-12T17:26:42.555 INFO:teuthology.orchestra.run.smithi033.stderr:bash: ntpq: command not found 2024-02-12T17:26:42.570 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-12T17:26:42.570 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-02-12T17:26:42.570 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 -1602us[-1602us] +/- 92ms 2024-02-12T17:26:42.570 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 +731us[ +752us] +/- 51ms 2024-02-12T17:26:42.570 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +272us[ +292us] +/- 68ms 2024-02-12T17:26:42.570 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T17:26:42.589 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-12T17:26:42.589 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-02-12T17:26:42.589 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 -1613us[-1608us] +/- 92ms 2024-02-12T17:26:42.589 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 +726us[ +731us] +/- 51ms 2024-02-12T17:26:42.589 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +233us[ +239us] +/- 68ms 2024-02-12T17:26:42.589 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T17:26:42.590 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-12T17:26:42.598 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-12T17:26:42.599 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-12T17:26:42.627 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-12T17:26:42.813 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-12T17:26:42.981 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-12T17:26:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T17:26:43.018 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-12T17:26:43.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T17:26:43.050 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-12T17:26:43.059 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-12T17:26:43.102 INFO:teuthology.task.internal:Duration was 3194.301984 seconds 2024-02-12T17:26:43.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-12T17:26:43.135 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-12T17:26:43.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-12T17:26:43.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-12T17:26:43.236 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-12T17:26:43.251 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-12T17:26:43.515 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-12T17:26:43.516 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-02-12T17:26:43.516 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-12T17:26:43.558 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-02-12T17:26:43.559 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-12T17:26:43.600 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-12T17:26:43.600 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-12T17:26:43.603 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-12T17:26:43.794 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-12T17:26:43.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-12T17:26:43.797 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-12T17:26:44.163 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-12T17:26:44.173 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-12T17:26:44.174 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-12T17:26:44.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-12T17:26:44.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-12T17:26:44.289 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:26:44.292 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:26:44.323 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-02-12T17:26:44.328 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-02-12T17:26:44.343 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:26:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T17:26:44.395 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:26:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T17:26:44.414 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-12T17:26:44.424 INFO:teuthology.task.internal:Transferring archived files... 2024-02-12T17:26:44.425 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705/remote/smithi033 2024-02-12T17:26:44.426 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-12T17:26:44.507 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705/remote/smithi094 2024-02-12T17:26:44.508 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-12T17:26:44.563 INFO:teuthology.task.internal:Removing archive directory... 2024-02-12T17:26:44.563 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-12T17:26:44.566 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-12T17:26:44.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-12T17:26:44.621 INFO:teuthology.task.internal:Not uploading archives. 2024-02-12T17:26:44.622 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-12T17:26:44.662 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-12T17:26:44.662 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-12T17:26:44.664 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-12T17:26:44.680 INFO:teuthology.orchestra.run.smithi033.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 17:26 /home/ubuntu/cephtest 2024-02-12T17:26:44.682 INFO:teuthology.orchestra.run.smithi094.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 17:26 /home/ubuntu/cephtest 2024-02-12T17:26:44.685 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-12T17:26:44.703 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-12T17:26:44.770 INFO:teuthology.nuke:Checking targets against current locks 2024-02-12T17:26:44.797 DEBUG:teuthology.nuke:shortname: smithi033 2024-02-12T17:26:44.798 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T17:26:44.824 DEBUG:teuthology.nuke:shortname: smithi094 2024-02-12T17:26:44.825 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T17:26:44.828 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 16:25:10.148164', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGNBgCOZxJx7VBzwEveLje+ynBZm42SIb2RE2Ir7sdFsc6o+jF9U86hq0QVYFO5P4erO08mcMnu0BKROB+gAIV8='} 2024-02-12T17:26:44.852 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556705', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 16:25:10.146465', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoot6WVbbPiPGVwo1bMdxjp4plJ1VsA3lBdEWR3Ar2pngRTH+vnPN00VB8rjh/eAQUbaEXdJ6Zn56XVs9YpSw0='} 2024-02-12T17:26:44.884 INFO:teuthology.orchestra.console.smithi033:Power off 2024-02-12T17:26:44.885 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-12T17:26:44.900 INFO:teuthology.orchestra.console.smithi094:Power off 2024-02-12T17:26:44.901 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-12T17:26:44.911 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2024-02-12T17:26:44.911 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:26:44.925 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2024-02-12T17:26:44.926 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:26:44.935 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-02-12T17:26:44.949 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-02-12T17:26:48.936 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:26:48.950 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:26:49.050 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-02-12T17:26:49.062 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-02-12T17:26:53.055 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:26:53.063 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:26:53.165 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-02-12T17:26:53.171 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2024-02-12T17:26:53.272 INFO:teuthology.orchestra.console.smithi094:Power off completed 2024-02-12T17:26:57.172 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:26:57.286 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2024-02-12T17:26:57.387 INFO:teuthology.orchestra.console.smithi033:Power off completed 2024-02-12T17:26:57.491 INFO:teuthology.run:Summary data: description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3194.30198431015 failure_reason: '"2024-02-12T16:46:05.953472+0000 mon.a (mon.0) 162 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-02-12T17:26:57.492 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-12T17:26:57.607 INFO:teuthology.run:FAIL